[
  {
    "path": ".coveragerc",
    "content": "[run]\nsource = teletext\n"
  },
  {
    "path": ".gitignore",
    "content": "/.idea\n/.coverage\n/venv\n/*.vbi\n*.pyc\n/build\n/dist\n/MANIFEST\n/*.egg-info\n/vbi\n/teletext/vbi/data/*-*\n\n"
  },
  {
    "path": "HOW_IT_WORKS.md",
    "content": "HOW IT WORKS\n------------\n\nTeletext is encoded as a non-return-to-zero signal with two levels representing\none and zero. This is a fancy way of saying that a line of teletext data is\na sequence of black and white \"pixels\" in the TV signal. Of course, since the\nsignal is analogue there are no individual pixels, the signal is continuous.\nBut you can imagine that there are pixels in the idealized \"perfect\" signal.\n\nThe problem of decoding teletext from a VHS recording is that VHS bandwidth is\nlower than teletext bandwidth. This means that the signal is effectively low\npass filtered, which in terms of an image is equivalent to gaussian blurring.\n\nThere are methods for reversing gaussian blur, but they are designed to work\nwith general image data. In the case of teletext we only have black or white\nlevels, so these methods are not optimal. We can exploit the limitations on\nthe input in order to get a better result. We can also exploit information\nabout the protocol to further improve efficiency and accuracy.\n\nWhen the black and white signal is blurred, the individual pixels are blurred \nin to each other. This makes the signal unreadable using normal methods, because\ninstead of a clean sequence like \"1010\" you something close to \"0.5 0.5 0.5 0.5\".\nBut all is not lost, because a sequence like \"1111\" or \"0000\" will be the same\nafter blurring. So if you see a signal like \"0.5 0.7 1.0 1.0\" you can guess that\nthe original was probably \"0 1 1 1\" or \"0 0 1 1\".  \n\nThere are 45 bytes in each teletext line, so the space of possible guesses is\n2^(45*8) which is a very big number, which makes trying every guess completely\nimpractical. However there are ways to reduce this number:\n\nFOUR RULES\n----------\n\n1. Nearly all bytes have a parity bit which means there are only 128 possible\ncombinations instead of 256.\n\n2. Some bytes are hamming encoded. These have even fewer possible combinations.\n\n3. The first three bytes in the signal are always the same. We can use this\nto find the start of the signal in the sample data (it moves a bit in each\nline, but the width is always the same.)\n\n4. The protocol itself defines rules about which bytes are allowed in which\npositions, reducing the problem space further.\n\n\nTRAINING\n--------\n\nA known signal is recorded to a tape using a Raspberry Pi with rpi-teletext.\nThis signal is played back into the computer, which builds a table of convolved\n-> original sequences.\n\n\nDECONVOLUTION\n-------------\n\nThe convolved training data can be compared to recorded tapes in order to determine\nwhat the data originally was. The line is first resampled to 1 sample per \"bit\". \nThen it is divided into \"bytes\". Each one is compared against the training\ntables, including a few bits before and after. The closest match is the most\nlikely original signal.\n\nThis algorithm can be performed in parallel using CUDA or OpenCL. This allows\ndeconvolution to run in near realtime with a GTX 780.\n\nSee TRAINING.md for more.\n\n\nSQUASHING\n---------\n\nThe algorithm outputs lots of teletext packets, but they will still not be\nperfect (even though they may be valid, they aren't necessarily correct.)\n\nSince the teletext pages are broadcast on a loop, any recording of more than a\nfew minutes will have multiple copies of every packet. This means, if two packets\nare received that only differ at a couple of bytes, they can be assumed to be the same.\n\nThe stream is first \"paginated\", ie split in to subpages.\n\nAll versions of the same subpage are compared, and for each byte, the most\nfrequent decoding is used so for example if you had these inputs:\n\nHELLO\nHELLP\nMELLO\n\nThen the result \"HELLO\" would be decoded, since those are the most frequent bytes\nin this position. For this to work well, you need a lot of copies of every packet.\n"
  },
  {
    "path": "LICENSE",
    "content": "                    GNU GENERAL PUBLIC LICENSE\n                       Version 3, 29 June 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>\n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n                            Preamble\n\n  The GNU General Public License is a free, copyleft license for\nsoftware and other kinds of works.\n\n  The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works.  By contrast,\nthe GNU General Public License is intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.  We, the Free Software Foundation, use the\nGNU General Public License for most of our software; it applies also to\nany other work released this way by its authors.  You can apply it to\nyour programs, too.\n\n  When we speak of free software, we are referring to freedom, not\nprice.  Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n  To protect your rights, we need to prevent others from denying you\nthese rights or asking you to surrender the rights.  Therefore, you have\ncertain responsibilities if you distribute copies of the software, or if\nyou modify it: responsibilities to respect the freedom of others.\n\n  For example, if you distribute copies of such a program, whether\ngratis or for a fee, you must pass on to the recipients the same\nfreedoms that you received.  You must make sure that they, too, receive\nor can get the source code.  And you must show them these terms so they\nknow their rights.\n\n  Developers that use the GNU GPL protect your rights with two steps:\n(1) assert copyright on the software, and (2) offer you this License\ngiving you legal permission to copy, distribute and/or modify it.\n\n  For the developers' and authors' protection, the GPL clearly explains\nthat there is no warranty for this free software.  For both users' and\nauthors' sake, the GPL requires that modified versions be marked as\nchanged, so that their problems will not be attributed erroneously to\nauthors of previous versions.\n\n  Some devices are designed to deny users access to install or run\nmodified versions of the software inside them, although the manufacturer\ncan do so.  This is fundamentally incompatible with the aim of\nprotecting users' freedom to change the software.  The systematic\npattern of such abuse occurs in the area of products for individuals to\nuse, which is precisely where it is most unacceptable.  Therefore, we\nhave designed this version of the GPL to prohibit the practice for those\nproducts.  If such problems arise substantially in other domains, we\nstand ready to extend this provision to those domains in future versions\nof the GPL, as needed to protect the freedom of users.\n\n  Finally, every program is threatened constantly by software patents.\nStates should not allow patents to restrict development and use of\nsoftware on general-purpose computers, but in those that do, we wish to\navoid the special danger that patents applied to a free program could\nmake it effectively proprietary.  To prevent this, the GPL assures that\npatents cannot be used to render the program non-free.\n\n  The precise terms and conditions for copying, distribution and\nmodification follow.\n\n                       TERMS AND CONDITIONS\n\n  0. Definitions.\n\n  \"This License\" refers to version 3 of the GNU General Public License.\n\n  \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n  \"The Program\" refers to any copyrightable work licensed under this\nLicense.  Each licensee is addressed as \"you\".  \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n  To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy.  The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n  A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n  To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy.  Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n  To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies.  Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n  An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License.  If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n  1. Source Code.\n\n  The \"source code\" for a work means the preferred form of the work\nfor making modifications to it.  \"Object code\" means any non-source\nform of a work.\n\n  A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n  The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form.  A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n  The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities.  However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work.  For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n  The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n  The Corresponding Source for a work in source code form is that\nsame work.\n\n  2. Basic Permissions.\n\n  All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met.  This License explicitly affirms your unlimited\npermission to run the unmodified Program.  The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work.  This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n  You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force.  You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright.  Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n  Conveying under any other circumstances is permitted solely under\nthe conditions stated below.  Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n  3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n  No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n  When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n  4. Conveying Verbatim Copies.\n\n  You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n  You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n  5. Conveying Modified Source Versions.\n\n  You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n    a) The work must carry prominent notices stating that you modified\n    it, and giving a relevant date.\n\n    b) The work must carry prominent notices stating that it is\n    released under this License and any conditions added under section\n    7.  This requirement modifies the requirement in section 4 to\n    \"keep intact all notices\".\n\n    c) You must license the entire work, as a whole, under this\n    License to anyone who comes into possession of a copy.  This\n    License will therefore apply, along with any applicable section 7\n    additional terms, to the whole of the work, and all its parts,\n    regardless of how they are packaged.  This License gives no\n    permission to license the work in any other way, but it does not\n    invalidate such permission if you have separately received it.\n\n    d) If the work has interactive user interfaces, each must display\n    Appropriate Legal Notices; however, if the Program has interactive\n    interfaces that do not display Appropriate Legal Notices, your\n    work need not make them do so.\n\n  A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit.  Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n  6. Conveying Non-Source Forms.\n\n  You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n    a) Convey the object code in, or embodied in, a physical product\n    (including a physical distribution medium), accompanied by the\n    Corresponding Source fixed on a durable physical medium\n    customarily used for software interchange.\n\n    b) Convey the object code in, or embodied in, a physical product\n    (including a physical distribution medium), accompanied by a\n    written offer, valid for at least three years and valid for as\n    long as you offer spare parts or customer support for that product\n    model, to give anyone who possesses the object code either (1) a\n    copy of the Corresponding Source for all the software in the\n    product that is covered by this License, on a durable physical\n    medium customarily used for software interchange, for a price no\n    more than your reasonable cost of physically performing this\n    conveying of source, or (2) access to copy the\n    Corresponding Source from a network server at no charge.\n\n    c) Convey individual copies of the object code with a copy of the\n    written offer to provide the Corresponding Source.  This\n    alternative is allowed only occasionally and noncommercially, and\n    only if you received the object code with such an offer, in accord\n    with subsection 6b.\n\n    d) Convey the object code by offering access from a designated\n    place (gratis or for a charge), and offer equivalent access to the\n    Corresponding Source in the same way through the same place at no\n    further charge.  You need not require recipients to copy the\n    Corresponding Source along with the object code.  If the place to\n    copy the object code is a network server, the Corresponding Source\n    may be on a different server (operated by you or a third party)\n    that supports equivalent copying facilities, provided you maintain\n    clear directions next to the object code saying where to find the\n    Corresponding Source.  Regardless of what server hosts the\n    Corresponding Source, you remain obligated to ensure that it is\n    available for as long as needed to satisfy these requirements.\n\n    e) Convey the object code using peer-to-peer transmission, provided\n    you inform other peers where the object code and Corresponding\n    Source of the work are being offered to the general public at no\n    charge under subsection 6d.\n\n  A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n  A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling.  In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage.  For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product.  A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n  \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source.  The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n  If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information.  But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n  The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed.  Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n  Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n  7. Additional Terms.\n\n  \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law.  If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n  When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit.  (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.)  You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n  Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n    a) Disclaiming warranty or limiting liability differently from the\n    terms of sections 15 and 16 of this License; or\n\n    b) Requiring preservation of specified reasonable legal notices or\n    author attributions in that material or in the Appropriate Legal\n    Notices displayed by works containing it; or\n\n    c) Prohibiting misrepresentation of the origin of that material, or\n    requiring that modified versions of such material be marked in\n    reasonable ways as different from the original version; or\n\n    d) Limiting the use for publicity purposes of names of licensors or\n    authors of the material; or\n\n    e) Declining to grant rights under trademark law for use of some\n    trade names, trademarks, or service marks; or\n\n    f) Requiring indemnification of licensors and authors of that\n    material by anyone who conveys the material (or modified versions of\n    it) with contractual assumptions of liability to the recipient, for\n    any liability that these contractual assumptions directly impose on\n    those licensors and authors.\n\n  All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10.  If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term.  If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n  If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n  Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n  8. Termination.\n\n  You may not propagate or modify a covered work except as expressly\nprovided under this License.  Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n  However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n  Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n  Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License.  If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n  9. Acceptance Not Required for Having Copies.\n\n  You are not required to accept this License in order to receive or\nrun a copy of the Program.  Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance.  However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work.  These actions infringe copyright if you do\nnot accept this License.  Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n  10. Automatic Licensing of Downstream Recipients.\n\n  Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License.  You are not responsible\nfor enforcing compliance by third parties with this License.\n\n  An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations.  If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n  You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License.  For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n  11. Patents.\n\n  A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based.  The\nwork thus licensed is called the contributor's \"contributor version\".\n\n  A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version.  For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n  Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n  In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement).  To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n  If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients.  \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n  If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n  A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License.  You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n  Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n  12. No Surrender of Others' Freedom.\n\n  If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License.  If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all.  For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n  13. Use with the GNU Affero General Public License.\n\n  Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU Affero General Public License into a single\ncombined work, and to convey the resulting work.  The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the special requirements of the GNU Affero General Public License,\nsection 13, concerning interaction through a network will apply to the\ncombination as such.\n\n  14. Revised Versions of this License.\n\n  The Free Software Foundation may publish revised and/or new versions of\nthe GNU General Public License from time to time.  Such new versions will\nbe similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n  Each version is given a distinguishing version number.  If the\nProgram specifies that a certain numbered version of the GNU General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation.  If the Program does not specify a version number of the\nGNU General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n  If the Program specifies that a proxy can decide which future\nversions of the GNU General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n  Later license versions may give you additional or different\npermissions.  However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n  15. Disclaimer of Warranty.\n\n  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n  16. Limitation of Liability.\n\n  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n  17. Interpretation of Sections 15 and 16.\n\n  If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n                     END OF TERMS AND CONDITIONS\n\n            How to Apply These Terms to Your New Programs\n\n  If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n  To do so, attach the following notices to the program.  It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n    <one line to give the program's name and a brief idea of what it does.>\n    Copyright (C) <year>  <name of author>\n\n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation, either version 3 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License\n    along with this program.  If not, see <https://www.gnu.org/licenses/>.\n\nAlso add information on how to contact you by electronic and paper mail.\n\n  If the program does terminal interaction, make it output a short\nnotice like this when it starts in an interactive mode:\n\n    <program>  Copyright (C) <year>  <name of author>\n    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.\n    This is free software, and you are welcome to redistribute it\n    under certain conditions; type `show c' for details.\n\nThe hypothetical commands `show w' and `show c' should show the appropriate\nparts of the General Public License.  Of course, your program's commands\nmight be different; for a GUI interface, you would use an \"about box\".\n\n  You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU GPL, see\n<https://www.gnu.org/licenses/>.\n\n  The GNU General Public License does not permit incorporating your program\ninto proprietary programs.  If your program is a subroutine library, you\nmay consider it more useful to permit linking proprietary applications with\nthe library.  If this is what you want to do, use the GNU Lesser General\nPublic License instead of this License.  But first, please read\n<https://www.gnu.org/licenses/why-not-lgpl.html>.\n"
  },
  {
    "path": "README.md",
    "content": "This is a suite of tools for processing teletext signals recorded on VHS, as\nwell as tools for processing teletext packet streams. The software has only\nbeen tested with bt8x8 capture hardware, but should work with any VBI capture\nhardware with appropriate configuration.\n\nThis is the second rewrite of the original software. The old versions are\nstill available in the `v1` and `v2` branches of this repo, or from the\nreleases page.\n\nYou can see my collection of pages recovered with this software at:\n\nhttps://al.zerostem.io/~al/teletext/\n\nAnd more at:\n\nhttp://www.teletextarchive.com\n\nAnd:\n\nhttp://archive.teletextart.co.uk/\n\nINSTALLATION\n------------\n\nIn order to use CUDA decoding you need to use the Nvidia proprietary driver.\n\nTo install with optional dependencies run:\n\n    pip3 install -e .[CUDA,spellcheck,viewer]\n\nIf CUDA or pyenchant are not available for your platform simply omit them\nfrom the install command.\n\nIn order to use OpenCL you need to install pyopencl and the appropriate\nopencl runtime for your card.  Then run the deconvolve command with the '-O'\noption.\n\nIn order for the output to be rendered correctly you need to use a specific\nfont and terminal:\n\n    sudo apt-get install tv-fonts rxvt-unicode\n\nThen enable bitmap fonts in your X server:\n\n    cd /etc/fonts/conf.d\n    sudo rm 70-no-bitmaps.conf\n    sudo ln -s ../conf.avail/70-yes-bitmaps.conf .\n\nAfter doing this you may need to rehash:\n\n    xset fp rehash\n\nFinally open a terminal with the required font:\n\n    urxvt -fg white -bg black -fn teletext -fb teletext -geometry 41x25 +sb &\n\n\nUSAGE\n-----\n\nFirst capture VBI from VHS:\n\n    teletext record -d /dev/vbi0 > capture.vbi\n\nDeconvolve the recording:\n\n    teletext deconvolve capture.vbi > stream.t42\n\nExamine the headers to find services on the tape:\n\n    teletext filter -r 0 stream.t42\n\nSplit capture into services:\n\n    teletext filter --start <N> --stop <N> stream.t42 > stream-1.t42\n\nDisplay all copies of a page in a stream:\n\n    teletext filter stream.t42 -p 100\n\nSquash duplicate subpages, which reduces errors:\n\n    teletext squash stream.t42 > output.t42\n\nGenerate HTML pages from a stream:\n\n    teletext html output/ stream.t42 \n\nInteractively view the pages in a t42 stream:\n\n    teletext service stream.t42 | teletext interactive\n\nIn the interactive viewer you can type page numbers, or '.' for hold.\n\nRun each command with '--help' for a complete list of options.\n"
  },
  {
    "path": "TRAINING.md",
    "content": "Training\n--------\n\n1. Record the training signal to a tape:\n\n```\nteletext training | raspi-teletext -\n```\n\n2. Record it back to `training.vbi`.\n\n3. Run the following script to process the training file into patterns:\n\n```\n#!/bin/sh\nSPLITS=$(mktemp -d -p .)\nteletext training split $SPLITS training.vbi\nteletext training squash $SPLITS training.dat\nteletext training build -m full -b 4 20 training.dat full.dat\nteletext training build -m parity -b 6 20 training.dat parity.dat\nteletext training build -m hamming -b 1 20 training.dat hamming.dat\ncp full.dat parity.dat hamming.dat ~/Source/vhs-teletext/teletext/vbi/data/\necho $SPLITS\n```\n\nTheory\n------\n\nThe idea behind training is to record a known teletext signal on to\ntape and then play it back into the computer in the same way as you\nwould when recovering a tape. Then the original and observed signal\ncan be compared to build a database of patterns.\n\nTo make sure we can identify the degraded training packets, each one\nhas an ID and checksum. These are encoded so that each bit of data is\nthree bits wide in the output. This makes recovery of the original\ntrivial. There are also fixed bytes which can be used to help with\nalignment.\n\nWe want to fit the most possible patterns into the least possible\ntape. A De Bruijn sequence is used to do this. This is defined as the\nshortest possible sequence which contains every sequence of input\ncharacters (0 and 1 in this case) up to length N.\n\nWe use the De Bruijn sequence [2, 24]. This means it contains every\npossible sequence of 1 or 0 of length 24 bits, which is about 16\nmillion patterns. The ID field stores an offset into this sequence.\n\nWhen recording it is possible for a run of whole frames to be lost,\nso we do not simply display the whole De Bruijn sequence from start\nto finish. Instead, for each packet, we add a prime number to the\noffset and modulo the sequence length. This way every part of the\nsequence is shown multiple times, and even a long run of frame drops\nis unlikely to cause total loss of any part of the pattern.\n\nAfter recording the signal back into the computer it is sliced into\npatterns representing 24 bits of data. For a specific 24 bit pattern\nthere will be multiple slices in the signal. An average is taken of\nevery occurence and saved along with the original data it represents.\nThis is the intermediate training data.\n\nFinally the pattern files are built. A pattern is describe like this:\n\n 1. Number of bits to match before.\n 2. Set of possible bytes to match.\n 3. Number of bits to match after.\n\nSo for example, the parity data file is like this:\n\n build_pattern(args.parity, 'parity.dat', 4, 18, parity_set)\n\n Means: \n\n 1. Match 4 bits before.\n 2. Match any byte with odd parity. (128 possibilities/7 bits)\n 3. Match 3 bits after.\n\ngiving 14 bits total, or 16384 patterns.\n\nTo build the pattern data the intermediate data is processed and any\npattern which matches the criteria is added into a list. Then the\naverage for each list is taken. That is the final pattern we will \nmatch against.\n\n\n\n"
  },
  {
    "path": "examples/filter",
    "content": "#!/usr/bin/env python3\n\n# An example of making a customized filter in Python.\n\n# This is almost a direct copy-paste of the filter subcommand\n# tweaked to run standalone, and with an extra filter that\n# removes non-page packets.\n\nimport click\n\nfrom teletext.cli.clihelpers import packetreader, packetwriter, paginated\nfrom teletext import pipeline\n\n\n@click.command()\n@packetwriter\n@paginated()\n@click.option('--pagecount', 'n', type=int, default=0, help='Stop after n pages. 0 = no limit. Implies -P.')\n@click.option('-k', '--keep-empty', is_flag=True, help='Keep empty packets in the output.')\n@packetreader()\ndef filter(packets, pages, subpages, paginate, n, keep_empty):\n\n    \"\"\"Demultiplex and display t42 packet streams.\"\"\"\n\n    if n:\n        paginate = True\n\n    if not keep_empty:\n        packets = (p for p in packets if not p.is_padding())\n\n    # customize the filtering:\n    packets = (p for p in packets if p.mrag.row not in [29, 30, 31])\n\n    if paginate:\n        for pn, pl in enumerate(pipeline.paginate(packets, pages=pages, subpages=subpages), start=1):\n            yield from pl\n            if pn == n:\n                return\n    else:\n        yield from packets\n\n\nif __name__ == '__main__':\n    filter()\n"
  },
  {
    "path": "examples/maze",
    "content": "#!/usr/bin/env python\n\nimport random\n\nimport click\nimport numpy as np\nfrom PIL import Image, ImageDraw\n\nfrom teletext.cli.clihelpers import packetwriter\nfrom teletext.service import Service\nfrom teletext.subpage import Subpage\n\n\nclass Maze:\n    directions = [(0, 1), (1, 0), (0, -1), (-1, 0)]\n\n    def __init__(self, width=8, height=6):\n        self.width = width\n        self.height = height\n        self.data = np.zeros((2*height-1, 2*width-1), dtype=np.uint8)\n        self.data[::2, ::2] = 1\n        self.connect(0, 0, 0, 1)\n        self.connect(self.width-1, self.height-1, 0, -1)\n        self.generate(self.width//2, self.height//2, {(0, 0), (self.width-1, self.height-1)})\n\n    def valid(self, px, py):\n        return 0 <= px < self.width and 0 <= py < self.height\n\n    def connect(self, px, py, dx, dy):\n        self.data[(2*py)+dy, (2*px)+dx] = 1\n\n    def generate(self, x, y, visited):\n        visited.add((x, y))\n        for d in random.sample(self.directions, 4):\n            print(x, y, d)\n            nx, ny = x+d[0], y+d[1]\n            if self.valid(nx, ny) and (nx, ny) not in visited:\n                self.connect(x, y, *d)\n                self.generate(nx, ny, visited)\n\n    def connections(self, px, py, dx, dy):\n        ldx, ldy = -dy, dx\n        rdx, rdy = dy, -dx\n        left = []\n        right = []\n        while True:\n            if self.valid(px+ldx, py+ldy):\n                left.append(self.data[(2*py)+ldy, (2*px)+ldx])\n            else:\n                left.append(0)\n            if self.valid(px+rdx, py+rdy):\n                right.append(self.data[(2*py)+rdy, (2*px)+rdx])\n            else:\n                right.append(0)\n            if self.valid(px+dx, py+dy) and self.data[(2*py)+dy, (2*px)+dx]:\n                px += dx\n                py += dy\n            else:\n                break\n        return left, right\n\n    def bitmap(self, left, right):\n        w, h = 39*2, 23*3\n        hh = h - 1\n        ww = w - 1\n        bitmap = Image.new(\"1\", (w, h))\n\n        def ox(o, l):\n            o = min(o, (ww//2)-4)\n            return o + 4 if l else ww-o-4\n\n        def oy(o, t):\n            o = min(o, (hh//2))\n            return o if t else hh-o\n\n        def draw_p(x1, y1, x2, y2, l):\n            x1 = ox(x1, l)\n            x2 = ox(x2, l)\n            for t in (True, False):\n                y1 = oy(y1, t)\n                y2 = oy(y2, t)\n                draw.line((x1, y1, x2, y2), fill=1)\n\n        def draw_h(x1, x2, y, l):\n            return draw_p(x1, y, x2, y, l)\n\n        def draw_d(xy1, xy2, l):\n            return draw_p(xy1, xy1, xy2, xy2, l)\n\n        def draw_v(xy, l):\n            x = ox(xy, l)\n            draw.line((x, oy(xy, True), x, oy(xy, False)), fill=1)\n\n        def draw_side(n, o1, o2, o3, l, p):\n            if p:\n                draw_h(o2, o3, o3, l)\n                draw_v(o2, l)\n                if n+1 < len(left):\n                    draw_v(o3, l)\n            else:\n                draw_d(o2, o3, l)\n            draw_d(o1, o2, l)\n\n        def calc_o(n):\n            return (n * 22) - 16 - ((n*(n+1)*2))\n\n        draw = ImageDraw.Draw(bitmap)\n        for n, (l, r) in enumerate(zip(left, right)):\n            o1 = calc_o(n)\n            o2 = o1 + max(0, 6 - n)\n            o3 = max(o1, calc_o(n+1))\n            draw_side(n, o1, o2, o3, True, l)\n            draw_side(n, o1, o2, o3, False, r)\n\n        o = calc_o(len(left))\n        draw_h(o, ww, o, True)\n        draw_h(o, ww, o, False)\n        if not left[-1]:\n            draw_v(o, True)\n        if not right[-1]:\n            draw_v(o, False)\n\n        return np.array(bitmap)\n\n    def view_to_mrag(self, px, py, dx, dy):\n        return self.directions.index((dx, dy)) + 2, py + (px << 4)\n\n    def view(self, px, py, dx, dy):\n        m, p = self.view_to_mrag(px, py, dx, dy)\n        page = Subpage(prefill=True, magazine=m)\n        page.header.page = p\n        page.header.control = 0\n        left, right = self.connections(px, py, dx, dy)\n\n        page.displayable.place_bitmap(np.array(self.bitmap(left[:5], right[:5])))\n\n        # cheat mode / debug dump\n        # put revealo maps on every wall\n        if True and len(left) == 1:\n            x = (38 - self.data.shape[1])//2\n            y = (23 - self.data.shape[0])//2\n            for n, r in enumerate(self.data[::-1]):\n                page.displayable.place_string('\\x07\\x18' + ''.join('.' if p else ' ' for p in r) + '\\x17', x=x, y=y+n)\n            dn = ['^', '>', 'v', '<'][self.directions.index((dx, dy))]\n            page.displayable.place_string(dn, x=px*2+x+2, y=self.data.shape[0]+y-1-(2*py))\n\n        # create fastext links\n        # TODO: add some helpers to make this easier\n        page.displayable.place_string('\\x01TurnLeft\\x02StepForward\\x03TurnRight\\x06StepBack', y=23)\n\n        page.init_packet(27, magazine=m)\n        page.packet(27).fastext.dc = 0\n        page.packet(27).fastext.control = 0xf\n\n        lm, lp = self.view_to_mrag(px, py, -dy, dx)\n        page.packet(27).fastext.links[0].magazine = lm\n        page.packet(27).fastext.links[0].page = lp\n\n        lm, lp = self.view_to_mrag(px, py, dy, -dx)\n        page.packet(27).fastext.links[2].magazine = lm\n        page.packet(27).fastext.links[2].page = lp\n\n        if self.valid(px+dx, py+dy) and self.data[(2*py)+dy, (2*px)+dx]:\n            lm, lp = self.view_to_mrag(px+dx, py+dy, dx, dy)\n        else:\n            lm, lp = m, p\n        page.packet(27).fastext.links[1].magazine = lm\n        page.packet(27).fastext.links[1].page = lp\n\n        if self.valid(px-dx, py-dy)and self.data[(2*py)-dy, (2*px)-dx]:\n            lm, lp = self.view_to_mrag(px-dx, py-dy, dx, dy)\n        else:\n            lm, lp = m, p\n        page.packet(27).fastext.links[3].magazine = lm\n        page.packet(27).fastext.links[3].page = lp\n\n        return page\n\n    def map(self):\n        page = Subpage(prefill=True, magazine=1)\n        page.header.page = 0\n        page.header.control = 0\n        page.displayable.place_string(\"\\x0d      You are trapped in a maze!       \", y=2)\n        page.displayable.place_string(\"\\x0d   Use the fastext buttons to move.    \", y=4)\n        page.displayable.place_string(\"\\x0d       Press reveal for a hint:        \", y=6)\n\n        page.displayable.place_string('\\x12\\x18\\x24', x=18+(self.data.shape[1]//4), y=8)\n        page.displayable.place_bitmap(self.data[::-1], x=19-(self.data.shape[1]//4), y=9, conceal=True)\n        page.displayable.place_string('\\x11\\x18\\x21', x=17-(self.data.shape[1]//4), y=10+(self.data.shape[0]//3))\n\n        page.displayable.place_string(\"\\x0d              Good luck!               \", y=14)\n        page.displayable.place_string(\"\\x0d       Press any button to begin.      \", y=16)\n        page.displayable.place_string(\"\\x01  Begin  \\x02  Begin  \\x03  Begin  \\x06  Begin  \", y=23)\n\n        page.init_packet(27, magazine=1)\n        page.packet(27).fastext.dc = 0\n        page.packet(27).fastext.control = 0xf\n        for n in range(4):\n            page.packet(27).fastext.links[n].magazine = 2\n            page.packet(27).fastext.links[n].page = 0\n        return page\n\n    def service(self):\n        service = Service(replace_headers=True, title=\"Maze!\")\n        service.insert_page(self.map())\n        for y in range(self.height):\n            for x in range(self.width):\n                for d in self.directions:\n                    service.insert_page(self.view(x, y, *d))\n        return service\n\n@click.command()\n@packetwriter\ndef maze():\n    return Maze().service()\n\nif __name__ == '__main__':\n    maze()\n"
  },
  {
    "path": "examples/service",
    "content": "#!/usr/bin/env python3\n\n# An example of generating a service from scratch.\n\nimport sys\n\nfrom teletext.service import Service\nfrom teletext.subpage import Subpage\n\n# Create a subpage\nsubpage = Subpage(prefill=True)\n\n# Fill with clock cracker\nsubpage.displayable[:,0::2] = 0xfe\nsubpage.displayable[:,1::2] = 0x7f\nsubpage.displayable[:,0] = 0x20\n\n# Put a message in the middle\nsubpage.displayable.place_string('Hello World', 15, 11)\n\n# Create the service\nservice = Service(replace_headers=True)\n\n# Add the subpage to the service.\nservice.magazines[1].pages[0].subpages[0] = subpage\n\n# Set magazine name and number\nservice.magazines[1].title = 'Example '\n\n# Broadcast it forever\nwhile True:\n    # Stream some packets\n    for packet in service.packets(32):\n        sys.stdout.buffer.write(packet.bytes)\n\n    # Modify the service if required\n"
  },
  {
    "path": "examples/terminal",
    "content": "#!/usr/bin/env python3\n\nimport datetime\nimport os\nimport pty\nimport select\nimport shlex\nimport time\n\nimport click\nimport pyte\n\nfrom teletext.cli.clihelpers import packetwriter\nfrom teletext.subpage import Subpage\n\n\n@click.command()\n@click.argument(\"command\", type=str)\n@packetwriter\ndef terminal(command):\n    \"\"\"\n    Runs a command in a simulated terminal and outputs a packet stream on page 100.\n    \"\"\"\n    columns, lines = 40, 24\n\n    # run the command\n    p_pid, p_fd = pty.fork()\n    if p_pid == 0:  # Child.\n        argv = shlex.split(command)\n        env = dict(TERM=\"linux\", LC_ALL=\"en_GB.UTF-8\",\n                   COLUMNS=str(columns), LINES=str(lines))\n        os.execvpe(argv[0], argv, env)\n\n    # set up virtual terminal\n    screen = pyte.Screen(columns, lines)\n    screen.set_mode(pyte.modes.LNM)\n    screen.write_process_input = lambda data: p_fd.write(data.encode())\n    stream = pyte.ByteStream()\n    stream.attach(screen)\n\n    # set up page\n    page = Subpage(prefill=True, magazine=1)\n    page.header.control = 1<<4\n    page.header.page = 0\n    # init fastext so we can use row 24\n    page.init_packet(27, magazine=1)\n    page.packet(27).fastext.dc = 0\n    page.packet(27).fastext.control = 0xf\n\n    # generation loop\n    prev_refresh = time.time()\n    try:\n        while True:\n            r, w, x = select.select([p_fd], [], [], 1.0)\n            if p_fd in r:\n                stream.feed(os.read(p_fd, 65536))\n\n            dt = datetime.datetime.now().strftime(\" %a %d %b\\x03%H:%M/%S\")\n            page.header.displayable.place_string('%-12s' % (command[:12]) + dt)\n\n            for y in screen.dirty:\n                line = screen.buffer[y]\n                data = ''.join(char.data for char in (line[x] for x in range(screen.columns)))\n                page.packet(y+1).displayable.place_string(data)\n\n            now = time.time()\n            elapsed = now - prev_refresh\n            if elapsed > 1.0:\n                prev_refresh = now\n                send_lines = range(lines)\n            else:\n                send_lines = screen.dirty\n\n            yield page.packet(0)\n            for y in send_lines:\n                yield page.packet(y+1)\n            yield page.packet(27)\n\n            screen.dirty.clear()\n    except OSError:\n        pass\n\n\nif __name__ == '__main__':\n    terminal()\n"
  },
  {
    "path": "examples/tti",
    "content": "#!/usr/bin/env python3\n\nimport json\nimport re\nimport requests\nfrom PIL import Image\nimport numpy as np\n\nfrom teletext.subpage import Subpage\n\n\ndef get_sensors():\n    url = \"http://hms.nottinghack.org.uk/api/spaceapi\"\n    dat = json.loads(requests.get(url).text)\n    return {s['location']: s[\"value\"] for s in dat['sensors']['temperature']}\n\n\ndef colour(t):\n    if t < 10:\n        return '\\x06'\n    elif t < 20:\n        return '\\x02'\n    elif t < 30:\n        return '\\x04'\n    else:\n        return '\\x01'\n\ndef clean_name(s):\n    s = s.split('/')[0].replace('-LLAP', '').replace('G5-', '')\n    return re.sub(r'([^\\s-])([A-Z])', r'\\1 \\2', s)\n\ndef tti():\n    with open('template.t42', 'rb') as f:\n        page = Subpage.from_file(f)\n\n    page.header.page = 0\n    page.header.control = 0\n\n    layout = (\n        ('Airlock', 'ComfyArea-LLAP'),\n        ('Studio-LLAP', 'CraftRoom-LLAP'),\n        ('Kitchen-LLAP', 'Workshop-LLAP'),\n        ('ClassRoom', 'G5-BlueRoom-LLAP'),\n    )\n\n    i = Image.open(\"sensors.png\").convert(mode=\"1\")\n    page.displayable.place_bitmap(np.array(i), 16, 2, 0x13)\n\n    sensors = get_sensors()\n    for n, l in enumerate(layout):\n        clean_names = (clean_name(s)[:10] + ':' for s in l)\n        string = '  \\x07'.join(f'{c:11s}{colour(sensors[s])}{sensors[s]:4.1f}C' for c, s in zip(clean_names, l))\n        page.displayable.place_string('\\x0d'+string, 1, (3*n)+8) # 0d = double height\n\n    page.displayable.place_string('\\x02\\x1d\\x04Temperature readings from the space', 0, 21)\n    page.displayable.place_string('\\x02\\x1d\\x04See P123 for info about the heating', 0, 22)\n    page.displayable.place_string('\\x01Main Index   \\x02News    \\x03Events    \\x06Tools', 0, 23)\n\n    with open('P100.tti', 'wb') as f:\n        f.write(page.to_tti())\n\n    print(\"https://zxnet.co.uk/teletext/editor/#\" + page.url)\n\nif __name__ == '__main__':\n    tti()\n"
  },
  {
    "path": "examples/video",
    "content": "#!/usr/bin/env python3\n\nimport datetime\nimport socket\n\nimport click\nimport cv2\nimport srt\n\nfrom teletext.cli.clihelpers import packetwriter\nfrom teletext.subpage import Subpage\n\n\n@click.command()\n@click.argument('videofile', type=click.Path(readable=True))\n@click.option('-s', '--subs', type=click.File('r'), help=\"Subtitle file (srt)\")\n@click.option('--start', type=int, default=0, help=\"Start at frame N\")\n@click.option('--end', type=int, default=None, help=\"End at frame N\")\n@packetwriter\ndef video(videofile, subs, start, end):\n    \"\"\"\n    Converts a video into a page stream. Also supports SRT subtitles.\n    Can set a start and end position in frames. Also works with images\n    because OpenCV treats them as videos with one frame.\n    \"\"\"\n    hostname = socket.gethostname()[:12]\n\n    # Set up page\n    page = Subpage(prefill=True, magazine=1)\n    page.header.control = 1<<4\n    page.header.page = 0\n    # init fastext so we can use row 24\n    page.init_packet(27, magazine=1)\n    page.packet(27).fastext.dc = 0\n    page.packet(27).fastext.control = 0xf\n\n    # Creating a VideoCapture object to read the video\n    cap = cv2.VideoCapture(videofile)\n    cap.set(cv2.CAP_PROP_POS_FRAMES, start)\n    fps = cap.get(cv2.CAP_PROP_FPS)\n    frameno = start\n\n    sub = None\n    if subs:\n        subs = srt.parse(subs.read())\n        sub = next(subs)\n\n    # Loop until the end of the video\n    while (cap.isOpened() and (end is None or frameno <= end)):\n        # Capture frame-by-frame\n        ret, frame = cap.read()\n        if not ret: # eof\n            break\n\n        frameno += 1\n\n        frame = cv2.resize(frame, (39*2, 24*3), fx = 0, fy = 0,\n                             interpolation = cv2.INTER_CUBIC)\n\n        # conversion of BGR to grayscale is necessary to apply this operation\n        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n\n        # adaptive thresholding to use different threshold\n        # values on different regions of the frame.\n        thresha = cv2.adaptiveThreshold(gray, 1, cv2.ADAPTIVE_THRESH_MEAN_C,\n                                               cv2.THRESH_BINARY, 3, 2)\n        threshb = cv2.threshold(gray, 16, 255, cv2.THRESH_BINARY)[1]\n        thresh = (thresha * threshb) > 0\n\n        dt = datetime.datetime.now().strftime(\" %a %d %b\\x03%H:%M/%S\")\n        page.header.displayable.place_string('%-12s' % (hostname) + dt)\n        page.displayable.place_bitmap(thresh)\n\n        if sub:\n            seconds = datetime.timedelta(seconds = frameno / fps)\n            try:\n                while seconds > sub.end:\n                    sub = next(subs)\n                if sub and seconds >= sub.start:\n                    x = sub.content.split('\\n')\n                    w = len(max(x, key=len))\n                    o = max(min(38 - w, 2), 0)\n                    for n, l in enumerate(x):\n                        page.packet(n+21).displayable.place_string(('\\x06' + l + '\\x17')[:(40-o)], x=o)\n            except StopIteration:\n                sub = None\n\n        yield from page.packets\n\n    # release the video capture object\n    cap.release()\n\nif __name__ == '__main__':\n    video()\n"
  },
  {
    "path": "misc/teletext-noscanlines.css",
    "content": "body {background: black;}\n\na {color: inherit; text-decoration: none;}\na:hover {color: orange; } a:active {color: red;}\n:focus {outline: 0;}\n@font-face {font-family:teletext2; src:url('teletext2.ttf')}\n@font-face {font-family:teletext4; src:url('teletext4.ttf')}\n\n.subpage {\n    position:relative; top:0; left:0;\n    float:left; white-space: pre; color:white; background:black;\n    font-family: teletext2; font-size:30px; line-height: 100%;\n    border: solid black 10px;\n    border-bottom: solid black 20px;\n    text-shadow: 0 0 0.05em;\n}\n\n.subpage{\n    filter:blur(0.5px) brightness(120%);\n}\n\n.row{\n    display: flex;\n}\n\n.subpage:after{\n    content: \"\";\n    display: block;\n    position: absolute;\n    top: 0;\n    left: 0;\n    height: 100%;\n    width: 100%;\n    z-index:10;\n    pointer-events:none;\n}\n\n.f0{color:black;} .f1{color:red;} .f2{color:#00ff00;} .f3{color:yellow;}\n.f4{color:blue;} .f5{color:magenta;} .f6{color:cyan;} .f7{color:white;}\n\n.b0{background:black;} .b1{background:red;} .b2{background:#00ff00;} .b3{background:yellow;}\n.b4{background:blue;} .b5{background:magenta;} .b6{background:cyan;} .b7{background:white;}\n\n.dh{font-family: teletext4; font-size:200%; line-height:100%;}\n\n.fl{text-decoration: blink}\n"
  },
  {
    "path": "misc/teletext.css",
    "content": "@import url(\"teletext-noscanlines.css\");\n\n/* scanlines */\n\n.subpage {\n    /* This gradient looks worse but can be rendered at 2 (physical) pixels. */\n    --gradient-small: repeating-linear-gradient(\n                            to top,\n                            rgba(0,0,0,0.8) 0px,\n                            transparent 1px,\n                            transparent 2px,\n                            rgba(0,0,0,0.8) 3px\n                        );\n    \n    /* This gradient looks better but can't be rendered at 2 (physical) pixels. */\n    --gradient-big: repeating-linear-gradient(\n                        to top,\n                        rgba(0,0,0,1),\n                        transparent 25%,\n                        transparent 75%,\n                        rgba(0,0,0,1) 100%\n                    );\n}\n\n\n/* Small size: 245px x 250px, 1 x 1 scaling */\n@media (max-width: 660px) {\n    .subpage { font-size:10px; }\n    .subpage:after { background-image: none; }\n}\n\n/* Medium size: 490px x 500px, 2 x 2 scaling, scanlines possible */\n@media (min-width: 660px) and (max-width: 980px) {\n    .subpage { font-size:20px; }\n    .subpage:after{\n        background-size: 2px 2px;\n        background-image: var(--gradient-small);\n    }\n    /* Use the nicer gradient on hidpi/retina displays. */\n    /* This is only necessary when 2x2 scaling. */\n    @media (min-resolution: 2dppx) {\n        .subpage:after{\n            background-image: var(--gradient-big);\n        }\n    }\n}\n\n/* Large size: 735 x 750px, 3 x 3 scaling, scanlines possible */\n@media (min-width: 980px) {\n    .subpage { font-size:30px; }\n    .subpage:after{\n        /* At 3x3 scaling and above, the big gradient looks better. */\n        background-image: var(--gradient-big);\n        background-size: 3px 3px;\n    }\n}\n"
  },
  {
    "path": "setup.py",
    "content": "from setuptools import setup\n\nsetup(\n    name='teletext',\n    version='3.1.99',\n    author='Alistair Buxton',\n    author_email='a.j.buxton@gmail.com',\n    url='http://github.com/ali1234/vhs-teletext',\n    packages=['teletext', 'teletext.vbi', 'teletext.cli', 'teletext.gui'],\n    package_data={\n        'teletext.vbi': [\n            'data/debruijn.dat',\n            'data/vhs/parity.dat',\n            'data/vhs/hamming.dat',\n            'data/vhs/full.dat',\n            'data/betamax/parity.dat',\n            'data/betamax/hamming.dat',\n            'data/betamax/full.dat'\n        ],\n        'teletext.gui': [\n            'decoder.qml',\n            'editor.ui',\n        ]\n    },\n    entry_points={\n        'console_scripts': [\n            'teletext = teletext.cli.teletext:teletext',\n        ],\n        'gui_scripts': [\n            'ttviewer = teletext.gui.editor:main',\n        ],\n    },\n    install_requires=[\n        'numpy<2', 'scipy', 'matplotlib', 'click', 'tqdm',  'pyzmq', 'watchdog', 'pyserial',\n        'windows-curses;platform_system==\"Windows\"',\n    ],\n    extras_require={\n        'spellcheck': ['pyenchant'],\n        'CUDA': ['pycuda'],\n        'OpenCL': ['pyopencl'],\n        'viewer': ['PyOpenGL'],\n        'profiler': ['plop'],\n        'qt': ['PyQt5'],\n        'audio': ['spectrum', 'miniaudio'],\n    }\n)\n"
  },
  {
    "path": "teletext/__init__.py",
    "content": ""
  },
  {
    "path": "teletext/__main__.py",
    "content": "from teletext.cli.teletext import teletext\n\n\nif __name__ == '__main__':\n    teletext()\n"
  },
  {
    "path": "teletext/celp.py",
    "content": "import numpy as np\nimport matplotlib.pyplot as plt\nfrom spectrum import lsf2poly\nimport numpy as np\nfrom scipy.signal import lfilter\nfrom collections import deque\nfrom tqdm import tqdm\n\nhamming7_dec = np.array([\n    [ 0x00, 0x10, 0x10, 0x14, 0x10, 0x11, 0x18, 0x12 ],\n    [ 0x10, 0x11, 0x13, 0x19, 0x11, 0x01, 0x15, 0x11 ],\n    [ 0x10, 0x1a, 0x13, 0x12, 0x16, 0x12, 0x12, 0x02 ],\n    [ 0x13, 0x17, 0x03, 0x13, 0x1b, 0x11, 0x13, 0x12 ],\n    [ 0x10, 0x14, 0x14, 0x04, 0x16, 0x1c, 0x15, 0x14 ],\n    [ 0x1d, 0x17, 0x15, 0x14, 0x15, 0x11, 0x05, 0x15 ],\n    [ 0x16, 0x17, 0x1e, 0x14, 0x06, 0x16, 0x16, 0x12 ],\n    [ 0x17, 0x07, 0x13, 0x17, 0x16, 0x17, 0x15, 0x1f ],\n    [ 0x10, 0x1a, 0x18, 0x19, 0x18, 0x1c, 0x08, 0x18 ],\n    [ 0x1d, 0x19, 0x19, 0x09, 0x1b, 0x11, 0x18, 0x19 ],\n    [ 0x1a, 0x0a, 0x1e, 0x1a, 0x1b, 0x1a, 0x18, 0x12 ],\n    [ 0x1b, 0x1a, 0x13, 0x19, 0x0b, 0x1b, 0x1b, 0x1f ],\n    [ 0x1d, 0x1c, 0x1e, 0x14, 0x1c, 0x0c, 0x18, 0x1c ],\n    [ 0x0d, 0x1d, 0x1d, 0x19, 0x1d, 0x1c, 0x15, 0x1f ],\n    [ 0x1e, 0x1a, 0x0e, 0x1e, 0x16, 0x1c, 0x1e, 0x1f ],\n    [ 0x1d, 0x17, 0x1e, 0x1f, 0x1b, 0x1f, 0x1f, 0x0f ],\n], dtype=np.uint8)\n\nclass LtpCodebook:\n    def __init__(self, subframe_length):\n        #self.buffer = np.zeros((147, subframe_length), dtype=np.double)\n        #self.pos = 0\n        self.buffer = deque(maxlen=147)\n\n    def insert(self, subframe):\n        self.buffer.extendleft(subframe)\n\n    def get(self, lag):\n        try:\n            return self.buffer[lag + 20]\n        except IndexError:\n            return 0\n\n\nclass CELPStats:\n    def __init__(self, decoder):\n        self.decoder = decoder\n\n    def __str__(self):\n        result = f', L:{self.decoder.lsf_error:.0f}%, VG:{self.decoder.vector_gain_error:.0f}%'\n        # reset the error counters\n        self.decoder.lsf_errors = 0\n        self.decoder.vector_gain_errors = 0\n        self.decoder.subframes = 0\n        return result\n\n\nclass CELPDecoder:\n    widths = np.array([\n        0,\n        3, 4, 4, 4, 4, 4, 4, 4, 3, 3, # 37 bytes - 10 x LPC params of (unknown?) variable size\n        5, 5, 5, 5,             # 4x5 = 20 bytes - pitch gain (LTP gain)\n        5, 5, 5, 5,             # 4x5 = 20 bytes - vector gain\n        7, 7, 7, 7,             # 4x7 = 28 bytes - pitch index (LTP lag)\n        8, 8, 8, 8,             # 4x8 = 32 bytes - vector index\n        3, 3, 3, 3,             # 4x3 = 12 bytes - error correction for vector gains?\n        3,                      # 3 bytes - always zero (except for recovery errors)\n    ])\n    offsets = np.cumsum(widths)\n\n\n    lsf_vector_quantizers = {\n        # Source Reliant Error Control For Low Bit Rate Speech Communications, Ong, p103\n        'ong': np.array([\n            [ 178,  218,  236,  267,  293,  332,  378,  420,    0,    0,    0,    0,    0,    0,    0,    0,],\n            [ 210,  235,  265,  295,  325,  360,  400,  440,  480,  520,  560,  610,  670,  740,  810,  880,],\n            [ 420,  460,  500,  540,  585,  640,  705,  775,  850,  950, 1050, 1150, 1250, 1350, 1450, 1550,],\n            [ 752,  844,  910,  968, 1016, 1064, 1110, 1155, 1202, 1249, 1295, 1349, 1409, 1498, 1616, 1808,],\n            [1041, 1174, 1274, 1340, 1407, 1466, 1514, 1559, 1611, 1658, 1714, 1773, 1834, 1906, 2008, 2166,],\n            [1438, 1583, 1671, 1740, 1804, 1855, 1905, 1947, 1988, 2034, 2081, 2135, 2193, 2267, 2369, 2476,],\n            [2005, 2115, 2176, 2222, 2260, 2297, 2333, 2365, 2394, 2427, 2463, 2501, 2551, 2625, 2728, 2851,],\n            [2286, 2410, 2480, 2528, 2574, 2613, 2650, 2689, 2723, 2758, 2790, 2830, 2879, 2957, 3049, 3197,],\n            [2775, 2908, 3000, 3086, 3159, 3234, 3331, 3453,    0,    0,    0,    0,    0,    0,    0,    0,],\n            [3150, 3272, 3354, 3415, 3473, 3531, 3580, 3676,    0,    0,    0,    0,    0,    0,    0,    0,],\n        ]),\n\n        # Speech Coding in Private and Broadcast Networks, Suddle, p121\n        # note the transposition of first 8 values in column 7/8\n        'suddle': np.array([\n            [ 143,  182,  214,  246,  284,  329,  389,  475,    0,    0,    0,    0,    0,    0,    0,    0,],\n            [ 211,  252,  285,  317,  349,  383,  419,  458,  503,  554,  608,  665,  731,  809,  912, 1072,],\n            [ 402,  470,  522,  571,  621,  671,  724,  778,  835,  902,  979, 1065, 1147, 1241, 1357, 1517,],\n            [ 617,  732,  819,  885,  944, 1001, 1060, 1121, 1186, 1260, 1342, 1425, 1514, 1613, 1723, 1885,],\n            [ 981, 1081, 1172, 1254, 1329, 1403, 1473, 1539, 1609, 1679, 1753, 1826, 1908, 1998, 2106, 2236,],\n            [1334, 1446, 1539, 1626, 1697, 1763, 1828, 1890, 1954, 2019, 2087, 2160, 2238, 2328, 2420, 2526,],\n            [1830, 1959, 2056, 2134, 2198, 2254, 2303, 2349, 2397, 2448, 2500, 2560, 2632, 2715, 2823, 2966,],\n            [2247, 2361, 2434, 2496, 2550, 2600, 2647, 2694, 2742, 2791, 2846, 2904, 2966, 3049, 3155, 3256,],\n            [2347, 2481, 2583, 2674, 2767, 2874, 3005, 3202,    0,    0,    0,    0,    0,    0,    0,    0,],\n            [3140, 3246, 3326, 3395, 3458, 3524, 3601, 3709,    0,    0,    0,    0,    0,    0,    0,    0,],\n        ]),\n    }\n\n    lsf_weights = np.array([1/8, 3/8, 5/8, 7/8])\n\n    vec_gain_quantizers = {\n        # Suddle, chapter 7\n        'audetel': np.array([\n            -1996, -1306, -990, -780, -628, -510, -418, -336,\n            -268, -204, -148, -96, -54, -20, -6, -2,\n            2, 6, 20, 54, 96, 148, 204, 268,\n            336, 418, 510, 628, 780, 990, 1306, 1996,\n        ]),\n\n        # Suddle, chapter 5\n        'unknown': np.array([\n            -1100,  -850,  -650,  -510,  -415,  -335,  -275,  -220,\n             -175,  -135,   -98,   -65,   -35,   -12,    -3,    -1,\n                1,     3,    12,    35,    65,    98,   135,   175,\n              220,   275,   335,   415,   510,   650,   850,  1100,\n        ]),\n    }\n\n    ltp_gain_quantization = np.array([\n        -0.993, -0.831, -0.693, -0.555, -0.414, -0.229,    0.0,  0.193,\n         0.255,  0.368,  0.457,  0.531,  0.601,  0.653,  0.702,  0.745,\n         0.780,  0.816,  0.850,  0.881,  0.915,  0.948,  0.983,  1.020,\n         1.062,  1.117,  1.193,  1.289,  1.394,  1.540,  1.765,  1.991,\n    ])\n\n    def __init__(self, lsf_lut='suddle', vec_gain_lut='audetel', sample_rate=8000):\n        self.lsf_lut = lsf_lut\n        self.lsf_vector_quantization = self.lsf_vector_quantizers[lsf_lut]\n        self.vec_gain_quantization = self.vec_gain_quantizers[vec_gain_lut]\n        self.sample_rate = sample_rate\n        self.subframe_length = sample_rate // 200\n        self.pos = 0\n        self.vector_gain_errors = 0\n        self.lsf_errors = 0\n        self.subframes = 0\n\n    @property\n    def lsf_error(self):\n        return 100.0 * self.lsf_errors / self.subframes\n\n    @property\n    def vector_gain_error(self):\n        return 100.0 * self.vector_gain_errors / self.subframes\n\n    def stats(self):\n        return CELPStats(self)\n\n    def vector_parity(self, data, parity):\n        hamm = hamming7_dec[data>>1, parity]\n        self.vector_gain_errors += np.sum(hamm >> 4)\n        return ((hamm&0xf)<<1)|(data&1)\n\n    def decode_params(self, raw_frame):\n        \"\"\"Extracts the parameters from the raw packet according to offsets and widths.\"\"\"\n        bits = np.unpackbits(raw_frame, bitorder='little')\n        decoded_frame = np.empty((30, ), dtype=np.uint8)\n        for n in range(len(self.offsets)-2):\n            slice = bits[self.offsets[n]:self.offsets[n+1]]\n            decoded_frame[n] = np.packbits(slice, bitorder='little')\n        lsf = self.lsf_vector_quantization[np.arange(10), decoded_frame[:10]]\n        if np.any(np.diff(lsf) < 0):\n            self.lsf_errors += 4\n        pitch_gain = self.ltp_gain_quantization[decoded_frame[10:14]]\n        #vector_gain = self.vec_gain_quantization[decoded_frame[14:18]]  # no hamming correction\n        vector_gain = self.vec_gain_quantization[self.vector_parity(decoded_frame[14:18], decoded_frame[26:30])]\n        pitch_idx = decoded_frame[18:22]\n        vector_idx = decoded_frame[22:26]\n        self.subframes += 4\n        return lsf, pitch_gain, vector_gain, pitch_idx, vector_idx\n\n    # wave we're going to play through the filter\n    wave = np.random.uniform(-1, 1, size=(8000, ))\n\n    def apply_lpc_filter(self, lsf, signal):\n        \"\"\"Convert line spectrum frequencies to a filter and apply to the signal.\"\"\"\n        a = lsf2poly(sorted(lsf * 2 * np.pi / self.sample_rate))\n        result, self.last_z = lfilter([1], a, signal, zi=self.last_z)\n        return result\n\n    def generate_audio(self, raw_frame):\n        \"\"\"Generate an audio frame from a raw frame.\"\"\"\n        lsf, pitch_gain, vector_gain, pitch_idx, vector_idx = self.decode_params(raw_frame)\n\n        # interpolate the LSFs\n        if self.last_lsf is None:\n            sub_lsf = np.repeat(lsf, 4).reshape(10, 4).T\n        else:\n            sub_lsf = (self.last_lsf[np.newaxis, :] * self.lsf_weights[::-1, np.newaxis]) + (lsf[np.newaxis, :] * self.lsf_weights[:, np.newaxis])\n\n        frame = np.empty((self.subframe_length * 4,), dtype=np.double)\n        subframe_buf = np.empty((self.subframe_length), dtype=np.double)\n        for subframe in range(4):\n            for n in range(self.subframe_length):\n                self.pos += 1\n                subframe_buf[n] = (self.wave[self.pos % self.sample_rate] * vector_gain[subframe]) + (self.ltp_codebook.get(pitch_idx[subframe] - n) * pitch_gain[subframe])\n            self.ltp_codebook.insert(subframe_buf)\n            p = subframe * self.subframe_length\n            frame[p:p + self.subframe_length] = self.apply_lpc_filter(sub_lsf[subframe], subframe_buf)\n\n        self.last_lsf = lsf\n        return np.clip(frame * 0.5, -32767, 32767).astype(np.int16)\n\n    def decode_packet_stream(self, packets, frame=None):\n        \"\"\"Decode an entire packet stream, yielding audio frames.\"\"\"\n        self.last_lsf = None\n        self.last_z = np.zeros((10, ))\n        self.ltp_codebook = LtpCodebook(self.subframe_length)\n        for p in packets:\n            if frame is None or frame == 0:\n                yield self.generate_audio(p._array[4:23])\n            if frame is None or frame == 1:\n                yield self.generate_audio(p._array[23:42])\n\n    def stream_pcm(self, packets, frame, device):\n        src = self.decode_packet_stream(packets, frame)\n        buf = []\n        buflen = 0\n        required_samples = yield b\"\"  # generator initialization\n        for frame in src:\n            buf.append(frame)\n            buflen += frame.shape[0]\n            if buflen >= required_samples:\n                tmp = np.concatenate(buf)\n                buf = [tmp[required_samples:]]\n                buflen = buf[0].shape[0]\n                required_samples = yield tmp[:required_samples].tobytes()\n        device.__running = False\n\n    def play(self, packets, frame=None):\n        \"\"\"Play a packet stream.\"\"\"\n        import miniaudio\n        import time\n\n        with miniaudio.PlaybackDevice(output_format=miniaudio.SampleFormat.SIGNED16,\n                                      nchannels=1, sample_rate=self.sample_rate,\n                                      buffersize_msec=500) as device:\n            device.__running = True\n            stream = self.stream_pcm(packets, frame, device)\n            next(stream)  # start the generator\n            device.start(stream)\n            while device.__running:\n                time.sleep(0.1)\n\n    def convert(self, output, packets, frame=None):\n        import wave\n        wf = wave.open(output, 'wb')\n        wf.setnchannels(1)\n        wf.setsampwidth(2)\n        wf.setframerate(8000)\n        for frame in self.decode_packet_stream(packets, frame):\n            wf.writeframes(frame.tobytes())\n        wf.close()\n\n    @classmethod\n    def plot(cls, packets):\n        \"\"\"Plot statistics of the raw packets.\"\"\"\n        datas = []\n        for p in packets:\n            datas.append(p._array[4:])\n        datas = np.concatenate(datas)\n\n        data = np.unpackbits(datas.reshape(-1, 2, 19), bitorder='little').reshape(-1, 2, 152)\n        d1 = np.sum(data, axis=0)\n        p = np.arange(152)\n\n        fig, ax = plt.subplots(6, 2)\n\n        # plot the bit counts (top plots)\n        for n in range(cls.offsets.shape[0]-1):\n            s = slice(cls.offsets[n], cls.offsets[n + 1])\n            ax[0][0].bar(p[s], d1[0][s], 0.8)\n            ax[0][1].bar(p[s], d1[1][s], 0.8)\n\n        # plot vector and pitch parameters over time\n        for x in range(2):\n            frame = data[:, x, :]\n            for y, o in enumerate([10, 14, 18, 22], start=2):\n                bits = frame[:,cls.offsets[o]:cls.offsets[o+4]].reshape(-1, cls.widths[o+1])\n                a = np.packbits(bits, axis=-1, bitorder='little').flatten()\n                ax[y][x].plot(a[:10000], linewidth=0.1)\n                if y == 3:\n                    hm = frame[:, cls.offsets[26]:cls.offsets[30]].reshape(-1, cls.widths[27])\n                    hm = np.packbits(hm, axis=-1, bitorder='little').flatten()\n                    for ham in range(8):\n                        h = np.histogram(a[np.where(hm == ham)]>>1, np.arange(17))\n                        ax[1][x].bar(np.arange(16), h[0])\n\n        fig.tight_layout()\n        plt.show()\n"
  },
  {
    "path": "teletext/charset.py",
    "content": "#\tName:   Map from Teletext G0 character set to Unicode\n#\tDate:   2018 April 20\n#\tAuthor: Rebecca Bettencourt <support@kreativekorp.com>\n\ng0 = {'default': {\n    0x20: chr(0x0020),  # SPACE\n    0x21: chr(0x0021),  # EXCLAMATION MARK\n    0x22: chr(0x0022),  # QUOTATION MARK\n    0x23: chr(0x00A3),  # POUND SIGN\n    0x24: chr(0x0024),  # DOLLAR SIGN\n    0x25: chr(0x0025),  # PERCENT SIGN\n    0x26: chr(0x0026),  # AMPERSAND\n    0x27: chr(0x0027),  # APOSTROPHE\n    0x28: chr(0x0028),  # LEFT PARENTHESIS\n    0x29: chr(0x0029),  # RIGHT PARENTHESIS\n    0x2A: chr(0x002A),  # ASTERISK\n    0x2B: chr(0x002B),  # PLUS SIGN\n    0x2C: chr(0x002C),  # COMMA\n    0x2D: chr(0x002D),  # HYPHEN-MINUS\n    0x2E: chr(0x002E),  # FULL STOP\n    0x2F: chr(0x002F),  # SOLIDUS\n    0x30: chr(0x0030),  # DIGIT ZERO\n    0x31: chr(0x0031),  # DIGIT ONE\n    0x32: chr(0x0032),  # DIGIT TWO\n    0x33: chr(0x0033),  # DIGIT THREE\n    0x34: chr(0x0034),  # DIGIT FOUR\n    0x35: chr(0x0035),  # DIGIT FIVE\n    0x36: chr(0x0036),  # DIGIT SIX\n    0x37: chr(0x0037),  # DIGIT SEVEN\n    0x38: chr(0x0038),  # DIGIT EIGHT\n    0x39: chr(0x0039),  # DIGIT NINE\n    0x3A: chr(0x003A),  # COLON\n    0x3B: chr(0x003B),  # SEMICOLON\n    0x3C: chr(0x003C),  # LESS-THAN SIGN\n    0x3D: chr(0x003D),  # EQUALS SIGN\n    0x3E: chr(0x003E),  # GREATER-THAN SIGN\n    0x3F: chr(0x003F),  # QUESTION MARK\n    0x40: chr(0x0040),  # COMMERCIAL AT\n    0x41: chr(0x0041),  # LATIN CAPITAL LETTER A\n    0x42: chr(0x0042),  # LATIN CAPITAL LETTER B\n    0x43: chr(0x0043),  # LATIN CAPITAL LETTER C\n    0x44: chr(0x0044),  # LATIN CAPITAL LETTER D\n    0x45: chr(0x0045),  # LATIN CAPITAL LETTER E\n    0x46: chr(0x0046),  # LATIN CAPITAL LETTER F\n    0x47: chr(0x0047),  # LATIN CAPITAL LETTER G\n    0x48: chr(0x0048),  # LATIN CAPITAL LETTER H\n    0x49: chr(0x0049),  # LATIN CAPITAL LETTER I\n    0x4A: chr(0x004A),  # LATIN CAPITAL LETTER J\n    0x4B: chr(0x004B),  # LATIN CAPITAL LETTER K\n    0x4C: chr(0x004C),  # LATIN CAPITAL LETTER L\n    0x4D: chr(0x004D),  # LATIN CAPITAL LETTER M\n    0x4E: chr(0x004E),  # LATIN CAPITAL LETTER N\n    0x4F: chr(0x004F),  # LATIN CAPITAL LETTER O\n    0x50: chr(0x0050),  # LATIN CAPITAL LETTER P\n    0x51: chr(0x0051),  # LATIN CAPITAL LETTER Q\n    0x52: chr(0x0052),  # LATIN CAPITAL LETTER R\n    0x53: chr(0x0053),  # LATIN CAPITAL LETTER S\n    0x54: chr(0x0054),  # LATIN CAPITAL LETTER T\n    0x55: chr(0x0055),  # LATIN CAPITAL LETTER U\n    0x56: chr(0x0056),  # LATIN CAPITAL LETTER V\n    0x57: chr(0x0057),  # LATIN CAPITAL LETTER W\n    0x58: chr(0x0058),  # LATIN CAPITAL LETTER X\n    0x59: chr(0x0059),  # LATIN CAPITAL LETTER Y\n    0x5A: chr(0x005A),  # LATIN CAPITAL LETTER Z\n    0x5B: chr(0x2190),  # LEFTWARDS ARROW\n    0x5C: chr(0x00BD),  # VULGAR FRACTION ONE HALF\n    0x5D: chr(0x2192),  # RIGHTWARDS ARROW\n    0x5E: chr(0x2191),  # UPWARDS ARROW\n    0x5F: chr(0x0023),  # NUMBER SIGN\n    #0x60: chr(0x2500),  # BOX DRAWINGS LIGHT HORIZONTAL\n    0x60: chr(0x2014),  # EM DASH\n    0x61: chr(0x0061),  # LATIN SMALL LETTER A\n    0x62: chr(0x0062),  # LATIN SMALL LETTER B\n    0x63: chr(0x0063),  # LATIN SMALL LETTER C\n    0x64: chr(0x0064),  # LATIN SMALL LETTER D\n    0x65: chr(0x0065),  # LATIN SMALL LETTER E\n    0x66: chr(0x0066),  # LATIN SMALL LETTER F\n    0x67: chr(0x0067),  # LATIN SMALL LETTER G\n    0x68: chr(0x0068),  # LATIN SMALL LETTER H\n    0x69: chr(0x0069),  # LATIN SMALL LETTER I\n    0x6A: chr(0x006A),  # LATIN SMALL LETTER J\n    0x6B: chr(0x006B),  # LATIN SMALL LETTER K\n    0x6C: chr(0x006C),  # LATIN SMALL LETTER L\n    0x6D: chr(0x006D),  # LATIN SMALL LETTER M\n    0x6E: chr(0x006E),  # LATIN SMALL LETTER N\n    0x6F: chr(0x006F),  # LATIN SMALL LETTER O\n    0x70: chr(0x0070),  # LATIN SMALL LETTER P\n    0x71: chr(0x0071),  # LATIN SMALL LETTER Q\n    0x72: chr(0x0072),  # LATIN SMALL LETTER R\n    0x73: chr(0x0073),  # LATIN SMALL LETTER S\n    0x74: chr(0x0074),  # LATIN SMALL LETTER T\n    0x75: chr(0x0075),  # LATIN SMALL LETTER U\n    0x76: chr(0x0076),  # LATIN SMALL LETTER V\n    0x77: chr(0x0077),  # LATIN SMALL LETTER W\n    0x78: chr(0x0078),  # LATIN SMALL LETTER X\n    0x79: chr(0x0079),  # LATIN SMALL LETTER Y\n    0x7A: chr(0x007A),  # LATIN SMALL LETTER Z\n    0x7B: chr(0x00BC),  # VULGAR FRACTION ONE QUARTER\n    0x7C: chr(0x2016),  # DOUBLE VERTICAL LINE\n    0x7D: chr(0x00BE),  # VULGAR FRACTION THREE QUARTERS\n    0x7E: chr(0x00F7),  # DIVISION SIGN\n    0x7F: chr(0x25A0),  # BLACK SQUARE\n}, 'cyr': {\n    0x20: chr(0x0020),  # SPACE\n    0x21: chr(0x0021),  # EXCLAMATION MARK\n    0x22: chr(0x0022),  # QUOTATION MARK\n    0x23: chr(0x00A3),  # POUND SIGN\n    0x24: chr(0x0024),  # DOLLAR SIGN\n    0x25: chr(0x0025),  # PERCENT SIGN\n    0x26: chr(0x044B),  # CYRILLIC SMALL LETTER YERU\n    0x27: chr(0x0027),  # APOSTROPHE\n    0x28: chr(0x0028),  # LEFT PARENTHESIS\n    0x29: chr(0x0029),  # RIGHT PARENTHESIS\n    0x2A: chr(0x002A),  # ASTERISK\n    0x2B: chr(0x002B),  # PLUS SIGN\n    0x2C: chr(0x002C),  # COMMA\n    0x2D: chr(0x002D),  # HYPHEN-MINUS\n    0x2E: chr(0x002E),  # FULL STOP\n    0x2F: chr(0x002F),  # SOLIDUS\n    0x30: chr(0x0030),  # DIGIT ZERO\n    0x31: chr(0x0031),  # DIGIT ONE\n    0x32: chr(0x0032),  # DIGIT TWO\n    0x33: chr(0x0033),  # DIGIT THREE\n    0x34: chr(0x0034),  # DIGIT FOUR\n    0x35: chr(0x0035),  # DIGIT FIVE\n    0x36: chr(0x0036),  # DIGIT SIX\n    0x37: chr(0x0037),  # DIGIT SEVEN\n    0x38: chr(0x0038),  # DIGIT EIGHT\n    0x39: chr(0x0039),  # DIGIT NINE\n    0x3A: chr(0x003A),  # COLON\n    0x3B: chr(0x003B),  # SEMICOLON\n    0x3C: chr(0x003C),  # LESS-THAN SIGN\n    0x3D: chr(0x003D),  # EQUALS SIGN\n    0x3E: chr(0x003E),  # GREATER-THAN SIGN\n    0x3F: chr(0x003F),  # QUESTION MARK\n    0x40: chr(0x042E),  # CYRILLIC CAPITAL LETTER YU\n    0x41: chr(0x0410),  # CYRILLIC CAPITAL LETTER A\n    0x42: chr(0x0411),  # CYRILLIC CAPITAL LETTER BE\n    0x43: chr(0x0426),  # CYRILLIC CAPITAL LETTER TSE\n    0x44: chr(0x0414),  # CYRILLIC CAPITAL LETTER DE\n    0x45: chr(0x0415),  # CYRILLIC CAPITAL LETTER IE\n    0x46: chr(0x0424),  # CYRILLIC CAPITAL LETTER EF\n    0x47: chr(0x0413),  # CYRILLIC CAPITAL LETTER GHE\n    0x48: chr(0x0425),  # CYRILLIC CAPITAL LETTER HA\n    0x49: chr(0x0418),  # CYRILLIC CAPITAL LETTER I\n    0x4A: chr(0x0419),  # CYRILLIC CAPITAL LETTER SHORT I\n    0x4B: chr(0x041A),  # CYRILLIC CAPITAL LETTER KA\n    0x4C: chr(0x041B),  # CYRILLIC CAPITAL LETTER EL\n    0x4D: chr(0x041C),  # CYRILLIC CAPITAL LETTER EМ\n    0x4E: chr(0x041D),  # CYRILLIC CAPITAL LETTER EN\n    0x4F: chr(0x041E),  # CYRILLIC CAPITAL LETTER O\n    0x50: chr(0x041F),  # CYRILLIC CAPITAL LETTER PE\n    0x51: chr(0x042F),  # CYRILLIC CAPITAL LETTER YA\n    0x52: chr(0x0420),  # CYRILLIC CAPITAL LETTER ER\n    0x53: chr(0x0421),  # CYRILLIC CAPITAL LETTER ES\n    0x54: chr(0x0422),  # CYRILLIC CAPITAL LETTER TE\n    0x55: chr(0x0423),  # CYRILLIC CAPITAL LETTER U\n    0x56: chr(0x0416),  # CYRILLIC CAPITAL LETTER ZHE\n    0x57: chr(0x0412),  # CYRILLIC CAPITAL LETTER BE\n    0x58: chr(0x042C),  # CYRILLIC CAPITAL LETTER SOFT SIGN\n    0x59: chr(0x042A),  # CYRILLIC CAPITAL LETTER HARD SIGN\n    0x5A: chr(0x0417),  # CYRILLIC CAPITAL LETTER ZE\n    0x5B: chr(0x0428),  # CYRILLIC CAPITAL LETTER SHA\n    0x5C: chr(0x042D),  # CYRILLIC CAPITAL LETTER E\n    0x5D: chr(0x0429),  # CYRILLIC CAPITAL LETTER SHCHA\n    0x5E: chr(0x0427),  # CYRILLIC CAPITAL LETTER CHA\n    0x5F: chr(0x042B),  # CYRILLIC CAPITAL LETTER YERU\n    0x60: chr(0x044E),  # CYRILLIC SMALL LETTER YU\n    0x61: chr(0x0430),  # CYRILLIC SMALL LETTER A\n    0x62: chr(0x0431),  # CYRILLIC SMALL LETTER BE\n    0x63: chr(0x0446),  # CYRILLIC SMALL LETTER TSE\n    0x64: chr(0x0434),  # CYRILLIC SMALL LETTER DE\n    0x65: chr(0x0435),  # CYRILLIC SMALL LETTER IE\n    0x66: chr(0x0444),  # CYRILLIC SMALL LETTER EF\n    0x67: chr(0x0433),  # CYRILLIC SMALL LETTER GHE\n    0x68: chr(0x0445),  # CYRILLIC SMALL LETTER HA\n    0x69: chr(0x0438),  # CYRILLIC SMALL LETTER I\n    0x6A: chr(0x0439),  # CYRILLIC SMALL LETTER SHORT I\n    0x6B: chr(0x043A),  # CYRILLIC SMALL LETTER KA\n    0x6C: chr(0x043B),  # CYRILLIC SMALL LETTER EL\n    0x6D: chr(0x043C),  # CYRILLIC SMALL LETTER EM\n    0x6E: chr(0x043D),  # CYRILLIC SMALL LETTER EN\n    0x6F: chr(0x043E),  # CYRILLIC SMALL LETTER O\n    0x70: chr(0x043F),  # CYRILLIC SMALL LETTER PE\n    0x71: chr(0x044F),  # CYRILLIC SMALL LETTER YA\n    0x72: chr(0x0440),  # CYRILLIC SMALL LETTER ER\n    0x73: chr(0x0441),  # CYRILLIC SMALL LETTER ES\n    0x74: chr(0x0442),  # CYRILLIC SMALL LETTER TE\n    0x75: chr(0x0443),  # CYRILLIC SMALL LETTER U\n    0x76: chr(0x0436),  # CYRILLIC SMALL LETTER ZHE\n    0x77: chr(0x0432),  # CYRILLIC SMALL LETTER BE\n    0x78: chr(0x044C),  # CYRILLIC SMALL LETTER SOFT SIGN\n    0x79: chr(0x044A),  # CYRILLIC SMALL LETTER HARD SIGN\n    0x7A: chr(0x0437),  # CYRILLIC SMALL LETTER ZE\n    0x7B: chr(0x0448),  # CYRILLIC SMALL LETTER SHA\n    0x7C: chr(0x044D),  # CYRILLIC SMALL LETTER E\n    0x7D: chr(0x0449),  # CYRILLIC SMALL LETTER SHCHA\n    0x7E: chr(0x0447),  # CYRILLIC SMALL LETTER CHE\n    0x7F: chr(0x25A0),  # BLACK SQUARE\n\n    # Swedish national subset (replaces characters 0x40, 0x5B-0x5F, 0x60, 0x7B-0x7E)\n}, 'swe': {\n    0x20: chr(0x0020),  # SPACE\n    0x21: chr(0x0021),  # EXCLAMATION MARK\n    0x22: chr(0x0022),  # QUOTATION MARK\n    0x23: chr(0x00A3),  # POUND SIGN\n    0x24: chr(0x0024),  # DOLLAR SIGN\n    0x25: chr(0x0025),  # PERCENT SIGN\n    0x26: chr(0x0026),  # AMPERSAND\n    0x27: chr(0x0027),  # APOSTROPHE\n    0x28: chr(0x0028),  # LEFT PARENTHESIS\n    0x29: chr(0x0029),  # RIGHT PARENTHESIS\n    0x2A: chr(0x002A),  # ASTERISK\n    0x2B: chr(0x002B),  # PLUS SIGN\n    0x2C: chr(0x002C),  # COMMA\n    0x2D: chr(0x002D),  # HYPHEN-MINUS\n    0x2E: chr(0x002E),  # FULL STOP\n    0x2F: chr(0x002F),  # SOLIDUS\n    0x30: chr(0x0030),  # DIGIT ZERO\n    0x31: chr(0x0031),  # DIGIT ONE\n    0x32: chr(0x0032),  # DIGIT TWO\n    0x33: chr(0x0033),  # DIGIT THREE\n    0x34: chr(0x0034),  # DIGIT FOUR\n    0x35: chr(0x0035),  # DIGIT FIVE\n    0x36: chr(0x0036),  # DIGIT SIX\n    0x37: chr(0x0037),  # DIGIT SEVEN\n    0x38: chr(0x0038),  # DIGIT EIGHT\n    0x39: chr(0x0039),  # DIGIT NINE\n    0x3A: chr(0x003A),  # COLON\n    0x3B: chr(0x003B),  # SEMICOLON\n    0x3C: chr(0x003C),  # LESS-THAN SIGN\n    0x3D: chr(0x003D),  # EQUALS SIGN\n    0x3E: chr(0x003E),  # GREATER-THAN SIGN\n    0x3F: chr(0x003F),  # QUESTION MARK\n    0x40: chr(0x00C9),  # CAPITAL E-ACUTE\n    0x41: chr(0x0041),  # LATIN CAPITAL LETTER A\n    0x42: chr(0x0042),  # LATIN CAPITAL LETTER B\n    0x43: chr(0x0043),  # LATIN CAPITAL LETTER C\n    0x44: chr(0x0044),  # LATIN CAPITAL LETTER D\n    0x45: chr(0x0045),  # LATIN CAPITAL LETTER E\n    0x46: chr(0x0046),  # LATIN CAPITAL LETTER F\n    0x47: chr(0x0047),  # LATIN CAPITAL LETTER G\n    0x48: chr(0x0048),  # LATIN CAPITAL LETTER H\n    0x49: chr(0x0049),  # LATIN CAPITAL LETTER I\n    0x4A: chr(0x004A),  # LATIN CAPITAL LETTER J\n    0x4B: chr(0x004B),  # LATIN CAPITAL LETTER K\n    0x4C: chr(0x004C),  # LATIN CAPITAL LETTER L\n    0x4D: chr(0x004D),  # LATIN CAPITAL LETTER M\n    0x4E: chr(0x004E),  # LATIN CAPITAL LETTER N\n    0x4F: chr(0x004F),  # LATIN CAPITAL LETTER O\n    0x50: chr(0x0050),  # LATIN CAPITAL LETTER P\n    0x51: chr(0x0051),  # LATIN CAPITAL LETTER Q\n    0x52: chr(0x0052),  # LATIN CAPITAL LETTER R\n    0x53: chr(0x0053),  # LATIN CAPITAL LETTER S\n    0x54: chr(0x0054),  # LATIN CAPITAL LETTER T\n    0x55: chr(0x0055),  # LATIN CAPITAL LETTER U\n    0x56: chr(0x0056),  # LATIN CAPITAL LETTER V\n    0x57: chr(0x0057),  # LATIN CAPITAL LETTER W\n    0x58: chr(0x0058),  # LATIN CAPITAL LETTER X\n    0x59: chr(0x0059),  # LATIN CAPITAL LETTER Y\n    0x5A: chr(0x005A),  # LATIN CAPITAL LETTER Z\n    0x5B: chr(0x00C4),  # LATIN CAPITAL A WITH DIAERESIS\n    0x5C: chr(0x00D6),  # LATIN CAPITAL O WITH DIAERESIS \n    0x5D: chr(0x00C5),  # LATIN CAPITAL A WITH OVERRING\n    0x5E: chr(0x00DC),  # LATIN CAPITAL U WITH DIAERESIS\n    0x5F: chr(0x005F),  # UNDERSCORE\n    #0x60: chr(0x2500),  # BOX DRAWINGS LIGHT HORIZONTAL\n    0x60: chr(0x00E9),  # LOWER-CASE E-ACUTE\n    0x61: chr(0x0061),  # LATIN SMALL LETTER A\n    0x62: chr(0x0062),  # LATIN SMALL LETTER B\n    0x63: chr(0x0063),  # LATIN SMALL LETTER C\n    0x64: chr(0x0064),  # LATIN SMALL LETTER D\n    0x65: chr(0x0065),  # LATIN SMALL LETTER E\n    0x66: chr(0x0066),  # LATIN SMALL LETTER F\n    0x67: chr(0x0067),  # LATIN SMALL LETTER G\n    0x68: chr(0x0068),  # LATIN SMALL LETTER H\n    0x69: chr(0x0069),  # LATIN SMALL LETTER I\n    0x6A: chr(0x006A),  # LATIN SMALL LETTER J\n    0x6B: chr(0x006B),  # LATIN SMALL LETTER K\n    0x6C: chr(0x006C),  # LATIN SMALL LETTER L\n    0x6D: chr(0x006D),  # LATIN SMALL LETTER M\n    0x6E: chr(0x006E),  # LATIN SMALL LETTER N\n    0x6F: chr(0x006F),  # LATIN SMALL LETTER O\n    0x70: chr(0x0070),  # LATIN SMALL LETTER P\n    0x71: chr(0x0071),  # LATIN SMALL LETTER Q\n    0x72: chr(0x0072),  # LATIN SMALL LETTER R\n    0x73: chr(0x0073),  # LATIN SMALL LETTER S\n    0x74: chr(0x0074),  # LATIN SMALL LETTER T\n    0x75: chr(0x0075),  # LATIN SMALL LETTER U\n    0x76: chr(0x0076),  # LATIN SMALL LETTER V\n    0x77: chr(0x0077),  # LATIN SMALL LETTER W\n    0x78: chr(0x0078),  # LATIN SMALL LETTER X\n    0x79: chr(0x0079),  # LATIN SMALL LETTER Y\n    0x7A: chr(0x007A),  # LATIN SMALL LETTER Z\n    0x7B: chr(0x00E4),  # LATIN SMALL A WITH DIAERESIS\n    0x7C: chr(0x00F6),  # LATIN SMALL O WITH DIAERESIS\n    0x7D: chr(0x00E5),  # LATIN SMALL A WITH OVERRING\n    0x7E: chr(0x00FC),  # LATIN SMALL U WITH DIAERESIS\n    0x7F: chr(0x25A0),  # BLACK SQUARE\n\n}}\n\n#       Name:   Map from Teletext G1 character set to Unicode\n#       Date:   2018 April 20\n#       Author: Rebecca Bettencourt <support@kreativekorp.com>\n\ng1 = {\n    0x20: chr(0x00A0), # NO-BREAK SPACE; unification of EMPTY BLOCK SEXTANT\n    0x21: chr(0x1FB00), # BLOCK SEXTANT-1\n    0x22: chr(0x1FB01), # BLOCK SEXTANT-2\n    0x23: chr(0x1FB02), # BLOCK SEXTANT-12\n    0x24: chr(0x1FB03), # BLOCK SEXTANT-3\n    0x25: chr(0x1FB04), # BLOCK SEXTANT-13\n    0x26: chr(0x1FB05), # BLOCK SEXTANT-23\n    0x27: chr(0x1FB06), # BLOCK SEXTANT-123\n    0x28: chr(0x1FB07), # BLOCK SEXTANT-4\n    0x29: chr(0x1FB08), # BLOCK SEXTANT-14\n    0x2A: chr(0x1FB09), # BLOCK SEXTANT-24\n    0x2B: chr(0x1FB0A), # BLOCK SEXTANT-124\n    0x2C: chr(0x1FB0B), # BLOCK SEXTANT-34\n    0x2D: chr(0x1FB0C), # BLOCK SEXTANT-134\n    0x2E: chr(0x1FB0D), # BLOCK SEXTANT-234\n    0x2F: chr(0x1FB0E), # BLOCK SEXTANT-1234\n    0x30: chr(0x1FB0F), # BLOCK SEXTANT-5\n    0x31: chr(0x1FB10), # BLOCK SEXTANT-15\n    0x32: chr(0x1FB11), # BLOCK SEXTANT-25\n    0x33: chr(0x1FB12), # BLOCK SEXTANT-125\n    0x34: chr(0x1FB13), # BLOCK SEXTANT-35\n    0x35: chr(0x258C), # LEFT HALF BLOCK; unification of BLOCK SEXTANT-135\n    0x36: chr(0x1FB14), # BLOCK SEXTANT-235\n    0x37: chr(0x1FB15), # BLOCK SEXTANT-1235\n    0x38: chr(0x1FB16), # BLOCK SEXTANT-45\n    0x39: chr(0x1FB17), # BLOCK SEXTANT-145\n    0x3A: chr(0x1FB18), # BLOCK SEXTANT-245\n    0x3B: chr(0x1FB19), # BLOCK SEXTANT-1245\n    0x3C: chr(0x1FB1A), # BLOCK SEXTANT-345\n    0x3D: chr(0x1FB1B), # BLOCK SEXTANT-1345\n    0x3E: chr(0x1FB1C), # BLOCK SEXTANT-2345\n    0x3F: chr(0x1FB1D), # BLOCK SEXTANT-12345\n    0x40: chr(0x0040), # COMMERCIAL AT\n    0x41: chr(0x0041), # LATIN CAPITAL LETTER A\n    0x42: chr(0x0042), # LATIN CAPITAL LETTER B\n    0x43: chr(0x0043), # LATIN CAPITAL LETTER C\n    0x44: chr(0x0044), # LATIN CAPITAL LETTER D\n    0x45: chr(0x0045), # LATIN CAPITAL LETTER E\n    0x46: chr(0x0046), # LATIN CAPITAL LETTER F\n    0x47: chr(0x0047), # LATIN CAPITAL LETTER G\n    0x48: chr(0x0048), # LATIN CAPITAL LETTER H\n    0x49: chr(0x0049), # LATIN CAPITAL LETTER I\n    0x4A: chr(0x004A), # LATIN CAPITAL LETTER J\n    0x4B: chr(0x004B), # LATIN CAPITAL LETTER K\n    0x4C: chr(0x004C), # LATIN CAPITAL LETTER L\n    0x4D: chr(0x004D), # LATIN CAPITAL LETTER M\n    0x4E: chr(0x004E), # LATIN CAPITAL LETTER N\n    0x4F: chr(0x004F), # LATIN CAPITAL LETTER O\n    0x50: chr(0x0050), # LATIN CAPITAL LETTER P\n    0x51: chr(0x0051), # LATIN CAPITAL LETTER Q\n    0x52: chr(0x0052), # LATIN CAPITAL LETTER R\n    0x53: chr(0x0053), # LATIN CAPITAL LETTER S\n    0x54: chr(0x0054), # LATIN CAPITAL LETTER T\n    0x55: chr(0x0055), # LATIN CAPITAL LETTER U\n    0x56: chr(0x0056), # LATIN CAPITAL LETTER V\n    0x57: chr(0x0057), # LATIN CAPITAL LETTER W\n    0x58: chr(0x0058), # LATIN CAPITAL LETTER X\n    0x59: chr(0x0059), # LATIN CAPITAL LETTER Y\n    0x5A: chr(0x005A), # LATIN CAPITAL LETTER Z\n    0x5B: chr(0x2190), # LEFTWARDS ARROW\n    0x5C: chr(0x00BD), # VULGAR FRACTION ONE HALF\n    0x5D: chr(0x2192), # RIGHTWARDS ARROW\n    0x5E: chr(0x2191), # UPWARDS ARROW\n    0x5F: chr(0x0023), # NUMBER SIGN\n    0x60: chr(0x1FB1E), # BLOCK SEXTANT-6\n    0x61: chr(0x1FB1F), # BLOCK SEXTANT-16\n    0x62: chr(0x1FB20), # BLOCK SEXTANT-26\n    0x63: chr(0x1FB21), # BLOCK SEXTANT-126\n    0x64: chr(0x1FB22), # BLOCK SEXTANT-36\n    0x65: chr(0x1FB23), # BLOCK SEXTANT-136\n    0x66: chr(0x1FB24), # BLOCK SEXTANT-236\n    0x67: chr(0x1FB25), # BLOCK SEXTANT-1236\n    0x68: chr(0x1FB26), # BLOCK SEXTANT-46\n    0x69: chr(0x1FB27), # BLOCK SEXTANT-146\n    0x6A: chr(0x2590), # RIGHT HALF BLOCK; unification of BLOCK SEXTANT-246\n    0x6B: chr(0x1FB28), # BLOCK SEXTANT-1246\n    0x6C: chr(0x1FB29), # BLOCK SEXTANT-346\n    0x6D: chr(0x1FB2A), # BLOCK SEXTANT-1346\n    0x6E: chr(0x1FB2B), # BLOCK SEXTANT-2346\n    0x6F: chr(0x1FB2C), # BLOCK SEXTANT-12346\n    0x70: chr(0x1FB2D), # BLOCK SEXTANT-56\n    0x71: chr(0x1FB2E), # BLOCK SEXTANT-156\n    0x72: chr(0x1FB2F), # BLOCK SEXTANT-256\n    0x73: chr(0x1FB30), # BLOCK SEXTANT-1256\n    0x74: chr(0x1FB31), # BLOCK SEXTANT-356\n    0x75: chr(0x1FB32), # BLOCK SEXTANT-1356\n    0x76: chr(0x1FB33), # BLOCK SEXTANT-2356\n    0x77: chr(0x1FB34), # BLOCK SEXTANT-12356\n    0x78: chr(0x1FB35), # BLOCK SEXTANT-456\n    0x79: chr(0x1FB36), # BLOCK SEXTANT-1456\n    0x7A: chr(0x1FB37), # BLOCK SEXTANT-2456\n    0x7B: chr(0x1FB38), # BLOCK SEXTANT-12456\n    0x7C: chr(0x1FB39), # BLOCK SEXTANT-3456\n    0x7D: chr(0x1FB3A), # BLOCK SEXTANT-13456\n    0x7E: chr(0x1FB3B), # BLOCK SEXTANT-23456\n    0x7F: chr(0x2588), # FULL BLOCK; unification of BLOCK SEXTANT-123456\n}\n\n\n#       Name:   Map from Teletext G2 character set to Unicode\n#       Date:   2018 April 20\n#       Author: Rebecca Bettencourt <support@kreativekorp.com>\n\ng2 = {\n    0x20: chr(0x0020), # SPACE\n    0x21: chr(0x00A1), # INVERTED EXCLAMATION MARK\n    0x22: chr(0x00A2), # CENT SIGN\n    0x23: chr(0x00A3), # POUND SIGN\n    0x24: chr(0x0024), # DOLLAR SIGN\n    0x25: chr(0x00A5), # YEN SIGN\n    0x26: chr(0x0023), # NUMBER SIGN\n    0x27: chr(0x00A7), # SECTION SIGN\n    0x28: chr(0x00A4), # CURRENCY SIGN\n    0x29: chr(0x2018), # LEFT SINGLE QUOTATION MARK\n    0x2A: chr(0x201C), # LEFT DOUBLE QUOTATION MARK\n    0x2B: chr(0x00AB), # LEFT-POINTING DOUBLE ANGLE QUOTATION MARK\n    0x2C: chr(0x2190), # LEFTWARDS ARROW\n    0x2D: chr(0x2191), # UPWARDS ARROW\n    0x2E: chr(0x2192), # RIGHTWARDS ARROW\n    0x2F: chr(0x2193), # DOWNWARDS ARROW\n    0x30: chr(0x00B0), # DEGREE SIGN\n    0x31: chr(0x00B1), # PLUS-MINUS SIGN\n    0x32: chr(0x00B2), # SUPERSCRIPT TWO\n    0x33: chr(0x00B3), # SUPERSCRIPT THREE\n    0x34: chr(0x00D7), # MULTIPLICATION SIGN\n    0x35: chr(0x00B5), # MICRO SIGN\n    0x36: chr(0x00B6), # PILCROW SIGN\n    0x37: chr(0x00B7), # MIDDLE DOT\n    0x38: chr(0x00F7), # DIVISION SIGN\n    0x39: chr(0x2019), # RIGHT SINGLE QUOTATION MARK\n    0x3A: chr(0x201D), # RIGHT DOUBLE QUOTATION MARK\n    0x3B: chr(0x00BB), # RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK\n    0x3C: chr(0x00BC), # VULGAR FRACTION ONE QUARTER\n    0x3D: chr(0x00BD), # VULGAR FRACTION ONE HALF\n    0x3E: chr(0x00BE), # VULGAR FRACTION THREE QUARTERS\n    0x3F: chr(0x00BF), # INVERTED QUESTION MARK\n    0x40: chr(0x00A0), # NO-BREAK SPACE\n    0x41: chr(0x02CB), # MODIFIER LETTER GRAVE ACCENT\n    0x42: chr(0x02CA), # MODIFIER LETTER ACUTE ACCENT\n    0x43: chr(0x02C6), # MODIFIER LETTER CIRCUMFLEX ACCENT\n    0x44: chr(0x02DC), # SMALL TILDE\n    0x45: chr(0x02C9), # MODIFIER LETTER MACRON\n    0x46: chr(0x02D8), # BREVE\n    0x47: chr(0x02D9), # DOT ABOVE\n    0x48: chr(0x00A8), # DIAERESIS\n    0x49: chr(0x02CC), # MODIFIER LETTER LOW VERTICAL LINE\n    0x4A: chr(0x02DA), # RING ABOVE\n    0x4B: chr(0x00B8), # CEDILLA\n    0x4C: chr(0x005F), # LOW LINE\n    0x4D: chr(0x02DD), # DOUBLE ACUTE ACCENT\n    0x4E: chr(0x02DB), # OGONEK\n    0x4F: chr(0x02C7), # CARON\n    0x50: chr(0x2500), # BOX DRAWINGS LIGHT HORIZONTAL\n    0x51: chr(0x00B9), # SUPERSCRIPT ONE\n    0x52: chr(0x00AE), # REGISTERED SIGN\n    0x53: chr(0x00A9), # COPYRIGHT SIGN\n    0x54: chr(0x2122), # TRADE MARK SIGN\n    0x55: chr(0x266A), # EIGHTH NOTE\n    0x56: chr(0x20A0), # EURO-CURRENCY SIGN\n    0x57: chr(0x2030), # PER MILLE SIGN\n    0x58: chr(0x03B1), # GREEK SMALL LETTER ALPHA\n    0x5C: chr(0x215B), # VULGAR FRACTION ONE EIGHTH\n    0x5D: chr(0x215C), # VULGAR FRACTION THREE EIGHTHS\n    0x5E: chr(0x215D), # VULGAR FRACTION FIVE EIGHTHS\n    0x5F: chr(0x215E), # VULGAR FRACTION SEVEN EIGHTHS\n    0x60: chr(0x03A9), # GREEK CAPITAL LETTER OMEGA\n    0x61: chr(0x00C6), # LATIN CAPITAL LETTER AE\n    0x62: chr(0x00D0), # LATIN CAPITAL LETTER ETH\n    0x63: chr(0x00AA), # FEMININE ORDINAL INDICATOR\n    0x64: chr(0x0126), # LATIN CAPITAL LETTER H WITH STROKE\n    0x66: chr(0x0132), # LATIN CAPITAL LIGATURE IJ\n    0x67: chr(0x013F), # LATIN CAPITAL LETTER L WITH MIDDLE DOT\n    0x68: chr(0x0141), # LATIN CAPITAL LETTER L WITH STROKE\n    0x69: chr(0x00D8), # LATIN CAPITAL LETTER O WITH STROKE\n    0x6A: chr(0x0152), # LATIN CAPITAL LIGATURE OE\n    0x6B: chr(0x00BA), # MASCULINE ORDINAL INDICATOR\n    0x6C: chr(0x00DE), # LATIN CAPITAL LETTER THORN\n    0x6D: chr(0x0166), # LATIN CAPITAL LETTER T WITH STROKE\n    0x6E: chr(0x014A), # LATIN CAPITAL LETTER ENG\n    0x6F: chr(0x0149), # LATIN SMALL LETTER N PRECEDED BY APOSTROPHE\n    0x70: chr(0x0138), # LATIN SMALL LETTER KRA\n    0x71: chr(0x00E6), # LATIN SMALL LETTER AE\n    0x72: chr(0x0111), # LATIN SMALL LETTER D WITH STROKE\n    0x73: chr(0x00F0), # LATIN SMALL LETTER ETH\n    0x74: chr(0x0127), # LATIN SMALL LETTER H WITH STROKE\n    0x75: chr(0x0131), # LATIN SMALL LETTER DOTLESS I\n    0x76: chr(0x0133), # LATIN SMALL LIGATURE IJ\n    0x77: chr(0x0140), # LATIN SMALL LETTER L WITH MIDDLE DOT\n    0x78: chr(0x0142), # LATIN SMALL LETTER L WITH STROKE\n    0x79: chr(0x00F8), # LATIN SMALL LETTER O WITH STROKE\n    0x7A: chr(0x0153), # LATIN SMALL LIGATURE OE\n    0x7B: chr(0x00DF), # LATIN SMALL LETTER SHARP S\n    0x7C: chr(0x00FE), # LATIN SMALL LETTER THORN\n    0x7D: chr(0x0167), # LATIN SMALL LETTER T WITH STROKE\n    0x7E: chr(0x014B), # LATIN SMALL LETTER ENG\n    0x7F: chr(0x25A0), # BLACK SQUARE\n}\n\n\n#       Name:   Map from Teletext G3 character set to Unicode\n#       Date:   2018 April 20\n#       Author: Rebecca Bettencourt <support@kreativekorp.com>\n\ng3 = {\n    0x20: chr(0x1FB3C), # LOWER LEFT BLOCK DIAGONAL LOWER MIDDLE LEFT TO LOWER CENTRE\n    0x21: chr(0x1FB3D), # LOWER LEFT BLOCK DIAGONAL LOWER MIDDLE LEFT TO LOWER RIGHT\n    0x22: chr(0x1FB3E), # LOWER LEFT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER CENTRE\n    0x23: chr(0x1FB3F), # LOWER LEFT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER RIGHT\n    0x24: chr(0x1FB40), # LOWER LEFT BLOCK DIAGONAL UPPER LEFT TO LOWER CENTRE\n    0x25: chr(0x25E3), # BLACK LOWER LEFT TRIANGLE\n    0x26: chr(0x1FB41), # LOWER RIGHT BLOCK DIAGONAL UPPER MIDDLE LEFT TO UPPER CENTRE\n    0x27: chr(0x1FB42), # LOWER RIGHT BLOCK DIAGONAL UPPER MIDDLE LEFT TO UPPER RIGHT\n    0x28: chr(0x1FB43), # LOWER RIGHT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER CENTRE\n    0x29: chr(0x1FB44), # LOWER RIGHT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER RIGHT\n    0x2A: chr(0x1FB45), # LOWER RIGHT BLOCK DIAGONAL LOWER LEFT TO UPPER CENTRE\n    0x2B: chr(0x1FB46), # LOWER RIGHT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER MIDDLE RIGHT\n    0x2C: chr(0x1FB68), # UPPER AND RIGHT AND LOWER TRIANGULAR THREE QUARTERS BLOCK\n    0x2D: chr(0x1FB69), # LEFT AND LOWER AND RIGHT TRIANGULAR THREE QUARTERS BLOCK\n    0x2E: chr(0x1FB70), # VERTICAL ONE EIGHTH BLOCK-2\n    0x2F: chr(0x2592), # MEDIUM SHADE\n    0x30: chr(0x1FB47), # LOWER RIGHT BLOCK DIAGONAL LOWER CENTRE TO LOWER MIDDLE RIGHT\n    0x31: chr(0x1FB48), # LOWER RIGHT BLOCK DIAGONAL LOWER LEFT TO LOWER MIDDLE RIGHT\n    0x32: chr(0x1FB49), # LOWER RIGHT BLOCK DIAGONAL LOWER CENTRE TO UPPER MIDDLE RIGHT\n    0x33: chr(0x1FB4A), # LOWER RIGHT BLOCK DIAGONAL LOWER LEFT TO UPPER MIDDLE RIGHT\n    0x34: chr(0x1FB4B), # LOWER RIGHT BLOCK DIAGONAL LOWER CENTRE TO UPPER RIGHT\n    0x35: chr(0x25E2), # BLACK LOWER RIGHT TRIANGLE\n    0x36: chr(0x1FB4C), # LOWER LEFT BLOCK DIAGONAL UPPER CENTRE TO UPPER MIDDLE RIGHT\n    0x37: chr(0x1FB4D), # LOWER LEFT BLOCK DIAGONAL UPPER LEFT TO UPPER MIDDLE RIGHT\n    0x38: chr(0x1FB4E), # LOWER LEFT BLOCK DIAGONAL UPPER CENTRE TO LOWER MIDDLE RIGHT\n    0x39: chr(0x1FB4F), # LOWER LEFT BLOCK DIAGONAL UPPER LEFT TO LOWER MIDDLE RIGHT\n    0x3A: chr(0x1FB50), # LOWER LEFT BLOCK DIAGONAL UPPER CENTRE TO LOWER RIGHT\n    0x3B: chr(0x1FB51), # LOWER LEFT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER MIDDLE RIGHT\n    0x3C: chr(0x1FB6A), # UPPER AND LEFT AND LOWER TRIANGULAR THREE QUARTERS BLOCK\n    0x3D: chr(0x1FB6B), # LEFT AND UPPER AND RIGHT TRIANGULAR THREE QUARTERS BLOCK\n    0x3E: chr(0x1FB75), # VERTICAL ONE EIGHTH BLOCK-7\n    0x3F: chr(0x2588), # FULL BLOCK\n    0x40: chr(0x2537), # BOX DRAWINGS UP LIGHT AND HORIZONTAL HEAVY\n    0x41: chr(0x252F), # BOX DRAWINGS DOWN LIGHT AND HORIZONTAL HEAVY\n    0x42: chr(0x251D), # BOX DRAWINGS VERTICAL LIGHT AND RIGHT HEAVY\n    0x43: chr(0x2525), # BOX DRAWINGS VERTICAL LIGHT AND LEFT HEAVY\n    0x44: chr(0x1FBA4), # BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE LEFT TO LOWER CENTRE\n    0x45: chr(0x1FBA5), # BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE RIGHT TO LOWER CENTRE\n    0x46: chr(0x1FBA6), # BOX DRAWINGS LIGHT DIAGONAL MIDDLE LEFT TO LOWER CENTRE TO MIDDLE RIGHT\n    0x47: chr(0x1FBA7), # BOX DRAWINGS LIGHT DIAGONAL MIDDLE LEFT TO UPPER CENTRE TO MIDDLE RIGHT\n    0x48: chr(0x1FBA0), # BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE LEFT\n    0x49: chr(0x1FBA1), # BOX DRAWINGS LIGHT DIAGONAL UPPER CENTRE TO MIDDLE RIGHT\n    0x4A: chr(0x1FBA2), # BOX DRAWINGS LIGHT DIAGONAL MIDDLE LEFT TO LOWER CENTRE\n    0x4B: chr(0x1FBA3), # BOX DRAWINGS LIGHT DIAGONAL MIDDLE RIGHT TO LOWER CENTRE\n    0x4C: chr(0x253F), # BOX DRAWINGS VERTICAL LIGHT AND HORIZONTAL HEAVY\n    0x4D: chr(0x2022), # BULLET\n    0x4E: chr(0x25CF), # BLACK CIRCLE\n    0x4F: chr(0x25CB), # WHITE CIRCLE\n    0x50: chr(0x2502), # BOX DRAWINGS LIGHT VERTICAL\n    0x51: chr(0x2500), # BOX DRAWINGS LIGHT HORIZONTAL\n    0x52: chr(0x250C), # BOX DRAWINGS LIGHT DOWN AND RIGHT\n    0x53: chr(0x2510), # BOX DRAWINGS LIGHT DOWN AND LEFT\n    0x54: chr(0x2514), # BOX DRAWINGS LIGHT UP AND RIGHT\n    0x55: chr(0x2518), # BOX DRAWINGS LIGHT UP AND LEFT\n    0x56: chr(0x251C), # BOX DRAWINGS LIGHT VERTICAL AND RIGHT\n    0x57: chr(0x2524), # BOX DRAWINGS LIGHT VERTICAL AND LEFT\n    0x58: chr(0x252C), # BOX DRAWINGS LIGHT DOWN AND HORIZONTAL\n    0x59: chr(0x2534), # BOX DRAWINGS LIGHT UP AND HORIZONTAL\n    0x5A: chr(0x253C), # BOX DRAWINGS LIGHT VERTICAL AND HORIZONTAL\n    0x5B: chr(0x2B62), # RIGHTWARDS TRIANGLE-HEADED ARROW\n    0x5C: chr(0x2B60), # LEFTWARDS TRIANGLE-HEADED ARROW\n    0x5D: chr(0x2B61), # UPWARDS TRIANGLE-HEADED ARROW\n    0x5E: chr(0x2B63), # DOWNWARDS TRIANGLE-HEADED ARROW\n    0x5F: chr(0x00A0), # NO-BREAK SPACE\n    0x60: chr(0x1FB52), # UPPER RIGHT BLOCK DIAGONAL LOWER MIDDLE LEFT TO LOWER CENTRE\n    0x61: chr(0x1FB53), # UPPER RIGHT BLOCK DIAGONAL LOWER MIDDLE LEFT TO LOWER RIGHT\n    0x62: chr(0x1FB54), # UPPER RIGHT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER CENTRE\n    0x63: chr(0x1FB55), # UPPER RIGHT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER RIGHT\n    0x64: chr(0x1FB56), # UPPER RIGHT BLOCK DIAGONAL UPPER LEFT TO LOWER CENTRE\n    0x65: chr(0x25E5), # BLACK UPPER RIGHT TRIANGLE\n    0x66: chr(0x1FB57), # UPPER LEFT BLOCK DIAGONAL UPPER MIDDLE LEFT TO UPPER CENTRE\n    0x67: chr(0x1FB58), # UPPER LEFT BLOCK DIAGONAL UPPER MIDDLE LEFT TO UPPER RIGHT\n    0x68: chr(0x1FB59), # UPPER LEFT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER CENTRE\n    0x69: chr(0x1FB5A), # UPPER LEFT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER RIGHT\n    0x6A: chr(0x1FB5B), # UPPER LEFT BLOCK DIAGONAL LOWER LEFT TO UPPER CENTRE\n    0x6B: chr(0x1FB5C), # UPPER LEFT BLOCK DIAGONAL LOWER MIDDLE LEFT TO UPPER MIDDLE RIGHT\n    0x6C: chr(0x1FB6C), # LEFT TRIANGULAR ONE QUARTER BLOCK\n    0x6D: chr(0x1FB6D), # UPPER TRIANGULAR ONE QUARTER BLOCK\n    0x70: chr(0x1FB5D), # UPPER LEFT BLOCK DIAGONAL LOWER CENTRE TO LOWER MIDDLE RIGHT\n    0x71: chr(0x1FB5E), # UPPER LEFT BLOCK DIAGONAL LOWER LEFT TO LOWER MIDDLE RIGHT\n    0x72: chr(0x1FB5F), # UPPER LEFT BLOCK DIAGONAL LOWER CENTRE TO UPPER MIDDLE RIGHT\n    0x73: chr(0x1FB60), # UPPER LEFT BLOCK DIAGONAL LOWER LEFT TO UPPER MIDDLE RIGHT\n    0x74: chr(0x1FB61), # UPPER LEFT BLOCK DIAGONAL LOWER CENTRE TO UPPER RIGHT\n    0x75: chr(0x25E4), # BLACK UPPER LEFT TRIANGLE\n    0x76: chr(0x1FB62), # UPPER RIGHT BLOCK DIAGONAL UPPER CENTRE TO UPPER MIDDLE RIGHT\n    0x77: chr(0x1FB63), # UPPER RIGHT BLOCK DIAGONAL UPPER LEFT TO UPPER MIDDLE RIGHT\n    0x78: chr(0x1FB64), # UPPER RIGHT BLOCK DIAGONAL UPPER CENTRE TO LOWER MIDDLE RIGHT\n    0x79: chr(0x1FB65), # UPPER RIGHT BLOCK DIAGONAL UPPER LEFT TO LOWER MIDDLE RIGHT\n    0x7A: chr(0x1FB66), # UPPER RIGHT BLOCK DIAGONAL UPPER CENTRE TO LOWER RIGHT\n    0x7B: chr(0x1FB67), # UPPER RIGHT BLOCK DIAGONAL UPPER MIDDLE LEFT TO LOWER MIDDLE RIGHT\n    0x7C: chr(0x1FB6E), # RIGHT TRIANGULAR ONE QUARTER BLOCK\n    0x7D: chr(0x1FB6F), # LOWER TRIANGULAR ONE QUARTER BLOCK\n}\n"
  },
  {
    "path": "teletext/cli/__init__.py",
    "content": ""
  },
  {
    "path": "teletext/cli/celp.py",
    "content": "import click\n\nfrom teletext.cli.clihelpers import packetreader\nfrom teletext.celp import CELPDecoder\n\n@click.group()\ndef celp():\n    \"\"\"Tools for analysing CELP audio packets.\"\"\"\n    pass\n\n\n@celp.command()\n@click.option('-f', '--frame', type=int, default=None, help='Frame selection.')\n@click.option('-o', '--output', type=click.File('wb'), help='Write audio to WAV file.')\n@click.option('-l', '--lsf-lut', type=click.Choice(CELPDecoder.lsf_vector_quantizers.keys()), default='suddle', help='LSF vector look-up table.')\n@click.option('-g', '--gain-lut', type=click.Choice(CELPDecoder.vec_gain_quantizers.keys()), default='audetel', help='LSF vector look-up table.')\n@packetreader(filtered='data', mag_hist=None, row_hist=None, err_hist=None, pass_progress=True)\ndef play(progress, frame, output, lsf_lut, gain_lut, packets):\n    \"\"\"Play data from CELP packets. Warning: Will make a horrible noise.\"\"\"\n    dec = CELPDecoder(lsf_lut=lsf_lut, vec_gain_lut=gain_lut)\n    progress.postfix.append(dec.stats())\n    if output is not None:\n        dec.convert(output, packets, frame=frame)\n    else:\n        dec.play(packets, frame=frame)\n\n\n@celp.command()\n@packetreader(filtered='data')\ndef plot(packets):\n    \"\"\"Plot data from CELP packets. Experimental code.\"\"\"\n    CELPDecoder.plot(packets)\n"
  },
  {
    "path": "teletext/cli/clihelpers.py",
    "content": "import cProfile\nimport os\nimport stat\nfrom functools import wraps\n\nimport click\nfrom tqdm import tqdm\n\nfrom teletext import pipeline\nfrom teletext.packet import Packet\nfrom teletext.stats import StatsList, MagHistogram, RowHistogram, ErrorHistogram\nfrom teletext.file import FileChunker\nfrom teletext.vbi.config import Config\n\ntry:\n    import plop.collector as plop\nexcept ImportError:\n    plop = None\n\n\nclass BasedIntType(click.ParamType):\n    name = \"integer\"\n\n    def convert(self, value, param, ctx):\n        if isinstance(value, int):\n            return value\n\n        try:\n            if value[:2].lower() == \"0x\":\n                return int(value[2:], 16)\n            return int(value, 10)\n        except ValueError:\n            self.fail(f\"{value!r} is not a valid integer\", param, ctx)\n\nBasedInt = BasedIntType()\n\ndef dcnparams(f):\n    return click.option('-d', '--dcn', 'dcn', type=int, required=True, help='Data channel to read from.')(f)\n\n\ndef filterparams(enabled=True):\n    def fp(f):\n        if enabled:\n            for d in [\n                click.option('-m', '--mag', 'mags', type=int, multiple=True, default=range(9), help='Limit output to specific magazines. Can be specified multiple times.'),\n                click.option('-r', '--row', 'rows', type=int, multiple=True, default=range(32), help='Limit output to specific rows. Can be specified multiple times.'),\n            ][::-1]:\n                f = d(f)\n        return f\n    return fp\n\ndef progressparams(progress=True, mag_hist=False, row_hist=False, err_hist=False):\n    def p(f):\n        if err_hist is not None:\n            f = click.option('--err-hist/--no-err-hist', default=err_hist, help='Display error distribution.')(f)\n        if row_hist is not None:\n            f = click.option('--row-hist/--no-row-hist', default=row_hist, help='Display row histogram.')(f)\n        if mag_hist is not None:\n            f = click.option('--mag-hist/--no-mag-hist', default=mag_hist, help='Display magazine histogram.')(f)\n        if progress is not None:\n            f = click.option('--progress/--no-progress', default=progress, help='Display progress bar.')(f)\n        return f\n    return p\n\n\ndef carduser(extended=False):\n    def c(f):\n        if extended:\n            for d in [\n                click.option('--sample-rate', type=float, default=None, help='Override capture card sample rate (Hz).'),\n                click.option('--sample-rate-adjust', type=float, default=0, help='Adjustment to default capture card sample rate (Hz).'),\n                click.option('--extra-roll', type=int, default=0, help='Shift line by N samples after locking to the packet.'),\n                click.option('--line-start-range', type=(int, int), default=(None, None), help='Override capture card line start offset.'),\n            ][::-1]:\n                f = d(f)\n\n        @click.option('-c', '--card', type=click.Choice(list(Config.cards.keys())), default='bt8x8', help='Capture device type. Default: bt8x8.')\n        @click.option('--line-length', type=int, default=None, help='Override capture card samples per line.')\n        @wraps(f)\n        def wrapper(card, line_length=None, sample_rate=None, sample_rate_adjust=0, line_start_range=None, extra_roll=0, *args, **kwargs):\n            if line_start_range == (None, None):\n                line_start_range = None\n            config = Config(card=card, line_length=line_length, sample_rate=sample_rate, sample_rate_adjust=sample_rate_adjust, line_start_range=line_start_range, extra_roll=extra_roll)\n            return f(config=config, *args,**kwargs)\n        return wrapper\n    return c\n\n\ndef chunkreader(loop=False, dup_stdin=False):\n    def cr(f):\n        @click.argument('input', type=click.File('rb'), default='-')\n        @click.option('--start', type=int, default=0, help='Start at the Nth line of the input file.')\n        @click.option('--stop', type=int, default=None, help='Stop before the Nth line of the input file.')\n        @click.option('--step', type=int, default=1, help='Process every Nth line from the input file.')\n        @click.option('--limit', type=int, default=None, help='Stop after processing N lines from the input file.')\n        @wraps(f)\n        def wrapper(input, start, stop, step, limit, *args, **kwargs):\n\n            if input.isatty():\n                raise click.UsageError('No input file and stdin is a tty - exiting.', )\n\n            if 'progress' in kwargs and kwargs['progress'] is None:\n                if hasattr(input, 'fileno') and stat.S_ISFIFO(os.fstat(input.fileno()).st_mode):\n                    kwargs['progress'] = False\n\n            chunker = lambda size, flines=16, frange=range(0, 16): FileChunker(input, size, start, stop, step, limit, flines, frange, loop=loop, dup_stdin=dup_stdin)\n\n            return f(chunker=chunker, *args, **kwargs)\n        return wrapper\n    return cr\n\ndef packetreader(filtered=True, progress=True, mag_hist=False, row_hist=False, err_hist=False, pass_progress=False, loop=False, dup_stdin=False):\n    if filtered == 'data':\n        filterdec = dcnparams\n    else:\n        filterdec = filterparams(filtered)\n\n    def pr(f):\n        @chunkreader(loop=loop, dup_stdin=dup_stdin)\n        @click.option('--wst', is_flag=True, default=False, help='Input is 43 bytes per packet (WST capture card format.)')\n        @click.option('--ts', type=BasedInt, default=None, help='Input is MPEG transport stream. (Specify PID to extract.)')\n        @filterdec\n        @progressparams(progress=(progress and not loop), mag_hist=mag_hist, row_hist=row_hist, err_hist=err_hist)\n        @wraps(f)\n        def wrapper(chunker, wst, ts, progress, *args, **kwargs):\n\n            if wst and (ts is not None):\n                raise click.UsageError('--wst and --ts can not be specified at the same time.')\n\n            if wst:\n                chunks = chunker(43)\n                chunks = ((c[0],c[1][:42]) for c in chunks if c[1][0] != 0)\n            elif ts is not None:\n                from teletext.ts import pidextract\n                chunks = chunker(188)\n                chunks = pidextract(chunks, ts)\n            else:\n                chunks = chunker(42)\n\n            if progress is None:\n                progress = True\n\n            mag_hist = kwargs.pop('mag_hist', None)\n            row_hist = kwargs.pop('row_hist', None)\n            err_hist = kwargs.pop('err_hist', None)\n\n            if progress:\n                chunks = tqdm(chunks, unit='P', dynamic_ncols=True)\n                if pass_progress or any((mag_hist, row_hist, err_hist)):\n                    chunks.postfix = StatsList()\n\n            packets = (Packet(data, number) for number, data in chunks)\n            if 'mags' in kwargs and 'rows' in kwargs:\n                mags = kwargs.pop('mags')\n                rows = kwargs.pop('rows')\n                packets = (p for p in packets if p.mrag.magazine in mags and p.mrag.row in rows)\n\n            elif 'dcn' in kwargs:\n                dcn = kwargs.pop('dcn')\n                mags = (dcn & 0x7,)\n                rows = (30 + (dcn>>3),)\n                packets = (p for p in packets if p.mrag.magazine in mags and p.mrag.row in rows)\n\n            if progress:\n                if mag_hist:\n                    packets = MagHistogram(packets)\n                    chunks.postfix.append(packets)\n                if row_hist:\n                    packets = RowHistogram(packets)\n                    chunks.postfix.append(packets)\n                if err_hist:\n                    packets = ErrorHistogram(packets)\n                    chunks.postfix.append(packets)\n\n            if pass_progress:\n                return f(progress=chunks, packets=packets, *args, **kwargs)\n            else:\n                return f(packets=packets, *args, **kwargs)\n\n        return wrapper\n\n    return pr\n\n\ndef paginated(always=False, filtered=True):\n    def _paginated(f):\n        @wraps(f)\n        def wrapper(*args, **kwargs):\n            paginate = always or kwargs['paginate']\n\n            if filtered:\n                pages = kwargs['pages']\n                if pages is None or len(pages) == 0:\n                    pages = range(0x900)\n                else:\n                    pages = {int(x, 16) for x in pages}\n                    paginate = True\n                kwargs['pages'] = pages\n\n                subpages = kwargs['subpages']\n                if subpages is None or len(subpages) == 0:\n                    subpages = range(0x3f80)\n                else:\n                    subpages = {int(x, 16) for x in subpages}\n                    paginate = True\n                kwargs['subpages'] = subpages\n\n            if paginate and 0 not in kwargs['rows']:\n                raise click.BadArgumentUsage(\"Can't paginate when row 0 is filtered.\")\n\n            if not always:\n                kwargs['paginate'] = paginate\n\n            return f(*args, **kwargs)\n\n        if filtered:\n            wrapper = click.option('-s', '--subpage', 'subpages', type=str, multiple=True,\n                      help='Limit output to specific subpage. Can be specified multiple times.')(wrapper)\n            wrapper = click.option('-p', '--page', 'pages', type=str, multiple=True,\n                      help='Limit output to specific page. Can be specified multiple times.')(wrapper)\n        if not always:\n            wrapper = click.option('-P', '--paginate', is_flag=True, help='Sort rows into contiguous pages.')(wrapper)\n\n        return wrapper\n    return _paginated\n\n\ndef packetwriter(f):\n    @click.option(\n        '-o', '--output', type=(click.Choice(['auto', 'text', 'ansi', 'debug', 'bar', 'bytes', 'hex', 'vbi']), click.File('wb')),\n        multiple=True, default=[('auto', '-')]\n    )\n    @wraps(f)\n    def wrapper(output, *args, **kwargs):\n\n        if 'progress' in kwargs and kwargs['progress'] is None:\n            for attr, o in output:\n                if o.isatty():\n                    kwargs['progress'] = False\n\n        packets = f(*args, **kwargs)\n\n        for attr, o in output:\n            packets = pipeline.to_file(packets, o, attr)\n\n        for p in packets:\n            pass\n\n    return wrapper\n"
  },
  {
    "path": "teletext/cli/teletext.py",
    "content": "import itertools\nimport multiprocessing\nimport os\nimport pathlib\nimport platform\n\nimport sys\nfrom collections import defaultdict\n\nimport click\nfrom tqdm import tqdm\n\nfrom teletext.charset import g0\nfrom teletext.cli.clihelpers import packetreader, packetwriter, paginated, \\\n    progressparams, filterparams, carduser, chunkreader\nfrom teletext.file import FileChunker\nfrom teletext.mp import itermap\nfrom teletext.packet import Packet, np\nfrom teletext.stats import StatsList, MagHistogram, RowHistogram, Rejects, ErrorHistogram\nfrom teletext.subpage import Subpage\nfrom teletext import pipeline\nfrom teletext.cli.training import training\nfrom teletext.cli.vbi import vbi\nfrom teletext.vbi.config import Config\n\n\nif os.name == 'nt' and platform.release() == '10' and platform.version() >= '10.0.14393':\n    # Fix ANSI color in Windows 10 version 10.0.14393 (Windows Anniversary Update)\n    import ctypes\n    kernel32 = ctypes.windll.kernel32\n    kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)\n\n\n@click.group(invoke_without_command=True, no_args_is_help=True)\n@click.option('-u', '--unicode', is_flag=True, help='Use experimental Unicode 13.0 Terminal graphics.')\n@click.version_option()\n@click.help_option()\n@click.option('--help-all', is_flag=True, help='Show help for all subcommands.')\n@click.pass_context\ndef teletext(ctx, unicode, help_all):\n    \"\"\"Teletext stream processing toolkit.\"\"\"\n    if help_all:\n        print(teletext.get_help(ctx))\n\n        def help_recurse(group, ctx):\n            for scmd in group.list_commands(ctx):\n                cmd = group.get_command(ctx, scmd)\n                nctx = click.Context(cmd, ctx, scmd)\n                if isinstance(cmd, click.Group):\n                    help_recurse(cmd, nctx)\n                else:\n                    click.echo()\n                    click.echo(cmd.get_help(nctx))\n\n        help_recurse(teletext, ctx)\n\n    if unicode:\n        from teletext import parser\n        parser._unicode13 = True\n\n\nteletext.add_command(training)\nteletext.add_command(vbi)\n\ntry:\n    from teletext.cli.celp import celp\n    teletext.add_command(celp)\nexcept ImportError:\n    pass\n\n\n@teletext.command()\n@packetwriter\n@paginated()\n@click.option('--pagecount', 'n', type=int, default=0, help='Stop after n pages. 0 = no limit. Implies -P.')\n@click.option('-k', '--keep-empty', is_flag=True, help='Keep empty packets in the output.')\n@packetreader()\ndef filter(packets, pages, subpages, paginate, n, keep_empty):\n\n    \"\"\"Demultiplex and display t42 packet streams.\"\"\"\n\n    if n:\n        paginate = True\n\n    if not keep_empty:\n        packets = (p for p in packets if not p.is_padding())\n\n    if paginate:\n        for pn, pl in enumerate(pipeline.paginate(packets, pages=pages, subpages=subpages), start=1):\n            yield from pl\n            if pn == n:\n                return\n    else:\n        yield from packets\n\n\n@teletext.command()\n@packetwriter\n@paginated()\n@click.argument('regex', type=str)\n@click.option('-v', is_flag=True, help='Invert matches.')\n@click.option('-i', is_flag=True, help='Ignore case.')\n@click.option('--pagecount', 'n', type=int, default=0, help='Stop after n pages. 0 = no limit. Implies -P.')\n@click.option('-k', '--keep-empty', is_flag=True, help='Keep empty packets in the output.')\n@packetreader()\ndef grep(packets, pages, subpages, paginate, regex, v, i, n, keep_empty):\n\n    \"\"\"Filter packets with a regular expression.\"\"\"\n\n    import re\n\n    pattern = re.compile(regex.encode('ascii'), re.IGNORECASE if i else 0)\n\n    if n:\n        paginate = True\n\n    if not keep_empty:\n        packets = (p for p in packets if not p.is_padding())\n\n    if paginate:\n        for pn, pl in enumerate(pipeline.paginate(packets, pages=pages, subpages=subpages), start=1):\n            for p in pl:\n                if bool(v) != bool(re.search(pattern, p.to_bytes_no_parity())):\n                    yield from pl\n                    if pn == n:\n                        return\n    else:\n        for p in packets:\n            if bool(v) != bool(re.search(pattern, p.to_bytes_no_parity())):\n                yield p\n\n\n@teletext.command(name='list')\n@click.option('-c', '--count', is_flag=True, help='Show counts of each entry.')\n@click.option('-s', '--subpages', is_flag=True, help='Also list subpages.')\n@paginated(always=True, filtered=False)\n@packetreader()\n@progressparams(progress=True, mag_hist=True)\ndef _list(packets, count, subpages):\n\n    \"\"\"List pages present in a t42 stream.\"\"\"\n\n    import textwrap\n\n    packets = (p for p in packets if not p.is_padding())\n\n    seen = {}\n    try:\n        for pl in pipeline.paginate(packets):\n            s = Subpage.from_packets(pl)\n            identifier = f'{s.mrag.magazine}{s.header.page:02x}'\n            if subpages:\n                identifier += f':{s.header.subpage:04x}'\n            if identifier in seen:\n                seen[identifier]+=1\n            else:\n                seen[identifier]=1\n    except KeyboardInterrupt:\n        print('\\n')\n    finally:\n        if count:\n            maxdigits = len(str(max(seen.values())))\n            formatstr=\"{page}/{count:0\" + str(maxdigits) +\"}\"\n        else:\n            formatstr=\"{page}\"\n        seen = list(map(lambda e: formatstr.format(page = e[0], count = e[1]), seen.items()))\n        print('\\n'.join(textwrap.wrap(' '.join(sorted(seen)))))\n\n\n@teletext.command()\n@click.argument('pattern')\n@paginated(always=True)\n@packetreader()\ndef split(packets, pattern, pages, subpages):\n\n    \"\"\"Split a t42 stream in to multiple files.\"\"\"\n\n    packets = (p for p in packets if not p.is_padding())\n    counts = defaultdict(int)\n\n    for pl in pipeline.paginate(packets, pages=pages, subpages=subpages):\n        subpage = Subpage.from_packets(pl)\n        m = subpage.mrag.magazine\n        p = subpage.header.page\n        s = subpage.header.subpage\n        c = counts[(m,p,s)]\n        counts[(m,p,s)] += 1\n        f = pathlib.Path(pattern.format(m=m, p=f'{p:02x}', s=f'{s:04x}', c=f'{c:04d}'))\n        f.parent.mkdir(parents=True, exist_ok=True)\n        with f.open('ab') as ff:\n            ff.write(b''.join(p.bytes for p in pl))\n\n\n@teletext.command()\n@click.argument('a', type=click.File('rb'))\n@click.argument('b', type=click.File('rb'))\n@filterparams()\ndef diff(a, b, mags, rows):\n    \"\"\"Show side by side difference of two t42 streams.\"\"\"\n    for chunka, chunkb in zip(FileChunker(a, 42), FileChunker(b, 42)):\n        pa = Packet(chunka[1], chunka[0])\n        pb = Packet(chunkb[1], chunkb[0])\n        if (pa.mrag.row in rows and pa.mrag.magazine in mags) or (pb.mrag.row in rows and pa.mrag.magazine in mags):\n            if any(pa[:] != pb[:]):\n                print(pa.to_ansi(), pb.to_ansi())\n\n\n@teletext.command()\n@packetwriter\n@packetreader()\ndef finders(packets):\n\n    \"\"\"Apply finders to fix up common packets.\"\"\"\n\n    for p in packets:\n        if p.type == 'header':\n            p.header.apply_finders()\n        yield p\n\n\n@teletext.command()\n@packetreader(filtered=False)\n@click.option('-l', '--lines', type=int, default=32, help='Number of recorded lines per frame.')\n@click.option('-f', '--frames', type=int, default=250, help='Number of frames to squash.')\ndef scan(packets, lines, frames):\n\n    \"\"\"Filter a t42 stream down to headers and bsdp, with squashing.\"\"\"\n\n    from teletext.pipeline import packet_squash, bsdp_squash_format1, bsdp_squash_format2\n    bars = '_:|I'\n\n    while True:\n        actives = np.zeros((lines,), dtype=np.uint32)\n        headers = [[], [], [], [], [], [], [], [], []]\n        service = [[], []]\n        start = None\n        try:\n            for i in range(frames):\n                for n, p in enumerate(itertools.islice(packets, lines)):\n                    if start is None:\n                        start = p._number\n                    if not p.is_padding():\n                        if p.type == 'header':\n                            p.header.apply_finders()\n                        actives[n] += 1\n                        if p.mrag.row == 0:\n                            headers[p.mrag.magazine].append(p)\n                        elif p.mrag.row == 30 and p.mrag.magazine == 8:\n                            if p.broadcast.dc in [0, 1]:\n                                service[0].append(p)\n                            elif p.broadcast.dc in [2, 3]:\n                                service[1].append(p)\n\n        except StopIteration:\n            pass\n        if start is None:\n            return\n        active_group = 1*(actives>0) + 1*(actives>(frames/2)) + 1*(actives==frames)\n        print(f'{start:8d}', '['+''.join(bars[a] for a in active_group)+']', end=' ')\n        for h in headers:\n            if h:\n                print(packet_squash(h).header.displayable.to_ansi(), end=' ')\n                break\n        for s in service:\n            if s:\n                print(packet_squash(s).broadcast.displayable.to_ansi(), end=' ')\n                break\n        if service[0]:\n            print(bsdp_squash_format1(service[0]), end=' ')\n        if service[1]:\n            print(bsdp_squash_format2(service[1]), end=' ')\n        print()\n\n\n@teletext.command()\n@click.option('-d', '--min-duplicates', type=int, default=3, help='Only squash and output subpages with at least N duplicates.')\n@click.option('-t', '--threshold', type=int, default=-1, help='Max difference for squashing.')\n@click.option('-i', '--ignore-empty', is_flag=True, default=False, help='Ignore the emptiest duplicate packets instead of the earliest.')\n@packetwriter\n@paginated(always=True)\n@packetreader()\ndef squash(packets, min_duplicates, threshold, pages, subpages, ignore_empty):\n\n    \"\"\"Reduce errors in t42 stream by using frequency analysis.\"\"\"\n\n    packets = (p for p in packets if not p.is_padding())\n    for sp in pipeline.subpage_squash(\n            pipeline.paginate(packets, pages=pages, subpages=subpages),\n            min_duplicates=min_duplicates, ignore_empty=ignore_empty,\n            threshold=threshold\n    ):\n        yield from sp.packets\n\n\n@teletext.command()\n@click.option('-l', '--language', default='en_GB', help='Language. Default: en_GB')\n@click.option('-b', '--both', is_flag=True, help='Show packet before and after corrections.')\n@click.option('-t', '--threads', type=int, default=multiprocessing.cpu_count(), help='Number of threads.')\n@packetwriter\n@packetreader()\ndef spellcheck(packets, language, both, threads):\n\n    \"\"\"Spell check a t42 stream.\"\"\"\n\n    try:\n        from teletext.spellcheck import spellcheck_packets\n    except ModuleNotFoundError as e:\n        if e.name == 'enchant':\n            raise click.UsageError(f'{e.msg}. PyEnchant is not installed. Spelling checker is not available.')\n        else:\n            raise e\n    else:\n        if both:\n            packets, orig_packets = itertools.tee(packets, 2)\n            packets = itermap(spellcheck_packets, packets, threads, language=language)\n            try:\n                while True:\n                    yield next(orig_packets)\n                    yield next(packets)\n            except StopIteration:\n                pass\n        else:\n            yield from itermap(spellcheck_packets, packets, threads, language=language)\n\n\n@teletext.command()\n@click.option('-r', '--replace_headers', 'replace_headers', is_flag=True, default=False, help='Replace headers with a live clock.')\n@click.option('-t', '--title', 'title', type=str, default=\"Teletext \", help='Replace header title field with this string.')\n@packetwriter\n@paginated(always=True, filtered=False)\n@packetreader()\ndef service(packets, replace_headers, title):\n\n    \"\"\"Build a service carousel from a t42 stream.\"\"\"\n\n    from teletext.service import Service\n    return Service.from_packets((p for p in packets if  not p.is_padding()), replace_headers, title)\n\n\n@teletext.command()\n@click.option('-r', '--replace_headers', 'replace_headers', is_flag=True, default=False, help='Replace headers with a live clock.')\n@click.option('-t', '--title', 'title', type=str, default=None, help='Replace header title field with this string.')\n@packetwriter\n@click.argument('directory', type=click.Path(exists=True, readable=True, file_okay=False, dir_okay=True))\ndef servicedir(directory, replace_headers, title):\n    \"\"\"Build a service from a directory of t42 files.\"\"\"\n\n    from teletext.servicedir import ServiceDir\n    with ServiceDir(directory, replace_headers, title) as s:\n        yield from s\n\n\n@teletext.command()\n@click.option('-i', '--initial_page', 'initial_page', type=str, default='100', help='Initial page.')\n@packetreader(loop=True, dup_stdin=True)\ndef interactive(packets, initial_page):\n\n    \"\"\"Interactive teletext emulator.\"\"\"\n\n    from teletext import interactive\n    interactive.main(packets, int(initial_page, 16))\n\n\n@teletext.command()\n@packetreader(loop=True)\n@click.option('-p', '--port', type=str, default=None)\ndef serial(packets, port):\n\n    \"\"\"Write escaped packets to serial inserter.\"\"\"\n\n    import serial.tools.list_ports\n    import time\n\n    if port is None:\n        for comport in serial.tools.list_ports.comports():\n            if comport.vid == 0x2e8a and (comport.pid == 0x000a or comport.pid == 0x0009):\n                port = comport.device\n\n    if port is None:\n        raise click.UsageError('No serial inserter found. Specify the path with -p')\n\n    port = serial.Serial(port, timeout=3, rtscts=True)\n\n    for p in packets:\n        buf = p.bytes\n        buf = buf.replace(b'\\xfe', b'\\xfe\\x00')\n        buf = buf.replace(b'\\xff', b'\\xfe\\x01')\n        buf = b'\\xff' + buf\n        port.write(buf)\n\n\n@teletext.command()\n@click.option('-e', '--editor', type=str, default='https://zxnet.co.uk/teletext/editor/#',\n              show_default=True, help='Teletext editor URL.')\n@paginated(always=True)\n@packetreader()\ndef urls(packets, editor, pages, subpages):\n\n    \"\"\"Paginate a t42 stream and print edit.tf URLs.\"\"\"\n\n    packets = (p for p in packets if  not p.is_padding())\n    subpages = (Subpage.from_packets(pl) for pl in pipeline.paginate(packets, pages=pages, subpages=subpages))\n\n    for s in subpages:\n        print(f'{editor}{s.url}')\n\n@teletext.command()\n@click.argument('outdir', type=click.Path(exists=True, file_okay=False, dir_okay=True, writable=True), required=True)\n@click.option('-f', '--font', type=click.File('rb'), help='PCF font for rendering.')\n@paginated(always=True)\n@packetreader()\ndef images(packets, outdir, font, pages, subpages):\n\n    \"\"\"Generate images for the input stream.\"\"\"\n\n    try:\n        from teletext.image import subpage_to_image, load_glyphs\n    except ModuleNotFoundError as e:\n        if e.name == 'PIL':\n            raise click.UsageError(\n                f'{e.msg}. PIL is not installed. Image generation is not available.')\n        else:\n            raise e\n\n    from teletext.service import Service\n\n    glyphs = load_glyphs(font)\n\n    packets = (p for p in packets if  not p.is_padding())\n    svc = Service.from_packets(p for p in packets if not p.is_padding())\n\n    subpages = tqdm(list(svc.all_subpages), unit=\"subpage\")\n    for s in subpages:\n        image = subpage_to_image(s, glyphs)\n        filename = f'P{s.mrag.magazine}{s.header.page:02x}-{s.header.subpage:04x}.png'\n        subpages.set_description(filename, refresh=False)\n        if image._flash_used:\n            opts = {\n                'save_all': True,\n                'append_images': [subpage_to_image(s, glyphs, flash_off=True)],\n                'duration': 500,\n                'loop': 0,\n                'disposal': 2,\n            }\n        else:\n            opts = {}\n        image.save(pathlib.Path(outdir) / filename, **opts)\n        if image._missing_glyphs:\n            missing = ', '.join(f'{repr(c)} {hex(ord(c))}' for c in image._missing_glyphs)\n            print(f'{filename} missing characters: {missing}')\n\n\n@teletext.command()\n@click.argument('outdir', type=click.Path(exists=True, file_okay=False, dir_okay=True, writable=True), required=True)\n@click.option('-t', '--template', type=click.File('r'), default=None, help='HTML template.')\n@click.option('--localcodepage', type=click.Choice(g0.keys()), default=None, help='Select codepage for Local Code of Practice')\n@paginated(always=True, filtered=False)\n@packetreader()\ndef html(packets, outdir, template, localcodepage):\n\n    \"\"\"Generate HTML files from the input stream.\"\"\"\n\n    from teletext.service import Service\n\n    if template is not None:\n        template = template.read()\n\n    svc = Service.from_packets(p for p in packets if not p.is_padding())\n    svc.to_html(outdir, template, localcodepage)\n\n\n@teletext.command()\n@click.argument('output', type=click.File('wb'), default='-')\n@click.option('-d', '--device', type=click.File('rb'), default='/dev/vbi0', help='Capture device.')\n@carduser()\ndef record(output, device, config):\n\n    \"\"\"Record VBI samples from a capture device.\"\"\"\n\n    import struct\n    import sys\n\n    if output.name.startswith('/dev/vbi'):\n        raise click.UsageError(f'Refusing to write output to VBI device. Did you mean -d?')\n\n    chunks = FileChunker(device, config.line_length*config.field_lines*2)\n    bar = tqdm(chunks, unit=' Frames')\n\n    prev_seq = None\n    dropped = 0\n\n    try:\n        for n, chunk in bar:\n            output.write(chunk)\n            if config.card == 'bt8x8':\n                seq, = struct.unpack('<I', chunk[-4:])\n                if prev_seq is not None and seq != (prev_seq + 1):\n                   dropped += 1\n                   sys.stderr.write('Frame drop? %d\\n' % dropped)\n                prev_seq = seq\n\n    except KeyboardInterrupt:\n        pass\n\n\n@teletext.command()\n@click.option('-p', '--pause', is_flag=True, help='Start the viewer paused.')\n@click.option('-f', '--tape-format', type=click.Choice(Config.tape_formats), default='vhs', help='Source VCR format.')\n@click.option('-n', '--n-lines', type=int, default=None, help='Number of lines to display. Overrides card config.')\n@carduser(extended=True)\n@chunkreader()\ndef vbiview(chunker, config, pause, tape_format, n_lines):\n\n    \"\"\"Display raw VBI samples with OpenGL.\"\"\"\n\n    try:\n        from teletext.vbi.viewer import VBIViewer\n    except ModuleNotFoundError as e:\n        if e.name.startswith('OpenGL'):\n            raise click.UsageError(f'{e.msg}. PyOpenGL is not installed. VBI viewer is not available.')\n        else:\n            raise e\n    else:\n        from teletext.vbi.line import Line\n\n        Line.configure(config, force_cpu=True, tape_format=tape_format)\n\n        if n_lines is not None:\n            chunks = chunker(config.line_bytes, n_lines, range(n_lines))\n        else:\n            chunks = chunker(config.line_bytes, config.field_lines, config.field_range)\n\n        lines = (Line(chunk, number) for number, chunk in chunks)\n\n        VBIViewer(lines, config, pause=pause, nlines=n_lines)\n\n\n@teletext.command()\n@click.option('-M', '--mode', type=click.Choice(['deconvolve', 'slice']), default='deconvolve', help='Deconvolution mode.')\n@click.option('-8', '--eight-bit', is_flag=True, help='Treat rows 1-25 as 8-bit data without parity check.')\n@click.option('-f', '--tape-format', type=click.Choice(Config.tape_formats), default='vhs', help='Source VCR format.')\n@click.option('-C', '--force-cpu', is_flag=True, help='Disable GPU even if it is available.')\n@click.option('-O', '--prefer-opencl', is_flag=True, default=False, help='Use OpenCL even if CUDA is available.')\n@click.option('-t', '--threads', type=int, default=multiprocessing.cpu_count(), help='Number of threads.')\n@click.option('-k', '--keep-empty', is_flag=True, help='Insert empty packets in the output when line could not be deconvolved.')\n@carduser(extended=True)\n@packetwriter\n@chunkreader()\n@filterparams()\n@paginated()\n@progressparams(progress=True, mag_hist=True)\n@click.option('--rejects/--no-rejects', default=True, help='Display percentage of lines rejected.')\ndef deconvolve(chunker, mags, rows, pages, subpages, paginate, config, mode, eight_bit, force_cpu, prefer_opencl, threads, keep_empty, progress, mag_hist, row_hist, err_hist, rejects, tape_format):\n\n    \"\"\"Deconvolve raw VBI samples into Teletext packets.\"\"\"\n\n    if keep_empty and paginate:\n        raise click.UsageError(\"Can't keep empty packets when paginating.\")\n\n    from teletext.vbi.line import process_lines\n\n    if force_cpu:\n        sys.stderr.write('GPU disabled by user request.\\n')\n\n    chunks = chunker(config.line_bytes, config.field_lines, config.field_range)\n\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n        if any((mag_hist, row_hist, rejects)):\n            chunks.postfix = StatsList()\n\n    packets = itermap(process_lines, chunks, threads,\n                      mode=mode, config=config,\n                      force_cpu=force_cpu, prefer_opencl=prefer_opencl,\n                      mags=mags, rows=rows,\n                      tape_format=tape_format,\n                      eight_bit=eight_bit)\n\n    if progress and rejects:\n        packets = Rejects(packets)\n        chunks.postfix.append(packets)\n\n    if keep_empty:\n        packets = (p if isinstance(p, Packet) else Packet() for p in packets)\n    else:\n        packets = (p for p in packets if isinstance(p, Packet))\n\n    if progress and mag_hist:\n        packets = MagHistogram(packets)\n        chunks.postfix.append(packets)\n    if progress and row_hist:\n        packets = RowHistogram(packets)\n        chunks.postfix.append(packets)\n    if progress and err_hist:\n        packets = ErrorHistogram(packets)\n        chunks.postfix.append(packets)\n\n    if paginate:\n        for p in pipeline.paginate(packets, pages=pages, subpages=subpages):\n            yield from p\n    else:\n        yield from packets\n"
  },
  {
    "path": "teletext/cli/training.py",
    "content": "import multiprocessing\nimport os\n\nimport click\nfrom tqdm import tqdm\n\nfrom teletext.cli.clihelpers import carduser, chunkreader\nfrom teletext.file import FileChunker\nfrom teletext.mp import itermap\nfrom teletext.packet import Packet, np\nfrom teletext.stats import StatsList, Rejects\n\n\n@click.group()\ndef training():\n    \"\"\"Training and calibration tools.\"\"\"\n    pass\n\n\n@training.command()\n@click.argument('output', type=click.File('wb'), default='-')\ndef generate(output):\n    \"\"\"Generate training samples for raspi-teletext.\"\"\"\n    from teletext.vbi.training import PatternGenerator\n    PatternGenerator().to_file(output)\n\n\n@training.command()\n@click.argument('outdir', type=click.Path(exists=True, file_okay=False, dir_okay=True, writable=True), required=True)\n@click.option('-t', '--threads', type=int, default=multiprocessing.cpu_count(), help='Number of threads.')\n@carduser(extended=True)\n@chunkreader()\n@click.option('--progress/--no-progress', default=True, help='Display progress bar.')\n@click.option('--rejects/--no-rejects', default=True, help='Display percentage of lines rejected.')\ndef split(chunker, outdir, config, threads, progress, rejects):\n    \"\"\"Split training recording into intermediate bins.\"\"\"\n    from teletext.vbi.training import process_training, split\n\n    chunks = chunker(config.line_length * np.dtype(config.dtype).itemsize, config.field_lines, config.field_range)\n\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n\n    results = itermap(process_training, chunks, threads, config=config)\n\n    if progress and rejects:\n        results = Rejects(results)\n        chunks.postfix = StatsList()\n        chunks.postfix.append(results)\n\n    results = (r for r in results if isinstance(r, tuple))\n\n    files = [open(os.path.join(outdir, f'training.{n:02x}.dat'), 'wb') for n in range(256)]\n\n    split(results, files)\n\n\n@training.command(name='squash')\n@click.argument('indir', type=click.Path(exists=True, file_okay=False, dir_okay=True), required=True)\n@click.argument('output', type=click.File('wb'), default='-')\ndef training_squash(output, indir):\n    \"\"\"Squash the intermediate bins into a single file.\"\"\"\n    from teletext.vbi.training import squash\n    squash(output, indir)\n\n\n@training.command()\n@chunkreader()\ndef showbin(chunker):\n    \"\"\"Visually display an intermediate training bin.\"\"\"\n    import numpy as np\n\n    bars = ' ▁▂▃▄▅▆▇█'\n    bits = ' █'\n\n    chunks = chunker(27)\n\n    for n, chunk in chunks:\n        arr = np.frombuffer(chunk, dtype=np.uint8)\n        bi = ''.join(bits[n] for n in np.unpackbits(arr[:3][::-1])[::-1])\n        by = ''.join(bars[n] for n in arr[3:]>>5)\n        print(f'[{bi}] [{by}]')\n\n\n@training.command()\n@click.argument('input', type=click.File('rb'), required=True)\n@click.argument('output', type=click.File('wb'), required=True)\n@click.option('-m', '--mode', type=click.Choice(['full', 'parity', 'hamming']), default='full')\n@click.option('-b', '--bits', type=(int, int), default=(3, 21))\ndef build(input, output, mode, bits):\n    \"\"\"Build pattern tables.\"\"\"\n    from teletext.coding import parity_encode, hamming8_enc\n    from teletext.vbi.pattern import build_pattern\n\n    if mode == 'parity':\n        pattern_set = set(parity_encode(range(0x80)))\n    elif mode == 'hamming':\n        pattern_set = set(hamming8_enc)\n    else:\n        pattern_set = range(256)\n\n    chunks = FileChunker(input, 27)\n    chunks = tqdm(chunks, unit='P', dynamic_ncols=True)\n\n    build_pattern(chunks, output, *bits, pattern_set)\n\n\n@training.command()\n@click.option('-f', '--tape-format', type=click.Choice(['vhs', 'betamax', 'grundig_2x4']), default='vhs', help='Source VCR format.')\ndef similarities(tape_format):\n    from teletext.vbi.pattern import Pattern\n\n    pattern = Pattern(os.path.dirname(__file__) + '/vbi/data-' + tape_format + '/parity.dat')\n\n    print(pattern.similarities())\n\n\n@training.command()\n@click.option('-t', '--threads', type=int, default=multiprocessing.cpu_count(), help='Number of threads.')\n@carduser(extended=True)\n@chunkreader()\n@click.option('--progress/--no-progress', default=True, help='Display progress bar.')\n@click.option('--rejects/--no-rejects', default=True, help='Display percentage of lines rejected.')\ndef crifc(chunker, config, threads, progress, rejects):\n    \"\"\"Split training recording into intermediate bins.\"\"\"\n    from teletext.vbi.training import process_crifc\n\n    chunks = chunker(config.line_length * np.dtype(config.dtype).itemsize, config.field_lines, config.field_range)\n\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n\n    process_crifc(chunks, config=config)\n"
  },
  {
    "path": "teletext/cli/vbi.py",
    "content": "import click\nimport pathlib\nimport numpy as np\nfrom tqdm import tqdm\n\nfrom teletext.cli.clihelpers import carduser, chunkreader\n\n@click.group()\ndef vbi():\n    \"\"\"Tools for analysing raw VBI samples.\"\"\"\n    pass\n\n\n@vbi.command()\n@click.argument('output', type=click.Path(writable=True))\n@click.option('-d', '--diff', is_flag=True, help='User first differential of samples.')\n@click.option('-s', '--show', is_flag=True, help='Show image when complete.')\n@click.option('-n', '--n-lines', type=int, default=None, help='Number of lines to display. Overrides card config.')\n@carduser(extended=True)\n@chunkreader()\ndef histogram(output, diff, show, chunker, config, n_lines):\n    from PIL import Image\n    import colorsys\n\n    n_lines = n_lines or len(list(config.field_range))*2\n    line_length = config.line_length - (1 if diff else 0)\n    result = np.zeros((n_lines, 256, line_length), dtype=np.uint32)\n    sel = np.arange(line_length)\n    chunks = chunker(config.line_length * np.dtype(config.dtype).itemsize, config.field_lines, config.field_range)\n    chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n    for n, d in chunks:\n        l = np.frombuffer(d, dtype=config.dtype) >> ((np.dtype(config.dtype).itemsize - 1) * 8)\n        if diff:\n            l = np.diff(l) + 128\n        result[n%n_lines, l, sel] += 1\n\n    for i in range(n_lines):\n        for j in range(line_length):\n            result[i,:,j] = 255*result[i,:,j]/np.max(result[i,:,j])\n\n    # flip vertically\n    result = result[:,::-1,:].reshape(-1, line_length)\n\n    palette = np.zeros((256, 3), dtype=np.uint8)\n    palette[0] = [0, 0, 0]\n    for c in range(1, 256):\n        palette[c] = [n * 255 for n in colorsys.hsv_to_rgb(c/1025, 1, 1)]\n\n    rgb = palette[result]\n    rgb[0::256, :] += 100\n    rgb[0::32, :] = np.maximum(rgb[0::32, :], 32)\n\n    i = Image.fromarray(rgb)\n    if show:\n        i.show()\n    i.convert('RGB').save(output)\n\n\n@vbi.command()\n@carduser(extended=True)\n@chunkreader()\ndef plot(chunker, config):\n    from teletext.gui.vbiplot import vbiplot\n    vbiplot(chunker, config)\n\n\n@vbi.command()\n@carduser(extended=True)\n@click.argument('input', type=click.Path(readable=True), required=True)\n@click.argument('sampledir', type=click.Path(writable=True), required=True)\n@click.option('-a', '--auto', is_flag=True)\ndef classifygui(input, sampledir, auto, config):\n    from teletext.gui.classify import classify_gui\n    classify_gui(input, sampledir, auto, config)\n\n\n@vbi.command()\n@carduser()\n@chunkreader()\n@click.argument('output', type=click.File('wb'))\n@click.option('--progress/--no-progress', default=True, help='Display progress bar.')\ndef copy(chunker, config, progress, output):\n    \"\"\"Copy input to output\"\"\"\n    chunks = chunker(config.line_length * np.dtype(config.dtype).itemsize, config.field_lines, config.field_range)\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n    for n, c in chunks:\n        output.write(c)\n\n\n@vbi.command()\n@carduser()\n@chunkreader()\n@click.argument('output', type=click.Path(), required=True)\n@click.option('--progress/--no-progress', default=True, help='Display progress bar.')\ndef linesplit(chunker, config, progress, output):\n    \"\"\"Split VBI file into one file per line\"\"\"\n    chunks = chunker(config.line_length * np.dtype(config.dtype).itemsize, config.field_lines, config.field_range)\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n    output = pathlib.Path(output)\n    output.mkdir(parents=True, exist_ok=True)\n    files = [(output / f'{n:02x}.vbi').open(\"wb\") for n in range(config.frame_lines)]\n    for number, chunk in chunks:\n        files[number % config.frame_lines].write(chunk)\n\n\n@vbi.command()\n@carduser()\n@chunkreader()\n@click.argument('output', type=click.Path(), required=True)\n@click.option('--progress/--no-progress', default=True, help='Display progress bar.')\n@click.option('--prefix', type=str, default=\"\", help='Prefix for cluster file names.')\ndef cluster(chunker, config, progress, output, prefix):\n    \"\"\"Split VBI file into clusters of similar lines\"\"\"\n    import teletext.vbi.clustering\n    chunks = chunker(config.line_bytes, config.field_lines, config.field_range)\n    if progress:\n        chunks = tqdm(chunks, unit='L', dynamic_ncols=True)\n    output = pathlib.Path(output)\n    output.mkdir(parents=True, exist_ok=True)\n    teletext.vbi.clustering.batch_cluster(chunks, output, prefix, config.field_lines * 2)\n\n\n@vbi.command()\n@carduser()\n@click.argument('map', type=click.File('rb'), required=True)\n@click.argument('output', type=click.File('wb'), required=True)\ndef rendermap(config, map, output):\n    \"\"\"Render cluster map to image\"\"\"\n    import teletext.vbi.clustering\n    teletext.vbi.clustering.rendermap(config, map, output)\n"
  },
  {
    "path": "teletext/coding.py",
    "content": "# * Copyright 2016 Alistair Buxton <a.j.buxton@gmail.com>\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\n\n\n\"\"\"Byte coding and error protection\n\nOdd parity:\nThe high bit of each byte is set such that there are an odd number of bits in the byte.\nSingle bit errors can be detected.\n\nHamming 8/4:\nP1 D1 P2 D2 P3 D3 P4 D4 (Transmission order, LSB first.)\nSingle bit errors can be identified and corrected. Double bit errors can be detected.\n\nHamming 24/16:\nP1 P2 D1 P3 D2 D3 D4 P4  D5 D6 D7 D8 D9 D10 D11 P5  D12 D13 D14 D15 D16 D17 D18 P6\nSingle bit errors can be identified and corrected. Double bit errors\ncan be detected.\n\n\"\"\"\nimport numpy as np\n\n\ndef thue_morse(n, even=True):\n    arr = np.array([even], dtype=bool)\n    for i in range(0, n):\n        arr = np.append(arr,~arr)\n    return arr\n\n# hamming 8/4 encoding look up table\nhamming8_enc = np.array([\n    0x15, 0x02, 0x49, 0x5e, 0x64, 0x73, 0x38, 0x2f, 0xd0, 0xc7, 0x8c, 0x9b, 0xa1, 0xb6, 0xfd, 0xea,\n], dtype=np.uint8)\nhamming8_enc.flags.writeable = False\n\n# hamming 8/4 correctable errors occur when the input has even parity\nhamming8_cor = thue_morse(8, even=True)\nhamming8_cor.flags.writeable = False\n\n# hamming 8/4 uncorrectable errors always have odd parity,\n# but so do valid bytes.\nhamming8_unc = thue_morse(8, even=False)\nhamming8_unc[hamming8_enc] = False\nhamming8_unc.flags.writeable = False\n\n# hamming 8/4 decoding lookup table\nhamming8_dec = np.array([\n    0x1, 0xf, 0x1, 0x1, 0xf, 0x0, 0x1, 0xf, 0xf, 0x2, 0x1, 0xf, 0xa, 0xf, 0xf, 0x7,\n    0xf, 0x0, 0x1, 0xf, 0x0, 0x0, 0xf, 0x0, 0x6, 0xf, 0xf, 0xb, 0xf, 0x0, 0x3, 0xf,\n    0xf, 0xc, 0x1, 0xf, 0x4, 0xf, 0xf, 0x7, 0x6, 0xf, 0xf, 0x7, 0xf, 0x7, 0x7, 0x7,\n    0x6, 0xf, 0xf, 0x5, 0xf, 0x0, 0xd, 0xf, 0x6, 0x6, 0x6, 0xf, 0x6, 0xf, 0xf, 0x7,\n    0xf, 0x2, 0x1, 0xf, 0x4, 0xf, 0xf, 0x9, 0x2, 0x2, 0xf, 0x2, 0xf, 0x2, 0x3, 0xf,\n    0x8, 0xf, 0xf, 0x5, 0xf, 0x0, 0x3, 0xf, 0xf, 0x2, 0x3, 0xf, 0x3, 0xf, 0x3, 0x3,\n    0x4, 0xf, 0xf, 0x5, 0x4, 0x4, 0x4, 0xf, 0xf, 0x2, 0xf, 0xf, 0x4, 0xf, 0xf, 0x7,\n    0xf, 0x5, 0x5, 0x5, 0x4, 0xf, 0xf, 0x5, 0x6, 0xf, 0xf, 0x5, 0xf, 0xe, 0x3, 0xf,\n    0xf, 0xc, 0x1, 0xf, 0xa, 0xf, 0xf, 0x9, 0xa, 0xf, 0xf, 0xb, 0xa, 0xa, 0xa, 0xf,\n    0x8, 0xf, 0xf, 0xb, 0xf, 0x0, 0xd, 0xf, 0xf, 0xb, 0xb, 0xb, 0xa, 0xf, 0xf, 0xb,\n    0xc, 0xc, 0xf, 0xc, 0xf, 0xc, 0xd, 0xf, 0xf, 0xc, 0xf, 0xf, 0xa, 0xf, 0xf, 0x7,\n    0xf, 0xc, 0xd, 0xf, 0xd, 0xf, 0xd, 0xd, 0x6, 0xf, 0xf, 0xb, 0xf, 0xe, 0xd, 0xf,\n    0x8, 0xf, 0xf, 0x9, 0xf, 0x9, 0x9, 0x9, 0xf, 0x2, 0xf, 0xf, 0xa, 0xf, 0xf, 0x9,\n    0x8, 0x8, 0x8, 0xf, 0x8, 0xf, 0xf, 0x9, 0x8, 0xf, 0xf, 0xb, 0xf, 0xe, 0x3, 0xf,\n    0xf, 0xc, 0xf, 0xf, 0x4, 0xf, 0xf, 0x9, 0xf, 0xf, 0xf, 0xf, 0xf, 0xe, 0xf, 0xf,\n    0x8, 0xf, 0xf, 0x5, 0xf, 0xe, 0xd, 0xf, 0xf, 0xe, 0xf, 0xf, 0xe, 0xe, 0xf, 0xe,\n], dtype=np.uint8)\nhamming8_dec.flags.writeable = False\n\n# odd parity bits\nparity_tab = thue_morse(7, even=True) * 0x80\nparity_tab.flags.writeable = False\n\n# bit reverse\nreverse8_tab = np.packbits(np.unpackbits(np.array(range(256), dtype=np.uint8)).reshape((-1, 8))[:,::-1])\nreverse8_tab.flags.writeable = False\n\n\ndef hamming8_encode(a):\n    return hamming8_enc[a]\n\n\ndef hamming8_decode(a):\n    return hamming8_dec[a]\n\n\ndef hamming16_encode(a):\n    return np.ravel(np.column_stack((\n        hamming8_enc[a & 0xf],\n        hamming8_enc[a >> 4],\n    )))\n\n\ndef hamming16_decode(a):\n    if len(a) == 2:\n        return hamming8_dec[a[0]] | (hamming8_dec[a[1]] << 4)\n    else:\n        return hamming8_dec[a[0::2]] | (hamming8_dec[a[1::2]] << 4)\n\n\ndef hamming8_correctable_errors(a):\n    return hamming8_cor[a]\n\n\ndef hamming8_uncorrectable_errors(a):\n    return hamming8_unc[a]\n\n\ndef hamming8_errors(a):\n    return (2 * hamming8_unc[a]) + hamming8_cor[a]\n\n\ndef parity_encode(a):\n    return a | parity_tab[a]\n\n\ndef parity_decode(a):\n    return a & 0x7f\n\n\ndef parity_errors(a):\n    return parity_tab[a&0x7f] != a&0x80\n\n\ndef bcd8_decode(a):\n    tens = ((a >> 4) - 1) & 0xf\n    units = ((a & 0xf) - 1) & 0xf\n    return (tens * 10) + units\n\n\ndef bcd8_encode(a):\n    tens = ((a / 10) + 1) & 0xf\n    units = ((a % 10) + 1) & 0xf\n    return (tens << 4) | units\n\n\ndef byte_reverse(a):\n    return reverse8_tab[a]\n\n\ndef crc(n, c):\n    for b in range(7, -1, -1):\n        r = ((n >> b) & 1) ^ ((c >> 6) & 1) ^ ((c >> 8) & 1) ^ ((c >> 11) & 1) ^ ((c >> 15) & 1)\n        c = r | ((c & 0x7FFF) << 1)\n    return c\n"
  },
  {
    "path": "teletext/elements.py",
    "content": "import datetime\n\nfrom .printer import PrinterANSI\nfrom .coding import *\n\nfrom . import finders\n\n\nclass Element(object):\n\n    def __init__(self, shape, array=None):\n        if array is None:\n            self._array = np.zeros(shape, dtype=np.uint8)\n        elif type(array) == bytes:\n            self._array = np.frombuffer(array, dtype=np.uint8).copy()\n        else:\n            self._array = array\n\n        if self._array.shape != shape:\n            raise IndexError('Element got wrong shaped data.')\n\n    def __getitem__(self, item):\n        return self._array[item]\n\n    def __setitem__(self, item, value):\n        self._array[item] = value\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}({repr(self._array)})'\n\n    @property\n    def bytes(self):\n        return self._array.tobytes()\n\n    @property\n    def sevenbit(self):\n        return np.packbits(np.unpackbits(self._array).reshape(-1, 8)[:, 1:].flatten()).tobytes()\n\n    @sevenbit.setter\n    def sevenbit(self, b):\n        a = np.frombuffer(b, dtype=np.uint8)\n        s = self._array.size * 7\n        u = np.unpackbits(a)[:s].reshape(-1, 7)\n        self._array[:] = np.packbits(u, axis=-1).reshape(self._array.shape) >> 1\n\n    @property\n    def errors(self):\n        raise NotImplementedError\n\n\nclass ElementParity(Element):\n\n    @property\n    def errors(self):\n        return parity_errors(self._array)\n\n\nclass ElementHamming(Element):\n\n    @property\n    def errors(self):\n        return hamming8_errors(self._array)\n\n\nclass Mrag(ElementHamming):\n\n    def __init__(self, array=None):\n        super().__init__((2,), array)\n\n    @property\n    def magazine(self):\n        magazine = hamming8_decode(self._array[0]) & 0x7\n        return magazine or 8\n\n    @property\n    def row(self):\n        return hamming16_decode(self._array[:2]) >> 3\n\n    @magazine.setter\n    def magazine(self, magazine):\n        if magazine < 0 or magazine > 8:\n            raise ValueError('Magazine numbers must be between 0 and 8.')\n        self._array[0] = hamming8_encode((magazine & 0x7) | ((self.row&0x1) << 3))\n\n    @row.setter\n    def row(self, row):\n        if row < 0 or row > 31:\n            raise ValueError('Row numbers must be between 0 and 31.')\n        self._array[0] = hamming8_encode((self.magazine & 0x7) | ((row & 0x1) << 3))\n        self._array[1] = hamming8_encode(row >> 1)\n\n    def __str__(self):\n        return f'{self.magazine} {self.row} {self.errors}'\n\n\nclass Displayable(ElementParity):\n\n    def place_string(self, string, x=0, y=None):\n        if isinstance(string, str):\n            string = string.encode('ascii')\n        a = np.frombuffer(string, dtype=np.uint8)\n        if y is None:\n            self._array[x:x+a.shape[0]] = parity_encode(a)\n        else:\n            self._array[y, x:x + a.shape[0]] = parity_encode(a)\n\n    def place_bitmap(self, bitmap, x=1, y=0, colour=0x17, conceal=False):\n        yp, xp = bitmap.shape\n        yr = yp % 3\n        xr = xp % 2\n        if yr or xr:\n            bitmap = np.pad(bitmap, ((0, (3-yr)%3), (0, (2-xr)%2)))\n            yp, xp = bitmap.shape\n        yc = yp // 3\n        xc = xp // 2\n        mosaic = bitmap.reshape(yc, 3, xc, 2).swapaxes(1, 2).reshape(yc, xc, 6)\n        mosaic = np.packbits(mosaic, axis=2, bitorder=\"little\").reshape(yc, xc)\n        mosaic = mosaic + ((mosaic >= 0x20) * 0x20) + 0x20\n        if conceal:\n            self._array[y:y+yc, x-2] = parity_encode(colour)\n            self._array[y:y+yc, x-1] = parity_encode(0x18)\n        else:\n            self._array[y:y+yc, x-1] = parity_encode(colour)\n        self._array[y:y+yc, x:x+xc] = parity_encode(mosaic)\n\n    def to_ansi(self, colour=True):\n        if len(self._array.shape) == 1:\n            return str(PrinterANSI(self._array, colour))\n        else:\n            return '\\n'.join(\n                [str(PrinterANSI(a, colour)) for a in self._array]\n            )\n\n    def _tti_escape(self, array):\n        return ''.join(f'\\x1b{chr(x | 0x40)}' if 0 <= x <= 0x1f else chr(x) for x in (array & 0x7f))\n\n    def to_tti(self):\n        if len(self._array.shape) == 1:\n            return self._tti_escape(self._array)\n        else:\n            return [self._tti_escape(a) for a in self._array]\n\n    @property\n    def bytes_no_parity(self):\n        return (self._array & 0x7f).tobytes()\n\n\nclass Page(Element):\n\n    @property\n    def page(self):\n        return hamming16_decode(self._array[:2])\n\n    @page.setter\n    def page(self, page):\n        if page < 0 or page > 0xff:\n            raise ValueError('Page numbers must be between 0 and 0xff.')\n        self._array[:2] = hamming16_encode(page)\n\n    @property\n    def errors(self):\n        e = np.zeros_like(self._array)\n        e[:2] = hamming8_errors(self._array[:2])\n        return e\n\n\nclass Header(Page):\n\n    def __init__(self, array):\n        super().__init__((40,), array)\n\n    @property\n    def subpage(self):\n        values = hamming16_decode(self._array[2:6])\n        return (values[0] & 0x7f) | ((values[1] & 0x3f) <<8)\n\n    @property\n    def control(self):\n        values = hamming16_decode(self._array[2:8])\n        return (values[0] >> 7) | ((values[1] >> 5) & 0x6) | (values[2] << 3)\n\n    @property\n    def displayable(self):\n        return Displayable((32,), self._array[8:])\n\n    @property\n    def codepage(self):\n        return (self.control >> 8) & 0x7\n\n    @property\n    def newsflash(self):\n        return bool(self.control & 0x2)\n\n    @property\n    def subtitle(self):\n        return bool(self.control & 0x4)\n\n    @property\n    def supress_header(self):\n        return bool(self.control & 0x8)\n\n    @subpage.setter\n    def subpage(self, subpage):\n        if subpage < 0 or subpage > 0x3f7f:\n            raise ValueError('Subpage numbers must be between 0 and 0x3f7f.')\n        control = self.control\n        self._array[2:6] = hamming16_encode(np.array([\n            (subpage & 0x7f) | ((control & 1) << 7),\n            (subpage >> 8) | ((control & 6) << 6),\n        ], dtype=np.uint8))\n\n    @control.setter\n    def control(self, control):\n        if control < 0 or control > 2047:\n            raise ValueError('Control bits must be between 0 and 2047.')\n        subpage = self.subpage\n        self._array[3] = hamming8_encode(((subpage >> 4) & 0x7) | ((control & 1) << 3))\n        self._array[5] = hamming8_encode(((subpage >> 12) & 0x3) | ((control & 6) << 1))\n        self._array[6:8] = hamming16_encode(control >> 3)\n\n    def to_ansi(self, colour=True):\n        return f'{self.page:02x} {self.displayable.to_ansi(colour)}'\n\n    def apply_finders(self):\n        ranks = [(f.match(self.displayable[:]),f) for f in finders.HeaderFinders]\n        ranks.sort(reverse=True, key=lambda x: x[0])\n        if ranks[0][0] > 20:\n            self.finder = ranks[0][1]\n            self.finder.fixup(self.displayable[:])\n\n    @property\n    def errors(self):\n        e = super().errors\n        e[2:8] = hamming8_errors(self._array[2:8])\n        e[8:] = self.displayable.errors\n        return e\n\n\nclass Triplet(Element):\n\n    def __init__(self, array):\n        super().__init__((3,), array)\n\n    def __str__(self):\n        return f'triplet'\n\n    @property\n    def errors(self):\n        e = super().errors\n        e[:] = hamming8_errors(self._array[:])\n        return e\n\n\nclass PageLink(Page):\n\n    def __init__(self, array, mrag):\n        super().__init__((6,), array)\n        self._mrag = mrag\n\n    @property\n    def subpage(self):\n        values = hamming16_decode(self._array[2:6])\n        return (values[0] & 0x7f) | ((values[1] & 0x3f) <<8)\n\n    @property\n    def magazine(self):\n        values = hamming16_decode(self._array[2:6])\n        magazine = ((values[0] >> 7) | ((values[1] >> 5) & 0x6)) ^ (self._mrag.magazine & 0x7)\n        return magazine or 8\n\n    @subpage.setter\n    def subpage(self, subpage):\n        if subpage < 0 or subpage > 0x3f7f:\n            raise ValueError('Subpage numbers must be between 0 and 0x3f7f.')\n        magazine = self.magazine\n        self._array[2:6] = hamming16_encode(np.array([\n            (subpage & 0x7f) | ((magazine & 1) << 7),\n            (subpage >> 8) | ((magazine & 6) << 6),\n        ]))\n\n    @magazine.setter\n    def magazine(self, magazine):\n        if magazine < 0 or magazine > 8:\n            raise ValueError('Magazine numbers must be between 0 and 8.')\n        magazine = magazine ^ self._mrag.magazine\n        subpage = self.subpage\n        self._array[3:6:2] = hamming8_encode([\n            ((subpage >> 4) & 0x7) | ((magazine & 1) << 3),\n            ((subpage >> 12) & 0x3) | ((magazine & 6) << 1),\n        ])\n\n    def __str__(self):\n        return f'{self.magazine}{self.page:02x}:{self.subpage:04x}'\n\n    @property\n    def errors(self):\n        e = super().errors\n        e[:] = hamming8_errors(self._array[:])\n        return e\n\n\nclass DesignationCode(Element):\n\n    @property\n    def dc(self):\n        return hamming8_decode(self._array[0])\n\n    @dc.setter\n    def dc(self, dc):\n        self._array[0] = hamming8_encode(dc)\n\n    @property\n    def errors(self):\n        e = np.zeros_like(self._array)\n        e[0] = hamming8_errors(self._array[0])\n        return e\n\n\nclass Triplets(DesignationCode):\n\n    def __init__(self, array, mrag):\n        super().__init__((40,), array)\n        self._mrag = mrag\n\n    @property\n    def triplets(self):\n        return tuple(Triplet(self._array[n:n+3]) for n in range(1, 40, 3))\n\n    def to_ansi(self, colour=True):\n        return f'DC={self.dc:x} ' + ' '.join((str(triplet) for triplet in self.triplets))\n\n    @property\n    def errors(self):\n        e = super().errors\n        for t,n in zip(self.triplets, range(1, 40, 3)):\n            e[n:n+3] = t.errors\n        return e\n\n\nclass Fastext(DesignationCode):\n\n    def __init__(self, array, mrag):\n        super().__init__((40,), array)\n        self._mrag = mrag\n\n    @property\n    def links(self):\n        return tuple(PageLink(self._array[n:n+6], self._mrag) for n in range(1, 36, 6))\n\n    @property\n    def control(self):\n        return hamming8_decode(self._array[37])\n\n    @control.setter\n    def control(self, value):\n        self._array[37] = hamming8_encode(value)\n\n    @property\n    def checksum(self):\n        return self._array[38]<<8 | self._array[39]\n\n    @checksum.setter\n    def checksum(self, value):\n        self._array[38] = value>>8\n        self._array[39] = value&0xff\n\n    def to_ansi(self, colour=True):\n        return f'DC={self.dc:x} ' + ' '.join((str(link) for link in self.links)) + f' CRC={self.checksum:04x}'\n\n    @property\n    def errors(self):\n        e = super().errors\n        for l,n in zip(self.links, range(1, 36, 6)):\n            e[n:n+6] = l.errors\n        return e\n\n\nclass Format1(Element):\n\n    epoch = datetime.date(1858, 11, 17)\n\n    def __init__(self, array):\n        super().__init__((9,), array)\n\n    @property\n    def network(self):\n        return (byte_reverse(self._array[0]) << 8) | byte_reverse(self._array[1])\n\n    @property\n    def offset(self):\n        hours = 0.5 * ((self._array[2] >> 1) & 0x1f)\n        if ((self._array[2] >> 6) & 0x01):\n            hours *= -1\n        return hours\n\n    @property\n    def mjd(self):\n        return (bcd8_decode((int(self._array[3])&0xf)|0x10) * 10000) + (bcd8_decode(int(self._array[4])) * 100) + bcd8_decode(int(self._array[5]))\n\n    @property\n    def date(self):\n        return self.epoch + datetime.timedelta(days=int(self.mjd))\n\n    @property\n    def hour(self):\n        return bcd8_decode(self._array[6])\n\n    @hour.setter\n    def hour(self, value):\n        self._array[6] = bcd8_encode(value)\n\n    @property\n    def minute(self):\n        return bcd8_decode(self._array[7])\n\n    @minute.setter\n    def minute(self, value):\n        self._array[7] = bcd8_encode(value)\n\n    @property\n    def second(self):\n        return bcd8_decode(self._array[8])\n\n    @second.setter\n    def second(self, value):\n        self._array[8] = bcd8_encode(value)\n\n    def to_ansi(self, colour=True):\n        return f'NI={self.network:04x} {self.date} {self.hour:02d}:{self.minute:02d}:{self.second:02d} {self.offset}'\n\n    @property\n    def errors(self):\n        #TODO: detect invalid dates and times\n        return 0\n\n\nclass Format2(Element):\n\n    def __init__(self, array):\n        super().__init__((13,), array)\n\n    @property\n    def day(self):\n        return byte_reverse(((hamming16_decode(self._array[3:5]) >> 2) & 0x1f)) >> 3\n\n    @property\n    def month(self):\n        return byte_reverse((hamming16_decode(self._array[4:6]) >> 3) & 0x0f) >> 4\n\n    @property\n    def hour(self):\n        return byte_reverse((hamming16_decode(self._array[5:7]) >> 3) & 0x1f) >> 3\n\n    @property\n    def minute(self):\n        return byte_reverse(hamming16_decode(self._array[7:9]) & 0x3f) >> 2\n\n    @property\n    def country(self):\n        return byte_reverse(hamming8_decode(self._array[2]) | ((hamming8_decode(self._array[8]) & 0xC) << 2) | ((hamming8_decode(self._array[9]) & 0x3) << 6))\n\n    @property\n    def network(self):\n        return byte_reverse((hamming8_decode(self._array[3]) & 0x3) | (hamming8_decode(self._array[9]) & 0xC) | (hamming8_decode(self._array[10]) << 4))\n\n    def to_ansi(self, colour=True):\n        return f'NI={self.network:02x} C={self.country:02x} {self.day}/{self.month} {self.hour:02d}:{self.minute:02d}'\n\n\nclass BroadcastData(DesignationCode):\n\n    def __init__(self, array, mrag):\n        super().__init__((40,), array)\n        self._mrag = mrag\n\n    @property\n    def displayable(self):\n        return Displayable((20,), self._array[20:])\n\n    @property\n    def initial_page(self):\n        return PageLink(self._array[1:7], self._mrag)\n\n    @property\n    def format1(self):\n        return Format1(self._array[7:16])\n\n    @property\n    def format2(self):\n        return Format2(self._array[7:20])\n\n    def to_ansi(self, colour=True):\n        if self.dc in [0, 1]:\n            return f'{self.displayable.to_ansi(colour)} DC={self.dc} IP={self.initial_page} {self.format1.to_ansi(colour)}'\n        elif self.dc in [2, 3]:\n            return f'{self.displayable.to_ansi(colour)} DC={self.dc} IP={self.initial_page} {self.format2.to_ansi(colour)}'\n        else:\n            return f'DC={self.dc}'\n\n    @property\n    def errors(self):\n        e = super().errors\n        e[1:7] = self.initial_page.errors\n        e[20:] = self.displayable.errors\n        return e\n\n\nclass Celp(Element):\n\n    dblevels = [0, 4, 8, 12, 18, 24, 30, 0]\n\n    servicetypes = [\n        'Single-channel mode using 1 VBI line per frame',\n        'Single-channel mode using 2 VBI lines per frame',\n        'Single-channel mode using 3 VBI lines per frame',\n        'Single-channel mode using 4 VBI lines per frame',\n        'Mute Channel 1',\n        'Two-channel Mode using 2 VBI lines per frame',\n        'Mute Channel 2',\n        'Two-channel Mode using 4 VBI lines per frame',\n    ]\n\n    fmt = 'DCN: {dcn}, {rel}, S: {svc:>7s}, C: {ctl} {frame0} {frame1}'\n\n    def __init__(self, array, mrag):\n        super().__init__((40,), array)\n        self._mrag = mrag\n\n    @property\n    def dcn(self):\n        return self._mrag.magazine + ((self._mrag.row & 1) << 3)\n\n    @property\n    def service(self):\n        return hamming8_decode(self._array[0])\n\n    @service.setter\n    def service(self, service):\n        self._array[0] = hamming8_encode(service)\n\n    @property\n    def control(self):\n        return hamming8_decode(self._array[1])\n\n    @control.setter\n    def control(self, service):\n        self._array[0] = hamming8_encode(service)\n\n    def to_ansi(self, colour=True):\n        frame0 = self._array[2:21].tobytes().hex()\n        frame1 = self._array[21:40].tobytes().hex()\n\n        if self.dcn == 4:\n            return self.fmt.format(\n                dcn = self.dcn,\n                rel = 'Programme-related audio',\n                svc = 'AUDETEL' if self.service == 0 else hex(self.service),\n                ctl = f'{hex(self.control)} {self.dblevels[self.control & 0x7]:2d} dB {\"muted\" if self.control & 0x8 else \"\"}',\n                frame0 = frame0,\n                frame1 = frame1,\n            )\n        elif self.dcn == 12:\n            if self.service & 0x8:\n                return self.fmt.format(\n                    dcn=self.dcn,\n                    rel='Programme-independent audio',\n                    svc=f'User-defined {hex(self.service&0x7)}',\n                    ctl=hex(self._array[1]),\n                    frame0=frame0,\n                    frame1=frame1,\n                )\n            else:\n                return self.fmt.format(\n                    dcn=self.dcn,\n                    rel='Programme-independent audio',\n                    svc=self.servicetypes[self.service],\n                    ctl=hex(self.control),\n                    frame0=frame0,\n                    frame1=frame1,\n                )\n        else:\n            raise ValueError(\"Unexpected data channel for CELP.\")\n\n    @property\n    def errors(self):\n        e = np.zeros_like(self._array)\n        e[0:2] = hamming8_errors(self._array[0:2])\n        return e\n"
  },
  {
    "path": "teletext/file.py",
    "content": "import io\nimport itertools\nimport os\nimport stat\n\n\ndef PossiblyInfiniteRange(start=0, stop=None, step=1, limit=None):\n    if stop is None:\n        if limit is None:\n            return itertools.count(start, step)\n        else:\n            return range(start, start + (limit * step), step)\n    else:\n        if limit is None:\n            return range(start, stop, step)\n        else:\n            return range(start, min(stop, start + (limit * step)), step)\n\n\nclass LenWrapper(object):\n    def __init__(self, i, l):\n        self.i = i\n        self.l = l\n\n    def __iter__(self):\n        return self.i\n\n    def __len__(self):\n        return self.l\n\n\ndef _chunks(f, size, flines, frange, seek):\n    while True:\n        if seek:\n            f.seek(size * frange.start, os.SEEK_CUR)\n        else:\n            f.read(size * frange.start)\n        for _ in frange:\n            b = f.read(size)\n            if len(b) < size:\n                return\n            yield b\n        if seek:\n            f.seek(size * (flines - frange.stop), os.SEEK_CUR)\n        else:\n            f.read(size * (flines - frange.stop))\n\n\ndef chunks(f, size, start, step, flines=16, frange=(0, 16), seek=True):\n    while True:\n        c = _chunks(f, size, flines, frange, seek)\n        try:\n            for _ in range(start):\n                next(c)\n            while True:\n                yield next(c)\n                for i in range(step-1):\n                    next(c)\n        except StopIteration:\n            if seek:\n                f.seek(0, os.SEEK_SET)\n            else:\n                return\n\ndef FileChunker(f, size, start=0, stop=None, step=1, limit=None, flines=16, frange=range(0, 16), loop=False, dup_stdin=False):\n    seekable = False\n    try:\n        if hasattr(f, 'fileno') and stat.S_ISFIFO(os.fstat(f.fileno()).st_mode):\n            if dup_stdin and f.fileno() == 0:\n                f = os.fdopen(os.dup(f.fileno()), 'rb')\n            raise io.UnsupportedOperation\n\n        f.seek(0, os.SEEK_END)\n        total_lines = f.tell() // size\n        total_fields = total_lines // flines\n        remainder = max(min((total_lines % flines) - frange.start, len(frange)), 0)\n        useful_lines = (total_fields * len(frange)) + remainder\n\n        if stop is None:\n            stop = useful_lines\n        else:\n            stop = min(stop, useful_lines)\n\n        seekable = True\n        f.seek(0, os.SEEK_SET)\n\n    except io.UnsupportedOperation:\n        # chunks() always seeks to the start\n        pass\n\n    r = PossiblyInfiniteRange(start, None if loop else stop, step, limit)\n    i = zip(r, chunks(f, size, start, step, flines, frange, seek=seekable))\n    if hasattr(r, '__len__'):\n        return LenWrapper(i, len(r))\n    else:\n        return i\n"
  },
  {
    "path": "teletext/finders.py",
    "content": "import itertools\n\nfrom .coding import parity_encode, parity_decode\n\n\nclass Finder(object):\n\n    groups = {\n        'c': b'abcdefghijklmnopqrstuvwxyz ',\n        'C': b'ABCDEFGHIJKLMNOPQRSTUVWXYZ ',\n        'D': b'MTWFS',\n        'd': b'mtwfs',\n        'A': b'OUEHRA',\n        'a': b'ouehra',\n        'Y': b'NEDUIT',\n        'y': b'neduit',\n        'M': b'JFMASOND',\n        'm': b'jfmasond',\n        'O': b'AEPUCO',\n        'o': b'aepuco',\n        'N': b'NBRYNLGPTVC',\n        'n': b'nbrynlgptvc',\n        'Z': b'12345678',\n        'T': b'0123456789ABCDEFabcdef',\n        'U': b'0123456789ABCDEFabcdef',\n        'F': b'0123 ',\n        'f': b'0123456789',\n        'H': b'012 ',\n        'h': b'0123456789',\n        'L': b'012345',\n        'l': b'0123456789',\n        'S': b'012345',\n        's': b'0123456789',\n        'e': b'', # exact match\n        '*': b'', # wildcard\n        ' ': b'\\x00\\x01\\x02\\x03\\x04\\x05\\x05\\x06\\x07\\x08\\x09\\x0a\\x0b\\x0c\\x0d\\x0e\\x0f\\x10\\x11\\x12\\x13\\x14\\x15\\x16\\x17\\x18\\x19\\x1a\\x1b\\x1c\\x1d\\x1e\\x1f ', # whitespace/spacing attributes\n    }\n\n    def __init__(self, match1, match2, name, years, channels):\n        if len(match1) != len(match2):\n            raise ValueError('Match fields must be equal length.')\n        self.match1 = match1.encode('ascii')\n        self.match2 = match2\n        self.name = name\n        self.years = years\n        self.channels = channels\n\n    def match(self, arr):\n        a = [2 if (m2 == 'e' and m1 == c) else (1 if c in self.groups[m2] else 0) for m1, m2, c in zip(self.match1, self.match2, parity_decode(arr))]\n        #print(f'{self.name[:20]:21s}', ''.join(str(n) for n in a))\n        return sum(a)\n\n    def fixup(self, arr):\n        for n, m1, m2 in zip(itertools.count(), self.match1, self.match2):\n            if m2 == 'e':\n                arr[n] = parity_encode(m1)\n\n\nHeaderFinders = [\nFinder(\"CEEFAX 217 \\x09Wed 25 Dec\\x03 18:29/53\",\n       \"eeeeeeeZTUee\"+\"DayeFfeMone\"+\"eHheLleSs\",\n       \"BBC\", (0,1996), ['BBC1', 'BBC2']),\n\nFinder(\"CEEFAX 1 217 Wed 25 Dec\\x0318:29/53\",\n       \"eeeeeeeeeZTUeDayeFfeMoneHhe\"+\"LleSs\",\n       \"BBC1\", (1996,3000), ['BBC1']),\n\nFinder(\"CEEFAX 2 217 Wed 25 Dec\\x0318:29/53\",\n       \"eeeeeeeeeZTUeDayeFfeMone\"+\"HheLleSs\",\n       \"BBC2\", (1996,3000), ['BBC2']),\n\nFinder(\"Central  217 Wed 25 Dec 18:29:53\",\n       \"eeeeeeeeeZTUeDayeFfeMoneHheLleSs\",\n       \"Central\", (0, 3000), ['ITV']),\n\nFinder(\"\\x02   ITV SUBTITLES               \",\n       \"e\"+\"eeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\",\n       \"ITV Subs.\", (0, 3000), ['ITV']),\n\nFinder(\"\\x01\\x1d\\x07 DBI STATUS PAGE   \\x1c  2059:27\",\n       \"e\"+\"e\"+\"e\"+\"eeeeeeeeeeeeeeeeeeee\"+\"eeHhLleSs\",\n       \"ITV DBI Stat.\", (0, 3000), ['ITV']),\n\nFinder(\"                         2059:27\",\n       \"eeeeeeeeeeeeeeeeeeeeeeeeeHhLleSs\",\n       \"ITV DBI Blank\", (0, 3000), ['ITV']),\n\nFinder(\"                                \",\n       \"eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\",\n       \"Subs Blank\", (0, 3000), ['BBC1', 'BBC2', 'ITV', 'C4', 'Five']),\n\nFinder(\"\\x01789 DBI TEST PAGE 789\\x07  2059:27\",\n       \"e\"+\"ZTUeeeeeeeeeeeeeeeZTUe\"+\"eeHhLleSs\",\n       \"ITV DBI Test.\", (0, 3000), ['ITV']),\n\nFinder(\"\\x01   DBI/CH4 - BCAST2  \\x09\\x07 2059:27\",\n       \"e\"+\"ZTUeeeeeeeeeeeeeeeeeee\"+\"e\"+\"eHhLleSs\",\n       \"C4 DBI Test.\", (0, 3000), ['C4']),\n\nFinder(\" 500     mon 12  may     2059:27\",\n       \"eZTUeeeeedayeFfeemoneeeeeHhLleSs\",\n       \"Five\", (1997,1997), ['Five']),\n\nFinder(\"\\x06   5 text   \\x07255 02 May\\x031835:21\",\n       \"e\"+\"eeeeeeeeeeeee\"+\"ZTUeFfeMone\"+\"HhLleSs\",\n       \"Five\", (1997, 2006), ['Five']),\n\nFinder(\"Five 500  27 Nov        20:59.27\",\n       \"eeeeeZTUeeFfeMonemoneeeeHheLleSs\",\n       \"Five\", (1999,3000), ['Five']),\n\nFinder(\"SOFTEL D1 SUBTITLE INSERTER     \",\n       \"eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\",\n       \"Five Subs.\", (0,3000), ['Five']),\n\nFinder(\"\\x04\\x1d\\x03Teletext\\x07 \\x1c100 May05\\x0318:29:53\",\n       \"e\"+\"e\"+\"e\"+\"eeeeeeeee\"+\"ee\"+\"ZTUeMonFfe\"+\"HheLleSs\",\n       \"Teletext Ltd.\", (1993, 3000), ['ITV', 'C4']),\n\nFinder(\"\\x04\\x1d\\x03Teletext \\x03\\x1c100 May05\\x0318:29:53\",\n       \"e\"+\"e\"+\"e\"+\"eeeeeeeeee\"+\"e\"+\"ZTUeMonFfe\"+\"HheLleSs\",\n       \"Teletext Ltd. (Five)\", (1999, 3000), ['Five']),\n\nFinder(\"\\x04\\x1d\\x03Teletext\\x07 \\x1c100\\x03May05\\x0318:29:53\",\n       \"e\"+\"e\"+\"e\"+\"eeeeeeeee\"+\"ee\"+\"ZTUe\"+\"MonFfe\"+\"HheLleSs\",\n       \"Teletext Ltd. (Five)\", (1999, 3000), ['Five']),\n\nFinder(\"4-Tel 307 Sun 26 May\\x03C4\\x0718:29:53\",\n       \"eeeeeeZTUeDayeFfeMone\"+\"eee\"+\"HheLleSs\",\n       \"4-Tel\", (0, 3000), ['C4']),\n\nFinder(\"PLEASE REFER TO PAGE 100 2001:01\",\n       \"eeeeeeeeeeeeeeeeeeeeeeeeeHhLleSs\",\n       \"Oracle Filler\", (0,1992), ['ITV', 'C4']),\n\nFinder(\"ORACLE 200 Sun27 Dec\\x03ITV\\x032001:01\",\n       \"eeeeeeeZTUeDayFfeMone\"+\"eeee\"+\"HhLleSs\",\n       \"Oracle (ITV)\", (0,1992), ['ITV']),\n\nFinder(\"Teletext on 4 100 Jan25\\x0320:01:01\",\n       \"eeeeeeeeeeeeeeZTUeMonFfe\"+\"HheLleSs\",\n       \"Teletext Ltd. (C4 - Early)\", (1993,1993), ['C4']),\n\nFinder(\"100\\x02ARD/ZDF\\x07Mo 26.12.88\\x0222:00:00\",\n       \"ZTUe\"+\"eeeeeeee\"+\"Daeeeeeeeeee\"+\"HheLleSs\",\n       \"ARD\", (0,3000), ['ARD']),\n\nFinder(\"102 BELTEK              22:07:06\",\n       \"ZTU CCCCCCCCCCCCCCCCCCC HheLleSs\",\n       \"TVR\", (0,3000), ['ARD']),\n\n]\n\n"
  },
  {
    "path": "teletext/gui/__init__.py",
    "content": ""
  },
  {
    "path": "teletext/gui/classify.py",
    "content": "import pathlib\nimport sys\n\nimport numpy as np\nfrom PyQt5 import QtWidgets, QtCore\n\nimport matplotlib.pyplot as plt\nfrom matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas\nfrom matplotlib.figure import Figure\n\n\nclass Window(QtWidgets.QMainWindow):\n    def __init__(self, dir, sampledir, auto, config, app, parent=None):\n        super(Window, self).__init__(parent)\n\n        self.app = app\n        self.auto = auto\n        self.config = config\n        self.dir = pathlib.Path(dir)\n        self.sampledir = pathlib.Path(sampledir)\n        self.files = []\n        for f in self.dir.iterdir():\n            if f.is_file() and f.suffix == '.vbi':\n                s = f.stat().st_size // self.config.line_bytes\n                self.files.append((f, s))\n            elif f.is_dir():\n                for g in f.iterdir():\n                    if g.is_file() and g.suffix == '.vbi':\n                        s = g.stat().st_size // self.config.line_bytes\n                        self.files.append((g, s))\n\n        print(len(self.files))\n        self.files.sort(key=lambda x: x[1], reverse=True)\n        #self.files = self.files[1000:]\n        self.current_file = 0\n\n        self.setWindowTitle(\"VBI Classify\")\n        w = QtWidgets.QWidget(self)\n        self.setCentralWidget(w)\n        layout = QtWidgets.QVBoxLayout(w)\n        w.setLayout(layout)\n\n        f = QtWidgets.QFrame()\n        f.setFrameShape(QtWidgets.QFrame.Shape.Panel)\n        f.setFrameShadow(QtWidgets.QFrame.Shadow.Sunken)\n        f.setLineWidth(2)\n        fl = QtWidgets.QVBoxLayout()\n        fl.setContentsMargins(0, 0, 0, 0)\n        f.setLayout(fl)\n\n        self.canvas = FigureCanvas()\n        fl.addWidget(self.canvas)\n        layout.addWidget(f, 1)\n\n        self.buttons = {}\n        btntmp = ['teletext', 'negatext', 'quiet', 'empty', 'noise', 'mixed']\n        if not self.auto:\n            self.g = QtWidgets.QWidget(w)\n            self.bbox = QtWidgets.QGridLayout(self.g)\n            self.g.setLayout(self.bbox)\n            layout.addWidget(self.g)\n\n            self.buttonMapper = QtCore.QSignalMapper(self)\n            self.buttonMapper.mappedString.connect(self.button_pressed)\n\n            for f in sorted(self.sampledir.iterdir()):\n                if f.is_dir():\n                    if f.name not in btntmp:\n                        btntmp.append(f.name)\n\n            btntmp.append('skip')\n\n            for y in range(10):\n                for x in range(5):\n                    try:\n                        self.add_button(btntmp[(y*5)+x], (y, x))\n                    except IndexError:\n                        break\n\n        self.progress = QtWidgets.QProgressBar()\n        self.progress.setVisible(False)\n        self.progress.setFixedWidth(200)\n        self.statusBar().addPermanentWidget(self.progress)\n\n        self.enable_buttons(False)\n        w.setLayout(layout)\n        self.resize(1000, 600)\n        self.show()\n        self.load()\n\n    def add_button(self, label, pos):\n        b = QtWidgets.QPushButton(self.g)\n        b.setText(label)\n        #b.setSizePolicy(QtWidgets.QSizePolicy.Policy.Expanding, QtWidgets.QSizePolicy.Policy.Expanding)\n        b.setMinimumHeight(b.height()+10)\n        b.clicked.connect(self.buttonMapper.map)\n        self.buttonMapper.setMapping(b, label)\n        self.bbox.addWidget(b, *pos)\n        self.buttons[label] = b\n\n    def enable_buttons(self, en):\n        for b in self.buttons.values():\n            b.setEnabled(en)\n\n    def button_pressed(self, label):\n        if label != 'skip':\n            src = self.files[self.current_file][0]\n            (self.sampledir / label).mkdir(parents=True, exist_ok=True)\n            src.rename(self.sampledir / label / src.name)\n            print(self.files[self.current_file], '->', label)\n        self.current_file += 1\n        if self.current_file >= len(self.files):\n            print(\"All done\")\n            self.app.quit()\n        else:\n            self.enable_buttons(False)\n            self.load()\n\n    def load(self):\n        f = self.files[self.current_file]\n        self.statusBar().showMessage(f'{self.current_file + 1}/{len(self.files)} - {f[0]} - {f[1]} lines')\n\n        try:\n            result = np.fromfile(f[0].with_suffix('.hist'), dtype=np.uint32).reshape(256, self.config.line_length)\n            self.plot(result)\n            if self.auto:\n                self.button_pressed('skip')\n        except FileNotFoundError:\n            self.load_thread = VBILoader(f[0], self.config)\n            self.load_thread.total.connect(self.progress.setMaximum)\n            self.load_thread.update.connect(self.progress.setValue)\n            self.load_thread.finished.connect(self.loaded)\n\n            self.progress.setValue(0)\n            self.load_thread.start()\n            self.progress.setVisible(True)\n\n    def loaded(self):\n        result = self.load_thread.result\n        del self.load_thread\n        self.progress.setVisible(False)\n        if self.auto:\n            f = self.files[self.current_file][0]\n            f = f.with_suffix('.hist')\n            result.tofile(f)\n            self.button_pressed('skip')\n        self.plot(result)\n\n    def plot(self, result):\n        fig = Figure()\n        ax = fig.subplots(1, 1)\n        aa = result == 0\n        mn = np.argmin(aa, axis=0)\n        mx = 255 - np.argmin(aa[::-1, :], axis=0)\n\n        ax.imshow(result, origin=\"lower\", cmap=\"hot\")\n        ax.plot(mn, linewidth=1)\n        ax.plot(mx, linewidth=1)\n        fig.tight_layout(pad=0)\n\n        self.canvas.figure = fig\n        x, y = self.width(), self.height()\n        self.resize(x+1, y+1)\n        self.resize(x, y)\n\n        # refresh canvas\n        self.canvas.draw()\n        plt.close(fig)\n        self.enable_buttons(True)\n\nclass VBILoader(QtCore.QThread):\n    total = QtCore.pyqtSignal(int)\n    update = QtCore.pyqtSignal(int)\n\n    def __init__(self, file, config):\n        self.file = file\n        self.config = config\n        super().__init__()\n\n    def run(self):\n        arr = np.memmap(self.file, dtype=self.config.dtype).reshape(-1, self.config.line_length)\n        h = np.zeros((256, self.config.line_length), dtype=np.uint32)\n        sel = np.arange(self.config.line_length)\n        shift = (np.dtype(self.config.dtype).itemsize - 1) * 8\n        self.total.emit(arr.shape[0])\n        for n in range(arr.shape[0]):\n            l = arr[n] >> shift\n            h[l, sel] += 1\n            self.update.emit(n)\n\n        for j in range(self.config.line_length):\n            h[:,j] = 255*h[:,j]/np.max(h[:,j])\n        self.result = h\n\ndef classify_gui(dir, sampledir, auto, config):\n    app = QtWidgets.QApplication.instance()\n    if app is None:\n        app = QtWidgets.QApplication(sys.argv)\n\n    main = Window(dir, sampledir, auto, config, app)\n    main.show()\n    app.exec_()\n"
  },
  {
    "path": "teletext/gui/decoder.py",
    "content": "import os\nimport random\nimport sys\nimport webbrowser\n\nimport numpy as np\nfrom PyQt5.QtCore import QSize, QObject, QUrl\nfrom PyQt5.QtGui import QFont, QColor\n\nfrom teletext.parser import Parser\n\n\nclass Palette(object):\n\n    def __init__(self, context):\n        self._context = context\n        self._palette = [\n            QColor(0, 0, 0),\n            QColor(255, 0, 0),\n            QColor(0, 255, 0),\n            QColor(255, 255, 0),\n            QColor(0, 0, 255),\n            QColor(255, 0, 255),\n            QColor(0, 255, 255),\n            QColor(255, 255, 255),\n        ]\n        self._context.setContextProperty('ttpalette', self._palette)\n\n    def __getitem__(self, item):\n        return (self._palette[item].red(), self._palette[item].green(), self._palette[item].blue())\n\n    def __setitem__(self, item, value):\n        self._palette[item].setRed(value[0])\n        self._palette[item].setGreen(value[1])\n        self._palette[item].setBlue(value[2])\n        self._context.setContextProperty('ttpalette', self._palette)\n\n\nclass ParserQML(Parser):\n\n    def __init__(self, tt, row, cells, nextrow):\n        self._row = row\n        self._cells = cells\n        self._nextrow = nextrow\n        super().__init__(tt)\n\n    def emitcharacter(self, c):\n        self._cells[self._cell].setProperty('c', c)\n        for state, value in self._state.items():\n            self._cells[self._cell].setProperty(state, value)\n        self._dh |= self._state['dh']\n        self._cell += 1\n\n    def parse(self):\n        self._cell = 0\n        self._dh = False\n        super().parse()\n        self._row.setProperty('rowheight', 2 if self._dh else 1)\n        if self._nextrow:\n            self._nextrow.setProperty('rowrendered', not (self._row.property('rowrendered') and self._dh))\n\n\nclass Decoder(object):\n\n    def __init__(self, widget):\n\n        self.widget = widget\n\n        self._fonts = [\n            [\n                [self.make_font(100), self.make_font(50)],\n                [self.make_font(200), self.make_font(100)]\n            ],\n            [\n                [self.make_font(120), self.make_font(60)],\n                [self.make_font(240), self.make_font(120)]\n            ]\n        ]\n\n        self.widget.rootContext().setContextProperty('ttfonts', self._fonts)\n        self._palette = Palette(self.widget.rootContext())\n\n        qml_file = os.path.join(os.path.dirname(__file__), 'decoder.qml')\n        self.widget.setSource(QUrl.fromLocalFile(qml_file))\n\n        self._rows = [self.widget.rootObject().findChild(QObject, 'rows').itemAt(x) for x in range(25)]\n        self._cells = [[r.findChild(QObject, 'cols').itemAt(x) for x in range(40)] for r in self._rows]\n        self._data = np.zeros((25, 40), dtype=np.uint8)\n        self._parsers = [ParserQML(self._data[x], self._rows[x], self._cells[x], self._rows[x+1] if x < 24 else None) for x in range(25)]\n\n        self.zoom = 2\n\n    def __setitem__(self, item, value):\n        self._data[item] = value\n        if isinstance(item, tuple):\n            item = item[0]\n        if isinstance(item, int):\n            self._parsers[item].parse()\n        else:\n            for p in self._parsers[item]:\n                p.parse()\n\n    def __getitem__(self, item):\n        return self._data[item]\n\n    def randomize(self):\n        self[1:] = np.random.randint(0, 256, size=(24, 40), dtype=np.uint8)\n\n    def make_font(self, stretch):\n        font = QFont('teletext2')\n        font.setStyleStrategy(QFont.NoSubpixelAntialias)\n        font.setHintingPreference(QFont.PreferNoHinting)\n        font.setStretch(stretch)\n        return font\n\n    @property\n    def palette(self):\n        return self._palette\n\n    @property\n    def zoom(self):\n        return self.widget.rootObject().property('zoom')\n\n    @zoom.setter\n    def zoom(self, zoom):\n        if 0 < zoom < 5:\n            self._fonts[0][0][0].setPixelSize(zoom * 10)\n            self._fonts[0][0][1].setPixelSize(zoom * 20)\n            self._fonts[0][1][0].setPixelSize(zoom * 10)\n            self._fonts[0][1][1].setPixelSize(zoom * 20)\n            self._fonts[1][0][0].setPixelSize(zoom * 10)\n            self._fonts[1][0][1].setPixelSize(zoom * 20)\n            self._fonts[1][1][0].setPixelSize(zoom * 10)\n            self._fonts[1][1][1].setPixelSize(zoom * 20)\n            self.widget.rootContext().setContextProperty('ttfonts', self._fonts)\n            self.widget.rootObject().setProperty('zoom', zoom)\n            self.widget.setFixedSize(self.size())\n\n    @property\n    def reveal(self):\n        return self.widget.rootObject().property('reveal')\n\n    @reveal.setter\n    def reveal(self, reveal):\n        self.widget.rootObject().setProperty('reveal', reveal)\n\n    @property\n    def crteffect(self):\n        return self.widget.rootObject().property('crteffect')\n\n    @crteffect.setter\n    def crteffect(self, crteffect):\n        self.widget.rootObject().setProperty('crteffect', crteffect)\n\n    def size(self):\n        sf = self.widget.rootObject().size()\n        return QSize(int(sf.width()), int(sf.height()))\n\n    def setEffect(self, e):\n        self._effect = bool(e)\n        self.widget.rootContext().setContextProperty('tteffect', self._effect)\n"
  },
  {
    "path": "teletext/gui/decoder.qml",
    "content": "import QtQuick 2.12\nimport QtQuick.Controls 2.12\nimport QtGraphicalEffects 1.12\n\n\nRectangle {\n    property int zoom: 2\n    property int borderSize: 10 * zoom\n    property bool crteffect: true\n    property bool flashsrc: true\n    property bool reveal: false\n    width: teletext.width + borderSize * 4\n    height: teletext.height + borderSize * 2\n    border.width: borderSize\n    border.color: \"black\"\n    color: \"black\"\n    Column {\n        id: teletext\n        objectName: \"teletext\"\n        width: 40 * 8 * zoom\n        height: 250 * zoom\n        x: borderSize * 2\n        y: borderSize\n        clip: true\n        Repeater {\n            objectName: \"rows\"\n            model: 25\n            Row {\n                property int rowheight: 1\n                property bool rowrendered: true\n                Repeater {\n                    objectName: \"cols\"\n                    model: 40\n                    Item {\n                        property string c: \"X\"\n                        property int bg: 1\n                        property int fg: 7\n                        property bool dw: false\n                        property bool dh: false\n                        property bool flash: false\n                        property bool mosaic: false\n                        property bool solid: true\n                        property bool boxed: false\n                        property bool conceal: false\n                        property bool rendered: true\n                        height: 10 * zoom\n                        width: 8 * zoom\n                        Rectangle {\n                            height: rowheight * 10 * zoom\n                            width: (dw?2:1) * 8 * zoom\n                            clip: true\n                            visible: rowrendered && rendered\n                            color: ttpalette[bg]\n                            Text {\n                                renderType: Text.NativeRendering\n                                anchors.top: parent.top\n                                anchors.horizontalCenter: parent.horizontalCenter\n                                color: ttpalette[fg]\n                                text: c\n                                font: ttfonts[(mosaic && solid && text[0] > \"\\ue000\")?1:0][dw?1:0][dh?1:0]\n                                visible: ((!flash) || flashsrc) && (conceal ? reveal : true)\n                            }\n                        }\n                    }\n                }\n            }\n        }\n        layer.enabled: crteffect && (zoom > 1)\n        layer.effect: ShaderEffect {\n            fragmentShader: \"\n                    uniform lowp sampler2D source;\n                    uniform lowp float qt_Opacity;\n                    varying highp vec2 qt_TexCoord0;\n                    varying lowp vec3 qt_FragCoord0;\n                    void main() {\n                        lowp vec4 tex = texture2D(source, qt_TexCoord0);\n                        int zoom = \" + zoom + \";\n                        int row = int(gl_FragCoord.y) % zoom;\n                        gl_FragColor = (0 < row && (row < 2 || row < (zoom-1))) ? tex : tex*0.6;\n                    }\n                \"\n        }\n    }\n    layer.enabled: crteffect && (zoom > 1)\n    layer.effect: GaussianBlur {\n        radius: 0.75 * zoom\n    }\n    SequentialAnimation on flashsrc {\n        loops: -1\n        running: true\n        PropertyAction { value: false }\n        PauseAnimation { duration: 333 }\n        PropertyAction { value: true }\n        PauseAnimation { duration: 1000 }\n    }\n}\n\n\n"
  },
  {
    "path": "teletext/gui/editor.py",
    "content": "import os\n\nfrom teletext.file import FileChunker\nfrom teletext.packet import Packet\n\n\ntry:\n    from PyQt5 import QtCore, QtGui, QtWidgets, QtQuickWidgets, uic\nexcept ImportError:\n    print('PyQt5 is not installed. Qt VBI Viewer not available.')\n\nfrom teletext.gui.decoder import Decoder\nfrom teletext.gui.service import ServiceModel, StdSubpage, ServiceModelLoader\n\n\nclass EditorWindow(QtWidgets.QMainWindow):\n    def __init__(self):\n        super(EditorWindow, self).__init__()\n        ui_file = os.path.join(os.path.dirname(__file__), 'editor.ui')\n        self.ui = uic.loadUi(ui_file)\n\n        self._tt = Decoder(self.ui.DecoderWidget)\n\n        self.ui.actionZoom_In.triggered.connect(lambda: setattr(self._tt, 'zoom', self._tt.zoom+1))\n        self.ui.actionZoom_Out.triggered.connect(lambda: setattr(self._tt, 'zoom', self._tt.zoom-1))\n\n        self.ui.action1x.triggered.connect(lambda: setattr(self._tt, 'zoom', 1))\n        self.ui.action2x.triggered.connect(lambda: setattr(self._tt, 'zoom', 2))\n        self.ui.action3x.triggered.connect(lambda: setattr(self._tt, 'zoom', 3))\n        self.ui.action4x.triggered.connect(lambda: setattr(self._tt, 'zoom', 4))\n\n        self.ui.actionImport_T42.triggered.connect(self.importt42)\n\n        self.ui.actionCRT_Effect.setProperty('checked', self._tt.crteffect)\n        self.ui.actionCRT_Effect.toggled.connect(lambda x: setattr(self._tt, 'crteffect', x))\n\n        self.ui.actionReveal.setProperty('checked', self._tt.reveal)\n        self.ui.actionReveal.toggled.connect(lambda x: setattr(self._tt, 'reveal', x))\n\n        self.ui.ServiceTree.doubleClicked.connect(self.showsubpage)\n        self.ui.ServiceTree.header().setSortIndicator(0, QtCore.Qt.AscendingOrder)\n\n        self.progress = QtWidgets.QProgressBar()\n        self.progress.setVisible(False)\n        self.progress.setFixedWidth(200)\n        self.ui.statusBar().addPermanentWidget(self.progress)\n\n\n        try:\n            self.importt42('/media/al/Teletext/test.t42')\n        except FileNotFoundError:\n            pass\n\n        self.ui.show()\n\n    def showsubpage(self, index):\n        item = self.ui.ServiceTree.model().itemFromIndex(index)\n        if isinstance(item, StdSubpage):\n            self._tt[1:] = item._subpage.displayable[:]\n            self._tt[0, :8] = 0x20\n            self._tt[0, 8:] = item._subpage.header.displayable[:]\n\n    def importt42(self, filename=None):\n        self.ui.actionImport_T42.setEnabled(False)\n\n        if not isinstance(filename, str):\n            filename = QtWidgets.QFileDialog.getOpenFileName(self, \"Open Teletext Page\", \"\", \"T42 Files (*.t42)\")[0]\n        if filename == '':\n            return\n\n        self.service_thread = ServiceModelLoader(filename)\n        self.service_thread.total.connect(self.progress.setMaximum)\n        self.service_thread.update.connect(self.progress.setValue)\n        self.service_thread.finished.connect(self.importt42done)\n\n        self.ui.statusBar().addPermanentWidget(self.progress)\n        self.service_thread.start()\n        self.progress.setVisible(True)\n\n    def importt42done(self):\n        model = self.service_thread.model\n        del self.service_thread\n        self.ui.ServiceTree.setModel(model)\n        i = model.invisibleRootItem().child(0).child(0).child(0).index()\n        self.ui.ServiceTree.scrollTo(i)\n        self.showsubpage(i)\n        self.progress.reset()\n        self.progress.setVisible(False)\n        self.ui.actionImport_T42.setEnabled(True)\n\n\ndef main():\n    import sys\n    app = QtWidgets.QApplication(sys.argv)\n    window = EditorWindow()\n    app.exec_()\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "teletext/gui/editor.ui",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ui version=\"4.0\">\n <class>MainWindow</class>\n <widget class=\"QMainWindow\" name=\"MainWindow\">\n  <property name=\"geometry\">\n   <rect>\n    <x>0</x>\n    <y>0</y>\n    <width>1056</width>\n    <height>654</height>\n   </rect>\n  </property>\n  <property name=\"windowTitle\">\n   <string>Teletext Editor</string>\n  </property>\n  <property name=\"documentMode\">\n   <bool>false</bool>\n  </property>\n  <widget class=\"QWidget\" name=\"centralwidget\">\n   <layout class=\"QHBoxLayout\" name=\"horizontalLayout_2\">\n    <property name=\"spacing\">\n     <number>0</number>\n    </property>\n    <property name=\"sizeConstraint\">\n     <enum>QLayout::SetDefaultConstraint</enum>\n    </property>\n    <property name=\"leftMargin\">\n     <number>1</number>\n    </property>\n    <property name=\"topMargin\">\n     <number>1</number>\n    </property>\n    <property name=\"rightMargin\">\n     <number>1</number>\n    </property>\n    <property name=\"bottomMargin\">\n     <number>1</number>\n    </property>\n    <item>\n     <widget class=\"QScrollArea\" name=\"scrollArea\">\n      <property name=\"frameShape\">\n       <enum>QFrame::Panel</enum>\n      </property>\n      <property name=\"lineWidth\">\n       <number>1</number>\n      </property>\n      <property name=\"widgetResizable\">\n       <bool>true</bool>\n      </property>\n      <property name=\"alignment\">\n       <set>Qt::AlignCenter</set>\n      </property>\n      <widget class=\"QWidget\" name=\"scrollAreaWidgetContents\">\n       <property name=\"geometry\">\n        <rect>\n         <x>0</x>\n         <y>0</y>\n         <width>855</width>\n         <height>608</height>\n        </rect>\n       </property>\n       <property name=\"sizePolicy\">\n        <sizepolicy hsizetype=\"Preferred\" vsizetype=\"Preferred\">\n         <horstretch>0</horstretch>\n         <verstretch>0</verstretch>\n        </sizepolicy>\n       </property>\n       <property name=\"styleSheet\">\n        <string notr=\"true\">background-color: rgb(0, 0, 0);</string>\n       </property>\n       <layout class=\"QHBoxLayout\" name=\"horizontalLayout_3\">\n        <property name=\"spacing\">\n         <number>0</number>\n        </property>\n        <property name=\"leftMargin\">\n         <number>0</number>\n        </property>\n        <property name=\"topMargin\">\n         <number>0</number>\n        </property>\n        <property name=\"rightMargin\">\n         <number>0</number>\n        </property>\n        <property name=\"bottomMargin\">\n         <number>0</number>\n        </property>\n        <item>\n         <widget class=\"QQuickWidget\" name=\"DecoderWidget\">\n          <property name=\"resizeMode\">\n           <enum>QQuickWidget::SizeViewToRootObject</enum>\n          </property>\n         </widget>\n        </item>\n       </layout>\n      </widget>\n     </widget>\n    </item>\n   </layout>\n  </widget>\n  <widget class=\"QMenuBar\" name=\"menubar\">\n   <property name=\"geometry\">\n    <rect>\n     <x>0</x>\n     <y>0</y>\n     <width>1056</width>\n     <height>20</height>\n    </rect>\n   </property>\n   <widget class=\"QMenu\" name=\"menuFile\">\n    <property name=\"title\">\n     <string>File</string>\n    </property>\n    <addaction name=\"actionImport_T42\"/>\n   </widget>\n   <widget class=\"QMenu\" name=\"menuView\">\n    <property name=\"title\">\n     <string>View</string>\n    </property>\n    <widget class=\"QMenu\" name=\"menuZoom\">\n     <property name=\"title\">\n      <string>Zoom</string>\n     </property>\n     <addaction name=\"actionZoom_In\"/>\n     <addaction name=\"actionZoom_Out\"/>\n     <addaction name=\"separator\"/>\n     <addaction name=\"action1x\"/>\n     <addaction name=\"action2x\"/>\n     <addaction name=\"action3x\"/>\n     <addaction name=\"action4x\"/>\n    </widget>\n    <addaction name=\"menuZoom\"/>\n    <addaction name=\"actionCRT_Effect\"/>\n    <addaction name=\"actionReveal\"/>\n   </widget>\n   <addaction name=\"menuFile\"/>\n   <addaction name=\"menuView\"/>\n  </widget>\n  <widget class=\"QDockWidget\" name=\"ServiceDock\">\n   <property name=\"windowTitle\">\n    <string>Service</string>\n   </property>\n   <attribute name=\"dockWidgetArea\">\n    <number>1</number>\n   </attribute>\n   <widget class=\"QWidget\" name=\"ServiceDockContents\">\n    <layout class=\"QVBoxLayout\" name=\"verticalLayout_3\">\n     <property name=\"spacing\">\n      <number>0</number>\n     </property>\n     <property name=\"leftMargin\">\n      <number>2</number>\n     </property>\n     <property name=\"topMargin\">\n      <number>2</number>\n     </property>\n     <property name=\"rightMargin\">\n      <number>2</number>\n     </property>\n     <property name=\"bottomMargin\">\n      <number>2</number>\n     </property>\n     <item>\n      <widget class=\"QTreeView\" name=\"ServiceTree\">\n       <property name=\"frameShape\">\n        <enum>QFrame::Panel</enum>\n       </property>\n       <property name=\"editTriggers\">\n        <set>QAbstractItemView::NoEditTriggers</set>\n       </property>\n       <property name=\"dragEnabled\">\n        <bool>false</bool>\n       </property>\n       <property name=\"dragDropMode\">\n        <enum>QAbstractItemView::NoDragDrop</enum>\n       </property>\n       <property name=\"defaultDropAction\">\n        <enum>Qt::MoveAction</enum>\n       </property>\n       <property name=\"selectionMode\">\n        <enum>QAbstractItemView::ExtendedSelection</enum>\n       </property>\n       <property name=\"selectionBehavior\">\n        <enum>QAbstractItemView::SelectItems</enum>\n       </property>\n       <property name=\"rootIsDecorated\">\n        <bool>true</bool>\n       </property>\n       <property name=\"sortingEnabled\">\n        <bool>true</bool>\n       </property>\n       <attribute name=\"headerVisible\">\n        <bool>false</bool>\n       </attribute>\n      </widget>\n     </item>\n    </layout>\n   </widget>\n  </widget>\n  <widget class=\"QStatusBar\" name=\"statusbar\"/>\n  <action name=\"actionImport_T42\">\n   <property name=\"text\">\n    <string>Import T42</string>\n   </property>\n   <property name=\"toolTip\">\n    <string>Import</string>\n   </property>\n  </action>\n  <action name=\"actionZoom_In\">\n   <property name=\"text\">\n    <string>Zoom In</string>\n   </property>\n   <property name=\"shortcut\">\n    <string>Ctrl++</string>\n   </property>\n  </action>\n  <action name=\"actionZoom_Out\">\n   <property name=\"text\">\n    <string>Zoom Out</string>\n   </property>\n   <property name=\"shortcut\">\n    <string>Ctrl+-</string>\n   </property>\n  </action>\n  <action name=\"action1x\">\n   <property name=\"text\">\n    <string>1x</string>\n   </property>\n  </action>\n  <action name=\"action2x\">\n   <property name=\"text\">\n    <string>2x</string>\n   </property>\n  </action>\n  <action name=\"action3x\">\n   <property name=\"text\">\n    <string>3x</string>\n   </property>\n  </action>\n  <action name=\"action4x\">\n   <property name=\"text\">\n    <string>4x</string>\n   </property>\n  </action>\n  <action name=\"actionCRT_Effect\">\n   <property name=\"checkable\">\n    <bool>true</bool>\n   </property>\n   <property name=\"text\">\n    <string>CRT Effect</string>\n   </property>\n  </action>\n  <action name=\"actionReveal\">\n   <property name=\"checkable\">\n    <bool>true</bool>\n   </property>\n   <property name=\"text\">\n    <string>Reveal</string>\n   </property>\n  </action>\n </widget>\n <customwidgets>\n  <customwidget>\n   <class>QQuickWidget</class>\n   <extends>QWidget</extends>\n   <header location=\"global\">QtQuickWidgets/QQuickWidget</header>\n  </customwidget>\n </customwidgets>\n <resources/>\n <connections/>\n</ui>\n"
  },
  {
    "path": "teletext/gui/qthelpers.py",
    "content": "from PyQt5 import QtWidgets\n\n\ndef build_menu(window, parent_menu, menu_defs):\n    for name, action, shortcut in menu_defs:\n        if name is None:\n            parent_menu.addSeparator()\n        elif isinstance(action, list):\n            m = parent_menu.addMenu(name)\n            build_menu(window, m, action)\n        else:\n            a = QtWidgets.QAction(name, window)\n            if shortcut:\n                a.setShortcut(shortcut)\n            if callable(action):\n                a.triggered.connect(action)\n            else:\n                print(f'Warning: menu item {name}: {action} is not callable.')\n            parent_menu.addAction((a))\n"
  },
  {
    "path": "teletext/gui/service.py",
    "content": "from PyQt5 import QtCore, QtGui, QtWidgets, QtQuickWidgets\nfrom PyQt5.QtCore import QVariant\n\nfrom teletext.file import FileChunker\nfrom teletext.packet import Packet\nfrom teletext.service import Service\n\n\nclass StdSubpage(QtGui.QStandardItem):\n    def __init__(self, subpage, number):\n        self._subpage = subpage\n        self._number = number\n        super().__init__(f'Subpage {self._subpage.addr}')\n        for s in self._subpage.duplicates:\n            self.appendRow(StdSubpage(s, self._number))\n\n\n\nclass StdPage(QtGui.QStandardItem):\n    def __init__(self, page, number):\n        self._page = page\n        self._number = number\n        super().__init__(f'Page {self._number:02X}')\n        for n, s in sorted(self._page.subpages.items()):\n            self.appendRow(StdSubpage(s, n))\n\n\nclass StdMagazine(QtGui.QStandardItem):\n    def __init__(self, magazine, number):\n        self._magazine = magazine\n        self._number = number\n        super().__init__(f'Magazine {self._number}')\n        self.setDragEnabled(False)\n        for n, p in sorted(self._magazine.pages.items()):\n            self.appendRow(StdPage(p, (0x100*self._number)+n))\n\n\nclass ServiceModel(QtGui.QStandardItemModel):\n    def __init__(self, service = None):\n        super().__init__()\n        self._service = service or Service()\n        for n, m in sorted(self._service.magazines.items()):\n            self.invisibleRootItem().appendRow(StdMagazine(m, n))\n\n\nclass ServiceModelLoader(QtCore.QThread):\n    total = QtCore.pyqtSignal(int)\n    update = QtCore.pyqtSignal(int)\n\n    def __init__(self, filename):\n        self._filename = filename\n        super().__init__()\n\n    def progress(self, chunks):\n        for n, d in chunks:\n            if n&0xfff == 0:\n                self.update.emit(n)\n            yield n, d\n\n    def run(self):\n        with open(self._filename, 'rb') as f:\n            chunks = FileChunker(f, 42)\n            self.total.emit(len(chunks))\n            packets = (Packet(data, number) for number, data in self.progress(chunks))\n            service = Service.from_packets(packets)\n            self.model = ServiceModel(service)\n"
  },
  {
    "path": "teletext/gui/vbiplot.py",
    "content": "import sys\n\nimport numpy as np\nfrom PyQt5 import QtWidgets\n\nfrom PyQt5.QtWidgets import QApplication, QPushButton, QMainWindow, QVBoxLayout\n\nimport matplotlib.pyplot as plt\nfrom matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas\nfrom matplotlib.backends.backend_qt5agg import NavigationToolbar2QT as NavigationToolbar\nfrom matplotlib.figure import Figure\n\nfrom itertools import islice\n\nfrom scipy import signal\n\nfrom teletext.vbi.line import Line\n\n\nclass Window(QMainWindow):\n    def __init__(self, chunker, config, parent=None):\n        super(Window, self).__init__(parent)\n\n        self.chunker = chunker\n        self.config = config\n\n        self.chunks = self.chunker(self.config.line_length * np.dtype(self.config.dtype).itemsize, self.config.field_lines, self.config.field_range)\n\n        self.canvas = FigureCanvas()\n\n        self.button = QPushButton('Next')\n        self.button.clicked.connect(self.plot)\n\n        self.n_lines = 16\n\n        w = QtWidgets.QWidget(self)\n        self.setCentralWidget(w)\n        layout = QVBoxLayout()\n        #layout.addWidget(self.toolbar)\n        layout.addWidget(self.canvas)\n        layout.addWidget(self.button)\n        w.setLayout(layout)\n        self.show()\n\n        self.plot()\n\n    def plot(self):\n        fig = Figure()\n        axs = fig.subplots(self.n_lines, 3, sharex='col', sharey='col')\n        for n, (o, d) in enumerate(islice(self.chunks, self.n_lines)):\n            ax = axs[n][0]\n            ax.set_ylabel(str(o))\n\n            line = Line(d)\n\n            xaxis_scaled = np.arange(self.config.line_length) * 8 * self.config.teletext_bitrate / self.config.sample_rate\n            xaxis = np.arange(len(line.resampled))\n\n            ax.plot(xaxis_scaled, line.original, color='green' if line.is_teletext else 'red', linewidth=0.5)\n\n            if line.start is not None:\n                ax.plot(line.start, line.resampled[line.start], 'x')\n                ax.plot(line.start + 128 + 12, line.resampled[line.start + 128 + 12], 'x')\n\n            ax = axs[n][1]\n            widths = np.array([8, 12, 16, 20, 24, 28, 32])\n            cwtmatr = signal.cwt(line.resampled, signal.morlet2, widths)\n            ll = np.sum(np.abs(cwtmatr), axis=0)\n\n            #ax.plot(ll)\n\n            ax.pcolormesh(np.abs(cwtmatr), cmap='viridis', shading='gouraud')\n            #ax.imshow(cwtmatr, extent=[-1, 1, 31, 1], cmap='PRGn', aspect='auto',\n            #           vmax=abs(cwtmatr).max(), vmin=-abs(cwtmatr).max())\n\n\n            ax = axs[n][2]\n\n            h = np.histogram(ll, np.arange(0, np.max(ll), 1))\n            c = np.cumsum(h[0])/len(ll)\n            t = np.array([np.argmax(c>m/10) for m in range(1,10)])\n            print(t)\n            l = 'green'\n            if t[-1] - t[0] < 200: # quiet line?\n                l = 'red'\n            elif t[1] < 75 and t[2] > 200:  # not teletext?\n                l = 'red'\n\n            ax.plot(h[1][:-1], c, color=l, linewidth=0.5)\n            ax.plot(t, c[t], 'x', color=l)\n\n\n        axs[-1][0].set_xlabel(f'samples, resampled to 8x {self.config.teletext_bitrate} Hz')\n        self.canvas.figure = fig\n        x, y = self.width(), self.height()\n        self.resize(x+1, y+1)\n        self.resize(x, y)\n\n        # refresh canvas\n        self.canvas.draw()\n\n        # close the figure so that we don't create too many figure instances\n        plt.close(fig)\n\n\ndef vbiplot(chunker, config):\n    # To prevent random crashes when rerunning the code,\n    # first check if there is instance of the app before creating another.\n    app = QApplication.instance()\n    if app is None:\n        app = QApplication(sys.argv)\n\n    main = Window(chunker, config)\n    main.show()\n    app.exec_()\n"
  },
  {
    "path": "teletext/image.py",
    "content": "import math\n\nimport numpy as np\nfrom teletext.parser import Parser\n\nfrom PIL import Image\nfrom PIL.PcfFontFile import *\n\n\nclass PcfFontFileUnicode(PcfFontFile):\n    def unicode_glyphs(self):\n        metrics = self._load_metrics()\n        bitmaps = self._load_bitmaps(metrics)\n\n        # map character code to bitmap index\n        encoding = {}\n\n        fp, format, i16, i32 = self._getformat(PCF_BDF_ENCODINGS)\n\n        firstCol, lastCol = i16(fp.read(2)), i16(fp.read(2))\n        firstRow, lastRow = i16(fp.read(2)), i16(fp.read(2))\n\n        i16(fp.read(2))  # default\n\n        nencoding = (lastCol - firstCol + 1) * (lastRow - firstRow + 1)\n\n        for i in range(nencoding):\n            encodingOffset = i16(fp.read(2))\n            if encodingOffset != 0xFFFF:\n                encoding[i + firstCol] = encodingOffset\n\n        glyphs = {}\n\n        for ch, ix in encoding.items():\n            if ix is not None:\n                x, y, l, r, w, a, d, f = metrics[ix]\n                glyph = (w, 0), (l, d - y, x + l, d), (0, 0, x, y), bitmaps[ix]\n                glyphs[ch] = glyph[3]\n\n        return glyphs\n\n\ndef load_glyphs(fp):\n    f = PcfFontFileUnicode(fp)\n    return f.unicode_glyphs()\n\n\nclass PrinterImage(Parser):\n\n    def __init__(self, tt, glyphs, box=False, flash_off=False, colour=True, codepage=0):\n        self.colour = colour\n        self.column = 0\n        self.glyphs = glyphs\n        self.image = Image.new(\"P\", (12*len(tt), 20), color=8)\n        self.missing = set()\n        self.flash_off = flash_off\n        self.flash_used = False\n        self.box = box\n        super().__init__(tt)\n\n    def emitcharacter(self, c):\n        if self._state['boxed'] or not self.box:\n            if self._state['conceal'] or (self.flash_off and self._state['flash']):\n                c = ' '\n            try:\n                glyph = self.glyphs[ord(c)]\n            except KeyError:\n                self.missing.add(c)\n            else:\n                data = np.choose(glyph, (self._state['bg'], self._state['fg']))\n                i = Image.fromarray(data.astype(np.uint8), \"P\")\n                i = i.resize((\n                    i.width * (2 if self._state['dw'] else 1),\n                    i.height * (2 if self._state['dh'] else 1),\n                ))\n                self.image.paste(i, (self.column*12, 0))\n        self.column += 1\n\n    def parse(self):\n        super().parse()\n        return self.missing\n\n\ndef subpage_to_image(s, glyphs, background=None, flash_off=False):\n    img = Image.new(\"P\", (12*40, 25*10), color=8)\n    missing = set()\n    img.putpalette([\n        0, 0, 0, 255,\n        255, 0, 0, 255,\n        0, 255, 0, 255,\n        255, 255, 0, 255,\n        0, 0, 255, 255,\n        255, 0, 255, 255,\n        0, 255, 255, 255,\n        255, 255, 255, 255,\n        0, 0, 0, 0,\n    ], \"RGBA\")\n\n    box = s.header.newsflash or s.header.subtitle\n    flash_used = False\n\n    if not s.header.supress_header:\n        prnt = PrinterImage(s.header.displayable._array, glyphs, flash_off=flash_off)\n        missing.update(prnt.parse())\n        img.paste(prnt.image, (12*8, 0))\n        if np.any(s.header.displayable == 0x08):\n            flash_used = True\n\n    for i in range(0, 24):\n        # only draw the line if previous line does not contain double height code\n        if i == 0 or np.all(s.displayable[i - 1, :] != 0x0d):\n            prnt = PrinterImage(s.displayable[i, :], glyphs, flash_off=flash_off, box=box)\n            missing.update(prnt.parse())\n            img.paste(prnt.image, (0, (i+1)*10))\n        if np.any(s.displayable[i - 1, :] == 0x08):\n            flash_used = True\n\n    img = img.convert(\"RGBA\").resize((img.width, img.height*2))\n\n    if background is None:\n        background = Image.new(\"RGBA\", (720, 576), color=(0, 0, 0, 0 if box else 255))\n    else:\n        background = background.convert(\"RGBA\").resize((720, 576))\n\n    background.alpha_composite(img, (\n        (background.width - img.width) // 2,\n        (background.height - img.height) // 2,\n    ))\n\n    background._missing_glyphs = missing\n    background._flash_used = flash_used\n    return background\n"
  },
  {
    "path": "teletext/interactive.py",
    "content": "import curses\nimport os\nimport time\nimport locale\n\nfrom .file import FileChunker\nfrom .packet import Packet\nfrom .parser import Parser\nfrom .printer import PrinterANSI\n\n\ndef setstyle(self, fg=None, bg=None):\n    return '\\033[' + chr(fg or self.fg) + chr(bg or self.bg) + chr(1 if self.flash else 0 + 2 if self.conceal else 0)\n\n\nPrinterANSI.setstyle = setstyle\n\n\nclass TerminalTooSmall(Exception):\n    pass\n\n\nclass ParserNcurses(Parser):\n    def __init__(self, tt, scr, row):\n        self._scr = scr\n        self._row = row\n        super().__init__(tt)\n\n    def emitcharacter(self, c):\n        colour = Interactive.colours[self._state['fg']] | Interactive.colours[self._state['bg']] << 3\n        if self._state['conceal']:\n            colour += 64\n        self._scr.addstr(self._row, self._pos, c, curses.color_pair(colour+1) | (curses.A_BLINK if self._state['flash'] else 0))\n        self._pos += 1\n\n    def parse(self):\n        self._pos = 0 if self._row else 8\n        super().parse()\n\n\nclass Interactive(object):\n    colours = {0: curses.COLOR_BLACK, 1: curses.COLOR_RED, 2: curses.COLOR_GREEN, 3: curses.COLOR_YELLOW,\n               4: curses.COLOR_BLUE, 5: curses.COLOR_MAGENTA, 6: curses.COLOR_CYAN, 7: curses.COLOR_WHITE}\n\n    def __init__(self, packet_iter, scr, initial_page=0x100):\n        self.scr = scr\n\n        self.packet_iter = packet_iter\n\n        if initial_page is None:\n            self.magazine = 1\n            self.page = 0\n        else:\n            self.magazine = initial_page >> 8\n            self.page = initial_page & 0xff\n        self.last_subpage = None\n        self.last_header = None\n        self.inputtmp = [None, None, None]\n        self.inputstate = 0\n        self.need_clear = False\n        self.hold = False\n        self.reveal = False\n        self.links = [(7, 255), (7, 255), (7, 255), (7, 255)]\n\n        y, x = self.scr.getmaxyx()\n        if x < 41 or y < 25:\n            raise TerminalTooSmall(x, y)\n\n        curses.start_color()\n        for n in range(64):\n            curses.init_pair(n + 1, Interactive.colours[n & 0x7], Interactive.colours[n >> 3])\n        self.set_concealed_pairs()\n\n        self.scr.nodelay(1)\n        curses.curs_set(0)\n\n        self.set_input_field('P%d%02x' % (self.magazine, self.page))\n\n    def set_concealed_pairs(self, show=False):\n        for n in range(16):\n            # workaround for ncurses bug where pairs are only refreshed if their previous\n            # fg and bg are both not black. one of the temp colours here must be not black\n            # and one must be different to the actual desired colours.\n            curses.init_pair(n + 1 + 64, 0, 1)\n        for n in range(64):\n            curses.init_pair(n + 1 + 64, Interactive.colours[n & 0x7] if show else Interactive.colours[n >> 3],\n                             Interactive.colours[n >> 3])\n\n    def set_input_field(self, str, clr=0):\n        self.scr.addstr(0, 3, str, curses.color_pair(clr))\n\n    def go_page(self, magazine, page):\n        self.inputstate = 0\n        self.magazine = magazine\n        self.page = page\n        self.set_input_field('P%d%02x' % (self.magazine, self.page))\n\n    def addstr(self, packet):\n        r = packet.mrag.row\n        if r:\n            ParserNcurses(packet.displayable[:], self.scr, r)\n        else:\n            ParserNcurses(packet.header.displayable[:], self.scr, r)\n\n    def do_alnum(self, i):\n        if self.inputstate == 0:\n            if i >= 1 and i <= 8:\n                self.inputtmp[0] = i\n                self.inputstate = 1\n        else:\n            self.inputtmp[self.inputstate] = i\n            self.inputstate += 1\n\n        if self.inputstate != 0:\n            self.set_input_field(\n                'P' + ''.join([('%1X' % self.inputtmp[x]) if self.inputtmp[x] is not None else '.' for x in range(3)]),\n                3 if self.inputstate < 3 else 0)\n\n        if self.inputstate == 3:\n            self.inputstate = 0\n            self.magazine = self.inputtmp[0]\n            self.page = (self.inputtmp[1] << 4) | self.inputtmp[2]\n            self.inputtmp = [None, None, None]\n            self.last_header = None\n            self.need_clear = True\n\n    def do_hold(self):\n        self.hold = not self.hold\n        if self.hold:\n            self.set_input_field('HOLD', 2)\n            self.inputstate = 0\n            self.inputtmp[0] = None\n            self.inputtmp[1] = None\n            self.inputtmp[2] = None\n        else:\n            self.set_input_field('P%d%02x' % (self.magazine, self.page))\n            self.need_clear = True\n\n    def do_reveal(self):\n        self.reveal = not self.reveal\n        self.set_concealed_pairs(self.reveal)\n\n    def do_input(self, c):\n        if c >= ord('0') and c <= ord('9'):\n            if self.hold:\n                self.do_hold()\n            self.do_alnum(c - ord('0'))\n        elif c >= ord('a') and c <= ord('f'):\n            if self.hold:\n                self.do_hold()\n            self.do_alnum(c + 10 - ord('a'))\n        elif c == ord('.'):\n            self.do_hold()\n        elif c == ord('r'):\n            self.do_reveal()\n        elif c == ord('h'):\n            self.go_page(self.links[0].magazine, self.links[0].page)\n        elif c == ord('j'):\n            self.go_page(self.links[1].magazine, self.links[1].page)\n        elif c == ord('k'):\n            self.go_page(self.links[2].magazine, self.links[2].page)\n        elif c == ord('l'):\n            self.go_page(self.links[3].magazine, self.links[3].page)\n        elif c == ord('q'):\n            self.running = False\n\n    def handle_one_packet(self):\n\n        packet = next(self.packet_iter)\n        if self.inputstate == 0 and not self.hold:\n            if packet.mrag.magazine == self.magazine:\n                if packet.mrag.row == 0:\n                    if packet.header.page == self.page:\n                        if self.need_clear or packet.header.control & 0x1:\n                            self.scr.erase()\n                            self.need_clear = False\n                    self.last_header = packet.header.page\n                    self.addstr(packet)\n                    self.set_input_field('P%d%02X' % (self.magazine, self.page))\n                elif self.last_header == self.page:\n                    if packet.mrag.row < 25:\n                        self.addstr(packet)\n                    elif packet.mrag.row == 27:\n                        self.links = packet.fastext.links\n                        #print(self.links)\n\n    def main(self):\n        self.running = True\n        while self.running:\n            for i in range(32):\n                self.handle_one_packet()\n\n            self.do_input(self.scr.getch())\n\n            self.scr.refresh()\n            time.sleep(0.01)\n\n\ndef main(packets, initial_page):\n    locale.setlocale(locale.LC_ALL, '')\n\n    if os.name == 'nt':\n        f = open(\"CON:\", 'r')\n    else:\n        f = open(\"/dev/tty\", 'r')\n    os.dup2(f.fileno(), 0)\n\n    def main(scr):\n        Interactive(packets, scr, initial_page=initial_page).main()\n\n    try:\n        curses.wrapper(main)\n    except TerminalTooSmall as e:\n        print(f'Your terminal is too small.\\nPlease make it at least 41x25.\\nCurrent size: {e.args[0]}x{e.args[1]}.')\n        exit(-1)\n"
  },
  {
    "path": "teletext/mp.py",
    "content": "import atexit\nimport itertools\nimport pickle\nimport queue\n\nimport multiprocessing as mp\n\nimport zmq\n\n\ndef denumerate(work, control, tmp_queue):\n\n    \"\"\"Strips sequence numbers from work_queue items and yields the work.\"\"\"\n\n    poller = zmq.Poller()\n    poller.register(work, zmq.POLLIN)\n    poller.register(control, zmq.POLLIN)\n\n    while True:\n        socks = dict(poller.poll())\n        if socks.get(work) == zmq.POLLIN:\n            n, item = work.recv_pyobj()\n            tmp_queue.put((n, len(item)))\n            yield from item\n        if socks.get(control) == zmq.POLLIN:\n            return\n\ndef renumerate(iterator, result, tmp_queue):\n\n    \"\"\"Recombines results with the sequence numbers stored in tmp_queue.\"\"\"\n\n    try:\n        while True:\n            r = [next(iterator)]\n            n, l = tmp_queue.get()\n            while len(r) < l:\n                r.append(next(iterator))\n            result.send_pyobj((n, r))\n    except StopIteration:\n        pass\n\n\ndef worker(work_port, result_port, control_port, status_port, function, args, kwargs):\n\n    \"\"\"Subprocess main. Runs a generator function on items from a pipe.\"\"\"\n\n    tmp_queue = queue.Queue()\n\n    ctx = zmq.Context()\n    work = ctx.socket(zmq.PULL)\n    work.set_hwm(10)\n    result = ctx.socket(zmq.PUSH)\n    status = ctx.socket(zmq.PUSH)\n    control = ctx.socket(zmq.SUB)\n\n    try:\n        work.connect(f'tcp://localhost:{work_port}')\n        result.connect(f'tcp://localhost:{result_port}')\n        status.connect(f\"tcp://localhost:{status_port}\")\n        control.connect(f\"tcp://localhost:{control_port}\")\n        control.setsockopt(zmq.SUBSCRIBE, b\"\")\n        status.send_string('CON')\n\n        renumerate(function(denumerate(work, control, tmp_queue), *args, **kwargs), result, tmp_queue)\n    except KeyboardInterrupt:\n        pass\n    finally:\n        status.send_string('DED')\n\n\nclass _PureGeneratorPoolMP(object):\n\n    def __init__(self, function, processes=1, *args, **kwargs):\n        self._processes = processes\n        self._function = function\n        self._args = args\n        self._kwargs = kwargs\n        self._procs = []\n\n        # Similar to how, on Linux, putting an unpickleable object on a Queue\n        # causes an uncatchable exception, passing unpickleable objects to\n        # ctx.Process does the same thing on Windows. So we must check that\n        # everything can be pickled before attempting to use it. Luckily this\n        # is only done once.\n        pickle.dumps(self._function)\n        pickle.dumps(self._args)\n        pickle.dumps(self._kwargs)\n\n    def __enter__(self):\n        mp_ctx = mp.get_context('spawn')\n\n        self._ctx = zmq.Context()\n\n        self._work = self._ctx.socket(zmq.PUSH)\n        work_port = self._work.bind_to_random_port('tcp://*')\n\n        self._result = self._ctx.socket(zmq.PULL)\n        result_port = self._result.bind_to_random_port('tcp://*')\n\n        self._status = self._ctx.socket(zmq.PULL)\n        status_port = self._status.bind_to_random_port('tcp://*')\n\n        self._control = self._ctx.socket(zmq.PUB)\n        control_port = self._control.bind_to_random_port('tcp://*')\n\n        try:\n\n            for id in range(self._processes):\n                p = mp_ctx.Process(target=worker, args=(\n                    work_port, result_port, control_port, status_port,\n                    self._function, self._args, self._kwargs\n                ))\n                self._procs.append(p)\n\n            for p in self._procs:\n                p.start()\n\n            atexit.register(self.__exit__)\n\n            for p in self._procs:\n                s = self._status.recv_string()\n                if s == 'DED':\n                    raise ChildProcessError(\"Worker failed to start.\")\n\n        except (KeyboardInterrupt, ChildProcessError):\n            self._control.send_string(\"DIE\")\n            raise\n\n        return self\n\n    def apply(self, iterable):\n        try:\n            chunksize = min(64, 1+(len(iterable)//len(self._procs)))\n        except TypeError:\n            chunksize = 64\n\n        it = iter(iterable)\n        iterable = enumerate(iter(lambda: list(itertools.islice(it, chunksize)), []))\n        received = {}\n        sent_count = 0\n        received_count = 0\n        done = False\n\n        poller = zmq.Poller()\n        poller.register(self._work, zmq.POLLOUT)\n        poller.register(self._status, zmq.POLLIN)\n        poller.register(self._result, zmq.POLLIN)\n\n        while True:\n            socks = dict(poller.poll())\n\n            if socks.get(self._status) == zmq.POLLIN:\n                raise ChildProcessError('Worker exited unexpectedly.')\n\n            if socks.get(self._result) == zmq.POLLIN:\n                n, item = self._result.recv_pyobj()\n                received[n] = item\n\n                while received_count in received:\n                    yield from received[received_count]\n                    del received[received_count]\n                    received_count += 1\n                    if not done and sent_count - received_count < self._processes * 3:\n                        poller.register(self._work, zmq.POLLOUT)\n\n                if done and sent_count == received_count:\n                    return\n\n            if socks.get(self._work) == zmq.POLLOUT:\n                try:\n                    self._work.send_pyobj(next(iterable))\n                    sent_count += 1\n                    if sent_count - received_count > self._processes * 4:\n                        poller.unregister(self._work)\n                except StopIteration:\n                    done = True\n                    if sent_count == 0:\n                        # we didn't send any work, so we will wait forever unless we exit now\n                        return\n\n    def __exit__(self, *args):\n        self._control.send_string(\"DIE\")\n        for proc in self._procs:\n            proc.join()\n        atexit.unregister(self.__exit__)\n\n\nclass _PureGeneratorPoolSingle(object):\n\n    \"\"\"\n    An implementation of PureGeneratorPool that doesn't use multiple processes.\n    \"\"\"\n\n    def __init__(self, function, *args, **kwargs):\n        self._function = function\n        self._args = args\n        self._kwargs = kwargs\n        self._work_queue = queue.Queue()\n        self._proc = self._function(self._work, *args, **kwargs)\n\n    @property\n    def _work(self):\n        while True:\n            try:\n                yield self._work_queue.get(block=False)\n            except queue.Empty:\n                return\n\n    def __enter__(self):\n        return self\n\n    def apply(self, iterable):\n        for item in iterable:\n            self._work_queue.put(item)\n            yield next(self._proc)\n\n    def __exit__(self, *args):\n        try:\n            next(self._proc)\n        except StopIteration:\n            pass\n\n\ndef PureGeneratorPool(function, processes, *args, **kwargs):\n\n    \"\"\"\n    Implements a parallel processing pool similar to multiprocessing.Pool. However,\n    Pool.map(f, i) calls f on every item in i individually. f is expected to return\n    the result. PureGeneratorPool.apply(f, i) calls f exactly once for each process\n    it starts, and then delivers an iterator containing work items. f is expected\n    to yield results. In practice, this means you can pass large objects to f and\n    they will only be pickled once rather than for every item in i. It also allows\n    you to do one-time setup at the beginning of f.\n\n    f must be a \"pure generator\". This means it must yield exactly one result for\n    each item in the iterator, and that result must only depend on the current\n    item being processed. It must not have any mutable state which affects the\n    output. For example, any function of the form:\n\n        itertools.partial(map, f)\n\n    is a pure generator if f is pure.\n\n    And further:\n\n        def gen(g, f, it):\n            g()\n            yield from f(it)\n\n    is a pure generator if f is a pure generator, regardless of whether or not g\n    is pure.\n\n    apply() preserves the ordering of items in the input iterator.\n    \"\"\"\n\n    if processes > 1:\n        return _PureGeneratorPoolMP(function, processes, *args, **kwargs)\n    else:\n        return _PureGeneratorPoolSingle(function, *args, **kwargs)\n\n\ndef itermap(function, iterable, processes=1, *args, **kwargs):\n\n    \"\"\"One-shot function to make a PureGeneratorPool and apply it.\"\"\"\n\n    with PureGeneratorPool(function, processes, *args, **kwargs) as pool:\n        yield from pool.apply(iterable)\n\n\nif __name__ in ['__main__', '__mp_main__']:\n\n    def f(iterator, *args, **kwargs):\n        # f first creates an unpickable, unsharable object. It must be done\n        # exactly once per process.\n        print('This line MUST be printed exactly once by each process.', args, kwargs)\n        for item in iterator:\n            #time.sleep(1)\n            yield item\n\n\nif __name__ == '__main__':\n\n    import click\n    from tqdm import tqdm\n\n    @click.command()\n    @click.option('-j', '--jobs', type=int, default=1000000)\n    @click.option('-t', '--threads', type=int, default=2)\n    @click.option('-v', '--verbose', is_flag=True)\n    def main(jobs, threads, verbose):\n        for result in itermap(f, iter(tqdm(range(jobs))), processes=threads, a=2, b=3):\n            if(verbose):\n                print(result, end=' ')\n        print('')\n\n    main()\n"
  },
  {
    "path": "teletext/packet.py",
    "content": "from .elements import *\n\n\nclass Packet(Element):\n\n    def __init__(self, array=None, number=None, original=None):\n        super().__init__((42, ), array)\n        self._number = number\n        self._original = original\n\n    def __getitem__(self, item):\n        return self._array[item]\n\n    def __setitem__(self, item, value):\n        self._array[item] = value\n\n    def is_padding(self):\n        return not np.any(self._array)\n\n    @property\n    def number(self):\n        return self._number\n\n    @property\n    def type(self):\n        row = self.mrag.row\n        if row == 0:\n            return 'header'\n        elif row < 26:\n            return 'display'\n        elif row == 27:\n            return 'fastext'\n        elif row == 30 and self.mrag.magazine == 8:\n            return 'broadcast'\n        elif row in [26, 28]:\n            return 'page enhancement'\n        elif row == 29:\n            return 'magazine enhancement'\n        elif row in [30, 31] and self.mrag.magazine == 4:\n            return 'celp'\n        elif row in [30, 31]:\n            return 'independent data'\n        else:\n            return 'unknown'\n\n    @property\n    def mrag(self):\n        return Mrag(self._array[:2])\n\n    @property\n    def dc(self):\n        return DesignationCode((1,), self._array[2:3])\n\n    @property\n    def header(self):\n        return Header(self._array[2:])\n\n    @property\n    def displayable(self):\n        return Displayable((40,), self._array[2:])\n\n    @property\n    def fastext(self):\n        return Fastext(self._array[2:], self.mrag)\n\n    @property\n    def triplets(self):\n        return Triplets(self._array[2:], self.mrag)\n\n    @property\n    def broadcast(self):\n        return BroadcastData(self._array[2:], self.mrag)\n\n    @property\n    def celp(self):\n        return Celp(self._array[2:], self.mrag)\n\n    def to_ansi(self, colour=True):\n        t = self.type\n\n        if t == 'header':\n            return f'   P{self.mrag.magazine}{self.header.to_ansi(colour)}'\n        elif t == 'display':\n            return self.displayable.to_ansi(colour)\n        elif t == 'page enhancement':\n            return self.triplets.to_ansi(colour)\n        elif t == 'fastext':\n            return self.fastext.to_ansi(colour)\n        elif t == 'broadcast':\n            return self.broadcast.to_ansi(colour)\n        elif t == 'celp':\n            return self.celp.to_ansi(colour)\n        elif t.endswith('enhancement'):\n            return f'{t} DC={self.dc.dc}'\n        else:\n            return f'{t}'\n\n    def to_bytes_no_parity(self):\n        t = self.type\n\n        if t == 'header':\n            return self.header.displayable.bytes_no_parity\n        elif t == 'display':\n            return self.displayable.bytes_no_parity\n        elif t == 'broadcast':\n            return self.broadcast.displayable.bytes_no_parity\n        else:\n            return b''\n\n    def to_binary(self):\n        b = np.unpackbits(self._array[::-1])[::-1]\n        x = b[0::2] | (b[1::2]<<1)\n        return ''.join([' ', chr(0x258C), chr(0x2590), chr(0x2588)][n] for n in x)\n\n    def to_bytes(self):\n        return self._array.tobytes()\n\n    @property\n    def ansi(self):\n        return self.to_ansi(colour=True).encode('utf8') + b'\\n'\n\n    @property\n    def text(self):\n        return self.to_ansi(colour=False).encode('utf8') + b'\\n'\n\n    @property\n    def bar(self):\n        return self.to_binary().encode('utf8') + b'\\n'\n\n    @property\n    def hex(self):\n        return self._array.tobytes().hex(' ').encode('utf8') + b'\\n'\n\n    @property\n    def debug(self):\n        if self.number is None:\n            return f'None     {self.mrag.magazine} {self.mrag.row:2d} {self.to_ansi(colour=True)} errors: {np.sum(self.errors)}\\n'.encode('utf8')\n        else:\n            return f'{self.number:8d} {self.mrag.magazine} {self.mrag.row:2d} {self.to_ansi(colour=True)} errors: {np.sum(self.errors)}\\n'.encode('utf8')\n\n    @property\n    def vbi(self):\n        if self._original is None:\n            raise Exception('Original VBI data is not available. Probably we are not deconvolving.')\n        return self._original\n\n    @property\n    def errors(self):\n        e = np.zeros_like(self._array)\n        e[:2] = self.mrag.errors\n        t = self.type\n\n        if t == 'header':\n            e[2:] = self.header.errors\n        elif t == 'display':\n            e[2:] = self.displayable.errors\n        elif t == 'fastext':\n            e[2:] = self.fastext.errors\n        elif t == 'broadcast':\n            e[2:] = self.broadcast.errors\n        elif t == 'celp':\n            e[2:] = self.celp.errors\n\n        return e\n"
  },
  {
    "path": "teletext/parser.py",
    "content": "from . import charset\n\n_unicode13 = False\n\n\nclass Parser(object):\n\n    \"Abstract base class for parsers\"\n\n    def __init__(self, tt, localcodepage=None, codepage=0):\n        self.tt = tt\n        self._state = {}\n        self.codepage = codepage\n        self.localcodepage = localcodepage\n        self.parse()\n\n    def reset(self):\n        self._state['fg'] = 7\n        self._state['bg'] = 0\n        self._state['dw'] = False\n        self._state['dh'] = False\n        self._state['mosaic'] = False\n        self._state['solid'] = True\n        self._state['flash'] = False\n        self._state['conceal'] = False\n        self._state['boxed'] = False\n        self._state['rendered'] = True\n\n        self._heldmosaic = ' '\n        self._heldsolid = True\n        self._held = False\n        self._esc = False\n        #self._codepage = 0 # not implemented\n\n    def setstate(self, **kwargs):\n        any = False\n        for state, value in kwargs.items():\n            if value != self._state[state]:\n                self._state[state] = value\n                any = True\n                if state in ['dw', 'dh']:\n                    self._heldmosaic = ' '\n                getattr(self, state+'Changed', lambda: None)()\n        if any:\n            getattr(self, 'stateChanged', lambda: None)()\n\n    def ttchar(self, c):\n        if self._state['mosaic'] and c not in range(0x40, 0x60):\n            if _unicode13:\n                return charset.g1[c]\n            else:\n                return chr(int(c)+0xee00) if self._state['solid'] else chr(int(c)+0xede0)\n        else:\n            if not self.localcodepage:\n                return charset.g0[\"default\"][c]\n            else:\n                if not self._esc and self.codepage:\n                    return charset.g0[self.localcodepage][c]\n                else:\n                    return charset.g0[\"default\"][c]\n\n    def _emitcharacter(self, c):\n        getattr(self, 'emitcharacter', lambda x: None)(c)\n        if self._state['dw']:\n            self._state['rendered'] = not self._state['rendered']\n        else:\n            self._state['rendered'] = True\n\n    def emitcode(self):\n        if self._held:\n            tmp = self._state['solid']\n            self._state['solid'] = self._heldsolid\n            self._emitcharacter(self._heldmosaic)\n            self._state['solid'] = tmp\n        else:\n            self._emitcharacter(' ')\n\n    def setat(self, **kwargs):\n        self.setstate(**kwargs)\n        self.emitcode()\n\n    def setafter(self, **kwargs):\n        self.emitcode()\n        self.setstate(**kwargs)\n\n    def parsebyte(self, b, prev):\n        h, l = int(b&0xf0), int(b&0x0f)\n        if h == 0x0:\n            if l < 8:\n                self.setafter(fg=l, mosaic=False, conceal=False)\n                self._heldmosaic = ' '\n            elif l == 0x8: # flashing\n                self.setafter(flash=True)\n            elif l == 0x9: # steady\n                self.setat(flash=False)\n            elif l == 0xa:\n                if prev == 0xa: # end box - set at because we're triggering on the second one\n                    self.setat(boxed=False)\n                else:\n                    self.emitcode()\n            elif l == 0xb:\n                if prev == 0xb: # start box - set at because we're triggering on the second one\n                    self.setat(boxed=True)\n                else:\n                    self.emitcode()\n            else: # sizes\n                dh, dw = bool(l&1), bool(l&2)\n                if dh or dw:\n                    self.setafter(dh=dh, dw=dw)\n                else:\n                    self.setat(dh=dh, dw=dw)\n\n        elif h == 0x10:\n            if l < 8:\n                self.setafter(fg=l, mosaic=True, conceal=False)\n            elif l == 0x8: # conceal\n                self.setat(conceal=True)\n            elif l == 0x9: # contiguous mosaic\n                self.setat(solid=True)\n            elif l == 0xa: # separated mosaic\n                self.setat(solid=False)\n            elif l == 0xb: # esc/switch\n                self.emitcode()\n                self._esc = not self._esc\n            elif l == 0xc: # black background\n                self.setat(bg = 0)\n            elif l == 0xd: # new background\n                self.setat(bg = self._state['fg'])\n            elif l == 0xe: # hold mosaic\n                self._held = True\n                self.emitcode()\n            elif l == 0xf: # release mosaic\n                self.emitcode()\n                self._held = False\n        else:\n            c = self.ttchar(b)\n            if self._state['mosaic'] and (b & 0x20):\n                self._heldmosaic = c\n                self._heldsolid = self._state['solid']\n            self._emitcharacter(c)\n\n    def parse(self):\n        self.reset()\n        prev = None\n        for c in self.tt&0x7f:\n            self.parsebyte(c, prev)\n            prev = c\n"
  },
  {
    "path": "teletext/pipeline.py",
    "content": "from collections import defaultdict\nfrom statistics import mode as pymode\n\nimport numpy as np\n\nfrom scipy.stats.mstats import mode\nfrom tqdm import tqdm\n\nfrom .subpage import Subpage\nfrom .packet import Packet\n\n\ndef check_buffer(mb, pages, subpages, min_rows=0):\n    if (len(mb) > min_rows) and mb[0].type == 'header':\n        page = mb[0].header.page | (mb[0].mrag.magazine * 0x100)\n        if page in pages or (page & 0x7ff) in pages:\n            if mb[0].header.subpage in subpages:\n                yield sorted(mb, key=lambda p: p.mrag.row)\n\n\ndef packet_squash(packets):\n    return Packet(mode(np.stack([p._array for p in packets]), axis=0)[0][0].astype(np.uint8))\n\n\ndef bsdp_squash_format1(packets):\n    date = pymode([p.broadcast.format1.date for p in packets])\n    hour = min(pymode([p.broadcast.format1.hour for p in packets]), 99)\n    minute = min(pymode([p.broadcast.format1.minute for p in packets]), 99)\n    second = min(pymode([p.broadcast.format1.second for p in packets]), 99)\n    return f'{date} {hour:02d}:{minute:02d}:{second:02d}'\n\n\ndef bsdp_squash_format2(packets):\n    day = min(pymode([p.broadcast.format2.day for p in packets]), 99)\n    month = min(pymode([p.broadcast.format2.month for p in packets]), 99)\n    hour = min(pymode([p.broadcast.format1.hour for p in packets]), 99)\n    minute = min(pymode([p.broadcast.format1.minute for p in packets]), 99)\n    return f'{month:02d}-{day:02d} {hour:02d}:{minute:02d}'\n\ndef paginate(packets, pages=range(0x900), subpages=range(0x3f80), drop_empty=False):\n\n    \"\"\"Yields packet lists containing contiguous rows.\"\"\"\n\n    magbuffers = [[],[],[],[],[],[],[],[]]\n    for packet in packets:\n        mag = packet.mrag.magazine & 0x7\n        if packet.type == 'header':\n            yield from check_buffer(magbuffers[mag], pages, subpages, 1 if drop_empty else 0)\n            magbuffers[mag] = []\n        magbuffers[mag].append(packet)\n    for mb in magbuffers:\n        yield from check_buffer(mb, pages, subpages, 1 if drop_empty else 0)\n\n\ndef subpage_group(packet_lists, threshold, ignore_empty):\n\n    \"\"\"Group similar subpages.\"\"\"\n    spdict = defaultdict(list)\n\n    for pl in packet_lists:\n        if len(pl) > 1:\n            subpage = Subpage.from_packets(pl, ignore_empty=ignore_empty)\n            group = spdict[(subpage.mrag.magazine, subpage.header.page, subpage.header.subpage)]\n            for op in group:\n                if threshold == -1:\n                    op.append(subpage)\n                    break\n                d = subpage.diff(op[0])\n                if d < threshold:\n                    op.append(subpage)\n                    break\n            else:\n                group.append([subpage])\n\n    groups = []\n    for group in spdict.values():\n        groups.extend(group)\n    return groups\n\n\ndef subpage_squash(packet_lists, threshold=-1, min_duplicates=3, ignore_empty=False):\n\n    \"\"\"Yields squashed subpages.\"\"\"\n\n    for splist in tqdm(subpage_group(packet_lists, threshold, ignore_empty), unit=' Groups'):\n        if len(splist) >= min_duplicates:\n            numbers = mode(np.stack([np.clip(sp.numbers, -100, -1) for sp in splist]), axis=0)[0][0].astype(np.int64)\n            s = Subpage(numbers=numbers)\n            for row in range(29):\n                if row in [26, 27, 28]:\n                    for dc in range(16):\n                        if s.has_packet(row, dc):\n                            packets = [sp.packet(row, dc) for sp in splist if sp.has_packet(row, dc)]\n                            arr = np.stack([p[3:] for p in packets])\n                            s.packet(row, dc)[:3] = packets[0][:3]\n                            if row == 27:\n                                s.packet(row, dc)[3:] = mode(arr, axis=0)[0][0].astype(np.uint8)\n                            else:\n                                t = arr.astype(np.uint32)\n                                t = t[:, 0::3] | (t[:, 1::3] << 8) | (t[:, 2::3] << 16)\n                                result = mode(t, axis=0)[0][0].astype(np.uint32)\n                                s.packet(row, dc)[3::3] = result & 0xff\n                                s.packet(row, dc)[4::3] = (result >> 8) & 0xff\n                                s.packet(row, dc)[5::3] = (result >> 16) & 0xff\n                else:\n                    if s.has_packet(row):\n                        packets = [sp.packet(row) for sp in splist if sp.has_packet(row)]\n                        arr = np.stack([p[2:] for p in packets])\n                        s.packet(row)[:2] = packets[0][:2]\n                        s.packet(row)[2:] = mode(arr, axis=0)[0][0].astype(np.uint8)\n\n            yield s\n\n\ndef to_file(packets, f, format):\n\n    \"\"\"Write packets to f as format.\"\"\"\n\n    if format == 'auto':\n        format = 'debug' if f.isatty() else 'bytes'\n    if f.isatty():\n        for p in packets:\n            with tqdm.external_write_mode():\n                f.write(getattr(p, format))\n            yield p\n    else:\n        for p in packets:\n            f.write(getattr(p, format))\n            yield p\n"
  },
  {
    "path": "teletext/printer.py",
    "content": "import re\n\nfrom .parser import Parser\n\n\nclass PrinterANSI(Parser):\n\n    def __init__(self, tt, colour=True, codepage=0):\n        self.colour = colour\n        super().__init__(tt)\n\n    def fgChanged(self):\n        if self.colour:\n            self._results.append('\\033[3{fg}m'.format(**self._state))\n\n    def bgChanged(self):\n        if self.colour:\n            self._results.append('\\033[4{bg}m'.format(**self._state))\n\n    def emitcharacter(self, c):\n        self._results.append(c)\n\n    def parse(self):\n        self._results = []\n        if self.colour:\n            self._results.append('\\033[37m\\033[40m')\n        super().parse()\n        if self.colour:\n            self._results.append('\\033[0m')\n\n    def __str__(self):\n        return ''.join(self._results)\n\n\nclass PrinterHTML(Parser):\n\n    def __init__(self, tt, fastext=None, pages_set=range(0x100), localcodepage=None, codepage=0):\n        self.flinkopen = False\n        self.fastext = fastext\n        self.pages_set = pages_set\n\n        # anchor for header links so we can bookmark a subpage\n        self.anchor = \"\"\n\n        super().__init__(tt, localcodepage, codepage)\n\n    def ttchar(self, c):\n        # Use the unicode characters produced by the base parser\n        # but escape < and > so as not to break the HTML.\n        c = Parser.ttchar(self, c)\n        if c == ord('<'):\n            return '&lt;'\n        elif c == ord('>'):\n            return '&gt;'\n        else:\n            return c\n\n    def stateChanged(self):\n        link = ''\n        linkclose = ''\n        if self.fastext:\n            if self.flinkopen:\n                linkclose = '</a>'\n                self.flinkopen = False\n            fg = self._state['fg']\n            if fg in [1,2,3,6] and self.fastext[[1,2,3,6].index(fg)] in self.pages_set:\n                link = '<a href=\"%s.html\">' % self.fastext[[1,2,3,6].index(fg)]\n                self.flinkopen = True\n\n        self._results.extend([\n            linkclose, '</span>',\n            '<span class=\"f{fg} b{bg}'.format(**self._state),\n            (' dh' if self._state['dh'] else ''),\n            (' fl' if self._state['flash'] else ''),\n            (' cn' if self._state['conceal'] else ''),\n            (' bx' if self._state['boxed'] else ' nx'),\n            '\">', link\n        ])\n\n    def emitcharacter(self, c):\n        self._results.append(c)\n\n    def linkify(self, html):\n        e = '([^0-9])([0-9]{3})([^0-9]|$)'\n        def repl(match):\n            if match.group(2) in self.pages_set:\n                return '%s<a href=\"%s.html%s\">%s</a>%s' % (match.group(1), match.group(2), self.anchor, match.group(2), match.group(3))\n            else:\n                return '%s%s%s' % (match.group(1), match.group(2), match.group(3))\n        p = re.compile(e)\n        return p.sub(repl, html)\n\n    def parse(self):\n        self._results = ['<span class=\"row\"><span class=\"f7 b0 nx\">']\n        super().parse()\n        self._results.append('</span></span>')\n        if self.flinkopen:\n            self._results.append('</a>')\n        self._string = ''.join(self._results)\n        if self.fastext is None:\n            self._string = self.linkify(self._string)\n\n    def __str__(self):\n        return self._string\n"
  },
  {
    "path": "teletext/service.py",
    "content": "import datetime\nimport os\nimport textwrap\n\nfrom collections import defaultdict\n\nfrom tqdm import tqdm\n\nfrom .subpage import Subpage\nfrom .file import FileChunker\nfrom .packet import Packet\nfrom . import pipeline\n\n\nclass Page(object):\n    def __init__(self):\n        self.subpages = {}\n        self._iter = self._gen()\n\n    def _gen(self):\n        while True:\n            if len(self.subpages) > 0:\n                yield from sorted(self.subpages.items())\n            yield 0x3f7f, None\n\n    def __iter__(self):\n        return self\n\n    def __next__(self):\n        return next(self._iter)\n\n\nclass Magazine(object):\n    def __init__(self, title=None):\n        self.title = title or \"Unnamed  \"\n        self.pages = defaultdict(Page)\n        self._iter = self._gen()\n\n    def _gen(self):\n        while True:\n            for pageno, page in sorted(self.pages.items()):\n                spno, subpage = next(page)\n                if subpage is None:\n                    p = Packet()\n                    p.mrag.row = 0\n                    p.header.page = 0xff\n                    p.header.subpage = spno\n                    yield p\n                else:\n                    subpage.header.page = pageno\n                    subpage.header.subpage = spno\n                    yield from subpage.packets\n\n    def __iter__(self):\n        return self\n\n    def __next__(self):\n        return next(self._iter)\n\n\nclass Service(object):\n    def __init__(self, replace_headers=False, title=None):\n        self.magazines = defaultdict(lambda: Magazine(title=title))\n        self.priorities = [1,1,1,1,1,1,1,1]\n        self.replace_headers = replace_headers\n        self._iter = self._gen()\n\n    def header(self, title, mag, page):\n        t = datetime.datetime.now()\n        return '%-9s%1d%02x' % (title, mag, page) + t.strftime(\" %a %d %b\\x03%H:%M/%S\")\n\n    def insert_page(self, page):\n        self.magazines[page.mrag.magazine].pages[page.header.page].subpages[page.header.subpage] = page\n\n    def _gen(self):\n        while True:\n            for n,m in sorted(self.magazines.items()):\n                for count in range(self.priorities[n&0x7]):\n                    packet = next(m)\n                    packet.mrag.magazine = n\n                    if packet.type == 'header':\n                        packet = Packet(packet._array)\n                        packet.header.control &= 0x77f # clear magazine serial\n                        if self.replace_headers:\n                            packet.header.displayable.place_string(self.header(m.title, n, packet.header.page))\n                    yield packet\n\n    def __iter__(self):\n        return self\n\n    def __next__(self):\n        return next(self._iter)\n\n    def packets(self, n):\n        for i in range(n):\n            yield next(self)\n\n    @property\n    def all_subpages(self):\n        for km, m in sorted(self.magazines.items()):\n            for kp, p in sorted(m.pages.items()):\n                for ks, s in sorted(p.subpages.items()):\n                    yield s\n\n    @property\n    def pages_set(self):\n        return set(f'{m}{p:02x}' for m, mag in self.magazines.items() for p, _ in mag.pages.items())\n\n    @classmethod\n    def from_packets(cls, packets, replace_headers=False, title=None):\n        svc = cls(replace_headers=replace_headers, title=title)\n        subpages = (Subpage.from_packets(pl) for pl in pipeline.paginate(packets, drop_empty=True))\n\n        for s in subpages:\n            page = svc.magazines[s.mrag.magazine].pages[s.header.page]\n            if s.header.subpage in page.subpages:\n                page.subpages[s.header.subpage].duplicates.append(s)\n            else:\n                page.subpages[s.header.subpage] = s\n\n        return svc\n\n    @classmethod\n    def from_file(cls, f):\n        chunks = FileChunker(f, 42)\n        packets = (Packet(data, number) for number, data in chunks)\n        return cls.from_packets(packets)\n\n    def to_html(self, outdir, template=None, localcodepage=None):\n\n        pages_set = self.pages_set\n\n        if template is None:\n            template = textwrap.dedent(\"\"\"\\\n                <html>\n                    <head>\n                        <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n                        <title>Page {page}</title>\n                        <link rel=\"stylesheet\" type=\"text/css\" href=\"teletext.css\" title=\"Default Style\"/>\n                        <link rel=\"alternative stylesheet\" type=\"text/css\" href=\"teletext-noscanlines.css\" title=\"No Scanlines\"/>\n                        <script type=\"text/javascript\" src=\"cssswitch.js\"></script>\n                    </head>\n                    <body onload=\"set_style_from_cookie()\">\n                    {body}\n                    </body>\n                </html>\n            \"\"\")\n\n        for magazineno, magazine in tqdm(self.magazines.items(), desc='Magazines', unit='M'):\n            for pageno, page in tqdm(magazine.pages.items(), desc='Pages', unit='P'):\n                pagestr = f'{magazineno}{pageno:02x}'\n                outfile = open(os.path.join(outdir, f'{pagestr}.html'), 'w', encoding='utf-8')\n                body = '\\n'.join(\n                    subpage.to_html(pages_set, localcodepage) for n, subpage in sorted(page.subpages.items())\n                )\n                outfile.write(template.format(page=pagestr, body=body))\n\n"
  },
  {
    "path": "teletext/servicedir.py",
    "content": "import pathlib\n\nfrom watchdog.events import FileModifiedEvent, FileDeletedEvent\nfrom watchdog.observers import Observer\n\nfrom .service import Service\nfrom .subpage import Subpage\n\n\nclass ServiceDir(Service):\n    \"\"\"\n    Implements a service backed by a directory of t42 files.\n\n    The files should be organized by page number with one subpage\n    per file, like this: 100/0000.t42\n\n    Whenever a file is modified it will be reloaded into the\n    service for broadcast in the next loop of the magazine.\n    \"\"\"\n    def __init__(self, directory, replace_headers, title):\n        super().__init__(replace_headers=replace_headers, title=title)\n        self._dir = directory\n\n    def file_changed(self, f, deleted=False):\n        try:\n            m = int(f.parent.name[0])\n            p = int(f.parent.name[1:], 16)\n            s = int(f.stem, 16)\n        except ValueError:\n            pass\n        else:\n            if deleted:\n                del self.magazines[m].pages[p].subpages[s]\n            else:\n                self.magazines[m].pages[p].subpages[s] = Subpage.from_file(f.open('rb'))\n\n    def __enter__(self):\n        self.observer = Observer()\n        self.observer.schedule(self, self._dir, recursive=True)\n        self.observer.start()\n\n        # perform initial scan of the pages\n        path = pathlib.Path(self._dir)\n        for f in path.rglob(\"*\"):\n            if f.is_file():\n                self.file_changed(f)\n\n        return self\n\n    def __exit__(self, *args, **kwargs):\n        self.observer.stop()\n        self.observer.join()\n\n    def dispatch(self, evt):\n        f = pathlib.Path(evt.src_path)\n        if isinstance(evt, FileModifiedEvent):\n            self.file_changed(f)\n        elif isinstance(evt, FileDeletedEvent):\n            self.file_changed(f, deleted=True)\n\n"
  },
  {
    "path": "teletext/sigint.py",
    "content": "import signal\n\nclass SigIntDefer(object):\n\n    \"\"\"\n    SigIntDefer is a context manager which catches SIGINT (aka KeyboardInterrupt)\n    and allows the code to check if it has happened or not, and then exit at an\n    appropriate time, instead of in the middle of doing something important.\n\n    Example: the goal here is to make sure that if we print \"hello\", we always\n    print \"goodbye\", even if a KeyboardInterrupt happens. If a KeyboardInterrupt\n    did happen, SigIntDefer will re-fire it upon exiting the context.\n\n    import time\n\n    def loop():\n        with SigIntDefer() as sigint:\n            while True:\n                if sigint.fired:\n                    return\n                print(\"hello\")\n                time.sleep(0.5)\n                print(\"goodbye\")\n\n    loop()\n    \"\"\"\n\n    def __init__(self, times=1):\n        self._times = 1\n        self._fired = None\n\n    def __enter__(self):\n        self._old_handler = signal.getsignal(signal.SIGINT)\n        signal.signal(signal.SIGINT, self.handler)\n        return self\n\n    def handler(self, f, n):\n        self._fired = (f, n)\n        self._times -= 1\n        if self._times == 0:\n            signal.signal(signal.SIGINT, self._old_handler)\n\n    @property\n    def fired(self):\n        return self._fired is not None\n\n    def __exit__(self, *args, **kwargs):\n        signal.signal(signal.SIGINT, self._old_handler)\n        if self._fired:\n            self._old_handler(*self._fired)\n"
  },
  {
    "path": "teletext/spellcheck.py",
    "content": "import itertools\n\nimport enchant\n\nfrom .coding import parity_encode\n\n\nclass SpellChecker(object):\n\n    common_errors = set(itertools.chain.from_iterable(\n        itertools.permutations(s, 2) for s in (\n            'ab', 'bd', 'ce', 'dh', 'ef', 'ei', 'ej', 'er', 'fj', 'ij', 'jl', 'jr', 'jt',\n            'km', 'ks', 'ku', 'lt', 'mn', 'qr', 'rt', 'tx', 'uv', 'uy', 'uz', 'vz', 'yz',\n        )\n    ))\n\n    def __init__(self, language='en_GB'):\n        self.dictionary = enchant.Dict(language)\n\n    def check_pair(self, x, y):\n        if x == y or (x, y) in self.common_errors:\n            return 0\n        return 1\n\n    def weighted_hamming(self, a, b):\n        return sum(self.check_pair(x, y) for x,y in zip(a, b))\n\n    def case_match(self, word, src):\n        return ''.join(c.lower() if d.islower() else c.upper() for c, d in zip(word, src))\n\n    def suggest(self, word):\n        if len(word) > 1:\n            lcword = word.lower()\n            if not self.dictionary.check(lcword):\n                for suggestion in self.dictionary.suggest(lcword):\n                    if len(suggestion) == len(lcword) and self.weighted_hamming(suggestion.lower(), lcword) == 0:\n                        return self.case_match(suggestion, word)\n        return word\n\n    def spellcheck(self, displayable):\n        words = ''.join(c if c.isalpha() else ' ' for c in displayable.to_ansi(colour=False)).split(' ')\n\n        words = [self.suggest(w) for w in words]\n\n        line = ' '.join(words).encode('ascii')\n        for n, b in enumerate(line):\n            if b != ord(b' '):\n                displayable[n] = parity_encode(b)\n\n\ndef spellcheck_packets(packets, language='en_GB'):\n\n    sc = SpellChecker(language)\n\n    for p in packets:\n        t = p.type\n        if t == 'display':\n            sc.spellcheck(p.displayable)\n        elif t == 'header':\n            sc.spellcheck(p.header.displayable)\n        yield p\n"
  },
  {
    "path": "teletext/stats.py",
    "content": "import numpy as np\n\n\nclass Histogram(object):\n\n    bars = ' ▁▂▃▄▅▆▇█'\n    label = 'H'\n    bins = range(2)\n\n    def __init__(self, shape=(1000, ), fill=255, dtype=np.uint8):\n        self._data = np.full(shape, fill_value=fill, dtype=dtype)\n        self._pos = 0\n\n    def insert(self, value):\n        self._data[self._pos] = value\n        self._pos += 1\n        self._pos %= self._data.shape[0]\n\n    @property\n    def histogram(self):\n        h,_ = np.histogram(self._data, bins=self.bins)\n        return h\n\n    @property\n    def render(self):\n        h = self.histogram\n        m = max(1, np.max(h)) # no div by zero\n        if m == 0:\n            return (' ' * len(self.bins))\n        else:\n            h2 = np.ceil(h * ((len(self.bars) - 1) / m)).astype(np.uint8)\n            return ''.join(self.bars[n] for n in h2)\n\n    def __str__(self):\n        return f', {self.label}:|{self.render}|'\n\n\nclass MagHistogram(Histogram):\n\n    label = 'M'\n    bins = range(1, 9)\n\n    def __init__(self, packets, size=1000):\n        super().__init__((size, ))\n        self._packets = packets\n\n    def __iter__(self):\n        for p in self._packets:\n            self.insert(p.mrag.magazine)\n            yield p\n\n\nclass RowHistogram(MagHistogram):\n\n    label = 'R'\n    bins = range(33)\n\n    def __iter__(self):\n        for p in self._packets:\n            self.insert(p.mrag.row)\n            yield p\n\n\nclass Rejects(Histogram):\n\n    label = 'R'\n    bins = range(3)\n\n    def __init__(self, lines, size=1000):\n        super().__init__((size, ))\n        self._lines = lines\n\n    def __iter__(self):\n        for l in self._lines:\n            self.insert(l == 'rejected')\n            yield l\n\n    def __str__(self):\n        h = self.histogram\n        total = max(1, np.sum(h))\n        return f', {self.label}:{100*h[1]/total:.0f}%'\n\n\nclass ErrorHistogram(Histogram):\n\n    label = 'E'\n\n    def __init__(self, packets, size=100):\n        super().__init__((size, 6), fill=0, dtype=np.uint32)\n        self._packets = packets\n\n    def __iter__(self):\n        for p in self._packets:\n            self.insert(np.sum(p.vector_gain_errors.reshape(6, -1), axis=1))\n            yield p\n\n    def __str__(self):\n        bins = np.sum(self._data, axis=0)\n        bins = np.ceil(bins * ((len(self.bars) - 1) * 2 / self._data.shape[0])).astype(np.uint8)\n        bins = np.clip(bins, 0, len(self.bars)-1)\n        return f', {self.label}: |{\"\".join(self.bars[n] for n in bins)}|'\n\n\nclass StatsList(list):\n    def __str__(self):\n        return ''.join(str(x) for x in self)\n"
  },
  {
    "path": "teletext/subpage.py",
    "content": "import base64\nimport itertools\n\nimport numpy as np\n\nfrom .coding import crc\nfrom .packet import Packet\nfrom .elements import Element, Displayable\nfrom .printer import PrinterHTML\nfrom .file import FileChunker\n\nclass Subpage(Element):\n\n    def __init__(self, array=None, numbers=None, prefill=False, magazine=1):\n        super().__init__((26 + (3*16), 42), array)\n        if numbers is None:\n            self._numbers = np.full((26 + (3*16),), fill_value=-100, dtype=np.int64)\n        else:\n            self._numbers = numbers\n\n        if prefill:\n            for i in range(0, 25):\n                self.init_packet(i, 0, magazine)\n            self.header.displayable[:] = 0x20\n            self.header.subpage = 0\n            self.header.control = 1<<0 # erase\n            self.displayable[:] = 0x20\n\n        self.duplicates = []\n\n    def diff(self, other):\n        \"\"\"Try to determine if two subpages are the same.\"\"\"\n        diff = np.sum(self._array != other._array, axis=1)\n        rows = (self._numbers != -100) & (other._numbers != -100)\n        return np.sum(diff * rows)\n\n    @property\n    def numbers(self):\n        return self._numbers[:]\n\n    def _slot(self, row, dc):\n        if row < 26:\n            return row\n        else:\n            return ((row-26)*16)+26+dc\n\n    def has_packet(self, row, dc=0):\n        return self._numbers[self._slot(row, dc)] > -100\n\n    def init_packet(self, row, dc=0, magazine=1):\n        self.packet(row, dc).mrag.row = row\n        self.packet(row, dc).mrag.magazine = magazine\n        self._numbers[self._slot(row, dc)] = -1\n\n    def packet(self, row, dc=0):\n        try:\n            return Packet(self._array[self._slot(row, dc), :])\n        except IndexError:\n            print(row, dc)\n            raise\n\n    @property\n    def mrag(self):\n        return self.packet(0).mrag\n\n    @property\n    def header(self):\n        return self.packet(0).header\n\n    @property\n    def codepage(self):\n        return self.packet(0).header.codepage\n\n    @property\n    def fastext(self):\n        return self.packet(27, 0).fastext\n\n    @property\n    def displayable(self):\n        return Displayable((24, 40), self._array[1:25,2:])\n\n    @property\n    def checksum(self):\n        '''Calculates the actual checksum of the subpage.'''\n        c = 0\n        if self.has_packet(0):\n            for b in self.header.displayable[:24]:\n                c = crc(b, c)\n        else:\n            for n in range(24):\n                c = crc(0x20, c)\n\n        for r in range(1, 26):\n            if self.has_packet(r):\n                for b in self.packet(r).displayable:\n                    c = crc(b, c)\n            else:\n                for n in range(40):\n                    c = crc(0x20, c)\n\n        return c\n\n    @property\n    def addr(self):\n        return f'{self.mrag.magazine}{self.header.page:02X}:{self.header.subpage:04X}'\n\n    @classmethod\n    def from_packets(cls, packets, ignore_empty=False):\n        s = cls()\n\n        for p in packets:\n            row = p.mrag.row\n            if row >= 29:\n                continue\n            dc = 0 if row < 26 else p.dc.dc\n            i = s._slot(row, dc)\n            if ignore_empty and s._numbers[i] > -100:\n                # we've already seen this packet\n                # if the new one is closer to all spaces than the old one, skip it\n                if np.sum(s._array[i, :] == 0x80) < np.sum(p[:] == 0x80):\n                    continue\n            s._array[i, :] = p[:]\n            s._numbers[i] = -1 if p.number is None else p.number\n        return s\n\n    @classmethod\n    def from_url(cls, url):\n        s = cls(prefill=True)\n        parts = url.split(':')\n        Element((25, 40), s._array[0:25, 2:]).sevenbit = base64.urlsafe_b64decode(parts[1]+'==')\n        for p in parts[2:]:\n            l, d = p.split('=', maxsplit=1)\n            if l == 'PN':\n                s.mrag.magazine = int(d[0], 16)\n                s.header.page = int(d[1:3], 16)\n            elif l == 'PS':\n                c = int(d, 16)\n                s.header.control = (c<<1) | ((c&1)>>14)\n            elif l == 'SC':\n                pass  # TODO\n            elif l == 'X25':\n                pass  # TODO\n            elif l == 'X270':\n                pass  # TODO\n            elif l == 'X280':\n                pass  # TODO\n            elif l == 'X284':\n                pass  # TODO\n        return s\n\n    @classmethod\n    def from_file(cls, f):\n        chunks = FileChunker(f, 42)\n        packets = (Packet(data, number) for number, data in chunks)\n        return cls.from_packets(packets)\n\n    @property\n    def packets(self):\n        for n, a in enumerate(self._array):\n            if self._numbers[n] > -100:\n                yield Packet(a, number=None if self._numbers[n] < 0 else self._numbers[n])\n\n    @property\n    def mrg_PN(self):\n        return f'{self.mrag.magazine}{self.header.page:02x}'\n\n    def mrg_PS(self, transmit=False):\n        c = self.header.control\n        c = (c>>1) | ((c&1)<<14)\n        if transmit:\n            c |= 1<<15\n        return f'{c:x}'\n\n    @property\n    def mrg_SC(self):\n        return f'{self.header.subpage:x}'\n\n\n\n    @property\n    def url(self):\n        data = self._array[0:25,2:].copy()\n        data[0, :8] = 0x20\n        parts = [\n            '0',\n            base64.urlsafe_b64encode(Element((25, 40), data).sevenbit).decode('ascii').rstrip('='),\n            f'PN={self.mrg_PN}',\n            f'PS={self.mrg_PS()}',\n            f'SC={self.mrg_SC}',\n        ]\n        if self.has_packet(25):\n            parts.append('X25=' + base64.urlsafe_b64encode(Element((1, 40), self._array[25:26,2:]).sevenbit).decode('ascii').rstrip('='))\n        for d in range(16):\n            if self.has_packet(26, d):\n                pass # TODO: X26\n        if self.has_packet(27, 0):\n            parts.append('X270=' + ''.join([f'{l.magazine}{l.page:02x}{l.subpage:04x}' for l in self.fastext.links]) + f'{self.fastext.control:1x}')\n        if self.has_packet(28, 0):\n            pass # TODO: X280\n        if self.has_packet(28, 4):\n            pass # TODO: X284\n\n        return ':'.join(parts)\n\n    def to_tti(self, cycle_time=None, transmit=True):\n        parts = [\n            f'PN,{self.mrg_PN}00',\n            f'SC,{self.mrg_SC}',\n            f'PS,{self.mrg_PS(transmit)}',\n        ]\n        if cycle_time is not None:\n            parts.append(f'CT,{cycle_time}')\n\n        parts.extend(f'OL,{line+1},{data}' for line, data in enumerate(self.displayable.to_tti()))\n        links = ','.join(f'{l.magazine}{l.page:02x}' for l in self.fastext.links)\n        parts.append(f'FL,{links}')\n        return ('\\r\\n'.join(parts) + '\\r\\n').encode('ascii')\n\n    def to_html(self, pages_set, localcodepage=None):\n        lines = []\n\n        lines.append(f'<div class=\"subpage\" id=\"{self.header.subpage:04x}\">')\n        buf = np.full((40,), fill_value=0x20, dtype=np.uint8)\n        buf[3:7] = np.fromstring(f'P{self.mrag.magazine}{self.header.page:02x}', dtype=np.uint8)\n        buf[8:] = self.header.displayable[:]\n        p = PrinterHTML(buf, localcodepage=localcodepage, codepage=self.codepage)\n        p.anchor = f'#{self.header.subpage:04x}'\n        lines.append(str(p))\n\n        for i in range(0,24):\n            # only draw the line if previous line does not contain double height code\n            if i == 0 or np.all(self.displayable[i-1,:] != 0x0d):\n                fastext = [f'{l.magazine}{l.page:02x}' for l in self.fastext.links] if i == 23 else None\n                p = PrinterHTML(self.displayable[i,:], fastext=fastext, pages_set=pages_set, localcodepage=localcodepage, codepage=self.codepage)\n                lines.append(str(p))\n\n        lines.append('</div>')\n\n        return ''.join(lines)\n\n"
  },
  {
    "path": "teletext/ts.py",
    "content": "# Based on https://github.com/fsphil/tstxtdump with assistance from the author :)\n\nimport itertools\nimport struct\n\nfrom .coding import byte_reverse\n\n\ndef parse_data(data):\n    pos = 0\n    while (len(data) - pos) >= 46:\n        if data[pos] in [2, 3]:\n            yield bytes(byte_reverse(b) for b in data[pos+4:pos+4+42])\n        pos += 46\n\n\ndef parse_pes(pes):\n    pos = 0\n    while (len(pes) - pos) >= 9:\n        l, o = struct.unpack('!xxxxHxxB', pes[pos:pos+9])\n        yield from parse_data(pes[pos+10+o:pos+l-o-4])\n        pos += l + 6\n\n\ndef pidextract(packets, pid):\n    pes = []\n    count = itertools.count()\n    start_seen = False\n    for n, packet in packets:\n        t, p, c = struct.unpack('!BHB', packet[:4])\n        o = 4\n        if t == 0x47 and (p&0x1fff) == pid:\n            if p & 0x4000:\n                if pes:\n                    yield from ((y, x) for x, y in zip(parse_pes(b''.join(pes)), count))\n                    pes = []\n                start_seen = True\n            if start_seen:\n                if c & 0x20:  # adaptation field\n                    o += packet[4] + 1\n                pes.append(packet[o:])\n\n\n"
  },
  {
    "path": "teletext/vbi/__init__.py",
    "content": ""
  },
  {
    "path": "teletext/vbi/clustering.py",
    "content": "import pathlib\nimport numpy as np\nfrom collections import defaultdict\nfrom itertools import islice\nfrom binascii import hexlify\n\n\ndef cluster(a, l, clusters=None, steps=None):\n    if clusters is None:\n        clusters = defaultdict(list)\n    if steps is None:\n        steps = np.floor(np.linspace(0, a.shape[1]-5, num=11)).astype(np.uint32)[[1, 5, 10]]\n    v = np.empty((a.shape[0], 5), dtype=np.uint8)\n    v[:, 0] = l\n    v[:, 1] = np.floor(np.mean(np.abs(np.diff(a.astype(np.int16), axis=1)), axis=1)).astype(np.uint8)\n    v[:, 2:] = np.diff(np.sort(a, axis=1)[:, steps] >> 4, axis=1, prepend=0)\n    for vv, aa in zip(v, a):\n        clusters[vv.tobytes()].append(aa)\n    return v, clusters\n\n\ndef batched(iterable, n):\n    \"Batch data into lists of length n. The last batch may be shorter.\"\n    # batched('ABCDEFG', 3) --> ABC DEF G\n    it = iter(iterable)\n    while True:\n        batch = list(islice(it, n))\n        if not batch:\n            return\n        yield batch\n\n\ndef batch_cluster(chunks, output, prefix=\"\", lpf=32):\n    output = pathlib.Path(output)\n\n    with (output / f'{prefix}map.bin').open('wb') as mapfile:\n\n        for batch in batched(chunks, 10000):\n            a = np.stack(list(np.frombuffer(i[1], dtype=np.uint8) for i in batch))\n            l = np.array(list(i[0] for i in batch)) % lpf\n            map, clusters = cluster(a, l)\n            mapfile.write(map.tobytes())\n            for k, v in clusters.items():\n                p = output / f'{prefix}{hexlify(k).decode(\"utf8\")}.vbi'\n                with p.open('ab') as f:\n                    for l in v:\n                        f.write(l.tobytes())\n\n\ndef rendermap(config, map, output):\n    from PIL import Image\n    import math\n    a = np.fromfile(map, dtype=np.uint8).reshape(-1, config.frame_lines, 5)\n    rows = []\n    frames = 25 * 60\n    for n in range(0, a.shape[0], frames):\n        r = a[n:n+frames]\n        if r.shape[0] < frames:\n            r = np.concatenate([r, np.zeros((frames-r.shape[0], config.frame_lines, 5), dtype=np.uint8)], axis=0)\n        r = np.swapaxes(r, 0, 1)\n        rows.append(r)\n    i = np.concatenate(rows, axis=0) * 20\n    i = Image.fromarray(i[:,:,[1, 3, 4]], mode=\"RGB\")\n    i.save(output)\n\n"
  },
  {
    "path": "teletext/vbi/config.py",
    "content": "import math\nimport pathlib\n\nimport numpy as np\n\nclass Config(object):\n\n    teletext_bitrate = 6937500.0\n    gauss = 4.0\n    std_thresh = 14\n\n    sample_rate: float\n    line_length: int\n    line_start_range: tuple\n    dtype: type\n    field_lines: int\n    field_range: range\n\n    extra_roll: int = 0\n    sample_rate_adjust: float = 0\n\n    # Clock run-in and framing code. These bits are set at the start of every teletext packet.\n    crifc = np.array((\n        1, -1, 1, -1, 1, -1, 1, -1,\n        1, -1, 1, -1, 1, -1, 1, -1,\n        1, 1, 1, -1, -1, 1, -1, -1,\n    ))\n\n    observed_crifc = np.array([\n        [133, 132, 129, 127, 124, 121, 119, 117],\n        [116, 115, 115, 115, 116, 117, 118, 119],\n        [120, 121, 121, 121, 121, 120, 119, 118],\n        [118, 117, 116, 116, 116, 117, 117, 118],\n        [119, 120, 120, 121, 121, 121, 120, 119],\n        [119, 118, 117, 116, 116, 116, 116, 117],\n        [118, 119, 120, 121, 122, 122, 122, 122],\n        [121, 120, 119, 118, 117, 117, 117, 117],\n        [118, 118, 119, 120, 121, 121, 121, 121],\n        [121, 120, 119, 119, 118, 118, 117, 117],\n        [118, 118, 119, 120, 121, 122, 122, 122],\n        [122, 121, 120, 119, 118, 118, 117, 117],\n        [117, 117, 118, 119, 120, 120, 121, 121],\n        [122, 122, 122, 122, 121, 121, 121, 121],\n        [120, 120, 119, 118, 116, 115, 113, 110],\n        [108, 105, 104, 103, 104, 107, 112, 119],\n        [128, 137, 147, 157, 166, 174, 179, 183],\n        [184, 183, 181, 178, 175, 171, 168, 166],\n        [164, 163, 162, 160, 159, 156, 153, 147],\n        [141, 133, 124, 114, 104,  96,  88,  82],\n        [ 78,  77,  79,  83,  90,  99, 108, 118],\n        [127, 134, 140, 144, 146, 145, 141, 136],\n        [128, 119, 110, 100,  91,  83,  76,  69],\n        [ 65,  61,  59,  57,  57,  57,  57,  58]\n    ], dtype=np.uint8)\n\n    observed_crifc_gradient = np.gradient(observed_crifc[8:24,:].flatten())\n\n    # Card specific default parameters:\n\n    cards = {\n        'bt8x8': {\n            'sample_rate': 35468950.0,\n            'line_length': 2048,\n            'line_start_range': (60, 130),\n            'dtype': np.uint8,\n            'field_lines': 16,\n            'field_range': range(0, 16),\n        },\n        'cx88': {\n            'sample_rate': 35468950.0,\n            'line_length': 2048,\n            'line_start_range': (90, 150),\n            'dtype': np.uint8,\n            'field_lines': 18,\n            'field_range': range(1, 17),\n        },\n        'tbc': { # ld-decode/vhs-decode tbc (full fields)\n            'sample_rate': 17730000.0,\n            'line_length': 1135,\n            'line_start_range': (160, 190),\n            'dtype': np.uint16,\n            'field_lines': 313,\n            'field_range': range(6, 22),\n        },\n        'tbc-vbi': {  # ld-decode/vhs-decode tbc (vbi only)\n            'sample_rate': 17730000.0,\n            'line_length': 1135,\n            'line_start_range': (160, 190),\n            'dtype': np.uint16,\n            'field_lines': 16,\n            'field_range': range(0, 16),\n        },\n        'saa7131': {\n            'sample_rate': 27000000.0,\n            'line_length': 1440,\n            'line_start_range': (0, 20),\n            'dtype': np.uint8,\n            'field_lines': 16,\n            'field_range': range(0, 16),\n        },\n    }\n\n    def __init__(self, card='bt8x8', **kwargs):\n        setattr(self, 'card', card)\n\n        for k, v in self.cards[card].items():\n            setattr(self, k, v)\n\n        for k, v in kwargs.items():\n            if v is not None:\n                setattr(self, k, v)\n\n        self.frame_lines = self.field_lines * 2\n\n        self.sample_rate += self.sample_rate_adjust\n\n        # width of a bit in samples (float)\n        self.bit_width = self.sample_rate / self.teletext_bitrate\n\n        results = []\n        for pad in range(500):\n            r = (self.line_length+pad) * 8 / self.bit_width\n            rs = round(r)\n            err = abs(r - rs)\n            results.append((err, pad, rs))\n\n        # resample params\n        self.resample_pad, self.resample_tgt = min(results)[1:]\n        self.resample_size = math.ceil(self.line_length * 8 / self.bit_width)\n\n        # region of the original line where the CRI begins, in samples\n        self.start_slice = slice(\n            math.floor(self.line_start_range[0] * 8 / self.bit_width),\n            math.ceil(self.line_start_range[1] * 8 / self.bit_width)\n        )\n\n        # last sample of original line where teletext may occur\n        self.line_trim = self.start_slice.stop + math.ceil(8 * 45 * 8)\n\n        # fft\n        self.fftbins = [0, 47, 54, 97, 104, 147, 154, 197, 204]\n\n    def __repr__(self):\n        return f'{type(self).__name__}: {self.__dict__}'\n\n    @property\n    def line_bytes(self):\n        return self.line_length * np.dtype(self.dtype).itemsize\n\n    __datadir = pathlib.Path(__file__).parent.parent / 'vbi' / 'data'\n    tape_formats = [f.name for f in __datadir.iterdir() if f.is_dir()]\n"
  },
  {
    "path": "teletext/vbi/line.py",
    "content": "# * Copyright 2016 Alistair Buxton <a.j.buxton@gmail.com>\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\n\nimport importlib\nimport math\nimport pathlib\nimport sys\nimport numpy as np\nfrom scipy.ndimage import gaussian_filter1d as gauss\nfrom scipy.signal import resample\n\nfrom teletext.packet import Packet\nfrom teletext.elements import Mrag, DesignationCode\n\nfrom .config import Config\n\n\ndef normalise(a, start=None, end=None):\n    mn = a[start:end].min()\n    mx = a[start:end].max()\n    r = (mx-mn)\n    if r == 0:\n        r = 1\n    return np.clip((a.astype(np.float32) - mn) * (255.0/r), 0, 255)\n\n\n# Line: Handles a single line of raw VBI samples.\n\nclass Line(object):\n    \"\"\"Container for a single line of raw samples.\"\"\"\n\n    config: Config\n\n    configured = False\n\n    @classmethod\n    def configure_patterns(cls, method, tape_format):\n        try:\n            module = importlib.import_module(\".pattern\" + method.lower(), __package__)\n            Pattern = getattr(module, \"Pattern\" + method)\n            datadir = pathlib.Path(__file__).parent / 'data' / tape_format\n            cls.h = Pattern(datadir / 'hamming.dat')\n            cls.p = Pattern(datadir / 'parity.dat')\n            cls.f = Pattern(datadir / 'full.dat')\n            return True\n        except Exception as e:\n            sys.stderr.write(str(e) + '\\n')\n            sys.stderr.write((method if method else 'CPU') + ' init failed.\\n')\n            return False\n\n    @classmethod\n    def configure(cls, config, force_cpu=False, prefer_opencl=False, tape_format='vhs'):\n        cls.config = config\n        if force_cpu:\n            methods = ['']\n        elif prefer_opencl:\n            methods = ['OpenCL', 'CUDA', '']\n        else:\n            methods = ['CUDA', 'OpenCL', '']\n        if any(cls.configure_patterns(method, tape_format) for method in methods):\n            cls.configured = True\n        else:\n            raise Exception('Could not initialize any deconvolution method.')\n\n    def __init__(self, data, number=None):\n        if not self.configured:\n            self.configure(Config())\n\n        self._number = number\n        self._original = np.frombuffer(data, dtype=self.config.dtype).astype(np.float32)\n        self._original /= 256 ** (np.dtype(self.config.dtype).itemsize-1)\n        self._original_bytes = data\n\n        resample_tmp = np.pad(self._original, (0, self.config.resample_pad), 'constant', constant_values=(0,0))\n\n        self._resampled = np.pad(resample(resample_tmp, self.config.resample_tgt)[:self.config.resample_size], (0, 64), 'edge')\n\n        self.reset()\n\n    def reset(self):\n        \"\"\"Reset line to original unknown state.\"\"\"\n        self.roll = 0\n\n        self._noisefloor = None\n        self._max = None\n        self._fft = None\n        self._gstart = None\n        self._is_teletext = None\n        self._start = None\n        self._reason = None\n\n    @property\n    def resampled(self):\n        \"\"\"The resampled line. 8 samples = 1 bit.\"\"\"\n        return self._resampled[:]\n\n    @property\n    def original(self):\n        \"\"\"The raw, untouched line.\"\"\"\n        return self._original[:]\n\n    @property\n    def rolled(self):\n        if self.start is not None:\n            return np.roll(self._resampled, 90-(self.start+self.roll))\n        else:\n            return self._resampled[:]\n\n    @property\n    def gradient(self):\n        return (np.gradient(gauss(self.rolled, 12))[20:300]>0)*255\n\n    def fchop(self, start, stop):\n        \"\"\"Chop the samples associated with each bit.\"\"\"\n        # This should use self.start not self._start so that self._start\n        # is calculated if it hasn't been already.\n        r = (self.start + self.roll)\n        # sys.stderr.write(f'{r}, {start}, {stop}, {d.shape}\\n')\n        return self._resampled[r + (start * 8):r + (stop * 8)]\n\n    def chop(self, start, stop):\n        \"\"\"Average the samples associated with each bit.\"\"\"\n        return np.mean(self.fchop(start, stop).reshape(-1, 8), 1)\n\n    @property\n    def chopped(self):\n        \"\"\"The whole chopped teletext line, for vbi viewer.\"\"\"\n        return self.chop(0, 360)\n\n    @property\n    def noisefloor(self):\n        if self._noisefloor is None:\n            if self.config.start_slice.start == 0:\n                self._noisefloor = np.max(gauss(self._resampled[self.config.line_trim:-4], self.config.gauss))\n            else:\n                self._noisefloor = np.max(gauss(self._resampled[:self.config.start_slice.start], self.config.gauss))\n        return self._noisefloor\n\n    @property\n    def fft(self):\n        \"\"\"The FFT of the original line.\"\"\"\n        if self._fft is None:\n            # This test only looks at the bins for the harmonics.\n            # It could be made smarter by looking at all bins.\n            self._fft = normalise(gauss(np.abs(np.fft.fft(np.diff(self._resampled[:3200], n=1))[:256]), 4))\n        return self._fft\n\n    def find_start(self):\n        # First try to detect by comparing pre-start noise floor to post-start levels.\n        # Store self._gstart so that self.start can re-use it.\n        self._gstart = gauss(self._resampled[self.config.start_slice], self.config.gauss)\n        smax = np.max(self._gstart)\n        if smax < 64:\n            self._is_teletext = False\n            self._reason = f'Signal max is {smax}'\n        elif self.noisefloor > 80:\n            self._is_teletext = False\n            self._reason = f'Noise is {self.noisefloor}'\n        elif smax < (self.noisefloor + 16):\n            # There is no interesting signal in the start_slice.\n            self._is_teletext = False\n            self._reason = f'Noise is higher than signal {smax} {self.noisefloor}'\n        else:\n            # There is some kind of signal in the line. Check if\n            # it is teletext by looking for harmonics of teletext\n            # symbol rate.\n            fftchop = np.add.reduceat(self.fft, self.config.fftbins)\n            self._is_teletext = np.sum(fftchop[1:-1:2]) > 1000\n        if not self._is_teletext:\n            return\n\n        # Find the steepest part of the line within start_slice.\n        # This gives a rough location of the start.\n        self._start = np.argmax(np.gradient(np.maximum.accumulate(self._gstart))) + self.config.start_slice.start\n        # Now find the extra roll needed to lock in the clock run-in and framing code.\n        confidence = []\n\n        for roll in range(max(-30, 8-self._start), 20):\n            self.roll = roll\n            # 15:20 is the last bit of CRI and first 4 bits of FC - 01110.\n            # This is the most distinctive part of the CRI/FC to look for.\n            c = self.chop(15, 21)\n            confidence.append((c[1] + c[2] + c[3] - c[0] - c[4] - c[5], roll))\n            #confidence.append((np.sum(self.chop(15, 20) * self.config.crifc[15:20]), roll))\n\n        self._start += max(confidence)[1]\n        self.roll = 0\n\n        # Use the observed CRIFC to lock to the framing code\n        confidence = []\n        for roll in range(-4, 4):\n            self.roll = roll\n            x = np.gradient(self.fchop(8, 24))\n            c = np.sum(np.square(x - self.config.observed_crifc_gradient))\n            confidence.append((c, roll))\n\n        self._start += min(confidence)[1]\n        self.roll = 0\n\n        self._start += self.config.extra_roll\n\n    @property\n    def is_teletext(self):\n        \"\"\"Determine whether the VBI data in this line contains a teletext signal.\"\"\"\n        if self._is_teletext is None:\n            self.find_start()\n        return self._is_teletext\n\n    @property\n    def start(self):\n        \"\"\"Find the offset in samples where teletext data begins in the line.\"\"\"\n        if self.is_teletext:\n            return self._start\n        else:\n            return None\n\n    def deconvolve(self, mags=range(9), rows=range(32), eight_bit=False):\n        \"\"\"Recover original teletext packet by pattern recognition.\"\"\"\n        if not self.is_teletext:\n            return 'rejected'\n\n        bytes_array = np.zeros((42,), dtype=np.uint8)\n\n        # Note: 368 (46*8) not 360 (45*8), because pattern matchers need an\n        # extra byte on either side of the input byte(s) we want to match for.\n        # The framing code serves this purpose at the beginning as we never\n        # need to match it. We need just an extra byte at the end.\n        bits_array = normalise(self.chop(0, 368))\n\n        # First match just the mrag and dc for the line.\n        bytes_array[:3] = self.h.match(bits_array[16:56])\n        m = Mrag(bytes_array[:2])\n        d = DesignationCode((1, ), bytes_array[2:3])\n        if m.magazine in mags and m.row in rows:\n            if m.row == 0:\n                bytes_array[3:10] = self.h.match(bits_array[40:112])\n                bytes_array[10:] = self.p.match(bits_array[96:368])\n            elif m.row < 26:\n                if eight_bit:\n                    bytes_array[2:] = self.f.match(bits_array[32:368])\n                else:\n                    bytes_array[2:] = self.p.match(bits_array[32:368])\n            elif m.row == 27:\n                if d.dc < 4:\n                    bytes_array[3:40] = self.h.match(bits_array[40:352])\n                    bytes_array[40:] = self.f.match(bits_array[336:368])\n                else:\n                    bytes_array[3:] = self.f.match(bits_array[40:368]) # TODO: proper codings\n            elif m.row < 30:\n                bytes_array[3:] = self.f.match(bits_array[40:368]) # TODO: proper codings\n            elif m.row == 30 and m.magazine == 8: # BDSP\n                bytes_array[3:9] = self.h.match(bits_array[40:104]) # initial page\n                if d.dc in [2, 3]:\n                    bytes_array[9:22] = self.h.match(bits_array[88:208]) # 8-bit data\n                else:\n                    bytes_array[9:22] = self.f.match(bits_array[88:208])  # 8-bit data\n                bytes_array[22:] = self.p.match(bits_array[192:368]) # status display\n            else:\n                bytes_array[3:] = self.f.match(bits_array[40:368]) # TODO: proper codings\n            return Packet(bytes_array, number=self._number, original=self._original_bytes)\n        else:\n            return 'filtered'\n\n    def slice(self, mags=range(9), rows=range(32), eight_bit=False):\n        \"\"\"Recover original teletext packet by threshold and differential.\"\"\"\n        if not self.is_teletext:\n            return 'rejected'\n\n        # Note: 23 (last bit of FC), not 24 (first bit of MRAG) because\n        # taking the difference reduces array length by 1. We cut the\n        # extra bit off when taking the threshold.\n        bits_array = normalise(self.chop(23, 360))\n        diff = np.diff(bits_array, n=1)\n        ones = (diff > 48)\n        zeros = (diff > -48)\n        result = ((bits_array[1:] > 127) | ones) & zeros\n\n        packet = Packet(np.packbits(result.reshape(-1,8)[:,::-1]), number=self._number, original=self._original_bytes)\n\n        m = packet.mrag\n        if m.magazine in mags and m.row in rows:\n            return packet\n        else:\n            return 'filtered'\n\ndef process_lines(chunks, mode, config, force_cpu=False, prefer_opencl=False, mags=range(9), rows=range(32), tape_format='vhs', eight_bit=False):\n    if mode == 'slice':\n        force_cpu = True\n    Line.configure(config, force_cpu, prefer_opencl, tape_format)\n    for number, chunk in chunks:\n        try:\n            yield getattr(Line(chunk, number), mode)(mags, rows, eight_bit)\n        except Exception:\n            sys.stderr.write(str(number) + '\\n')\n            raise\n"
  },
  {
    "path": "teletext/vbi/pattern.py",
    "content": "# * Copyright 2016 Alistair Buxton <a.j.buxton@gmail.com>\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\n\nimport itertools\nimport struct\nfrom collections import defaultdict\n\nimport numpy as np\n\nfrom tqdm import tqdm\n\n\nclass Pattern(object):\n\n    def __init__(self, filename):\n        with open(filename, 'rb') as f:\n            self.inlen,self.outlen,self.n,self.start,self.end = struct.unpack('>IIIBB', f.read(14))\n            self.patterns = np.fromfile(f, dtype=np.uint8, count=self.inlen*self.n)\n            self.patterns = self.patterns.reshape((self.n, self.inlen))\n            self.patterns = self.patterns.astype(np.float32)\n            self.bytes = np.fromfile(f, dtype=np.uint8, count=self.outlen*self.n)\n            self.bytes = self.bytes.reshape((self.n, self.outlen))\n        self.pslice = self.patterns[:, self.start:self.end]\n\n    def match(self, inp):\n        l = (len(inp)//8)-2\n        idx = np.empty((l,), dtype=np.uint32)\n        for i in range(l):\n            start = (i*8) + self.start\n            end = (i*8) + self.end\n            diffs = self.pslice - inp[start:end]\n            diffs = diffs * diffs\n            idx[i] = np.argmin(np.sum(diffs, axis=1))\n        return self.bytes[idx][:,0]\n\n    def similarities(self):\n\n        def norm(arr):\n            mn = np.nanmin(arr)\n            mx = np.nanmax(arr)\n            print(mn, mx)\n            r = (mx - mn)\n            if r == 0:\n                r = 1\n            return np.clip((arr - mn) * (255.0 / r), 0, 255)\n\n        s = defaultdict(list)\n        for x in tqdm(range(0, self.n)):\n            for y in range(x+1, self.n):\n                d = np.sum(np.square(self.pslice[x] - self.pslice[y]))\n                s[(self.bytes[x][0]&0x7f, self.bytes[y][0]&0x7f)].append(d)\n\n        result = np.full((256, 256, 3), dtype=np.float32, fill_value=float('nan'))\n\n        for k, v in s.items():\n            if v:\n                x, y = sorted(k)\n                result[x, y, 0] = min(v)\n                result[x, y, 1] = sum(v)/len(v)\n                result[x, y, 2] = max(v)\n\n        result = norm(result)\n\n        def get(x, y):\n            x, y = sorted((x, y))\n            return (x, y), result[ord(x), ord(y)].astype(np.uint8), len(s[ord(x), ord(y)])\n\n        errors = []\n        for c, d in itertools.combinations('abcdefghijklmnopqrstuvwxyz', 2):\n            r = get(c, d)\n            if r[1][0] < 5:\n                errors.append(c+d)\n\n        errorsu = []\n        for c, d in itertools.combinations('ABCDEFGHIJKLMNOPQRSTUVWXYZ', 2):\n            r = get(c, d)\n            if r[1][0] < 5:\n                errorsu.append(c+d)\n\n        print(errorsu)\n\n        return errors\n\n# Classes used to build pattern files from training data.\n# Not used during normal decoding.\n\nclass PatternBuilder(object):\n\n    def __init__(self, inwidth):\n        self.patterns = defaultdict(list)\n        self.inwidth = inwidth\n\n    def write_patterns(self, f, start, end):\n        flat_patterns = []\n        for k, v in tqdm(self.patterns.items(), unit='P', desc='Squashing'):\n            pattn = np.mean(np.frombuffer(b''.join(v), dtype=np.uint8).reshape((len(v), self.inwidth)), axis=0).astype(np.uint8)\n            flat_patterns.append((pattn, k[1:2]))\n\n        header = struct.pack('>IIIBB', len(flat_patterns[0][0]), len(flat_patterns[0][1]), len(flat_patterns), start, end)\n        f.write(header)\n\n        for (p,b) in flat_patterns:\n            f.write(p)\n        for (p,b) in flat_patterns:\n            f.write(b)\n\n        f.close()\n\n    def add_pattern(self, key, pattern):\n        self.patterns[key].append(pattern)\n\n\ndef build_pattern(chunks, output, start, end, pattern_set=range(256)):\n\n    #build_pattern(squashed, 'full.dat', 3, 19)\n    #build_pattern(squashed, 'parity.dat', 4, 18, parity_set)\n    #build_pattern(squashed, 'hamming.dat', 1, 20, hamming_set)\n\n    pb = PatternBuilder(24)\n\n    def key(s):\n        pre = s[0]&(0xff<<start)\n        post = s[2]&(0xff>>(24-end))\n        return bytes((pre, line[1], post))\n\n    for n, line in chunks:\n        if line[1] in pattern_set:\n            pb.add_pattern(key(line), line[3:])\n\n    pb.write_patterns(output, start, end)\n"
  },
  {
    "path": "teletext/vbi/patterncuda.py",
    "content": "# * Copyright 2016 Alistair Buxton <a.j.buxton@gmail.com>\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\nimport atexit\n\nimport numpy as np\n\nimport pycuda.driver as cuda\nimport pycuda.gpuarray as gpuarray\nfrom pycuda.compiler import SourceModule\nfrom pycuda.driver import ctx_flags\n\nfrom .pattern import Pattern\n\ncuda.init()\ncudadevice = cuda.Device(0)\ncudacontext = cudadevice.make_context(flags=ctx_flags.SCHED_YIELD)\natexit.register(cudacontext.pop)\n\n\nclass PatternCUDA(Pattern):\n\n    correlate = SourceModule(\"\"\"\n        __global__ void correlate(float *input, float *patterns, float *result, int range_low, int range_high)\n        {\n            int x = (threadIdx.x + (blockDim.x*blockIdx.x));\n            int y = (threadIdx.y + (blockDim.y*blockIdx.y));\n            int iidx = x * 8;\n            int ridx = (x * blockDim.y * gridDim.y) + y;\n            int pidx = y * 24;\n        \n            float d;\n            result[ridx] = 0;\n        \n            for (int i=range_low;i<range_high;i++) {\n                d = input[iidx+i] - patterns[pidx+i];\n                result[ridx] += (d*d);\n            }\n        }            \n    \"\"\").get_function(\"correlate\")\n\n    # argmin from scikit-cuda/blob/master/skcuda/misc.py\n    argmin = SourceModule(\"\"\"\n        /*\n        Copyright (c) 2009-2019, Lev E. Givon. All rights reserved.\n\n        Redistribution and use in source and binary forms, with or without modification, are \n        permitted provided that the following conditions are met:\n\n            Redistributions of source code must retain the above copyright notice, this list of \n            conditions and the following disclaimer.\n            Redistributions in binary form must reproduce the above copyright notice, this list \n            of conditions and the following disclaimer in the documentation and/or other materials \n            provided with the distribution.\n            Neither the name of Lev E. Givon nor the names of any contributors may be used to \n            endorse or promote products derived from this software without specific prior \n            written permission.\n\n        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY \n        EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES \n        OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT \n        SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, \n        SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT \n        OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) \n        HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR \n        TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, \n        EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n        */\n \n        __global__ void minmax_row_kernel(float* mat, float* target,\n                                          unsigned int* idx_target,\n                                          unsigned int width,\n                                          unsigned int height) {\n            __shared__ float max_vals[32];\n            __shared__ unsigned int max_idxs[32];\n            float cur_max = 3.4028235e+38;\n            unsigned int cur_idx = 0;\n            float val = 0;\n    \n            for (unsigned int i = threadIdx.x; i < width; i += 32) {\n                val = mat[blockIdx.x * width + i];\n    \n                if (val < cur_max) {\n                    cur_max = val;\n                    cur_idx = i;\n                }\n            }\n            max_vals[threadIdx.x] = cur_max;\n            max_idxs[threadIdx.x] = cur_idx;\n            __syncthreads();\n    \n            if (threadIdx.x == 0) {\n                cur_max = 3.4028235e+38;\n                cur_idx = 0;\n    \n                for (unsigned int i = 0; i < 32; i++)\n                    if (max_vals[i] < cur_max) {\n                        cur_max = max_vals[i];\n                        cur_idx = max_idxs[i];\n                    }\n    \n                target[blockIdx.x] = cur_max;\n                idx_target[blockIdx.x] = cur_idx;\n            }\n        }\n    \"\"\").get_function(\"minmax_row_kernel\")\n\n    def __init__(self, filename):\n        Pattern.__init__(self, filename)\n\n        if self.n&1023 != 0:\n            raise ValueError('Number of patterns must be a multiple of 1024.')\n\n        self.patterns_gpu = cuda.mem_alloc(self.patterns.nbytes)\n        cuda.memcpy_htod(self.patterns_gpu, self.patterns)\n\n        self.input_match = cuda.mem_alloc(4*((40*8)+16))\n        self.result_match = gpuarray.empty((40,self.n), dtype=np.float32, allocator=cuda.mem_alloc)\n\n        self.result_min = gpuarray.empty((40,), dtype=np.float32, allocator=cuda.mem_alloc)\n        self.result_argmin = gpuarray.empty((40,), dtype=np.uint32, allocator=cuda.mem_alloc)\n\n    def match(self, inp):\n        l = (len(inp)//8)-2\n        x = l & -l # highest power of two which divides l, up to 8\n        y = min(1024//x, self.n)\n        cuda.memcpy_htod(self.input_match, inp.astype(np.float32))\n\n        PatternCUDA.correlate(\n            self.input_match, self.patterns_gpu, self.result_match,\n            np.int32(self.start), np.int32(self.end),\n            block=(x, y, 1), grid=(l//x, self.n//y)\n        )\n\n        PatternCUDA.argmin(\n            self.result_match, self.result_min, self.result_argmin,\n            np.uint32(self.n), np.uint32(l),\n            block=(32, 1, 1), grid=(l, 1, 1), stream=None\n        )\n\n        result = self.result_argmin.get()\n        return self.bytes[result[:l],0]\n\n"
  },
  {
    "path": "teletext/vbi/patternopencl.py",
    "content": "# * Copyright 2023 Dr. David Alan Gilbert <dave@treblig.org>\n# *   based on Alistair's patterncuda.py\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\n\nimport numpy as np\nimport pyopencl as cl\nimport sys\n\nfrom .pattern import Pattern\n\nopenclctx = cl.create_some_context(interactive=False)\n\nclass PatternOpenCL(Pattern):\n    prg = cl.Program(openclctx, \"\"\"\n    __kernel void correlate(global float* restrict input, global float* restrict patterns, global float* restrict result,\n                            int range_low, int range_high)\n    {\n      int x = get_global_id(0);\n      int y = get_global_id(1);\n      int iidx = x * 8;\n      int ridx = (x * get_global_size(1)) + y;\n      int pidx = y * 24;\n      float d;\n\n      input+=iidx+range_low;\n      patterns+=pidx+range_low;\n      range_high-=range_low;\n\n      int i=0;\n      float res = 0;\n      float4 vd;\n      for (;(i+3)<range_high;i+=4) {\n        vd = vload4(0, input+i) - vload4(0, patterns+i);\n        res += dot(vd, vd);\n      }\n\n      for (;i<range_high;i++) {\n        d = input[i] - patterns[i];\n        res += (d*d);\n      }\n      result[ridx] = res;\n    }\n\n    // Each workitem takes one character x npatterns/minpar values\n    // and finds the minimum, writing one value and index into the\n    // temporaries\n    // The temporaries are 40 characters wide\n    // Done as a 2D parallel, X is character,\n    // Y is npatterns/minpar chunk of correlate results\n    __kernel void minerr1(global float* restrict input,\n                         global float* restrict tmp_val, global int* restrict tmp_idx,\n                         int npatterns, int minpar)\n    {\n      int ch = get_global_id(0);\n      int width = get_global_size(0);\n      int patblock = get_global_id(1);\n      int patstep = npatterns / minpar;\n      int patstart = patblock * patstep;\n      int patend = patstart + patstep;\n\n      int inindex = patstart + npatterns*ch;\n      int bestidx = patstart;\n      float bestval = input[inindex];\n      float4 vv;\n\n      for (int p=patstart; (p+3) < patend; p+=4, inindex+=4) {\n        vv = vload4(0, input+inindex);\n        if (any(vv < bestval)) {\n          // Someone is negative, figure out who\n          if (vv.s0 < bestval) {\n            bestidx = p;\n            bestval = vv.s0;\n          }\n          if (vv.s1 < bestval) {\n            bestidx = p+1;\n            bestval = vv.s1;\n          }\n          if (vv.s2 < bestval) {\n            bestidx = p+2;\n            bestval = vv.s2;\n          }\n          if (vv.s3 < bestval) {\n            bestidx = p+3;\n            bestval = vv.s3;\n          }\n        }\n      }\n\n      int tidx = patblock*width + ch;\n      tmp_idx[tidx] = bestidx;\n      tmp_val[tidx] = bestval;\n    }\n\n    // Each workitem takes one character x minpar values and finds the\n    // minimum of the temporary minima and writes the index\n    // Done as a 1D parallel over the characters\n    __kernel void minerr2(global float* restrict tmp_val, global int* restrict tmp_idx,\n                          global int* restrict indexes,\n                          int minpar)\n    {\n      int ch = get_global_id(0);\n      int width = get_global_size(0);\n\n      int iidx = ch;\n      int bestidx = tmp_idx[iidx];\n      float bestval = tmp_val[iidx];\n      float val;\n\n      int i = 0;\n\n      for (;(i+3)<minpar;i+=4,iidx+=4*width) {\n        float4 vv = (float4)(tmp_val[iidx+0*width], tmp_val[iidx+1*width], tmp_val[iidx+2*width], tmp_val[iidx+3*width]);\n        if (any(vv < bestval)) {\n          // Someone is negative, figure out who\n          if (vv.s0 < bestval) {\n            bestidx = tmp_idx[iidx];\n            bestval = vv.s0;\n          }\n          if (vv.s1 < bestval) {\n            bestidx = tmp_idx[iidx+width];\n            bestval = vv.s1;\n          }\n          if (vv.s2 < bestval) {\n            bestidx = tmp_idx[iidx+2*width];\n            bestval = vv.s2;\n          }\n          if (vv.s3 < bestval) {\n            bestidx = tmp_idx[iidx+3*width];\n            bestval = vv.s3;\n          }\n        }\n      }\n\n      indexes[ch] = bestidx;\n    }\n    \"\"\").build()\n\n    def __init__(self, filename):\n        Pattern.__init__(self, filename)\n\n        self.profile = 0\n\n        if self.profile:\n          self.queue = cl.CommandQueue(openclctx, properties = cl.command_queue_properties.PROFILING_ENABLE)\n        else:\n          self.queue = cl.CommandQueue(openclctx)\n\n        mf = cl.mem_flags\n\n        self.kernel_correlate = self.prg.correlate\n        self.kernel_min1 = self.prg.minerr1\n        self.kernel_min2 = self.prg.minerr2\n\n        # patterns is already float32 (see Pattern __init__)\n        self.patterns_gpu = cl.Buffer(openclctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=self.patterns)\n\n        # the input of the correlate -\n        # size copying CUDA code, something like 40chars, 8 bits each, float??\n        self.input_match = cl.Buffer(openclctx, mf.READ_WRITE, 4*((40*8)+16))\n\n        # output of the correlate\n        self.result_match = cl.Buffer(openclctx, mf.HOST_NO_ACCESS, 4*40*self.n)\n\n        # How much to split the min search by vertically\n        # DANGER: Heuristic, probably varies by hardware, possibly want to\n        # vary on len(inp) as well and opencl hardware config\n        if self.n > 32768:\n            self.minpar = 1024\n        else:\n            self.minpar = 512\n\n        # Temporaries used during parallel min (value and index)\n        self.mintmp_val = cl.Buffer(openclctx, mf.HOST_NO_ACCESS, 4*40*self.minpar)\n        self.mintmp_idx = cl.Buffer(openclctx, mf.HOST_NO_ACCESS, 4*40*self.minpar)\n\n        # output of the min pass - an integer index to which pattern was best\n        # for each character\n        self.result_minidx = cl.Buffer(openclctx, mf.WRITE_ONLY, 4*40)\n        # and a copy of that for np\n        self.result_minidx_np = np.zeros(40, dtype=np.uint32)\n\n    def match(self, inp):\n        l = (len(inp)//8)-2\n        x = l & -l # highest power of two which divides l, up to 8\n        y = min(1024//x, self.n)\n\n        # copy data in\n        e_copy = cl.enqueue_copy(self.queue, self.input_match, inp.astype(np.float32), is_blocking = False)\n        # call corellate\n        # Output is one row per character, with one value per pattern\n        self.kernel_correlate.set_args(self.input_match, self.patterns_gpu, self.result_match,\n                                       np.int32(self.start), np.int32(self.end))\n\n        e_corr = cl.enqueue_nd_range_kernel(self.queue, self.kernel_correlate,\n                                            (l, self.n), None,\n                                            wait_for = (e_copy,))\n\n        # Run min pass 1\n        # squashes the set of patterns down into minpar minima\n        assert (self.n % self.minpar) == 0\n\n        self.kernel_min1.set_args(self.result_match,\n                                  self.mintmp_val, self.mintmp_idx,\n                                  np.int32(self.n), np.int32(self.minpar))\n\n        e_min1 = cl.enqueue_nd_range_kernel(self.queue, self.kernel_min1,\n                                            (l,self.minpar), None,\n                                            wait_for = (e_corr,))\n\n        # Run min pass 2\n        # squashes the temporaries down to a final minimum index for each char\n        self.kernel_min2.set_args(self.mintmp_val, self.mintmp_idx,\n                                  self.result_minidx,\n                                  np.int32(self.minpar))\n\n        e_min2 = cl.enqueue_nd_range_kernel(self.queue, self.kernel_min2,\n                                            (l,), None,\n                                            wait_for = (e_min1,))\n\n\n        # and get the index values back from OpenCL\n        e_out = cl.enqueue_copy(self.queue, self.result_minidx_np, self.result_minidx, wait_for = (e_min2,))\n        e_out.wait()\n\n        if self.profile:\n          print('s/e: {}/{} n: {} len: {}  / total: {} Copy: {} correlate: {} min1: {} min2: {} copy-out: {}'.format(\n              self.start, self.end,\n              self.n, len(inp),\n              e_out.profile.end-e_copy.profile.start,\n              e_copy.profile.end-e_copy.profile.start,\n              e_corr.profile.end-e_corr.profile.start,\n              e_min1.profile.end-e_min1.profile.start,\n              e_min2.profile.end-e_min2.profile.start,\n              e_out.profile.end-e_out.profile.start),\n\n              file=sys.stderr)\n        return self.bytes[self.result_minidx_np[:l],0]\n\n"
  },
  {
    "path": "teletext/vbi/training.py",
    "content": "# * Copyright 2016 Alistair Buxton <a.j.buxton@gmail.com>\n# *\n# * License: This program is free software; you can redistribute it and/or\n# * modify it under the terms of the GNU General Public License as published\n# * by the Free Software Foundation; either version 3 of the License, or (at\n# * your option) any later version. This program is distributed in the hope\n# * that it will be useful, but WITHOUT ANY WARRANTY; without even the implied\n# * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# * GNU General Public License for more details.\n\n\nimport os\nimport itertools\nimport sys\n\nimport numpy as np\n\nfrom tqdm import tqdm\n\nfrom teletext.file import FileChunker\nfrom teletext.coding import parity_encode, hamming8_enc as hamming_set\nfrom teletext.vbi.line import Line, normalise\n\nfrom .pattern import build_pattern\n\n\nclass PatternGenerator(object):\n    pattern = None\n\n    # pattern_length is the number of bytes in the teletext data available for patterns.\n    pattern_length = 27\n\n    def __init__(self):\n        if self.pattern is None:\n            self.load_pattern()\n\n    @classmethod\n    def load_pattern(cls):\n        with open(os.path.join(os.path.dirname(__file__), 'data', 'debruijn.dat'), 'rb') as db:\n            data = db.read()\n        cls.pattern = np.frombuffer(data + data[:cls.pattern_length], dtype=np.uint8)\n\n    def checksum(self, array):\n        return array[0] ^ array[1] ^ array[2] ^ 0xf0\n\n    def generate_line(self, offset):\n        line = np.zeros((42,), dtype=np.uint8)\n\n        # constant bytes. can be used for horizontal alignment.\n        line[0] = 0x18\n        line[1 + self.pattern_length] = 0x18\n        line[41] = 0x18\n\n        # insert pattern slice into line\n        line[1:1 + self.pattern_length] = self.pattern[offset:offset + self.pattern_length]\n\n        # encode the offset for maximum readability\n        offset_list = [(offset >> n) & 0xff for n in range(0, 24, 8)]\n        # add a checksum\n        offset_list.append(self.checksum(offset_list))\n        # convert to a list of bits, LSB first\n        offset_arr = np.array(offset_list, dtype=np.uint8)\n        # repeat each bit 3 times, then convert back in to t42 bytes\n        offset_arr = np.packbits(np.repeat(np.unpackbits(offset_arr[::-1])[::-1], 3)[::-1])[::-1]\n\n        # insert encoded offset into line\n        line[2 + self.pattern_length:14 + self.pattern_length] = offset_arr\n\n        return line\n\n    def to_file(self, file):\n        offset = 0\n        while True:\n            line = self.generate_line(offset)\n\n            # calculate next offset for maximum distance\n            offset += 65521  # greatest prime less than 2097152/32\n            offset &= 0x1fffff  # mod 2097152\n\n            # write to stdout\n            file.write(line.tobytes())\n\n\ndef de_bruijn(k, n):\n    a = [0] * k * n\n    sequence = []\n\n    def db(t, p):\n        if t > n:\n            if n % p == 0:\n                sequence.extend(a[1:p + 1])\n        else:\n            a[t] = a[t - p]\n            db(t + 1, p)\n            for j in range(a[t - p] + 1, k):\n                a[t] = j\n\n                db(t + 1, t)\n\n    db(1, 1)\n\n    return sequence\n\n\ndef save_pattern(filename):\n    pattern = np.packbits(np.array(de_bruijn(2, 24), dtype=np.uint8)[::-1])[::-1]\n    with open(filename, 'wb') as data:\n        pattern.tofile(data)\n\n\nclass TrainingLine(Line):\n\n    pgen = PatternGenerator()\n\n    def tchop(self, start, stop):\n        s = np.sum(self.chop(256+(start*24), 256+(stop*24)).reshape(-1, 3), 1)\n        s = (s > 384).astype(np.uint8)\n        return np.packbits(s[::-1])[::-1]\n\n    def lock(self, offset):\n        orig = np.empty((45*8), dtype=np.float)\n        orig[:24] = self.config.crifc * 255\n        orig[24:] = np.unpackbits(self.pgen.generate_line(offset)[::-1])[::-1] * 255\n        x = []\n        for roll in range(-10, 10):\n            self.roll = roll\n            t = np.sum(np.square(self.chop(0, 360)-orig))\n            x.append((t, roll))\n\n        roll = min(x)[1]\n        self.roll = 0\n        self._start += roll\n        #print(roll)\n\n\n    @property\n    def checksum(self):\n        return self.tchop(3, 4)[0]\n\n    @property\n    def offset(self):\n        for roll in range(-8, 8):\n            self.roll = roll\n            bytes = self.tchop(0, 3)\n            if self.pgen.checksum(bytes) == self.checksum:\n                offset = bytes[0] | (bytes[1] << 8) | (bytes[2] << 16)\n                if offset < 0x1fffff:\n                    self._start += roll\n                    self.roll = 0\n                    self.lock(offset)\n                    return offset\n        #sys.stderr.write(f'Warning: bad line {self._number}\\n')\n\n\ndef process_training(chunks, config):\n    TrainingLine.configure(config, force_cpu=True)\n    lines = (TrainingLine(chunk, n) for n, chunk in chunks)\n\n    for l in lines:\n        if l.is_teletext:\n            offset = l.offset\n            if offset is not None:\n                yield (offset, normalise(l.chop(32, 32+(8*TrainingLine.pgen.pattern_length))).astype(np.uint8))\n                continue\n        yield 'rejected'\n\n\ndef process_crifc(chunks, config):\n    TrainingLine.configure(config, force_cpu=True)\n    lines = (TrainingLine(chunk, n) for n, chunk in chunks)\n\n    n = 1000\n    crifc = np.empty((n, 192), dtype=np.float)\n\n    for l in lines:\n        if l.is_teletext:\n            offset = l.offset\n            if offset is not None:\n                crifc[n-1] = l.fchop(0, 24)\n                n -= 1\n                if n == 0:\n                    break\n\n    mean = np.mean(crifc, 0).reshape(-1, 8)\n    print(repr(mean.astype(np.uint8)))\n\ndef split(data, files):\n    pgen = PatternGenerator()\n\n    chopped_indexer = np.arange(24)[None, :] + np.arange((8 * pgen.pattern_length) - 23)[:, None]\n    pattern_indexer = chopped_indexer[::-1,:]\n\n    for offset, chopped in data:\n        # Fetch the pattern block corresponding to this line.\n        block = np.unpackbits(pgen.pattern[offset:offset + pgen.pattern_length][::-1])\n        # Sliding window through the pattern block.\n        patterns = np.packbits(block[pattern_indexer], axis=1)[:, ::-1]\n        # Sliding window through the chopped line.\n        choppeds = chopped[chopped_indexer]\n        # Append chopped samples to pattern bytes.\n        result = np.append(patterns, choppeds, axis=1)\n        for p in result:\n            files[p[0]].write(p.tobytes())\n\n\ndef squash(output, indir):\n    for n in tqdm(range(256), unit='File'):\n        with open(os.path.join(indir, f'training.{n:02x}.dat'), 'rb') as f:\n            chunks = FileChunker(f, 27)\n            chunks = sorted(chunk for n, chunk in chunks)\n            for k, g in itertools.groupby(chunks, lambda x: x[:3]):\n                a = list(g)\n                b = np.frombuffer(b''.join(a), dtype=np.uint8).reshape((len(a), 27))\n                b = np.mean(b, axis=0).astype(np.uint8)\n                output.write(b.tobytes())\n"
  },
  {
    "path": "teletext/vbi/viewer.py",
    "content": "import time\n\nimport numpy as np\nfrom itertools import islice\n\nfrom OpenGL.GLUT import *\nfrom OpenGL.GL import *\n\n\nclass VBIViewer(object):\n\n    def __init__(self, lines, config, name = \"VBI Viewer\", width=800, height=512, nlines=32, tint=True, show_grid=True, show_slices=False, pause=False):\n        self.config = config\n        self.show_grid = show_grid\n        self.tint = tint\n        self.pause = pause\n        self.single_step = False\n        self.name = name\n\n        self.line_attr = 'resampled'\n\n        if nlines is None:\n            self.nlines = 32\n        else:\n            self.nlines = nlines\n\n        self.lines_src = lines\n        self.lines = list(islice(self.lines_src, 0, self.nlines))\n\n        glutInit(sys.argv)\n        glutInitDisplayMode(GLUT_SINGLE | GLUT_RGB)\n        glutInitWindowSize(width,height)\n        glutCreateWindow(name)\n        self.set_title()\n\n        glutDisplayFunc(self.display)\n        glutReshapeFunc(self.reshape)\n        glutKeyboardFunc(self.keyboard)\n        glutMouseFunc(self.mouse)\n\n        glMatrixMode(GL_PROJECTION)\n        glOrtho(0, config.resample_size, 0, self.nlines, -1, 1)\n        glMatrixMode(GL_MODELVIEW)\n        glLoadIdentity()\n\n        glDisable(GL_LIGHTING)\n        glDisable(GL_DEPTH_TEST)\n        glEnable(GL_BLEND)\n        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)\n        glEnable(GL_TEXTURE_2D)\n        glBindTexture(GL_TEXTURE_2D, glGenTextures(1))\n        glPixelStorei(GL_UNPACK_ALIGNMENT,1)\n\n        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)\n        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)\n        glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE)\n\n        glutMainLoop()\n\n    def reshape(self, width, height):\n        self.width = width\n        self.height = height\n        glViewport(0, 0, width, height)\n\n    def keyboard(self, key, x, y):\n        if key == b'g':\n            self.show_grid ^= True\n        elif key == b'c':\n            self.tint ^= True\n        elif key == b'p':\n            self.pause ^= True\n        elif key == b'n':\n            self.single_step = True\n        elif key == b'r':\n            self.dumpline(x, y, teletext=False)\n        elif key == b't':\n            self.dumpline(x, y, teletext=True)\n        elif key == b'R':\n            self.dumpall(teletext=False)\n        elif key == b'T':\n            self.dumpall(teletext=True)\n        elif key == b'1':\n            self.line_attr = 'resampled'\n        elif key == b'2':\n            self.line_attr = 'fft'\n        elif key == b'3':\n            self.line_attr = 'rolled'\n        elif key == b'4':\n            self.line_attr = 'gradient'\n        elif key == b'q':\n            exit(0)\n        self.set_title()\n\n    def mouse(self, button, state, x, y):\n        if state == GLUT_DOWN:\n            l = self.lines[self.nlines * y//self.height]\n            if button == 3:\n                l.roll += 1\n            elif button == 4:\n                l.roll -= 1\n            if l.is_teletext:\n                print(l.deconvolve().debug.decode('utf8')[:-1], 'er:', l.roll, l._reason)\n            else:\n                print(l._reason)\n            a = np.frombuffer(l._original_bytes, dtype=np.uint8)\n            d = np.diff(a.astype(np.int16))\n            md = np.mean(np.abs(d))\n            steps = np.floor(np.linspace(0, 2048 - 5, num=11)).astype(np.uint32)[[1, 5, 9]]\n            s = np.sort(a)\n            print(md, s[steps])\n            sys.stdout.flush()\n\n    def dumpline(self, x, y, teletext):\n        if teletext:\n            print('Writing to teletext.vbi')\n            fn = 'teletext.vbi'\n        else:\n            print('Writing to reject.vbi')\n            fn = 'reject.vbi'\n        l = self.lines[self.nlines * y // self.height]\n        with open(fn, 'ab') as f:\n            f.write(l._original_bytes)\n\n    def dumpall(self, teletext):\n        if teletext:\n            print('Writing all to teletext.vbi')\n            fn = 'teletext.vbi'\n        else:\n            print('Writing all to reject.vbi')\n            fn = 'reject.vbi'\n        with open(fn, 'ab') as f:\n            for l in self.lines:\n                f.write(l._original_bytes)\n\n    def set_title(self):\n        glutSetWindowTitle(f'{self.name} - {self.line_attr}{\" (paused)\" if self.pause else \"\"}')\n\n    def draw_slice(self, slice, r, g, b, a=1.0):\n        glColor4f(r, g, b, a)\n        glBegin(GL_LINES)\n        glVertex2f(slice.start, 0)\n        glVertex2f(slice.start, self.nlines)\n        glVertex2f(slice.stop, 0)\n        glVertex2f(slice.stop, self.nlines)\n        glEnd()\n\n    def draw_h_grid(self, r, g, b, a=1.0):\n        glColor4f(r, g, b, a)\n        glBegin(GL_LINES)\n        for x in range(self.nlines):\n            glVertex2f(0, x)\n            glVertex2f(self.config.resample_size, x)\n        glEnd()\n\n    def draw_bits(self, r, g, b, a=1.0):\n        glColor4f(r, g, b, a)\n        glBegin(GL_LINES)\n        for x in range(0, 368,8):\n            glVertex2f((x*8)+90, 0)\n            glVertex2f((x*8)+90, self.nlines)\n        glEnd()\n\n    def draw_freq_bins(self, n, r, g, b, a=1.0):\n        glColor4f(r, g, b, a)\n        glBegin(GL_LINES)\n        for x in self.config.fftbins:\n            glVertex2f(self.config.resample_size*x/256, 0)\n            glVertex2f(self.config.resample_size*x/256, self.nlines)\n        glEnd()\n\n    def draw_lines(self):\n\n        glEnable(GL_TEXTURE_2D)\n        for n,l in enumerate(self.lines[::-1]):\n            array = getattr(l, self.line_attr)\n            glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, array.size, 1, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, np.clip(array, 0, 255).astype(np.uint8).tostring())\n            if self.tint:\n                if l.is_teletext:\n                    glColor4f(0.5, 1.0, 0.7, 1.0)\n                else:\n                    glColor4f(1.0, 0.5, 0.5, 1.0)\n            else:\n                glColor4f(1.0, 1.0, 1.0, 1.0)\n\n            glBegin(GL_QUADS)\n\n            glTexCoord2f(0, 1)\n            glVertex2f(0, n)\n\n            glTexCoord2f(0, 0)\n            glVertex2f(0, (n+1))\n\n            glTexCoord2f(1, 0)\n            glVertex2f(self.config.resample_size, (n+1))\n\n            glTexCoord2f(1, 1)\n            glVertex2f(self.config.resample_size, n)\n\n            glEnd()\n\n        glDisable(GL_TEXTURE_2D)\n\n    def display(self):\n\n        self.draw_lines()\n\n        if self.height / self.nlines > 3:\n            self.draw_h_grid(0, 0, 0, 0.25)\n\n        if self.show_grid:\n            if self.line_attr == 'fft':\n                self.draw_freq_bins(256, 1, 1, 1, 0.5)\n            elif self.line_attr == 'rolled' and self.width / 42 > 5:\n                self.draw_bits(1, 1, 1, 0.5)\n            elif self.line_attr == 'resampled':\n                self.draw_slice(self.config.start_slice, 0, 1, 0, 0.5)\n\n        glutSwapBuffers()\n        glutPostRedisplay()\n\n        if self.pause and not self.single_step:\n            time.sleep(0.1)\n        else:\n            next_lines = list(islice(self.lines_src, 0, self.nlines))\n\n            if len(next_lines) > 0:\n                self.lines = next_lines\n            self.single_step = False\n"
  },
  {
    "path": "tests/__init__.py",
    "content": ""
  },
  {
    "path": "tests/test_cli.py",
    "content": "import unittest\n\nfrom click.testing import CliRunner\n\nimport teletext.cli.teletext\nimport teletext.cli.training\n\n\nclass TestCommandTeletext(unittest.TestCase):\n    cmd = teletext.cli.teletext.teletext\n\n    def setUp(self):\n        self.runner = CliRunner()\n\n    def test_help(self):\n        result = self.runner.invoke(self.cmd, ['--help'])\n        self.assertEqual(result.exit_code, 0)\n\n\nclass TestCmdFilter(TestCommandTeletext):\n    cmd = teletext.cli.teletext.filter\n\n\nclass TestCmdDiff(TestCommandTeletext):\n    cmd = teletext.cli.teletext.diff\n\n\nclass TestCmdFinders(TestCommandTeletext):\n    cmd = teletext.cli.teletext.finders\n\n\nclass TestCmdSquash(TestCommandTeletext):\n    cmd = teletext.cli.teletext.squash\n\n\nclass TestCmdSpellcheck(TestCommandTeletext):\n    cmd = teletext.cli.teletext.spellcheck\n\n\nclass TestCmdService(TestCommandTeletext):\n    cmd = teletext.cli.teletext.service\n\n\nclass TestCmdInteractive(TestCommandTeletext):\n    cmd = teletext.cli.teletext.interactive\n\n\nclass TestCmdUrls(TestCommandTeletext):\n    cmd = teletext.cli.teletext.urls\n\n\nclass TestCmdHtml(TestCommandTeletext):\n    cmd = teletext.cli.teletext.html\n\n\nclass TestCmdRecord(TestCommandTeletext):\n    cmd = teletext.cli.teletext.record\n\n\nclass TestCmdVBIView(TestCommandTeletext):\n    cmd = teletext.cli.teletext.vbiview\n\n\nclass TestCmdDeconvolve(TestCommandTeletext):\n    cmd = teletext.cli.teletext.deconvolve\n\n\nclass TestCmdTraining(TestCommandTeletext):\n    cmd = teletext.cli.training.training\n\n\nclass TestCmdGenerate(TestCommandTeletext):\n    cmd = teletext.cli.training.generate\n\n\nclass TestCmdTrainingSquash(TestCommandTeletext):\n    cmd = teletext.cli.training.training_squash\n\n\nclass TestCmdShowBin(TestCommandTeletext):\n    cmd = teletext.cli.training.showbin\n\n\nclass TestCmdBuild(TestCommandTeletext):\n    cmd = teletext.cli.training.build\n"
  },
  {
    "path": "tests/test_coding.py",
    "content": "import unittest\n\nimport numpy as np\n\nfrom teletext.coding import parity_encode, parity_decode, parity_errors, hamming8_encode, hamming8_decode, \\\n    hamming8_correctable_errors, hamming8_uncorrectable_errors, byte_reverse, crc\n\n\nclass ParityEncodeTestCase(unittest.TestCase):\n\n    def _test_array(self, array: np.ndarray):\n\n        encoded = parity_encode(array)\n        self.assertEqual(encoded.dtype, int)\n        self.assertEqual(array.shape, encoded.shape, 'Encoded data has wrong shape')\n\n        #bitcounts = np.sum(np.unpackbits(encoded, axis=1), axis=1)\n        #self.assertTrue(all(bitcounts & 1), 'Encoded data has wrong parity.')\n\n        errors = parity_errors(encoded)\n        self.assertFalse(any(errors), 'Encoded data has false errors.')\n\n        decoded = parity_decode(encoded)\n        self.assertEqual(decoded.dtype, int)\n        self.assertTrue(all(decoded == array), 'Decoded data does not match original.')\n\n        for b in range(8):\n            oneerr = encoded ^ (1 << b)\n            errors = parity_errors(oneerr)\n            self.assertTrue(all(errors), 'Error not detected in encoded data.')\n\n    def test_full_array(self):\n        self._test_array(array=np.array(range(0x80), dtype=np.uint8))\n\n    def test_unit_arrays(self):\n        for i in range(0x80):\n            self._test_array(array=np.array([i], dtype=np.uint8))\n\n    def test_array_type(self):\n        encoded = parity_encode(np.array(range(0x80), dtype=np.uint8))\n        self.assertIsInstance(encoded, np.ndarray)\n        #self.assertEqual(encoded.dtype, np.int)\n\n    def test_list_type(self):\n        encoded = parity_encode(list(range(0x80)))\n        self.assertIsInstance(encoded, np.ndarray)\n        #self.assertEqual(encoded.dtype, np.int)\n\n    def test_unit_array_type(self):\n        encoded = parity_encode(np.array([0], dtype=np.uint8))\n        self.assertIsInstance(encoded, np.ndarray)\n        #self.assertEqual(encoded.dtype, np.int)\n\n    def test_unit_list_type(self):\n        encoded = parity_encode([0])\n        self.assertIsInstance(encoded, np.ndarray)\n        #self.assertEqual(encoded.dtype, np.int)\n\n    def test_int_type(self):\n        encoded = parity_encode(0)\n        #self.assertIsInstance(encoded, np.int64)\n\n    def test_full_list(self):\n        data = list(range(0x80))\n        encoded = parity_encode(data)\n        self.assertEqual(encoded.shape, (len(data),), 'Encoded data has wrong shape')\n\n    def test_unit_list(self):\n        data = [0]\n        encoded = parity_encode(data)\n        self.assertEqual(encoded.shape, (1, ), 'Encoded data has wrong shape')\n\n    def test_ints(self):\n        for i in range(0x80):\n            encoded = parity_encode(i)\n\n            errors = parity_errors(encoded)\n            self.assertFalse(errors, 'Encoded data has false errors.')\n\n            decoded = parity_decode(encoded)\n            self.assertEqual(decoded, i, 'Decoded data does not match original.')\n\n            for b in range(8):\n                oneerr = encoded ^ (1 << b)\n                errors = parity_errors(oneerr)\n                self.assertTrue(errors, 'Error not detected in encoded data.')\n\n\nclass Hamming8TestCase(unittest.TestCase):\n\n    def test_all(self):\n\n        def h8_manual(d):\n            d1 = d & 1\n            d2 = (d >> 1) & 1\n            d3 = (d >> 2) & 1\n            d4 = (d >> 3) & 1\n\n            p1 = (1 + d1 + d3 + d4) & 1\n            p2 = (1 + d1 + d2 + d4) & 1\n            p3 = (1 + d1 + d2 + d3) & 1\n            p4 = (1 + p1 + d1 + p2 + d2 + p3 + d3 + d4) & 1\n\n            return (p1 | (d1 << 1) | (p2 << 2) | (d2 << 3)\n                    | (p3 << 4) | (d3 << 5) | (p4 << 6) | (d4 << 7))\n\n        for i in range(0x10):\n            self.assertTrue(hamming8_encode(i) == h8_manual(i))\n\n        data = np.arange(0x10, dtype=np.uint8)\n        encoded = hamming8_encode(data)\n        self.assertTrue(all(hamming8_decode(encoded) == data))\n        self.assertTrue(not any(hamming8_correctable_errors(encoded)))\n        self.assertTrue(not any(hamming8_uncorrectable_errors(encoded)))\n\n        for b1 in range(8):\n            oneerr = encoded ^ (1 << b1)\n            self.assertTrue(all(hamming8_decode(oneerr) == data))\n            self.assertTrue(all(hamming8_correctable_errors(oneerr)))\n            self.assertTrue(not any(hamming8_uncorrectable_errors(oneerr)))\n            for b2 in range(8):\n                if b2 != b1:\n                    twoerr = oneerr ^ (1 << b2)\n                    self.assertTrue(not any(hamming8_correctable_errors(twoerr)))\n                    self.assertTrue(all(hamming8_uncorrectable_errors(twoerr)))\n\n\nclass Reverse8TestCase(unittest.TestCase):\n\n    def test_all(self):\n\n        for i in range(256):\n            reversed = 0\n            for n in range(8):\n                reversed |= ((i>>n)&1) << (7-n)\n            self.assertEqual(byte_reverse(i), reversed)\n\n\nclass CRCTestCase(unittest.TestCase):\n\n    def test_all(self):\n        self.assertEqual(crc(0, 0), 0)\n        self.assertEqual(crc(0x55, 0xaa), 0xaa5e)\n"
  },
  {
    "path": "tests/test_elements.py",
    "content": "import itertools\nimport unittest\n\nfrom teletext.elements import *\n\n\n\nclass TestElement(unittest.TestCase):\n\n    cls = Element\n    shape = (2,)\n    sized = False\n    needsmrag = False\n\n    def make_element(self, array):\n        args = []\n        if not self.sized:\n            args.append(self._array.shape)\n        args.append(array)\n        if self.needsmrag:\n            mrag = Mrag()\n            mrag.magazine = 1\n            mrag.row = 0\n            args.append(mrag)\n        return self.cls(*args)\n\n    def setUp(self):\n        self._array = np.zeros(self.shape, dtype=np.uint8)\n        self.element = self.make_element(self._array)\n\n    def test_type(self):\n        self.assertIsInstance(self.element, self.cls)\n\n    def test_wrong_shape(self):\n        with self.assertRaises(IndexError):\n            array = np.zeros((1,1), dtype=np.uint8)\n            self.make_element(array)\n\n    def test_getitem(self):\n        for i, j in itertools.product(range(self.shape[0]), range(256)):\n            self._array[i] = j\n            self.assertEqual(self.element[i], j)\n            self.assertEqual(self.element[i], self._array[i])\n\n    def test_setitem(self):\n        for i, j in itertools.product(range(self.shape[0]), range(256)):\n            self.element[i] = j\n            self.assertEqual(self.element[i], j)\n            self.assertEqual(self.element[i], self._array[i])\n\n    def test_repr(self):\n        self.assertEqual(repr(self.element), f'{self.cls.__name__}({repr(self._array)})')\n\n    def test_errors(self):\n        with self.assertRaises(NotImplementedError):\n            self.element.errors()\n\n    def test_bytes(self):\n        self.assertEqual(self._array.tobytes(), self.element.bytes)\n\n\nclass TestElementParity(TestElement):\n\n    cls = ElementParity\n    shape = (2, )\n\n    def test_errors(self):\n        self.assertTrue(all(self.element.errors))\n        self._array[:] = 0x20 # correct parity\n        self.assertFalse(any(self.element.errors))\n\n\nclass TestElementHamming(TestElementParity):\n\n    cls = ElementHamming\n    shape = (2, )\n\n    def test_errors(self):\n        self.assertTrue(all(self.element.errors))\n        self._array[:] = 0x15  # correct hamming8\n        self.assertFalse(any(self.element.errors))\n\n\nclass TestMrag(TestElementHamming):\n\n    cls = Mrag\n    shape = (2, )\n    sized = True\n\n    def test_magazine(self):\n        for i in range(1, 9):\n            self._array[:] = 0\n            self.element.magazine = i\n            self.assertEqual(self.element.magazine, i)\n            self.assertTrue(any(self._array))\n\n    def test_row(self):\n        for i in range(32):\n            self._array[:] = 0\n            self.element.row = i\n            self.assertEqual(self.element.row, i)\n            self.assertTrue(any(self._array))\n\n\nclass TestDisplayable(TestElementParity):\n\n    cls = Displayable\n    shape = (11, )\n\n    def test_place_string(self):\n        self.element.place_string('Hello World', x=0)\n        self.assertFalse(any(self.element.errors))\n\n\nclass TestPage(TestElementHamming):\n\n    cls = Page\n    shape = (2,)\n\n\nclass TestHeader(TestPage):\n\n    cls = Header\n    shape = (40,)\n    sized = True\n\n\nclass TestPageLink(TestPage):\n\n    cls = PageLink\n    shape = (6,)\n    sized = True\n    needsmrag = True\n\n\nclass TestDesignationCode(TestElementHamming):\n\n    cls = DesignationCode\n    shape = (1, )\n\n    def test_set_dc(self):\n        for i in range(16):\n            self.element.dc = i\n            self.assertEqual(self.element.dc, i)\n\n\nclass TestFastext(TestDesignationCode):\n\n    cls = Fastext\n    shape = (40,)\n    sized = True\n    needsmrag = True\n\n    @unittest.skip(\"Not implemented yet.\")\n    def test_errors(self):\n        pass # TODO\n"
  },
  {
    "path": "tests/test_file.py",
    "content": "import io\nimport unittest\n\nimport numpy as np\n\nfrom teletext.file import FileChunker\n\nclass TestChunker(unittest.TestCase):\n    def setUp(self):\n        self.data = np.arange(0, 256, dtype=np.uint8)\n        self.file = io.BytesIO(self.data.tobytes())\n\n    def test_basic(self):\n        result = list(FileChunker(self.file, 1))\n        self.assertEqual(len(result), len(self.data))\n        for n in range(256):\n            self.assertEqual(result[n], (n, bytes([n])))\n\n    def test_step(self):\n        result = list(FileChunker(self.file, 1, step=2))\n        self.assertEqual(len(result), len(self.data[::2]))\n        for n in range(128):\n            self.assertEqual(result[n], (n*2, bytes([n*2])))\n"
  },
  {
    "path": "tests/test_mp.py",
    "content": "from multiprocessing import current_process\nimport unittest\nfrom functools import wraps\nfrom itertools import count, islice\nimport os\nimport sys\nimport time\n\nfrom teletext.mp import itermap, PureGeneratorPool, _PureGeneratorPoolSingle, _PureGeneratorPoolMP\n\nfrom .test_sigint import ctrl_c\n\n\ndef multiply(it, a):\n    for x in it:\n        yield x*a\n\n\ndef null(it, a):\n    for x in it:\n        yield (x, a)\n\n\ncallcounter = 0\ndef callcount(it):\n    global callcounter\n    callcounter += 1\n    for x in it:\n        yield callcounter\n\n\ndef crashy(it):\n    for x in it:\n        if x:\n            raise ValueError('Crashed on purpose.')\n        else:\n            yield x\n\n\ndef early_crash(it):\n    raise ValueError('Crashed early on purpose.')\n\n\ndef not_generator(it):\n    return 23\n\n\nclass TestMPSingle(unittest.TestCase):\n    procs = 1\n    desired_type = _PureGeneratorPoolSingle\n\n    def setUp(self):\n        global callcounter\n        callcounter = 0\n\n    def test_single(self):\n        input = list(range(100))\n        expected = list(multiply(input, 3))\n        result = list(itermap(multiply, input, self.procs, 3))\n        self.assertListEqual(result, expected)\n\n    def test_called_once_single(self):\n        result = list(itermap(callcount, [None] * (self.procs + 1), processes=self.procs))\n        self.assertListEqual(result, [1] * (self.procs + 1))\n\n    def test_reuse(self):\n        input = list(range(100))\n        expected = list(multiply(input, 3))\n        with PureGeneratorPool(multiply, self.procs, 3) as pool:\n            self.assertIsInstance(pool, self.desired_type)\n            result = list(pool.apply(input[:50]))\n            self.assertListEqual(result, expected[:50])\n            result = list(pool.apply(input[50:]))\n            self.assertListEqual(result, expected[50:])\n\n    def test_called_once_reuse(self):\n        with PureGeneratorPool(callcount, processes=self.procs) as pool:\n            for n in range(self.procs + 1): # ensure at least one process is used twice\n                result = list(pool.apply([None]))\n                self.assertListEqual(result, [1])\n\n    def _crashing_iter(self, n):\n        with self.assertRaises(ChildProcessError if self.procs > 1 else ValueError):\n            list(itermap(crashy, ([False]*n) + [True], processes=self.procs))\n\n    def test_crashing_iter(self):\n        self._crashing_iter(0)\n        self._crashing_iter(self.procs + 1)\n        self._crashing_iter(40)\n\n    def test_early_crash(self):\n        with self.assertRaises(ChildProcessError if self.procs > 1 else ValueError):\n            list(itermap(early_crash, ([False]*3), self.procs))\n\n    def test_not_generator(self):\n        with self.assertRaises(ChildProcessError if self.procs > 1 else TypeError):\n            list(itermap(not_generator, ([False]*3), self.procs))\n\n    def test_too_many_args(self):\n        with self.assertRaises(ChildProcessError if self.procs > 1 else TypeError):\n            list(itermap(multiply, ([False]*3), self.procs, 3, 4))\n\n\nclass TestMPMulti(TestMPSingle):\n    procs = 2\n    desired_type = _PureGeneratorPoolMP\n\n    def test_unpickleable_function(self):\n        with self.assertRaises(AttributeError):\n            list(itermap(lambda x: x, ([False] * 3), self.procs))\n\n    def test_unpickleable_item_in_args(self):\n        with self.assertRaises(AttributeError):\n            list(itermap(null, ([None]*10), self.procs, lambda x: x))\n\n    def test_unpickleable_item_in_iter(self):\n        with self.assertRaises(AttributeError):\n            list(itermap(null, ([None]*10) + [lambda x: x], self.procs, None))\n\n    def test_empty_iter(self):\n        result = list(itermap(callcount, [], processes=self.procs))\n        self.assertListEqual(result, [])\n\n\nclass TestMPMultiSigInt(unittest.TestCase):\n\n    pool_size = 4\n\n    def items(self):\n        with PureGeneratorPool(multiply, self.pool_size, 1) as pool:\n            self.pool = pool\n            yield from pool.apply(islice(count(), 2000))\n\n    def test_sigint_to_self(self):\n        result = self.items()\n        next(result)\n        with self.assertRaises(KeyboardInterrupt):\n            ctrl_c(os.getpid())\n\n    @unittest.skipIf(sys.platform.startswith('win'), \"Can't send ctrl-c to an individual process on Windows\")\n    def test_sigint_to_child(self):\n        result = self.items()\n        next(result)\n        with self.assertRaises(ChildProcessError):\n            for i in range(self.pool_size):\n                ctrl_c(self.pool._procs[i].pid)\n            for r in result:\n                pass\n"
  },
  {
    "path": "tests/test_packet.py",
    "content": "import itertools\nimport unittest\n\nfrom teletext.packet import *\n\n\nclass TestPacket(unittest.TestCase):\n\n    packet = Packet()\n\n    def setUp(self):\n        pass\n\n    def test_type(self):\n        self.packet.mrag.row = 0\n        self.assertEqual(self.packet.type, 'header')\n        self.packet.mrag.row = 1\n        self.assertEqual(self.packet.type, 'display')\n        self.packet.mrag.row = 27\n        self.assertEqual(self.packet.type, 'fastext')\n        self.packet.mrag.row = 28\n        self.assertEqual(self.packet.type, 'page enhancement')\n        self.packet.mrag.row = 29\n        self.assertEqual(self.packet.type, 'magazine enhancement')\n        self.packet.mrag.row = 31\n        self.assertEqual(self.packet.type, 'independent data')\n        self.packet.mrag.row = 30\n        self.assertEqual(self.packet.type, 'independent data')\n        self.packet.mrag.magazine = 8\n        self.assertEqual(self.packet.type, 'broadcast')\n"
  },
  {
    "path": "tests/test_sigint.py",
    "content": "import os\nimport signal\nimport sys\nimport time\nimport unittest\n\nfrom teletext.sigint import SigIntDefer\n\n\ndef ctrl_c(pid):\n    if sys.platform.startswith('win'):\n        # Note: on Windows this doesn't get delivered immediately.\n        os.kill(pid, signal.CTRL_C_EVENT)\n        time.sleep(0.05)\n    else:\n        os.kill(pid, signal.SIGINT)\n\n\nclass TestSigInt(unittest.TestCase):\n\n    def test_ctrl_c(self):\n        with self.assertRaises(KeyboardInterrupt):\n            ctrl_c(os.getpid())\n\n    def test_interrupt(self):\n        with self.assertRaises(KeyboardInterrupt):\n            with self.assertRaises(ValueError):\n                with SigIntDefer() as s:\n                    self.assertFalse(s.fired)\n                    ctrl_c(os.getpid())\n                    self.assertTrue(s.fired)\n                    raise ValueError\n"
  },
  {
    "path": "tests/test_spellcheck.py",
    "content": "import unittest\n\nfrom teletext.spellcheck import *\nfrom teletext.elements import Displayable\n\n\nclass TestSpellCheck(unittest.TestCase):\n\n    def setUp(self):\n        self.sc = SpellChecker(language='en_GB')\n\n    def test_case_match(self):\n        src = 'AaAaA'\n        word = 'bbbbb'\n        self.assertEqual(self.sc.case_match(word, src), 'BbBbB')\n\n    def test_suggest(self):\n        # correctly spelled word should be unchanged\n        self.assertEqual(self.sc.suggest('hello'), 'hello')\n        # incorrectly spelled word with known substitutions should be fixed\n        self.assertEqual(self.sc.suggest('dello'), 'hello')\n\n    def test_spellcheck(self):\n        d = Displayable((17,), 'dello dello dello'.encode('ascii'))\n        self.sc.spellcheck(d)\n        self.assertEqual(d.to_ansi(colour=False), 'hello hello hello')\n"
  },
  {
    "path": "tests/test_stats.py",
    "content": "import unittest\n\nfrom teletext.stats import *\n\n\nclass TestStatsList(unittest.TestCase):\n\n    def test_str(self):\n        l = StatsList()\n        l.append('a')\n        l.append('b')\n        self.assertEqual(str(l), 'ab')\n"
  },
  {
    "path": "tests/test_subpage.py",
    "content": "import unittest\n\nimport numpy as np\n\nfrom teletext.subpage import Subpage\n\n\nclass SubpageTestCase(unittest.TestCase):\n\n    def test_checksum(self):\n        p = Subpage()\n        self.assertEqual(0xe23d, p.checksum)\n        p = Subpage(prefill=True)\n        self.assertEqual(0xe23d, p.checksum)\n\n"
  },
  {
    "path": "tests/vbi/__init__.py",
    "content": ""
  },
  {
    "path": "tests/vbi/test_line.py",
    "content": "import os\nimport unittest\n\nimport numpy as np\n\nfrom teletext.file import FileChunker\nfrom teletext.vbi.line import Line\nfrom teletext.vbi.config import Config\n\n\nclass LineTestCase(unittest.TestCase):\n\n    def noisegen(self, max_loc, max_scale):\n        for n in range(10):\n            for loc in range(0, max_loc+1, max(1, max_loc//8)):\n                for scale in range(0, max_scale+1, max(1, max_scale//8)):\n                    yield (\n                        np.clip(np.random.normal(loc, scale, size=(2048,)), 0, 255).astype(np.uint8).tobytes(),\n                        {'loc':loc, 'scale':scale}\n                    )\n\n    def setUp(self):\n        Line.configure(Config(), force_cpu=True)\n\n    def test_empty_rejection(self):\n        lines = ((Line(data), params) for data, params in self.noisegen(256, 8))\n        lines = ((line, params) for line, params in lines if line.is_teletext)\n        for line, params in lines:\n            self.assertFalse(line.is_teletext, f'Noise interpreted as teletext: {params}')\n\n    @unittest.expectedFailure\n    def test_known_teletext(self):\n        try:\n            with open(os.path.join(os.path.dirname(__file__), 'data', 'teletext.vbi'), 'rb') as f:\n                lines = (Line(data, number) for number, data in FileChunker(f, 2048))\n                for line in lines:\n                    self.assertTrue(line.is_teletext, f'Line {line._number} false negative.')\n        except FileNotFoundError:\n            self.skipTest('Known teletext data not available.')\n\n    @unittest.expectedFailure\n    def test_known_reject(self):\n        try:\n            with open(os.path.join(os.path.dirname(__file__), 'data', 'reject.vbi'), 'rb') as f:\n                lines = (Line(data, number) for number, data in FileChunker(f, 2048))\n                for line in lines:\n                    self.assertFalse(line.is_teletext, f'Line {line._number} false positive.')\n        except FileNotFoundError:\n            self.skipTest('Known reject data not available.')\n\n"
  },
  {
    "path": "tests/vbi/test_patterncuda.py",
    "content": "import pathlib\nimport unittest\n\nimport numpy as np\n\nfrom teletext.vbi.pattern import Pattern\ntry:\n    from teletext.vbi.patterncuda import PatternCUDA\n\n\n    class PatternCUDATestCase(unittest.TestCase):\n\n        def setUp(self):\n            p = pathlib.Path(__file__).parent.parent.parent / 'teletext' / 'vbi' / 'data' / 'vhs' / 'parity.dat'\n            self.pattern = Pattern(p)\n            self.patterncuda = PatternCUDA(p)\n\n        def test_equal_to_cpu(self):\n            arr = np.arange(256, dtype=np.uint8)\n            a = self.pattern.match(arr)\n            b = self.patterncuda.match(arr)\n\n            self.assertTrue(all(a==b), 'CPU and CUDA pattern matching produced different results.')\n\nexcept ModuleNotFoundError as e:\n    if e.name != 'pycuda':\n        raise\n\n"
  },
  {
    "path": "tests/vbi/test_patternopencl.py",
    "content": "import pathlib\nimport unittest\n\nimport numpy as np\n\nfrom teletext.vbi.pattern import Pattern\ntry:\n    from teletext.vbi.patternopencl import PatternOpenCL\n\n\n    class PatternOpenCLTestCase(unittest.TestCase):\n\n        def setUp(self):\n            p = pathlib.Path(__file__).parent.parent.parent / 'teletext' / 'vbi' / 'data' / 'vhs' / 'parity.dat'\n            self.pattern = Pattern(p)\n            self.patternopencl = PatternOpenCL(p)\n\n        def test_equal_to_cpu(self):\n            arr = np.arange(256, dtype=np.uint8)\n            a = self.pattern.match(arr)\n            b = self.patternopencl.match(arr)\n\n            #self.assertTrue(all(a==b), 'CPU and OpenCL pattern matching produced different results.')\n            self.assertEqual(a.tolist(), b.tolist(), 'CPU and OpenCL pattern matching produced different results.')\n\nexcept ModuleNotFoundError as e:\n    if e.name != 'pyopencl':\n        raise\n\n"
  },
  {
    "path": "tests/vbi/test_training.py",
    "content": "import io\nimport unittest\n\nimport numpy as np\n\nfrom teletext.vbi.training import *\nfrom teletext.vbi.config import Config\n\n\nclass TrainingTestCase(unittest.TestCase):\n\n    def setUp(self):\n        pass\n\n    @unittest.skip\n    def test_split(self):\n        files = [io.BytesIO() for _ in range(256)]\n        pattern = PatternGenerator.load_pattern()\n        max_n = 10\n\n        data = [(n, np.unpackbits(pattern[n:n+PatternGenerator.pattern_length][::-1])[::-1]) for n in range(max_n)]\n        split(data, files)\n\n        pattern_bits = np.unpackbits(pattern[:max_n+PatternGenerator.pattern_length][::-1])[::-1]\n        patterns_present = set()\n        for x in range(len(pattern_bits) - 23):\n            patterns_present.add(\n                np.packbits(pattern_bits[x:x+24][::-1])[::-1].tobytes()\n            )\n\n        for f in files[:1]:\n            arr = np.frombuffer(f.getvalue(), dtype=np.uint8).reshape(-1, 27)\n            for l in arr:\n                # Assert that pattern matches samples.\n                self.assertTrue(all(l[:3] == np.packbits(l[3:][::-1])[::-1]))\n                # Assert that pattern is a pattern we actually put in to split.\n                self.assertIn(l[:3].tobytes(), patterns_present)\n\n"
  }
]