[ { "path": ".gitattributes", "content": "* linguist-vendored\n*.hs linguist-vendored=false\n" }, { "path": ".gitignore", "content": ".stack-work\n.hie\n" }, { "path": "LICENSE", "content": "Copyright Ruben Berenguel (c) 2021\n\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n * Redistributions of source code must retain the above copyright\n notice, this list of conditions and the following disclaimer.\n\n * Redistributions in binary form must reproduce the above\n copyright notice, this list of conditions and the following\n disclaimer in the documentation and/or other materials provided\n with the distribution.\n\n * Neither the name of Author name here nor the names of other\n contributors may be used to endorse or promote products derived\n from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\nLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\nA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\nOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\nSPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\nLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\nDATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\nTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE." }, { "path": "README.md", "content": "# Glancer\n\n> **glancer**:\n\n> NOUN _informal_ a person who glances\n\n> **glance**:\n\n> VERB If you glance at something or someone, you look at them very quickly and then look away again immediately.\n\n> VERB If you glance through or at a newspaper, report, or book, you **spend a short time looking at it without reading it very carefully**.\n\n---\n\n[Regularly updated examples](https://github.com/rberenguel/glances)\n\n---\n\n- [Glancer](#glancer)\n - [Installation](#installation)\n - [Usage](#usage)\n - [Notes and TODOs](#notes-and-todos)\n\n---\n\nThe amount of online conferences has skyrocketed lately, I wonder why. This has caused my _Pending to watch_ list to balloon from 30-ish pending technical videos (which are already a lot) to more than 100.\n\nThere are then 2 problems:\n\n1. I have too many techie conference videos to watch\n2. In a lot of cases I realise halfway through that the subject wasn't that interesting or that I already know the area to be covered.\n\nFor a long while I have had a similar problem with written articles. I solved it by:\n\n1. Forcing me to read a substantial amount by [writing a weekly list of the best ones](https://mostlymaths.net/tags/readings/)\n2. Brutally stop reading any article that is not good enough to _possibly be_ in that list.\n\nThis is easy in writing: you can quickly scan the text and decide if it looks interesting enough for a deep dive in a few seconds (tech article reads range from a few minutes to around half an hour, depending of how technical it may get). But there is no way of doing it in videos! You need to watch maybe 15-20 minutes to then realise \"meh\".\n\n_Glancer_ should help with this. Given a YouTube url, it will:\n\n- Download the corresponding video (to a temporary folder),\n- Download the auto-generated subtitles (assumes English, hardcoded),\n- Capture images from the video every N=30 seconds (hardcoded for the moment),\n- Convert the images to base64,\n- Create a standalone webpage with the screenshots on the left and the corresponding text on the right.\n\nThe goal is to be able to glance at the talk to decide if you really want to watch it or not. The _standalone_ part of the created webpage is to make it easier to \"watch\"/\"share\" to my iPad/iPhone without having to move a folder full of images. The whole talk becomes just a 5-15 Mb HTML file.\n\nA couple of additional neat (for me at least) features:\n\n- Clicking/tapping on the image will enlarge it, in case you want to see some code block larger (I wanted hover, but it was too tricky on mobile).\n- Clicking on the arrow on the lower-right of the slid block will open the video on Youtube, at that moment in time.\n\n## Installation\n\n```bash\ngit clone https://github.com/rberenguel/glancer\ncd glancer\nstack install\n```\n\nYou will need to have installed/available in the path:\n\n- The `base64` executable (should be in all IX systems by default)\n- `cat` in `/bin/cat` (likewise)\n- [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) installed. Note the _p_. There is currently a bug in the normal one with downloading auto-generated subtitles (again, this also happened a long time ago).\n- The [stack](https://docs.haskellstack.org/en/stable/install_and_upgrade/) Haskell build tool\n\n## Usage\n\n```\nUsage: glancer URL FILEPATH\n Glancer\n\nAvailable options:\n URL Youtube URL\n FILEPATH HTML file name (don't add extension)\n -h,--help Show this help text\n```\n\nIn other words, `glancer https://www.youtube.com/watch?v=JWQxd3YKWhs internals-pyspark-arrow` would create the webpage `internals-pyspark-arrow.html` in the current folder, after processing the talk I gave at Spark Summit 2019. You can see the generated file [here](https://www.mostlymaths.net/glancer/example/internals-pyspark-arrow.html).\n\nSometimes `youtube-dlc` won't be able to find the embedded youtube video (I've seen this happen randomly in Spark Summit North America 2020 videos in databricks.com), in this case the process will fail. Try to feed it youtube urls directly.\n\n## Notes and TODOs\n\n- [ ] Making the time between images customizable via the CLI (if I find out 30 is not good enough in general).\n- [ ] Add a test suite to harden subtitle parsing. I always think parsers will be small enough and that it will be \"obvious\" they work. It's never the case, at least I did it [right](https://github.com/rberenguel/haskset/blob/master/test/Spec.hs) [twice](https://github.com/rberenguel/bear-note-graph/blob/master/tests/test_parser.py).\n- [ ] Make the still images video-dependent (so several `glancer` commands can run concurrently, even if it's a bad idea)\n- [x] ~Some additional tweaks to the HTML/CSS (possibly adding some JS as well)~\n\n## Similar projects\n\n### [natural-language-youtube-search](https://github.com/haltakov/natural-language-youtube-search)\n\nThis project downloads the YouTube video, extracts every N-th frame and uses neural networks to classify the content of each slide, allowing you to search by text. Impressive!\n\n---\n\n_Note_: This README is long and winding on purpose.\n" }, { "path": "Setup.hs", "content": "import Distribution.Simple\nmain = defaultMain\n" }, { "path": "example/internals-pyspark-arrow.html", "content": "\n\n\n\n\n \n \n\n\t\n\t\t
\n\t\t\t

Internals of Speeding up PySpark with Arrow - Ruben Berenguel (Consultant)\n

\n\t\t\t

Created with glancer

\n
\n\t
\n\t\t\n\t
\n\t
\n\t\ta lot of people here are you're working\n\t\twith a spark\n\t\tright your hands yeah it's nice so I\n\t\twill start with what is pandas but if\n\t\tyou are working with Python you probably\n\t\talready know this one then what is our\n\t\trole this may be Neil or maybe it's not\n\t\tNeil then a bit of how this part works I\n\t\tmean we are in sparks emits so probably\n\t\tyou all know this but it's always good\n\t\tto remind how the pieces fit together\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tto remind how the pieces fit together\n\t\tbecause I when I explained how Pais part\n\t\tworks because maybe you don't know that\n\t\tone it useful and then finally I'll\n\t\tcover how our row is making a pie spark\n\t\tway way faster so I'm Ruben a\n\t\tmathematician I work as a consultant\n\t\tfrom advertising company and I think\n\t\twriting Python for something like 15\n\t\tyears I mean I don't know an amp I\n\t\tdidn't exist back then then I move to\n\t\tother languages but I still write a lot\n\t\tof Python day-to-day scale and Python\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tof Python day-to-day scale and Python\n\t\tnow so I'm Pi spark code base it's kind\n\t\tof a sweet spot because I have a bit of\n\t\tPython every Java Scala it's a cool\n\t\tplace to be so what is pandas I mean you\n\t\tmay know it already it's a Python that\n\t\tanalysis library if you see a job offer\n\t\tthat says Python data pandas is involved\n\t\tin the code base somewhere and it's\n\t\tefficient wide efficient it because it's\n\t\tcolumnar when it has a CNS - back and so\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tcolumnar when it has a CNS - back and so\n\t\tthe mechanism written is specifically in\n\t\tPython I mean the high-level API is but\n\t\tthe rest is written in a very efficient\n\t\timplementation corner appears a lot in\n\t\tmodern databases modern storage systems\n\t\tand what's the point of something being\n\t\tcolumnar same machine we have a table\n\t\twith a lot of columns and we want to sum\n\t\tone of the columns if you Resta curve\n\t\tyou - you have a cpu and you have a row\n\t\tyou have a cache line inside of the CPU\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tif you're thinking in rows and you want\n\t\tto some one column you need to go row by\n\t\trow loading the data and a little bit to\n\t\tthe point where you can actually sum it\n\t\twhen you can something is stored column\n\t\twise you don't need to do it slowly you\n\t\tcan just get everything from\n\t\tthe column bringing it into the cash and\n\t\tsome that was a bit first if you were in\n\t\tthe previous table ie you kick that's\n\t\treally heartening to pronounce but he\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\treally heartening to pronounce but he\n\t\twas doing the integration of spark are\n\t\twith ro and he also explained how CN the\n\t\tinstructions make this fast so Panda is\n\t\tcolumnar how does it work internally in\n\t\ta high-level because I don't want to get\n\t\tinto the details because I they are kind\n\t\tof complicated you have a data frame if\n\t\tyou use pandas you know what it is it's\n\t\tbasically a table with some names for\n\t\tthe columns you have different types\n\t\tinternally pan that is going to group\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tthese types in memory by column index s\n\t\tof up to one place and then you bring\n\t\ttogether all the floats put them\n\t\tsideways in the fitting this way and\n\t\tbring them in memory same for Strings\n\t\tobjects anything that is not an internal\n\t\tdata type for pandas it's stored as an\n\t\tobject block and integers there are\n\t\tseveral pandas data types that are\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tstored differently in memory altogether\n\t\tthe blocks form a hundred by a block\n\t\tmanager and when you request something\n\t\tbeing done to a problem you tell to the\n\t\tblock manager give me this column and\n\t\trun this operation now we've done\n\t\tquickly to our Oh what is our always a\n\t\tlibrary so you are unlikely to be using\n\t\tit unless you are writing your own\n\t\tlibrary I mean you are just a client\n\t\tthrough pandas or through spark to the\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tother library it cause language it's\n\t\tcolumnar it in memory the key point is\n\t\tthat it's across language so you can use\n\t\tthe same internal structure in different\n\t\tlanguages and pass from one to the other\n\t\tand it's a bit optimized I mean the\n\t\twhole point of it it is written in C and\n\t\tthere are implementations in Grasse go C\n\t\tJava obviously are\n\t\tthere are not a lot of projects that are\n\t\tusing it internally but pandas is one of\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tusing it internally but pandas is one of\n\t\tusing it internally but pandas is one of\nthem\n\t\tthem\n\t\tspark park were to use it is for\n\t\tsomething and Romero for what task which\n\t\tis a distributed computation framework\n\t\tkind of similar to spark but written\n\t\tentirely in Python gray which is basic\n\t\tsomething similar to dusk there is a POC\n\t\tof data processing written in rust using\n\t\tonly a row there's a connector from Java\n\t\tto Scala with Python that is still\n\t\texperimental and it's using our role as\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\texperimental and it's using our role as\n\t\twell an integration between our own\n\t\tpandas is seamless this by design I mean\n\t\tro was created by the same developers\n\t\tthat sorry that started pandas so the\n\t\tinternal memory layouts are similar\n\t\tenough that the operations are really\n\t\tenough that the operations are really\nfast\n\t\tfast\n\t\thow does it look like in a row again you\n\t\thave a table this time I didn't bother\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\twith the types and each set of rows is\n\t\tcalled a tracker patch I mean you can\n\t\talways call our track or throw a\n\t\trecording database link also throws\n\t\trecords and you bring all the columns\n\t\tput them sideways and submit the data\n\t\tand consider that track or batch that's\n\t\tdone for a certain number of rows and\n\t\tthis can be done in a streaming\n\t\tbasically the metadata says up to this\n\t\tpoint is this column up to this point is\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tpoint is this column up to this point is\n\t\tthat other column and you can\n\t\tessentially skip the first if you only\n\t\tone the last one so you do can do it in\n\t\tstreaming by just getting blocks of rows\n\t\tone to the other so at the high level\n\t\tyou can think of a narrow table as set\n\t\tof record matches it's not exactly that\n\t\tthat internal internal it's more\n\t\tcomplicated than this one from a high\n\t\tlevel this is good enough to understand\n\t\thow our looks like well our olaf's\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\t\n\t\tif you think about the destruction so\n\t\twe're using record matches in one side\n\t\tso blocks of growth on the other side we\n\t\thave a block manager that its handling\n\t\tvertical blocks they don't look exactly\n\t\tthe same but in the end you can think of\n\t\tthem as the same you can convert another\n\t\ttable into a pandas dataframe real\n\t\teasily there's a method called two\n\t\tpandas I mean if you think of it that\n\t\tway it's super easy but internally if\n\t\tyou look at implementation it's like you\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tyou look at implementation it's like you\n\t\tget something from one site and shop it\n\t\taround the idea is that if you squint\n\t\tlike hard enough these are just vertical\n\t\tthings that have similar colors that\n\t\tmotors the idea you're can convert\n\t\tcracker batches to block managers and\n\t\tyou can convert block managers to\n\t\tfreaking batches now further what is a\n\t\tspark we're gonna spark some it's you\n\t\tknow this I'll do this quicker than the\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tknow this I'll do this quicker than the\n\t\tprevious one which was looking off\n\t\tinterest with a computation framework\n\t\topen source it's somewhat easy to use\n\t\tand scale horizontally and vertically\n\t\thow does it work you have a cluster\n\t\tmanager that is going to give you\n\t\tcomputational resources you have some\n\t\tkind of distributed storage can be\n\t\tHadoop in general HDFS in general your\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\t\n\t\tanother is going to request resources to\n\t\tthe cluster manager to compute some\n\t\ttasks so you have tasks you record\n\t\tresources and then some executors appear\n\t\tin the resources that you have been\n\t\tgiven but this doesn't really tell us\n\t\tmuch about spark the important part that\n\t\tmade spark spark is the RDD it's the\n\t\tability to recompute things when they\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tfailed that before in MapReduce wasn't\n\t\tas easy so here we have a happier\n\t\tas easy so here we have a happier\nleading\n\t\tleading\n\t\tand when you have an RDD you have a\n\t\tbasically five properties or five if you\n\t\tcheck out the class at the class level\n\t\tyou have five required but three\n\t\trequired methods that you need to\n\t\tprovide for an RDD to be not abstract\n\t\tbasically one is partitions this is\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\twhere the data is or how do they make\n\t\twhere the data is or how do they make\nsense\n\t\t\n\t\tthis one is optional it's the preferred\n\t\tlocations so when you have these data\n\t\tyou have here this can be I don't know\n\t\tyou have low data file or you have\n\t\tcreated something in memory so it gets\n\t\tpartitioned by executor preferred\n\t\tlocation is when you have some kind of\n\t\tsort or some operation that it's closer\n\t\tto the load the loaded data or the\n\t\tprocess data and basically it says where\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tprocess data and basically it says where\n\t\tyou can find it because sometimes you\n\t\twant to compute something on a group\n\t\tit's better if you do the computation in\n\t\tthe machine that holds that group you\n\t\tneed a computed a computing method\n\t\ttrying to bring partitions to other\n\t\tpartitions after you have computation\n\t\tfor each partition the new result of the\n\t\tcomputation of compute is a new RDD that\n\t\thas a computation being applied to the\n\t\tprevious video you have the dependencies\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tthis is the key point of the RDD when\n\t\tone of the partition disappears because\n\t\tthe machine that was holding it\n\t\tdisappears because of the internet being\n\t\tlarge you can recompute only the\n\t\tpartition because you can trace back\n\t\tthrough compute and the dependencies\n\t\twhere this was coming from so eventually\n\t\tyou can go back to the first original\n\t\tdata source and compute just only that\n\t\tuntil you make it to the lost machine\n\t\tand dependencies it's basically RT these\n\t\ttracing back in time we are compute\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\ttracing back in time we are compute\n\t\tuntil you go through some data source or\n\t\tdata provider and fellow you have a\n\t\tpartition er as part is going to do\n\t\tsomething sensible if you don't provide\n\t\tit I need to be a one-to-one method that\n\t\tcan distribute the partitions across the\n\t\tdifferent machines so\n\t\tgo back to buy spark I'll refer to PI\n\t\tspark as the Python API on top the spark\n\t\totherwise I will say just a spark or a\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tScala spark a spark is a Python API for\n\t\tthe core of a spark which is waiting in\n\t\ta scalos you already know how is it\n\t\tworking inside on one side you are\n\t\twearing Python Python does not have a\n\t\tJVM I mean - runs in the Python built on\n\t\tmachine using a library known as a PI 4j\n\t\twhich as the Jay can tell you it's\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tsomething to do with Java every time you\n\t\tcreate some object in PI spark these get\n\t\tassociated an object in the JVM so when\n\t\tyou start the Python driver in Python in\n\t\tPI spark this happens at some point I\n\t\twanted to show the code because this\n\t\tsuper short thing it's basically telling\n\t\tto the JVM start spark and keep\n\t\tlistening and that's it\n\t\tand I'm super super sure because this is\n\t\tdoing a lot of work basically the\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tdoing a lot of work basically the\n\t\tstarting spark it's connecting back to\n\t\tyour Python driver and it's keeping a\n\t\treference in memory to that JVM so you\n\t\tcan keep track of whatever you dream\n\t\tdown there now we have a happy RDD in\n\t\tPython like like we've had our diseases\n\t\tclasses in a Scala we have a class\n\t\tcalled RDD in Python you may have\n\t\t\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tinside of it you can look it up by\n\t\tcreating your I mean you can open a bit\n\t\toknot booklet an RTD and check this each\n\t\tother it is going to have a private in\n\t\tPython style underscore jRD this is a\n\t\treference to the object in JVM so you\n\t\tcan always trace back the oddity that\n\t\tyou created in Python didn't really\n\t\tcreate it in Python you created dignity\n\t\tin the JVM and it had a reference to\n\t\t\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tand well these J's appear everywhere in\n\t\ta lot of private properties on methods\n\t\tin the Python API and they are standard\n\t\tfor Java I mean I'm not a huge fan of\n\t\tjvm languages I like a Scala and I like\n\t\tPython and that's it a Python it's not\n\t\tin the IBM of course the connection\n\t\tbetween these two is through this\n\t\tgateway this could be object it's\n\t\tcreated at the beginning and if it's\n\t\tpassed through every time you create a\n\t\tnew RDD\n\t\tI mean it's the main connection that\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tI mean it's the main connection that\n\t\tgets you back from Python to whatever is\n\t\tin spark in past part they are two entry\n\t\tpoints to create this so you create an\n\t\tRDD that is it you just type blah blah\n\t\tblah did he paralyzed whatever and then\n\t\tyou have pi blind RDD this one it's kind\n\t\tof internal so every time you want to\n\t\tmap the partitions of an RDD in python a\n\t\tpipeline target is going to be created\n\t\tand pipeline that leads the one that is\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tactually doing the drawing the work so\n\t\tit's a bit like the this matreoshka\n\t\tanimated gif it because uh you are\n\t\tcreating things in python and they are\n\t\tinside other things that are inside\n\t\tother things inside of the JVM it gets a\n\t\tbit confusing in the end you have Python\n\t\tare deities in the scholar side and\n\t\tpipelined rdd's in the python side this\n\t\tis transparent for the PI spark user it\n\t\tis just right Rd didn't map F when it\n\t\tworks but it's important to know what it\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tinside this material I think because why\n\t\tnot get into the play into the picture\n\t\tit will explain why it's Astro so we\n\t\thave an angry RDD because he's a bit it\n\t\tin Python we want to apply a map F F is\n\t\ta method in Python whatever this create\n\t\ta pipeline that ID its internal the F\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tit's pulled into our internal property\n\t\tcalled funk\n\t\tand we get a pointer to the previous RTD\n\t\tand we have a J or diddly of course this\n\t\tJ our DD it's where the magic is\n\t\thappening we have a Python or DD in the\n\t\tJVM these are passing a Scala\n\t\tobviously it has some dependencies\n\t\tbecause all our T's need to have a\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tstarting point this dependency goes back\n\t\tto the our DD we had originally because\n\t\twe are going to map it with an F it in\n\t\tPython whatever it is still an RDD in\n\t\tscholars you need to have dependencies\n\t\tand we have a computer methods because I\n\t\tneed to be there the computing method\n\t\tneeds to do something with F even if\n\t\tit's in Python you need to compute it it\n\t\tdoesn't matter it what the magic is\n\t\thappening in the end all the\n\t\tcomplication it's how do you compute\n\t\tsomething in the JVM that it's actually\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tin Python well you want to run something\n\t\tin the executors and that's going to be\n\t\tdone by something called Python rather\n\t\tPython run is going to start a Volker\n\t\tit basically creates Python process\n\t\tthrough a connects to be a socket send\n\t\tthrough the socket a lot of the data\n\t\tit makes the worker load all the\n\t\tlibraries that you need and I had\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tdrawings for this one but otherwise it\n\t\tmakes no sense we are in Scala this time\n\t\tPython Runner is inside of call it\n\t\tcreated by compute so as soon as you\n\t\twant to materialize the the computation\n\t\tthis creates a Python worker in the\n\t\texecutor and data goes through a socket\n\t\tso the F is being applied in the Python\n\t\tworker and the data is being sent back\n\t\tto the JVM as you can imagine there's a\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tlot of serialization going both ways and\n\t\tas you can imagine serializing is\n\t\texpensive always I mean basically you\n\t\tare sending pickles down and pickles up\n\t\tin Python and pickling and unpicking is\n\t\ta slow very slow so workers process the\n\t\tstreams of data connects back to the JVM\n\t\tload all the libraries the libraries\n\t\tneed to match the driver\n\t\tthis utilized pickle things and applies\n\t\tthe function at Cynthia pull back yeah\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tdo it to back this was covering our\n\t\tdisease but dying yes you are not using\n\t\tour do this directly anymore\n\t\tI'm not use data frames I use data\n\t\tframes because then I can optimize my\n\t\twell I don't need to optimize it I mean\n\t\tcatalyst is going to optimize code for\n\t\tme I want I defined a D a G and the D a\n\t\tG get optimized so if you're using data\n\t\tframes it when you get the advantages of\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tusing a spark after 2.0 basically a plan\n\t\tis generated so you write all the\n\t\ttransformations you want eventually Duke\n\t\tright or your count or you basically\n\t\tcreate an action this plan will convert\n\t\tinto a logical plan optimized logical\n\t\tplan physical plan when you were to the\n\t\tphysical plan something get executed and\n\t\tall the compute methods start running\n\t\tbecause underlying data frames that are\n\t\tstill are get is the planner optimize my\n\t\tcatalyst which is basically something\n\t\tthat removes branches from trees if\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tpossible so it when to it gets this\n\t\tlogical plan and when you have a filter\n\t\tat the end it has to push it down as\n\t\tmuch as possible because the less data\n\t\tyou move around the less data you're\n\t\tcomputing now depending on what you did\n\t\tin Python catalyst is going to choose\n\t\teither the Python UTF runner or the\n\t\tPython ro runner that's the good one the\n\t\tone that uses our row the other one it's\n\t\tthe old version of what is using all\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tthese pickling things so you have UDF\n\t\tRunner serialize be serialize and send\n\t\tback this is basically the same as the\n\t\tPython runner before and the only thing\n\t\tis that this is the data frame\n\t\timplementation that the first two they\n\t\tare good implementation but when you're\n\t\tusing the other one I mean we just\n\t\tchange the names we are sending record\n\t\tchange the names we are sending record\nmatches\n\t\tmatches\n\t\tso what internally the in the worker\n\t\twe're converting onto pandas remember\n\t\tthis is super fast because our own\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tpandas look the same internally we play\n\t\tf2 the call\n\t\tit is fast because it's columnar so\n\t\teverything it's kind of optimized for\n\t\tspeed and we send a letter back that's\n\t\tagain kind of cheap because arrow has a\n\t\tserializing format so all this movement\n\t\tand the computation in the Python side\n\t\tare faster already and it's like well\n\t\t\n\t\taccording to the kind of official /\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tunofficial benchmarks you can get from\n\t\t3x to 100x optimization just by changing\n\t\ta setting which is enabling arrow and\n\t\tusing UV f defined as pandas either\n\t\tscholar or group maps or group what\n\t\tthere are forth and you can use\n\t\tbasically take the advantage of what you\n\t\tcan do in pandas by converting to a rule\n\t\tI want to show a couple of examples so\n\t\tyou see what you can do and what speed\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tups you get basic example is just two\n\t\tpandas basically if you do have two\n\t\tpandas in general I mean what I'm doing\n\t\there is just creating a range a dataset\n\t\twith a power of two amount of later so\n\t\there it's a two to the 20 with a random\n\t\tcolor column and just converting it to\n\t\tpandas converting to pandas brings all\n\t\tthe data to the driver so it's always a\n\t\tkind of a bad idea but sometimes you\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tkind of a bad idea but sometimes you\n\t\tneed to do something in pandas I need to\n\t\tdo in the driver and that's what you get\n\t\tI'm just doing this this is with arrow\n\t\tdisabled and I want to compare how much\n\t\tfaster is when I just change that to\n\t\tenabled the only thing you need to do if\n\t\tyou want to take advantage of ro\n\t\taside from having a spark at least two\n\t\t\n\t\twell I time lists for different powers\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tof of - it was in my local machine with\n\t\tlocal runners easiest benchmark possible\n\t\tand this gets to 18 seconds I couldn't\n\t\tdo 23 or 24 because I the driver was\n\t\tdying and then I did the reversion\n\t\tdarrel version is consistently 10 times\n\t\tfaster just in the local thing super\n\t\teasy in just changing one setting so if\n\t\tyou ever do to pandas just do this you\n\t\tget a awesome\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tget a awesome\n\t\tspeed up and it works for larger\n\t\tdatasets because of the compression of\n\t\thow are you handle things now a little\n\t\tbit of a more complicated one which is a\n\t\tgroup map you may know that doing group\n\t\tbias is always well it's not always\n\t\tabout idea if you're using the scale API\n\t\tgroup I had a lot of improvement in 2.2\n\t\t2.4 when the aggregations are critic are\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tcomputed they are computed\n\t\tincrementalist so you never store the\n\t\twhole group in memory one problem that\n\t\tthese still has is that it needs to hold\n\t\tthe whole group in memory so it's a bit\n\t\tproblematic but what anyway what we want\n\t\tto do here is that I have created kind\n\t\tof some fake transactions with some\n\t\tspent amounts and I want to I don't know\n\t\tdo some photo analysis I compare each\n\t\ttransaction with a mean for the same\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tuser because maybe if you are spending a\n\t\tlot more than the mean it's like hey\n\t\tyou've been hacked and you are doing\n\t\tsomething you shouldn't do this is very\n\t\teasy with the new way you can write\n\t\tthese pandas aggregations with are\n\t\tenabled they find a planner UDF it's a\n\t\tgroup map because for each group you\n\t\twant to map and the map is basically a\n\t\tpandas method that it's assigning\n\t\tsomething so in this case it's just the\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tmean what's this doing so here we have\n\t\tsome executors with a driver and we are\n\t\tgrouping and aggregating we have some\n\t\tlettuce had lettuce ended it in the\n\t\tdriver because it was created in the\n\t\tdriver it is distributed because when\n\t\tyou do a group by operation you need to\n\t\t\n\t\t\n\t\tproblem everything is grouped in\n\t\tmemories so each of the groups need to\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tfit in the executors there's kind of\n\t\tongoing work or there should be ongoing\n\t\twork soon about fixing this\n\t\tso it's incremental and not needing to\n\t\tusual the memory the thing is the design\n\t\tin the workers\n\t\tthese are rows in DFS are they are row\n\t\tare parallel rows so it being done by a\n\t\trow it's computed in pandas and it's\n\t\tsent back and sent in the example the\n\t\tlast thing was a two pandas to the\n\t\tdriver did you send to the rival this is\n\t\talready computed it's on per group\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\talready computed it's on per group\n\t\tyou're finished so I wrote a similar\n\t\tversion without using a pandas EDF I\n\t\twrote this it's kind of bad it could be\n\t\tdone a bit better but if you the problem\n\t\tis that if you want to do it well it\n\t\tgets really really messy so this is just\n\t\tgroup everything correct with a collect\n\t\tlist so you have for each group or so\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tlist so you have for each group or so\n\t\tfor each user all the transactions for\n\t\tthat user and then you convert\n\t\teverything to pandas pass-through numpy\n\t\tand get the mean the problem is that the\n\t\tgroup dot two pandas it's applied to the\n\t\tgroups that are coming with a collect\n\t\tlist and collect list if it works it is\n\t\tlo one when it's too large it blows\n\t\texecutors well yeah in this case we are\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tstill grouping so we are at least in the\n\t\tdata as before we are sending the\n\t\tcomputation the problem is that right\n\t\tnow we are doing the group in each\n\t\tmachine as before this time it\n\t\tincremental so you don't need to fit\n\t\teverything in memory it's fine but when\n\t\tyou get here you are sending all the\n\t\tdata to the driver and still a lot of\n\t\twork to be done in the driver because\n\t\tyou just have grouped you still need to\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tdo the mean for each group so this needs\n\t\tto be sent to panders to the worker and\n\t\tis actually\n\t\tblows up I've it a quick benchmark of\n\t\tthis one material in graphene because\n\t\tit's kind of a slow let me go back\n\t\tbecause it was in the so this\n\t\timplementation for two to the twenty\n\t\tfive fails it can do it\n\t\tand for twenty million records which is\n\t\twhat's written here it takes around\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\twhat's written here it takes around\n\t\tthree minutes the producing the the\n\t\tother implementation with pandas was\n\t\ttaking a couple of minutes for the two\n\t\tten twenty two two two two two twenty\n\t\tfive and just took one minute for twenty\n\t\tmillions it's not a huge speed up it's\n\t\tlike a three times x it's a bad code\n\t\twise it's much better and actually works\n\t\tbecause you can do to to to to to to the\n\t\ttwenty five which you couldn't do in\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tthis case so the too long didn't read or\n\t\ttoo long didn't watch is that you just\n\t\tneed to use our oh right pandas UDF's\n\t\tusually the problem is that you need to\n\t\twrite your UDF's pandas UDS that may not\n\t\tbe straightforward I mean that may be a\n\t\tproblem but if you are doing two pandas\n\t\tjust a navel down rule I mean that's a\n\t\tteam win you can get ten paint\n\t\timprovements just for that part it\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tdepends on your use case I mean if you\n\t\tare using PI spark for long enough you\n\t\tcan use you can kind of figure out how\n\t\tto do this otherwise I mean there are\n\t\tother solutions but I think using PI's\n\t\tpart it's getting better and better\n\t\treally really fast so I mean when I\n\t\tstarted with a spark that was like three\n\t\tand something years ago I used this\n\t\tcolor because back then it was like\n\t\tPython it to its low even if I was a\n\t\tseasoned Python developer if I had to\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tseasoned Python developer if I had to\n\t\tchoose now I would use spice part\n\t\tbecause why not it's almost as fast as a\n\t\tscholar well that some reason you can\n\t\tfind some resources in the presentation\n\t\tI recommend that you become a\n\t\tcontributor to spark if you are not one\n\t\talready I mean you learn a lot of the\n\t\tinternals and what's nice what's not\n\t\tI think it's a it's a good way to to be\n\t\tpart of the community and I think it's\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\ttime for questions thank you so much for\n\t\tthis session I had a question so when\n\t\tusing pandas UDF it would use arrow for\n\t\tits optimization for the serialization\n\t\tand deserialization of data but in the\n\t\tactual execution plan of a spark job is\n\t\tthere any way to validate that it is\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tactually using apache arrow because when\n\t\tyou for example do two pandas or create\n\t\ta part data frame from a panda's data\n\t\tframe it actually states in the\n\t\texecution plan that it's using arrow\n\t\tevil python but not while in the pandas\n\t\tUDF that I've seen I don't remember for\n\t\tthis color one but for the group map\n\t\tthat flood map group in pandas that's\n\t\tthe physical execution note okay and the\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tflat map group it basically suggests\n\t\tthat it is using Apache arrow I think so\n\t\tmaybe it has all robots all in the name\n\t\tbecause because when I turned the\n\t\tbasically configuration off for the\n\t\tarrow execution enabled it still uses\n\t\tthe same in the execution plan the flat\n\t\tmap group yeah that that's a good\n\t\tquestion I didn't check that I didn't\n\t\ttry to check the plan I just check the\n\t\tplan when I was using a robot either and\n\t\tcheck what it's doing I mean yeah this\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tcheck what it's doing I mean yeah this\n\t\twas something that I was working on then\n\t\tyeah yeah I don't know if there's a way\n\t\tto confirm that if you are using it from\n\t\tthe plan only okay there should but\n\t\tmaybe there's not sure all right okay\n\t\tlet's let's check later maybe we can\n\t\topen a pull request or something yeah\n\t\t\n\t\thi spark and sparked airframes and not\n\t\tpandas or maybe the working class is\n\t\tthere any advantage to era or is arrow\n\t\tonly if you trying to put anything into\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tonly if you trying to put anything into\n\t\tpandas koalas is going to use our row if\n\t\tyou enable it and it may be enabling it\n\t\tby default I don't know if it's by\n\t\tdefault but internally the colors code\n\t\thas the art row settings for the tests\n\t\tat least so it went to be an advantage\n\t\there because it's doing two pandas at\n\t\tsome point any other questions I think\n\t\tfor the walkthrough to the basic of ice\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tfor the walkthrough to the basic of ice\n\t\tpart was very useful one of the things\n\t\twe found using panda UDF's even enabling\n\t\tarrows is the problem supporting complex\n\t\tdata types as you move between Scotland\n\t\tPython even more I know the current\n\t\tworkaround is put everything with JSON\n\t\tserializes as fast as possible do you\n\t\thave any updates or any clue other is\n\t\tmoving in the error\n\t\tI've seen some cheetah tickets for a\n\t\tspark 3 that maybe are working around\n\t\tthat but I haven't checked in a few\n\t
⇰
\n
\n\t
\n\t\t\n\t
\n\t
\n\t\tthat but I haven't checked in a few\n\t\tmonths so I'm not sure but yeah I mean\n\t\tdata types I mean our data type has had\n\t\ta lot of JIRA tickets on its own so yeah\n\t\tthat's a problematic one definitive so\n\t\tyou're using not pandas what improvement\n\t\tyou can get there I mean if you are not\n\t\tusing the PI part of a spark it\n\t\tshouldn't affect you so much because it\n\t\tnever get into Python so it's just in\n\t
⇰
\n\n" }, { "path": "glancer.cabal", "content": "cabal-version: 2.2\nname: glancer\nversion: 0.1.0.0\n-- synopsis:\n-- description:\nhomepage: https://github.com/rberenguel/glancer#readme\nlicense: BSD-3-Clause\nlicense-file: LICENSE\nauthor: Ruben Berenguel\ncopyright: 2021 Ruben Berenguel\ncategory: Misc\nbuild-type: Simple\nextra-source-files: README.md\n\ncommon deps\n build-depends: base >= 4.7 && < 5, text, \n bytestring, raw-strings-qq, \n megaparsec, directory, filepath, mtl,\n html-entities\n default-language: Haskell2010\n ghc-options: -fwarn-incomplete-patterns \n -fwarn-unused-imports \n -Werror=incomplete-patterns\n -fwrite-ide-info\n -hiedir=.hie\n\nexecutable glancer\n import: deps\n hs-source-dirs: src\n main-is: Main.hs\n other-modules: Parser, Process, Html, Captions\n build-depends: optparse-applicative, hspec, containers,\n process, temporary,random, process-extras\n\ntest-suite tests\n import: deps\n type: exitcode-stdio-1.0\n hs-source-dirs: test, src\n main-is: Spec.hs\n build-depends: hspec, QuickCheck, hspec-megaparsec" }, { "path": "src/Captions.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n\nmodule Captions where\n\nimport Data.Coerce (coerce)\nimport Data.Foldable (Foldable (toList))\nimport Data.Sequence (fromList, mapWithIndex)\nimport Data.Text (intercalate, strip)\nimport qualified Data.Text as T\nimport Data.Text.Encoding (decodeUtf8)\nimport Html (embody, heading)\nimport Parser (Caption (block, txt), TimeBlock (..), TimeDef (..), inBlock, secs)\nimport Process\n ( Dir (..),\n Url (..),\n Video (..),\n deleteImages,\n )\nimport System.IO\n ( hPutStrLn,\n stderr,\n )\nimport qualified System.Process.ByteString as B\nimport Text.Printf (printf)\n\nconvertToHTML :: Show a => Video -> Dir -> Either a [Caption] -> IO T.Text\nconvertToHTML video dir parsed = case parsed of\n Right captions -> do\n html <- captionsToHTML video dir captions\n hPutStrLn stderr \"Finished\"\n return html\n Left err -> return (T.pack (show err))\n\ncaptionsToHTML :: Video -> Dir -> [Caption] -> IO T.Text\ncaptionsToHTML video dir captions = do\n caps <- formatCaptions captions (coerce $ url video) dir\n deleteImages dir\n let embodied = embody video caps\n return (heading <> embodied)\n\nformatCaptions :: [Caption] -> Url -> Dir -> IO T.Text\nformatCaptions captions (Url url) (Dir dir) = do\n let toMap = fromList $ captionsPerSlide captions\n let listy = toList (mapWithIndex (imgCaps url (T.pack dir)) toMap)\n intercalate \"\\n\" <$> sequence listy\n\nimgCaps :: Integral a => T.Text -> T.Text -> a -> [Caption] -> IO T.Text\nimgCaps url dir ind captions = do\n --let next = toInteger ind + 1\n img <- slideBlock url dir (toInteger ind) --next\n let toVideo = toVideoBlock url (toInteger ind)\n return (img <> caps captions <> toVideo <> \"
\")\n\nslideBlock :: T.Text -> T.Text -> Integer -> IO T.Text\nslideBlock url dir shot = do\n let imgPath = T.unpack (dir <> \"/glancer-img\" <> T.pack (printf \"%04d.jpg\" shot))\n (_, jpg, _) <- B.readProcessWithExitCode \"/bin/cat\" [imgPath] \"\"\n (_, base64, _) <- B.readProcessWithExitCode \"base64\" [] jpg\n let slide = \"
\\n\"\n let div = \"\\t
\\n\"\n let img = \"\\t\\t\\n\"\n let close = \"\\t
\\n\"\n let formatted = slide <> div <> img <> close\n return formatted\n\ntoVideoBlock :: T.Text -> Integer -> T.Text\ntoVideoBlock url shot = do\n let when = T.pack $ show (shotSeconds shot 30)\n let title = \"title='Go to video at timestamp \" <> when <> \"s'\"\n let diva = \"
title <> \"href='\" <> url <> \"&t=\" <> when <> \"s'>⇰
\"\n diva\n\ncaps :: [Caption] -> T.Text\ncaps captions = \"\\t
\\n\" <> intercalate \"\\n\" (stripped captions) <> \"\\n\\t
\"\n where\n stripped captions = map ((\"\\t\\t\" <>) . strip . txt) captions\n\ncaptionsPerSlide :: [Caption] -> [[Caption]]\ncaptionsPerSlide captions = map (capsForShot (filtering captions) 30) (shots (filtering captions))\n where\n filtering lst = filter (not . (T.isInfixOf \"\" . txt)) lst\n shots captions = [1 .. numShots captions 30 + 1]\n\nshotSeconds :: Integer -> Integer -> Integer\nshotSeconds shotNumber secsPerShot = shotNumber * secsPerShot\n\nhourMinuteSeconds :: Integer -> TimeDef\nhourMinuteSeconds seconds = TimeDef h m s 0\n where\n h = seconds `div` 3600\n m = seconds `mod` 3600 `div` 60\n s = seconds `mod` 3600 `mod` 60\n\nshotTime :: Integer -> Integer -> TimeBlock\nshotTime shotNumber secsPerShot = TimeBlock startTime endTime\n where\n startTime = hourMinuteSeconds start_\n endTime = hourMinuteSeconds end_\n start_ = shotSeconds (shotNumber - 1) secsPerShot\n end_ = shotSeconds shotNumber secsPerShot\n\nnumShots :: [Caption] -> Integer -> Integer\nnumShots caps secsPerShot = secs (end . block . last $ caps) `div` secsPerShot\n\ncapsForShot :: [Caption] -> Integer -> Integer -> [Caption]\ncapsForShot caps secsPerShot shotNumber = filter (inTimeBlock shotBlock) caps\n where\n shotBlock = shotTime shotNumber secsPerShot\n inTimeBlock blck cap = inBlock (start . block $ cap) blck && inBlock (end . block $ cap) blck\n" }, { "path": "src/Html.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n{-# LANGUAGE QuasiQuotes #-}\n\nmodule Html where\n\nimport qualified Data.Text as T\nimport qualified HTMLEntities.Text as H\nimport Process (Title (..), Url (..), Video (..))\nimport Text.RawString.QQ (r)\n\nheading :: T.Text\nheading =\n T.pack\n [r|\n\n\n\n\n \n \n \n\n|]\n\nembody :: Video -> T.Text -> T.Text\nembody (Video (Url url) (Title title) _) body =\n T.intercalate\n \"\\n\"\n [ \"\\t\",\n \"\\t\\t
\",\n \"\\t\\t\\t

\" <> H.text title <> \"

\",\n \"\\t\\t\\t

Created with glancer

\",\n body,\n \"\",\n \"\"\n ]\n" }, { "path": "src/Main.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n\nmodule Main where\n\nimport Captions (convertToHTML)\nimport Data.Coerce (coerce)\nimport Data.Text (pack)\nimport qualified Data.Text as T\nimport GHC.IO.Encoding (utf8)\nimport qualified Options.Applicative as Ap\nimport Parser (subsP)\nimport Process\n ( Dir (..),\n Filename (..),\n Url (..),\n Video (..),\n processURL,\n )\nimport System.Directory (getHomeDirectory)\nimport System.FilePath (joinPath, splitPath, (-<.>), ())\nimport System.IO\n ( IOMode (ReadMode),\n hGetContents,\n hPutStrLn,\n hSetEncoding,\n openFile,\n stderr,\n )\nimport Text.Megaparsec (parse)\nimport Prelude\n\ndata CLIConfig = CLIConfig\n { _url :: String,\n _filename :: String\n }\n\ncliConfig :: Ap.Parser CLIConfig\ncliConfig =\n CLIConfig\n <$> Ap.strArgument (Ap.metavar \"URL\" <> Ap.help \"Youtube URL\")\n <*> Ap.strArgument (Ap.metavar \"FILEPATH\" <> Ap.help \"HTML file name (don't add extension)\")\n\ngetFullPath :: FilePath -> IO FilePath\ngetFullPath s = case splitPath s of\n \"~/\" : t -> joinPath . (: t) <$> getHomeDirectory\n _ -> return s\n\nstart :: CLIConfig -> IO ()\nstart (CLIConfig url filename) = do\n hPutStrLn stderr (\"Looking for video in \" ++ url)\n (dir_, video) <- processURL (Url $ T.pack url)\n let videoName = coerce (file video)\n let dir = coerce dir_\n capsPath <- getFullPath (dir videoName -<.> \"en.vtt\")\n handle <- openFile capsPath ReadMode\n hSetEncoding handle utf8\n contents <- hGetContents handle\n let parsed = parse subsP \"\" (pack contents)\n html <- convertToHTML video dir_ parsed\n destinationPath <- getFullPath (filename -<.> \"html\")\n hPutStrLn stderr (\"Writing html to \" ++ destinationPath)\n writeFile destinationPath (T.unpack html)\n hPutStrLn stderr (\"Data written to \" ++ destinationPath)\n\nmain :: IO ()\nmain = do\n Main.start =<< Ap.execParser opts\n where\n opts =\n Ap.info\n (cliConfig Ap.<**> Ap.helper)\n ( Ap.fullDesc\n <> Ap.progDesc \"Glancer\"\n <> Ap.header \"Why not\"\n )\n" }, { "path": "src/Parser.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n\nmodule Parser where\n\nimport Control.Monad (void)\nimport qualified Data.Text as T\nimport Data.Void (Void)\nimport Text.Megaparsec\n ( MonadParsec (eof, lookAhead, try),\n Parsec,\n anySingle,\n manyTill,\n (<|>),\n )\nimport Text.Megaparsec.Char\n ( char,\n eol,\n numberChar,\n space,\n spaceChar,\n string,\n )\n\ntype Parser = Parsec Void T.Text\n\ndata TimeDef = TimeDef\n { hours :: Integer,\n minutes :: Integer,\n seconds :: Integer,\n cents :: Integer\n }\n deriving (Show, Eq)\n\nsecs :: TimeDef -> Integer\nsecs td = 3600 * hours td + 60 * minutes td + seconds td + 1\n\ninstance Ord TimeDef where\n compare a b = compare (secs a) (secs b)\n\ndata TimeBlock = TimeBlock\n { start :: TimeDef,\n end :: TimeDef\n }\n deriving (Show, Eq)\n\ninBlock :: TimeDef -> TimeBlock -> Bool\ninBlock td tb = (secs . start $ tb) <= secstd && secstd <= (secs . end $ tb)\n where\n secstd = secs td\n\ndata Caption = Caption\n { block :: TimeBlock,\n txt :: T.Text\n }\n deriving (Show, Eq)\n\nhourBlockP :: Parser TimeDef\nhourBlockP = do\n hours <- colonSeparated\n minutes <- colonSeparated\n seconds <- read <$> manyTill numberChar (char '.')\n millis <- read <$> manyTill numberChar (void spaceChar <|> try (lookAhead (void eol)))\n return (TimeDef hours minutes seconds millis)\n where\n colonSeparated = read <$> manyTill numberChar (char ':')\n\narrowP :: Parser T.Text\narrowP = do\n string \"-->\"\n\nanyChar :: Parser Char\nanyChar = anySingle\n\ntimeLineP :: Parser TimeBlock\ntimeLineP = do\n start <- hourBlockP\n arrowP\n space\n end <- hourBlockP\n manyTill anyChar eol\n return (TimeBlock start end)\n\ncaptionP :: Parser Caption\ncaptionP = do\n space\n block <- timeLineP\n caption <- T.pack <$> manyTill anyChar (try $ lookAhead (void timeLineP <|> void eof))\n return (Caption block caption)\n\nsubsP :: Parser [Caption]\nsubsP = do\n string \"WEBVTT\"\n manyTill anyChar (try $ lookAhead timeLineP)\n space\n manyTill captionP eof\n" }, { "path": "src/Process.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n\nmodule Process where\n\nimport Control.Monad (replicateM)\nimport qualified Data.ByteString.Char8 as B\nimport Data.Coerce (coerce)\nimport qualified Data.Text as T\nimport qualified Data.Text.Encoding as TE\nimport System.Directory (getTemporaryDirectory)\nimport System.FilePath ((-<.>), ())\nimport System.IO\n ( hPutStrLn,\n stderr,\n )\nimport System.Process (callCommand, callProcess, readProcess)\nimport System.Random (Random (randomRIO))\n\nnewtype Url = Url T.Text\n\nnewtype Title = Title T.Text\n\nnewtype Id = Id T.Text\n\nnewtype Dir = Dir String\n\nnewtype Filename = Filename String\n\ndata Video = Video\n { url :: Url,\n title :: Title,\n file :: Filename\n }\n\ngetTitle :: Url -> IO Title\ngetTitle (Url url) = do\n title <- readProcess \"yt-dlp\" [\"-e\", \"--no-warnings\", \"--no-playlist\", T.unpack url] []\n return (Title $ TE.decodeUtf8 $ B.pack title) -- This deletes badly encoded characters, which is better than having them but worse than properly encoding them. Help appreciated\n\ngetId :: Url -> IO Id\ngetId (Url url) = do\n id <- readProcess \"yt-dlp\" [\"--get-id\", \"--no-warnings\", \"--no-playlist\", T.unpack url] []\n return (Id $ T.pack id)\n\nyoutubeURL :: Id -> Url\nyoutubeURL (Id id) = Url (\"https://www.youtube.com/watch?v=\" <> id)\n\ngenerateVideo :: Video -> Dir -> IO ()\ngenerateVideo (Video (Url url) _ (Filename videoName)) (Dir dir) = do\n callProcess command arguments\n where\n command = \"yt-dlp\"\n arguments = [\"-q\", \"--no-playlist\", \"-f mp4\", coerce (\"-o\" <> dir videoName -<.> \"mp4\"), \"--sub-langs\", \"en\", \"--write-auto-sub\", \"--write-sub\", \"--no-warnings\", \"-k\", \"--no-cache-dir\", T.unpack url]\n\nargs :: Dir -> Filename -> [String] -> String -> [String]\nargs (Dir dir) (Filename videoName) selector suffix = [ \"-i\",\n coerce (dir videoName -<.> \"mp4\")]\n ++ selector ++ [\n coerce (dir \"glancer-img\" <> suffix -<.> \"jpg\"),\n \"-hide_banner\",\n \"-loglevel\",\n \"panic\"\n ]\n\ngenerateShots :: Dir -> Filename -> IO ()\ngenerateShots dir video = do\n callProcess \"ffmpeg\" (args dir video [\"-vf\", \"fps=1/30\"] \"%04d\")\n callProcess \"ffmpeg\" (args dir video [\"-vframes\", \"1\", \"-ss\", \"3\"] \"0000\" )\n\ndeleteVideo :: Dir -> Filename -> IO ()\ndeleteVideo (Dir dir) (Filename videoName) = callProcess \"rm\" [coerce (dir videoName -<.> \"mp4\")]\n\ndeleteImages :: Dir -> IO ()\ndeleteImages (Dir dir) = callCommand $ T.unpack (\"rm \" <> T.pack (dir \"glancer-img*\"))\n\nprocessURL :: Url -> IO (Dir, Video)\nprocessURL url = do\n dir <- Dir <$> getTemporaryDirectory\n videoName <- Filename <$> replicateM 10 (randomRIO ('a', 'z'))\n title <- getTitle url\n hPutStrLn stderr $ T.unpack (\"The video is titled '\" <> T.strip (coerce title) <> \"'\")\n id <- getId url\n let yourl = youtubeURL id\n hPutStrLn stderr $ T.unpack (T.strip (\"Seems like the video is in \" <> coerce yourl))\n let video = Video yourl title videoName\n hPutStrLn stderr \"Downloading video (this may take a while)\"\n generateVideo video dir\n hPutStrLn stderr (T.unpack (\"Downloaded video to \" <> (T.pack . coerce) dir <> (T.pack . coerce) videoName <> \"(.mp4|en.vtt)\"))\n hPutStrLn stderr \"Generating still images from video (this may take a while)\"\n generateShots dir videoName\n hPutStrLn stderr \"Generated images\"\n deleteVideo dir videoName\n return (dir, video)\n" }, { "path": "stack.yaml", "content": "# This file was automatically generated by 'stack init'\n#\n# Some commonly used options have been documented as comments in this file.\n# For advanced use and comprehensive documentation of the format, please see:\n# https://docs.haskellstack.org/en/stable/yaml_configuration/\n\n# Resolver to choose a 'specific' stackage snapshot or a compiler version.\n# A snapshot resolver dictates the compiler version and the set of packages\n# to be used for project dependencies. For example:\n#\n# resolver: lts-3.5\n# resolver: nightly-2015-09-21\n# resolver: ghc-7.10.2\n#\n# The location of a snapshot can be provided as a file or url. Stack assumes\n# a snapshot provided as a file might change, whereas a url resource does not.\n#\n# resolver: ./custom-snapshot.yaml\n# resolver: https://example.com/snapshots/2018-01-01.yaml\nresolver: lts-16.31\n\n\n# User packages to be built.\n# Various formats can be used as shown in the example below.\n#\n# packages:\n# - some-directory\n# - https://example.com/foo/bar/baz-0.0.2.tar.gz\n# subdirs:\n# - auto-update\n# - wai\npackages:\n- .\n# Dependency packages to be pulled from upstream that are not in the resolver.\n# These entries can reference officially published versions as well as\n# forks / in-progress versions pinned to a git hash. For example:\n#\n# extra-deps:\n# - acme-missiles-0.3\n# - git: https://github.com/commercialhaskell/stack.git\n# commit: e7b331f14bcffb8367cd58fbfc8b40ec7642100a\n#\n# extra-deps: []\n\n# Override default flag values for local packages and extra-deps\n# flags: {}\n\n# Extra package databases containing global packages\n# extra-package-dbs: []\n\n# Control whether we use the GHC we find on the path\n# system-ghc: true\n#\n# Require a specific version of stack, using version ranges\n# require-stack-version: -any # Default\n# require-stack-version: \">=2.3\"\n#\n# Override the architecture used by stack, especially useful on Windows\n# arch: i386\n# arch: x86_64\n#\n# Extra directories used by stack for building\n# extra-include-dirs: [/path/to/dir]\n# extra-lib-dirs: [/path/to/dir]\n#\n# Allow a newer minor version of GHC than the snapshot specifies\n# compiler-check: newer-minor\n" }, { "path": "test/Spec.hs", "content": "{-# LANGUAGE OverloadedStrings #-}\n{-# LANGUAGE QuasiQuotes #-}\n\nimport Data.Text (pack)\nimport qualified Parser as P\nimport Test.Hspec ( hspec, describe, it, Spec )\nimport Test.Hspec.Megaparsec ( shouldParse )\nimport Text.Megaparsec ( parse )\nimport Text.RawString.QQ ( r )\n\nmain :: IO ()\nmain = hspec spec\n\nhourBlock = \"01:07:46.029 \"\n\nparsedHourBlock = P.TimeDef 1 7 46 29\n\ntimeBlock =\n pack\n [r|01:07:46.029 --> 01:07:52.319 align:start position:0%\n|]\n\nparsedTimeBlock = P.TimeBlock parsedHourBlock (P.TimeDef 1 7 52 319)\n\ncaptionBlock =\n pack\n [r|\n01:07:46.029 --> 01:07:52.319 align:start position:0%\nreally look into but we have considered\n|]\n\nparsedCaptionBlock = P.Caption parsedTimeBlock \"really look into but we have considered\\n\"\n\nspec :: Spec\nspec = do\n describe \"arrowP\" $ do\n it \"parses a time arrow\" $\n parse P.arrowP \"\" \"-->\" `shouldParse` \"-->\"\n describe \"hourBlockP\" $ do\n it \"parses a hourBlock\" $\n parse P.hourBlockP \"\" hourBlock `shouldParse` parsedHourBlock\n describe \"timeBlockP\" $ do\n it \"parses a timeBlock\" $\n parse P.timeLineP \"\" timeBlock `shouldParse` parsedTimeBlock\n describe \"captionP\" $ do\n it \"parses a captionBlock\" $ parse P.captionP \"\" captionBlock `shouldParse` parsedCaptionBlock\n" }, { "path": "this.code-workspace", "content": "{\n\t\"folders\": [\n\t\t{\n\t\t\t\"path\": \".\"\n\t\t}\n\t],\n\t\"settings\": {\n\t\t\"files.watcherExclude\": {\n\t\t\t\"**/target\": true\n\t\t}\n\t}\n}" } ]