[ { "path": ".github/workflows/python-package.yml", "content": "# This workflow will install Python dependencies, run tests and lint with a variety of Python versions\n\nname: develop\n\non:\n push:\n branches: [ develop ]\n pull_request:\n branches: [ develop ]\n\njobs:\n build:\n\n runs-on: ubuntu-latest\n strategy:\n fail-fast: false\n matrix:\n python-version: [3.8, 3.9, '3.10', '3.11', '3.12', '3.13', pypy-3.8, pypy-3.9, pypy-3.10]\n\n steps:\n - uses: actions/checkout@v4\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python-version }}\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip setuptools\n pip install flake8\n if [ -f requirements.txt ]; then pip install -r requirements.txt; fi\n - name: Lint with flake8\n run: |\n # stop the build if there are Python syntax errors or undefined names\n flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics\n # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide\n flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics\n - name: Test with unittest\n run: |\n make test\n" }, { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*,cover\n.hypothesis/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Misc\n_*\n*_\n*.3gp\n*.asf\n*.download\n*.f4v\n*.flv\n*.gif\n*.html\n*.jpg\n*.lrc\n*.mkv\n*.mp3\n*.mp4\n*.mpg\n*.png\n*.srt\n*.ts\n*.webm\n*.xml\n*.json\n/.env\n/.idea\n*.m4a\n*.DS_Store\n*.txt\n*.sw[a-p]\n\n*.zip\n\n.emacs*\n.vscode\n" }, { "path": "CHANGELOG.rst", "content": "Changelog\n=========\n\n0.3.36\n------\n\n*Date: 2015-10-05*\n\n* New command-line option: --json\n* New site support:\n - Internet Archive\n* Bug fixes:\n - iQIYI\n - SoundCloud\n\n0.3.35\n------\n\n*Date: 2015-09-21*\n\n* New site support:\n - 755 http://7gogo.jp/ (via #659 by @soimort)\n - Funshion http://www.fun.tv/ (via #619 by @cnbeining)\n - iQilu http://v.iqilu.com/ (via #636 by @cnbeining)\n - Metacafe http://www.metacafe.com/ (via #620 by @cnbeining)\n - Qianmo http://qianmo.com/ (via #600 by @cnbeining)\n - Weibo Miaopai http://weibo.com/ (via #605 by @cnbeining)\n* Bug fixes:\n - 163 (by @lilydjwg)\n - CNTV (by @Red54)\n - Dailymotion (by @jackyzy823 and @ddumitran)\n - iQIYI (by @jackyzy823 and others)\n - QQ (by @soimort)\n - SoundCloud (by @soimort)\n - Tudou (by @CzBiX)\n - Vimeo channel (by @cnbeining)\n - YinYueTai (by @soimort)\n - Youku (by @junzh0u)\n - Embedded Youku/Tudou player (by @zhangn1985)\n\n0.3.34\n------\n\n*Date: 2015-07-12*\n\n* Bug fix release\n\n0.3.33\n------\n\n*Date: 2015-06-10*\n\n* Many bug fixes by our awesome contributors\n\n0.3.32\n------\n\n*Date: 2014-12-10*\n\n* New site support:\n - baomihua.com\n - zhanqi.tv\n* Bug fixes:\n - DouyuTV\n - Tudou\n - Tumblr\n - Vine\n - Youku\n\n0.3.31\n------\n\n*Date: 2014-11-01*\n\n* New site support:\n - Dongting (by @lilydjwg)\n - DouyuTV (by @0x00-pl)\n - LeTV cloud (by @cnbeining)\n* Bug fixes:\n - AcFun\n - Bilibili\n - Niconico\n - iQIYI\n\n0.3.30\n------\n\n*Date: 2014-09-21*\n\n* First Alpha release\n* Support PyPy3\n* Bug fixes:\n - YouTube\n - Youku\n - Tudou\n - Niconico\n - AcFun\n\n0.3.30dev-20140907\n------------------\n\n*Date: 2014-09-07*\n\n* Bug fixes:\n - AcFun\n - iQIYI\n - MioMio\n - QQ\n\n0.3.30dev-20140820\n------------------\n\n*Date: 2014-08-20*\n\n* Bug fix release\n\n0.3.30dev-20140812\n------------------\n\n*Date: 2014-08-12*\n\n* Bug fixes:\n - Youku\n* New site support:\n - VideoBam (by @cnbeining)\n\n0.3.30dev-20140806\n------------------\n\n*Date: 2014-08-06*\n\n* Bug fixes:\n - Youku\n - Nicovideo\n - Bilibili\n - Letv\n* New site support:\n - Tucao.cc\n* Use FFmpeg concat demuxer to join video segments (ffmpeg>=1.1)\n\n0.3.30dev-20140730\n------------------\n\n*Date: 2014-07-30*\n\n* YouTube: support fixed\n* Youku: password-protected video support\n\n0.3.30dev-20140723\n------------------\n\n*Date: 2014-07-23*\n\n* YouTube: (experimental) video format selection\n* Youku: playlist support\n* NetEase Music: high quality download (by @farseer90718)\n* PPTV: support fixed (by @jackyzy823)\n* Catfun.tv: new site support (by @jackyzy823)\n* AcFun.tv: domain name fixed\n\n0.3.30dev-20140716\n------------------\n\n*Date: 2014-07-16*\n\n* Bug fix release for:\n - YouTube\n - Youku\n\n* New site support: (by @jackyzy823)\n - MTV 81 http://www.mtv81.com\n - Kugou (酷狗音乐) http://www.kugou.com\n - Kuwo (酷我音乐) http://www.kuwo.cn\n - NetEase Music (网易云音乐) http://music.163.com\n\n0.3.30dev-20140629\n------------------\n\n*Date: 2014-06-29*\n\n* Bug fix release for:\n - Youku\n - YouTube\n - TED\n - Bilibili\n* (Experimental) Video format selection (for Youku only)\n\n0.3.29\n------\n\n*Date: 2014-05-29*\n\n* Bug fix release\n\n0.3.28.3\n--------\n\n*Date: 2014-05-18*\n\n* New site support:\n - CBS.com\n\n0.3.28.2\n--------\n\n*Date: 2014-04-13*\n\n* Bug fix release\n\n0.3.28.1\n--------\n\n*Date: 2014-02-28*\n\n* Bug fix release\n\n0.3.28\n------\n\n*Date: 2014-02-21*\n\n* New site support:\n - Magisto.com\n - VK.com\n\n0.3.27\n------\n\n*Date: 2014-02-14*\n\n* Bug fix release\n\n0.3.26\n------\n\n*Date: 2014-02-08*\n\n* New features:\n - Play video in players (#286)\n - LeTV support (#289)\n - Youku 1080P support\n* Bug fixes:\n - YouTube (#282, #292)\n - Sina (#246, #280)\n - Mixcloud\n - NetEase\n - QQ\n - Vine\n\n0.3.25\n------\n\n*Date: 2013-12-20*\n\n* Bug fix release\n\n0.3.24\n------\n\n*Date: 2013-10-30*\n\n* Experimental: Sogou proxy server\n* Fix issues for:\n - Vimeo\n\n0.3.23\n------\n\n*Date: 2013-10-23*\n\n* Support YouTube playlists\n* Support general short URLs\n* Fix issues for:\n - Sina\n\n0.3.22\n------\n\n*Date: 2013-10-18*\n\n* Fix issues for:\n - Baidu\n - Bilibili\n - JPopsuki TV\n - Niconico\n - PPTV\n - TED\n - Tumblr\n - YinYueTai\n - YouTube\n - ...\n\n0.3.21\n------\n\n*Date: 2013-08-17*\n\n* Fix issues for:\n - YouTube\n - YinYueTai\n - pan.baidu.com\n\n0.3.20\n------\n\n*Date: 2013-08-16*\n\n* Add support for:\n - eHow\n - Khan Academy\n - TED\n - 5sing\n* Fix issues for:\n - Tudou\n\n0.3.18\n------\n\n*Date: 2013-07-19*\n\n* Fix issues for:\n - Dailymotion\n - Youku\n - Sina\n - AcFun\n - bilibili\n\n0.3.17\n------\n\n*Date: 2013-07-12*\n\n* Fix issues for:\n - YouTube\n - 163\n - bilibili\n* Code cleanup.\n\n0.3.16\n------\n\n*Date: 2013-06-28*\n\n* Fix issues for:\n - YouTube\n - Sohu\n - Google+ (enable HTTPS proxy)\n\n0.3.15\n------\n\n*Date: 2013-06-21*\n\n* Add support for:\n - Instagram\n\n0.3.14\n------\n\n*Date: 2013-06-14*\n\n* Add support for:\n - Alive.in.th\n* Remove support of:\n - JPopsuki\n* Fix issues for:\n - AcFun\n - iQIYI\n\n0.3.13\n------\n\n*Date: 2013-06-07*\n\n* Add support for:\n - Baidu Wangpan (video only)\n* Fix issue for:\n - Google+\n\n0.3.12\n------\n\n*Date: 2013-05-19*\n\n* Fix issues for:\n - Google+\n - Mixcloud\n - Tudou\n\n0.3.11\n------\n\n*Date: 2013-04-26*\n\n* Add support for:\n - Google Drive (Google Docs)\n\n0.3.10\n------\n\n*Date: 2013-04-19*\n\n* Add support for:\n - SongTaste\n* Support Libav as well as FFmpeg.\n\n0.3.9\n-----\n\n*Date: 2013-04-12*\n\n* Add support for:\n - Freesound\n\n0.3.8\n-----\n\n*Date: 2013-04-05*\n\n* Add support for:\n - Coursera\n\n0.3.7\n-----\n\n*Date: 2013-03-29*\n\n* Add support for:\n - Baidu\n\n0.3.6\n-----\n\n*Date: 2013-03-22*\n\n* Add support for:\n - Vine\n* Fix issue for:\n - YouTube\n\n0.3.5\n-----\n\n*Date: 2013-03-15*\n\n* Default to use FFmpeg for merging .flv files.\n\n0.3.4\n-----\n\n*Date: 2013-03-08*\n\n* Add support for:\n - Blip\n - VID48\n\n0.3.3\n-----\n\n*Date: 2013-03-01*\n\n* Add support for:\n - Douban\n - MioMio\n* Fix issues for:\n - Tudou\n - Vimeo\n\n0.3.2\n-----\n\n*Date: 2013-02-22*\n\n* Add support for:\n - JPopsuki\n* Fix issue for Xiami.\n\n0.3.1\n-----\n\n*Date: 2013-02-15*\n\n* Fix issues for Google+ and Mixcloud.\n* API changed.\n\n0.3.0\n-----\n\n*Date: 2013-02-08*\n\n* Add support for:\n - Niconico\n\n0.3dev-20130201\n---------------\n\n*Date: 2013-02-01*\n\n* Add support for:\n - Mixcloud\n - Facebook\n - Joy.cn\n\n0.3dev-20130125\n---------------\n\n*Date: 2013-01-25*\n\n* Dailymotion: downloading best quality available now.\n* iQIYI: fix `#77 `_.\n\n0.3dev-20130118\n---------------\n\n*Date: 2013-01-18*\n\n* YinYueTai: downloading best quality available now.\n* Sohu: fix `#69 `_.\n\n0.3dev-20130111\n---------------\n\n*Date: 2013-01-11*\n\n* Add support for:\n - NetEase (v.163.com)\n - YouTube short URLs\n* Vimeo: downloading best quality available now.\n\n0.3dev-20130104\n---------------\n\n*Date: 2013-01-04*\n\n* Sohu:\n - fix `#53 `_.\n - merge pull request `#54 `_; downloading best quality available now.\n\n0.3dev-20121228\n---------------\n\n*Date: 2012-12-28*\n\n* Add support for:\n - Xiami\n - Tumblr audios\n\n0.3dev-20121221\n---------------\n\n*Date: 2012-12-21*\n\n* YouTube: fix `#45 `_.\n* Merge pull request `#46 `_; fix title parsing issue on Tudou.\n\n0.3dev-20121220\n---------------\n\n*Date: 2012-12-20*\n\n* YouTube: quick dirty fix to `#45 `_.\n\n0.3dev-20121219\n---------------\n\n*Date: 2012-12-19*\n\n* Add support for:\n - Tumblr\n\n0.3dev-20121217\n---------------\n\n*Date: 2012-12-17*\n\n* Google+: downloading best quality available now.\n* Fix issues `#42 `_, `#43 `_ for Google+.\n* Merge pull request `#40 `_; fix some issues for Ku6, Sina and 56.\n\n0.3dev-20121212\n---------------\n\n*Date: 2012-12-12*\n\n* YouTube: fix some major issues on parsing video titles.\n\n0.3dev-20121210\n---------------\n\n*Date: 2012-12-10*\n\n* YouTube: downloading best quality available now.\n* Add support for:\n - SoundCloud\n\n0.2.16\n------\n\n*Date: 2012-12-01*\n\n* Add support for:\n - QQ\n* Small fixes merged from youku-lixian.\n\n0.2.15\n------\n\n*Date: 2012-11-30*\n\n* Fix issue `#30 `_ for bilibili.\n\n0.2.14\n------\n\n*Date: 2012-11-29*\n\n* Fix issue `#28 `_ for Tudou.\n* Better support for AcFun.\n\n0.2.13\n------\n\n*Date: 2012-10-30*\n\n* Nothing new.\n\n0.2.12\n------\n\n*Date: 2012-10-30*\n\n* Fix issue `#20 `_ for AcFun.\n\n0.2.11\n------\n\n*Date: 2012-10-23*\n\n* Move on to Python 3.3!\n* Fix issues:\n - `#17 `_\n - `#18 `_\n - `#19 `_\n\n0.2.10\n------\n\n*Date: 2012-10-16*\n\n* Add support for:\n - Google+\n\n0.2.9\n-----\n\n*Date: 2012-10-09*\n\n* Fix issue `#16 `_.\n\n0.2.8\n-----\n\n*Date: 2012-10-02*\n\n* Fix issue `#15 `_ for AcFun.\n\n0.2.7\n-----\n\n*Date: 2012-09-28*\n\n* Fix issue `#6 `_ for YouTube.\n\n0.2.6\n-----\n\n*Date: 2012-09-26*\n\n* Fix issue `#5 `_ for YinYueTai.\n\n0.2.5\n-----\n\n*Date: 2012-09-25*\n\n* Add support for:\n - Dailymotion\n\n0.2.4\n-----\n\n*Date: 2012-09-18*\n\n* Use FFmpeg for converting and joining video files.\n* Add '--url' and '--debug' options.\n\n0.2.2\n-----\n\n*Date: 2012-09-17*\n\n* Add danmaku support for AcFun and bilibili.\n* Fix issue `#2 `_ and `#4 `_ for YouTube.\n* Temporarily fix issue for iQIYI (use .ts instead of .f4v).\n\n0.2.1\n-----\n\n*Date: 2012-09-02*\n\n* Add support for:\n - ifeng\n\n0.2\n---\n\n*Date: 2012-09-02*\n\n* Add support for:\n - Vimeo\n - AcFun\n - bilibili\n - CNTV\n - iQIYI\n - Ku6\n - PPTV\n - Sina\n - Sohu\n - 56\n\n0.1.3\n-----\n\n*Date: 2012-09-01*\n\n* Playlist URLs are now automatically handled. ('--playlist' option is no longer needed)\n* Handle KeyboardInterrupt silently.\n* Fix Unicode character display on code pages.\n\n0.1\n---\n\n*Date: 2012-09-01*\n\n* First PyPI release.\n* Fix issue `#1 `_.\n\n0.0.1\n-----\n\n*Date: 2012-08-21*\n\n* Initial release, forked from `iambus/youku-lixian `_; add:\n - YouTube support.\n - Pausing and resuming of downloads.\n - HTTP proxy settings.\n" }, { "path": "CONTRIBUTING.md", "content": "# How to Report an Issue\n\nIf you would like to report a problem you find when using `you-get`, please open a [Pull Request](https://github.com/soimort/you-get/pulls), which should include:\n\n1. A detailed description of the encountered problem;\n2. At least one commit, addressing the problem through some unit test(s).\n * Examples of good commits: [#2675](https://github.com/soimort/you-get/pull/2675/files), [#2680](https://github.com/soimort/you-get/pull/2680/files), [#2685](https://github.com/soimort/you-get/pull/2685/files)\n\nPRs that fail to meet the above criteria may be closed summarily with no further action.\n\nA valid PR will remain open until its addressed problem is fixed.\n\n\n\n# 如何汇报问题\n\n为了防止对 GitHub Issues 的滥用，本项目不接受一般的 Issue。\n\n如您在使用 `you-get` 的过程中发现任何问题，请开启一个 [Pull Request](https://github.com/soimort/you-get/pulls)。该 PR 应当包含：\n\n1. 详细的问题描述；\n2. 至少一个 commit，其内容是**与问题相关的**单元测试。**不要通过随意修改无关文件的方式来提交 PR！**\n * 有效的 commit 示例：[#2675](https://github.com/soimort/you-get/pull/2675/files), [#2680](https://github.com/soimort/you-get/pull/2680/files), [#2685](https://github.com/soimort/you-get/pull/2685/files)\n\n不符合以上条件的 PR 可能被直接关闭。\n\n有效的 PR 将会被一直保留，直至相应的问题得以修复。\n" }, { "path": "MANIFEST.in", "content": "include *.rst\ninclude *.txt\ninclude Makefile\ninclude CONTRIBUTING.md\ninclude README.md\ninclude you-get\ninclude you-get.json\ninclude you-get.plugin.zsh\nrecursive-include contrib *\n" }, { "path": "Makefile", "content": ".PHONY: default i test clean all html rst build install release\n\ndefault: i\n\ni:\n\t@(cd src; python -i -c 'import you_get; print(\"You-Get %s\\n>>> import you_get\" % you_get.version.__version__)')\n\ntest:\n\t(cd src; python -m unittest discover -s ../tests)\n\nclean:\n\tzenity --question\n\trm -fr build/ dist/ src/*.egg-info/\n\tfind . | grep __pycache__ | xargs rm -fr\n\tfind . | grep .pyc | xargs rm -f\n\nall: build\n\nhtml:\n\tpandoc README.md > README.html\n\nrst:\n\tpandoc -s -t rst README.md > README.rst\n\nbuild:\n\tpython -m build\n\ninstall:\n\tpython -m pip install .\n\nrelease: build\n\t@echo 'Upload new version to PyPI using:'\n\t@echo '\ttwine upload --sign dist/you_get-VERSION*'\n" }, { "path": "README.md", "content": "# You-Get\n\n[![Build Status](https://github.com/soimort/you-get/workflows/develop/badge.svg)](https://github.com/soimort/you-get/actions)\n[![PyPI version](https://img.shields.io/pypi/v/you-get.svg)](https://pypi.python.org/pypi/you-get/)\n[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/soimort/you-get?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n\n**NOTICE (30 May 2022): Support for Python 3.5, 3.6 and 3.7 will eventually be dropped. ([see details here](https://github.com/soimort/you-get/wiki/TLS-1.3-post-handshake-authentication-(PHA)))**\n\n**NOTICE (8 Mar 2019): Read [this](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md) if you are looking for the conventional \"Issues\" tab.**\n\n---\n\n[You-Get](https://you-get.org/) is a tiny command-line utility to download media contents (videos, audios, images) from the Web, in case there is no other handy way to do it.\n\nHere's how you use `you-get` to download a video from [YouTube](https://www.youtube.com/watch?v=jNQXAC9IVRw):\n\n```console\n$ you-get 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\nsite: YouTube\ntitle: Me at the zoo\nstream:\n - itag: 43\n container: webm\n quality: medium\n size: 0.5 MiB (564215 bytes)\n # download-with: you-get --itag=43 [URL]\n\nDownloading Me at the zoo.webm ...\n 100% ( 0.5/ 0.5MB) ├██████████████████████████████████┤[1/1] 6 MB/s\n\nSaving Me at the zoo.en.srt ... Done.\n```\n\nAnd here's why you might want to use it:\n\n* You enjoyed something on the Internet, and just want to download them for your own pleasure.\n* You watch your favorite videos online from your computer, but you are prohibited from saving them. You feel that you have no control over your own computer. (And it's not how an open Web is supposed to work.)\n* You want to get rid of any closed-source technology or proprietary JavaScript code, and disallow things like Flash running on your computer.\n* You are an adherent of hacker culture and free software.\n\nWhat `you-get` can do for you:\n\n* Download videos / audios from popular websites such as YouTube, Youku, Niconico, and a bunch more. (See the [full list of supported sites](#supported-sites))\n* Stream an online video in your media player. No web browser, no more ads.\n* Download images (of interest) by scraping a web page.\n* Download arbitrary non-HTML contents, i.e., binary files.\n\nInterested? [Install it](#installation) now and [get started by examples](#getting-started).\n\nAre you a Python programmer? Then check out [the source](https://github.com/soimort/you-get) and fork it!\n\n![](https://i.imgur.com/GfthFAz.png)\n\n## Installation\n\n### Prerequisites\n\nThe following dependencies are recommended:\n\n* **[Python](https://www.python.org/downloads/)** 3.7.4 or above\n* **[FFmpeg](https://www.ffmpeg.org/)** 1.0 or above\n* (Optional) [RTMPDump](https://rtmpdump.mplayerhq.hu/)\n\n### Option 1: Install via pip\n\nThe official release of `you-get` is distributed on [PyPI](https://pypi.python.org/pypi/you-get), and can be installed easily from a PyPI mirror via the [pip](https://en.wikipedia.org/wiki/Pip_\\(package_manager\\)) package manager: (Note that you must use the Python 3 version of `pip`)\n\n $ pip install you-get\n\n### Option 2: Install via [Antigen](https://github.com/zsh-users/antigen) (for Zsh users)\n\nAdd the following line to your `.zshrc`:\n\n antigen bundle soimort/you-get\n\n### Option 3: Download from GitHub\n\nYou may either download the [stable](https://github.com/soimort/you-get/archive/master.zip) (identical with the latest release on PyPI) or the [develop](https://github.com/soimort/you-get/archive/develop.zip) (more hotfixes, unstable features) branch of `you-get`. Unzip it, and put the directory containing the `you-get` script into your `PATH`.\n\nAlternatively, run\n\n```\n$ cd path/to/you-get\n$ [sudo] python -m pip install .\n```\n\nOr\n\n```\n$ cd path/to/you-get\n$ python -m pip install . --user\n```\n\nto install `you-get` to a permanent path. (And don't omit the dot `.` representing the current directory)\n\nYou can also use the [pipenv](https://pipenv.pypa.io/en/latest) to install the `you-get` in the Python virtual environment.\n\n```\n$ pipenv install -e .\n$ pipenv run you-get --version\nyou-get: version 0.4.1555, a tiny downloader that scrapes the web.\n```\n\n### Option 4: Git clone\n\nThis is the recommended way for all developers, even if you don't often code in Python.\n\n```\n$ git clone git://github.com/soimort/you-get.git\n```\n\nThen put the cloned directory into your `PATH`, or run `python -m pip install path/to/you-get` to install `you-get` to a permanent path.\n\n### Option 5: Homebrew (Mac only)\n\nYou can install `you-get` easily via:\n\n```\n$ brew install you-get\n```\n\n### Option 6: pkg (FreeBSD only)\n\nYou can install `you-get` easily via:\n\n```\n# pkg install you-get\n```\n\n### Option 7: Flox (Mac, Linux, and Windows WSL)\n\nYou can install `you-get` easily via:\n\n```\n$ flox install you-get\n```\n\n### Shell completion\n\nCompletion definitions for Bash, Fish and Zsh can be found in [`contrib/completion`](https://github.com/soimort/you-get/tree/develop/contrib/completion). Please consult your shell's manual for how to take advantage of them.\n\n## Upgrading\n\nBased on which option you chose to install `you-get`, you may upgrade it via:\n\n```\n$ pip install --upgrade you-get\n```\n\nor download the latest release via:\n\n```\n$ you-get https://github.com/soimort/you-get/archive/master.zip\n```\n\nIn order to get the latest ```develop``` branch without messing up the PIP, you can try:\n\n```\n$ pip install --upgrade --force-reinstall git+https://github.com/soimort/you-get@develop\n```\n\n## Getting Started\n\n### Download a video\n\nWhen you get a video of interest, you might want to use the `--info`/`-i` option to see all available quality and formats:\n\n```\n$ you-get -i 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\nsite: YouTube\ntitle: Me at the zoo\nstreams: # Available quality and codecs\n [ DASH ] ____________________________________\n - itag: 242\n container: webm\n quality: 320x240\n size: 0.6 MiB (618358 bytes)\n # download-with: you-get --itag=242 [URL]\n\n - itag: 395\n container: mp4\n quality: 320x240\n size: 0.5 MiB (550743 bytes)\n # download-with: you-get --itag=395 [URL]\n\n - itag: 133\n container: mp4\n quality: 320x240\n size: 0.5 MiB (498558 bytes)\n # download-with: you-get --itag=133 [URL]\n\n - itag: 278\n container: webm\n quality: 192x144\n size: 0.4 MiB (392857 bytes)\n # download-with: you-get --itag=278 [URL]\n\n - itag: 160\n container: mp4\n quality: 192x144\n size: 0.4 MiB (370882 bytes)\n # download-with: you-get --itag=160 [URL]\n\n - itag: 394\n container: mp4\n quality: 192x144\n size: 0.4 MiB (367261 bytes)\n # download-with: you-get --itag=394 [URL]\n\n [ DEFAULT ] _________________________________\n - itag: 43\n container: webm\n quality: medium\n size: 0.5 MiB (568748 bytes)\n # download-with: you-get --itag=43 [URL]\n\n - itag: 18\n container: mp4\n quality: small\n # download-with: you-get --itag=18 [URL]\n\n - itag: 36\n container: 3gp\n quality: small\n # download-with: you-get --itag=36 [URL]\n\n - itag: 17\n container: 3gp\n quality: small\n # download-with: you-get --itag=17 [URL]\n```\n\nBy default, the one on the top is the one you will get. If that looks cool to you, download it:\n\n```\n$ you-get 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\nsite: YouTube\ntitle: Me at the zoo\nstream:\n - itag: 242\n container: webm\n quality: 320x240\n size: 0.6 MiB (618358 bytes)\n # download-with: you-get --itag=242 [URL]\n\nDownloading Me at the zoo.webm ...\n 100% ( 0.6/ 0.6MB) ├██████████████████████████████████████████████████████████████████████████████┤[2/2] 2 MB/s\nMerging video parts... Merged into Me at the zoo.webm\n\nSaving Me at the zoo.en.srt ... Done.\n```\n\n(If a YouTube video has any closed captions, they will be downloaded together with the video file, in SubRip subtitle format.)\n\nOr, if you prefer another format (mp4), just use whatever the option `you-get` shows to you:\n\n```\n$ you-get --itag=18 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\n```\n\n**Note:**\n\n* At this point, format selection has not been generally implemented for most of our supported sites; in that case, the default format to download is the one with the highest quality.\n* `ffmpeg` is a required dependency, for downloading and joining videos streamed in multiple parts (e.g. on some sites like Youku), and for YouTube videos of 1080p or high resolution.\n* If you don't want `you-get` to join video parts after downloading them, use the `--no-merge`/`-n` option.\n\n### Download anything else\n\nIf you already have the URL of the exact resource you want, you can download it directly with:\n\n```\n$ you-get https://stallman.org/rms.jpg\nSite: stallman.org\nTitle: rms\nType: JPEG Image (image/jpeg)\nSize: 0.06 MiB (66482 Bytes)\n\nDownloading rms.jpg ...\n 100% ( 0.1/ 0.1MB) ├████████████████████████████████████████┤[1/1] 127 kB/s\n```\n\nOtherwise, `you-get` will scrape the web page and try to figure out if there's anything interesting to you:\n\n```\n$ you-get https://kopasas.tumblr.com/post/69361932517\nSite: Tumblr.com\nTitle: [tumblr] tumblr_mxhg13jx4n1sftq6do1_640\nType: Portable Network Graphics (image/png)\nSize: 0.11 MiB (118484 Bytes)\n\nDownloading [tumblr] tumblr_mxhg13jx4n1sftq6do1_640.png ...\n 100% ( 0.1/ 0.1MB) ├████████████████████████████████████████┤[1/1] 22 MB/s\n```\n\n**Note:**\n\n* This feature is an experimental one and far from perfect. It works best on scraping large-sized images from popular websites like Tumblr and Blogger, but there is really no universal pattern that can apply to any site on the Internet.\n\n### Search on Google Videos and download\n\nYou can pass literally anything to `you-get`. If it isn't a valid URL, `you-get` will do a Google search and download the most relevant video for you. (It might not be exactly the thing you wish to see, but still very likely.)\n\n```\n$ you-get \"Richard Stallman eats\"\n```\n\n### Pause and resume a download\n\nYou may use Ctrl+C to interrupt a download.\n\nA temporary `.download` file is kept in the output directory. Next time you run `you-get` with the same arguments, the download progress will resume from the last session. In case the file is completely downloaded (the temporary `.download` extension is gone), `you-get` will just skip the download.\n\nTo enforce re-downloading, use the `--force`/`-f` option. (**Warning:** doing so will overwrite any existing file or temporary file with the same name!)\n\n### Set the path and name of downloaded file\n\nUse the `--output-dir`/`-o` option to set the path, and `--output-filename`/`-O` to set the name of the downloaded file:\n\n```\n$ you-get -o ~/Videos -O zoo.webm 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\n```\n\n**Tips:**\n\n* These options are helpful if you encounter problems with the default video titles, which may contain special characters that do not play well with your current shell / operating system / filesystem.\n* These options are also helpful if you write a script to batch download files and put them into designated folders with designated names.\n\n### Proxy settings\n\nYou may specify an HTTP proxy for `you-get` to use, via the `--http-proxy`/`-x` option:\n\n```\n$ you-get -x 127.0.0.1:8087 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\n```\n\nHowever, the system proxy setting (i.e. the environment variable `http_proxy`) is applied by default. To disable any proxy, use the `--no-proxy` option.\n\n**Tips:**\n\n* If you need to use proxies a lot (in case your network is blocking certain sites), you might want to use `you-get` with [proxychains](https://github.com/rofl0r/proxychains-ng) and set `alias you-get=\"proxychains -q you-get\"` (in Bash).\n* For some websites (e.g. Youku), if you need access to some videos that are only available in mainland China, there is an option of using a specific proxy to extract video information from the site: `--extractor-proxy`/`-y`.\n\n### Watch a video\n\nUse the `--player`/`-p` option to feed the video into your media player of choice, e.g. `mpv` or `vlc`, instead of downloading it:\n\n```\n$ you-get -p vlc 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\n```\n\nOr, if you prefer to watch the video in a browser, just without ads or comment section:\n\n```\n$ you-get -p chromium 'https://www.youtube.com/watch?v=jNQXAC9IVRw'\n```\n\n**Tips:**\n\n* It is possible to use the `-p` option to start another download manager, e.g., `you-get -p uget-gtk 'https://www.youtube.com/watch?v=jNQXAC9IVRw'`, though they may not play together very well.\n\n### Load cookies\n\nNot all videos are publicly available to anyone. If you need to log in your account to access something (e.g., a private video), it would be unavoidable to feed the browser cookies to `you-get` via the `--cookies`/`-c` option.\n\n**Note:**\n\n* As of now, we are supporting two formats of browser cookies: Mozilla `cookies.sqlite` and Netscape `cookies.txt`.\n\n### Reuse extracted data\n\nUse `--url`/`-u` to get a list of downloadable resource URLs extracted from the page. Use `--json` to get an abstract of extracted data in the JSON format.\n\n**Warning:**\n\n* For the time being, this feature has **NOT** been stabilized and the JSON schema may have breaking changes in the future.\n\n## Supported Sites\n\n| Site | URL | Videos? | Images? | Audios? |\n| :--: | :-- | :-----: | :-----: | :-----: |\n| **YouTube** | |✓| | |\n| **X (Twitter)** | |✓|✓| |\n| VK | |✓|✓| |\n| Vimeo | |✓| | |\n| Veoh | |✓| | |\n| **Tumblr** | |✓|✓|✓|\n| TED | |✓| | |\n| SoundCloud | | | |✓|\n| SHOWROOM | |✓| | |\n| Pinterest | | |✓| |\n| MTV81 | |✓| | |\n| Mixcloud | | | |✓|\n| Metacafe | |✓| | |\n| Magisto | |✓| | |\n| Khan Academy | |✓| | |\n| Internet Archive | |✓| | |\n| **Instagram** | |✓|✓| |\n| InfoQ | |✓| | |\n| Imgur | | |✓| |\n| Heavy Music Archive | | | |✓|\n| Freesound | | | |✓|\n| Flickr | |✓|✓| |\n| FC2 Video | |✓| | |\n| Facebook | |✓| | |\n| eHow | |✓| | |\n| Dailymotion | |✓| | |\n| Coub | |✓| | |\n| CBS | |✓| | |\n| Bandcamp | | | |✓|\n| AliveThai | |✓| | |\n| interest.me | |✓| | |\n| **755
ナナゴーゴー** | |✓|✓| |\n| **niconico
ニコニコ動画** | |✓| | |\n| **163
网易视频
网易云音乐** |
|✓| |✓|\n| 56网 | |✓| | |\n| **AcFun** | |✓| | |\n| **Baidu
百度贴吧** | |✓|✓| |\n| 爆米花网 | |✓| | |\n| **bilibili
哔哩哔哩** | |✓|✓|✓|\n| 豆瓣 | |✓| |✓|\n| 斗鱼 | |✓| | |\n| 凤凰视频 | |✓| | |\n| 风行网 | |✓| | |\n| iQIYI
爱奇艺 | |✓| | |\n| 激动网 | |✓| | |\n| 酷6网 | |✓| | |\n| 酷狗音乐 | | | |✓|\n| 酷我音乐 | | | |✓|\n| 乐视网 | |✓| | |\n| 荔枝FM | | | |✓|\n| 懒人听书 | | | |✓|\n| 秒拍 | |✓| | |\n| MioMio弹幕网 | |✓| | |\n| MissEvan
猫耳FM | | | |✓|\n| 痞客邦 | |✓| | |\n| PPTV聚力 | |✓| | |\n| 齐鲁网 | |✓| | |\n| QQ
腾讯视频 | |✓| | |\n| 企鹅直播 | |✓| | |\n| Sina
新浪视频
微博秒拍视频 |
|✓| | |\n| Sohu
搜狐视频 | |✓| | |\n| **Tudou
土豆** | |✓| | |\n| 阳光卫视 | |✓| | |\n| **Youku
优酷** | |✓| | |\n| 战旗TV | |✓| | |\n| 央视网 | |✓| | |\n| Naver
네이버 | |✓| | |\n| 芒果TV | |✓| | |\n| 火猫TV | |✓| | |\n| 阳光宽频网 | |✓| | |\n| 西瓜视频 | |✓| | |\n| 新片场 | |✓| | |\n| 快手 | |✓|✓| |\n| 抖音 | |✓| | |\n| TikTok | |✓| | |\n| 中国体育(TV) |
|✓| | |\n| 知乎 | |✓| | |\n\nFor all other sites not on the list, the universal extractor will take care of finding and downloading interesting resources from the page.\n\n### Known bugs\n\nIf something is broken and `you-get` can't get you things you want, don't panic. (Yes, this happens all the time!)\n\nCheck if it's already a known problem on . If not, follow the guidelines on [how to report an issue](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md).\n\n## Getting Involved\n\nYou can reach us on the Gitter channel [#soimort/you-get](https://gitter.im/soimort/you-get) (here's how you [set up your IRC client](https://irc.gitter.im) for Gitter). If you have a quick question regarding `you-get`, ask it there.\n\nIf you are seeking to report an issue or contribute, please make sure to read [the guidelines](https://github.com/soimort/you-get/blob/develop/CONTRIBUTING.md) first.\n\n## Legal Issues\n\nThis software is distributed under the [MIT license](https://raw.github.com/soimort/you-get/master/LICENSE.txt).\n\nIn particular, please be aware that\n\n> THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\nTranslated to human words:\n\n*In case your use of the software forms the basis of copyright infringement, or you use the software for any other illegal purposes, the authors cannot take any responsibility for you.*\n\nWe only ship the code here, and how you are going to use it is left to your own discretion.\n\n## Authors\n\nMade by [@soimort](https://github.com/soimort), who is in turn powered by :coffee:, :beer: and :ramen:.\n\nYou can find the [list of all contributors](https://github.com/soimort/you-get/graphs/contributors) here.\n" }, { "path": "README.rst", "content": "You-Get\n=======\n\n|PyPI version| |Build Status| |Gitter|\n\n`You-Get `__ is a tiny command-line utility to\ndownload media contents (videos, audios, images) from the Web, in case\nthere is no other handy way to do it.\n\nHere's how you use ``you-get`` to download a video from `this web\npage `__:\n\n.. code:: console\n\n $ you-get http://www.fsf.org/blogs/rms/20140407-geneva-tedx-talk-free-software-free-society\n Site: fsf.org\n Title: TEDxGE2014_Stallman05_LQ\n Type: WebM video (video/webm)\n Size: 27.12 MiB (28435804 Bytes)\n\n Downloading TEDxGE2014_Stallman05_LQ.webm ...\n 100.0% ( 27.1/27.1 MB) ├████████████████████████████████████████┤[1/1] 12 MB/s\n\nAnd here's why you might want to use it:\n\n- You enjoyed something on the Internet, and just want to download them\n for your own pleasure.\n- You watch your favorite videos online from your computer, but you are\n prohibited from saving them. You feel that you have no control over\n your own computer. (And it's not how an open Web is supposed to\n work.)\n- You want to get rid of any closed-source technology or proprietary\n JavaScript code, and disallow things like Flash running on your\n computer.\n- You are an adherent of hacker culture and free software.\n\nWhat ``you-get`` can do for you:\n\n- Download videos / audios from popular websites such as YouTube,\n Youku, Niconico, and a bunch more. (See the `full list of supported\n sites <#supported-sites>`__)\n- Stream an online video in your media player. No web browser, no more\n ads.\n- Download images (of interest) by scraping a web page.\n- Download arbitrary non-HTML contents, i.e., binary files.\n\nInterested? `Install it <#installation>`__ now and `get started by\nexamples <#getting-started>`__.\n\nAre you a Python programmer? Then check out `the\nsource `__ and fork it!\n\n.. |PyPI version| image:: https://badge.fury.io/py/you-get.png\n :target: http://badge.fury.io/py/you-get\n.. |Build Status| image:: https://github.com/soimort/you-get/workflows/develop/badge.svg\n :target: https://github.com/soimort/you-get/actions\n.. |Gitter| image:: https://badges.gitter.im/Join%20Chat.svg\n :target: https://gitter.im/soimort/you-get?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge\n" }, { "path": "SECURITY.md", "content": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease report security issues to .\n" }, { "path": "contrib/completion/you-get-completion.bash", "content": "# Bash completion definition for you-get.\n\n_you-get () {\n COMPREPLY=()\n local IFS=$' \\n'\n local cur=$2 prev=$3\n local -a opts_without_arg opts_with_arg\n opts_without_arg=(\n -V --version -h --help -i --info -u --url --json -n --no-merge\n --no-caption -f --force --no-proxy -d --debug\n )\n opts_with_arg=(\n -F --format -O --output-filename -o --output-dir -p --player\n -c --cookies -x --http-proxy -y --extractor-proxy -t --timeout\n )\n\n # Do not complete non option names\n [[ $cur == -* ]] || return 1\n\n # Do not complete when the previous arg is an option expecting an argument\n for opt in \"${opts_with_arg[@]}\"; do\n [[ $opt == $prev ]] && return 1\n done\n\n # Complete option names\n COMPREPLY=( $(compgen -W \"${opts_without_arg[*]} ${opts_with_arg[*]}\" \\\n -- \"$cur\") )\n return 0\n}\n\ncomplete -F _you-get you-get\n" }, { "path": "contrib/completion/you-get.fish", "content": "# Fish completion definition for you-get.\n\ncomplete -c you-get -s V -l version -d 'print version and exit'\ncomplete -c you-get -s h -l help -d 'print help and exit'\ncomplete -c you-get -s i -l info -d 'print extracted information'\ncomplete -c you-get -s u -l url -d 'print extracted information'\ncomplete -c you-get -l json -d 'print extracted URLs in JSON format'\ncomplete -c you-get -s n -l no-merge -d 'do not merge video parts'\ncomplete -c you-get -l no-caption -d 'do not download captions'\ncomplete -c you-get -s f -l force -d 'force overwrite existing files'\ncomplete -c you-get -s F -l format -x -d 'set video format to the specified stream id'\ncomplete -c you-get -s O -l output-filename -d 'set output filename' \\\n -x -a '(__fish_complete_path (commandline -ct) \"output filename\")'\ncomplete -c you-get -s o -l output-dir -d 'set output directory' \\\n -x -a '(__fish_complete_directories (commandline -ct) \"output directory\")'\ncomplete -c you-get -s p -l player -x -d 'stream extracted URL to the specified player'\ncomplete -c you-get -s c -l cookies -d 'load cookies.txt or cookies.sqlite' \\\n -x -a '(__fish_complete_path (commandline -ct) \"cookies.txt or cookies.sqlite\")'\ncomplete -c you-get -s x -l http-proxy -x -d 'use the specified HTTP proxy for downloading'\ncomplete -c you-get -s y -l extractor-proxy -x -d 'use the specified HTTP proxy for extraction only'\ncomplete -c you-get -l no-proxy -d 'do not use a proxy'\ncomplete -c you-get -s t -l timeout -x -d 'set socket timeout'\ncomplete -c you-get -s d -l debug -d 'show traceback and other debug info'\n" }, { "path": "setup.cfg", "content": "[build]\nforce = 0\n\n[global]\nverbose = 0\n\n[egg_info]\ntag_build = \ntag_date = 0\ntag_svn_revision = 0\n" }, { "path": "setup.py", "content": "#!/usr/bin/env python3\n\nPROJ_NAME = 'you-get'\nPACKAGE_NAME = 'you_get'\n\nPROJ_METADATA = '%s.json' % PROJ_NAME\n\nimport importlib.util\nimport importlib.machinery\n\ndef load_source(modname, filename):\n loader = importlib.machinery.SourceFileLoader(modname, filename)\n spec = importlib.util.spec_from_file_location(modname, filename, loader=loader)\n module = importlib.util.module_from_spec(spec)\n # The module is always executed and not cached in sys.modules.\n # Uncomment the following line to cache the module.\n # sys.modules[module.__name__] = module\n loader.exec_module(module)\n return module\n\nimport os, json\nhere = os.path.abspath(os.path.dirname(__file__))\nproj_info = json.loads(open(os.path.join(here, PROJ_METADATA), encoding='utf-8').read())\ntry:\n README = open(os.path.join(here, 'README.rst'), encoding='utf-8').read()\nexcept:\n README = \"\"\nCHANGELOG = open(os.path.join(here, 'CHANGELOG.rst'), encoding='utf-8').read()\nVERSION = load_source('version', os.path.join(here, 'src/%s/version.py' % PACKAGE_NAME)).__version__\n\nfrom setuptools import setup, find_packages\nsetup(\n name = proj_info['name'],\n version = VERSION,\n\n author = proj_info['author'],\n author_email = proj_info['author_email'],\n url = proj_info['url'],\n license = proj_info['license'],\n\n description = proj_info['description'],\n keywords = proj_info['keywords'],\n\n long_description = README,\n\n packages = find_packages('src'),\n package_dir = {'' : 'src'},\n\n test_suite = 'tests',\n\n platforms = 'any',\n zip_safe = True,\n include_package_data = True,\n\n classifiers = proj_info['classifiers'],\n\n entry_points = {'console_scripts': proj_info['console_scripts']},\n\n install_requires = ['dukpy'],\n extras_require = {\n 'socks': ['PySocks'],\n }\n)\n" }, { "path": "src/you_get/cli_wrapper/player/dragonplayer.py", "content": "" }, { "path": "src/you_get/cli_wrapper/player/gnome_mplayer.py", "content": "" }, { "path": "src/you_get/cli_wrapper/player/mplayer.py", "content": "" }, { "path": "src/you_get/cli_wrapper/player/vlc.py", "content": "#!/usr/bin/env python\n" }, { "path": "src/you_get/cli_wrapper/player/wmp.py", "content": "" }, { "path": "src/you_get/cli_wrapper/transcoder/ffmpeg.py", "content": "" }, { "path": "src/you_get/cli_wrapper/transcoder/libav.py", "content": "" }, { "path": "src/you_get/cli_wrapper/transcoder/mencoder.py", "content": "" }, { "path": "src/you_get/common.py", "content": "#!/usr/bin/env python\n\nimport io\nimport os\nimport re\nimport sys\nimport time\nimport json\nimport socket\nimport locale\nimport logging\nimport argparse\nimport ssl\nfrom http import cookiejar\nfrom importlib import import_module\nfrom urllib import request, parse, error\n\nfrom .version import __version__\nfrom .util import log, term\nfrom .util.git import get_version\nfrom .util.strings import get_filename, unescape_html\nfrom . import json_output as json_output_\nsys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8')\n\nSITES = {\n '163' : 'netease',\n '56' : 'w56',\n '365yg' : 'toutiao',\n 'acfun' : 'acfun',\n 'archive' : 'archive',\n 'baidu' : 'baidu',\n 'bandcamp' : 'bandcamp',\n 'baomihua' : 'baomihua',\n 'bigthink' : 'bigthink',\n 'bilibili' : 'bilibili',\n 'cctv' : 'cntv',\n 'cntv' : 'cntv',\n 'cbs' : 'cbs',\n 'coub' : 'coub',\n 'dailymotion' : 'dailymotion',\n 'douban' : 'douban',\n 'douyin' : 'douyin',\n 'douyu' : 'douyutv',\n 'ehow' : 'ehow',\n 'facebook' : 'facebook',\n 'fc2' : 'fc2video',\n 'flickr' : 'flickr',\n 'freesound' : 'freesound',\n 'fun' : 'funshion',\n 'google' : 'google',\n 'giphy' : 'giphy',\n 'heavy-music' : 'heavymusic',\n 'huomao' : 'huomaotv',\n 'iask' : 'sina',\n 'icourses' : 'icourses',\n 'ifeng' : 'ifeng',\n 'imgur' : 'imgur',\n 'in' : 'alive',\n 'infoq' : 'infoq',\n 'instagram' : 'instagram',\n 'interest' : 'interest',\n 'iqilu' : 'iqilu',\n 'iqiyi' : 'iqiyi',\n 'ixigua' : 'ixigua',\n 'isuntv' : 'suntv',\n 'iwara' : 'iwara',\n 'joy' : 'joy',\n 'kankanews' : 'bilibili',\n 'kakao' : 'kakao',\n 'khanacademy' : 'khan',\n 'ku6' : 'ku6',\n 'kuaishou' : 'kuaishou',\n 'kugou' : 'kugou',\n 'kuwo' : 'kuwo',\n 'le' : 'le',\n 'letv' : 'le',\n 'lizhi' : 'lizhi',\n 'longzhu' : 'longzhu',\n 'lrts' : 'lrts',\n 'magisto' : 'magisto',\n 'metacafe' : 'metacafe',\n 'mgtv' : 'mgtv',\n 'miomio' : 'miomio',\n 'missevan' : 'missevan',\n 'mixcloud' : 'mixcloud',\n 'mtv81' : 'mtv81',\n 'miaopai' : 'yixia',\n 'naver' : 'naver',\n '7gogo' : 'nanagogo',\n 'nicovideo' : 'nicovideo',\n 'pinterest' : 'pinterest',\n 'pixnet' : 'pixnet',\n 'pptv' : 'pptv',\n 'qingting' : 'qingting',\n 'qq' : 'qq',\n 'showroom-live' : 'showroom',\n 'sina' : 'sina',\n 'smgbb' : 'bilibili',\n 'sohu' : 'sohu',\n 'soundcloud' : 'soundcloud',\n 'ted' : 'ted',\n 'theplatform' : 'theplatform',\n 'tiktok' : 'tiktok',\n 'tucao' : 'tucao',\n 'tudou' : 'tudou',\n 'tumblr' : 'tumblr',\n 'twimg' : 'twitter',\n 'twitter' : 'twitter',\n 'ucas' : 'ucas',\n 'vimeo' : 'vimeo',\n 'wanmen' : 'wanmen',\n 'weibo' : 'miaopai',\n 'veoh' : 'veoh',\n 'vk' : 'vk',\n 'x' : 'twitter',\n 'xiaokaxiu' : 'yixia',\n 'xiaojiadianvideo' : 'fc2video',\n 'ximalaya' : 'ximalaya',\n 'xinpianchang' : 'xinpianchang',\n 'yizhibo' : 'yizhibo',\n 'youku' : 'youku',\n 'youtu' : 'youtube',\n 'youtube' : 'youtube',\n 'zhanqi' : 'zhanqi',\n 'zhibo' : 'zhibo',\n 'zhihu' : 'zhihu',\n}\n\ndry_run = False\njson_output = False\nforce = False\nskip_existing_file_size_check = False\nplayer = None\nextractor_proxy = None\ncookies = None\noutput_filename = None\nauto_rename = False\ninsecure = False\nm3u8 = False\npostfix = False\nprefix = None\n\nfake_headers = {\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',\n 'Accept-Charset': 'UTF-8,*;q=0.5',\n 'Accept-Encoding': 'gzip,deflate,sdch',\n 'Accept-Language': 'en-US,en;q=0.8',\n 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/126.0.2592.113' # Latest Edge\n}\n\nif sys.stdout.isatty():\n default_encoding = sys.stdout.encoding.lower()\nelse:\n default_encoding = locale.getpreferredencoding().lower()\n\n\ndef rc4(key, data):\n # all encryption algo should work on bytes\n assert type(key) == type(data) and type(key) == type(b'')\n state = list(range(256))\n j = 0\n for i in range(256):\n j += state[i] + key[i % len(key)]\n j &= 0xff\n state[i], state[j] = state[j], state[i]\n\n i = 0\n j = 0\n out_list = []\n for char in data:\n i += 1\n i &= 0xff\n j += state[i]\n j &= 0xff\n state[i], state[j] = state[j], state[i]\n prn = state[(state[i] + state[j]) & 0xff]\n out_list.append(char ^ prn)\n\n return bytes(out_list)\n\n\ndef general_m3u8_extractor(url, headers={}):\n m3u8_list = get_content(url, headers=headers).split('\\n')\n urls = []\n for line in m3u8_list:\n line = line.strip()\n if line and not line.startswith('#'):\n if line.startswith('http'):\n urls.append(line)\n else:\n seg_url = parse.urljoin(url, line)\n urls.append(seg_url)\n return urls\n\n\ndef maybe_print(*s):\n try:\n print(*s)\n except:\n pass\n\n\ndef tr(s):\n if default_encoding == 'utf-8':\n return s\n else:\n return s\n # return str(s.encode('utf-8'))[2:-1]\n\n\n# DEPRECATED in favor of match1()\ndef r1(pattern, text):\n m = re.search(pattern, text)\n if m:\n return m.group(1)\n\n\n# DEPRECATED in favor of match1()\ndef r1_of(patterns, text):\n for p in patterns:\n x = r1(p, text)\n if x:\n return x\n\n\ndef match1(text, *patterns):\n \"\"\"Scans through a string for substrings matched some patterns (first-subgroups only).\n\n Args:\n text: A string to be scanned.\n patterns: Arbitrary number of regex patterns.\n\n Returns:\n When only one pattern is given, returns a string (None if no match found).\n When more than one pattern are given, returns a list of strings ([] if no match found).\n \"\"\"\n\n if len(patterns) == 1:\n pattern = patterns[0]\n match = re.search(pattern, text)\n if match:\n return match.group(1)\n else:\n return None\n else:\n ret = []\n for pattern in patterns:\n match = re.search(pattern, text)\n if match:\n ret.append(match.group(1))\n return ret\n\n\ndef matchall(text, patterns):\n \"\"\"Scans through a string for substrings matched some patterns.\n\n Args:\n text: A string to be scanned.\n patterns: a list of regex pattern.\n\n Returns:\n a list if matched. empty if not.\n \"\"\"\n\n ret = []\n for pattern in patterns:\n match = re.findall(pattern, text)\n ret += match\n\n return ret\n\n\ndef launch_player(player, urls):\n import subprocess\n import shlex\n urls = list(urls)\n for url in urls.copy():\n if type(url) is list:\n urls.extend(url)\n urls = [url for url in urls if type(url) is str]\n assert urls\n if (sys.version_info >= (3, 3)):\n import shutil\n exefile=shlex.split(player)[0]\n if shutil.which(exefile) is not None:\n subprocess.call(shlex.split(player) + urls)\n else:\n log.wtf('[Failed] Cannot find player \"%s\"' % exefile)\n else:\n subprocess.call(shlex.split(player) + urls)\n\n\ndef parse_query_param(url, param):\n \"\"\"Parses the query string of a URL and returns the value of a parameter.\n\n Args:\n url: A URL.\n param: A string representing the name of the parameter.\n\n Returns:\n The value of the parameter.\n \"\"\"\n\n try:\n return parse.parse_qs(parse.urlparse(url).query)[param][0]\n except:\n return None\n\n\ndef unicodize(text):\n return re.sub(\n r'\\\\u([0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f])',\n lambda x: chr(int(x.group(0)[2:], 16)),\n text\n )\n\n\n# DEPRECATED in favor of util.legitimize()\ndef escape_file_path(path):\n path = path.replace('/', '-')\n path = path.replace('\\\\', '-')\n path = path.replace('*', '-')\n path = path.replace('?', '-')\n return path\n\n\ndef ungzip(data):\n \"\"\"Decompresses data for Content-Encoding: gzip.\n \"\"\"\n from io import BytesIO\n import gzip\n buffer = BytesIO(data)\n f = gzip.GzipFile(fileobj=buffer)\n return f.read()\n\n\ndef undeflate(data):\n \"\"\"Decompresses data for Content-Encoding: deflate.\n (the zlib compression is used.)\n \"\"\"\n import zlib\n decompressobj = zlib.decompressobj(-zlib.MAX_WBITS)\n return decompressobj.decompress(data)+decompressobj.flush()\n\n\n# an http.client implementation of get_content()\n# because urllib does not support \"Connection: keep-alive\"\ndef getHttps(host, url, headers, debuglevel=0):\n import http.client\n\n conn = http.client.HTTPSConnection(host)\n conn.set_debuglevel(debuglevel)\n conn.request(\"GET\", url, headers=headers)\n resp = conn.getresponse()\n logging.debug('getHttps: %s' % resp.getheaders())\n set_cookie = resp.getheader('set-cookie')\n\n data = resp.read()\n try:\n data = ungzip(data) # gzip\n data = undeflate(data) # deflate\n except:\n pass\n\n conn.close()\n return str(data, encoding='utf-8'), set_cookie # TODO: support raw data\n\n\n# DEPRECATED in favor of get_content()\ndef get_response(url, faker=False):\n logging.debug('get_response: %s' % url)\n ctx = None\n if insecure:\n # ignore ssl errors\n ctx = ssl.create_default_context()\n ctx.check_hostname = False\n ctx.verify_mode = ssl.CERT_NONE\n # install cookies\n if cookies:\n opener = request.build_opener(request.HTTPCookieProcessor(cookies))\n request.install_opener(opener)\n\n if faker:\n response = request.urlopen(\n request.Request(url, headers=fake_headers), None, context=ctx,\n )\n else:\n response = request.urlopen(url, context=ctx)\n\n data = response.read()\n if response.info().get('Content-Encoding') == 'gzip':\n data = ungzip(data)\n elif response.info().get('Content-Encoding') == 'deflate':\n data = undeflate(data)\n response.data = data\n return response\n\n\n# DEPRECATED in favor of get_content()\ndef get_html(url, encoding=None, faker=False):\n content = get_response(url, faker).data\n return str(content, 'utf-8', 'ignore')\n\n\n# DEPRECATED in favor of get_content()\ndef get_decoded_html(url, faker=False):\n response = get_response(url, faker)\n data = response.data\n charset = r1(r'charset=([\\w-]+)', response.headers['content-type'])\n if charset:\n return data.decode(charset, 'ignore')\n else:\n return data\n\n\ndef get_location(url, headers=None, get_method='HEAD'):\n logging.debug('get_location: %s' % url)\n\n if headers:\n req = request.Request(url, headers=headers)\n else:\n req = request.Request(url)\n req.get_method = lambda: get_method\n res = urlopen_with_retry(req)\n return res.geturl()\n\n\ndef urlopen_with_retry(*args, **kwargs):\n retry_time = 3\n for i in range(retry_time):\n try:\n if insecure:\n # ignore ssl errors\n ctx = ssl.create_default_context()\n ctx.check_hostname = False\n ctx.verify_mode = ssl.CERT_NONE\n return request.urlopen(*args, context=ctx, **kwargs)\n else:\n return request.urlopen(*args, **kwargs)\n except socket.timeout as e:\n logging.debug('request attempt %s timeout' % str(i + 1))\n if i + 1 == retry_time:\n raise e\n # try to tackle youku CDN fails\n except error.HTTPError as http_error:\n logging.debug('HTTP Error with code{}'.format(http_error.code))\n if i + 1 == retry_time:\n raise http_error\n\n\ndef get_content(url, headers={}, decoded=True):\n \"\"\"Gets the content of a URL via sending a HTTP GET request.\n\n Args:\n url: A URL.\n headers: Request headers used by the client.\n decoded: Whether decode the response body using UTF-8 or the charset specified in Content-Type.\n\n Returns:\n The content as a string.\n \"\"\"\n\n logging.debug('get_content: %s' % url)\n\n req = request.Request(url, headers=headers)\n if cookies:\n # NOTE: Do not use cookies.add_cookie_header(req)\n # #HttpOnly_ cookies were not supported by CookieJar and MozillaCookieJar properly until python 3.10\n # See also:\n # - https://github.com/python/cpython/pull/17471\n # - https://bugs.python.org/issue2190\n # Here we add cookies to the request headers manually\n cookie_strings = []\n for cookie in list(cookies):\n cookie_strings.append(cookie.name + '=' + cookie.value)\n cookie_headers = {'Cookie': '; '.join(cookie_strings)}\n req.headers.update(cookie_headers)\n\n response = urlopen_with_retry(req)\n data = response.read()\n\n # Handle HTTP compression for gzip and deflate (zlib)\n content_encoding = response.getheader('Content-Encoding')\n if content_encoding == 'gzip':\n data = ungzip(data)\n elif content_encoding == 'deflate':\n data = undeflate(data)\n\n # Decode the response body\n if decoded:\n charset = match1(\n response.getheader('Content-Type', ''), r'charset=([\\w-]+)'\n )\n if charset is not None:\n data = data.decode(charset, 'ignore')\n else:\n data = data.decode('utf-8', 'ignore')\n\n return data\n\n\ndef post_content(url, headers={}, post_data={}, decoded=True, **kwargs):\n \"\"\"Post the content of a URL via sending a HTTP POST request.\n\n Args:\n url: A URL.\n headers: Request headers used by the client.\n decoded: Whether decode the response body using UTF-8 or the charset specified in Content-Type.\n\n Returns:\n The content as a string.\n \"\"\"\n if kwargs.get('post_data_raw'):\n logging.debug('post_content: %s\\npost_data_raw: %s' % (url, kwargs['post_data_raw']))\n else:\n logging.debug('post_content: %s\\npost_data: %s' % (url, post_data))\n\n req = request.Request(url, headers=headers)\n if cookies:\n # NOTE: Do not use cookies.add_cookie_header(req)\n # #HttpOnly_ cookies were not supported by CookieJar and MozillaCookieJar properly until python 3.10\n # See also:\n # - https://github.com/python/cpython/pull/17471\n # - https://bugs.python.org/issue2190\n # Here we add cookies to the request headers manually\n cookie_strings = []\n for cookie in list(cookies):\n cookie_strings.append(cookie.name + '=' + cookie.value)\n cookie_headers = {'Cookie': '; '.join(cookie_strings)}\n req.headers.update(cookie_headers)\n if kwargs.get('post_data_raw'):\n post_data_enc = bytes(kwargs['post_data_raw'], 'utf-8')\n else:\n post_data_enc = bytes(parse.urlencode(post_data), 'utf-8')\n response = urlopen_with_retry(req, data=post_data_enc)\n data = response.read()\n\n # Handle HTTP compression for gzip and deflate (zlib)\n content_encoding = response.getheader('Content-Encoding')\n if content_encoding == 'gzip':\n data = ungzip(data)\n elif content_encoding == 'deflate':\n data = undeflate(data)\n\n # Decode the response body\n if decoded:\n charset = match1(\n response.getheader('Content-Type'), r'charset=([\\w-]+)'\n )\n if charset is not None:\n data = data.decode(charset)\n else:\n data = data.decode('utf-8')\n\n return data\n\n\ndef url_size(url, faker=False, headers={}):\n if faker:\n response = urlopen_with_retry(\n request.Request(url, headers=fake_headers)\n )\n elif headers:\n response = urlopen_with_retry(request.Request(url, headers=headers))\n else:\n response = urlopen_with_retry(url)\n\n size = response.headers['content-length']\n return int(size) if size is not None else float('inf')\n\n\ndef urls_size(urls, faker=False, headers={}):\n return sum([url_size(url, faker=faker, headers=headers) for url in urls])\n\n\ndef get_head(url, headers=None, get_method='HEAD'):\n logging.debug('get_head: %s' % url)\n\n if headers:\n req = request.Request(url, headers=headers)\n else:\n req = request.Request(url)\n req.get_method = lambda: get_method\n res = urlopen_with_retry(req)\n return res.headers\n\n\ndef url_info(url, faker=False, headers={}):\n logging.debug('url_info: %s' % url)\n\n if faker:\n response = urlopen_with_retry(\n request.Request(url, headers=fake_headers)\n )\n elif headers:\n response = urlopen_with_retry(request.Request(url, headers=headers))\n else:\n response = urlopen_with_retry(request.Request(url))\n\n headers = response.headers\n\n type = headers['content-type']\n if type == 'image/jpg; charset=UTF-8' or type == 'image/jpg':\n type = 'audio/mpeg' # fix for netease\n mapping = {\n 'video/3gpp': '3gp',\n 'video/f4v': 'flv',\n 'video/mp4': 'mp4',\n 'video/MP2T': 'ts',\n 'video/quicktime': 'mov',\n 'video/webm': 'webm',\n 'video/x-flv': 'flv',\n 'video/x-ms-asf': 'asf',\n 'audio/mp4': 'mp4',\n 'audio/mpeg': 'mp3',\n 'audio/wav': 'wav',\n 'audio/x-wav': 'wav',\n 'audio/wave': 'wav',\n 'image/jpeg': 'jpg',\n 'image/png': 'png',\n 'image/gif': 'gif',\n 'application/pdf': 'pdf',\n }\n if type in mapping:\n ext = mapping[type]\n else:\n type = None\n if headers['content-disposition']:\n try:\n filename = parse.unquote(\n r1(r'filename=\"?([^\"]+)\"?', headers['content-disposition'])\n )\n if len(filename.split('.')) > 1:\n ext = filename.split('.')[-1]\n else:\n ext = None\n except:\n ext = None\n else:\n ext = None\n\n if headers['transfer-encoding'] != 'chunked':\n size = headers['content-length'] and int(headers['content-length'])\n else:\n size = None\n\n return type, ext, size\n\n\ndef url_locations(urls, faker=False, headers={}):\n locations = []\n for url in urls:\n logging.debug('url_locations: %s' % url)\n\n if faker:\n response = urlopen_with_retry(\n request.Request(url, headers=fake_headers)\n )\n elif headers:\n response = urlopen_with_retry(\n request.Request(url, headers=headers)\n )\n else:\n response = urlopen_with_retry(request.Request(url))\n\n locations.append(response.url)\n return locations\n\n\ndef url_save(\n url, filepath, bar, refer=None, is_part=False, faker=False,\n headers=None, timeout=None, **kwargs\n):\n tmp_headers = headers.copy() if headers is not None else {}\n # When a referer specified with param refer,\n # the key must be 'Referer' for the hack here\n if refer is not None:\n tmp_headers['Referer'] = refer\n if type(url) is list:\n chunk_sizes = [url_size(url, faker=faker, headers=tmp_headers) for url in url]\n file_size = sum(chunk_sizes)\n is_chunked, urls = True, url\n else:\n file_size = url_size(url, faker=faker, headers=tmp_headers)\n chunk_sizes = [file_size]\n is_chunked, urls = False, [url]\n\n continue_renameing = True\n while continue_renameing:\n continue_renameing = False\n if os.path.exists(filepath):\n if not force and (file_size == os.path.getsize(filepath) or skip_existing_file_size_check):\n if not is_part:\n if bar:\n bar.done()\n if skip_existing_file_size_check:\n log.w(\n 'Skipping {} without checking size: file already exists'.format(\n tr(os.path.basename(filepath))\n )\n )\n else:\n log.w(\n 'Skipping {}: file already exists'.format(\n tr(os.path.basename(filepath))\n )\n )\n else:\n if bar:\n bar.update_received(file_size)\n return\n else:\n if not is_part:\n if bar:\n bar.done()\n if not force and auto_rename:\n path, ext = os.path.basename(filepath).rsplit('.', 1)\n finder = re.compile(r' \\([1-9]\\d*?\\)$')\n if (finder.search(path) is None):\n thisfile = path + ' (1).' + ext\n else:\n def numreturn(a):\n return ' (' + str(int(a.group()[2:-1]) + 1) + ').'\n thisfile = finder.sub(numreturn, path) + ext\n filepath = os.path.join(os.path.dirname(filepath), thisfile)\n print('Changing name to %s' % tr(os.path.basename(filepath)), '...')\n continue_renameing = True\n continue\n if log.yes_or_no('File with this name already exists. Overwrite?'):\n log.w('Overwriting %s ...' % tr(os.path.basename(filepath)))\n else:\n return\n elif not os.path.exists(os.path.dirname(filepath)):\n os.mkdir(os.path.dirname(filepath))\n\n temp_filepath = filepath + '.download' if file_size != float('inf') \\\n else filepath\n received = 0\n if not force:\n open_mode = 'ab'\n\n if os.path.exists(temp_filepath):\n received += os.path.getsize(temp_filepath)\n if bar:\n bar.update_received(os.path.getsize(temp_filepath))\n else:\n open_mode = 'wb'\n\n chunk_start = 0\n chunk_end = 0\n for i, url in enumerate(urls):\n received_chunk = 0\n chunk_start += 0 if i == 0 else chunk_sizes[i - 1]\n chunk_end += chunk_sizes[i]\n if received < file_size and received < chunk_end:\n if faker:\n tmp_headers = fake_headers\n '''\n if parameter headers passed in, we have it copied as tmp_header\n elif headers:\n headers = headers\n else:\n headers = {}\n '''\n if received:\n # chunk_start will always be 0 if not chunked\n tmp_headers['Range'] = 'bytes=' + str(received - chunk_start) + '-'\n if refer:\n tmp_headers['Referer'] = refer\n\n if timeout:\n response = urlopen_with_retry(\n request.Request(url, headers=tmp_headers), timeout=timeout\n )\n else:\n response = urlopen_with_retry(\n request.Request(url, headers=tmp_headers)\n )\n try:\n range_start = int(\n response.headers[\n 'content-range'\n ][6:].split('/')[0].split('-')[0]\n )\n end_length = int(\n response.headers['content-range'][6:].split('/')[1]\n )\n range_length = end_length - range_start\n except:\n content_length = response.headers['content-length']\n range_length = int(content_length) if content_length is not None \\\n else float('inf')\n\n if is_chunked: # always append if chunked\n open_mode = 'ab'\n elif file_size != received + range_length: # is it ever necessary?\n received = 0\n if bar:\n bar.received = 0\n open_mode = 'wb'\n\n with open(temp_filepath, open_mode) as output:\n while True:\n buffer = None\n try:\n buffer = response.read(1024 * 256)\n except socket.timeout:\n pass\n if not buffer:\n if file_size == float('+inf'): # Prevent infinite downloading\n break\n if is_chunked and received_chunk == range_length:\n break\n elif not is_chunked and received == file_size: # Download finished\n break\n # Unexpected termination. Retry request\n tmp_headers['Range'] = 'bytes=' + str(received - chunk_start) + '-'\n response = urlopen_with_retry(\n request.Request(url, headers=tmp_headers)\n )\n continue\n output.write(buffer)\n received += len(buffer)\n received_chunk += len(buffer)\n if bar:\n bar.update_received(len(buffer))\n\n assert received == os.path.getsize(temp_filepath), '%s == %s == %s' % (\n received, os.path.getsize(temp_filepath), temp_filepath\n )\n\n if os.access(filepath, os.W_OK) and file_size != float('inf'):\n # on Windows rename could fail if destination filepath exists\n # we should simply choose a new name instead of brutal os.remove(filepath)\n filepath = filepath + \" (2)\"\n os.rename(temp_filepath, filepath)\n\n\nclass SimpleProgressBar:\n term_size = term.get_terminal_size()[1]\n\n def __init__(self, total_size, total_pieces=1):\n self.displayed = False\n self.total_size = total_size\n self.total_pieces = total_pieces\n self.current_piece = 1\n self.received = 0\n self.speed = ''\n self.last_updated = time.time()\n\n total_pieces_len = len(str(total_pieces))\n # 38 is the size of all statically known size in self.bar\n total_str = '%5s' % round(self.total_size / 1048576, 1)\n total_str_width = max(len(total_str), 5)\n self.bar_size = self.term_size - 28 - 2 * total_pieces_len \\\n - 2 * total_str_width\n self.bar = '{:>4}%% ({:>%s}/%sMB) ├{:─<%s}┤[{:>%s}/{:>%s}] {}' % (\n total_str_width, total_str, self.bar_size, total_pieces_len,\n total_pieces_len\n )\n\n def update(self):\n self.displayed = True\n bar_size = self.bar_size\n percent = round(self.received * 100 / self.total_size, 1)\n if percent >= 100:\n percent = 100\n dots = bar_size * int(percent) // 100\n plus = int(percent) - dots // bar_size * 100\n if plus > 0.8:\n plus = '█'\n elif plus > 0.4:\n plus = '>'\n else:\n plus = ''\n bar = '█' * dots + plus\n bar = self.bar.format(\n percent, round(self.received / 1048576, 1), bar,\n self.current_piece, self.total_pieces, self.speed\n )\n sys.stdout.write('\\r' + bar)\n sys.stdout.flush()\n\n def update_received(self, n):\n self.received += n\n time_diff = time.time() - self.last_updated\n bytes_ps = n / time_diff if time_diff else 0\n if bytes_ps >= 1024 ** 3:\n self.speed = '{:4.0f} GB/s'.format(bytes_ps / 1024 ** 3)\n elif bytes_ps >= 1024 ** 2:\n self.speed = '{:4.0f} MB/s'.format(bytes_ps / 1024 ** 2)\n elif bytes_ps >= 1024:\n self.speed = '{:4.0f} kB/s'.format(bytes_ps / 1024)\n else:\n self.speed = '{:4.0f} B/s'.format(bytes_ps)\n self.last_updated = time.time()\n self.update()\n\n def update_piece(self, n):\n self.current_piece = n\n\n def done(self):\n if self.displayed:\n print()\n self.displayed = False\n\n\nclass PiecesProgressBar:\n def __init__(self, total_size, total_pieces=1):\n self.displayed = False\n self.total_size = total_size\n self.total_pieces = total_pieces\n self.current_piece = 1\n self.received = 0\n\n def update(self):\n self.displayed = True\n bar = '{0:>5}%[{1:<40}] {2}/{3}'.format(\n '', '=' * 40, self.current_piece, self.total_pieces\n )\n sys.stdout.write('\\r' + bar)\n sys.stdout.flush()\n\n def update_received(self, n):\n self.received += n\n self.update()\n\n def update_piece(self, n):\n self.current_piece = n\n\n def done(self):\n if self.displayed:\n print()\n self.displayed = False\n\n\nclass DummyProgressBar:\n def __init__(self, *args):\n pass\n\n def update_received(self, n):\n pass\n\n def update_piece(self, n):\n pass\n\n def done(self):\n pass\n\n\ndef get_output_filename(urls, title, ext, output_dir, merge, **kwargs):\n # lame hack for the --output-filename option\n global output_filename\n if output_filename:\n result = output_filename\n if kwargs.get('part', -1) >= 0:\n result = '%s[%02d]' % (result, kwargs.get('part'))\n if ext:\n result = '%s.%s' % (result, ext)\n return result\n\n merged_ext = ext\n if (len(urls) > 1) and merge:\n from .processor.ffmpeg import has_ffmpeg_installed\n if ext in ['flv', 'f4v']:\n if has_ffmpeg_installed():\n merged_ext = 'mp4'\n else:\n merged_ext = 'flv'\n elif ext == 'mp4':\n merged_ext = 'mp4'\n elif ext == 'ts':\n if has_ffmpeg_installed():\n merged_ext = 'mkv'\n else:\n merged_ext = 'ts'\n result = title\n if kwargs.get('part', -1) >= 0:\n result = '%s[%02d]' % (result, kwargs.get('part'))\n result = '%s.%s' % (result, merged_ext)\n return result.replace(\"'\", \"_\")\n\ndef print_user_agent(faker=False):\n urllib_default_user_agent = 'Python-urllib/%d.%d' % sys.version_info[:2]\n user_agent = fake_headers['User-Agent'] if faker else urllib_default_user_agent\n print('User Agent: %s' % user_agent)\n\ndef download_urls(\n urls, title, ext, total_size, output_dir='.', refer=None, merge=True,\n faker=False, headers={}, **kwargs\n):\n assert urls\n if json_output:\n json_output_.download_urls(\n urls=urls, title=title, ext=ext, total_size=total_size,\n refer=refer\n )\n return\n if dry_run:\n print_user_agent(faker=faker)\n try:\n print('Real URLs:\\n%s' % '\\n'.join(urls))\n except:\n print('Real URLs:\\n%s' % '\\n'.join([j for i in urls for j in i]))\n return\n\n if player:\n launch_player(player, urls)\n return\n\n if not total_size:\n try:\n total_size = urls_size(urls, faker=faker, headers=headers)\n except:\n import traceback\n traceback.print_exc(file=sys.stdout)\n pass\n\n title = tr(get_filename(title))\n if postfix and 'vid' in kwargs:\n title = \"%s [%s]\" % (title, kwargs['vid'])\n if prefix is not None:\n title = \"[%s] %s\" % (prefix, title)\n output_filename = get_output_filename(urls, title, ext, output_dir, merge)\n output_filepath = os.path.join(output_dir, output_filename)\n\n if total_size:\n if not force and os.path.exists(output_filepath) and not auto_rename\\\n and (os.path.getsize(output_filepath) >= total_size * 0.9\\\n or skip_existing_file_size_check):\n if skip_existing_file_size_check:\n log.w('Skipping %s without checking size: file already exists' % output_filepath)\n else:\n log.w('Skipping %s: file already exists' % output_filepath)\n print()\n return\n bar = SimpleProgressBar(total_size, len(urls))\n else:\n bar = PiecesProgressBar(total_size, len(urls))\n\n if len(urls) == 1:\n url = urls[0]\n print('Downloading %s ...' % tr(output_filename))\n bar.update()\n url_save(\n url, output_filepath, bar, refer=refer, faker=faker,\n headers=headers, **kwargs\n )\n bar.done()\n else:\n parts = []\n print('Downloading %s ...' % tr(output_filename))\n bar.update()\n for i, url in enumerate(urls):\n output_filename_i = get_output_filename(urls, title, ext, output_dir, merge, part=i)\n output_filepath_i = os.path.join(output_dir, output_filename_i)\n parts.append(output_filepath_i)\n # print 'Downloading %s [%s/%s]...' % (tr(filename), i + 1, len(urls))\n bar.update_piece(i + 1)\n url_save(\n url, output_filepath_i, bar, refer=refer, is_part=True, faker=faker,\n headers=headers, **kwargs\n )\n bar.done()\n\n if not merge:\n print()\n return\n\n if 'av' in kwargs and kwargs['av']:\n from .processor.ffmpeg import has_ffmpeg_installed\n if has_ffmpeg_installed():\n from .processor.ffmpeg import ffmpeg_concat_av\n ret = ffmpeg_concat_av(parts, output_filepath, ext)\n print('Merged into %s' % output_filename)\n if ret == 0:\n for part in parts:\n os.remove(part)\n\n elif ext in ['flv', 'f4v']:\n try:\n from .processor.ffmpeg import has_ffmpeg_installed\n if has_ffmpeg_installed():\n from .processor.ffmpeg import ffmpeg_concat_flv_to_mp4\n ffmpeg_concat_flv_to_mp4(parts, output_filepath)\n else:\n from .processor.join_flv import concat_flv\n concat_flv(parts, output_filepath)\n print('Merged into %s' % output_filename)\n except:\n raise\n else:\n for part in parts:\n os.remove(part)\n\n elif ext == 'mp4':\n try:\n from .processor.ffmpeg import has_ffmpeg_installed\n if has_ffmpeg_installed():\n from .processor.ffmpeg import ffmpeg_concat_mp4_to_mp4\n ffmpeg_concat_mp4_to_mp4(parts, output_filepath)\n else:\n from .processor.join_mp4 import concat_mp4\n concat_mp4(parts, output_filepath)\n print('Merged into %s' % output_filename)\n except:\n raise\n else:\n for part in parts:\n os.remove(part)\n\n elif ext == 'ts':\n try:\n from .processor.ffmpeg import has_ffmpeg_installed\n if has_ffmpeg_installed():\n from .processor.ffmpeg import ffmpeg_concat_ts_to_mkv\n ffmpeg_concat_ts_to_mkv(parts, output_filepath)\n else:\n from .processor.join_ts import concat_ts\n concat_ts(parts, output_filepath)\n print('Merged into %s' % output_filename)\n except:\n raise\n else:\n for part in parts:\n os.remove(part)\n\n elif ext == 'mp3':\n try:\n from .processor.ffmpeg import has_ffmpeg_installed\n\n assert has_ffmpeg_installed()\n from .processor.ffmpeg import ffmpeg_concat_mp3_to_mp3\n ffmpeg_concat_mp3_to_mp3(parts, output_filepath)\n print('Merged into %s' % output_filename)\n except:\n raise\n else:\n for part in parts:\n os.remove(part)\n\n else:\n print(\"Can't merge %s files\" % ext)\n\n print()\n\n\ndef download_rtmp_url(\n url, title, ext, params={}, total_size=0, output_dir='.', refer=None,\n merge=True, faker=False\n):\n assert url\n if dry_run:\n print_user_agent(faker=faker)\n print('Real URL:\\n%s\\n' % [url])\n if params.get('-y', False): # None or unset -> False\n print('Real Playpath:\\n%s\\n' % [params.get('-y')])\n return\n\n if player:\n from .processor.rtmpdump import play_rtmpdump_stream\n play_rtmpdump_stream(player, url, params)\n return\n\n from .processor.rtmpdump import (\n has_rtmpdump_installed, download_rtmpdump_stream\n )\n assert has_rtmpdump_installed(), 'RTMPDump not installed.'\n download_rtmpdump_stream(url, title, ext, params, output_dir)\n\n\ndef download_url_ffmpeg(\n url, title, ext, params={}, total_size=0, output_dir='.', refer=None,\n merge=True, faker=False, stream=True\n):\n assert url\n if dry_run:\n print_user_agent(faker=faker)\n print('Real URL:\\n%s\\n' % [url])\n if params.get('-y', False): # None or unset ->False\n print('Real Playpath:\\n%s\\n' % [params.get('-y')])\n return\n\n if player:\n launch_player(player, [url])\n return\n\n from .processor.ffmpeg import has_ffmpeg_installed, ffmpeg_download_stream\n assert has_ffmpeg_installed(), 'FFmpeg not installed.'\n\n global output_filename\n if output_filename:\n dotPos = output_filename.rfind('.')\n if dotPos > 0:\n title = output_filename[:dotPos]\n ext = output_filename[dotPos+1:]\n else:\n title = output_filename\n\n title = tr(get_filename(title))\n\n ffmpeg_download_stream(url, title, ext, params, output_dir, stream=stream)\n\n\ndef playlist_not_supported(name):\n def f(*args, **kwargs):\n raise NotImplementedError('Playlist is not supported for ' + name)\n return f\n\n\ndef print_info(site_info, title, type, size, **kwargs):\n if json_output:\n json_output_.print_info(\n site_info=site_info, title=title, type=type, size=size\n )\n return\n if type:\n type = type.lower()\n if type in ['3gp']:\n type = 'video/3gpp'\n elif type in ['asf', 'wmv']:\n type = 'video/x-ms-asf'\n elif type in ['flv', 'f4v']:\n type = 'video/x-flv'\n elif type in ['mkv']:\n type = 'video/x-matroska'\n elif type in ['mp3']:\n type = 'audio/mpeg'\n elif type in ['mp4']:\n type = 'video/mp4'\n elif type in ['mov']:\n type = 'video/quicktime'\n elif type in ['ts']:\n type = 'video/MP2T'\n elif type in ['webm']:\n type = 'video/webm'\n\n elif type in ['jpg']:\n type = 'image/jpeg'\n elif type in ['png']:\n type = 'image/png'\n elif type in ['gif']:\n type = 'image/gif'\n\n if type in ['video/3gpp']:\n type_info = '3GPP multimedia file (%s)' % type\n elif type in ['video/x-flv', 'video/f4v']:\n type_info = 'Flash video (%s)' % type\n elif type in ['video/mp4', 'video/x-m4v']:\n type_info = 'MPEG-4 video (%s)' % type\n elif type in ['video/MP2T']:\n type_info = 'MPEG-2 transport stream (%s)' % type\n elif type in ['video/webm']:\n type_info = 'WebM video (%s)' % type\n # elif type in ['video/ogg']:\n # type_info = 'Ogg video (%s)' % type\n elif type in ['video/quicktime']:\n type_info = 'QuickTime video (%s)' % type\n elif type in ['video/x-matroska']:\n type_info = 'Matroska video (%s)' % type\n # elif type in ['video/x-ms-wmv']:\n # type_info = 'Windows Media video (%s)' % type\n elif type in ['video/x-ms-asf']:\n type_info = 'Advanced Systems Format (%s)' % type\n # elif type in ['video/mpeg']:\n # type_info = 'MPEG video (%s)' % type\n elif type in ['audio/mp4', 'audio/m4a']:\n type_info = 'MPEG-4 audio (%s)' % type\n elif type in ['audio/mpeg']:\n type_info = 'MP3 (%s)' % type\n elif type in ['audio/wav', 'audio/wave', 'audio/x-wav']:\n type_info = 'Waveform Audio File Format ({})'.format(type)\n\n elif type in ['image/jpeg']:\n type_info = 'JPEG Image (%s)' % type\n elif type in ['image/png']:\n type_info = 'Portable Network Graphics (%s)' % type\n elif type in ['image/gif']:\n type_info = 'Graphics Interchange Format (%s)' % type\n elif type in ['m3u8']:\n if 'm3u8_type' in kwargs:\n if kwargs['m3u8_type'] == 'master':\n type_info = 'M3U8 Master {}'.format(type)\n else:\n type_info = 'M3U8 Playlist {}'.format(type)\n else:\n type_info = 'Unknown type (%s)' % type\n\n maybe_print('Site: ', site_info)\n maybe_print('Title: ', unescape_html(tr(title)))\n print('Type: ', type_info)\n if type != 'm3u8':\n print(\n 'Size: ', round(size / 1048576, 2),\n 'MiB (' + str(size) + ' Bytes)'\n )\n if type == 'm3u8' and 'm3u8_url' in kwargs:\n print('M3U8 Url: {}'.format(kwargs['m3u8_url']))\n print()\n\n\ndef mime_to_container(mime):\n mapping = {\n 'video/3gpp': '3gp',\n 'video/mp4': 'mp4',\n 'video/webm': 'webm',\n 'video/x-flv': 'flv',\n }\n if mime in mapping:\n return mapping[mime]\n else:\n return mime.split('/')[1]\n\n\ndef parse_host(host):\n \"\"\"Parses host name and port number from a string.\n \"\"\"\n if re.match(r'^(\\d+)$', host) is not None:\n return (\"0.0.0.0\", int(host))\n if re.match(r'^(\\w+)://', host) is None:\n host = \"//\" + host\n o = parse.urlparse(host)\n hostname = o.hostname or \"0.0.0.0\"\n port = o.port or 0\n return (hostname, port)\n\n\ndef set_proxy(proxy):\n proxy_handler = request.ProxyHandler({\n 'http': '%s:%s' % proxy,\n 'https': '%s:%s' % proxy,\n })\n opener = request.build_opener(proxy_handler)\n request.install_opener(opener)\n\n\ndef unset_proxy():\n proxy_handler = request.ProxyHandler({})\n opener = request.build_opener(proxy_handler)\n request.install_opener(opener)\n\n\n# DEPRECATED in favor of set_proxy() and unset_proxy()\ndef set_http_proxy(proxy):\n if proxy is None: # Use system default setting\n proxy_support = request.ProxyHandler()\n elif proxy == '': # Don't use any proxy\n proxy_support = request.ProxyHandler({})\n else: # Use proxy\n proxy_support = request.ProxyHandler(\n {'http': '%s' % proxy, 'https': '%s' % proxy}\n )\n opener = request.build_opener(proxy_support)\n request.install_opener(opener)\n\n\ndef print_more_compatible(*args, **kwargs):\n import builtins as __builtin__\n \"\"\"Overload default print function as py (<3.3) does not support 'flush' keyword.\n Although the function name can be same as print to get itself overloaded automatically,\n I'd rather leave it with a different name and only overload it when importing to make less confusion.\n \"\"\"\n # nothing happens on py3.3 and later\n if sys.version_info[:2] >= (3, 3):\n return __builtin__.print(*args, **kwargs)\n\n # in lower pyver (e.g. 3.2.x), remove 'flush' keyword and flush it as requested\n doFlush = kwargs.pop('flush', False)\n ret = __builtin__.print(*args, **kwargs)\n if doFlush:\n kwargs.get('file', sys.stdout).flush()\n return ret\n\n\ndef download_main(download, download_playlist, urls, playlist, **kwargs):\n for url in urls:\n if re.match(r'https?://', url) is None:\n url = 'http://' + url\n\n if m3u8:\n if output_filename:\n title = output_filename\n else:\n title = \"m3u8file\"\n download_url_ffmpeg(url=url, title=title,ext = 'mp4',output_dir = '.')\n elif playlist:\n download_playlist(url, **kwargs)\n else:\n download(url, **kwargs)\n\n\ndef load_cookies(cookiefile):\n global cookies\n if cookiefile.endswith('.txt'):\n # MozillaCookieJar treats prefix '#HttpOnly_' as comments incorrectly!\n # do not use its load()\n # see also:\n # - https://docs.python.org/3/library/http.cookiejar.html#http.cookiejar.MozillaCookieJar\n # - https://github.com/python/cpython/blob/4b219ce/Lib/http/cookiejar.py#L2014\n # - https://curl.haxx.se/libcurl/c/CURLOPT_COOKIELIST.html#EXAMPLE\n #cookies = cookiejar.MozillaCookieJar(cookiefile)\n #cookies.load()\n from http.cookiejar import Cookie\n cookies = cookiejar.MozillaCookieJar()\n now = time.time()\n ignore_discard, ignore_expires = False, False\n with open(cookiefile, 'r', encoding='utf-8') as f:\n for line in f:\n # last field may be absent, so keep any trailing tab\n if line.endswith(\"\\n\"): line = line[:-1]\n\n # skip comments and blank lines XXX what is $ for?\n if (line.strip().startswith((\"#\", \"$\")) or\n line.strip() == \"\"):\n if not line.strip().startswith('#HttpOnly_'): # skip for #HttpOnly_\n continue\n\n domain, domain_specified, path, secure, expires, name, value = \\\n line.split(\"\\t\")\n secure = (secure == \"TRUE\")\n domain_specified = (domain_specified == \"TRUE\")\n if name == \"\":\n # cookies.txt regards 'Set-Cookie: foo' as a cookie\n # with no name, whereas http.cookiejar regards it as a\n # cookie with no value.\n name = value\n value = None\n\n initial_dot = domain.startswith(\".\")\n if not line.strip().startswith('#HttpOnly_'): # skip for #HttpOnly_\n assert domain_specified == initial_dot\n\n discard = False\n if expires == \"\":\n expires = None\n discard = True\n\n # assume path_specified is false\n c = Cookie(0, name, value,\n None, False,\n domain, domain_specified, initial_dot,\n path, False,\n secure,\n expires,\n discard,\n None,\n None,\n {})\n if not ignore_discard and c.discard:\n continue\n if not ignore_expires and c.is_expired(now):\n continue\n cookies.set_cookie(c)\n\n elif cookiefile.endswith(('.sqlite', '.sqlite3')):\n import sqlite3, shutil, tempfile\n temp_dir = tempfile.gettempdir()\n temp_cookiefile = os.path.join(temp_dir, 'temp_cookiefile.sqlite')\n shutil.copy2(cookiefile, temp_cookiefile)\n\n cookies = cookiejar.MozillaCookieJar()\n con = sqlite3.connect(temp_cookiefile)\n cur = con.cursor()\n cur.execute(\"\"\"SELECT host, path, isSecure, expiry, name, value\n FROM moz_cookies\"\"\")\n for item in cur.fetchall():\n c = cookiejar.Cookie(\n 0, item[4], item[5], None, False, item[0],\n item[0].startswith('.'), item[0].startswith('.'),\n item[1], False, item[2], item[3], item[3] == '', None,\n None, {},\n )\n cookies.set_cookie(c)\n\n else:\n log.e('[error] unsupported cookies format')\n # TODO: Chromium Cookies\n # SELECT host_key, path, secure, expires_utc, name, encrypted_value\n # FROM cookies\n # http://n8henrie.com/2013/11/use-chromes-cookies-for-easier-downloading-with-python-requests/\n\n\ndef set_socks_proxy(proxy):\n try:\n import socks\n if '@' in proxy:\n proxy_info = proxy.split(\"@\")\n socks_proxy_addrs = proxy_info[1].split(':')\n socks_proxy_auth = proxy_info[0].split(\":\")\n socks.set_default_proxy(\n socks.SOCKS5,\n socks_proxy_addrs[0],\n int(socks_proxy_addrs[1]),\n True,\n socks_proxy_auth[0],\n socks_proxy_auth[1]\n )\n else:\n socks_proxy_addrs = proxy.split(':')\n socks.set_default_proxy(\n socks.SOCKS5,\n socks_proxy_addrs[0],\n int(socks_proxy_addrs[1]),\n )\n socket.socket = socks.socksocket\n\n def getaddrinfo(*args):\n return [\n (socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))\n ]\n socket.getaddrinfo = getaddrinfo\n except ImportError:\n log.w(\n 'Error importing PySocks library, socks proxy ignored.'\n 'In order to use use socks proxy, please install PySocks.'\n )\n\n\ndef script_main(download, download_playlist, **kwargs):\n logging.basicConfig(format='[%(levelname)s] %(message)s')\n\n def print_version():\n version = get_version(\n kwargs['repo_path'] if 'repo_path' in kwargs else __version__\n )\n log.i(\n 'version {}, a tiny downloader that scrapes the web.'.format(\n version\n )\n )\n\n parser = argparse.ArgumentParser(\n prog='you-get',\n usage='you-get [OPTION]... URL...',\n description='A tiny downloader that scrapes the web',\n add_help=False,\n )\n parser.add_argument(\n '-V', '--version', action='store_true',\n help='Print version and exit'\n )\n parser.add_argument(\n '-h', '--help', action='store_true',\n help='Print this help message and exit'\n )\n\n dry_run_grp = parser.add_argument_group(\n 'Dry-run options', '(no actual downloading)'\n )\n dry_run_grp = dry_run_grp.add_mutually_exclusive_group()\n dry_run_grp.add_argument(\n '-i', '--info', action='store_true', help='Print extracted information'\n )\n dry_run_grp.add_argument(\n '-u', '--url', action='store_true',\n help='Print extracted information with URLs'\n )\n dry_run_grp.add_argument(\n '--json', action='store_true',\n help='Print extracted URLs in JSON format'\n )\n\n download_grp = parser.add_argument_group('Download options')\n download_grp.add_argument(\n '-n', '--no-merge', action='store_true', default=False,\n help='Do not merge video parts'\n )\n download_grp.add_argument(\n '--no-caption', action='store_true',\n help='Do not download captions (subtitles, lyrics, danmaku, ...)'\n )\n download_grp.add_argument(\n '--post', '--postfix', dest='postfix', action='store_true', default=False,\n help='Postfix downloaded files with unique identifiers'\n )\n download_grp.add_argument(\n '--pre', '--prefix', dest='prefix', metavar='PREFIX', default=None,\n help='Prefix downloaded files with string'\n )\n download_grp.add_argument(\n '-f', '--force', action='store_true', default=False,\n help='Force overwriting existing files'\n )\n download_grp.add_argument(\n '--skip-existing-file-size-check', action='store_true', default=False,\n help='Skip existing file without checking file size'\n )\n download_grp.add_argument(\n '-F', '--format', metavar='STREAM_ID',\n help='Set video format to STREAM_ID'\n )\n download_grp.add_argument(\n '-O', '--output-filename', metavar='FILE', help='Set output filename'\n )\n download_grp.add_argument(\n '-o', '--output-dir', metavar='DIR', default='.',\n help='Set output directory'\n )\n download_grp.add_argument(\n '-p', '--player', metavar='PLAYER',\n help='Stream extracted URL to a PLAYER'\n )\n download_grp.add_argument(\n '-c', '--cookies', metavar='COOKIES_FILE',\n help='Load cookies.txt or cookies.sqlite'\n )\n download_grp.add_argument(\n '-t', '--timeout', metavar='SECONDS', type=int, default=600,\n help='Set socket timeout'\n )\n download_grp.add_argument(\n '-d', '--debug', action='store_true',\n help='Show traceback and other debug info'\n )\n download_grp.add_argument(\n '-I', '--input-file', metavar='FILE', type=argparse.FileType('r'),\n help='Read non-playlist URLs from FILE'\n )\n download_grp.add_argument(\n '-P', '--password', help='Set video visit password to PASSWORD'\n )\n download_grp.add_argument(\n '-l', '--playlist', action='store_true',\n help='Prefer to download a playlist'\n )\n\n playlist_grp = parser.add_argument_group('Playlist optional options')\n playlist_grp.add_argument(\n '--first', metavar='FIRST',\n help='the first number'\n )\n playlist_grp.add_argument(\n '--last', metavar='LAST',\n help='the last number'\n )\n playlist_grp.add_argument(\n '--size', '--page-size', metavar='PAGE_SIZE',\n help='the page size number'\n )\n\n download_grp.add_argument(\n '-a', '--auto-rename', action='store_true', default=False,\n help='Auto rename same name different files'\n )\n\n download_grp.add_argument(\n '-k', '--insecure', action='store_true', default=False,\n help='ignore ssl errors'\n )\n\n proxy_grp = parser.add_argument_group('Proxy options')\n proxy_grp = proxy_grp.add_mutually_exclusive_group()\n proxy_grp.add_argument(\n '-x', '--http-proxy', metavar='HOST:PORT',\n help='Use an HTTP proxy for downloading'\n )\n proxy_grp.add_argument(\n '-y', '--extractor-proxy', metavar='HOST:PORT',\n help='Use an HTTP proxy for extracting only'\n )\n proxy_grp.add_argument(\n '--no-proxy', action='store_true', help='Never use a proxy'\n )\n proxy_grp.add_argument(\n '-s', '--socks-proxy', metavar='HOST:PORT or USERNAME:PASSWORD@HOST:PORT',\n help='Use an SOCKS5 proxy for downloading'\n )\n\n download_grp.add_argument('--stream', help=argparse.SUPPRESS)\n download_grp.add_argument('--itag', help=argparse.SUPPRESS)\n\n download_grp.add_argument('-m', '--m3u8', action='store_true', default=False,\n help = 'download video using an m3u8 url')\n\n\n parser.add_argument('URL', nargs='*', help=argparse.SUPPRESS)\n\n args = parser.parse_args()\n\n if args.help:\n print_version()\n parser.print_help()\n sys.exit()\n if args.version:\n print_version()\n sys.exit()\n\n if args.debug:\n # Set level of root logger to DEBUG\n logging.getLogger().setLevel(logging.DEBUG)\n\n global force\n global skip_existing_file_size_check\n global dry_run\n global json_output\n global player\n global extractor_proxy\n global output_filename\n global auto_rename\n global insecure\n global m3u8\n global postfix\n global prefix\n output_filename = args.output_filename\n extractor_proxy = args.extractor_proxy\n\n info_only = args.info\n if args.force:\n force = True\n if args.skip_existing_file_size_check:\n skip_existing_file_size_check = True\n if args.auto_rename:\n auto_rename = True\n if args.url:\n dry_run = True\n if args.json:\n json_output = True\n # to fix extractors not use VideoExtractor\n dry_run = True\n info_only = False\n\n if args.cookies:\n load_cookies(args.cookies)\n\n if args.m3u8:\n m3u8 = True\n\n caption = True\n stream_id = args.format or args.stream or args.itag\n if args.no_caption:\n caption = False\n if args.player:\n player = args.player\n caption = False\n\n if args.insecure:\n # ignore ssl\n insecure = True\n\n postfix = args.postfix\n prefix = args.prefix\n\n if args.no_proxy:\n set_http_proxy('')\n else:\n set_http_proxy(args.http_proxy)\n if args.socks_proxy:\n set_socks_proxy(args.socks_proxy)\n\n URLs = []\n if args.input_file:\n logging.debug('you are trying to load urls from %s', args.input_file)\n if args.playlist:\n log.e(\n \"reading playlist from a file is unsupported \"\n \"and won't make your life easier\"\n )\n sys.exit(2)\n URLs.extend(args.input_file.read().splitlines())\n args.input_file.close()\n URLs.extend(args.URL)\n\n if not URLs:\n parser.print_help()\n sys.exit()\n\n socket.setdefaulttimeout(args.timeout)\n\n try:\n extra = {'args': args}\n if extractor_proxy:\n extra['extractor_proxy'] = extractor_proxy\n if stream_id:\n extra['stream_id'] = stream_id\n download_main(\n download, download_playlist,\n URLs, args.playlist,\n output_dir=args.output_dir, merge=not args.no_merge,\n info_only=info_only, json_output=json_output, caption=caption,\n password=args.password,\n **extra\n )\n except KeyboardInterrupt:\n if args.debug:\n raise\n else:\n sys.exit(1)\n except UnicodeEncodeError:\n if args.debug:\n raise\n log.e(\n '[error] oops, the current environment does not seem to support '\n 'Unicode.'\n )\n log.e('please set it to a UTF-8-aware locale first,')\n log.e(\n 'so as to save the video (with some Unicode characters) correctly.'\n )\n log.e('you can do it like this:')\n log.e(' (Windows) % chcp 65001 ')\n log.e(' (Linux) $ LC_CTYPE=en_US.UTF-8')\n sys.exit(1)\n except Exception:\n if not args.debug:\n log.e('[error] oops, something went wrong.')\n log.e(\n 'don\\'t panic, c\\'est la vie. please try the following steps:'\n )\n log.e(' (1) Rule out any network problem.')\n log.e(' (2) Make sure you-get is up-to-date.')\n log.e(' (3) Check if the issue is already known, on')\n log.e(' https://github.com/soimort/you-get/wiki/Known-Bugs')\n log.e(' https://github.com/soimort/you-get/issues')\n log.e(' (4) Run the command with \\'--debug\\' option,')\n log.e(' and report this issue with the full output.')\n else:\n print_version()\n log.i(args)\n raise\n sys.exit(1)\n\n\ndef google_search(url):\n keywords = r1(r'https?://(.*)', url)\n url = 'https://www.google.com/search?tbm=vid&q=%s' % parse.quote(keywords)\n page = get_content(url, headers=fake_headers)\n videos = re.findall(\n r'(https://www\\.youtube\\.com/watch\\?v=[\\w-]+)', page\n )\n print('Best matched result:')\n return(videos[0])\n\n\ndef url_to_module(url):\n try:\n video_host = r1(r'https?://([^/]+)/', url)\n video_url = r1(r'https?://[^/]+(.*)', url)\n assert video_host and video_url\n except AssertionError:\n url = google_search(url)\n video_host = r1(r'https?://([^/]+)/', url)\n video_url = r1(r'https?://[^/]+(.*)', url)\n\n if video_host.endswith('.com.cn') or video_host.endswith('.ac.cn'):\n video_host = video_host[:-3]\n domain = r1(r'(\\.[^.]+\\.[^.]+)$', video_host) or video_host\n assert domain, 'unsupported url: ' + url\n\n # all non-ASCII code points must be quoted (percent-encoded UTF-8)\n url = ''.join([ch if ord(ch) in range(128) else parse.quote(ch) for ch in url])\n video_host = r1(r'https?://([^/]+)/', url)\n video_url = r1(r'https?://[^/]+(.*)', url)\n\n k = r1(r'([^.]+)', domain)\n if k in SITES:\n return (\n import_module('.'.join(['you_get', 'extractors', SITES[k]])),\n url\n )\n else:\n try:\n try:\n location = get_location(url) # t.co isn't happy with fake_headers\n except:\n location = get_location(url, headers=fake_headers)\n except:\n location = get_location(url, headers=fake_headers, get_method='GET')\n\n if location and location != url and not location.startswith('/'):\n return url_to_module(location)\n else:\n return import_module('you_get.extractors.universal'), url\n\n\ndef any_download(url, **kwargs):\n m, url = url_to_module(url)\n m.download(url, **kwargs)\n\n\ndef any_download_playlist(url, **kwargs):\n m, url = url_to_module(url)\n m.download_playlist(url, **kwargs)\n\n\ndef main(**kwargs):\n script_main(any_download, any_download_playlist, **kwargs)\n" }, { "path": "src/you_get/extractor.py", "content": "#!/usr/bin/env python\n\nfrom .common import match1, maybe_print, download_urls, get_filename, parse_host, set_proxy, unset_proxy, get_content, dry_run, player\nfrom .common import print_more_compatible as print\nfrom .util import log\nfrom . import json_output\nimport os\nimport sys\n\nclass Extractor():\n def __init__(self, *args):\n self.url = None\n self.title = None\n self.vid = None\n self.streams = {}\n self.streams_sorted = []\n\n if args:\n self.url = args[0]\n\nclass VideoExtractor():\n def __init__(self, *args):\n self.url = None\n self.title = None\n self.vid = None\n self.m3u8_url = None\n self.streams = {}\n self.streams_sorted = []\n self.audiolang = None\n self.password_protected = False\n self.dash_streams = {}\n self.caption_tracks = {}\n self.out = False\n self.ua = None\n self.referer = None\n self.danmaku = None\n self.lyrics = None\n\n if args:\n self.url = args[0]\n\n def download_by_url(self, url, **kwargs):\n self.url = url\n self.vid = None\n\n if 'extractor_proxy' in kwargs and kwargs['extractor_proxy']:\n set_proxy(parse_host(kwargs['extractor_proxy']))\n self.prepare(**kwargs)\n if self.out:\n return\n if 'extractor_proxy' in kwargs and kwargs['extractor_proxy']:\n unset_proxy()\n\n try:\n self.streams_sorted = [dict([('id', stream_type['id'])] + list(self.streams[stream_type['id']].items())) for stream_type in self.__class__.stream_types if stream_type['id'] in self.streams]\n except:\n self.streams_sorted = [dict([('itag', stream_type['itag'])] + list(self.streams[stream_type['itag']].items())) for stream_type in self.__class__.stream_types if stream_type['itag'] in self.streams]\n\n self.extract(**kwargs)\n\n self.download(**kwargs)\n\n def download_by_vid(self, vid, **kwargs):\n self.url = None\n self.vid = vid\n\n if 'extractor_proxy' in kwargs and kwargs['extractor_proxy']:\n set_proxy(parse_host(kwargs['extractor_proxy']))\n self.prepare(**kwargs)\n if 'extractor_proxy' in kwargs and kwargs['extractor_proxy']:\n unset_proxy()\n\n try:\n self.streams_sorted = [dict([('id', stream_type['id'])] + list(self.streams[stream_type['id']].items())) for stream_type in self.__class__.stream_types if stream_type['id'] in self.streams]\n except:\n self.streams_sorted = [dict([('itag', stream_type['itag'])] + list(self.streams[stream_type['itag']].items())) for stream_type in self.__class__.stream_types if stream_type['itag'] in self.streams]\n\n self.extract(**kwargs)\n\n self.download(**kwargs)\n\n def prepare(self, **kwargs):\n pass\n #raise NotImplementedError()\n\n def extract(self, **kwargs):\n pass\n #raise NotImplementedError()\n\n def p_stream(self, stream_id):\n if stream_id in self.streams:\n stream = self.streams[stream_id]\n else:\n stream = self.dash_streams[stream_id]\n\n if 'itag' in stream:\n print(\" - itag: %s\" % log.sprint(stream_id, log.NEGATIVE))\n else:\n print(\" - format: %s\" % log.sprint(stream_id, log.NEGATIVE))\n\n if 'container' in stream:\n print(\" container: %s\" % stream['container'])\n\n if 'video_profile' in stream:\n maybe_print(\" video-profile: %s\" % stream['video_profile'])\n\n if 'quality' in stream:\n print(\" quality: %s\" % stream['quality'])\n\n if 'size' in stream and 'container' in stream and stream['container'].lower() != 'm3u8':\n if stream['size'] != float('inf') and stream['size'] != 0:\n print(\" size: %s MiB (%s bytes)\" % (round(stream['size'] / 1048576, 1), stream['size']))\n\n if 'm3u8_url' in stream:\n print(\" m3u8_url: {}\".format(stream['m3u8_url']))\n\n if 'itag' in stream:\n print(\" # download-with: %s\" % log.sprint(\"you-get --itag=%s [URL]\" % stream_id, log.UNDERLINE))\n else:\n print(\" # download-with: %s\" % log.sprint(\"you-get --format=%s [URL]\" % stream_id, log.UNDERLINE))\n\n print()\n\n def p_i(self, stream_id):\n if stream_id in self.streams:\n stream = self.streams[stream_id]\n else:\n stream = self.dash_streams[stream_id]\n\n maybe_print(\" - title: %s\" % self.title)\n print(\" size: %s MiB (%s bytes)\" % (round(stream['size'] / 1048576, 1), stream['size']))\n print(\" url: %s\" % self.url)\n print()\n\n sys.stdout.flush()\n\n def p(self, stream_id=None):\n maybe_print(\"site: %s\" % self.__class__.name)\n maybe_print(\"title: %s\" % self.title)\n if stream_id:\n # Print the stream\n print(\"stream:\")\n self.p_stream(stream_id)\n\n elif stream_id is None:\n # Print stream with best quality\n print(\"stream: # Best quality\")\n stream_id = self.streams_sorted[0]['id'] if 'id' in self.streams_sorted[0] else self.streams_sorted[0]['itag']\n self.p_stream(stream_id)\n\n elif stream_id == []:\n print(\"streams: # Available quality and codecs\")\n # Print DASH streams\n if self.dash_streams:\n print(\" [ DASH ] %s\" % ('_' * 36))\n itags = sorted(self.dash_streams,\n key=lambda i: -self.dash_streams[i]['size'])\n for stream in itags:\n self.p_stream(stream)\n # Print all other available streams\n if self.streams_sorted:\n print(\" [ DEFAULT ] %s\" % ('_' * 33))\n for stream in self.streams_sorted:\n self.p_stream(stream['id'] if 'id' in stream else stream['itag'])\n\n if self.audiolang:\n print(\"audio-languages:\")\n for i in self.audiolang:\n print(\" - lang: {}\".format(i['lang']))\n print(\" download-url: {}\\n\".format(i['url']))\n\n sys.stdout.flush()\n\n def p_playlist(self, stream_id=None):\n maybe_print(\"site: %s\" % self.__class__.name)\n print(\"playlist: %s\" % self.title)\n print(\"videos:\")\n\n def download(self, **kwargs):\n if 'json_output' in kwargs and kwargs['json_output']:\n json_output.output(self)\n elif 'info_only' in kwargs and kwargs['info_only']:\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Display the stream\n stream_id = kwargs['stream_id']\n if 'index' not in kwargs:\n self.p(stream_id)\n else:\n self.p_i(stream_id)\n else:\n # Display all available streams\n if 'index' not in kwargs:\n self.p([])\n else:\n stream_id = self.streams_sorted[0]['id'] if 'id' in self.streams_sorted[0] else self.streams_sorted[0]['itag']\n self.p_i(stream_id)\n\n else:\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Download the stream\n stream_id = kwargs['stream_id']\n else:\n # Download stream with the best quality\n from .processor.ffmpeg import has_ffmpeg_installed\n if has_ffmpeg_installed() and player is None and self.dash_streams or not self.streams_sorted:\n #stream_id = list(self.dash_streams)[-1]\n itags = sorted(self.dash_streams,\n key=lambda i: -self.dash_streams[i]['size'])\n stream_id = itags[0]\n else:\n stream_id = self.streams_sorted[0]['id'] if 'id' in self.streams_sorted[0] else self.streams_sorted[0]['itag']\n\n if 'index' not in kwargs:\n self.p(stream_id)\n else:\n self.p_i(stream_id)\n\n if stream_id in self.streams:\n urls = self.streams[stream_id]['src']\n ext = self.streams[stream_id]['container']\n total_size = self.streams[stream_id]['size']\n else:\n urls = self.dash_streams[stream_id]['src']\n ext = self.dash_streams[stream_id]['container']\n total_size = self.dash_streams[stream_id]['size']\n\n if ext == 'm3u8' or ext == 'm4a':\n ext = 'mp4'\n\n if not urls:\n log.wtf('[Failed] Cannot extract video source.')\n # For legacy main()\n headers = {}\n if self.ua is not None:\n headers['User-Agent'] = self.ua\n if self.referer is not None:\n headers['Referer'] = self.referer\n download_urls(urls, self.title, ext, total_size, headers=headers,\n output_dir=kwargs['output_dir'],\n merge=kwargs['merge'],\n av=stream_id in self.dash_streams,\n vid=self.vid)\n\n if 'caption' not in kwargs or not kwargs['caption']:\n print('Skipping captions or danmaku.')\n return\n\n for lang in self.caption_tracks:\n filename = '%s.%s.srt' % (get_filename(self.title), lang)\n print('Saving %s ... ' % filename, end=\"\", flush=True)\n srt = self.caption_tracks[lang]\n with open(os.path.join(kwargs['output_dir'], filename),\n 'w', encoding='utf-8') as x:\n x.write(srt)\n print('Done.')\n\n if self.danmaku is not None and not dry_run:\n filename = '{}.cmt.xml'.format(get_filename(self.title))\n print('Downloading {} ...\\n'.format(filename))\n with open(os.path.join(kwargs['output_dir'], filename), 'w', encoding='utf8') as fp:\n fp.write(self.danmaku)\n\n if self.lyrics is not None and not dry_run:\n filename = '{}.lrc'.format(get_filename(self.title))\n print('Downloading {} ...\\n'.format(filename))\n with open(os.path.join(kwargs['output_dir'], filename), 'w', encoding='utf8') as fp:\n fp.write(self.lyrics)\n\n # For main_dev()\n #download_urls(urls, self.title, self.streams[stream_id]['container'], self.streams[stream_id]['size'])\n keep_obj = kwargs.get('keep_obj', False)\n if not keep_obj:\n self.__init__()\n" }, { "path": "src/you_get/extractors/acfun.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom ..extractor import VideoExtractor\n\nclass AcFun(VideoExtractor):\n name = \"AcFun\"\n\n stream_types = [\n {'id': '2160P', 'qualityType': '2160p'},\n {'id': '1080P60', 'qualityType': '1080p60'},\n {'id': '720P60', 'qualityType': '720p60'},\n {'id': '1080P+', 'qualityType': '1080p+'},\n {'id': '1080P', 'qualityType': '1080p'},\n {'id': '720P', 'qualityType': '720p'},\n {'id': '540P', 'qualityType': '540p'},\n {'id': '360P', 'qualityType': '360p'}\n ] \n\n def prepare(self, **kwargs):\n assert re.match(r'https?://[^\\.]*\\.*acfun\\.[^\\.]+/(\\D|bangumi)/\\D\\D(\\d+)', self.url)\n\n if re.match(r'https?://[^\\.]*\\.*acfun\\.[^\\.]+/\\D/\\D\\D(\\d+)', self.url):\n html = get_content(self.url, headers=fake_headers)\n json_text = match1(html, r\"(?s)videoInfo\\s*=\\s*(\\{.*?\\});\")\n json_data = json.loads(json_text)\n vid = json_data.get('currentVideoInfo').get('id')\n up = json_data.get('user').get('name')\n self.title = json_data.get('title')\n video_list = json_data.get('videoList')\n if len(video_list) > 1:\n self.title += \" - \" + [p.get('title') for p in video_list if p.get('id') == vid][0]\n currentVideoInfo = json_data.get('currentVideoInfo')\n\n elif re.match(r\"https?://[^\\.]*\\.*acfun\\.[^\\.]+/bangumi/aa(\\d+)\", self.url):\n html = get_content(self.url, headers=fake_headers)\n tag_script = match1(html, r'')\n json_text = tag_script[tag_script.find('{') : tag_script.find('};') + 1]\n json_data = json.loads(json_text)\n self.title = json_data['bangumiTitle'] + \" \" + json_data['episodeName'] + \" \" + json_data['title']\n vid = str(json_data['videoId'])\n up = \"acfun\"\n currentVideoInfo = json_data.get('currentVideoInfo')\n\n else:\n raise NotImplementedError() \n\n if 'ksPlayJson' in currentVideoInfo:\n durationMillis = currentVideoInfo['durationMillis']\n ksPlayJson = ksPlayJson = json.loads( currentVideoInfo['ksPlayJson'] )\n representation = ksPlayJson.get('adaptationSet')[0].get('representation')\n stream_list = representation\n\n for stream in stream_list:\n m3u8_url = stream[\"url\"]\n size = durationMillis * stream[\"avgBitrate\"] / 8\n # size = float('inf')\n container = 'mp4'\n stream_id = stream[\"qualityLabel\"]\n quality = stream[\"qualityType\"]\n \n stream_data = dict(src=m3u8_url, size=size, container=container, quality=quality)\n self.streams[stream_id] = stream_data\n\n assert self.title and m3u8_url\n self.title = unescape_html(self.title)\n self.title = escape_file_path(self.title)\n p_title = r1('active\">([^<]+)', html)\n self.title = '%s (%s)' % (self.title, up)\n if p_title:\n self.title = '%s - %s' % (self.title, p_title) \n\n\n def download(self, **kwargs):\n if 'json_output' in kwargs and kwargs['json_output']:\n json_output.output(self)\n elif 'info_only' in kwargs and kwargs['info_only']:\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Display the stream\n stream_id = kwargs['stream_id']\n if 'index' not in kwargs:\n self.p(stream_id)\n else:\n self.p_i(stream_id)\n else:\n # Display all available streams\n if 'index' not in kwargs:\n self.p([])\n else:\n stream_id = self.streams_sorted[0]['id'] if 'id' in self.streams_sorted[0] else self.streams_sorted[0]['itag']\n self.p_i(stream_id)\n\n else:\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Download the stream\n stream_id = kwargs['stream_id']\n else:\n stream_id = self.streams_sorted[0]['id'] if 'id' in self.streams_sorted[0] else self.streams_sorted[0]['itag']\n\n if 'index' not in kwargs:\n self.p(stream_id)\n else:\n self.p_i(stream_id)\n if stream_id in self.streams:\n url = self.streams[stream_id]['src']\n ext = self.streams[stream_id]['container']\n total_size = self.streams[stream_id]['size']\n\n\n if ext == 'm3u8' or ext == 'm4a':\n ext = 'mp4'\n\n if not url:\n log.wtf('[Failed] Cannot extract video source.')\n # For legacy main()\n headers = {}\n if self.ua is not None:\n headers['User-Agent'] = self.ua\n if self.referer is not None:\n headers['Referer'] = self.referer\n\n download_url_ffmpeg(url, self.title, ext, output_dir=kwargs['output_dir'], merge=kwargs['merge']) \n\n if 'caption' not in kwargs or not kwargs['caption']:\n print('Skipping captions or danmaku.')\n return\n\n for lang in self.caption_tracks:\n filename = '%s.%s.srt' % (get_filename(self.title), lang)\n print('Saving %s ... ' % filename, end=\"\", flush=True)\n srt = self.caption_tracks[lang]\n with open(os.path.join(kwargs['output_dir'], filename),\n 'w', encoding='utf-8') as x:\n x.write(srt)\n print('Done.')\n\n if self.danmaku is not None and not dry_run:\n filename = '{}.cmt.xml'.format(get_filename(self.title))\n print('Downloading {} ...\\n'.format(filename))\n with open(os.path.join(kwargs['output_dir'], filename), 'w', encoding='utf8') as fp:\n fp.write(self.danmaku)\n\n if self.lyrics is not None and not dry_run:\n filename = '{}.lrc'.format(get_filename(self.title))\n print('Downloading {} ...\\n'.format(filename))\n with open(os.path.join(kwargs['output_dir'], filename), 'w', encoding='utf8') as fp:\n fp.write(self.lyrics)\n\n # For main_dev()\n #download_urls(urls, self.title, self.streams[stream_id]['container'], self.streams[stream_id]['size'])\n keep_obj = kwargs.get('keep_obj', False)\n if not keep_obj:\n self.__init__()\n\n\n def acfun_download(self, url, output_dir='.', merge=True, info_only=False, **kwargs):\n assert re.match(r'https?://[^\\.]*\\.*acfun\\.[^\\.]+/(\\D|bangumi)/\\D\\D(\\d+)', url)\n\n def getM3u8UrlFromCurrentVideoInfo(currentVideoInfo):\n if 'playInfos' in currentVideoInfo:\n return currentVideoInfo['playInfos'][0]['playUrls'][0]\n elif 'ksPlayJson' in currentVideoInfo:\n ksPlayJson = json.loads( currentVideoInfo['ksPlayJson'] )\n representation = ksPlayJson.get('adaptationSet')[0].get('representation')\n reps = []\n for one in representation:\n reps.append( (one['width']* one['height'], one['url'], one['backupUrl']) )\n return max(reps)[1]\n\n\n if re.match(r'https?://[^\\.]*\\.*acfun\\.[^\\.]+/\\D/\\D\\D(\\d+)', url):\n html = get_content(url, headers=fake_headers)\n json_text = match1(html, r\"(?s)videoInfo\\s*=\\s*(\\{.*?\\});\")\n json_data = json.loads(json_text)\n vid = json_data.get('currentVideoInfo').get('id')\n up = json_data.get('user').get('name')\n title = json_data.get('title')\n video_list = json_data.get('videoList')\n if len(video_list) > 1:\n title += \" - \" + [p.get('title') for p in video_list if p.get('id') == vid][0]\n currentVideoInfo = json_data.get('currentVideoInfo')\n m3u8_url = getM3u8UrlFromCurrentVideoInfo(currentVideoInfo)\n elif re.match(r'https?://[^\\.]*\\.*acfun\\.[^\\.]+/bangumi/aa(\\d+)', url):\n html = get_content(url, headers=fake_headers)\n tag_script = match1(html, r'')\n json_text = tag_script[tag_script.find('{') : tag_script.find('};') + 1]\n json_data = json.loads(json_text)\n title = json_data['bangumiTitle'] + \" \" + json_data['episodeName'] + \" \" + json_data['title']\n vid = str(json_data['videoId'])\n up = \"acfun\"\n\n currentVideoInfo = json_data.get('currentVideoInfo')\n m3u8_url = getM3u8UrlFromCurrentVideoInfo(currentVideoInfo)\n\n else:\n raise NotImplementedError()\n\n assert title and m3u8_url\n title = unescape_html(title)\n title = escape_file_path(title)\n p_title = r1('active\">([^<]+)', html)\n title = '%s (%s)' % (title, up)\n if p_title:\n title = '%s - %s' % (title, p_title)\n\n print_info(site_info, title, 'm3u8', float('inf'))\n if not info_only:\n download_url_ffmpeg(m3u8_url, title, 'mp4', output_dir=output_dir, merge=merge)\n\nsite = AcFun()\nsite_info = \"AcFun.cn\"\ndownload = site.download_by_url\ndownload_playlist = playlist_not_supported('acfun')\n" }, { "path": "src/you_get/extractors/alive.py", "content": "#!/usr/bin/env python\n\n__all__ = ['alive_download']\n\nfrom ..common import *\n\ndef alive_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n html = get_html(url)\n \n title = r1(r'(.+?)<\\/h2>', html)\n artist = r1(r'', html)\n output_dir = '%s/%s - %s' % (output_dir, artist, album_name)\n ids = json.loads(r1(r'',\n html).replace('"', '').replace(';', '\"'))['ids']\n track_nr = 1\n for id in ids:\n song_data = baidu_get_song_data(id)\n song_url = baidu_get_song_url(song_data)\n song_title = baidu_get_song_title(song_data)\n song_lrc = baidu_get_song_lyric(song_data)\n file_name = '%02d.%s' % (track_nr, song_title)\n\n type, ext, size = url_info(song_url, faker=True)\n print_info(site_info, song_title, type, size)\n if not info_only:\n download_urls([song_url], file_name, ext, size,\n output_dir, merge=merge, faker=True)\n\n if song_lrc:\n type, ext, size = url_info(song_lrc, faker=True)\n print_info(site_info, song_title, type, size)\n if not info_only:\n download_urls([song_lrc], file_name, ext,\n size, output_dir, faker=True)\n\n track_nr += 1\n\n\ndef baidu_download(url, output_dir='.', stream_type=None, merge=True, info_only=False, **kwargs):\n\n if re.match(r'https?://pan.baidu.com', url):\n real_url, title, ext, size = baidu_pan_download(url)\n print_info('BaiduPan', title, ext, size)\n if not info_only:\n print('Hold on...')\n time.sleep(5)\n download_urls([real_url], title, ext, size,\n output_dir, url, merge=merge, faker=True)\n elif re.match(r'https?://music.baidu.com/album/\\d+', url):\n id = r1(r'https?://music.baidu.com/album/(\\d+)', url)\n baidu_download_album(id, output_dir, merge, info_only)\n\n elif re.match(r'https?://music.baidu.com/song/\\d+', url):\n id = r1(r'https?://music.baidu.com/song/(\\d+)', url)\n baidu_download_song(id, output_dir, merge, info_only)\n\n elif re.match('https?://tieba.baidu.com/', url):\n try:\n # embedded videos\n embed_download(url, output_dir, merge=merge, info_only=info_only, **kwargs)\n except:\n # images\n html = get_html(url)\n title = r1(r'title:\"([^\"]+)\"', html)\n\n vhsrc = re.findall(r'\"BDE_Image\"[^>]+src=\"([^\"]+\\.mp4)\"', html) or \\\n re.findall(r'vhsrc=\"([^\"]+)\"', html)\n if len(vhsrc) > 0:\n ext = 'mp4'\n size = url_size(vhsrc[0])\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(vhsrc, title, ext, size,\n output_dir=output_dir, merge=False)\n\n items = re.findall(\n r'//tiebapic.baidu.com/forum/w[^\"]+/([^/\"]+)', html)\n urls = ['http://tiebapic.baidu.com/forum/pic/item/' + i\n for i in set(items)]\n\n # handle albums\n kw = r1(r'kw=([^&]+)', html) or r1(r\"kw:'([^']+)'\", html)\n tid = r1(r'tid=(\\d+)', html) or r1(r\"tid:'([^']+)'\", html)\n album_url = 'http://tieba.baidu.com/photo/g/bw/picture/list?kw=%s&tid=%s&pe=%s' % (kw, tid, 1000)\n album_info = json.loads(get_content(album_url))\n for i in album_info['data']['pic_list']:\n urls.append(\n 'http://tiebapic.baidu.com/forum/pic/item/' + i['pic_id'] + '.jpg')\n\n ext = 'jpg'\n size = float('Inf')\n print_info(site_info, title, ext, size)\n\n if not info_only:\n download_urls(urls, title, ext, size,\n output_dir=output_dir, merge=False)\n\n\ndef baidu_pan_download(url):\n errno_patt = r'errno\":([^\"]+),'\n refer_url = \"\"\n fake_headers = {\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',\n 'Accept-Charset': 'UTF-8,*;q=0.5',\n 'Accept-Encoding': 'gzip,deflate,sdch',\n 'Accept-Language': 'en-US,en;q=0.8',\n 'Host': 'pan.baidu.com',\n 'Origin': 'http://pan.baidu.com',\n 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:13.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2500.0 Safari/537.36',\n 'Referer': refer_url\n }\n if cookies:\n print('Use user specified cookies')\n else:\n print('Generating cookies...')\n fake_headers['Cookie'] = baidu_pan_gen_cookies(url)\n refer_url = \"http://pan.baidu.com\"\n html = get_content(url, fake_headers, decoded=True)\n isprotected = False\n sign, timestamp, bdstoken, appid, primary_id, fs_id, uk = baidu_pan_parse(\n html)\n if sign is None:\n if re.findall(r'\\baccess-code\\b', html):\n isprotected = True\n sign, timestamp, bdstoken, appid, primary_id, fs_id, uk, fake_headers, psk = baidu_pan_protected_share(\n url)\n # raise NotImplementedError(\"Password required!\")\n if isprotected != True:\n raise AssertionError(\"Share not found or canceled: %s\" % url)\n if bdstoken is None:\n bdstoken = \"\"\n if isprotected != True:\n sign, timestamp, bdstoken, appid, primary_id, fs_id, uk = baidu_pan_parse(\n html)\n request_url = \"http://pan.baidu.com/api/sharedownload?sign=%s×tamp=%s&bdstoken=%s&channel=chunlei&clienttype=0&web=1&app_id=%s\" % (\n sign, timestamp, bdstoken, appid)\n refer_url = url\n post_data = {\n 'encrypt': 0,\n 'product': 'share',\n 'uk': uk,\n 'primaryid': primary_id,\n 'fid_list': '[' + fs_id + ']'\n }\n if isprotected == True:\n post_data['sekey'] = psk\n response_content = post_content(request_url, fake_headers, post_data, True)\n errno = match1(response_content, errno_patt)\n if errno != \"0\":\n raise AssertionError(\n \"Server refused to provide download link! (Errno:%s)\" % errno)\n real_url = r1(r'dlink\":\"([^\"]+)\"', response_content).replace('\\\\/', '/')\n title = r1(r'server_filename\":\"([^\"]+)\"', response_content)\n assert real_url\n type, ext, size = url_info(real_url, faker=True)\n title_wrapped = json.loads('{\"wrapper\":\"%s\"}' % title)\n title = title_wrapped['wrapper']\n logging.debug(real_url)\n return real_url, title, ext, size\n\n\ndef baidu_pan_parse(html):\n sign_patt = r'sign\":\"([^\"]+)\"'\n timestamp_patt = r'timestamp\":([^\"]+),'\n appid_patt = r'app_id\":\"([^\"]+)\"'\n bdstoken_patt = r'bdstoken\":\"([^\"]+)\"'\n fs_id_patt = r'fs_id\":([^\"]+),'\n uk_patt = r'uk\":([^\"]+),'\n errno_patt = r'errno\":([^\"]+),'\n primary_id_patt = r'shareid\":([^\"]+),'\n sign = match1(html, sign_patt)\n timestamp = match1(html, timestamp_patt)\n appid = match1(html, appid_patt)\n bdstoken = match1(html, bdstoken_patt)\n fs_id = match1(html, fs_id_patt)\n uk = match1(html, uk_patt)\n primary_id = match1(html, primary_id_patt)\n return sign, timestamp, bdstoken, appid, primary_id, fs_id, uk\n\n\ndef baidu_pan_gen_cookies(url, post_data=None):\n from http import cookiejar\n cookiejar = cookiejar.CookieJar()\n opener = request.build_opener(request.HTTPCookieProcessor(cookiejar))\n resp = opener.open('http://pan.baidu.com')\n if post_data != None:\n resp = opener.open(url, bytes(parse.urlencode(post_data), 'utf-8'))\n return cookjar2hdr(cookiejar)\n\n\ndef baidu_pan_protected_share(url):\n print('This share is protected by password!')\n inpwd = input('Please provide unlock password: ')\n inpwd = inpwd.replace(' ', '').replace('\\t', '')\n print('Please wait...')\n post_pwd = {\n 'pwd': inpwd,\n 'vcode': None,\n 'vstr': None\n }\n from http import cookiejar\n import time\n cookiejar = cookiejar.CookieJar()\n opener = request.build_opener(request.HTTPCookieProcessor(cookiejar))\n resp = opener.open('http://pan.baidu.com')\n resp = opener.open(url)\n init_url = resp.geturl()\n verify_url = 'http://pan.baidu.com/share/verify?%s&t=%s&channel=chunlei&clienttype=0&web=1' % (\n init_url.split('?', 1)[1], int(time.time()))\n refer_url = init_url\n fake_headers = {\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',\n 'Accept-Charset': 'UTF-8,*;q=0.5',\n 'Accept-Encoding': 'gzip,deflate,sdch',\n 'Accept-Language': 'en-US,en;q=0.8',\n 'Host': 'pan.baidu.com',\n 'Origin': 'http://pan.baidu.com',\n 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:13.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2500.0 Safari/537.36',\n 'Referer': refer_url\n }\n opener.addheaders = dict2triplet(fake_headers)\n pwd_resp = opener.open(verify_url, bytes(\n parse.urlencode(post_pwd), 'utf-8'))\n pwd_resp_str = ungzip(pwd_resp.read()).decode('utf-8')\n pwd_res = json.loads(pwd_resp_str)\n if pwd_res['errno'] != 0:\n raise AssertionError(\n 'Server returned an error: %s (Incorrect password?)' % pwd_res['errno'])\n pg_resp = opener.open('http://pan.baidu.com/share/link?%s' %\n init_url.split('?', 1)[1])\n content = ungzip(pg_resp.read()).decode('utf-8')\n sign, timestamp, bdstoken, appid, primary_id, fs_id, uk = baidu_pan_parse(\n content)\n psk = query_cookiejar(cookiejar, 'BDCLND')\n psk = parse.unquote(psk)\n fake_headers['Cookie'] = cookjar2hdr(cookiejar)\n return sign, timestamp, bdstoken, appid, primary_id, fs_id, uk, fake_headers, psk\n\n\ndef cookjar2hdr(cookiejar):\n cookie_str = ''\n for i in cookiejar:\n cookie_str = cookie_str + i.name + '=' + i.value + ';'\n return cookie_str[:-1]\n\n\ndef query_cookiejar(cookiejar, name):\n for i in cookiejar:\n if i.name == name:\n return i.value\n\n\ndef dict2triplet(dictin):\n out_triplet = []\n for i in dictin:\n out_triplet.append((i, dictin[i]))\n return out_triplet\n\nsite_info = \"Baidu.com\"\ndownload = baidu_download\ndownload_playlist = playlist_not_supported(\"baidu\")\n" }, { "path": "src/you_get/extractors/bandcamp.py", "content": "#!/usr/bin/env python\n\n__all__ = ['bandcamp_download']\n\nfrom ..common import *\n\ndef bandcamp_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url)\n trackinfo = json.loads(r1(r'(\\[{\"(video_poster_url|video_caption)\".*}\\]),', html))\n for track in trackinfo:\n track_num = track['track_num']\n title = '%s. %s' % (track_num, track['title'])\n file_url = 'http:' + track['file']['mp3-128']\n mime, ext, size = url_info(file_url)\n\n print_info(site_info, title, mime, size)\n if not info_only:\n download_urls([file_url], title, ext, size, output_dir, merge=merge)\n\nsite_info = \"Bandcamp.com\"\ndownload = bandcamp_download\ndownload_playlist = bandcamp_download\n" }, { "path": "src/you_get/extractors/baomihua.py", "content": "#!/usr/bin/env python\n\n__all__ = ['baomihua_download', 'baomihua_download_by_id']\n\nfrom ..common import *\n\nimport urllib\n\ndef baomihua_headers(referer=None, cookie=None):\n\t# a reasonable UA\n\tua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36'\n\theaders = {'Accept': '*/*', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': ua}\n\tif referer is not None:\n\t\theaders.update({'Referer': referer})\n\tif cookie is not None:\n\t\theaders.update({'Cookie': cookie})\n\treturn headers\n\t\ndef baomihua_download_by_id(id, title=None, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html('http://play.baomihua.com/getvideourl.aspx?flvid=%s&devicetype=phone_app' % id)\n host = r1(r'host=([^&]*)', html)\n assert host\n type = r1(r'videofiletype=([^&]*)', html)\n assert type\n vid = r1(r'&stream_name=([^&]*)', html)\n assert vid\n dir_str = r1(r'&dir=([^&]*)', html).strip()\n url = \"http://%s/%s/%s.%s\" % (host, dir_str, vid, type)\n _, ext, size = url_info(url, headers=baomihua_headers())\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir, merge = merge, headers=baomihua_headers())\n\ndef baomihua_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url)\n title = r1(r'(.*)', html)\n assert title\n id = r1(r'flvid\\s*=\\s*(\\d+)', html)\n assert id\n baomihua_download_by_id(id, title, output_dir=output_dir, merge=merge, info_only=info_only)\n\nsite_info = \"baomihua.com\"\ndownload = baomihua_download\ndownload_playlist = playlist_not_supported('baomihua')\n" }, { "path": "src/you_get/extractors/bigthink.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom ..extractor import VideoExtractor\n\nimport json\n\nclass Bigthink(VideoExtractor):\n name = \"Bigthink\"\n\n stream_types = [ #this is just a sample. Will make it in prepare()\n # {'id': '1080'},\n # {'id': '720'},\n # {'id': '360'},\n # {'id': '288'},\n # {'id': '190'},\n # {'id': '180'},\n \n ]\n\n @staticmethod\n def get_streams_by_id(account_number, video_id):\n \"\"\"\n int, int->list\n \n Get the height of the videos.\n \n Since brightcove is using 3 kinds of links: rtmp, http and https,\n we will be using the HTTPS one to make it secure.\n \n If somehow akamaihd.net is blocked by the Great Fucking Wall,\n change the \"startswith https\" to http.\n \"\"\"\n endpoint = 'https://edge.api.brightcove.com/playback/v1/accounts/{account_number}/videos/{video_id}'.format(account_number = account_number, video_id = video_id)\n fake_header_id = fake_headers\n #is this somehow related to the time? Magic....\n fake_header_id['Accept'] ='application/json;pk=BCpkADawqM1cc6wmJQC2tvoXZt4mrB7bFfi6zGt9QnOzprPZcGLE9OMGJwspQwKfuFYuCjAAJ53JdjI8zGFx1ll4rxhYJ255AXH1BQ10rnm34weknpfG-sippyQ'\n\n html = get_content(endpoint, headers= fake_header_id)\n html_json = json.loads(html)\n\n link_list = []\n\n for i in html_json['sources']:\n if 'src' in i: #to avoid KeyError\n if i['src'].startswith('https'):\n link_list.append((str(i['height']), i['src']))\n\n return link_list\n\n def prepare(self, **kwargs):\n\n html = get_content(self.url)\n\n self.title = match1(html, r' bangumi/play/ep\n # redirect: bangumi.bilibili.com/anime -> bangumi/play/ep\n elif re.match(r'https?://(www\\.)?bilibili\\.com/bangumi/play/ss(\\d+)', self.url) or \\\n re.match(r'https?://bangumi\\.bilibili\\.com/anime/(\\d+)/play', self.url):\n initial_state_text = match1(html_content, r'__INITIAL_STATE__=(.*?);\\(function\\(\\)') # FIXME\n initial_state = json.loads(initial_state_text)\n ep_id = initial_state['epList'][0]['id']\n self.url = 'https://www.bilibili.com/bangumi/play/ep%s' % ep_id\n html_content = get_content(self.url, headers=self.bilibili_headers(referer=self.url))\n\n # redirect: s\n elif re.match(r'https?://(www\\.)?bilibili\\.com/s/(.+)', self.url):\n self.url = 'https://www.bilibili.com/%s' % match1(self.url, r'/s/(.+)')\n html_content = get_content(self.url, headers=self.bilibili_headers())\n\n # redirect: festival\n elif re.match(r'https?://(www\\.)?bilibili\\.com/festival/(.+)', self.url):\n self.url = 'https://www.bilibili.com/video/%s' % match1(self.url, r'bvid=([^&]+)')\n html_content = get_content(self.url, headers=self.bilibili_headers())\n\n # sort it out\n if re.match(r'https?://(www\\.)?bilibili\\.com/audio/au(\\d+)', self.url):\n sort = 'audio'\n elif re.match(r'https?://(www\\.)?bilibili\\.com/bangumi/play/ep(\\d+)', self.url):\n sort = 'bangumi'\n elif match1(html_content, r'))', html)\n json_data = json.loads(coub_data)\n return json_data\n\n\ndef get_file_path(merge, output_dir, title, url):\n mime, ext, size = url_info(url)\n file_name = get_output_filename([], title, ext, output_dir, merge)\n file_path = os.path.join(output_dir, file_name)\n return file_name, file_path\n\n\ndef get_loop_file_path(title, output_dir):\n return os.path.join(output_dir, get_output_filename([], title, \"txt\", None, False))\n\n\ndef cleanup_files(files):\n for file in files:\n os.remove(file)\n\n\nsite_info = \"coub.com\"\ndownload = coub_download\ndownload_playlist = playlist_not_supported('coub')\n" }, { "path": "src/you_get/extractors/dailymotion.py", "content": "#!/usr/bin/env python\n\n__all__ = ['dailymotion_download']\n\nfrom ..common import *\nimport urllib.parse\n\ndef rebuilt_url(url):\n path = urllib.parse.urlparse(url).path\n aid = path.split('/')[-1].split('_')[0]\n return 'http://www.dailymotion.com/embed/video/{}?autoplay=1'.format(aid)\n\ndef dailymotion_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n \"\"\"Downloads Dailymotion videos by URL.\n \"\"\"\n\n html = get_content(rebuilt_url(url))\n info = json.loads(match1(html, r'qualities\":({.+?}),\"'))\n title = match1(html, r'\"video_title\"\\s*:\\s*\"([^\"]+)\"') or \\\n match1(html, r'\"title\"\\s*:\\s*\"([^\"]+)\"')\n title = unicodize(title)\n\n for quality in ['1080','720','480','380','240','144','auto']:\n try:\n real_url = info[quality][1][\"url\"]\n if real_url:\n break\n except KeyError:\n pass\n\n mime, ext, size = url_info(real_url)\n\n print_info(site_info, title, mime, size)\n if not info_only:\n download_urls([real_url], title, ext, size, output_dir=output_dir, merge=merge)\n\nsite_info = \"Dailymotion.com\"\ndownload = dailymotion_download\ndownload_playlist = playlist_not_supported('dailymotion')\n" }, { "path": "src/you_get/extractors/douban.py", "content": "#!/usr/bin/env python\n\n__all__ = ['douban_download']\n\nimport urllib.request, urllib.parse\nfrom ..common import *\n\ndef douban_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n html = get_html(url)\n\n if re.match(r'https?://movie', url):\n title = match1(html, 'name=\"description\" content=\"([^\"]+)')\n tid = match1(url, r'trailer/(\\d+)')\n real_url = 'https://movie.douban.com/trailer/video_url?tid=%s' % tid\n type, ext, size = url_info(real_url)\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([real_url], title, ext, size, output_dir, merge = merge)\n\n elif 'subject' in url:\n titles = re.findall(r'data-title=\"([^\"]*)\">', html)\n song_id = re.findall(r'
  • (.+?)'\n\n hit = re.search(patt, url)\n if hit is None:\n log.wtf('Unknown url pattern')\n vid = hit.group(1)\n\n page = get_content(url, headers=headers)\n hit = re.search(title_patt, page)\n if hit is None:\n title = vid\n else:\n title = hit.group(1)\n\n meta = json.loads(get_content(ep + vid))\n if meta['error'] != 0:\n log.wtf('Error from API server')\n m3u8_url = meta['data']['video_url']\n print_info('Douyu Video', title, 'm3u8', 0, m3u8_url=m3u8_url)\n if not info_only:\n urls = general_m3u8_extractor(m3u8_url)\n download_urls(urls, title, 'ts', 0, output_dir=output_dir, merge=merge, **kwargs)\n\n\ndef douyutv_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n if 'v.douyu.com/show/' in url:\n douyutv_video_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n\n url = re.sub(r'.*douyu.com','https://m.douyu.com/room', url)\n html = get_content(url, headers)\n room_id_patt = r'\"rid\"\\s*:\\s*(\\d+),'\n room_id = match1(html, room_id_patt)\n if room_id == \"0\":\n room_id = url[url.rfind('/') + 1:]\n\n api_url = \"http://www.douyutv.com/api/v1/\"\n args = \"room/%s?aid=wp&client_sys=wp&time=%d\" % (room_id, int(time.time()))\n auth_md5 = (args + \"zNzMV1y4EMxOHS6I5WKm\").encode(\"utf-8\")\n auth_str = hashlib.md5(auth_md5).hexdigest()\n json_request_url = \"%s%s&auth=%s\" % (api_url, args, auth_str)\n\n content = get_content(json_request_url, headers)\n json_content = json.loads(content)\n data = json_content['data']\n server_status = json_content.get('error', 0)\n if server_status != 0:\n raise ValueError(\"Server returned error:%s\" % server_status)\n\n title = data.get('room_name')\n show_status = data.get('show_status')\n if show_status != \"1\":\n raise ValueError(\"The live stream is not online! (Errno:%s)\" % server_status)\n\n real_url = data.get('rtmp_url') + '/' + data.get('rtmp_live')\n\n print_info(site_info, title, 'flv', float('inf'))\n if not info_only:\n download_url_ffmpeg(real_url, title, 'flv', params={}, output_dir=output_dir, merge=merge)\n\n\nsite_info = \"douyu.com\"\ndownload = douyutv_download\ndownload_playlist = playlist_not_supported('douyu')\n" }, { "path": "src/you_get/extractors/ehow.py", "content": "#!/usr/bin/env python\n\n__all__ = ['ehow_download']\n\nfrom ..common import *\n\ndef ehow_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n\t\n\tassert re.search(r'http://www.ehow.com/video_', url), \"URL you entered is not supported\"\n\n\thtml = get_html(url)\n\tcontentid = r1(r'', html)\n\tvid = r1(r'\"demand_ehow_videoid\":\"([^\"]+)\"', html)\n\tassert vid\n\n\txml = get_html('http://www.ehow.com/services/video/series.xml?demand_ehow_videoid=%s' % vid)\n \n\tfrom xml.dom.minidom import parseString\n\tdoc = parseString(xml)\n\ttab = doc.getElementsByTagName('related')[0].firstChild\n\n\tfor video in tab.childNodes:\n\t\tif re.search(contentid, video.attributes['link'].value):\n\t\t\turl = video.attributes['flv'].value\n\t\t\tbreak\n\n\ttitle = video.attributes['title'].value\n\tassert title \n\n\ttype, ext, size = url_info(url)\n\tprint_info(site_info, title, type, size)\n\t\n\tif not info_only:\n\t\tdownload_urls([url], title, ext, size, output_dir, merge = merge)\n\nsite_info = \"ehow.com\"\ndownload = ehow_download\ndownload_playlist = playlist_not_supported('ehow')\n" }, { "path": "src/you_get/extractors/embed.py", "content": "__all__ = ['embed_download']\n\nimport urllib.parse\n\nfrom ..common import *\n\nfrom .bilibili import bilibili_download\nfrom .dailymotion import dailymotion_download\nfrom .iqiyi import iqiyi_download_by_vid\nfrom .le import letvcloud_download_by_vu\nfrom .netease import netease_download\nfrom .qq import qq_download_by_vid\nfrom .sina import sina_download_by_vid\nfrom .tudou import tudou_download_by_id\nfrom .vimeo import vimeo_download_by_id\nfrom .youku import youku_download_by_vid\nfrom . import iqiyi\nfrom . import bokecc\n\n\"\"\"\nrefer to http://open.youku.com/tools\n\"\"\"\nyouku_embed_patterns = [ r'youku\\.com/v_show/id_([a-zA-Z0-9=]+)',\n r'player\\.youku\\.com/player\\.php/sid/([a-zA-Z0-9=]+)/v\\.swf',\n r'loader\\.swf\\?VideoIDS=([a-zA-Z0-9=]+)',\n r'player\\.youku\\.com/embed/([a-zA-Z0-9=]+)',\n r'YKU.Player\\(\\'[a-zA-Z0-9]+\\',{ client_id: \\'[a-zA-Z0-9]+\\', vid: \\'([a-zA-Z0-9]+)\\''\n ]\n\n\"\"\"\nhttp://www.tudou.com/programs/view/html5embed.action?type=0&code=3LS_URGvl54&lcode=&resourceId=0_06_05_99\n\"\"\"\ntudou_embed_patterns = [ r'tudou\\.com[a-zA-Z0-9\\/\\?=\\&\\.\\;]+code=([a-zA-Z0-9_-]+)\\&',\n r'www\\.tudou\\.com/v/([a-zA-Z0-9_-]+)/[^\"]*v\\.swf'\n ]\n\n\"\"\"\nrefer to http://open.tudou.com/wiki/video/info\n\"\"\"\ntudou_api_patterns = [ ]\n\niqiyi_embed_patterns = [ r'player\\.video\\.qiyi\\.com/([^/]+)/[^/]+/[^/]+/[^/]+\\.swf[^\"]+tvId=(\\d+)' ]\n\nnetease_embed_patterns = [ r'(http://\\w+\\.163\\.com/movie/[^\\'\"]+)' ]\n\nvimeo_embed_patters = [ r'player\\.vimeo\\.com/video/(\\d+)' ]\n\ndailymotion_embed_patterns = [ r'www\\.dailymotion\\.com/embed/video/(\\w+)' ]\n\n\"\"\"\ncheck the share button on http://www.bilibili.com/video/av5079467/\n\"\"\"\nbilibili_embed_patterns = [ r'static\\.hdslb\\.com/miniloader\\.swf.*aid=(\\d+)' ]\n\n\n'''\nhttp://open.iqiyi.com/lib/player.html\n'''\niqiyi_patterns = [r'(?:\\\"|\\')(https?://dispatcher\\.video\\.qiyi\\.com\\/disp\\/shareplayer\\.swf\\?.+?)(?:\\\"|\\')',\n r'(?:\\\"|\\')(https?://open\\.iqiyi\\.com\\/developer\\/player_js\\/coopPlayerIndex\\.html\\?.+?)(?:\\\"|\\')']\n\nbokecc_patterns = [r'bokecc\\.com/flash/pocle/player\\.swf\\?siteid=(.+?)&vid=(.{32})']\n\nrecur_limit = 3\n\n\ndef embed_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n content = get_content(url, headers=fake_headers)\n found = False\n title = match1(content, '([^<>]+)')\n\n vids = matchall(content, youku_embed_patterns)\n for vid in set(vids):\n found = True\n youku_download_by_vid(vid, title=title, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n vids = matchall(content, tudou_embed_patterns)\n for vid in set(vids):\n found = True\n tudou_download_by_id(vid, title=title, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n vids = matchall(content, iqiyi_embed_patterns)\n for vid in vids:\n found = True\n iqiyi_download_by_vid((vid[1], vid[0]), title=title, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n urls = matchall(content, netease_embed_patterns)\n for url in urls:\n found = True\n netease_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n urls = matchall(content, vimeo_embed_patters)\n for url in urls:\n found = True\n vimeo_download_by_id(url, title=title, output_dir=output_dir, merge=merge, info_only=info_only, referer=url, **kwargs)\n\n urls = matchall(content, dailymotion_embed_patterns)\n for url in urls:\n found = True\n dailymotion_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n aids = matchall(content, bilibili_embed_patterns)\n for aid in aids:\n found = True\n url = 'http://www.bilibili.com/video/av%s/' % aid\n bilibili_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n iqiyi_urls = matchall(content, iqiyi_patterns)\n for url in iqiyi_urls:\n found = True\n iqiyi.download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n bokecc_metas = matchall(content, bokecc_patterns)\n for meta in bokecc_metas:\n found = True\n bokecc.bokecc_download_by_id(meta[1], output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n if found:\n return True\n\n # Try harder, check all iframes\n if 'recur_lv' not in kwargs or kwargs['recur_lv'] < recur_limit:\n r = kwargs.get('recur_lv')\n if r is None:\n r = 1\n else:\n r += 1\n iframes = matchall(content, [r'(.+)', html)\n\n if title is None:\n title = url\n\n sd_urls = list(set([\n unicodize(str.replace(i, '\\\\/', '/'))\n for i in re.findall(r'sd_src_no_ratelimit:\"([^\"]*)\"', html)\n ]))\n hd_urls = list(set([\n unicodize(str.replace(i, '\\\\/', '/'))\n for i in re.findall(r'hd_src_no_ratelimit:\"([^\"]*)\"', html)\n ]))\n urls = hd_urls if hd_urls else sd_urls\n\n type, ext, size = url_info(urls[0], True)\n size = urls_size(urls)\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls(urls, title, ext, size, output_dir, merge=False)\n\nsite_info = \"Facebook.com\"\ndownload = facebook_download\ndownload_playlist = playlist_not_supported('facebook')\n" }, { "path": "src/you_get/extractors/fc2video.py", "content": "#!/usr/bin/env python\n\n__all__ = ['fc2video_download']\n\nfrom ..common import *\nfrom hashlib import md5\nfrom urllib.parse import urlparse\n\n#----------------------------------------------------------------------\ndef makeMimi(upid):\n \"\"\"From http://cdn37.atwikiimg.com/sitescript/pub/dksitescript/FC2.site.js\n Also com.hps.util.fc2.FC2EncrptUtil.makeMimiLocal\n L110\"\"\"\n strSeed = \"gGddgPfeaf_gzyr\"\n prehash = upid + \"_\" + strSeed\n return md5(prehash.encode('utf-8')).hexdigest()\n\n#----------------------------------------------------------------------\ndef fc2video_download_by_upid(upid, output_dir = '.', merge = True, info_only = False, **kwargs):\n \"\"\"\"\"\"\n fake_headers = {\n 'DNT': '1',\n 'Accept-Encoding': 'gzip, deflate, sdch',\n 'Accept-Language': 'en-CA,en;q=0.8,en-US;q=0.6,zh-CN;q=0.4,zh;q=0.2',\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.58 Safari/537.36',\n 'Accept': '*/*',\n 'X-Requested-With': 'ShockwaveFlash/19.0.0.245',\n 'Connection': 'keep-alive',\n }\n api_base = 'http://video.fc2.com/ginfo.php?upid={upid}&mimi={mimi}'.format(upid = upid, mimi = makeMimi(upid))\n html = get_content(api_base, headers=fake_headers)\n\n video_url = match1(html, r'filepath=(.+)&sec')\n video_url = video_url.replace('&mid', '?mid')\n\n title = match1(html, r'&title=([^&]+)')\n\n type, ext, size = url_info(video_url, headers=fake_headers)\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([video_url], title, ext, size, output_dir, merge=merge, headers = fake_headers)\n\n#----------------------------------------------------------------------\ndef fc2video_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n \"\"\"wrapper\"\"\"\n #'http://video.fc2.com/en/content/20151021bTVKnbEw'\n #'http://xiaojiadianvideo.asia/content/20151021bTVKnbEw'\n #'http://video.fc2.com/ja/content/20151021bTVKnbEw'\n #'http://video.fc2.com/tw/content/20151021bTVKnbEw'\n hostname = urlparse(url).hostname\n if not ('fc2.com' in hostname or 'xiaojiadianvideo.asia' in hostname):\n return False\n upid = match1(url, r'.+/content/(\\w+)')\n\n fc2video_download_by_upid(upid, output_dir, merge, info_only)\n\nsite_info = \"FC2Video\"\ndownload = fc2video_download\ndownload_playlist = playlist_not_supported('fc2video')\n" }, { "path": "src/you_get/extractors/flickr.py", "content": "#!/usr/bin/env python\n\n__all__ = ['flickr_download_main']\n\nfrom ..common import *\n\nimport json\n\npattern_url_photoset = r'https?://www\\.flickr\\.com/photos/.+/(?:(?:sets)|(?:albums))?/([^/]+)'\npattern_url_photostream = r'https?://www\\.flickr\\.com/photos/([^/]+)(?:/|(?:/page))?$'\npattern_url_single_photo = r'https?://www\\.flickr\\.com/photos/[^/]+/(\\d+)'\npattern_url_gallery = r'https?://www\\.flickr\\.com/photos/[^/]+/galleries/(\\d+)'\npattern_url_group = r'https?://www\\.flickr\\.com/groups/([^/]+)'\npattern_url_favorite = r'https?://www\\.flickr\\.com/photos/([^/]+)/favorites'\n\npattern_inline_title = r'([^<]*)'\npattern_inline_api_key = r'api\\.site_key\\s*=\\s*\"([^\"]+)\"'\npattern_inline_img_url = r'\"url\":\"([^\"]+)\",\"key\":\"[^\"]+\"}}'\npattern_inline_NSID = r'\"nsid\"\\s*:\\s*\"([^\"]+)\"'\npattern_inline_video_mark = r'(\"mediaType\":\"video\")'\n\n# (api_key, method, ext, page)\ntmpl_api_call = (\n 'https://api.flickr.com/services/rest?'\n '&format=json&nojsoncallback=1'\n # UNCOMMENT FOR TESTING\n #'&per_page=5'\n '&per_page=500'\n # this parameter CANNOT take control of 'flickr.galleries.getPhotos'\n # though the doc said it should.\n # it's always considered to be 500\n '&api_key=%s'\n '&method=flickr.%s'\n '&extras=url_sq,url_q,url_t,url_s,url_n,url_m,url_z,url_c,url_l,url_h,url_k,url_o,media'\n '%s&page=%d'\n)\n\ntmpl_api_call_video_info = (\n 'https://api.flickr.com/services/rest?'\n '&format=json&nojsoncallback=1'\n '&method=flickr.video.getStreamInfo'\n '&api_key=%s'\n '&photo_id=%s'\n '&secret=%s'\n)\n\ntmpl_api_call_photo_info = (\n 'https://api.flickr.com/services/rest?'\n '&format=json&nojsoncallback=1'\n '&method=flickr.photos.getInfo'\n '&api_key=%s'\n '&photo_id=%s'\n)\n\n# looks that flickr won't return urls for all sizes\n# we required in 'extras field without a acceptable header\ndummy_header = {\n 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0'\n}\ndef get_content_headered(url):\n return get_content(url, dummy_header)\n\ndef get_photoset_id(url, page):\n return match1(url, pattern_url_photoset)\n\ndef get_photo_id(url, page):\n return match1(url, pattern_url_single_photo)\n\ndef get_gallery_id(url, page):\n return match1(url, pattern_url_gallery)\n\ndef get_api_key(page):\n match = match1(page, pattern_inline_api_key)\n # this happens only when the url points to a gallery page\n # that contains no inline api_key(and never makes xhr api calls)\n # in fact this might be a better approach for getting a temporary api key\n # since there's no place for a user to add custom information that may\n # misguide the regex in the homepage\n if not match:\n return match1(get_html('https://flickr.com'), pattern_inline_api_key)\n return match\n\ndef get_NSID(url, page):\n return match1(page, pattern_inline_NSID)\n\n# [\n# (\n# regex_match_url,\n# remote_api_method,\n# additional_query_parameter_for_method,\n# parser_for_additional_parameter,\n# field_where_photourls_are_saved\n# )\n# ]\nurl_patterns = [\n # www.flickr.com/photos/{username|NSID}/sets|albums/{album-id}\n (\n pattern_url_photoset,\n 'photosets.getPhotos',\n 'photoset_id',\n get_photoset_id,\n 'photoset'\n ),\n # www.flickr.com/photos/{username|NSID}/{pageN}?\n (\n pattern_url_photostream,\n # according to flickr api documentation, this method needs to be\n # authenticated in order to filter photo visible to the calling user\n # but it seems works fine anonymously as well\n 'people.getPhotos',\n 'user_id',\n get_NSID,\n 'photos'\n ),\n # www.flickr.com/photos/{username|NSID}/galleries/{gallery-id}\n (\n pattern_url_gallery,\n 'galleries.getPhotos',\n 'gallery_id',\n get_gallery_id,\n 'photos'\n ),\n # www.flickr.com/groups/{groupname|groupNSID}/\n (\n pattern_url_group,\n 'groups.pools.getPhotos',\n 'group_id',\n get_NSID,\n 'photos'\n ),\n # www.flickr.com/photos/{username|NSID}/favorites/*\n (\n pattern_url_favorite,\n 'favorites.getList',\n 'user_id',\n get_NSID,\n 'photos'\n )\n]\n\ndef flickr_download_main(url, output_dir = '.', merge = False, info_only = False, **kwargs):\n urls = None\n size = 'o' # works for collections only\n title = None\n if 'stream_id' in kwargs:\n size = kwargs['stream_id']\n if match1(url, pattern_url_single_photo):\n url, title = get_single_photo_url(url)\n urls = [url]\n else:\n urls, title = fetch_photo_url_list(url, size)\n index = 0\n for url in urls:\n mime, ext, size = url_info(url)\n print_info('Flickr.com', title, mime, size)\n if not info_only:\n suffix = '[%d]' % index\n download_urls([url], title + suffix, ext, False, output_dir, None, False, False)\n index = index + 1\n\ndef fetch_photo_url_list(url, size):\n for pattern in url_patterns:\n # FIXME: fix multiple matching since the match group is dropped\n if match1(url, pattern[0]):\n return fetch_photo_url_list_impl(url, size, *pattern[1:])\n raise NotImplementedError('Flickr extractor is not supported for %s.' % url)\n\ndef fetch_photo_url_list_impl(url, size, method, id_field, id_parse_func, collection_name):\n page = get_html(url)\n api_key = get_api_key(page)\n ext_field = ''\n if id_parse_func:\n ext_field = '&%s=%s' % (id_field, id_parse_func(url, page))\n page_number = 1\n urls = []\n while True:\n call_url = tmpl_api_call % (api_key, method, ext_field, page_number)\n photoset = json.loads(get_content_headered(call_url))[collection_name]\n pagen = photoset['page']\n pages = photoset['pages']\n for info in photoset['photo']:\n url = get_url_of_largest(info, api_key, size)\n urls.append(url)\n page_number = page_number + 1\n # the typeof 'page' and 'pages' may change in different methods\n if str(pagen) == str(pages):\n break\n return urls, match1(page, pattern_inline_title)\n\n# image size suffixes used in inline json 'key' field\n# listed in descending order\nsize_suffixes = ['o', 'k', 'h', 'l', 'c', 'z', 'm', 'n', 's', 't', 'q', 'sq']\n\ndef get_orig_video_source(api_key, pid, secret):\n parsed = json.loads(get_content_headered(tmpl_api_call_video_info % (api_key, pid, secret)))\n for stream in parsed['streams']['stream']:\n if stream['type'] == 'orig':\n return stream['_content'].replace('\\\\', '')\n return None\n\ndef get_url_of_largest(info, api_key, size):\n if info['media'] == 'photo':\n sizes = size_suffixes\n if size in sizes:\n sizes = sizes[sizes.index(size):]\n for suffix in sizes:\n if 'url_' + suffix in info:\n return info['url_' + suffix].replace('\\\\', '')\n return None\n else:\n return get_orig_video_source(api_key, info['id'], info['secret'])\n\ndef get_single_photo_url(url):\n page = get_html(url)\n pid = get_photo_id(url, page)\n title = match1(page, pattern_inline_title)\n if match1(page, pattern_inline_video_mark):\n api_key = get_api_key(page)\n reply = get_content(tmpl_api_call_photo_info % (api_key, get_photo_id(url, page)))\n secret = json.loads(reply)['photo']['secret']\n return get_orig_video_source(api_key, pid, secret), title\n #last match always has the best resolution\n match = match1(page, pattern_inline_img_url)\n return 'https:' + match.replace('\\\\', ''), title\n\nsite_info = \"Flickr.com\"\ndownload = flickr_download_main\ndownload_playlist = playlist_not_supported('flickr');\n" }, { "path": "src/you_get/extractors/freesound.py", "content": "#!/usr/bin/env python\n\n__all__ = ['freesound_download']\n\nfrom ..common import *\n\ndef freesound_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n page = get_html(url)\n \n title = r1(r' 0:\n res.append(self.mapping_table[num % self.base])\n num = num // self.base\n return ''.join(res[::-1])\n\n\nclass Funshion(VideoExtractor):\n name = \"funshion\"\n stream_types = [\n {'id': 'sdvd'},\n {'id': 'sdvd_h265'},\n {'id': 'hd'},\n {'id': 'hd_h265'},\n {'id': 'dvd'},\n {'id': 'dvd_h265'},\n {'id': 'tv'},\n {'id': 'tv_h265'}\n ]\n a_mobile_url = 'http://m.fun.tv/implay/?mid=302555'\n video_ep = 'http://pv.funshion.com/v7/video/play/?id={}&cl=mweb&uc=111'\n media_ep = 'http://pm.funshion.com/v7/media/play/?id={}&cl=mweb&uc=111'\n coeff = None\n\n @classmethod\n def fetch_magic(cls, url):\n def search_dict(a_dict, target):\n for key, val in a_dict.items():\n if val == target:\n return key\n\n magic_list = []\n\n page = get_content(url)\n src = re.findall(r'src=\"(.+?)\"', page)\n js = [path for path in src if path.endswith('.js')]\n\n host = 'http://' + urllib.parse.urlparse(url).netloc\n js_path = [urllib.parse.urljoin(host, rel_path) for rel_path in js]\n\n for p in js_path:\n if 'mtool' in p or 'mcore' in p:\n js_text = get_content(p)\n hit = re.search(r'\\(\\'(.+?)\\',(\\d+),(\\d+),\\'(.+?)\\'\\.split\\(\\'\\|\\'\\),\\d+,\\{\\}\\)', js_text)\n\n code = hit.group(1)\n base = hit.group(2)\n size = hit.group(3)\n names = hit.group(4).split('|')\n\n mapping = KBaseMapping(base=int(base))\n sym_to_name = {}\n for no in range(int(size), 0, -1):\n no_in_base = mapping.mapping(no)\n val = names[no] if no < len(names) and names[no] else no_in_base\n sym_to_name[no_in_base] = val\n\n moz_ec_name = search_dict(sym_to_name, 'mozEcName')\n push = search_dict(sym_to_name, 'push')\n patt = r'{}\\.{}\\(\"(.+?)\"\\)'.format(moz_ec_name, push)\n ec_list = re.findall(patt, code)\n [magic_list.append(sym_to_name[ec]) for ec in ec_list]\n return magic_list\n\n @classmethod\n def get_coeff(cls, magic_list):\n magic_set = set(magic_list)\n no_dup = []\n for item in magic_list:\n if item in magic_set:\n magic_set.remove(item)\n no_dup.append(item)\n # really necessary?\n\n coeff = [0, 0, 0, 0]\n for num_pair in no_dup:\n idx = int(num_pair[-1])\n val = int(num_pair[:-1], 16)\n coeff[idx] = val\n\n return coeff\n\n @classmethod\n def funshion_decrypt(cls, a_bytes, coeff):\n res_list = []\n pos = 0\n while pos < len(a_bytes):\n a = a_bytes[pos]\n if pos == len(a_bytes) - 1:\n res_list.append(a)\n pos += 1\n else:\n b = a_bytes[pos + 1]\n m = a * coeff[0] + b * coeff[2]\n n = a * coeff[1] + b * coeff[3]\n res_list.append(m & 0xff)\n res_list.append(n & 0xff)\n pos += 2\n return bytes(res_list).decode('utf8')\n\n @classmethod\n def funshion_decrypt_str(cls, a_str, coeff):\n # r'.{27}0' pattern, untested\n if len(a_str) == 28 and a_str[-1] == '0':\n data_bytes = base64.b64decode(a_str[:27] + '=')\n clear = cls.funshion_decrypt(data_bytes, coeff)\n return binascii.hexlify(clear.encode('utf8')).upper()\n\n data_bytes = base64.b64decode(a_str[2:])\n return cls.funshion_decrypt(data_bytes, coeff)\n\n @classmethod\n def checksum(cls, sha1_str):\n if len(sha1_str) != 41:\n return False\n if not re.match(r'[0-9A-Za-z]{41}', sha1_str):\n return False\n sha1 = sha1_str[:-1]\n if (15 & sum([int(char, 16) for char in sha1])) == int(sha1_str[-1], 16):\n return True\n return False\n\n @classmethod\n def get_cdninfo(cls, hashid):\n url = 'http://jobsfe.funshion.com/query/v1/mp4/{}.json'.format(hashid)\n meta = json.loads(get_content(url, decoded=False).decode('utf8'))\n return meta['playlist'][0]['urls']\n\n @classmethod\n def dec_playinfo(cls, info, coeff):\n res = None\n clear = cls.funshion_decrypt_str(info['infohash'], coeff)\n if cls.checksum(clear):\n res = dict(hashid=clear[:40], token=cls.funshion_decrypt_str(info['token'], coeff))\n else:\n clear = cls.funshion_decrypt_str(info['infohash_prev'], coeff)\n if cls.checksum(clear):\n res = dict(hashid=clear[:40], token=cls.funshion_decrypt_str(info['token_prev'], coeff))\n return res\n\n def prepare(self, **kwargs):\n if self.__class__.coeff is None:\n magic_list = self.__class__.fetch_magic(self.__class__.a_mobile_url)\n self.__class__.coeff = self.__class__.get_coeff(magic_list)\n\n if 'title' not in kwargs:\n url = 'http://pv.funshion.com/v5/video/profile/?id={}&cl=mweb&uc=111'.format(self.vid)\n meta = json.loads(get_content(url))\n self.title = meta['name']\n else:\n self.title = kwargs['title']\n\n ep_url = self.__class__.video_ep if 'single_video' in kwargs else self.__class__.media_ep\n\n url = ep_url.format(self.vid)\n meta = json.loads(get_content(url))\n streams = meta['playlist']\n for stream in streams:\n definition = stream['code']\n for s in stream['playinfo']:\n codec = 'h' + s['codec'][2:]\n # h.264 -> h264\n for st in self.__class__.stream_types:\n s_id = '{}_{}'.format(definition, codec)\n if codec == 'h264':\n s_id = definition\n if s_id == st['id']:\n clear_info = self.__class__.dec_playinfo(s, self.__class__.coeff)\n cdn_list = self.__class__.get_cdninfo(clear_info['hashid'])\n base_url = cdn_list[0]\n vf = urllib.parse.quote(s['vf'])\n video_size = int(s['filesize'])\n token = urllib.parse.quote(base64.b64encode(clear_info['token'].encode('utf8')))\n video_url = '{}?token={}&vf={}'.format(base_url, token, vf)\n self.streams[s_id] = dict(size=video_size, src=[video_url], container='mp4')\n\n\ndef funshion_download(url, **kwargs):\n if re.match(r'http://www.fun.tv/vplay/v-(\\w+)', url):\n vid = re.search(r'http://www.fun.tv/vplay/v-(\\w+)', url).group(1)\n Funshion().download_by_vid(vid, single_video=True, **kwargs)\n elif re.match(r'http://www.fun.tv/vplay/.*g-(\\w+)', url):\n epid = re.search(r'http://www.fun.tv/vplay/.*g-(\\w+)', url).group(1)\n url = 'http://pm.funshion.com/v5/media/episode?id={}&cl=mweb&uc=111'.format(epid)\n meta = json.loads(get_content(url))\n drama_name = meta['name']\n\n extractor = Funshion()\n for ep in meta['episodes']:\n title = '{}_{}_{}'.format(drama_name, ep['num'], ep['name'])\n extractor.download_by_vid(ep['id'], title=title, **kwargs)\n else:\n log.wtf('Unknown url pattern')\n\nsite_info = \"funshion\"\ndownload = funshion_download\ndownload_playlist = playlist_not_supported('funshion')\n" }, { "path": "src/you_get/extractors/giphy.py", "content": "#!/usr/bin/env python\n\n__all__ = ['giphy_download']\n\nfrom ..common import *\n\ndef giphy_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url)\n\n url = list(set([\n unicodize(str.replace(i, '\\\\/', '/'))\n for i in re.findall(r'', html)\n ]))\n\n title = r1(r'', html)\n\n if title is None:\n title = url[0]\n\n type, ext, size = url_info(url[0], True)\n size = urls_size(url)\n\n type = \"video/mp4\"\n ext = \"mp4\"\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls(url, title, ext, size, output_dir, merge=False)\n\nsite_info = \"Giphy.com\"\ndownload = giphy_download\ndownload_playlist = playlist_not_supported('giphy')\n" }, { "path": "src/you_get/extractors/google.py", "content": "#!/usr/bin/env python\n\n__all__ = ['google_download']\n\nfrom ..common import *\n\nimport re\n\n# YouTube media encoding options, in descending quality order.\n# taken from http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs, 3/22/2013.\nyoutube_codecs = [\n {'itag': 38, 'container': 'MP4', 'video_resolution': '3072p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3.5-5', 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': 46, 'container': 'WebM', 'video_resolution': '1080p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n {'itag': 37, 'container': 'MP4', 'video_resolution': '1080p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3-4.3', 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': 102, 'container': 'WebM', 'video_resolution': '720p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '2', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n {'itag': 45, 'container': 'WebM', 'video_resolution': '720p', 'video_encoding': '', 'video_profile': '', 'video_bitrate': '', 'audio_encoding': '', 'audio_bitrate': ''},\n {'itag': 22, 'container': 'MP4', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '2-2.9', 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': 84, 'container': 'MP4', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '2-2.9', 'audio_encoding': 'AAC', 'audio_bitrate': '152'},\n {'itag': 120, 'container': 'FLV', 'video_resolution': '720p', 'video_encoding': 'AVC', 'video_profile': 'Main@L3.1', 'video_bitrate': '2', 'audio_encoding': 'AAC', 'audio_bitrate': '128'},\n {'itag': 85, 'container': 'MP4', 'video_resolution': '520p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '2-2.9', 'audio_encoding': 'AAC', 'audio_bitrate': '152'},\n {'itag': 44, 'container': 'WebM', 'video_resolution': '480p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '1', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': 35, 'container': 'FLV', 'video_resolution': '480p', 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.8-1', 'audio_encoding': 'AAC', 'audio_bitrate': '128'},\n {'itag': 101, 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n {'itag': 100, 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': 43, 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '0.5', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': 34, 'container': 'FLV', 'video_resolution': '360p', 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '128'},\n {'itag': 82, 'container': 'MP4', 'video_resolution': '360p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': 18, 'container': 'MP4', 'video_resolution': '270p/360p', 'video_encoding': 'H.264', 'video_profile': 'Baseline', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': 6, 'container': 'FLV', 'video_resolution': '270p', 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.8', 'audio_encoding': 'MP3', 'audio_bitrate': '64'},\n {'itag': 83, 'container': 'MP4', 'video_resolution': '240p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': 13, 'container': '3GP', 'video_resolution': '', 'video_encoding': 'MPEG-4 Visual', 'video_profile': '', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': ''},\n {'itag': 5, 'container': 'FLV', 'video_resolution': '240p', 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.25', 'audio_encoding': 'MP3', 'audio_bitrate': '64'},\n {'itag': 36, 'container': '3GP', 'video_resolution': '240p', 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.17', 'audio_encoding': 'AAC', 'audio_bitrate': '38'},\n {'itag': 17, 'container': '3GP', 'video_resolution': '144p', 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.05', 'audio_encoding': 'AAC', 'audio_bitrate': '24'},\n]\nfmt_level = dict(\n zip(\n [str(codec['itag'])\n for codec in\n youtube_codecs],\n range(len(youtube_codecs))))\n\ndef google_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n # Percent-encoding Unicode URL\n url = parse.quote(url, safe = ':/+%?=')\n\n service = url.split('/')[2].split('.')[0]\n\n if service == 'plus': # Google Plus\n\n # attempt to extract images first\n # TBD: posts with > 4 images\n # TBD: album links\n html = get_html(parse.unquote(url), faker=True)\n real_urls = []\n for src in re.findall(r'src=\"([^\"]+)\"[^>]*itemprop=\"image\"', html):\n t = src.split('/')\n t[0], t[-2] = t[0] or 'https:', 's0-d'\n u = '/'.join(t)\n real_urls.append(u)\n if not real_urls:\n real_urls = re.findall(r']*>([^<\\n]+)', post_html)\n\n if title is None:\n response = request.urlopen(request.Request(real_url))\n if response.headers['content-disposition']:\n filename = parse.unquote(r1(r'filename=\"?(.+)\"?', response.headers['content-disposition'])).split('.')\n title = ''.join(filename[:-1])\n except: pass\n\n for (i, real_url) in enumerate(real_urls):\n title_i = \"%s[%s]\" % (title, i) if len(real_urls) > 1 else title\n type, ext, size = url_info(real_url)\n if ext is None: ext = 'mp4'\n\n print_info(site_info, title_i, ext, size)\n if not info_only:\n download_urls([real_url], title_i, ext, size, output_dir, merge = merge)\n\n elif service in ['docs', 'drive'] : # Google Docs\n\n html = get_content(url, headers=fake_headers)\n\n title = r1(r'\"title\":\"([^\"]*)\"', html) or r1(r' 1:\n title = \".\".join(title.split('.')[:-1])\n\n docid = r1('/file/d/([^/]+)', url)\n\n request.install_opener(request.build_opener(request.HTTPCookieProcessor()))\n\n real_url = \"https://docs.google.com/uc?export=download&confirm=no_antivirus&id=%s\" % docid\n redirected_url = get_location(real_url)\n if real_url != redirected_url:\n# tiny file - get real url here\n type, ext, size = url_info(redirected_url)\n real_url = redirected_url\n else:\n# huge file - the real_url is a confirm page and real url is in it\n confirm_page = get_content(real_url)\n hrefs = re.findall(r'href=\"(.+?)\"', confirm_page)\n for u in hrefs:\n if u.startswith('/uc?export=download'):\n rel = unescape_html(u)\n confirm_url = 'https://docs.google.com' + rel\n real_url = get_location(confirm_url)\n _, ext, size = url_info(real_url, headers=fake_headers)\n if size is None:\n size = 0\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls([real_url], title, ext, size, output_dir, merge = merge)\n\nsite_info = \"Google.com\"\ndownload = google_download\ndownload_playlist = playlist_not_supported('google')\n" }, { "path": "src/you_get/extractors/heavymusic.py", "content": "#!/usr/bin/env python\n\n__all__ = ['heavymusic_download']\n\nfrom ..common import *\n\ndef heavymusic_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url)\n tracks = re.findall(r'href=\"(online2\\.php[^\"]+)\"', html)\n for track in tracks:\n band = r1(r'band=([^&]*)', track)\n album = r1(r'album=([^&]*)', track)\n title = r1(r'track=([^&]*)', track)\n file_url = 'http://www.heavy-music.ru/online2.php?band=%s&album=%s&track=%s' % (parse.quote(band), parse.quote(album), parse.quote(title))\n _, _, size = url_info(file_url)\n\n print_info(site_info, title, 'mp3', size)\n if not info_only:\n download_urls([file_url], title[:-4], 'mp3', size, output_dir, merge=merge)\n\nsite_info = \"heavy-music.ru\"\ndownload = heavymusic_download\ndownload_playlist = heavymusic_download\n" }, { "path": "src/you_get/extractors/huomaotv.py", "content": "#!/usr/bin/env python\n\n__all__ = ['huomaotv_download']\n\nfrom ..common import *\n\n\ndef get_mobile_room_url(room_id):\n return 'http://www.huomao.com/mobile/mob_live/%s' % room_id\n\n\ndef get_m3u8_url(stream_id):\n return 'http://live-ws.huomaotv.cn/live/%s/playlist.m3u8' % stream_id\n\n\ndef huomaotv_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n room_id_pattern = r'huomao.com/(\\d+)'\n room_id = match1(url, room_id_pattern)\n html = get_content(get_mobile_room_url(room_id))\n\n stream_id_pattern = r'id=\"html_stream\" value=\"(\\w+)\"'\n stream_id = match1(html, stream_id_pattern)\n\n m3u8_url = get_m3u8_url(stream_id)\n\n title = match1(html, r'([^<]{1,9999})')\n\n print_info(site_info, title, 'm3u8', float('inf'))\n\n if not info_only:\n download_url_ffmpeg(m3u8_url, title, 'm3u8', None, output_dir=output_dir, merge=merge)\n\n\nsite_info = 'huomao.com'\ndownload = huomaotv_download\ndownload_playlist = playlist_not_supported('huomao')\n" }, { "path": "src/you_get/extractors/icourses.py", "content": "#!/usr/bin/env python\nfrom ..common import *\nfrom urllib import parse, error\nimport random\nfrom time import sleep\nimport datetime\nimport hashlib\nimport base64\nimport logging\nimport re\nfrom xml.dom.minidom import parseString\n\n__all__ = ['icourses_download', 'icourses_playlist_download']\n\n\ndef icourses_download(url, output_dir='.', **kwargs):\n if 'showResDetail.action' in url:\n hit = re.search(r'id=(\\d+)&courseId=(\\d+)', url)\n url = 'http://www.icourses.cn/jpk/changeforVideo.action?resId={}&courseId={}'.format(hit.group(1), hit.group(2))\n if re.match(r'http://www.icourses.cn/coursestatic/course_(\\d+).html', url):\n raise Exception('You can download it with -l flag')\n icourses_parser = ICousesExactor(url=url)\n icourses_parser.basic_extract()\n title = icourses_parser.title\n size = None\n for i in range(5):\n try:\n # use this url only for size\n size_url = icourses_parser.generate_url(0)\n _, type_, size = url_info(size_url, headers=fake_headers)\n except error.HTTPError:\n logging.warning('Failed to fetch the video file! Retrying...')\n sleep(random.Random().randint(2, 5)) # Prevent from blockage\n else:\n print_info(site_info, title, type_, size)\n break\n\n if size is None:\n raise Exception(\"Failed\")\n\n if not kwargs['info_only']:\n real_url = icourses_parser.update_url(0)\n headers = fake_headers.copy()\n headers['Referer'] = url\n download_urls_icourses(real_url, title, 'flv',total_size=size, output_dir=output_dir, max_size=15728640, dyn_callback=icourses_parser.update_url)\n return\n\n\ndef get_course_title(url, course_type, page=None):\n if page is None:\n try:\n # shard course page could be gbk but with charset=\"utf-8\"\n page = get_content(url, decoded=False).decode('gbk')\n except UnicodeDecodeError:\n page = get_content(url, decoded=False).decode('utf8')\n\n if course_type == 'shared_old':\n patt = r'(.+?)<\\/div>'\n elif course_type == 'shared_new':\n patt = r'

    (.+?)<\\/h1>'\n else:\n patt = r'(.+?)<\\/div>'\n\n return re.search(patt, page).group(1)\n\n\ndef public_course_playlist(url, page=None):\n host = 'http://www.icourses.cn/'\n patt = r'(?:.|\\n)+?'\n\n if page is None:\n page = get_content(url)\n playlist = re.findall(patt, page)\n return [(host+i[0], i[1]) for i in playlist]\n\n\ndef public_course_get_title(url, page=None):\n patt = r'.+?第(\\d+)讲'\n\n if page is None:\n page = get_content(url)\n seq_num = int(re.search(patt, page).group(1)) - 1\n course_main_title = get_course_title(url, 'public', page)\n return '{}_第{}讲_{}'.format(course_main_title, seq_num+1, public_course_playlist(url, page)[seq_num][1])\n\n\ndef icourses_playlist_download(url, output_dir='.', **kwargs):\n page_type_patt = r'showSectionNode\\(this,(\\d+),(\\d+)\\)'\n resid_courseid_patt = r'changeforvideo\\(\\'(\\d+)\\',\\'(\\d+)\\',\\'(\\d+)\\'\\)'\n ep = 'http://www.icourses.cn/jpk/viewCharacterDetail.action?sectionId={}&courseId={}'\n change_for_video_ip = 'http://www.icourses.cn/jpk/changeforVideo.action?resId={}&courseId={}'\n video_list = []\n\n if 'viewVCourse' in url:\n playlist = public_course_playlist(url)\n for video in playlist:\n icourses_download(video[0], output_dir=output_dir, **kwargs)\n return\n elif 'coursestatic' in url:\n course_page = get_content(url)\n page_navi_vars = re.search(page_type_patt, course_page)\n\n if page_navi_vars is None: # type 2 shared course\n video_list = icourses_playlist_new(url, course_page)\n else: # type 1 shared course\n sec_page = get_content(ep.format(page_navi_vars.group(2), page_navi_vars.group(1)))\n video_list = re.findall(resid_courseid_patt, sec_page)\n elif 'viewCharacterDetail.action' in url or 'changeforVideo.action' in url:\n page = get_content(url)\n video_list = re.findall(resid_courseid_patt, page)\n\n if not video_list:\n raise Exception('Unknown url pattern')\n\n for video in video_list:\n video_url = change_for_video_ip.format(video[0], video[1])\n sleep(random.Random().randint(0, 5)) # Prevent from blockage\n icourses_download(video_url, output_dir=output_dir, **kwargs)\n\n\ndef icourses_playlist_new(url, page=None):\n # 2 helpers using same interface in the js code\n def to_chap(course_id, chap_id, mod):\n ep = 'http://www.icourses.cn/jpk/viewCharacterDetail2.action?courseId={}&characId={}&mod={}'\n req = post_content(ep.format(course_id, chap_id, mod), post_data={})\n return req\n\n def to_sec(course_id, chap_id, mod):\n ep = 'http://www.icourses.cn/jpk/viewCharacterDetail2.action?courseId={}&characId={}&mod={}'\n req = post_content(ep.format(course_id, chap_id, mod), post_data={})\n return req\n\n def show_sec(course_id, chap_id):\n ep = 'http://www.icourses.cn/jpk/getSectionNode.action?courseId={}&characId={}&mod=2'\n req = post_content(ep.format(course_id, chap_id), post_data={})\n return req\n\n if page is None:\n page = get_content(url)\n chap_patt = r'

    .+?id=\"parent_row_(\\d+)\".+?onclick=\"(\\w+)\\((.+)\\)\"'\n to_chap_patt = r'this,(\\d+),(\\d+),(\\d)'\n show_sec_patt = r'this,(\\d+),(\\d+)'\n res_patt = r'res_showResDetail\\(\\'(\\d+)\\',\\'.+?\\',\\'\\d+\\',\\'mp4\\',\\'(\\d+)\\'\\)'\n l = re.findall(chap_patt, page)\n for i in l:\n if i[1] == 'ajaxtocharac':\n hit = re.search(to_chap_patt, i[2])\n page = to_chap(hit.group(1), hit.group(2), hit.group(3))\n hit_list = re.findall(res_patt, page)\n if hit_list:\n return get_playlist(hit_list[0][0], hit_list[0][1])\n for hit in hit_list:\n print(hit)\n elif i[1] == 'showSectionNode2':\n hit = re.search(show_sec_patt, i[2])\n page = show_sec(hit.group(1), hit.group(2))\n # print(page)\n patt = r'ajaxtosection\\(this,(\\d+),(\\d+),(\\d+)\\)'\n hit_list = re.findall(patt, page)\n # print(hit_list)\n for hit in hit_list:\n page = to_sec(hit[0], hit[1], hit[2])\n vlist = re.findall(res_patt, page)\n if vlist:\n return get_playlist(vlist[0][0], vlist[0][1])\n raise Exception(\"No video found in this playlist\")\n\n\ndef get_playlist(res_id, course_id):\n ep = 'http://www.icourses.cn/jpk/changeforVideo.action?resId={}&courseId={}'\n req = get_content(ep.format(res_id, course_id))\n\n patt = r' (.*?)'\n title_b_patt = r'
    ((.|\\n)*?)
    '\n title_a = match1(self.page, title_a_patt).strip()\n title_b = match1(self.page, title_b_patt).strip()\n title = title_a + title_b\n title = re.sub('( +|\\n|\\t|\\r| )', '', unescape_html(title).replace(' ', ''))\n self.title = title\n\n def get_flashvars(self):\n patt = r'var flashvars\\s*=\\s*(\\{(?:.|\\n)+?\\});'\n hit = re.search(patt, self.page)\n if hit is None:\n raise Exception('Cannot find flashvars')\n flashvar_str = hit.group(1)\n\n uuid = re.search(r'uuid\\s*:\\s*\\\"?(\\w+)\\\"?', flashvar_str).group(1)\n other = re.search(r'other\\s*:\\s*\"(.*?)\"', flashvar_str).group(1)\n isvc = re.search(r'IService\\s*:\\s*\\'(.+?)\\'', flashvar_str).group(1)\n\n player_time_patt = r'MPlayer.swf\\?v\\=(\\d+)'\n player_time = re.search(player_time_patt, self.page).group(1)\n\n self.flashvars = dict(IService=isvc, uuid=uuid, other=other, v=player_time)\n\n def api_req(self, url):\n xml_str = get_content(url)\n dom = parseString(xml_str)\n status = dom.getElementsByTagName('result')[0].getAttribute('status')\n if status != 'success':\n raise Exception('API returned fail')\n\n api_res = {}\n meta = dom.getElementsByTagName('metadata')\n for m in meta:\n key = m.getAttribute('name')\n val = m.firstChild.nodeValue\n api_res[key] = val\n self.api_data = api_res\n\n def basic_extract(self):\n self.get_title()\n self.get_flashvars()\n api_req_url = '{}?{}'.format(self.flashvars['IService'], parse.urlencode(self.flashvars))\n self.api_req(api_req_url)\n\n def do_extract(self, received=0):\n self.basic_extract()\n return self.generate_url(received)\n\n def update_url(self, received):\n args = self.common_args.copy()\n play_type = 'seek' if received else 'play'\n received = received if received else -1\n args['ls'] = play_type\n args['start'] = received + 1\n args['lt'] = self.get_date_str()\n if self.enc_mode:\n ssl_ts, sign = self.get_sign(self.media_url)\n extra_args = dict(h=sign, r=ssl_ts, p=self.__class__.ENCRYPT_MOD_VER)\n args.update(extra_args)\n return '{}?{}'.format(self.media_url, parse.urlencode(args))\n\n @classmethod\n def get_date_str(self):\n fmt_str = '%-m-%-d/%-H:%-M:%-S'\n now = datetime.datetime.now()\n try:\n date_str = now.strftime(fmt_str)\n except ValueError: # msvcrt\n date_str = '{}-{}/{}:{}:{}'.format(now.month, now.day, now.hour, now.minute, now.second)\n return date_str\n\n def generate_url(self, received):\n media_host = self.get_media_host(self.api_data['host'])\n media_url = media_host + self.api_data['url']\n self.media_url = media_url\n\n common_args = dict(lv=self.__class__.PLAYER_BASE_VER)\n h = self.api_data.get('h')\n r = self.api_data.get('p', self.__class__.ENCRYPT_MOD_VER)\n\n if self.api_data['ssl'] != 'true':\n self.enc_mode = False\n common_args.update(dict(h=h, r=r))\n else:\n self.enc_mode = True\n common_args['p'] = self.__class__.ENCRYPT_MOD_VER\n self.common_args = common_args\n return self.update_url(received)\n\n def get_sign(self, media_url):\n media_host = parse.urlparse(media_url).netloc\n ran = random.randint(0, 9999999)\n ssl_callback = get_content('http://{}/ssl/ssl.shtml?r={}'.format(media_host, ran)).split(',')\n ssl_ts = int(datetime.datetime.strptime(ssl_callback[1], \"%b %d %H:%M:%S %Y\").timestamp() + int(ssl_callback[0]))\n sign_this = self.__class__.ENCRYPT_SALT + parse.urlparse(media_url).path + str(ssl_ts)\n arg_h = base64.b64encode(hashlib.md5(bytes(sign_this, 'utf-8')).digest(), altchars=b'-_')\n return ssl_ts, arg_h.decode('utf-8').strip('=')\n\n def get_media_host(self, ori_host):\n res = get_content(ori_host + '/ssl/host.shtml').strip()\n path = parse.urlparse(ori_host).path\n return ''.join([res, path])\n\n\ndef download_urls_icourses(url, title, ext, total_size, output_dir='.', headers=None, **kwargs):\n if dry_run or player:\n log.wtf('Non standard protocol')\n\n title = get_filename(title)\n\n filename = '%s.%s' % (title, ext)\n filepath = os.path.join(output_dir, filename)\n if not force and os.path.exists(filepath):\n print('Skipping {}: file already exists\\n'.format(filepath))\n return\n bar = SimpleProgressBar(total_size, 1)\n print('Downloading %s ...' % tr(filename))\n url_save_icourses(url, filepath, bar, total_size, headers=headers, **kwargs)\n bar.done()\n\n print()\n\n\ndef url_save_icourses(url, filepath, bar, total_size, dyn_callback=None, is_part=False, max_size=0, headers=None):\n def dyn_update_url(received):\n if callable(dyn_callback):\n logging.debug('Calling callback %s for new URL from %s' % (dyn_callback.__name__, received))\n return dyn_callback(received)\n if bar is None:\n bar = DummyProgressBar()\n if os.path.exists(filepath):\n if not force:\n if not is_part:\n bar.done()\n print('Skipping %s: file already exists' % tr(os.path.basename(filepath)))\n else:\n filesize = os.path.getsize(filepath)\n bar.update_received(filesize)\n return\n else:\n if not is_part:\n bar.done()\n print('Overwriting %s' % os.path.basename(filepath), '...')\n elif not os.path.exists(os.path.dirname(filepath)):\n os.mkdir(os.path.dirname(filepath))\n\n temp_filepath = filepath + '.download'\n received = 0\n if not force:\n open_mode = 'ab'\n\n if os.path.exists(temp_filepath):\n tempfile_size = os.path.getsize(temp_filepath)\n received += tempfile_size\n bar.update_received(tempfile_size)\n else:\n open_mode = 'wb'\n\n if received:\n url = dyn_update_url(received)\n\n if headers is None:\n headers = {}\n response = urlopen_with_retry(request.Request(url, headers=headers))\n# Do not update content-length here.\n# Only the 1st segment's content-length is the content-length of the file.\n# For other segments, content-length is the standard one, 15 * 1024 * 1024\n\n with open(temp_filepath, open_mode) as output:\n before_this_uri = received\n# received - before_this_uri is size of the buf we get from one uri\n while True:\n update_bs = 256 * 1024\n left_bytes = total_size - received\n to_read = left_bytes if left_bytes <= update_bs else update_bs\n# calc the block size to read -- The server can fail to send an EOF\n buffer = response.read(to_read)\n if not buffer:\n logging.debug('Got EOF from server')\n break\n output.write(buffer)\n received += len(buffer)\n bar.update_received(len(buffer))\n if received >= total_size:\n break\n if max_size and (received - before_this_uri) >= max_size:\n url = dyn_update_url(received)\n before_this_uri = received\n response = urlopen_with_retry(request.Request(url, headers=headers))\n\n assert received == os.path.getsize(temp_filepath), '%s == %s' % (received, os.path.getsize(temp_filepath))\n\n if os.access(filepath, os.W_OK):\n os.remove(filepath) # on Windows rename could fail if destination filepath exists\n os.rename(temp_filepath, filepath)\n\nsite_info = 'icourses.cn'\ndownload = icourses_download\ndownload_playlist = icourses_playlist_download\n" }, { "path": "src/you_get/extractors/ifeng.py", "content": "#!/usr/bin/env python\n\n__all__ = ['ifeng_download', 'ifeng_download_by_id']\n\nfrom ..common import *\n\ndef ifeng_download_by_id(id, title = None, output_dir = '.', merge = True, info_only = False):\n assert r1(r'([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})', id), id\n url = 'http://vxml.ifengimg.com/video_info_new/%s/%s/%s.xml' % (id[-2], id[-2:], id)\n xml = get_html(url, 'utf-8')\n title = r1(r'Name=\"([^\"]+)\"', xml)\n title = unescape_html(title)\n url = r1(r'VideoPlayUrl=\"([^\"]+)\"', xml)\n from random import randint\n r = randint(10, 19)\n url = url.replace('http://wideo.ifeng.com/', 'http://ips.ifeng.com/wideo.ifeng.com/')\n type, ext, size = url_info(url)\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir, merge = merge)\n\ndef ifeng_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n# old pattern /uuid.shtml\n# now it could be #uuid\n id = r1(r'([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})', url)\n if id:\n return ifeng_download_by_id(id, None, output_dir = output_dir, merge = merge, info_only = info_only)\n\n html = get_content(url)\n uuid_pattern = r'\"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\"'\n id = r1(r'var vid=\"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\"', html)\n if id is None:\n video_pattern = r'\"vid\"\\s*:\\s*' + uuid_pattern\n id = match1(html, video_pattern)\n assert id, \"can't find video info\"\n return ifeng_download_by_id(id, None, output_dir = output_dir, merge = merge, info_only = info_only)\n\nsite_info = \"ifeng.com\"\ndownload = ifeng_download\ndownload_playlist = playlist_not_supported('ifeng')\n" }, { "path": "src/you_get/extractors/imgur.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom ..extractor import VideoExtractor\nfrom .universal import *\n\nclass Imgur(VideoExtractor):\n name = \"Imgur\"\n\n stream_types = [\n {'id': 'original'},\n {'id': 'thumbnail'},\n ]\n\n def prepare(self, **kwargs):\n self.ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/123.0.2420.97'\n\n if re.search(r'imgur\\.com/a/', self.url):\n # album\n content = get_content(self.url, headers=fake_headers)\n album = match1(content, r'album\\s*:\\s*({.*}),') or \\\n match1(content, r'image\\s*:\\s*({.*}),')\n album = json.loads(album)\n count = album['album_images']['count']\n images = album['album_images']['images']\n ext = images[0]['ext']\n self.streams = {\n 'original': {\n 'src': ['http://i.imgur.com/%s%s' % (i['hash'], ext)\n for i in images],\n 'size': sum([i['size'] for i in images]),\n 'container': ext[1:]\n },\n 'thumbnail': {\n 'src': ['http://i.imgur.com/%ss%s' % (i['hash'], '.jpg')\n for i in images],\n 'container': 'jpg'\n }\n }\n self.title = album['title']\n\n elif re.search(r'i\\.imgur\\.com/', self.url):\n # direct image\n _, container, size = url_info(self.url, faker=True)\n self.streams = {\n 'original': {\n 'src': [self.url],\n 'size': size,\n 'container': container\n }\n }\n self.title = r1(r'i\\.imgur\\.com/([^./]*)', self.url)\n\n else:\n # gallery image\n content = get_content(self.url, headers=fake_headers)\n url = match1(content, r'meta property=\"og:video\"[^>]+(https?://i.imgur.com/[^\"?]+)') or \\\n match1(content, r'meta property=\"og:image\"[^>]+(https?://i.imgur.com/[^\"?]+)')\n _, container, size = url_info(url, headers={'User-Agent': fake_headers['User-Agent']})\n self.streams = {\n 'original': {\n 'src': [url],\n 'size': size,\n 'container': container\n }\n }\n self.title = r1(r'i\\.imgur\\.com/([^./]*)', url)\n\n def extract(self, **kwargs):\n if 'stream_id' in kwargs and kwargs['stream_id']:\n i = kwargs['stream_id']\n if 'size' not in self.streams[i]:\n self.streams[i]['size'] = urls_size(self.streams[i]['src'])\n\nsite = Imgur()\ndownload = site.download_by_url\ndownload_playlist = site.download_by_url\n" }, { "path": "src/you_get/extractors/infoq.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom ..extractor import VideoExtractor\n\nimport ssl\n\nclass Infoq(VideoExtractor):\n name = \"InfoQ\"\n\n stream_types = [\n {'id': 'video'},\n {'id': 'audio'},\n {'id': 'slides'}\n ]\n\n def prepare(self, **kwargs):\n content = get_content(self.url)\n self.title = match1(content, r'([^<]+)')\n s = match1(content, r'P\\.s\\s*=\\s*\\'([^\\']+)\\'')\n scp = match1(content, r'InfoQConstants\\.scp\\s*=\\s*\\'([^\\']+)\\'')\n scs = match1(content, r'InfoQConstants\\.scs\\s*=\\s*\\'([^\\']+)\\'')\n sck = match1(content, r'InfoQConstants\\.sck\\s*=\\s*\\'([^\\']+)\\'')\n\n mp3 = match1(content, r'name=\"filename\"\\s*value=\"([^\"]+\\.mp3)\"')\n if mp3: mp3 = 'http://res.infoq.com/downloads/mp3downloads/%s' % mp3\n\n pdf = match1(content, r'name=\"filename\"\\s*value=\"([^\"]+\\.pdf)\"')\n if pdf: pdf = 'http://res.infoq.com/downloads/pdfdownloads/%s' % pdf\n\n # cookie handler\n ssl_context = request.HTTPSHandler(\n context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))\n cookie_handler = request.HTTPCookieProcessor()\n opener = request.build_opener(ssl_context, cookie_handler)\n opener.addheaders = [\n ('Referer', self.url),\n ('Cookie',\n 'CloudFront-Policy=%s;CloudFront-Signature=%s;CloudFront-Key-Pair-Id=%s' % (scp, scs, sck))\n ]\n request.install_opener(opener)\n\n if s: self.streams['video'] = {'url': s }\n if mp3: self.streams['audio'] = { 'url': mp3 }\n if pdf: self.streams['slides'] = { 'url': pdf }\n\n def extract(self, **kwargs):\n for i in self.streams:\n s = self.streams[i]\n _, s['container'], s['size'] = url_info(s['url'])\n s['src'] = [s['url']]\n\nsite = Infoq()\ndownload = site.download_by_url\ndownload_playlist = site.download_by_url\n" }, { "path": "src/you_get/extractors/instagram.py", "content": "#!/usr/bin/env python\n\n__all__ = ['instagram_download']\n\nfrom ..common import *\n\ndef instagram_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n headers = {\n 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.2592.87',\n 'sec-fetch-mode': 'navigate' # important\n }\n\n url = r1(r'([^?]*)', url)\n cont = get_content(url, headers=headers)\n\n vid = r1(r'instagram.com/\\w+/([^/]+)', url)\n description = r1(r'([^<]*)', cont) # with logged-in cookies\n title = \"{} [{}]\".format(description.replace(\"\\n\", \" \"), vid)\n\n appId = r1(r'\"appId\":\"(\\d+)\"', cont)\n media_id = r1(r'\"media_id\":\"(\\d+)\"', cont)\n logging.debug('appId: %s' % appId)\n logging.debug('media_id: %s' % media_id)\n\n api_url = 'https://i.instagram.com/api/v1/media/%s/info/' % media_id\n try:\n api_cont = get_content(api_url, headers={**fake_headers, **{'x-ig-app-id': appId}})\n post = json.loads(api_cont)\n except:\n log.wtf('[Error] Please specify a cookie file.')\n\n for item in post['items']:\n code = item['code']\n carousel_media = item.get('carousel_media') or [item]\n for i, media in enumerate(carousel_media):\n title = '%s [%s]' % (code, i)\n image_url = media['image_versions2']['candidates'][0]['url']\n ext = image_url.split('?')[0].split('.')[-1]\n size = int(get_head(image_url)['Content-Length'])\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(urls=[image_url],\n title=title,\n ext=ext,\n total_size=size,\n output_dir=output_dir)\n\n # download videos (if any)\n if 'video_versions' in media:\n video_url = media['video_versions'][0]['url']\n ext = video_url.split('?')[0].split('.')[-1]\n size = int(get_head(video_url)['Content-Length'])\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(urls=[video_url],\n title=title,\n ext=ext,\n total_size=size,\n output_dir=output_dir)\n\nsite_info = \"Instagram.com\"\ndownload = instagram_download\ndownload_playlist = playlist_not_supported('instagram')\n" }, { "path": "src/you_get/extractors/interest.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom json import loads\n\ndef interest_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n #http://ch.interest.me/zhtv/VOD/View/114789\n #http://program.interest.me/zhtv/sonja/8/Vod/View/15794\n html = get_content(url)\n #get title\n title = match1(html, r' http://www.iqiyi.com/common/flashplayer/20150916/MainPlayer_5_2_28_c3_3_7_4.swf\n use @fffonion 's method in #617.\n Add trace AVM(asasm) code in Iqiyi's encode function where the salt is put into the encode array and reassemble by RABCDasm(or WinRABCDasm),then use Fiddler to response modified file to replace the src file with its AutoResponder function ,set browser Fiddler proxy and play with !debug version! Flash Player ,finially get result in flashlog.txt(its location can be easily found in search engine).\n Code Like (without letters after #comment:),it just do the job : trace(\"{IQIYI_SALT}:\"+salt_array.join(\"\"))\n ```(Position After getTimer)\n findpropstrict QName(PackageNamespace(\"\"), \"trace\")\n pushstring \"{IQIYI_SALT}:\" #comment for you to locate the salt\n getscopeobject 1\n getslot 17 #comment: 17 is the salt slots number defined in code\n pushstring \"\"\n callproperty QName(Namespace(\"http://adobe.com/AS3/2006/builtin\"), \"join\"), 1\n add\n callpropvoid QName(PackageNamespace(\"\"), \"trace\"), 1\n ```\n\n-> http://www.iqiyi.com/common/flashplayer/20150820/MainPlayer_5_2_27_2_c3_3_7_3.swf\n some small changes in Zombie.bite function\n\n'''\n\n'''\ncom.qiyi.player.core.model.def.DefinitonEnum\nbid meaning for quality\n0 none\n1 standard\n2 high\n3 super\n4 suprt-high\n5 fullhd\n10 4k\n96 topspeed\n\n'''\n'''\ndef mix(tvid):\n salt = '4a1caba4b4465345366f28da7c117d20'\n tm = str(randint(2000,4000))\n sc = hashlib.new('md5', bytes(salt + tm + tvid, 'utf-8')).hexdigest()\n return tm, sc, 'eknas'\n\ndef getVRSXORCode(arg1,arg2):\n loc3=arg2 %3\n if loc3 == 1:\n return arg1^121\n if loc3 == 2:\n return arg1^72\n return arg1^103\n\n\ndef getVrsEncodeCode(vlink):\n loc6=0\n loc2=''\n loc3=vlink.split(\"-\")\n loc4=len(loc3)\n # loc5=loc4-1\n for i in range(loc4-1,-1,-1):\n loc6=getVRSXORCode(int(loc3[loc4-i-1],16),i)\n loc2+=chr(loc6)\n return loc2[::-1]\n\ndef getDispathKey(rid):\n tp=\")(*&^flash@#$%a\" #magic from swf\n time=json.loads(get_content(\"http://data.video.qiyi.com/t?tn=\"+str(random())))[\"t\"]\n t=str(int(floor(int(time)/(10*60.0))))\n return hashlib.new(\"md5\",bytes(t+tp+rid,\"utf-8\")).hexdigest()\n'''\ndef getVMS(tvid, vid):\n t = int(time.time() * 1000)\n src = '76f90cbd92f94a2e925d83e8ccd22cb7'\n key = 'd5fb4bd9d50c4be6948c97edd7254b0e'\n sc = hashlib.new('md5', bytes(str(t) + key + vid, 'utf-8')).hexdigest()\n vmsreq= url = 'http://cache.m.iqiyi.com/tmts/{0}/{1}/?t={2}&sc={3}&src={4}'.format(tvid,vid,t,sc,src)\n return json.loads(get_content(vmsreq))\n\nclass Iqiyi(VideoExtractor):\n name = \"爱奇艺 (Iqiyi)\"\n\n stream_types = [\n {'id': '4k', 'container': 'm3u8', 'video_profile': '4k'},\n {'id': 'BD', 'container': 'm3u8', 'video_profile': '1080p'},\n {'id': 'TD', 'container': 'm3u8', 'video_profile': '720p'},\n {'id': 'TD_H265', 'container': 'm3u8', 'video_profile': '720p H265'},\n {'id': 'HD', 'container': 'm3u8', 'video_profile': '540p'},\n {'id': 'HD_H265', 'container': 'm3u8', 'video_profile': '540p H265'},\n {'id': 'SD', 'container': 'm3u8', 'video_profile': '360p'},\n {'id': 'LD', 'container': 'm3u8', 'video_profile': '210p'},\n ]\n '''\n supported_stream_types = [ 'high', 'standard']\n\n\n stream_to_bid = { '4k': 10, 'fullhd' : 5, 'suprt-high' : 4, 'super' : 3, 'high' : 2, 'standard' :1, 'topspeed' :96}\n '''\n ids = ['4k','BD', 'TD', 'HD', 'SD', 'LD']\n vd_2_id = {10: '4k', 19: '4k', 5:'BD', 18: 'BD', 21: 'HD_H265', 2: 'HD', 4: 'TD', 17: 'TD_H265', 96: 'LD', 1: 'SD', 14: 'TD'}\n id_2_profile = {'4k':'4k', 'BD': '1080p','TD': '720p', 'HD': '540p', 'SD': '360p', 'LD': '210p', 'HD_H265': '540p H265', 'TD_H265': '720p H265'}\n\n\n\n def download_playlist_by_url(self, url, **kwargs):\n self.url = url\n\n video_page = get_content(url)\n videos = set(re.findall(r' np\n try:\n if info[\"data\"]['vp'][\"tkl\"]=='' :\n raise ValueError\n except:\n log.e(\"[Error] Do not support for iQIYI VIP video.\")\n exit(-1)\n\n vs = info[\"data\"][\"vp\"][\"tkl\"][0][\"vs\"]\n self.baseurl=info[\"data\"][\"vp\"][\"du\"].split(\"/\")\n\n for stream in self.stream_types:\n for i in vs:\n if self.stream_to_bid[stream['id']] == i['bid']:\n video_links=i[\"fs\"] #now in i[\"flvs\"] not in i[\"fs\"]\n if not i[\"fs\"][0][\"l\"].startswith(\"/\"):\n tmp = getVrsEncodeCode(i[\"fs\"][0][\"l\"])\n if tmp.endswith('mp4'):\n video_links = i[\"flvs\"]\n self.stream_urls[stream['id']] = video_links\n size = 0\n for l in video_links:\n size += l['b']\n self.streams[stream['id']] = {'container': stream['container'], 'video_profile': stream['video_profile'], 'size' : size}\n break\n\n def extract(self, **kwargs):\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Extract the stream\n stream_id = kwargs['stream_id']\n\n if stream_id not in self.streams:\n log.e('[Error] Invalid video format.')\n log.e('Run \\'-i\\' command with no specific video format to view all available formats.')\n exit(2)\n else:\n # Extract stream with the best quality\n stream_id = self.streams_sorted[0]['id']\n\n urls=[]\n for i in self.stream_urls[stream_id]:\n vlink=i[\"l\"]\n if not vlink.startswith(\"/\"):\n #vlink is encode\n vlink=getVrsEncodeCode(vlink)\n key=getDispathKey(vlink.split(\"/\")[-1].split(\".\")[0])\n baseurl = [x for x in self.baseurl]\n baseurl.insert(-1,key)\n url=\"/\".join(baseurl)+vlink+'?su='+self.gen_uid+'&qyid='+uuid4().hex+'&client=&z=&bt=&ct=&tn='+str(randint(10000,20000))\n urls.append(json.loads(get_content(url))[\"l\"])\n #download should be complete in 10 minutes\n #because the url is generated before start downloading\n #and the key may be expired after 10 minutes\n self.streams[stream_id]['src'] = urls\n'''\n\nsite = Iqiyi()\ndownload = site.download_by_url\niqiyi_download_by_vid = site.download_by_vid\ndownload_playlist = site.download_playlist_by_url\n" }, { "path": "src/you_get/extractors/iwara.py", "content": "#!/usr/bin/env python\n__all__ = ['iwara_download']\nfrom ..common import *\nheaders = {\n 'DNT': '1',\n 'Accept-Encoding': 'gzip, deflate, sdch, br',\n 'Accept-Language': 'en-CA,en;q=0.8,en-US;q=0.6,zh-CN;q=0.4,zh;q=0.2',\n 'Upgrade-Insecure-Requests': '1',\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36',\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',\n 'Cache-Control': 'max-age=0',\n 'Connection': 'keep-alive',\n 'Save-Data': 'on',\n 'Cookie':'has_js=1;show_adult=1',\n}\nstream_types = [\n {'id': 'Source', 'container': 'mp4', 'video_profile': '原始'},\n {'id': '540p', 'container': 'mp4', 'video_profile': '540p'},\n {'id': '360p', 'container': 'mp4', 'video_profile': '360P'},\n ]\ndef iwara_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n global headers\n video_hash = match1(url, r'https?://\\w+.iwara.tv/videos/(\\w+)')\n video_url = match1(url, r'(https?://\\w+.iwara.tv)/videos/\\w+')\n html = get_content(url, headers=headers)\n title = r1(r'(.*)', html)\n api_url = video_url + '/api/video/' + video_hash\n content = get_content(api_url, headers=headers)\n data = json.loads(content)\n if len(data)<1 :\n print('Maybe is Private Video?'+'['+title+']')\n return True;\n down_urls = 'https:' + data[0]['uri']\n type, ext, size = url_info(down_urls, headers=headers)\n print_info(site_info, title+data[0]['resolution'], type, size)\n\n if not info_only:\n download_urls([down_urls], title, ext, size, output_dir, merge=merge, headers=headers)\n\ndef download_playlist_by_url( url, **kwargs):\n video_page = get_html(url)\n url_first=match1(url, r\"(http[s]?://[^/]+)\")\n videos = set(re.findall(r'0):\n for video in videos:\n iwara_download(url_first+video, **kwargs)\n else:\n maybe_print('this page not found any videos')\nsite_info = \"Iwara\"\ndownload = iwara_download\ndownload_playlist = download_playlist_by_url\n" }, { "path": "src/you_get/extractors/ixigua.py", "content": "#!/usr/bin/env python\nimport base64\n\nfrom ..common import *\nfrom json import loads\nfrom urllib import request\n\n__all__ = ['ixigua_download', 'ixigua_download_playlist_by_url']\n\nheaders = {\n \"user-agent\": \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 \"\n \"Safari/537.36\",\n}\n\n\ndef ixigua_download(url, output_dir='.', merge=True, info_only=False, stream_id='', **kwargs):\n # example url: https://www.ixigua.com/i6631065141750268420/#mid=63024814422\n headers['cookie'] = \"MONITOR_WEB_ID=7892c49b-296e-4499-8704-e47c1b15123; \" \\\n \"ixigua-a-s=1; ttcid=af99669b6304453480454f1507011d5c234; BD_REF=1; \" \\\n \"__ac_nonce=060d88ff000a75e8d17eb; __ac_signature=_02B4Z6wo100f01kX9ZpgAAIDAKIBBQUIPYT5F2WIAAPG2ad; \" \\\n \"ttwid=1%7CcIsVF_3vqSIk4XErhPB0H2VaTxT0tdsTMRbMjrJOPN8%7C1624806049%7C08ce7dd6f7d20506a41ba0a331ef96a6505d96731e6ad9f6c8c709f53f227ab1; \"\n\n resp = urlopen_with_retry(request.Request(url, headers=headers))\n html = resp.read().decode('utf-8')\n\n _cookies = []\n for c in resp.getheader('Set-Cookie').split(\"httponly,\"):\n _cookies.append(c.strip().split(' ')[0])\n headers['cookie'] += ' '.join(_cookies)\n\n match_txt = match1(html, r\"', home_page)[-1]\n\n client_id = get_content(js_url)\n return re.search(r'client_id:\"(.+?)\"', client_id).group(1)\n\n\ndef get_resource_info(resource_url, client_id):\n cont = get_content(resource_url, decoded=True)\n\n x = re.escape('forEach(function(e){n(e)})}catch(e){}})},')\n x = re.search(r'' + x + r'(.*)\\);', cont)\n\n info = json.loads(x.group(1))[-1]['data'][0]\n\n info = info['tracks'] if info.get('track_count') else [info]\n\n ids = [i['id'] for i in info if i.get('comment_count') is None]\n ids = list(map(str, ids))\n ids_split = ['%2C'.join(ids[i:i+10]) for i in range(0, len(ids), 10)]\n api_url = 'https://api-v2.soundcloud.com/tracks?ids={ids}&client_id={client_id}&%5Bobject%20Object%5D=&app_version=1584348206&app_locale=en'\n\n res = []\n for ids in ids_split:\n uri = api_url.format(ids=ids, client_id=client_id)\n cont = get_content(uri, decoded=True)\n res += json.loads(cont)\n\n res = iter(res)\n info = [next(res) if i.get('comment_count') is None else i for i in info]\n\n return info\n\n\ndef sndcd_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n client_id = get_sndcd_apikey()\n\n r_info = get_resource_info(url, client_id)\n\n for info in r_info:\n title = info['title']\n metadata = info.get('publisher_metadata')\n\n transcodings = info['media']['transcodings']\n sq = [i for i in transcodings if i['quality'] == 'sq']\n hq = [i for i in transcodings if i['quality'] == 'hq']\n # source url\n surl = sq[0] if hq == [] else hq[0]\n surl = surl['url']\n\n uri = surl + '?client_id=' + client_id\n r = get_content(uri)\n surl = json.loads(r)['url']\n\n m3u8 = get_content(surl)\n # url list\n urll = re.findall(r'http.*?(?=\\n)', m3u8)\n\n size = urls_size(urll)\n print_info(site_info, title, 'audio/mpeg', size)\n print(end='', flush=True)\n\n if not info_only:\n download_urls(urll, title=title, ext='mp3', total_size=size, output_dir=output_dir, merge=True)\n\n\nsite_info = \"SoundCloud.com\"\ndownload = sndcd_download\ndownload_playlist = sndcd_download\n" }, { "path": "src/you_get/extractors/suntv.py", "content": "#!/usr/bin/env python\n\n__all__ = ['suntv_download']\n\nfrom ..common import *\nimport urllib\nimport re\n\ndef suntv_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n if re.match(r'http://www.isuntv.com/\\w+', url):\n API_URL = \"http://www.isuntv.com/ajaxpro/SunTv.pro_vod_playcatemp4,App_Web_playcatemp4.ascx.9f08f04f.ashx\"\n \n itemid = match1(url, r'http://www.isuntv.com/pro/ct(\\d+).html')\n values = {\"itemid\" : itemid, \"vodid\": \"\"}\n \n data = str(values).replace(\"'\", '\"')\n data = data.encode('utf-8')\n req = urllib.request.Request(API_URL, data)\n req.add_header('AjaxPro-Method', 'ToPlay') #important!\n resp = urllib.request.urlopen(req)\n respData = resp.read()\n respData = respData.decode('ascii').strip('\"') #Ahhhhhhh!\n \n video_url = 'http://www.isuntv.com' + str(respData)\n \n html = get_content(url, decoded=False)\n html = html.decode('gbk')\n title = match1(html, '([^<]+)').strip() #get rid of \\r\\n s\n \n size = 0\n type, ext, size = url_info(video_url)\n \n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([url], title, 'mp4', size, output_dir, merge=merge)\n\nsite_info = \"SunTV\"\ndownload = suntv_download\ndownload_playlist = playlist_not_supported('suntv')\n" }, { "path": "src/you_get/extractors/ted.py", "content": "#!/usr/bin/env python\n\n__all__ = ['ted_download']\n\nfrom ..common import *\nimport json\n\ndef ted_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url)\n patt = r'\"__INITIAL_DATA__\"\\s*:\\s*\\{(.+)\\}'\n metadata = json.loads('{' + match1(html, patt) + '}')\n title = metadata['talks'][0]['title']\n nativeDownloads = metadata['talks'][0]['downloads']['nativeDownloads']\n for quality in ['high', 'medium', 'low']:\n if quality in nativeDownloads:\n url = nativeDownloads[quality]\n type, ext, size = url_info(url)\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir, merge=merge)\n break\n\nsite_info = \"TED.com\"\ndownload = ted_download\ndownload_playlist = playlist_not_supported('ted')\n" }, { "path": "src/you_get/extractors/theplatform.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\n\ndef theplatform_download_by_pid(pid, title, output_dir='.', merge=True, info_only=False, **kwargs):\n smil_url = \"http://link.theplatform.com/s/dJ5BDC/%s/meta.smil?format=smil&mbr=true\" % pid\n smil = get_content(smil_url)\n smil_base = unescape_html(match1(smil, r'<meta base=\"([^\"]+)\"'))\n smil_videos = {y:x for x,y in dict(re.findall(r'<video src=\"([^\"]+)\".+height=\"([^\"]+)\"', smil)).items()}\n for height in ['1080', '720', '480', '360', '240', '216']:\n if height in smil_videos:\n smil_video = smil_videos[height]\n break\n assert smil_video\n\n type, ext, size = 'mp4', 'mp4', 0\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_rtmp_url(url=smil_base, title=title, ext=ext,params={\"-y\":ext+':'+smil_video}, output_dir=output_dir)\n\nsite_info = \"thePlatform.com\"\ndownload = theplatform_download_by_pid\ndownload_playlist = playlist_not_supported('theplatform')\n" }, { "path": "src/you_get/extractors/tiktok.py", "content": "#!/usr/bin/env python\n\n__all__ = ['tiktok_download']\n\nfrom ..common import *\n\ndef tiktok_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n headers = {\n 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0',\n 'Accept-Encoding': 'gzip, deflate',\n 'Accept': '*/*',\n 'Referer': 'https://www.tiktok.com/',\n 'Connection': 'keep-alive' # important\n }\n\n m = re.match('(https?://)?([^/]+)(/.*)', url)\n host = m.group(2)\n if host != 'www.tiktok.com': # non-canonical URL\n if host == 'vt.tiktok.com': # short URL\n url = get_location(url)\n vid = r1(r'/video/(\\d+)', url)\n url = 'https://www.tiktok.com/@/video/%s/' % vid\n host = 'www.tiktok.com'\n else:\n url = m.group(3).split('?')[0]\n vid = url.split('/')[3] # should be a string of numbers\n\n html, set_cookie = getHttps(host, url, headers=headers)\n tt_chain_token = r1('tt_chain_token=([^;]+);', set_cookie)\n headers['Cookie'] = 'tt_chain_token=%s' % tt_chain_token\n\n data = r1(r'<script id=\"__UNIVERSAL_DATA_FOR_REHYDRATION__\" type=\"application/json\">(.*?)</script>', html)\n info = json.loads(data)\n itemStruct = info['__DEFAULT_SCOPE__']['webapp.video-detail']['itemInfo']['itemStruct']\n downloadAddr = itemStruct['video']['downloadAddr']\n author = itemStruct['author']['uniqueId']\n nickname = itemStruct['author']['nickname']\n title = '%s [%s]' % (nickname or author, vid)\n\n mime, ext, size = url_info(downloadAddr, headers=headers)\n\n print_info(site_info, title, mime, size)\n if not info_only:\n download_urls([downloadAddr], title, ext, size, output_dir=output_dir, merge=merge, headers=headers)\n\nsite_info = \"TikTok.com\"\ndownload = tiktok_download\ndownload_playlist = playlist_not_supported('tiktok')\n" }, { "path": "src/you_get/extractors/toutiao.py", "content": "#!/usr/bin/env python\nimport binascii\nimport random\nfrom json import loads\nfrom urllib.parse import urlparse\n\nfrom ..common import *\n\ntry:\n from base64 import decodebytes\nexcept ImportError:\n from base64 import decodestring\n\n decodebytes = decodestring\n\n__all__ = ['toutiao_download', ]\n\n\ndef random_with_n_digits(n):\n return random.randint(10 ** (n - 1), (10 ** n) - 1)\n\n\ndef sign_video_url(vid):\n r = str(random_with_n_digits(16))\n\n url = 'https://ib.365yg.com/video/urls/v/1/toutiao/mp4/{vid}'.format(vid=vid)\n n = urlparse(url).path + '?r=' + r\n b_n = bytes(n, encoding=\"utf-8\")\n s = binascii.crc32(b_n)\n aid = 1364\n ts = int(time.time() * 1000)\n return url + '?r={r}&s={s}&aid={aid}&vfrom=xgplayer&callback=axiosJsonpCallback1&_={ts}'.format(r=r, s=s, aid=aid,\n ts=ts)\n\n\nclass ToutiaoVideoInfo(object):\n\n def __init__(self):\n self.bitrate = None\n self.definition = None\n self.size = None\n self.height = None\n self.width = None\n self.type = None\n self.url = None\n\n def __str__(self):\n return json.dumps(self.__dict__)\n\n\ndef get_file_by_vid(video_id):\n vRet = []\n url = sign_video_url(video_id)\n ret = get_content(url)\n ret = loads(ret[20:-1])\n vlist = ret.get('data').get('video_list')\n if len(vlist) > 0:\n vInfo = vlist.get(sorted(vlist.keys(), reverse=True)[0])\n vUrl = vInfo.get('main_url')\n vUrl = decodebytes(vUrl.encode('ascii')).decode('ascii')\n videoInfo = ToutiaoVideoInfo()\n videoInfo.bitrate = vInfo.get('bitrate')\n videoInfo.definition = vInfo.get('definition')\n videoInfo.size = vInfo.get('size')\n videoInfo.height = vInfo.get('vheight')\n videoInfo.width = vInfo.get('vwidth')\n videoInfo.type = vInfo.get('vtype')\n videoInfo.url = vUrl\n vRet.append(videoInfo)\n return vRet\n\n\ndef toutiao_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n html = get_html(url, faker=True)\n video_id = match1(html, r\".*?videoId: '(?P<vid>.*)'\")\n title = match1(html, '.*?<title>(?P<title>.*?)')\n video_file_list = get_file_by_vid(video_id) # 调api获取视频源文件\n type, ext, size = url_info(video_file_list[0].url, faker=True)\n print_info(site_info=site_info, title=title, type=type, size=size)\n if not info_only:\n download_urls([video_file_list[0].url], title, ext, size, output_dir, merge=merge, faker=True)\n\n\nsite_info = \"Toutiao.com\"\ndownload = toutiao_download\ndownload_playlist = playlist_not_supported(\"toutiao\")\n" }, { "path": "src/you_get/extractors/tucao.py", "content": "#!/usr/bin/env python\n\n__all__ = ['tucao_download']\nfrom ..common import *\n# import re\nimport random\nimport time\nfrom xml.dom import minidom\n#possible raw list types\n#1.
  • type=tudou&vid=199687639
    • \n#2.
  • type=tudou&vid=199506910|
    • \n#3.
  • type=video&file=http://xiaoshen140731.qiniudn.com/lovestage04.flv|
    • \n#4 may ?
  • type=video&file=http://xiaoshen140731.qiniudn.com/lovestage04.flv|xx**type=&vid=?
    • \n#5.
  • type=tudou&vid=200003098|07**type=tudou&vid=200000350|08
    • \n#6.
  • vid=49454694&type=sina|
    • \n#7.
  • type=189&vid=513031813243909|
    • \n# re_pattern=re.compile(r\"(type=(.+?)&(vid|file)=(.*?))[\\|<]\")\n\ndef tucao_single_download(type_link, title, output_dir=\".\", merge=True, info_only=False):\n if \"file\" in type_link:\n url=type_link[type_link.find(\"file=\")+5:]\n vtype, ext, size=url_info(url)\n print_info(site_info, title, vtype, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir)\n #fix for 189 video source, see raw list types 7\n elif \"189\" in type_link:\n vid = match1(type_link, r\"vid=(\\d+)\")\n assert vid, \"vid not exsits\"\n url = \"http://api.tucao.tv/api/down/{}\".format(vid)\n vtype, ext, size=url_info(url)\n print_info(site_info, title, vtype, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir)\n else:\n u=\"http://www.tucao.tv/api/playurl.php?{}&key=tucao{:07x}.cc&r={}\".format(type_link,random.getrandbits(28),int(time.time()*1000))\n xml=minidom.parseString(get_content(u))\n urls=[]\n size=0\n for i in xml.getElementsByTagName(\"url\"):\n urls.append(i.firstChild.nodeValue)\n vtype, ext, _size=url_info(i.firstChild.nodeValue)\n size+=_size\n print_info(site_info, title, vtype, size)\n if not info_only:\n download_urls(urls, title, ext, size, output_dir)\n\ndef tucao_download(url, output_dir=\".\", merge=True, info_only=False, **kwargs):\n html=get_content(url)\n title=match1(html,r'

    (.*?)<\\w')\n #fix for raw list that vid goes before type, see raw list types 6\n raw_list=match1(html,r\"
  • \\s*(type=.+?|vid=.+?)
    • \")\n raw_l=raw_list.split(\"**\")\n if len(raw_l)==1:\n format_link=raw_l[0][:-1] if raw_l[0].endswith(\"|\") else raw_l[0]\n tucao_single_download(format_link,title,output_dir,merge,info_only)\n else:\n for i in raw_l:\n format_link,sub_title=i.split(\"|\")\n tucao_single_download(format_link,title+\"-\"+sub_title,output_dir,merge,info_only)\n\n\nsite_info = \"tucao.tv\"\ndownload = tucao_download\ndownload_playlist = playlist_not_supported(\"tucao\")\n" }, { "path": "src/you_get/extractors/tudou.py", "content": "#!/usr/bin/env python\n\n__all__ = ['tudou_download', 'tudou_download_playlist', 'tudou_download_by_id', 'tudou_download_by_iid']\n\nfrom ..common import *\nfrom xml.dom.minidom import parseString\nimport you_get.extractors.acfun\n\ndef tudou_download_by_iid(iid, title, output_dir = '.', merge = True, info_only = False):\n data = json.loads(get_decoded_html('http://www.tudou.com/outplay/goto/getItemSegs.action?iid=%s' % iid))\n temp = max([data[i] for i in data if 'size' in data[i][0]], key=lambda x:sum([part['size'] for part in x]))\n vids, size = [t[\"k\"] for t in temp], sum([t[\"size\"] for t in temp])\n\n urls = []\n for vid in vids:\n for i in parseString(get_html('http://ct.v2.tudou.com/f?id=%s' % vid)).getElementsByTagName('f'):\n urls.append(i.firstChild.nodeValue.strip())\n\n ext = r1(r'http://[\\w.]*/(\\w+)/[\\w.]*', urls[0])\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(urls, title, ext, size, output_dir=output_dir, merge = merge)\n\ndef tudou_download_by_id(id, title, output_dir = '.', merge = True, info_only = False):\n html = get_html('http://www.tudou.com/programs/view/%s/' % id)\n\n iid = r1(r'iid\\s*[:=]\\s*(\\S+)', html)\n try:\n title = r1(r'kw\\s*[:=]\\s*[\\'\\\"]([^\\n]+?)\\'\\s*\\n', html).replace(\"\\\\'\", \"\\'\")\n except AttributeError:\n title = ''\n tudou_download_by_iid(iid, title, output_dir = output_dir, merge = merge, info_only = info_only)\n\ndef tudou_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n if 'acfun.tudou.com' in url: #wrong way!\n url = url.replace('acfun.tudou.com', 'www.acfun.tv')\n you_get.extractors.acfun.acfun_download(url, output_dir,\n merge,\n info_only)\n return #throw you back\n\n # Embedded player\n id = r1(r'http://www.tudou.com/v/([^/]+)/', url)\n if id:\n return tudou_download_by_id(id, title=\"\", info_only=info_only)\n\n html = get_content(url)\n\n try:\n title = r1(r'\\Wkw\\s*[:=]\\s*[\\'\\\"]([^\\n]+?)\\'\\s*\\n', html).replace(\"\\\\'\", \"\\'\")\n assert title\n title = unescape_html(title)\n except AttributeError:\n title = match1(html, r'id=\\\"subtitle\\\"\\s*title\\s*=\\s*\\\"([^\\\"]+)\\\"')\n if title is None:\n title = ''\n\n vcode = r1(r'vcode\\s*[:=]\\s*\\'([^\\']+)\\'', html)\n if vcode is None:\n vcode = match1(html, r'viden\\s*[:=]\\s*\\\"([\\w+/=]+)\\\"')\n if vcode:\n from .youku import youku_download_by_vid\n return youku_download_by_vid(vcode, title=title, output_dir=output_dir, merge=merge, info_only=info_only, src='tudou', **kwargs)\n\n iid = r1(r'iid\\s*[:=]\\s*(\\d+)', html)\n if not iid:\n return tudou_download_playlist(url, output_dir, merge, info_only)\n\n tudou_download_by_iid(iid, title, output_dir = output_dir, merge = merge, info_only = info_only)\n\n# obsolete?\ndef parse_playlist(url):\n aid = r1(r'http://www.tudou.com/playlist/p/a(\\d+)(?:i\\d+)?\\.html', url)\n html = get_decoded_html(url)\n if not aid:\n aid = r1(r\"aid\\s*[:=]\\s*'(\\d+)'\", html)\n if re.match(r'http://www.tudou.com/albumcover/', url):\n atitle = r1(r\"title\\s*:\\s*'([^']+)'\", html)\n elif re.match(r'http://www.tudou.com/playlist/p/', url):\n atitle = r1(r'atitle\\s*=\\s*\"([^\"]+)\"', html)\n else:\n raise NotImplementedError(url)\n assert aid\n assert atitle\n import json\n #url = 'http://www.tudou.com/playlist/service/getZyAlbumItems.html?aid='+aid\n url = 'http://www.tudou.com/playlist/service/getAlbumItems.html?aid='+aid\n return [(atitle + '-' + x['title'], str(x['itemId'])) for x in json.loads(get_html(url))['message']]\n\ndef parse_plist(url):\n html = get_decoded_html(url)\n lcode = r1(r\"lcode:\\s*'([^']+)'\", html)\n plist_info = json.loads(get_content('http://www.tudou.com/crp/plist.action?lcode=' + lcode))\n return ([(item['kw'], item['iid']) for item in plist_info['items']])\n\ndef tudou_download_playlist(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n videos = parse_plist(url)\n for i, (title, id) in enumerate(videos):\n print('Processing %s of %s videos...' % (i + 1, len(videos)))\n tudou_download_by_iid(id, title, output_dir = output_dir, merge = merge, info_only = info_only)\n\nsite_info = \"Tudou.com\"\ndownload = tudou_download\ndownload_playlist = tudou_download_playlist\n" }, { "path": "src/you_get/extractors/tumblr.py", "content": "#!/usr/bin/env python\n\n__all__ = ['tumblr_download']\n\nfrom ..common import *\nfrom .universal import *\nfrom .dailymotion import dailymotion_download\nfrom .vimeo import vimeo_download\n\ndef tumblr_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n if re.match(r'https?://\\d+\\.media\\.tumblr\\.com/', url):\n universal_download(url, output_dir, merge=merge, info_only=info_only)\n return\n\n import ssl\n ssl_context = request.HTTPSHandler(context=ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)) # server requires TLS v1.2\n cookie_handler = request.HTTPCookieProcessor()\n opener = request.build_opener(ssl_context, cookie_handler)\n request.install_opener(opener)\n\n page = get_html(url)\n form_key = match1(page, r'id=\"tumblr_form_key\" content=\"([^\"]+)\"')\n if form_key is not None:\n # bypass GDPR consent page\n referer = 'https://www.tumblr.com/privacy/consent?redirect=%s' % parse.quote_plus(url)\n post_content('https://www.tumblr.com/svc/privacy/consent',\n headers={\n 'Content-Type': 'application/json',\n 'User-Agent': fake_headers['User-Agent'],\n 'Referer': referer,\n 'X-tumblr-form-key': form_key,\n 'X-Requested-With': 'XMLHttpRequest'\n },\n post_data_raw='{\"eu_resident\":true,\"gdpr_is_acceptable_age\":true,\"gdpr_consent_core\":true,\"gdpr_consent_first_party_ads\":true,\"gdpr_consent_third_party_ads\":true,\"gdpr_consent_search_history\":true,\"redirect_to\":\"%s\",\"gdpr_reconsent\":false}' % url)\n page = get_html(url, faker=True)\n\n html = parse.unquote(page).replace(r'\\/', '/')\n feed = r1(r'', html)\n\n if feed in ['photo', 'photoset', 'entry'] or feed is None:\n # try to extract photos\n page_title = r1(r'([^<\\n]*)', html)\n urls = re.findall(r'(https?://[^;\"&]+/tumblr_[^;\"&]+_\\d+\\.jpg)', html) +\\\n re.findall(r'(https?://[^;\"&]+/tumblr_[^;\"&]+_\\d+\\.png)', html) +\\\n re.findall(r'(https?://[^;\"&]+/tumblr_[^;\"&]+_\\d+\\.gif)', html) +\\\n re.findall(r'(https?://\\d+\\.media\\.tumblr\\.com/[^;\"&]+/s\\d+x\\d+/[^;\"&]+\\.jpg)', html) +\\\n re.findall(r'(https?://\\d+\\.media\\.tumblr\\.com/[^;\"&]+/s\\d+x\\d+/[^;\"&]+\\.png)', html) +\\\n re.findall(r'(https?://\\d+\\.media\\.tumblr\\.com/[^;\"&]+/s\\d+x\\d+/[^;\"&]+\\.gif)', html)\n\n tuggles = {}\n for url in urls:\n if url.endswith('.gif'):\n hd_url = url\n elif url.endswith('.jpg'):\n hd_url = url # FIXME: decide actual quality # r1(r'(.+)_\\d+\\.jpg$', url) + '_1280.jpg'\n elif url.endswith('.png'):\n hd_url = url # FIXME: decide actual quality # r1(r'(.+)_\\d+\\.png$', url) + '_1280.png'\n else:\n continue\n filename = parse.unquote(hd_url.split('/')[-1])\n title = '.'.join(filename.split('.')[:-1])\n tumblr_id = r1(r'^tumblr_(.+)_\\d+$', title) or title\n try:\n quality = int(r1(r'^tumblr_.+_(\\d+)$', title))\n except:\n quality = int(r1(r'/s(\\d+)x\\d+/', hd_url))\n ext = filename.split('.')[-1]\n\n try:\n size = int(get_head(hd_url)['Content-Length'])\n if tumblr_id not in tuggles or tuggles[tumblr_id]['quality'] < quality:\n tuggles[tumblr_id] = {\n 'title': title,\n 'url': hd_url,\n 'quality': quality,\n 'ext': ext,\n 'size': size,\n }\n except: pass\n\n if tuggles:\n #size = sum([tuggles[t]['size'] for t in tuggles])\n #print_info(site_info, page_title, None, size)\n\n for t in tuggles:\n title = '[tumblr] ' + tuggles[t]['title']\n ext = tuggles[t]['ext']\n size = tuggles[t]['size']\n url = tuggles[t]['url']\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls([url], title, ext, size,\n output_dir=output_dir)\n return\n\n # feed == 'audio' or feed == 'video' or feed is None\n # try to extract video / audio\n real_url = r1(r'source src=\\\\x22([^\\\\]+)\\\\', html)\n if not real_url:\n real_url = r1(r'audio_file=([^&]+)&', html)\n if real_url:\n real_url = real_url + '?plead=please-dont-download-this-or-our-lawyers-wont-let-us-host-audio'\n if not real_url:\n real_url = r1(r']+tumblr_video_container[^>]+>]+src=[\\'\"]([^\\'\"]*)[\\'\"]', html)\n\n if iframe_url is None:\n universal_download(url, output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n\n if iframe_url:\n iframe_html = get_content(iframe_url, headers=fake_headers)\n real_url = r1(r']*>[\\n ]*]+src=[\\'\"]([^\\'\"]*)[\\'\"]', iframe_html)\n else:\n iframe_url = r1(r']+src=[\\'\"]([^\\'\"]*)[\\'\"]', html)\n if iframe_url[:2] == '//': iframe_url = 'http:' + iframe_url\n if re.search(r'player\\.vimeo\\.com', iframe_url):\n vimeo_download(iframe_url, output_dir, merge=merge, info_only=info_only,\n referer='http://tumblr.com/', **kwargs)\n return\n elif re.search(r'dailymotion\\.com', iframe_url):\n dailymotion_download(iframe_url, output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n else:\n iframe_html = get_content(iframe_url)\n real_url = r1(r'', html) or\n r1(r'', html) or\n r1(r'([^<\\n]*)', html) or url.split(\"/\")[4]).replace('\\n', '')\n\n # this is better\n vcode = r1(r'tumblr_(\\w+)', real_url)\n real_url = 'https://vt.media.tumblr.com/tumblr_%s.mp4' % vcode\n\n type, ext, size = url_info(real_url, faker=True)\n\n print_info(site_info, title, type, size)\n if not info_only:\n download_urls([real_url], title, ext, size, output_dir, merge=merge)\n\nsite_info = \"Tumblr.com\"\ndownload = tumblr_download\ndownload_playlist = playlist_not_supported('tumblr')\n" }, { "path": "src/you_get/extractors/twitter.py", "content": "#!/usr/bin/env python\n\n__all__ = ['twitter_download']\n\nfrom ..common import *\nfrom .universal import *\n\ndef extract_m3u(source):\n r1 = get_content(source)\n s1 = re.findall(r'(/ext_tw_video/.*)', r1)\n s1 += re.findall(r'(/amplify_video/.*)', r1)\n r2 = get_content('https://video.twimg.com%s' % s1[-1])\n s2 = re.findall(r'(/ext_tw_video/.*)', r2)\n s2 += re.findall(r'(/amplify_video/.*)', r2)\n return ['https://video.twimg.com%s' % i for i in s2]\n\ndef twitter_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n headers = {\n 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0',\n 'Accept-Encoding': 'gzip, deflate',\n 'Accept': '*/*'\n }\n\n if re.match(r'https?://pbs\\.twimg\\.com', url):\n universal_download(url, output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n\n if re.match(r'https?://mobile', url): # normalize mobile URL\n url = 'https://' + match1(url, r'//mobile\\.(.+)')\n\n if re.match(r'https?://twitter\\.com/i/moments/', url): # FIXME: moments\n html = get_html(url, faker=True)\n paths = re.findall(r'data-permalink-path=\"([^\"]+)\"', html)\n for path in paths:\n twitter_download('https://twitter.com' + path,\n output_dir=output_dir,\n merge=merge,\n info_only=info_only,\n **kwargs)\n return\n\n m = re.match(r'^https?://(mobile\\.)?(x|twitter)\\.com/([^/]+)/status/(\\d+)', url)\n assert m\n screen_name, item_id = m.group(3), m.group(4)\n page_title = \"{} [{}]\".format(screen_name, item_id)\n\n # FIXME: this API won't work for protected or nsfw contents\n api_url = 'https://cdn.syndication.twimg.com/tweet-result?id=%s&token=!' % item_id\n content = get_content(api_url)\n info = json.loads(content)\n\n author = info['user']['name']\n url = 'https://twitter.com/%s/status/%s' % (info['user']['screen_name'], item_id)\n full_text = info['text']\n\n if 'photos' in info:\n for photo in info['photos']:\n photo_url = photo['url']\n title = item_id + '_' + photo_url.split('.')[-2].split('/')[-1]\n urls = [ photo_url + ':orig' ]\n size = urls_size(urls, headers=headers)\n ext = photo_url.split('.')[-1]\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(urls, title, ext, size, output_dir, merge=merge)\n\n if 'video' in info:\n for mediaDetail in info['mediaDetails']:\n if 'video_info' not in mediaDetail: continue\n variants = mediaDetail['video_info']['variants']\n variants = sorted(variants, key=lambda kv: kv.get('bitrate', 0))\n title = item_id + '_' + variants[-1]['url'].split('/')[-1].split('?')[0].split('.')[0]\n urls = [ variants[-1]['url'] ]\n size = urls_size(urls, headers=headers)\n mime, ext = variants[-1]['content_type'], 'mp4'\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls(urls, title, ext, size, output_dir, merge=merge, headers=headers)\n\n # TODO: should we deal with quoted tweets?\n\n\nsite_info = \"X.com\"\ndownload = twitter_download\ndownload_playlist = playlist_not_supported('twitter')\n" }, { "path": "src/you_get/extractors/ucas.py", "content": "#!/usr/bin/env python\n\n__all__ = ['ucas_download', 'ucas_download_single', 'ucas_download_playlist']\n\nfrom ..common import *\nimport http.client\nfrom time import time\nfrom random import random\nimport xml.etree.ElementTree as ET\nfrom copy import copy\n\n\"\"\"\nDo not replace http.client with get_content\nfor UCAS's server is not correctly returning data!\n\"\"\"\n\ndef dictify(r,root=True):\n \"\"\"http://stackoverflow.com/a/30923963/2946714\"\"\"\n if root:\n return {r.tag : dictify(r, False)}\n d=copy(r.attrib)\n if r.text:\n d[\"_text\"]=r.text\n for x in r.findall(\"./*\"):\n if x.tag not in d:\n d[x.tag]=[]\n d[x.tag].append(dictify(x,False))\n return d\n\ndef _get_video_query_url(resourceID):\n # has to be like this\n headers = {\n 'DNT': '1',\n 'Accept-Encoding': 'gzip, deflate',\n 'Accept-Language': 'en-CA,en;q=0.8,en-US;q=0.6,zh-CN;q=0.4,zh;q=0.2',\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.47 Safari/537.36',\n 'Accept': '*/*',\n 'Referer': 'http://v.ucas.ac.cn/',\n 'Connection': 'keep-alive',\n }\n conn = http.client.HTTPConnection(\"210.76.211.10\")\n \n conn.request(\"GET\", \"/vplus/remote.do?method=query2&loginname=videocas&pwd=af1c7a4c5f77f790722f7cae474c37e281203765d423a23b&resource=%5B%7B%22resourceID%22%3A%22\" + resourceID + \"%22%2C%22on%22%3A1%2C%22time%22%3A600%2C%22eid%22%3A100%2C%22w%22%3A800%2C%22h%22%3A600%7D%5D&timeStamp=\" + str(int(time())), headers=headers)\n res = conn.getresponse()\n data = res.read()\n\n info = data.decode(\"utf-8\")\n return match1(info, r'video\":\"(.+)\"')\n\ndef _get_virtualPath(video_query_url):\n #getResourceJsCode2\n html = get_content(video_query_url)\n \n return match1(html, r\"function\\s+getVirtualPath\\(\\)\\s+{\\s+return\\s+'(\\w+)'\")\n\n\ndef _get_video_list(resourceID):\n \"\"\"\"\"\"\n conn = http.client.HTTPConnection(\"210.76.211.10\")\n \n conn.request(\"GET\", '/vplus/member/resource.do?isyulan=0&method=queryFlashXmlByResourceId&resourceId={resourceID}&randoms={randoms}'.format(resourceID = resourceID,\n randoms = random()))\n res = conn.getresponse()\n data = res.read()\n\n video_xml = data.decode(\"utf-8\")\n\n root = ET.fromstring(video_xml.split('___!!!___')[0])\n\n r = dictify(root)\n\n huge_list = []\n # main\n huge_list.append([i['value'] for i in sorted(r['video']['mainUrl'][0]['_flv'][0]['part'][0]['video'], key=lambda k: int(k['index']))])\n\n # sub\n if '_flv' in r['video']['subUrl'][0]:\n huge_list.append([i['value'] for i in sorted(r['video']['subUrl'][0]['_flv'][0]['part'][0]['video'], key=lambda k: int(k['index']))])\n\n return huge_list\n\ndef _ucas_get_url_lists_by_resourceID(resourceID):\n video_query_url = _get_video_query_url(resourceID)\n assert video_query_url != '', 'Cannot find video GUID!'\n \n virtualPath = _get_virtualPath(video_query_url)\n assert virtualPath != '', 'Cannot find virtualPath!'\n \n url_lists = _get_video_list(resourceID)\n assert url_lists, 'Cannot find any URL to download!'\n\n # make real url\n # credit to a mate in UCAS\n for video_type_id, video_urls in enumerate(url_lists):\n for k, path in enumerate(video_urls):\n url_lists[video_type_id][k] = 'http://210.76.211.10/vplus/member/resource.do?virtualPath={virtualPath}&method=getImgByStream&imgPath={path}'.format(virtualPath = virtualPath,\n path = path)\n\n return url_lists\n\ndef ucas_download_single(url, output_dir = '.', merge = False, info_only = False, **kwargs):\n '''video page'''\n html = get_content(url)\n # resourceID is UUID\n resourceID = re.findall( r'resourceID\":\"([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})', html)[0]\n assert resourceID != '', 'Cannot find resourceID!'\n\n title = match1(html, r'<div class=\"bc-h\">(.+)</div>')\n url_lists = _ucas_get_url_lists_by_resourceID(resourceID)\n assert url_lists, 'Cannot find any URL of such class!'\n \n for k, part in enumerate(url_lists):\n part_title = title + '_' + str(k)\n print_info(site_info, part_title, 'flv', 0)\n if not info_only:\n download_urls(part, part_title, 'flv', total_size=None, output_dir=output_dir, merge=merge)\n\ndef ucas_download_playlist(url, output_dir = '.', merge = False, info_only = False, **kwargs):\n '''course page'''\n html = get_content(url)\n\n parts = re.findall( r'(getplaytitle.do\\?.+)\"', html)\n assert parts, 'No part found!'\n\n for part_path in parts:\n ucas_download('http://v.ucas.ac.cn/course/' + part_path, output_dir=output_dir, merge=merge, info_only=info_only)\n\ndef ucas_download(url, output_dir = '.', merge = False, info_only = False, **kwargs):\n if 'classid=' in url and 'getplaytitle.do' in url:\n ucas_download_single(url, output_dir=output_dir, merge=merge, info_only=info_only)\n elif 'CourseIndex.do' in url:\n ucas_download_playlist(url, output_dir=output_dir, merge=merge, info_only=info_only)\n\nsite_info = \"UCAS\"\ndownload = ucas_download\ndownload_playlist = ucas_download_playlist" }, { "path": "src/you_get/extractors/universal.py", "content": "#!/usr/bin/env python\n\n__all__ = ['universal_download']\n\nfrom ..common import *\nfrom .embed import *\n\ndef universal_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n try:\n content_type = get_head(url, headers=fake_headers)['Content-Type']\n except:\n content_type = get_head(url, headers=fake_headers, get_method='GET')['Content-Type']\n if content_type.startswith('text/html'):\n try:\n embed_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n except Exception:\n pass\n else:\n return\n\n domains = url.split('/')[2].split('.')\n if len(domains) > 2: domains = domains[1:]\n site_info = '.'.join(domains)\n\n if content_type.startswith('text/html'):\n # extract an HTML page\n response = get_response(url, faker=True)\n page = str(response.data)\n\n page_title = r1(r'<title>([^<]*)', page)\n if page_title:\n page_title = unescape_html(page_title)\n\n meta_videos = re.findall(r'<meta property=\"og:video:url\" content=\"([^\"]*)\"', page)\n if meta_videos:\n try:\n for meta_video in meta_videos:\n meta_video_url = unescape_html(meta_video)\n type_, ext, size = url_info(meta_video_url)\n print_info(site_info, page_title, type_, size)\n if not info_only:\n download_urls([meta_video_url], page_title,\n ext, size,\n output_dir=output_dir, merge=merge,\n faker=True)\n except:\n pass\n else:\n return\n\n hls_urls = re.findall(r'(https?://[^;\"\\'\\\\]+' + r'\\.m3u8?' +\n r'[^;\"\\'\\\\]*)', page)\n if hls_urls:\n try:\n for hls_url in hls_urls:\n type_, ext, size = url_info(hls_url)\n print_info(site_info, page_title, type_, size)\n if not info_only:\n download_url_ffmpeg(url=hls_url, title=page_title,\n ext='mp4', output_dir=output_dir)\n except:\n pass\n else:\n return\n\n # most common media file extensions on the Internet\n media_exts = [r'\\.flv', r'\\.mp3', r'\\.mp4', r'\\.webm',\n r'[-_]1\\d\\d\\d\\.jpe?g', r'[-_][6-9]\\d\\d\\.jpe?g', # tumblr\n r'[-_]1\\d\\d\\dx[6-9]\\d\\d\\.jpe?g',\n r'[-_][6-9]\\d\\dx1\\d\\d\\d\\.jpe?g',\n r'[-_][6-9]\\d\\dx[6-9]\\d\\d\\.jpe?g',\n r's1600/[\\w%]+\\.jpe?g', # blogger\n r'blogger\\.googleusercontent\\.com/img/a/\\w*', # blogger\n r'img[6-9]\\d\\d/[\\w%]+\\.jpe?g' # oricon?\n ]\n\n urls = []\n for i in media_exts:\n urls += re.findall(r'(https?://[^ ;&\"\\'\\\\<>]*' + i + r'[^ =?;&\"\\'\\\\<>]*)', page)\n\n p_urls = re.findall(r'(https?%3A%2F%2F[^;&\"]+' + i + r'[^;&\"]*)', page)\n urls += [parse.unquote(url) for url in p_urls]\n\n q_urls = re.findall(r'(https?:\\\\\\\\/\\\\\\\\/[^ ;\"\\'<>]+' + i + r'[^ ;\"\\'<>]*)', page)\n urls += [url.replace('\\\\\\\\/', '/') for url in q_urls]\n\n # a link href to an image is often an interesting one\n urls += re.findall(r'href=\"(https?://[^\"]+\\.jpe?g)\"', page, re.I)\n urls += re.findall(r'href=\"(https?://[^\"]+\\.png)\"', page, re.I)\n urls += re.findall(r'href=\"(https?://[^\"]+\\.gif)\"', page, re.I)\n\n # <img> with high widths\n urls += re.findall(r'<img src=\"([^\"]*)\"[^>]*width=\"\\d\\d\\d+\"', page, re.I)\n\n # relative path\n rel_urls = []\n rel_urls += re.findall(r'href=\"(\\.[^\"]+\\.jpe?g)\"', page, re.I)\n rel_urls += re.findall(r'href=\"(\\.[^\"]+\\.png)\"', page, re.I)\n rel_urls += re.findall(r'href=\"(\\.[^\"]+\\.gif)\"', page, re.I)\n for rel_url in rel_urls:\n urls += [ r1(r'(.*/)', url) + rel_url ]\n\n # site-relative path\n rel_urls = []\n rel_urls += re.findall(r'href=\"(/[^\"]+\\.jpe?g)\"', page, re.I)\n rel_urls += re.findall(r'href=\"(/[^\"]+\\.png)\"', page, re.I)\n rel_urls += re.findall(r'href=\"(/[^\"]+\\.gif)\"', page, re.I)\n for rel_url in rel_urls:\n urls += [ r1(r'(https?://[^/]+)', url) + rel_url ]\n\n # sometimes naive\n urls += re.findall(r'data-original=\"(https?://[^\"]+\\.jpe?g)\"', page, re.I)\n urls += re.findall(r'data-original=\"(https?://[^\"]+\\.png)\"', page, re.I)\n urls += re.findall(r'data-original=\"(https?://[^\"]+\\.gif)\"', page, re.I)\n\n # MPEG-DASH MPD\n mpd_urls = re.findall(r'src=\"(https?://[^\"]+\\.mpd)\"', page)\n for mpd_url in mpd_urls:\n cont = get_content(mpd_url)\n base_url = r1(r'<BaseURL>(.*)</BaseURL>', cont)\n urls += [ r1(r'(.*/)[^/]*', mpd_url) + base_url ]\n\n # have some candy!\n candies = []\n i = 1\n for url in set(urls):\n filename = parse.unquote(url.split('/')[-1])\n if 5 <= len(filename) <= 80:\n title = '.'.join(filename.split('.')[:-1]) or filename\n else:\n title = '%s' % i\n i += 1\n\n if r1(r'(https://pinterest.com/pin/)', url):\n continue\n\n candies.append({'url': url,\n 'title': title})\n\n for candy in candies:\n try:\n try:\n mime, ext, size = url_info(candy['url'], faker=False)\n assert size\n except:\n mime, ext, size = url_info(candy['url'], faker=True)\n if not size: size = float('Inf')\n except:\n continue\n else:\n print_info(site_info, candy['title'], ext, size)\n if not info_only:\n try:\n download_urls([candy['url']], candy['title'], ext, size,\n output_dir=output_dir, merge=merge,\n faker=False)\n except:\n download_urls([candy['url']], candy['title'], ext, size,\n output_dir=output_dir, merge=merge,\n faker=True)\n return\n\n else:\n # direct download\n url_trunk = url.split('?')[0] # strip query string\n filename = parse.unquote(url_trunk.split('/')[-1]) or parse.unquote(url_trunk.split('/')[-2])\n title = '.'.join(filename.split('.')[:-1]) or filename\n _, ext, size = url_info(url, faker=True)\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls([url], title, ext, size,\n output_dir=output_dir, merge=merge,\n faker=True)\n return\n\nsite_info = None\ndownload = universal_download\ndownload_playlist = playlist_not_supported('universal')\n" }, { "path": "src/you_get/extractors/veoh.py", "content": "#!/usr/bin/env python\n\n__all__ = ['veoh_download']\n\nfrom ..common import *\n\ndef veoh_download(url, output_dir = '.', merge = False, info_only = False, **kwargs):\n '''Get item_id'''\n if re.match(r'http://www.veoh.com/watch/\\w+', url):\n item_id = match1(url, r'http://www.veoh.com/watch/(\\w+)')\n elif re.match(r'http://www.veoh.com/m/watch.php\\?v=\\.*', url):\n item_id = match1(url, r'http://www.veoh.com/m/watch.php\\?v=(\\w+)')\n else:\n raise NotImplementedError('Cannot find item ID')\n veoh_download_by_id(item_id, output_dir = '.', merge = False, info_only = info_only, **kwargs)\n\n#----------------------------------------------------------------------\ndef veoh_download_by_id(item_id, output_dir = '.', merge = False, info_only = False, **kwargs):\n \"\"\"Source: Android mobile\"\"\"\n webpage_url = 'http://www.veoh.com/m/watch.php?v={item_id}&quality=1'.format(item_id = item_id)\n\n #grab download URL\n a = get_content(webpage_url, decoded=True)\n url = match1(a, r'<source src=\"(.*?)\\\"\\W')\n\n #grab title\n title = match1(a, r'<meta property=\"og:title\" content=\"([^\"]*)\"')\n\n type_, ext, size = url_info(url)\n print_info(site_info, title, type_, size)\n if not info_only:\n download_urls([url], title, ext, total_size=None, output_dir=output_dir, merge=merge)\n\n\nsite_info = \"Veoh\"\ndownload = veoh_download\ndownload_playlist = playlist_not_supported('veoh')\n" }, { "path": "src/you_get/extractors/vimeo.py", "content": "#!/usr/bin/env python\n\n__all__ = ['vimeo_download', 'vimeo_download_by_id', 'vimeo_download_by_channel', 'vimeo_download_by_channel_id']\n\nfrom ..common import *\nfrom ..util.log import *\nfrom ..extractor import VideoExtractor\nfrom json import loads\nimport urllib.error\nimport urllib.parse\n\naccess_token = 'f6785418277b72c7c87d3132c79eec24' #By Beining\n\n#----------------------------------------------------------------------\ndef vimeo_download_by_channel(url, output_dir='.', merge=False, info_only=False, **kwargs):\n \"\"\"str->None\"\"\"\n # https://vimeo.com/channels/464686\n channel_id = match1(url, r'http://vimeo.com/channels/(\\w+)')\n vimeo_download_by_channel_id(channel_id, output_dir, merge, info_only, **kwargs)\n\n#----------------------------------------------------------------------\ndef vimeo_download_by_channel_id(channel_id, output_dir='.', merge=False, info_only=False, **kwargs):\n \"\"\"str/int->None\"\"\"\n html = get_content('https://api.vimeo.com/channels/{channel_id}/videos?access_token={access_token}'.format(channel_id=channel_id, access_token=access_token))\n data = loads(html)\n id_list = []\n\n #print(data)\n for i in data['data']:\n id_list.append(match1(i['uri'], r'/videos/(\\w+)'))\n\n for id in id_list:\n try:\n vimeo_download_by_id(id, None, output_dir, merge, info_only, **kwargs)\n except urllib.error.URLError as e:\n log.w('{} failed with {}'.format(id, e))\n\nclass VimeoExtractor(VideoExtractor):\n stream_types = [\n {'id': '2160p', 'video_profile': '3840x2160'},\n {'id': '1440p', 'video_profile': '2560x1440'},\n {'id': '1080p', 'video_profile': '1920x1080'},\n {'id': '720p', 'video_profile': '1280x720'},\n {'id': '540p', 'video_profile': '960x540'},\n {'id': '360p', 'video_profile': '640x360'}\n ]\n name = 'Vimeo'\n\n def prepare(self, **kwargs):\n headers = fake_headers.copy()\n if 'referer' in kwargs:\n headers['Referer'] = kwargs['referer']\n\n try:\n page = get_content('https://vimeo.com/{}'.format(self.vid))\n cfg_patt = r'clip_page_config\\s*=\\s*(\\{.+?\\});'\n cfg = json.loads(match1(page, cfg_patt))\n video_page = get_content(cfg['player']['config_url'], headers=headers)\n self.title = cfg['clip']['title']\n info = json.loads(video_page)\n except Exception as e:\n page = get_content('https://player.vimeo.com/video/{}'.format(self.vid))\n self.title = r1(r'<title>([^<]+)', page)\n info = json.loads(match1(page, r'var t=(\\{.+?\\});'))\n\n plain = info['request']['files']['progressive']\n for s in plain:\n meta = dict(src=[s['url']], container='mp4')\n meta['video_profile'] = '{}x{}'.format(s['width'], s['height'])\n for stream in self.__class__.stream_types:\n if s['quality'] == stream['id']:\n self.streams[s['quality']] = meta\n self.master_m3u8 = info['request']['files']['hls']['cdns']\n\n def extract(self, **kwargs):\n for s in self.streams:\n self.streams[s]['size'] = urls_size(self.streams[s]['src'])\n\n master_m3u8s = []\n for m in self.master_m3u8:\n master_m3u8s.append(self.master_m3u8[m]['url'])\n\n master_content = None\n master_url = None\n\n for master_u in master_m3u8s:\n try:\n master_content = get_content(master_u).split('\\n')\n except urllib.error.URLError:\n continue\n else:\n master_url = master_u\n\n if master_content is None:\n return\n\n lines = []\n for line in master_content:\n if len(line.strip()) > 0:\n lines.append(line.strip())\n\n pos = 0\n while pos < len(lines):\n if lines[pos].startswith('#EXT-X-STREAM-INF'):\n patt = r'RESOLUTION=(\\d+)x(\\d+)'\n hit = re.search(patt, lines[pos])\n if hit is None:\n continue\n width = hit.group(1)\n height = hit.group(2)\n\n if height in ('2160', '1440'):\n m3u8_url = urllib.parse.urljoin(master_url, lines[pos+1])\n meta = dict(m3u8_url=m3u8_url, container='m3u8')\n if height == '1440':\n meta['video_profile'] = '2560x1440'\n else:\n meta['video_profile'] = '3840x2160'\n meta['size'] = 0\n meta['src'] = general_m3u8_extractor(m3u8_url)\n self.streams[height+'p'] = meta\n\n pos += 2\n else:\n pos += 1\n self.streams_sorted = []\n for stream_type in self.stream_types:\n if stream_type['id'] in self.streams:\n item = [('id', stream_type['id'])] + list(self.streams[stream_type['id']].items())\n self.streams_sorted.append(dict(item))\n\n\n\ndef vimeo_download_by_id(id, title=None, output_dir='.', merge=True, info_only=False, **kwargs):\n site = VimeoExtractor()\n site.download_by_vid(id, info_only=info_only, output_dir=output_dir, merge=merge, **kwargs)\n\ndef vimeo_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n if re.match(r'https?://vimeo.com/channels/\\w+', url):\n vimeo_download_by_channel(url, output_dir, merge, info_only)\n else:\n id = r1(r'https?://[\\w.]*vimeo.com[/\\w]*/(\\d+)', url)\n if id is None:\n video_page = get_content(url, headers=fake_headers)\n id = r1(r'\"clip_id\":(\\d+)', video_page)\n assert id\n\n vimeo_download_by_id(id, None, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\nsite_info = \"Vimeo.com\"\ndownload = vimeo_download\ndownload_playlist = vimeo_download_by_channel\n" }, { "path": "src/you_get/extractors/vk.py", "content": "#!/usr/bin/env python\n\n__all__ = ['vk_download']\n\nfrom ..common import *\n\n\ndef get_video_info(url):\n video_page = get_content(url)\n title = r1(r'
    (.[^>]+?)]+?)\"', video_page)\n\n for quality in ['.1080.', '.720.', '.480.', '.360.', '.240.']:\n for source in sources:\n if source.find(quality) != -1:\n url = source\n break\n assert url\n type, ext, size = url_info(url)\n print_info(site_info, title, type, size)\n\n return url, title, ext, size\n\n\ndef get_video_from_user_videolist(url):\n ep = 'https://vk.com/al_video.php'\n to_post = dict(act='show', al=1, module='direct', video=re.search(r'video(\\d+_\\d+)', url).group(1))\n page = post_content(ep, post_data=to_post)\n video_pt = r'(.+?)
    ', page).group(1)\n mime, ext, size = url_info(url)\n print_info(site_info, title, mime, size)\n\n return url, title, ext, size\n\n\ndef get_image_info(url):\n image_page = get_content(url)\n # used for title - vk page owner\n page_of = re.findall(r'Sender:
    (.[^>]+?)(.[^>]+?)Download full size', image_page)\n type, ext, size = url_info(image_link)\n print_info(site_info, title, type, size)\n\n return image_link, title, ext, size\n\n\ndef vk_download(url, output_dir='.', stream_type=None, merge=True, info_only=False, **kwargs):\n link = None\n if re.match(r'(.+)z\\=video(.+)', url):\n link, title, ext, size = get_video_info(url)\n elif re.match(r'(.+)vk\\.com\\/photo(.+)', url):\n link, title, ext, size = get_image_info(url)\n elif re.search(r'vk\\.com\\/video\\d+_\\d+', url):\n link, title, ext, size = get_video_from_user_videolist(url)\n else:\n raise NotImplementedError('Nothing to download here')\n\n if not info_only and link is not None:\n download_urls([link], title, ext, size, output_dir, merge=merge)\n\n\nsite_info = \"VK.com\"\ndownload = vk_download\ndownload_playlist = playlist_not_supported('vk')\n" }, { "path": "src/you_get/extractors/w56.py", "content": "#!/usr/bin/env python\n\n__all__ = ['w56_download', 'w56_download_by_id']\n\nfrom ..common import *\n\nfrom .sohu import sohu_download\n\nimport json\n\ndef w56_download_by_id(id, title = None, output_dir = '.', merge = True, info_only = False):\n content = json.loads(get_html('http://vxml.56.com/json/%s/?src=site' % id))\n info = content['info']\n title = title or info['Subject']\n assert title\n hd = info['hd']\n assert hd in (0, 1, 2)\n hd_types = [['normal', 'qvga'], ['clear', 'vga'], ['super', 'wvga']][hd]\n files = [x for x in info['rfiles'] if x['type'] in hd_types]\n assert len(files) == 1\n size = int(files[0]['filesize'])\n url = files[0]['url'] + '&prod=56'\n ext = 'mp4'\n\n print_info(site_info, title, ext, size)\n if not info_only:\n download_urls([url], title, ext, size, output_dir = output_dir, merge = merge)\n\ndef w56_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n content = get_content(url)\n sohu_url = r1(r\"url:\\s*'([^']*)'\", content)\n if sohu_url:\n sohu_download(sohu_url, output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n\n id = r1(r'http://www.56.com/u\\d+/v_(\\w+).html', url) or \\\n r1(r'http://www.56.com/.*vid-(\\w+).html', url)\n w56_download_by_id(id, output_dir = output_dir, merge = merge, info_only = info_only)\n\nsite_info = \"56.com\"\ndownload = w56_download\ndownload_playlist = playlist_not_supported('56')\n" }, { "path": "src/you_get/extractors/wanmen.py", "content": "#!/usr/bin/env python\n\n__all__ = ['wanmen_download', 'wanmen_download_by_course', 'wanmen_download_by_course_topic', 'wanmen_download_by_course_topic_part']\n\nfrom ..common import *\nfrom .bokecc import bokecc_download_by_id\nfrom json import loads\n\n\n##Helper functions\ndef _wanmen_get_json_api_content_by_courseID(courseID):\n \"\"\"int->JSON\n \n Return a parsed JSON tree of WanMen's API.\"\"\"\n\n return loads(get_content('http://api.wanmen.org/course/getCourseNested/{courseID}'.format(courseID = courseID)))\n\ndef _wanmen_get_title_by_json_topic_part(json_content, tIndex, pIndex):\n \"\"\"JSON, int, int, int->str\n \n Get a proper title with courseid+topicID+partID.\"\"\"\n\n return '_'.join([json_content[0]['name'],\n json_content[0]['Topics'][tIndex]['name'],\n json_content[0]['Topics'][tIndex]['Parts'][pIndex]['name']])\n\n\ndef _wanmen_get_boke_id_by_json_topic_part(json_content, tIndex, pIndex):\n \"\"\"JSON, int, int, int->str\n \n Get one BokeCC video ID with courseid+topicID+partID.\"\"\"\n\n return json_content[0]['Topics'][tIndex]['Parts'][pIndex]['ccVideoLink']\n\n\n##Parsers\ndef wanmen_download_by_course(json_api_content, output_dir='.', merge=True, info_only=False, **kwargs):\n \"\"\"int->None\n \n Download a WHOLE course.\n Reuse the API call to save time.\"\"\"\n\n for tIndex in range(len(json_api_content[0]['Topics'])):\n for pIndex in range(len(json_api_content[0]['Topics'][tIndex]['Parts'])):\n wanmen_download_by_course_topic_part(json_api_content,\n tIndex,\n pIndex,\n output_dir=output_dir,\n merge=merge,\n info_only=info_only,\n **kwargs)\n\n\ndef wanmen_download_by_course_topic(json_api_content, tIndex, output_dir='.', merge=True, info_only=False, **kwargs):\n \"\"\"int, int->None\n \n Download a TOPIC of a course.\n Reuse the API call to save time.\"\"\"\n\n for pIndex in range(len(json_api_content[0]['Topics'][tIndex]['Parts'])):\n wanmen_download_by_course_topic_part(json_api_content,\n tIndex,\n pIndex, \n output_dir=output_dir,\n merge=merge,\n info_only=info_only,\n **kwargs)\n\ndef wanmen_download_by_course_topic_part(json_api_content, tIndex, pIndex, output_dir='.', merge=True, info_only=False, **kwargs):\n \"\"\"int, int, int->None\n \n Download ONE PART of the course.\"\"\"\n\n html = json_api_content\n\n title = _wanmen_get_title_by_json_topic_part(html, \n tIndex, \n pIndex)\n\n bokeccID = _wanmen_get_boke_id_by_json_topic_part(html,\n tIndex, \n pIndex)\n\n bokecc_download_by_id(vid = bokeccID, title = title, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n\n\n##Main entrance\ndef wanmen_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n\n if not 'wanmen.org' in url:\n log.wtf('You are at the wrong place dude. This is for WanMen University!')\n raise\n\n courseID = int(match1(url, r'course\\/(\\d+)'))\n assert courseID > 0 #without courseID we cannot do anything\n\n tIndex = int(match1(url, r'tIndex=(\\d+)'))\n\n pIndex = int(match1(url, r'pIndex=(\\d+)'))\n\n json_api_content = _wanmen_get_json_api_content_by_courseID(courseID)\n\n if pIndex: #only download ONE single part\n assert tIndex >= 0\n wanmen_download_by_course_topic_part(json_api_content, tIndex, pIndex, \n output_dir = output_dir, \n merge = merge, \n info_only = info_only)\n elif tIndex: #download a topic\n wanmen_download_by_course_topic(json_api_content, tIndex, \n output_dir = output_dir, \n merge = merge, \n info_only = info_only)\n else: #download the whole course\n wanmen_download_by_course(json_api_content,\n output_dir = output_dir, \n merge = merge, \n info_only = info_only)\n\n\nsite_info = \"WanMen University\"\ndownload = wanmen_download\ndownload_playlist = wanmen_download_by_course" }, { "path": "src/you_get/extractors/ximalaya.py", "content": "#!/usr/bin/env python\n\n__all__ = ['ximalaya_download_playlist', 'ximalaya_download', 'ximalaya_download_by_id']\n\nfrom ..common import *\n\nimport json\nimport re\n\nstream_types = [\n {'itag': '1', 'container': 'm4a', 'bitrate': 'default'},\n {'itag': '2', 'container': 'm4a', 'bitrate': '32'},\n {'itag': '3', 'container': 'm4a', 'bitrate': '64'}\n ]\n\ndef ximalaya_download_by_id(id, title = None, output_dir = '.', info_only = False, stream_id = None):\n BASE_URL = 'http://www.ximalaya.com/tracks/'\n json_url = BASE_URL + id + '.json'\n json_data = json.loads(get_content(json_url, headers=fake_headers))\n if 'res' in json_data:\n if json_data['res'] == False:\n raise ValueError('Server reported id %s is invalid' % id)\n if 'is_paid' in json_data and json_data['is_paid']:\n if 'is_free' in json_data and not json_data['is_free']:\n raise ValueError('%s is paid item' % id)\n if (not title) and 'title' in json_data:\n title = json_data['title']\n#no size data in the json. should it be calculated?\n size = 0\n url = json_data['play_path_64']\n if stream_id:\n if stream_id == '1':\n url = json_data['play_path_32']\n elif stream_id == '0':\n url = json_data['play_path']\n logging.debug('ximalaya_download_by_id: %s' % url)\n ext = 'm4a' \n urls = [url]\n print('Site: %s' % site_info)\n print('title: %s' % title)\n if info_only:\n if stream_id:\n print_stream_info(stream_id)\n else:\n for item in range(0, len(stream_types)):\n print_stream_info(item)\n if not info_only:\n print('Type: MPEG-4 audio m4a')\n print('Size: N/A')\n download_urls(urls, title, ext, size, output_dir = output_dir, merge = False)\n\ndef ximalaya_download(url, output_dir = '.', info_only = False, stream_id = None, **kwargs):\n if re.match(r'http://www\\.ximalaya\\.com/(\\d+)/sound/(\\d+)', url):\n id = match1(url, r'http://www\\.ximalaya\\.com/\\d+/sound/(\\d+)')\n else:\n raise NotImplementedError(url)\n ximalaya_download_by_id(id, output_dir = output_dir, info_only = info_only, stream_id = stream_id)\n\ndef ximalaya_download_page(playlist_url, output_dir = '.', info_only = False, stream_id = None, **kwargs):\n if re.match(r'http://www\\.ximalaya\\.com/(\\d+)/album/(\\d+)', playlist_url):\n page_content = get_content(playlist_url)\n pattern = re.compile(r'
  • (\\w+)')\n\n #video_url = match1(html, r'
    400\n # change it to the dispatcher of aliCDN can do better\n # at least a little more recoverable from HTTP 403\n if cls.dispatcher_url in url:\n return url\n elif 'k.youku.com' in url:\n return url\n else:\n url_seg_list = list(urllib.parse.urlsplit(url))\n url_seg_list[1] = cls.dispatcher_url\n return urllib.parse.urlunsplit(url_seg_list)\n\n def get_vid_from_url(self):\n # It's unreliable. check #1633\n b64p = r'([a-zA-Z0-9=]+)'\n p_list = [r'youku\\.com/v_show/id_'+b64p,\n r'player\\.youku\\.com/player\\.php/sid/'+b64p+r'/v\\.swf',\n r'loader\\.swf\\?VideoIDS='+b64p,\n r'player\\.youku\\.com/embed/'+b64p]\n if not self.url:\n raise Exception('No url')\n for p in p_list:\n hit = re.search(p, self.url)\n if hit is not None:\n self.vid = hit.group(1)\n return\n\n def get_vid_from_page(self):\n if not self.url:\n raise Exception('No url')\n self.page = get_content(self.url)\n hit = re.search(r'videoId2:\"([A-Za-z0-9=]+)\"', self.page)\n if hit is not None:\n self.vid = hit.group(1)\n\n def prepare(self, **kwargs):\n assert self.url or self.vid\n\n if self.url and not self.vid:\n self.get_vid_from_url()\n\n if self.vid is None:\n self.get_vid_from_page()\n\n if self.vid is None:\n log.wtf('Cannot fetch vid')\n\n if kwargs.get('src') and kwargs['src'] == 'tudou':\n self.ccode = '0512'\n\n if kwargs.get('password') and kwargs['password']:\n self.password_protected = True\n self.password = kwargs['password']\n\n self.utid = fetch_cna()\n time.sleep(3)\n self.youku_ups()\n\n if self.api_data.get('stream') is None:\n if self.api_error_code == -6001: # wrong vid parsed from the page\n vid_from_url = self.vid\n self.get_vid_from_page()\n if vid_from_url == self.vid:\n log.wtf(self.api_error_msg)\n self.youku_ups()\n\n if self.api_data.get('stream') is None:\n if self.api_error_code == -2002: # wrong password\n self.password_protected = True\n # it can be True already(from cli). offer another chance to retry\n self.password = input(log.sprint('Password: ', log.YELLOW))\n self.youku_ups()\n\n if self.api_data.get('stream') is None:\n if self.api_error_msg:\n log.wtf(self.api_error_msg)\n else:\n log.wtf('Unknown error')\n\n self.title = self.api_data['video']['title']\n stream_types = dict([(i['id'], i) for i in self.stream_types])\n audio_lang = self.api_data['stream'][0]['audio_lang']\n\n for stream in self.api_data['stream']:\n stream_id = stream['stream_type']\n is_preview = False\n if stream_id in stream_types and stream['audio_lang'] == audio_lang:\n if 'alias-of' in stream_types[stream_id]:\n stream_id = stream_types[stream_id]['alias-of']\n\n if stream_id not in self.streams:\n self.streams[stream_id] = {\n 'container': stream_types[stream_id]['container'],\n 'video_profile': stream_types[stream_id]['video_profile'],\n 'size': stream['size'],\n 'pieces': [{\n 'segs': stream['segs']\n }],\n 'm3u8_url': stream['m3u8_url']\n }\n src = []\n for seg in stream['segs']:\n if seg.get('cdn_url'):\n src.append(self.__class__.change_cdn(seg['cdn_url']))\n else:\n is_preview = True\n self.streams[stream_id]['src'] = src\n else:\n self.streams[stream_id]['size'] += stream['size']\n self.streams[stream_id]['pieces'].append({\n 'segs': stream['segs']\n })\n src = []\n for seg in stream['segs']:\n if seg.get('cdn_url'):\n src.append(self.__class__.change_cdn(seg['cdn_url']))\n else:\n is_preview = True\n self.streams[stream_id]['src'].extend(src)\n if is_preview:\n log.w('{} is a preview'.format(stream_id))\n\n # Audio languages\n if 'dvd' in self.api_data:\n al = self.api_data['dvd'].get('audiolang')\n if al:\n self.audiolang = al\n for i in self.audiolang:\n i['url'] = 'http://v.youku.com/v_show/id_{}'.format(i['vid'])\n\n\ndef youku_download_playlist_by_url(url, **kwargs):\n video_page_pt = 'https?://v.youku.com/v_show/id_([A-Za-z0-9=]+)'\n js_cb_pt = r'\\(({.+})\\)'\n if re.match(video_page_pt, url):\n youku_obj = Youku()\n youku_obj.url = url\n youku_obj.prepare(**kwargs)\n total_episode = None\n try:\n total_episode = youku_obj.api_data['show']['episode_total']\n except KeyError:\n log.wtf('Cannot get total_episode for {}'.format(url))\n next_vid = youku_obj.vid\n for _ in range(total_episode):\n this_extractor = Youku()\n this_extractor.download_by_vid(next_vid, keep_obj=True, **kwargs)\n next_vid = this_extractor.video_next['encodevid']\n '''\n if youku_obj.video_list is None:\n log.wtf('Cannot find video list for {}'.format(url))\n else:\n vid_list = [v['encodevid'] for v in youku_obj.video_list]\n for v in vid_list:\n Youku().download_by_vid(v, **kwargs)\n '''\n\n elif re.match('https?://list.youku.com/show/id_', url):\n # http://list.youku.com/show/id_z2ae8ee1c837b11e18195.html\n # official playlist\n page = get_content(url)\n show_id = re.search(r'showid:\"(\\d+)\"', page).group(1)\n ep = 'http://list.youku.com/show/module?id={}&tab=showInfo&callback=jQuery'.format(show_id)\n xhr_page = get_content(ep).replace(r'\\/', '/').replace(r'\\\"', '\"')\n video_url = re.search(r'(v.youku.com/v_show/id_(?:[A-Za-z0-9=]+)\\.html)', xhr_page).group(1)\n youku_download_playlist_by_url('http://'+video_url, **kwargs)\n return\n elif re.match(r'https?://list.youku.com/albumlist/show/id_(\\d+)\\.html', url):\n # http://list.youku.com/albumlist/show/id_2336634.html\n # UGC playlist\n list_id = re.search(r'https?://list.youku.com/albumlist/show/id_(\\d+)\\.html', url).group(1)\n ep = 'http://list.youku.com/albumlist/items?id={}&page={}&size=20&ascending=1&callback=tuijsonp6'\n\n first_u = ep.format(list_id, 1)\n xhr_page = get_content(first_u)\n json_data = json.loads(re.search(js_cb_pt, xhr_page).group(1))\n video_cnt = json_data['data']['total']\n xhr_html = json_data['html']\n v_urls = re.findall(r'(v.youku.com/v_show/id_(?:[A-Za-z0-9=]+)\\.html)', xhr_html)\n\n if video_cnt > 20:\n req_cnt = video_cnt // 20\n for i in range(2, req_cnt+2):\n req_u = ep.format(list_id, i)\n xhr_page = get_content(req_u)\n json_data = json.loads(re.search(js_cb_pt, xhr_page).group(1).replace(r'\\/', '/'))\n xhr_html = json_data['html']\n page_videos = re.findall(r'(v.youku.com/v_show/id_(?:[A-Za-z0-9=]+)\\.html)', xhr_html)\n v_urls.extend(page_videos)\n for u in v_urls[0::2]:\n url = 'http://' + u\n Youku().download_by_url(url, **kwargs)\n return\n\n\ndef youku_download_by_url(url, **kwargs):\n Youku().download_by_url(url, **kwargs)\n\n\ndef youku_download_by_vid(vid, **kwargs):\n Youku().download_by_vid(vid, **kwargs)\n\ndownload = youku_download_by_url\ndownload_playlist = youku_download_playlist_by_url\n" }, { "path": "src/you_get/extractors/youtube.py", "content": "#!/usr/bin/env python\n\nfrom ..common import *\nfrom ..extractor import VideoExtractor\n\ntry:\n import dukpy\nexcept ImportError:\n log.e('Please install dukpy in order to extract videos from YouTube:')\n log.e('$ pip install dukpy')\n exit(0)\nfrom urllib.parse import urlparse, parse_qs, urlencode\nfrom xml.dom.minidom import parseString\n\nclass YouTube(VideoExtractor):\n name = \"YouTube\"\n\n # Non-DASH YouTube media encoding options, in descending quality order.\n # http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs. Retrieved July 17, 2014.\n stream_types = [\n {'itag': '38', 'container': 'MP4', 'video_resolution': '3072p',\n 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3.5-5',\n 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n #{'itag': '85', 'container': 'MP4', 'video_resolution': '1080p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '3-4', 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': '46', 'container': 'WebM', 'video_resolution': '1080p',\n 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '',\n 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n {'itag': '37', 'container': 'MP4', 'video_resolution': '1080p',\n 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '3-4.3',\n 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n #{'itag': '102', 'container': 'WebM', 'video_resolution': '720p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n {'itag': '45', 'container': 'WebM', 'video_resolution': '720p',\n 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '2',\n 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n #{'itag': '84', 'container': 'MP4', 'video_resolution': '720p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '2-3', 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': '22', 'container': 'MP4', 'video_resolution': '720p',\n 'video_encoding': 'H.264', 'video_profile': 'High', 'video_bitrate': '2-3',\n 'audio_encoding': 'AAC', 'audio_bitrate': '192'},\n {'itag': '120', 'container': 'FLV', 'video_resolution': '720p',\n 'video_encoding': 'H.264', 'video_profile': 'Main@L3.1', 'video_bitrate': '2',\n 'audio_encoding': 'AAC', 'audio_bitrate': '128'}, # Live streaming only\n {'itag': '44', 'container': 'WebM', 'video_resolution': '480p',\n 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '1',\n 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': '35', 'container': 'FLV', 'video_resolution': '480p',\n 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.8-1',\n 'audio_encoding': 'AAC', 'audio_bitrate': '128'},\n #{'itag': '101', 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '192'},\n #{'itag': '100', 'container': 'WebM', 'video_resolution': '360p', 'video_encoding': 'VP8', 'video_profile': '3D', 'video_bitrate': '', 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': '43', 'container': 'WebM', 'video_resolution': '360p',\n 'video_encoding': 'VP8', 'video_profile': '', 'video_bitrate': '0.5',\n 'audio_encoding': 'Vorbis', 'audio_bitrate': '128'},\n {'itag': '34', 'container': 'FLV', 'video_resolution': '360p',\n 'video_encoding': 'H.264', 'video_profile': 'Main', 'video_bitrate': '0.5',\n 'audio_encoding': 'AAC', 'audio_bitrate': '128'},\n #{'itag': '82', 'container': 'MP4', 'video_resolution': '360p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': '18', 'container': 'MP4', 'video_resolution': '360p',\n 'video_encoding': 'H.264', 'video_profile': 'Baseline', 'video_bitrate': '0.5',\n 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': '6', 'container': 'FLV', 'video_resolution': '270p',\n 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.8',\n 'audio_encoding': 'MP3', 'audio_bitrate': '64'},\n #{'itag': '83', 'container': 'MP4', 'video_resolution': '240p', 'video_encoding': 'H.264', 'video_profile': '3D', 'video_bitrate': '0.5', 'audio_encoding': 'AAC', 'audio_bitrate': '96'},\n {'itag': '13', 'container': '3GP', 'video_resolution': '',\n 'video_encoding': 'MPEG-4 Visual', 'video_profile': '', 'video_bitrate': '0.5',\n 'audio_encoding': 'AAC', 'audio_bitrate': ''},\n {'itag': '5', 'container': 'FLV', 'video_resolution': '240p',\n 'video_encoding': 'Sorenson H.263', 'video_profile': '', 'video_bitrate': '0.25',\n 'audio_encoding': 'MP3', 'audio_bitrate': '64'},\n {'itag': '36', 'container': '3GP', 'video_resolution': '240p',\n 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.175',\n 'audio_encoding': 'AAC', 'audio_bitrate': '32'},\n {'itag': '17', 'container': '3GP', 'video_resolution': '144p',\n 'video_encoding': 'MPEG-4 Visual', 'video_profile': 'Simple', 'video_bitrate': '0.05',\n 'audio_encoding': 'AAC', 'audio_bitrate': '24'},\n ]\n\n def dethrottle(js, url):\n def n_to_n(js, n):\n # Examples:\n # yma - https://www.youtube.com/s/player/84314bef/player_ias.vflset/en_US/base.js\n # Xka - https://www.youtube.com/s/player/dc0c6770/player_ias.vflset/sv_SE/base.js\n # jma - https://www.youtube.com/s/player/8d9f6215/player_ias.vflset/sv_SE/base.js\n f1 = match1(js, r',[$\\w]+\\.length\\|\\|([$\\w]+)\\(\"\"\\)\\)}};')\n\n # Examples:\n # Yla, ida - https://www.youtube.com/s/player/fb725ac8/player-plasma-ias-phone-sv_SE.vflset/base.js\n # Hla, eda - https://www.youtube.com/s/player/2f238d39/player-plasma-ias-phone-en_US.vflset/base.js\n # WyE, bE7, Gsn - https://www.youtube.com/s/player/3bb1f723/player-plasma-ias-phone-sv_SE.vflset/base.js\n if not f1:\n f0 = match1(js, r'\\w=([$\\w]+)\\[0\\]\\(\\w\\),\\w\\.set\\(\\w,\\w\\)')\n f1 = match1(js, r'%s=\\[([$\\w]+)\\]' % f0)\n\n f1def = match1(js, r'\\W%s=(function\\(\\w+\\).+?\\)});' % re.escape(f1))\n v1 = match1(f1def, r'if\\(typeof ([$\\w]+)===\"undefined\"\\)')\n v1def = match1(js, r'(var %s=[^;]+;)' % v1)\n if not v1def:\n v1def = ''\n n = dukpy.evaljs('%s(%s)(\"%s\")' % (v1def, f1def, n))\n return n\n\n u = urlparse(url)\n qs = parse_qs(u.query)\n n = n_to_n(js, qs['n'][0])\n qs['n'] = [n]\n return u._replace(query=urlencode(qs, doseq=True)).geturl()\n\n def s_to_sig(js, s):\n # Examples:\n # BPa - https://www.youtube.com/s/player/84314bef/player_ias.vflset/en_US/base.js\n # Xva - https://www.youtube.com/s/player/dc0c6770/player_ias.vflset/sv_SE/base.js\n js_code = ''\n f1 = match1(js, r'=([$\\w]+)\\(decodeURIComponent\\(')\n f1def = match1(js, r'\\W%s=function(\\(\\w+\\)\\{[^\\{]+\\})' % re.escape(f1))\n f1def = re.sub(r'([$\\w]+\\.)([$\\w]+\\(\\w+,\\d+\\))', r'\\2', f1def) # remove . prefix\n f1def = 'function %s%s' % (f1, f1def)\n f2s = set(re.findall(r'([$\\w]+)\\(\\w+,\\d+\\)', f1def)) # find all invoked function names\n for f2 in f2s:\n f2e = re.escape(f2)\n f2def = re.search(r'[^$\\w]%s:function\\((\\w+,\\w+)\\)(\\{[^\\{\\}]+\\})' % f2e, js)\n if f2def:\n f2def = 'function {}({}){}'.format(f2e, f2def.group(1), f2def.group(2))\n else:\n f2def = re.search(r'[^$\\w]%s:function\\((\\w+)\\)(\\{[^\\{\\}]+\\})' % f2e, js)\n f2def = 'function {}({},b){}'.format(f2e, f2def.group(1), f2def.group(2))\n js_code += f2def + ';'\n js_code += f1def + ';%s(\"%s\")' % (f1, s)\n sig = dukpy.evaljs(js_code)\n return sig\n\n def chunk_by_range(url, size):\n urls = []\n chunk_size = 10485760\n start, end = 0, chunk_size - 1\n urls.append('%s&range=%s-%s' % (url, start, end))\n while end + 1 < size: # processed size < expected size\n start, end = end + 1, end + chunk_size\n urls.append('%s&range=%s-%s' % (url, start, end))\n return urls\n\n def get_url_from_vid(vid):\n return 'https://youtu.be/{}'.format(vid)\n\n def get_vid_from_url(url):\n \"\"\"Extracts video ID from URL.\n \"\"\"\n return match1(url, r'youtu\\.be/([^?/]+)') or \\\n match1(url, r'youtube\\.com/embed/([^/?]+)') or \\\n match1(url, r'youtube\\.com/shorts/([^/?]+)') or \\\n match1(url, r'youtube\\.com/v/([^/?]+)') or \\\n match1(url, r'youtube\\.com/watch/([^/?]+)') or \\\n parse_query_param(url, 'v') or \\\n parse_query_param(parse_query_param(url, 'u'), 'v')\n\n def get_playlist_id_from_url(url):\n \"\"\"Extracts playlist ID from URL.\n \"\"\"\n return parse_query_param(url, 'list') or \\\n parse_query_param(url, 'p')\n\n def download_playlist_by_url(self, url, **kwargs):\n self.url = url\n\n playlist_id = self.__class__.get_playlist_id_from_url(self.url)\n if playlist_id is None:\n log.wtf('[Failed] Unsupported URL pattern.')\n\n video_page = get_content('https://www.youtube.com/playlist?list=%s' % playlist_id)\n playlist_json_serialized = match1(video_page, r'window\\[\"ytInitialData\"\\]\\s*=\\s*(.+);', r'var\\s+ytInitialData\\s*=\\s*([^;]+);')\n\n if len(playlist_json_serialized) == 0:\n log.wtf('[Failed] Unable to extract playlist data')\n\n ytInitialData = json.loads(playlist_json_serialized[0])\n\n tab0 = ytInitialData['contents']['twoColumnBrowseResultsRenderer']['tabs'][0]\n itemSection0 = tab0['tabRenderer']['content']['sectionListRenderer']['contents'][0]\n playlistVideoList0 = itemSection0['itemSectionRenderer']['contents'][0]\n videos = playlistVideoList0['playlistVideoListRenderer']['contents']\n\n self.title = re.search(r'|var )', video_page).group(1))\n self.check_playability_response(ytInitialPlayerResponse)\n\n # Get the video title\n self.title = ytInitialPlayerResponse[\"videoDetails\"][\"title\"]\n\n # Check the status\n playabilityStatus = ytInitialPlayerResponse['playabilityStatus']\n status = playabilityStatus['status']\n logging.debug('status: %s' % status)\n if status != 'OK':\n # If cookies are loaded, status should be OK\n try:\n subreason = playabilityStatus['errorScreen']['playerErrorMessageRenderer']['subreason']['runs'][0]['text']\n log.e('[Error] %s (%s)' % (playabilityStatus['reason'], subreason))\n except:\n log.e('[Error] %s' % playabilityStatus['reason'])\n if status == 'LOGIN_REQUIRED':\n log.e('View the video from a browser and export the cookies, then use --cookies to load cookies.')\n exit(1)\n\n stream_list = ytInitialPlayerResponse['streamingData']['formats']\n\n for stream in stream_list:\n logging.debug('Found format: itag=%s' % stream['itag'])\n if 'signatureCipher' in stream:\n logging.debug(' Parsing signatureCipher for itag=%s...' % stream['itag'])\n qs = parse_qs(stream['signatureCipher'])\n #logging.debug(qs)\n sp = qs['sp'][0]\n sig = self.__class__.s_to_sig(self.js, qs['s'][0])\n url = qs['url'][0] + '&{}={}'.format(sp, sig)\n elif 'url' in stream:\n url = stream['url']\n else:\n log.wtf(' No signatureCipher or url for itag=%s' % stream['itag'])\n url = self.__class__.dethrottle(self.js, url)\n\n self.streams[str(stream['itag'])] = {\n 'itag': str(stream['itag']),\n 'url': url,\n 'quality': stream['quality'],\n 'type': stream['mimeType'],\n 'mime': stream['mimeType'].split(';')[0],\n 'container': mime_to_container(stream['mimeType'].split(';')[0]),\n }\n\n # FIXME: Prepare caption tracks\n try:\n caption_tracks = ytInitialPlayerResponse['captions']['playerCaptionsTracklistRenderer']['captionTracks']\n for ct in caption_tracks:\n ttsurl, lang = ct['baseUrl'], ct['languageCode']\n\n if ttsurl.startswith('/'):\n ttsurl = 'https://www.youtube.com' + ttsurl\n tts_xml = parseString(get_content(ttsurl))\n transcript = tts_xml.getElementsByTagName('transcript')[0]\n texts = transcript.getElementsByTagName('text')\n srt = \"\"; seq = 0\n for text in texts:\n if text.firstChild is None: continue # empty element\n seq += 1\n start = float(text.getAttribute('start'))\n if text.getAttribute('dur'):\n dur = float(text.getAttribute('dur'))\n else: dur = 1.0 # could be ill-formed XML\n finish = start + dur\n m, s = divmod(start, 60); h, m = divmod(m, 60)\n start = '{:0>2}:{:0>2}:{:06.3f}'.format(int(h), int(m), s).replace('.', ',')\n m, s = divmod(finish, 60); h, m = divmod(m, 60)\n finish = '{:0>2}:{:0>2}:{:06.3f}'.format(int(h), int(m), s).replace('.', ',')\n content = unescape_html(text.firstChild.nodeValue)\n\n srt += '%s\\n' % str(seq)\n srt += '%s --> %s\\n' % (start, finish)\n srt += '%s\\n\\n' % content\n\n if 'kind' in ct:\n self.caption_tracks[ct['vssId']] = srt # autogenerated\n else:\n self.caption_tracks[lang] = srt\n except: pass\n\n # Prepare DASH streams\n if 'adaptiveFormats' in ytInitialPlayerResponse['streamingData']:\n streams = ytInitialPlayerResponse['streamingData']['adaptiveFormats']\n\n # FIXME: dead code?\n # streams without contentLength got broken urls, just remove them (#2767)\n streams = [stream for stream in streams if 'contentLength' in stream]\n\n for stream in streams:\n logging.debug('Found adaptiveFormat: itag=%s' % stream['itag'])\n stream['itag'] = str(stream['itag'])\n if 'qualityLabel' in stream:\n stream['quality_label'] = stream['qualityLabel']\n del stream['qualityLabel']\n logging.debug(' quality_label: \\t%s' % stream['quality_label'])\n if 'width' in stream:\n stream['size'] = '{}x{}'.format(stream['width'], stream['height'])\n del stream['width']\n del stream['height']\n logging.debug(' size: \\t%s' % stream['size'])\n stream['type'] = stream['mimeType']\n logging.debug(' type: \\t%s' % stream['type'])\n stream['clen'] = stream['contentLength']\n stream['init'] = '{}-{}'.format(\n stream['initRange']['start'],\n stream['initRange']['end'])\n stream['index'] = '{}-{}'.format(\n stream['indexRange']['start'],\n stream['indexRange']['end'])\n del stream['mimeType']\n del stream['contentLength']\n del stream['initRange']\n del stream['indexRange']\n\n if 'signatureCipher' in stream:\n logging.debug(' Parsing signatureCipher for itag=%s...' % stream['itag'])\n qs = parse_qs(stream['signatureCipher'])\n #logging.debug(qs)\n sp = qs['sp'][0]\n sig = self.__class__.s_to_sig(self.js, qs['s'][0])\n url = qs['url'][0] + '&ratebypass=yes&{}={}'.format(sp, sig)\n elif 'url' in stream:\n url = stream['url']\n else:\n log.wtf('No signatureCipher or url for itag=%s' % stream['itag'])\n url = self.__class__.dethrottle(self.js, url)\n stream['url'] = url\n\n for stream in streams: # audio\n if stream['type'].startswith('audio/mp4'):\n dash_mp4_a_url = stream['url']\n dash_mp4_a_size = stream['clen']\n elif stream['type'].startswith('audio/webm'):\n dash_webm_a_url = stream['url']\n dash_webm_a_size = stream['clen']\n for stream in streams: # video\n if 'size' in stream:\n if stream['type'].startswith('video/mp4'):\n mimeType = 'video/mp4'\n dash_url = stream['url']\n dash_size = stream['clen']\n itag = stream['itag']\n dash_urls = self.__class__.chunk_by_range(dash_url, int(dash_size))\n dash_mp4_a_urls = self.__class__.chunk_by_range(dash_mp4_a_url, int(dash_mp4_a_size))\n self.dash_streams[itag] = {\n 'quality': '%s (%s)' % (stream['size'], stream['quality_label']),\n 'itag': itag,\n 'type': mimeType,\n 'mime': mimeType,\n 'container': 'mp4',\n 'src': [dash_urls, dash_mp4_a_urls],\n 'size': int(dash_size) + int(dash_mp4_a_size)\n }\n elif stream['type'].startswith('video/webm'):\n mimeType = 'video/webm'\n dash_url = stream['url']\n dash_size = stream['clen']\n itag = stream['itag']\n audio_url = None\n audio_size = None\n try:\n audio_url = dash_webm_a_url\n audio_size = int(dash_webm_a_size)\n except UnboundLocalError as e:\n audio_url = dash_mp4_a_url\n audio_size = int(dash_mp4_a_size)\n dash_urls = self.__class__.chunk_by_range(dash_url, int(dash_size))\n audio_urls = self.__class__.chunk_by_range(audio_url, int(audio_size))\n self.dash_streams[itag] = {\n 'quality': '%s (%s)' % (stream['size'], stream['quality_label']),\n 'itag': itag,\n 'type': mimeType,\n 'mime': mimeType,\n 'container': 'webm',\n 'src': [dash_urls, audio_urls],\n 'size': int(dash_size) + int(audio_size)\n }\n\n def extract(self, **kwargs):\n if not self.streams_sorted:\n # No stream is available\n return\n\n if 'stream_id' in kwargs and kwargs['stream_id']:\n # Extract the stream\n stream_id = kwargs['stream_id']\n if stream_id not in self.streams and stream_id not in self.dash_streams:\n log.e('[Error] Invalid video format.')\n log.e('Run \\'-i\\' command with no specific video format to view all available formats.')\n exit(2)\n else:\n # Extract stream with the best quality\n stream_id = self.streams_sorted[0]['itag']\n\n if stream_id in self.streams:\n src = self.streams[stream_id]['url']\n\n self.streams[stream_id]['src'] = [src]\n self.streams[stream_id]['size'] = urls_size(self.streams[stream_id]['src'])\n\nsite = YouTube()\ndownload = site.download_by_url\ndownload_playlist = site.download_playlist_by_url\n" }, { "path": "src/you_get/extractors/zhanqi.py", "content": "#!/usr/bin/env python\n\n__all__ = ['zhanqi_download']\n\nfrom ..common import *\nimport json\nimport base64\nfrom urllib.parse import urlparse\n\ndef zhanqi_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n path = urlparse(url).path[1:]\n\n if not (path.startswith('videos') or path.startswith('v2/videos')): #url = \"https://www.zhanqi.tv/huashan?param_s=1_0.2.0\"\n path_list = path.split('/')\n room_id = path_list[1] if path_list[0] == 'topic' else path_list[0]\n zhanqi_live(room_id, merge=merge, output_dir=output_dir, info_only=info_only, **kwargs)\n else: #url = 'https://www.zhanqi.tv/videos/Lyingman/2017/01/182308.html'\n # https://www.zhanqi.tv/v2/videos/215593.html\n video_id = path.split('.')[0].split('/')[-1]\n zhanqi_video(video_id, merge=merge, output_dir=output_dir, info_only=info_only, **kwargs)\n\ndef zhanqi_live(room_id, merge=True, output_dir='.', info_only=False, **kwargs):\n api_url = \"https://www.zhanqi.tv/api/static/v2.1/room/domain/{}.json\".format(room_id)\n json_data = json.loads(get_content(api_url))['data']\n status = json_data['status']\n if status != '4':\n raise Exception(\"The live stream is not online!\")\n\n nickname = json_data['nickname']\n title = nickname + \": \" + json_data['title']\n video_levels = base64.b64decode(json_data['flashvars']['VideoLevels']).decode('utf8')\n m3u8_url = json.loads(video_levels)['streamUrl']\n\n print_info(site_info, title, 'm3u8', 0, m3u8_url=m3u8_url, m3u8_type='master')\n if not info_only:\n download_url_ffmpeg(m3u8_url, title, 'mp4', output_dir=output_dir, merge=merge)\n\ndef zhanqi_video(video_id, output_dir='.', info_only=False, merge=True, **kwargs):\n api_url = 'https://www.zhanqi.tv/api/static/v2.1/video/{}.json'.format(video_id)\n json_data = json.loads(get_content(api_url))['data']\n\n title = json_data['title']\n vid = json_data['flashvars']['VideoID']\n m3u8_url = 'http://dlvod.cdn.zhanqi.tv/' + vid\n urls = general_m3u8_extractor(m3u8_url)\n print_info(site_info, title, 'm3u8', 0)\n if not info_only:\n download_urls(urls, title, 'ts', 0, output_dir=output_dir, merge=merge, **kwargs)\n\nsite_info = \"www.zhanqi.tv\"\ndownload = zhanqi_download\ndownload_playlist = playlist_not_supported('zhanqi')\n" }, { "path": "src/you_get/extractors/zhibo.py", "content": "#!/usr/bin/env python\n\n__all__ = ['zhibo_download']\n\nfrom ..common import *\n\ndef zhibo_vedio_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n # http://video.zhibo.tv/video/details/d103057f-663e-11e8-9d83-525400ccac43.html\n\n html = get_html(url)\n title = r1(r'([\\s\\S]*)', html)\n total_size = 0\n part_urls= []\n\n video_html = r1(r'', html)\n\n # video_guessulike = r1(r\"window.xgData =([s\\S'\\s\\.]*)\\'\\;[\\s\\S]*window.vouchData\", video_html)\n video_url = r1(r\"window.vurl = \\'([s\\S'\\s\\.]*)\\'\\;[\\s\\S]*window.imgurl\", video_html)\n part_urls.append(video_url)\n ext = video_url.split('.')[-1]\n\n print_info(site_info, title, ext, total_size)\n if not info_only:\n download_urls(part_urls, title, ext, total_size, output_dir=output_dir, merge=merge)\n\n\ndef zhibo_download(url, output_dir = '.', merge = True, info_only = False, **kwargs):\n if 'video.zhibo.tv' in url:\n zhibo_vedio_download(url, output_dir=output_dir, merge=merge, info_only=info_only, **kwargs)\n return\n\n # if 'v.zhibo.tv' in url:\n # http://v.zhibo.tv/31609372\n html = get_html(url)\n title = r1(r'([\\s\\S]*)', html)\n is_live = r1(r\"window.videoIsLive=\\'([s\\S'\\s\\.]*)\\'\\;[\\s\\S]*window.resDomain\", html)\n if is_live != \"1\":\n raise ValueError(\"The live stream is not online! (Errno:%s)\" % is_live)\n\n match = re.search(r\"\"\"\n ourStreamName .*?\n '(.*?)' .*?\n rtmpHighSource .*?\n '(.*?)' .*?\n '(.*?)'\n \"\"\", html, re.S | re.X)\n real_url = match.group(3) + match.group(1) + match.group(2)\n\n print_info(site_info, title, 'flv', float('inf'))\n if not info_only:\n download_url_ffmpeg(real_url, title, 'flv', params={}, output_dir=output_dir, merge=merge)\n\nsite_info = \"zhibo.tv\"\ndownload = zhibo_download\ndownload_playlist = playlist_not_supported('zhibo')\n" }, { "path": "src/you_get/extractors/zhihu.py", "content": "#!/usr/bin/env python\n\n__all__ = ['zhihu_download', 'zhihu_download_playlist']\n\nfrom ..common import *\nimport json\n\n\ndef zhihu_download(url, output_dir='.', merge=True, info_only=False, **kwargs):\n paths = url.split(\"/\")\n # question or column\n if len(paths) < 3 and len(paths) < 6:\n raise TypeError(\"URL does not conform to specifications, Support column and question only.\"\n \"Example URL: https://zhuanlan.zhihu.com/p/51669862 or \"\n \"https://www.zhihu.com/question/267782048/answer/490720324\")\n\n if (\"question\" not in paths or \"answer\" not in paths) and \"zhuanlan.zhihu.com\" not in paths:\n raise TypeError(\"URL does not conform to specifications, Support column and question only.\"\n \"Example URL: https://zhuanlan.zhihu.com/p/51669862 or \"\n \"https://www.zhihu.com/question/267782048/answer/490720324\")\n\n html = get_html(url, faker=True)\n title = match1(html, r'data-react-helmet=\"true\">(.*?)')\n for index, video_id in enumerate(matchall(html, [r' '0') or (vers[0] == 'avconv')\n try:\n v = vers[2][1:] if vers[2][0] == 'n' else vers[2]\n version = [int(i) for i in v.split('.')]\n except:\n version = [1, 0]\n return cmd, 'ffprobe', version\n except:\n return None\n\nFFMPEG, FFPROBE, FFMPEG_VERSION = get_usable_ffmpeg('ffmpeg') or get_usable_ffmpeg('avconv') or (None, None, None)\nif logging.getLogger().isEnabledFor(logging.DEBUG):\n LOGLEVEL = ['-loglevel', 'info']\n STDIN = None\nelse:\n LOGLEVEL = ['-loglevel', 'quiet']\n STDIN = DEVNULL\n\ndef has_ffmpeg_installed():\n return FFMPEG is not None\n\n# Given a list of segments and the output path, generates the concat\n# list and returns the path to the concat list.\ndef generate_concat_list(files, output):\n concat_list_path = output + '.txt'\n concat_list_dir = os.path.dirname(concat_list_path)\n with open(concat_list_path, 'w', encoding='utf-8') as concat_list:\n for file in files:\n if os.path.isfile(file):\n relpath = os.path.relpath(file, start=concat_list_dir)\n concat_list.write('file %s\\n' % parameterize(relpath))\n return concat_list_path\n\ndef ffmpeg_concat_av(files, output, ext):\n print('Merging video parts... ', end=\"\", flush=True)\n params = [FFMPEG] + LOGLEVEL\n for file in files:\n if os.path.isfile(file): params.extend(['-i', file])\n params.extend(['-c', 'copy'])\n params.extend(['--', output])\n if subprocess.call(params, stdin=STDIN):\n print('Merging without re-encode failed.\\nTry again re-encoding audio... ', end=\"\", flush=True)\n try: os.remove(output)\n except FileNotFoundError: pass\n params = [FFMPEG] + LOGLEVEL\n for file in files:\n if os.path.isfile(file): params.extend(['-i', file])\n params.extend(['-c:v', 'copy'])\n if ext == 'mp4':\n params.extend(['-c:a', 'aac'])\n params.extend(['-strict', 'experimental'])\n elif ext == 'webm':\n params.extend(['-c:a', 'opus'])\n params.extend(['--', output])\n return subprocess.call(params, stdin=STDIN)\n else:\n return 0\n\ndef ffmpeg_convert_ts_to_mkv(files, output='output.mkv'):\n for file in files:\n if os.path.isfile(file):\n params = [FFMPEG] + LOGLEVEL\n params.extend(['-y', '-i', file])\n params.extend(['--', output])\n subprocess.call(params, stdin=STDIN)\n\n return\n\ndef ffmpeg_concat_mp4_to_mpg(files, output='output.mpg'):\n # Use concat demuxer on FFmpeg >= 1.1\n if FFMPEG == 'ffmpeg' and (FFMPEG_VERSION[0] >= 2 or (FFMPEG_VERSION[0] == 1 and FFMPEG_VERSION[1] >= 1)):\n concat_list = generate_concat_list(files, output)\n params = [FFMPEG] + LOGLEVEL + ['-y', '-f', 'concat', '-safe', '0',\n '-i', concat_list, '-c', 'copy']\n params.extend(['--', output])\n if subprocess.call(params, stdin=STDIN) == 0:\n os.remove(output + '.txt')\n return True\n else:\n raise\n\n for file in files:\n if os.path.isfile(file):\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.extend([file, file + '.mpg'])\n subprocess.call(params, stdin=STDIN)\n\n inputs = [open(file + '.mpg', 'rb') for file in files]\n with open(output + '.mpg', 'wb') as o:\n for input in inputs:\n o.write(input.read())\n\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append(output + '.mpg')\n params += ['-vcodec', 'copy', '-acodec', 'copy']\n params.extend(['--', output])\n\n if subprocess.call(params, stdin=STDIN) == 0:\n for file in files:\n os.remove(file + '.mpg')\n os.remove(output + '.mpg')\n return True\n else:\n raise\n\ndef ffmpeg_concat_ts_to_mkv(files, output='output.mkv'):\n print('Merging video parts... ', end=\"\", flush=True)\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append('concat:')\n for file in files:\n if os.path.isfile(file):\n params[-1] += file + '|'\n params += ['-f', 'matroska', '-c', 'copy']\n params.extend(['--', output])\n\n try:\n if subprocess.call(params, stdin=STDIN) == 0:\n return True\n else:\n return False\n except:\n return False\n\ndef ffmpeg_concat_flv_to_mp4(files, output='output.mp4'):\n print('Merging video parts... ', end=\"\", flush=True)\n # Use concat demuxer on FFmpeg >= 1.1\n if FFMPEG == 'ffmpeg' and (FFMPEG_VERSION[0] >= 2 or (FFMPEG_VERSION[0] == 1 and FFMPEG_VERSION[1] >= 1)):\n concat_list = generate_concat_list(files, output)\n params = [FFMPEG] + LOGLEVEL + ['-y', '-f', 'concat', '-safe', '0',\n '-i', concat_list, '-c', 'copy',\n '-bsf:a', 'aac_adtstoasc']\n params.extend(['--', output])\n subprocess.check_call(params, stdin=STDIN)\n os.remove(output + '.txt')\n return True\n\n for file in files:\n if os.path.isfile(file):\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append(file)\n params += ['-map', '0', '-c', 'copy', '-f', 'mpegts', '-bsf:v', 'h264_mp4toannexb']\n params.append(file + '.ts')\n\n subprocess.call(params, stdin=STDIN)\n\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append('concat:')\n for file in files:\n f = file + '.ts'\n if os.path.isfile(f):\n params[-1] += f + '|'\n if FFMPEG == 'avconv':\n params += ['-c', 'copy']\n else:\n params += ['-c', 'copy', '-bsf:a', 'aac_adtstoasc']\n params.extend(['--', output])\n\n if subprocess.call(params, stdin=STDIN) == 0:\n for file in files:\n os.remove(file + '.ts')\n return True\n else:\n raise\n\ndef ffmpeg_concat_mp3_to_mp3(files, output='output.mp3'):\n print('Merging video parts... ', end=\"\", flush=True)\n\n files = 'concat:' + '|'.join(files)\n\n params = [FFMPEG] + LOGLEVEL + ['-y']\n params += ['-i', files, '-acodec', 'copy']\n params.extend(['--', output])\n\n subprocess.call(params)\n\n return True\n\ndef ffmpeg_concat_mp4_to_mp4(files, output='output.mp4'):\n print('Merging video parts... ', end=\"\", flush=True)\n # Use concat demuxer on FFmpeg >= 1.1\n if FFMPEG == 'ffmpeg' and (FFMPEG_VERSION[0] >= 2 or (FFMPEG_VERSION[0] == 1 and FFMPEG_VERSION[1] >= 1)):\n concat_list = generate_concat_list(files, output)\n params = [FFMPEG] + LOGLEVEL + ['-y', '-f', 'concat', '-safe', '0',\n '-i', concat_list, '-c', 'copy',\n '-bsf:a', 'aac_adtstoasc']\n params.extend(['--', output])\n subprocess.check_call(params, stdin=STDIN)\n os.remove(output + '.txt')\n return True\n\n for file in files:\n if os.path.isfile(file):\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append(file)\n params += ['-c', 'copy', '-f', 'mpegts', '-bsf:v', 'h264_mp4toannexb']\n params.append(file + '.ts')\n\n subprocess.call(params, stdin=STDIN)\n\n params = [FFMPEG] + LOGLEVEL + ['-y', '-i']\n params.append('concat:')\n for file in files:\n f = file + '.ts'\n if os.path.isfile(f):\n params[-1] += f + '|'\n if FFMPEG == 'avconv':\n params += ['-c', 'copy']\n else:\n params += ['-c', 'copy', '-bsf:a', 'aac_adtstoasc']\n params.extend(['--', output])\n\n subprocess.check_call(params, stdin=STDIN)\n for file in files:\n os.remove(file + '.ts')\n return True\n\ndef ffmpeg_download_stream(files, title, ext, params={}, output_dir='.', stream=True):\n \"\"\"str, str->True\n WARNING: NOT THE SAME PARMS AS OTHER FUNCTIONS!!!!!!\n You can basically download anything with this function\n but better leave it alone with\n \"\"\"\n output = title + '.' + ext\n\n if not (output_dir == '.'):\n output = output_dir + '/' + output\n\n print('Downloading streaming content with FFmpeg, press q to stop recording...')\n if stream:\n ffmpeg_params = [FFMPEG] + ['-y', '-re', '-i']\n else:\n ffmpeg_params = [FFMPEG] + ['-y', '-i']\n ffmpeg_params.append(files) #not the same here!!!!\n\n if FFMPEG == 'avconv': #who cares?\n ffmpeg_params += ['-c', 'copy']\n else:\n ffmpeg_params += ['-c', 'copy', '-bsf:a', 'aac_adtstoasc']\n\n if params is not None:\n if len(params) > 0:\n for k, v in params:\n ffmpeg_params.append(k)\n ffmpeg_params.append(v)\n\n ffmpeg_params.extend(['--', output])\n\n print(' '.join(ffmpeg_params))\n\n try:\n a = subprocess.Popen(ffmpeg_params, stdin= subprocess.PIPE)\n a.communicate()\n except KeyboardInterrupt:\n try:\n a.stdin.write('q'.encode('utf-8'))\n except:\n pass\n\n return True\n\n\ndef ffmpeg_concat_audio_and_video(files, output, ext):\n print('Merging video and audio parts... ', end=\"\", flush=True)\n if has_ffmpeg_installed:\n params = [FFMPEG] + LOGLEVEL\n params.extend(['-f', 'concat'])\n params.extend(['-safe', '0']) # https://stackoverflow.com/questions/38996925/ffmpeg-concat-unsafe-file-name\n for file in files:\n if os.path.isfile(file):\n params.extend(['-i', file])\n params.extend(['-c:v', 'copy'])\n params.extend(['-c:a', 'aac'])\n params.extend(['-strict', 'experimental'])\n params.extend(['--', output + \".\" + ext])\n return subprocess.call(params, stdin=STDIN)\n else:\n raise EnvironmentError('No ffmpeg found')\n\n\ndef ffprobe_get_media_duration(file):\n print('Getting {} duration'.format(file))\n params = [FFPROBE]\n params.extend(['-i', file])\n params.extend(['-show_entries', 'format=duration'])\n params.extend(['-v', 'quiet'])\n params.extend(['-of', 'csv=p=0'])\n return subprocess.check_output(params, stdin=STDIN, stderr=subprocess.STDOUT).decode().strip()\n" }, { "path": "src/you_get/processor/join_flv.py", "content": "#!/usr/bin/env python\n\nimport struct\nfrom io import BytesIO\n\nTAG_TYPE_METADATA = 18\n\n##################################################\n# AMF0\n##################################################\n\nAMF_TYPE_NUMBER = 0x00\nAMF_TYPE_BOOLEAN = 0x01\nAMF_TYPE_STRING = 0x02\nAMF_TYPE_OBJECT = 0x03\nAMF_TYPE_MOVIECLIP = 0x04\nAMF_TYPE_NULL = 0x05\nAMF_TYPE_UNDEFINED = 0x06\nAMF_TYPE_REFERENCE = 0x07\nAMF_TYPE_MIXED_ARRAY = 0x08\nAMF_TYPE_END_OF_OBJECT = 0x09\nAMF_TYPE_ARRAY = 0x0A\nAMF_TYPE_DATE = 0x0B\nAMF_TYPE_LONG_STRING = 0x0C\nAMF_TYPE_UNSUPPORTED = 0x0D\nAMF_TYPE_RECORDSET = 0x0E\nAMF_TYPE_XML = 0x0F\nAMF_TYPE_CLASS_OBJECT = 0x10\nAMF_TYPE_AMF3_OBJECT = 0x11\n\nclass ECMAObject:\n def __init__(self, max_number):\n self.max_number = max_number\n self.data = []\n self.map = {}\n def put(self, k, v):\n self.data.append((k, v))\n self.map[k] = v\n def get(self, k):\n return self.map[k]\n def set(self, k, v):\n for i in range(len(self.data)):\n if self.data[i][0] == k:\n self.data[i] = (k, v)\n break\n else:\n raise KeyError(k)\n self.map[k] = v\n def keys(self):\n return self.map.keys()\n def __str__(self):\n return 'ECMAObject<' + repr(self.map) + '>'\n def __eq__(self, other):\n return self.max_number == other.max_number and self.data == other.data\n\ndef read_amf_number(stream):\n return struct.unpack('>d', stream.read(8))[0]\n\ndef read_amf_boolean(stream):\n b = read_byte(stream)\n assert b in (0, 1)\n return bool(b)\n\ndef read_amf_string(stream):\n xx = stream.read(2)\n if xx == b'':\n # dirty fix for the invalid Qiyi flv\n return None\n n = struct.unpack('>H', xx)[0]\n s = stream.read(n)\n assert len(s) == n\n return s.decode('utf-8')\n\ndef read_amf_object(stream):\n obj = {}\n while True:\n k = read_amf_string(stream)\n if not k:\n assert read_byte(stream) == AMF_TYPE_END_OF_OBJECT\n break\n v = read_amf(stream)\n obj[k] = v\n return obj\n\ndef read_amf_mixed_array(stream):\n max_number = read_uint(stream)\n mixed_results = ECMAObject(max_number)\n while True:\n k = read_amf_string(stream)\n if k is None:\n # dirty fix for the invalid Qiyi flv\n break\n if not k:\n assert read_byte(stream) == AMF_TYPE_END_OF_OBJECT\n break\n v = read_amf(stream)\n mixed_results.put(k, v)\n assert len(mixed_results.data) == max_number\n return mixed_results\n\ndef read_amf_array(stream):\n n = read_uint(stream)\n v = []\n for i in range(n):\n v.append(read_amf(stream))\n return v\n\namf_readers = {\n AMF_TYPE_NUMBER: read_amf_number,\n AMF_TYPE_BOOLEAN: read_amf_boolean,\n AMF_TYPE_STRING: read_amf_string,\n AMF_TYPE_OBJECT: read_amf_object,\n AMF_TYPE_MIXED_ARRAY: read_amf_mixed_array,\n AMF_TYPE_ARRAY: read_amf_array,\n}\n\ndef read_amf(stream):\n return amf_readers[read_byte(stream)](stream)\n\ndef write_amf_number(stream, v):\n stream.write(struct.pack('>d', v))\n\ndef write_amf_boolean(stream, v):\n if v:\n stream.write(b'\\x01')\n else:\n stream.write(b'\\x00')\n\ndef write_amf_string(stream, s):\n s = s.encode('utf-8')\n stream.write(struct.pack('>H', len(s)))\n stream.write(s)\n\ndef write_amf_object(stream, o):\n for k in o:\n write_amf_string(stream, k)\n write_amf(stream, o[k])\n write_amf_string(stream, '')\n write_byte(stream, AMF_TYPE_END_OF_OBJECT)\n\ndef write_amf_mixed_array(stream, o):\n write_uint(stream, o.max_number)\n for k, v in o.data:\n write_amf_string(stream, k)\n write_amf(stream, v)\n write_amf_string(stream, '')\n write_byte(stream, AMF_TYPE_END_OF_OBJECT)\n\ndef write_amf_array(stream, o):\n write_uint(stream, len(o))\n for v in o:\n write_amf(stream, v)\n\namf_writers_tags = {\n float: AMF_TYPE_NUMBER,\n bool: AMF_TYPE_BOOLEAN,\n str: AMF_TYPE_STRING,\n dict: AMF_TYPE_OBJECT,\n ECMAObject: AMF_TYPE_MIXED_ARRAY,\n list: AMF_TYPE_ARRAY,\n}\n\namf_writers = {\n AMF_TYPE_NUMBER: write_amf_number,\n AMF_TYPE_BOOLEAN: write_amf_boolean,\n AMF_TYPE_STRING: write_amf_string,\n AMF_TYPE_OBJECT: write_amf_object,\n AMF_TYPE_MIXED_ARRAY: write_amf_mixed_array,\n AMF_TYPE_ARRAY: write_amf_array,\n}\n\ndef write_amf(stream, v):\n if isinstance(v, ECMAObject):\n tag = amf_writers_tags[ECMAObject]\n else:\n tag = amf_writers_tags[type(v)]\n write_byte(stream, tag)\n amf_writers[tag](stream, v)\n\n##################################################\n# FLV\n##################################################\n\ndef read_int(stream):\n return struct.unpack('>i', stream.read(4))[0]\n\ndef read_uint(stream):\n return struct.unpack('>I', stream.read(4))[0]\n\ndef write_uint(stream, n):\n stream.write(struct.pack('>I', n))\n\ndef read_byte(stream):\n return ord(stream.read(1))\n\ndef write_byte(stream, b):\n stream.write(bytes([b]))\n\ndef read_unsigned_medium_int(stream):\n x1, x2, x3 = struct.unpack('BBB', stream.read(3))\n return (x1 << 16) | (x2 << 8) | x3\n\ndef read_tag(stream):\n # header size: 15 bytes\n header = stream.read(15)\n if len(header) == 4:\n return\n x = struct.unpack('>IBBBBBBBBBBB', header)\n previous_tag_size = x[0]\n data_type = x[1]\n body_size = (x[2] << 16) | (x[3] << 8) | x[4]\n assert body_size < 1024 * 1024 * 128, 'tag body size too big (> 128MB)'\n timestamp = (x[5] << 16) | (x[6] << 8) | x[7]\n timestamp += x[8] << 24\n assert x[9:] == (0, 0, 0)\n body = stream.read(body_size)\n return (data_type, timestamp, body_size, body, previous_tag_size)\n #previous_tag_size = read_uint(stream)\n #data_type = read_byte(stream)\n #body_size = read_unsigned_medium_int(stream)\n #assert body_size < 1024*1024*128, 'tag body size too big (> 128MB)'\n #timestamp = read_unsigned_medium_int(stream)\n #timestamp += read_byte(stream) << 24\n #assert read_unsigned_medium_int(stream) == 0\n #body = stream.read(body_size)\n #return (data_type, timestamp, body_size, body, previous_tag_size)\n\ndef write_tag(stream, tag):\n data_type, timestamp, body_size, body, previous_tag_size = tag\n write_uint(stream, previous_tag_size)\n write_byte(stream, data_type)\n write_byte(stream, body_size>>16 & 0xff)\n write_byte(stream, body_size>>8 & 0xff)\n write_byte(stream, body_size & 0xff)\n write_byte(stream, timestamp>>16 & 0xff)\n write_byte(stream, timestamp>>8 & 0xff)\n write_byte(stream, timestamp & 0xff)\n write_byte(stream, timestamp>>24 & 0xff)\n stream.write(b'\\0\\0\\0')\n stream.write(body)\n\ndef read_flv_header(stream):\n assert stream.read(3) == b'FLV'\n header_version = read_byte(stream)\n assert header_version == 1\n type_flags = read_byte(stream)\n assert type_flags == 5\n data_offset = read_uint(stream)\n assert data_offset == 9\n\ndef write_flv_header(stream):\n stream.write(b'FLV')\n write_byte(stream, 1)\n write_byte(stream, 5)\n write_uint(stream, 9)\n\ndef read_meta_data(stream):\n meta_type = read_amf(stream)\n meta = read_amf(stream)\n return meta_type, meta\n\ndef read_meta_tag(tag):\n data_type, timestamp, body_size, body, previous_tag_size = tag\n assert data_type == TAG_TYPE_METADATA\n assert timestamp == 0\n assert previous_tag_size == 0\n return read_meta_data(BytesIO(body))\n\n#def write_meta_data(stream, meta_type, meta_data):\n# assert isinstance(meta_type, basesting)\n# write_amf(meta_type)\n# write_amf(meta_data)\n\ndef write_meta_tag(stream, meta_type, meta_data):\n buffer = BytesIO()\n write_amf(buffer, meta_type)\n write_amf(buffer, meta_data)\n body = buffer.getvalue()\n write_tag(stream, (TAG_TYPE_METADATA, 0, len(body), body, 0))\n\n\n##################################################\n# main\n##################################################\n\ndef guess_output(inputs):\n import os.path\n inputs = map(os.path.basename, inputs)\n n = min(map(len, inputs))\n for i in reversed(range(1, n)):\n if len(set(s[:i] for s in inputs)) == 1:\n return inputs[0][:i] + '.flv'\n return 'output.flv'\n\ndef concat_flv(flvs, output = None):\n assert flvs, 'no flv file found'\n import os.path\n if not output:\n output = guess_output(flvs)\n elif os.path.isdir(output):\n output = os.path.join(output, guess_output(flvs))\n \n print('Merging video parts...')\n ins = [open(flv, 'rb') for flv in flvs]\n for stream in ins:\n read_flv_header(stream)\n meta_tags = map(read_tag, ins)\n metas = list(map(read_meta_tag, meta_tags))\n meta_types, metas = zip(*metas)\n assert len(set(meta_types)) == 1\n meta_type = meta_types[0]\n \n # must merge fields: duration\n # TODO: check other meta info, update other meta info\n total_duration = sum(meta.get('duration') for meta in metas)\n meta_data = metas[0]\n meta_data.set('duration', total_duration)\n \n out = open(output, 'wb')\n write_flv_header(out)\n write_meta_tag(out, meta_type, meta_data)\n timestamp_start = 0\n for stream in ins:\n while True:\n tag = read_tag(stream)\n if tag:\n data_type, timestamp, body_size, body, previous_tag_size = tag\n timestamp += timestamp_start\n tag = data_type, timestamp, body_size, body, previous_tag_size\n write_tag(out, tag)\n else:\n break\n timestamp_start = timestamp\n write_uint(out, previous_tag_size)\n \n return output\n\ndef usage():\n print('Usage: [python3] join_flv.py --output TARGET.flv flv...')\n\ndef main():\n import sys, getopt\n try:\n opts, args = getopt.getopt(sys.argv[1:], \"ho:\", [\"help\", \"output=\"])\n except getopt.GetoptError as err:\n usage()\n sys.exit(1)\n output = None\n for o, a in opts:\n if o in (\"-h\", \"--help\"):\n usage()\n sys.exit()\n elif o in (\"-o\", \"--output\"):\n output = a\n else:\n usage()\n sys.exit(1)\n if not args:\n usage()\n sys.exit(1)\n \n concat_flv(args, output)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/you_get/processor/join_mp4.py", "content": "#!/usr/bin/env python\n\n# reference: c041828_ISO_IEC_14496-12_2005(E).pdf\n\n##################################################\n# reader and writer\n##################################################\n\nimport struct\nfrom io import BytesIO\n\ndef skip(stream, n):\n stream.seek(stream.tell() + n)\n\ndef skip_zeros(stream, n):\n assert stream.read(n) == b'\\x00' * n\n\ndef read_int(stream):\n return struct.unpack('>i', stream.read(4))[0]\n\ndef read_uint(stream):\n return struct.unpack('>I', stream.read(4))[0]\n\ndef write_uint(stream, n):\n stream.write(struct.pack('>I', n))\n\ndef write_ulong(stream, n):\n stream.write(struct.pack('>Q', n))\n\ndef read_ushort(stream):\n return struct.unpack('>H', stream.read(2))[0]\n\ndef read_ulong(stream):\n return struct.unpack('>Q', stream.read(8))[0]\n\ndef read_byte(stream):\n return ord(stream.read(1))\n\ndef copy_stream(source, target, n):\n buffer_size = 1024 * 1024\n while n > 0:\n to_read = min(buffer_size, n)\n s = source.read(to_read)\n assert len(s) == to_read, 'no enough data'\n target.write(s)\n n -= to_read\n\nclass Atom:\n def __init__(self, type, size, body):\n assert len(type) == 4\n self.type = type\n self.size = size\n self.body = body\n def __str__(self):\n #return '' % (self.type, repr(self.body))\n return '' % (self.type, '')\n def __repr__(self):\n return str(self)\n def write1(self, stream):\n write_uint(stream, self.size)\n stream.write(self.type)\n def write(self, stream):\n assert type(self.body) == bytes, '%s: %s' % (self.type, type(self.body))\n assert self.size == 8 + len(self.body)\n self.write1(stream)\n stream.write(self.body)\n def calsize(self):\n return self.size\n\nclass CompositeAtom(Atom):\n def __init__(self, type, size, body):\n assert isinstance(body, list)\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n assert type(self.body) == list\n self.write1(stream)\n for atom in self.body:\n atom.write(stream)\n def calsize(self):\n self.size = 8 + sum([atom.calsize() for atom in self.body])\n return self.size\n def get1(self, k):\n for a in self.body:\n if a.type == k:\n return a\n else:\n raise Exception('atom not found: ' + k)\n def get(self, *keys):\n atom = self\n for k in keys:\n atom = atom.get1(k)\n return atom\n def get_all(self, k):\n return list(filter(lambda x: x.type == k, self.body))\n\nclass VariableAtom(Atom):\n def __init__(self, type, size, body, variables):\n assert isinstance(body, bytes)\n Atom.__init__(self, type, size, body)\n self.variables = variables\n def write(self, stream):\n self.write1(stream)\n i = 0\n n = 0\n for name, offset, value, bsize in self.variables:\n stream.write(self.body[i:offset])\n if bsize == 4:\n write_uint(stream, value)\n elif bsize == 8:\n write_ulong(stream, value)\n else:\n raise NotImplementedError()\n n += offset - i + bsize\n i = offset + bsize\n stream.write(self.body[i:])\n n += len(self.body) - i\n assert n == len(self.body)\n def get(self, k):\n for v in self.variables:\n if v[0] == k:\n return v[2]\n else:\n raise Exception('field not found: ' + k)\n def set(self, k, v):\n for i in range(len(self.variables)):\n variable = self.variables[i]\n if variable[0] == k:\n self.variables[i] = (k, variable[1], v, variable[3])\n break\n else:\n raise Exception('field not found: '+k)\n\ndef read_raw(stream, size, left, type):\n assert size == left + 8\n body = stream.read(left)\n return Atom(type, size, body)\n\ndef read_udta(stream, size, left, type):\n assert size == left + 8\n body = stream.read(left)\n class Udta(Atom):\n def write(self, stream):\n return\n def calsize(self):\n return 0\n return Udta(type, size, body)\n\ndef read_body_stream(stream, left):\n body = stream.read(left)\n assert len(body) == left\n return body, BytesIO(body)\n\ndef read_full_atom(stream):\n value = read_uint(stream)\n version = value >> 24\n flags = value & 0xffffff\n assert version == 0\n return value\n\ndef read_full_atom2(stream):\n value = read_uint(stream)\n version = value >> 24\n flags = value & 0xffffff\n return version, value\n\ndef read_mvhd(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n value = read_full_atom(stream)\n left -= 4\n \n # new Date(movieTime * 1000 - 2082850791998L); \n creation_time = read_uint(stream)\n modification_time = read_uint(stream)\n time_scale = read_uint(stream)\n duration = read_uint(stream)\n left -= 16\n \n qt_preferred_fate = read_uint(stream)\n qt_preferred_volume = read_ushort(stream)\n assert stream.read(10) == b'\\x00' * 10\n qt_matrixA = read_uint(stream)\n qt_matrixB = read_uint(stream)\n qt_matrixU = read_uint(stream)\n qt_matrixC = read_uint(stream)\n qt_matrixD = read_uint(stream)\n qt_matrixV = read_uint(stream)\n qt_matrixX = read_uint(stream)\n qt_matrixY = read_uint(stream)\n qt_matrixW = read_uint(stream)\n qt_previewTime = read_uint(stream)\n qt_previewDuration = read_uint(stream)\n qt_posterTime = read_uint(stream)\n qt_selectionTime = read_uint(stream)\n qt_selectionDuration = read_uint(stream)\n qt_currentTime = read_uint(stream)\n nextTrackID = read_uint(stream)\n left -= 80\n assert left == 0\n return VariableAtom(b'mvhd', size, body, [('duration', 16, duration, 4)])\n\ndef read_tkhd(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n value = read_full_atom(stream)\n left -= 4\n \n # new Date(movieTime * 1000 - 2082850791998L); \n creation_time = read_uint(stream)\n modification_time = read_uint(stream)\n track_id = read_uint(stream)\n assert stream.read(4) == b'\\x00' * 4\n duration = read_uint(stream)\n left -= 20\n \n assert stream.read(8) == b'\\x00' * 8\n qt_layer = read_ushort(stream)\n qt_alternate_group = read_ushort(stream)\n qt_volume = read_ushort(stream)\n assert stream.read(2) == b'\\x00\\x00'\n qt_matrixA = read_uint(stream)\n qt_matrixB = read_uint(stream)\n qt_matrixU = read_uint(stream)\n qt_matrixC = read_uint(stream)\n qt_matrixD = read_uint(stream)\n qt_matrixV = read_uint(stream)\n qt_matrixX = read_uint(stream)\n qt_matrixY = read_uint(stream)\n qt_matrixW = read_uint(stream)\n qt_track_width = read_uint(stream)\n width = qt_track_width >> 16\n qt_track_height = read_uint(stream)\n height = qt_track_height >> 16\n left -= 60\n assert left == 0\n return VariableAtom(b'tkhd', size, body, [('duration', 20, duration, 4)])\n\ndef read_mdhd(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n ver, value = read_full_atom2(stream)\n left -= 4\n\n if ver == 1:\n creation_time = read_ulong(stream)\n modification_time = read_ulong(stream)\n time_scale = read_uint(stream)\n duration = read_ulong(stream)\n var = [('duration', 24, duration, 8)]\n left -= 28\n else: \n assert ver == 0, \"ver=%d\" % ver\n creation_time = read_uint(stream)\n modification_time = read_uint(stream)\n time_scale = read_uint(stream)\n duration = read_uint(stream)\n var = [('duration', 16, duration, 4)]\n left -= 16\n \n packed_language = read_ushort(stream)\n qt_quality = read_ushort(stream)\n left -= 4\n \n assert left == 0\n return VariableAtom(b'mdhd', size, body, var)\n\ndef read_hdlr(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n value = read_full_atom(stream)\n left -= 4\n \n qt_component_type = read_uint(stream)\n handler_type = read_uint(stream)\n qt_component_manufacturer = read_uint(stream)\n qt_component_flags = read_uint(stream)\n qt_component_flags_mask = read_uint(stream)\n left -= 20\n \n track_name = stream.read(left)\n #assert track_name[-1] == b'\\x00'\n \n return Atom(b'hdlr', size, body)\n\ndef read_vmhd(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n value = read_full_atom(stream)\n left -= 4\n \n assert left == 8\n graphic_mode = read_ushort(stream)\n op_color_read = read_ushort(stream)\n op_color_green = read_ushort(stream)\n op_color_blue = read_ushort(stream)\n \n return Atom(b'vmhd', size, body)\n\ndef read_stsd(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n left -= 4\n \n children = []\n for i in range(entry_count):\n atom = read_atom(stream)\n children.append(atom)\n left -= atom.size\n \n assert left == 0\n #return Atom('stsd', size, children)\n class stsd_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for atom in self.body[1]:\n atom.write(stream)\n def calsize(self):\n oldsize = self.size # TODO: remove\n self.size = 8 + 4 + 4 + sum([atom.calsize() for atom in self.body[1]])\n assert oldsize == self.size, '%s: %d, %d' % (self.type, oldsize, self.size) # TODO: remove\n return self.size\n return stsd_atom(b'stsd', size, (value, children))\n\ndef read_avc1(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n \n skip_zeros(stream, 6)\n data_reference_index = read_ushort(stream)\n skip_zeros(stream, 2)\n skip_zeros(stream, 2)\n skip_zeros(stream, 12)\n width = read_ushort(stream)\n height = read_ushort(stream)\n horizontal_rez = read_uint(stream) >> 16\n vertical_rez = read_uint(stream) >> 16\n assert stream.read(4) == b'\\x00' * 4\n frame_count = read_ushort(stream)\n string_len = read_byte(stream)\n compressor_name = stream.read(31)\n depth = read_ushort(stream)\n assert stream.read(2) == b'\\xff\\xff'\n left -= 78\n \n child = read_atom(stream)\n assert child.type in (b'avcC', b'pasp'), 'if the sub atom is not avcC or pasp (actual %s), you should not cache raw body' % child.type\n left -= child.size\n stream.read(left) # XXX\n return Atom(b'avc1', size, body)\n\ndef read_avcC(stream, size, left, type):\n stream.read(left)\n return Atom(b'avcC', size, None)\n\ndef read_stts(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n #assert entry_count == 1\n left -= 4\n \n samples = []\n for i in range(entry_count):\n sample_count = read_uint(stream)\n sample_duration = read_uint(stream)\n samples.append((sample_count, sample_duration))\n left -= 8\n\n assert left == 0\n #return Atom('stts', size, None)\n class stts_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for sample_count, sample_duration in self.body[1]:\n write_uint(stream, sample_count)\n write_uint(stream, sample_duration)\n def calsize(self):\n #oldsize = self.size # TODO: remove\n self.size = 8 + 4 + 4 + len(self.body[1]) * 8\n #assert oldsize == self.size, '%s: %d, %d' % (self.type, oldsize, self.size) # TODO: remove\n return self.size\n return stts_atom(b'stts', size, (value, samples))\n\ndef read_stss(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n left -= 4\n \n samples = []\n for i in range(entry_count):\n sample = read_uint(stream)\n samples.append(sample)\n left -= 4\n \n assert left == 0\n #return Atom('stss', size, None)\n class stss_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for sample in self.body[1]:\n write_uint(stream, sample)\n def calsize(self):\n self.size = 8 + 4 + 4 + len(self.body[1]) * 4\n return self.size\n return stss_atom(b'stss', size, (value, samples))\n\ndef read_stsc(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n left -= 4\n \n chunks = []\n for i in range(entry_count):\n first_chunk = read_uint(stream)\n samples_per_chunk = read_uint(stream)\n sample_description_index = read_uint(stream)\n assert sample_description_index == 1 # what is it?\n chunks.append((first_chunk, samples_per_chunk, sample_description_index))\n left -= 12\n #chunks, samples = zip(*chunks)\n #total = 0\n #for c, s in zip(chunks[1:], samples):\n #\ttotal += c*s\n #print 'total', total\n \n assert left == 0\n #return Atom('stsc', size, None)\n class stsc_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for first_chunk, samples_per_chunk, sample_description_index in self.body[1]:\n write_uint(stream, first_chunk)\n write_uint(stream, samples_per_chunk)\n write_uint(stream, sample_description_index)\n def calsize(self):\n self.size = 8 + 4 + 4 + len(self.body[1]) * 12\n return self.size\n return stsc_atom(b'stsc', size, (value, chunks))\n\ndef read_stsz(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n sample_size = read_uint(stream)\n sample_count = read_uint(stream)\n left -= 8\n \n assert sample_size == 0\n total = 0\n sizes = []\n if sample_size == 0:\n for i in range(sample_count):\n entry_size = read_uint(stream)\n sizes.append(entry_size)\n total += entry_size\n left -= 4\n \n assert left == 0\n #return Atom('stsz', size, None)\n class stsz_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, self.body[1])\n write_uint(stream, self.body[2])\n for entry_size in self.body[3]:\n write_uint(stream, entry_size)\n def calsize(self):\n self.size = 8 + 4 + 8 + len(self.body[3]) * 4\n return self.size\n return stsz_atom(b'stsz', size, (value, sample_size, sample_count, sizes))\n\ndef read_stco(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n left -= 4\n \n offsets = []\n for i in range(entry_count):\n chunk_offset = read_uint(stream)\n offsets.append(chunk_offset)\n left -= 4\n \n assert left == 0\n #return Atom('stco', size, None)\n class stco_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for chunk_offset in self.body[1]:\n write_uint(stream, chunk_offset)\n def calsize(self):\n self.size = 8 + 4 + 4 + len(self.body[1]) * 4\n return self.size\n return stco_atom(b'stco', size, (value, offsets))\n\ndef read_ctts(stream, size, left, type):\n value = read_full_atom(stream)\n left -= 4\n \n entry_count = read_uint(stream)\n left -= 4\n \n samples = []\n for i in range(entry_count):\n sample_count = read_uint(stream)\n sample_offset = read_uint(stream)\n samples.append((sample_count, sample_offset))\n left -= 8\n \n assert left == 0\n class ctts_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n write_uint(stream, self.body[0])\n write_uint(stream, len(self.body[1]))\n for sample_count, sample_offset in self.body[1]:\n write_uint(stream, sample_count)\n write_uint(stream, sample_offset)\n def calsize(self):\n self.size = 8 + 4 + 4 + len(self.body[1]) * 8\n return self.size\n return ctts_atom(b'ctts', size, (value, samples))\n\ndef read_smhd(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n value = read_full_atom(stream)\n left -= 4\n \n balance = read_ushort(stream)\n assert stream.read(2) == b'\\x00\\x00'\n left -= 4\n \n assert left == 0\n return Atom(b'smhd', size, body)\n\ndef read_mp4a(stream, size, left, type):\n body, stream = read_body_stream(stream, left)\n \n assert stream.read(6) == b'\\x00' * 6\n data_reference_index = read_ushort(stream)\n assert stream.read(8) == b'\\x00' * 8\n channel_count = read_ushort(stream)\n sample_size = read_ushort(stream)\n assert stream.read(4) == b'\\x00' * 4\n time_scale = read_ushort(stream)\n assert stream.read(2) == b'\\x00' * 2\n left -= 28\n \n atom = read_atom(stream)\n assert atom.type == b'esds'\n left -= atom.size\n \n assert left == 0\n return Atom(b'mp4a', size, body)\n\ndef read_descriptor(stream):\n tag = read_byte(stream)\n raise NotImplementedError()\n\ndef read_esds(stream, size, left, type):\n value = read_uint(stream)\n version = value >> 24\n assert version == 0\n flags = value & 0xffffff\n left -= 4\n \n body = stream.read(left)\n return Atom(b'esds', size, None)\n\ndef read_composite_atom(stream, size, left, type):\n children = []\n while left > 0:\n atom = read_atom(stream)\n children.append(atom)\n left -= atom.size\n assert left == 0, left\n return CompositeAtom(type, size, children)\n\ndef read_mdat(stream, size, left, type):\n source_start = stream.tell()\n source_size = left\n skip(stream, left)\n #return Atom(type, size, None)\n #raise NotImplementedError()\n class mdat_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n self.write2(stream)\n def write2(self, stream):\n source, source_start, source_size = self.body\n original = source.tell()\n source.seek(source_start)\n copy_stream(source, stream, source_size)\n def calsize(self):\n return self.size\n return mdat_atom(b'mdat', size, (stream, source_start, source_size))\n\natom_readers = {\n b'mvhd': read_mvhd, # merge duration\n b'tkhd': read_tkhd, # merge duration\n b'mdhd': read_mdhd, # merge duration\n b'hdlr': read_hdlr, # nothing\n b'vmhd': read_vmhd, # nothing\n b'stsd': read_stsd, # nothing\n b'avc1': read_avc1, # nothing\n b'avcC': read_avcC, # nothing\n b'stts': read_stts, # sample_count, sample_duration\n b'stss': read_stss, # join indexes\n b'stsc': read_stsc, # merge # sample numbers\n b'stsz': read_stsz, # merge # samples\n b'stco': read_stco, # merge # chunk offsets\n b'ctts': read_ctts, # merge\n b'smhd': read_smhd, # nothing\n b'mp4a': read_mp4a, # nothing\n b'esds': read_esds, # noting\n \n b'ftyp': read_raw,\n b'yqoo': read_raw,\n b'moov': read_composite_atom,\n b'trak': read_composite_atom,\n b'mdia': read_composite_atom,\n b'minf': read_composite_atom,\n b'dinf': read_composite_atom,\n b'stbl': read_composite_atom,\n b'iods': read_raw,\n b'dref': read_raw,\n b'free': read_raw,\n b'edts': read_raw,\n b'pasp': read_raw,\n\n b'mdat': read_mdat,\n b'udta': read_udta,\n}\n#stsd sample descriptions (codec types, initialization etc.) \n#stts (decoding) time-to-sample \n#ctts (composition) time to sample \n#stsc sample-to-chunk, partial data-offset information \n#stsz sample sizes (framing) \n#stz2 compact sample sizes (framing) \n#stco chunk offset, partial data-offset information \n#co64 64-bit chunk offset \n#stss sync sample table (random access points) \n#stsh shadow sync sample table \n#padb sample padding bits \n#stdp sample degradation priority \n#sdtp independent and disposable samples \n#sbgp sample-to-group \n#sgpd sample group description \n#subs sub-sample information\n\n\ndef read_atom(stream):\n header = stream.read(8)\n if not header:\n return\n assert len(header) == 8\n n = 0\n size = struct.unpack('>I', header[:4])[0]\n assert size > 0\n n += 4\n type = header[4:8]\n n += 4\n assert type != b'uuid'\n if size == 1:\n size = read_ulong(stream)\n n += 8\n \n left = size - n\n if type in atom_readers:\n return atom_readers[type](stream, size, left, type)\n raise NotImplementedError('%s: %d' % (type, left))\n\ndef write_atom(stream, atom):\n atom.write(stream)\n\ndef parse_atoms(stream):\n atoms = []\n while True:\n atom = read_atom(stream)\n if atom:\n atoms.append(atom)\n else:\n break\n return atoms\n\ndef read_mp4(stream):\n print(stream.name)\n atoms = parse_atoms(stream)\n moov = list(filter(lambda x: x.type == b'moov', atoms))\n mdat = list(filter(lambda x: x.type == b'mdat', atoms))\n assert len(moov) == 1\n assert len(mdat) == 1\n moov = moov[0]\n mdat = mdat[0]\n return atoms, moov, mdat\n\n##################################################\n# merge\n##################################################\n\ndef merge_stts(samples_list):\n sample_list = []\n for samples in samples_list:\n #assert len(samples) == 1\n #sample_list.append(samples[0])\n sample_list += samples\n counts, durations = zip(*sample_list)\n #assert len(set(durations)) == 1, 'not all durations equal'\n if len(set(durations)) == 1:\n return [(sum(counts), durations[0])]\n return sample_list\n\ndef merge_stss(samples, sample_number_list):\n results = []\n start = 0\n for samples, sample_number_list in zip(samples, sample_number_list):\n results.extend(map(lambda x: start + x, samples))\n start += sample_number_list\n return results\n\ndef merge_stsc(chunks_list, total_chunk_number_list):\n results = []\n chunk_index = 1\n for chunks, total in zip(chunks_list, total_chunk_number_list):\n for i in range(len(chunks)):\n if i < len(chunks) - 1:\n chunk_number = chunks[i + 1][0] - chunks[i][0]\n else:\n chunk_number = total + 1 - chunks[i][0]\n sample_number = chunks[i][1]\n description = chunks[i][2]\n results.append((chunk_index, sample_number, description))\n chunk_index += chunk_number\n return results\n\ndef merge_stco(offsets_list, mdats):\n offset = 0\n results = []\n for offsets, mdat in zip(offsets_list, mdats):\n results.extend(offset + x - mdat.body[1] for x in offsets)\n offset += mdat.size - 8\n return results\n\ndef merge_stsz(sizes_list):\n return sum(sizes_list, [])\n\ndef merge_mdats(mdats):\n total_size = sum(x.size - 8 for x in mdats) + 8\n class multi_mdat_atom(Atom):\n def __init__(self, type, size, body):\n Atom.__init__(self, type, size, body)\n def write(self, stream):\n self.write1(stream)\n self.write2(stream)\n def write2(self, stream):\n for mdat in self.body:\n mdat.write2(stream)\n def calsize(self):\n return self.size\n return multi_mdat_atom(b'mdat', total_size, mdats)\n\ndef merge_moov(moovs, mdats):\n mvhd_duration = 0\n for x in moovs:\n mvhd_duration += x.get(b'mvhd').get('duration')\n tkhd_durations = [0, 0]\n mdhd_durations = [0, 0]\n for x in moovs:\n traks = x.get_all(b'trak')\n assert len(traks) == 2\n tkhd_durations[0] += traks[0].get(b'tkhd').get('duration')\n tkhd_durations[1] += traks[1].get(b'tkhd').get('duration')\n mdhd_durations[0] += traks[0].get(b'mdia', b'mdhd').get('duration')\n mdhd_durations[1] += traks[1].get(b'mdia', b'mdhd').get('duration')\n #mvhd_duration = min(mvhd_duration, tkhd_durations)\n \n trak0s = [x.get_all(b'trak')[0] for x in moovs]\n trak1s = [x.get_all(b'trak')[1] for x in moovs]\n \n stts0 = merge_stts(x.get(b'mdia', b'minf', b'stbl', b'stts').body[1] for x in trak0s)\n stts1 = merge_stts(x.get(b'mdia', b'minf', b'stbl', b'stts').body[1] for x in trak1s)\n \n stss = merge_stss((x.get(b'mdia', b'minf', b'stbl', b'stss').body[1] for x in trak0s), (len(x.get(b'mdia', b'minf', b'stbl', b'stsz').body[3]) for x in trak0s))\n \n stsc0 = merge_stsc((x.get(b'mdia', b'minf', b'stbl', b'stsc').body[1] for x in trak0s), (len(x.get(b'mdia', b'minf', b'stbl', b'stco').body[1]) for x in trak0s))\n stsc1 = merge_stsc((x.get(b'mdia', b'minf', b'stbl', b'stsc').body[1] for x in trak1s), (len(x.get(b'mdia', b'minf', b'stbl', b'stco').body[1]) for x in trak1s))\n \n stco0 = merge_stco((x.get(b'mdia', b'minf', b'stbl', b'stco').body[1] for x in trak0s), mdats)\n stco1 = merge_stco((x.get(b'mdia', b'minf', b'stbl', b'stco').body[1] for x in trak1s), mdats)\n \n stsz0 = merge_stsz((x.get(b'mdia', b'minf', b'stbl', b'stsz').body[3] for x in trak0s))\n stsz1 = merge_stsz((x.get(b'mdia', b'minf', b'stbl', b'stsz').body[3] for x in trak1s))\n \n ctts = sum((x.get(b'mdia', b'minf', b'stbl', b'ctts').body[1] for x in trak0s), [])\n \n moov = moovs[0]\n \n moov.get(b'mvhd').set('duration', mvhd_duration)\n trak0 = moov.get_all(b'trak')[0]\n trak1 = moov.get_all(b'trak')[1]\n trak0.get(b'tkhd').set('duration', tkhd_durations[0])\n trak1.get(b'tkhd').set('duration', tkhd_durations[1])\n trak0.get(b'mdia', b'mdhd').set('duration', mdhd_durations[0])\n trak1.get(b'mdia', b'mdhd').set('duration', mdhd_durations[1])\n \n stts_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stts')\n stts_atom.body = stts_atom.body[0], stts0\n stts_atom = trak1.get(b'mdia', b'minf', b'stbl', b'stts')\n stts_atom.body = stts_atom.body[0], stts1\n \n stss_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stss')\n stss_atom.body = stss_atom.body[0], stss\n \n stsc_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stsc')\n stsc_atom.body = stsc_atom.body[0], stsc0\n stsc_atom = trak1.get(b'mdia', b'minf', b'stbl', b'stsc')\n stsc_atom.body = stsc_atom.body[0], stsc1\n \n stco_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stco')\n stco_atom.body = stss_atom.body[0], stco0\n stco_atom = trak1.get(b'mdia', b'minf', b'stbl', b'stco')\n stco_atom.body = stss_atom.body[0], stco1\n \n stsz_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stsz')\n stsz_atom.body = stsz_atom.body[0], stsz_atom.body[1], len(stsz0), stsz0\n stsz_atom = trak1.get(b'mdia', b'minf', b'stbl', b'stsz')\n stsz_atom.body = stsz_atom.body[0], stsz_atom.body[1], len(stsz1), stsz1\n \n ctts_atom = trak0.get(b'mdia', b'minf', b'stbl', b'ctts')\n ctts_atom.body = ctts_atom.body[0], ctts\n \n old_moov_size = moov.size\n new_moov_size = moov.calsize()\n new_mdat_start = mdats[0].body[1] + new_moov_size - old_moov_size\n stco0 = list(map(lambda x: x + new_mdat_start, stco0))\n stco1 = list(map(lambda x: x + new_mdat_start, stco1))\n stco_atom = trak0.get(b'mdia', b'minf', b'stbl', b'stco')\n stco_atom.body = stss_atom.body[0], stco0\n stco_atom = trak1.get(b'mdia', b'minf', b'stbl', b'stco')\n stco_atom.body = stss_atom.body[0], stco1\n \n return moov\n\ndef merge_mp4s(files, output):\n assert files\n ins = [open(mp4, 'rb') for mp4 in files]\n mp4s = list(map(read_mp4, ins))\n moovs = list(map(lambda x: x[1], mp4s))\n mdats = list(map(lambda x: x[2], mp4s))\n moov = merge_moov(moovs, mdats)\n mdat = merge_mdats(mdats)\n with open(output, 'wb') as output:\n for x in mp4s[0][0]:\n if x.type == b'moov':\n moov.write(output)\n elif x.type == b'mdat':\n mdat.write(output)\n else:\n x.write(output)\n\n##################################################\n# main\n##################################################\n\n# TODO: FIXME: duplicate of join_flv\n\ndef guess_output(inputs):\n import os.path\n inputs = map(os.path.basename, inputs)\n n = min(map(len, inputs))\n for i in reversed(range(1, n)):\n if len(set(s[:i] for s in inputs)) == 1:\n return inputs[0][:i] + '.mp4'\n return 'output.mp4'\n\ndef concat_mp4(mp4s, output = None):\n assert mp4s, 'no mp4 file found'\n import os.path\n if not output:\n output = guess_output(mp4s)\n elif os.path.isdir(output):\n output = os.path.join(output, guess_output(mp4s))\n \n print('Merging video parts...')\n merge_mp4s(mp4s, output)\n \n return output\n\ndef usage():\n print('Usage: [python3] join_mp4.py --output TARGET.mp4 mp4...')\n\ndef main():\n import sys, getopt\n try:\n opts, args = getopt.getopt(sys.argv[1:], \"ho:\", [\"help\", \"output=\"])\n except getopt.GetoptError as err:\n usage()\n sys.exit(1)\n output = None\n for o, a in opts:\n if o in (\"-h\", \"--help\"):\n usage()\n sys.exit()\n elif o in (\"-o\", \"--output\"):\n output = a\n else:\n usage()\n sys.exit(1)\n if not args:\n usage()\n sys.exit(1)\n \n concat_mp4(args, output)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/you_get/processor/join_ts.py", "content": "#!/usr/bin/env python\n\nimport struct\nfrom io import BytesIO\n\n##################################################\n# main\n##################################################\n\ndef guess_output(inputs):\n import os.path\n inputs = map(os.path.basename, inputs)\n n = min(map(len, inputs))\n for i in reversed(range(1, n)):\n if len(set(s[:i] for s in inputs)) == 1:\n return inputs[0][:i] + '.ts'\n return 'output.ts'\n\ndef concat_ts(ts_parts, output = None):\n assert ts_parts, 'no ts files found'\n import os.path\n if not output:\n output = guess_output(ts_parts)\n elif os.path.isdir(output):\n output = os.path.join(output, guess_output(ts_parts))\n \n print('Merging video parts...')\n \n ts_out_file = open(output, \"wb\")\n for ts_in in ts_parts:\n ts_in_file = open(ts_in, \"rb\")\n ts_in_data = ts_in_file.read()\n ts_in_file.close()\n ts_out_file.write(ts_in_data)\n ts_out_file.close()\n return output\n\ndef usage():\n print('Usage: [python3] join_ts.py --output TARGET.ts ts...')\n\ndef main():\n import sys, getopt\n try:\n opts, args = getopt.getopt(sys.argv[1:], \"ho:\", [\"help\", \"output=\"])\n except getopt.GetoptError as err:\n usage()\n sys.exit(1)\n output = None\n for o, a in opts:\n if o in (\"-h\", \"--help\"):\n usage()\n sys.exit()\n elif o in (\"-o\", \"--output\"):\n output = a\n else:\n usage()\n sys.exit(1)\n if not args:\n usage()\n sys.exit(1)\n \n concat_ts(args, output)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "src/you_get/processor/rtmpdump.py", "content": "#!/usr/bin/env python\n\nimport os.path\nimport subprocess\n\ndef get_usable_rtmpdump(cmd):\n try:\n p = subprocess.Popen([cmd], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n out, err = p.communicate()\n return cmd\n except:\n return None\n\nRTMPDUMP = get_usable_rtmpdump('rtmpdump')\n\ndef has_rtmpdump_installed():\n return RTMPDUMP is not None\n\n#\n#params ={\"-y\":\"playlist\",\"-q\":None,}\n#if Only Key ,Value should be None\n#-r -o should not be included in params\n\ndef download_rtmpdump_stream(url, title, ext,params={},output_dir='.'):\n filename = '%s.%s' % (title, ext)\n filepath = os.path.join(output_dir, filename)\n\n cmdline = [RTMPDUMP, '-r']\n cmdline.append(url)\n cmdline.append('-o')\n cmdline.append(filepath)\n\n for key in params.keys():\n cmdline.append(key)\n if params[key]!=None:\n cmdline.append(params[key])\n\n # cmdline.append('-y')\n # cmdline.append(playpath)\n print(\"Call rtmpdump:\\n\"+\" \".join(cmdline)+\"\\n\")\n subprocess.call(cmdline)\n return\n\n#\ndef play_rtmpdump_stream(player, url, params={}):\n \n #construct left side of pipe\n cmdline = [RTMPDUMP, '-r']\n cmdline.append(url)\n \n #append other params if exist\n for key in params.keys():\n cmdline.append(key)\n if params[key]!=None:\n cmdline.append(params[key])\n\n cmdline.append('-o')\n cmdline.append('-')\n\n #pipe start\n cmdline.append('|')\n cmdline.append(player)\n cmdline.append('-')\n\n #logging\n print(\"Call rtmpdump:\\n\"+\" \".join(cmdline)+\"\\n\")\n\n #call RTMPDump!\n subprocess.call(cmdline)\n \n # os.system(\"rtmpdump -r '%s' -y '%s' -o - | %s -\" % (url, playpath, player))\n return\n" }, { "path": "src/you_get/util/fs.py", "content": "#!/usr/bin/env python\n\nfrom .os import detect_os\n\ndef legitimize(text, os=detect_os()):\n \"\"\"Converts a string to a valid filename.\n \"\"\"\n\n # POSIX systems\n text = text.translate({\n 0: None,\n ord('/'): '-',\n ord('|'): '-',\n })\n\n # FIXME: do some filesystem detection\n if os == 'windows' or os == 'cygwin' or os == 'wsl':\n # Windows (non-POSIX namespace)\n text = text.translate({\n # Reserved in Windows VFAT and NTFS\n ord(':'): '-',\n ord('*'): '-',\n ord('?'): '-',\n ord('\\\\'): '-',\n ord('\\\"'): '\\'',\n # Reserved in Windows VFAT\n ord('+'): '-',\n ord('<'): '-',\n ord('>'): '-',\n ord('['): '(',\n ord(']'): ')',\n ord('\\t'): ' ',\n })\n else:\n # *nix\n if os == 'mac':\n # Mac OS HFS+\n text = text.translate({\n ord(':'): '-',\n })\n\n # Remove leading .\n if text.startswith(\".\"):\n text = text[1:]\n\n text = text[:80] # Trim to 82 Unicode characters long\n return text\n" }, { "path": "src/you_get/util/git.py", "content": "#!/usr/bin/env python\n\nimport os\nimport subprocess\nfrom ..version import __version__\n\ndef get_head(repo_path):\n \"\"\"Get (branch, commit) from HEAD of a git repo.\"\"\"\n try:\n ref = open(os.path.join(repo_path, '.git', 'HEAD'), 'r').read().strip()[5:].split('/')\n branch = ref[-1]\n commit = open(os.path.join(repo_path, '.git', *ref), 'r').read().strip()[:7]\n return branch, commit\n except:\n return None\n\ndef get_version(repo_path):\n try:\n version = __version__.split('.')\n major, minor, cn = [int(i) for i in version]\n p = subprocess.Popen(['git',\n '--git-dir', os.path.join(repo_path, '.git'),\n '--work-tree', repo_path,\n 'rev-list', 'HEAD', '--count'],\n stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n raw, err = p.communicate()\n c_head = int(raw.decode('ascii'))\n q = subprocess.Popen(['git',\n '--git-dir', os.path.join(repo_path, '.git'),\n '--work-tree', repo_path,\n 'rev-list', 'master', '--count'],\n stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n raw, err = q.communicate()\n c_master = int(raw.decode('ascii'))\n cc = c_head - c_master\n assert cc\n return '%s.%s.%s' % (major, minor, cn + cc)\n except:\n return __version__\n" }, { "path": "src/you_get/util/log.py", "content": "#!/usr/bin/env python\n# This file is Python 2 compliant.\n\nfrom ..version import script_name\n\nimport os, sys\n\nTERM = os.getenv('TERM', '')\nIS_ANSI_TERMINAL = TERM in (\n 'eterm-color',\n 'linux',\n 'screen',\n 'vt100',\n) or TERM.startswith('xterm')\n\n# ANSI escape code\n# See \nRESET = 0\nBOLD = 1\nUNDERLINE = 4\nNEGATIVE = 7\nNO_BOLD = 21\nNO_UNDERLINE = 24\nPOSITIVE = 27\nBLACK = 30\nRED = 31\nGREEN = 32\nYELLOW = 33\nBLUE = 34\nMAGENTA = 35\nCYAN = 36\nLIGHT_GRAY = 37\nDEFAULT = 39\nBLACK_BACKGROUND = 40\nRED_BACKGROUND = 41\nGREEN_BACKGROUND = 42\nYELLOW_BACKGROUND = 43\nBLUE_BACKGROUND = 44\nMAGENTA_BACKGROUND = 45\nCYAN_BACKGROUND = 46\nLIGHT_GRAY_BACKGROUND = 47\nDEFAULT_BACKGROUND = 49\nDARK_GRAY = 90 # xterm\nLIGHT_RED = 91 # xterm\nLIGHT_GREEN = 92 # xterm\nLIGHT_YELLOW = 93 # xterm\nLIGHT_BLUE = 94 # xterm\nLIGHT_MAGENTA = 95 # xterm\nLIGHT_CYAN = 96 # xterm\nWHITE = 97 # xterm\nDARK_GRAY_BACKGROUND = 100 # xterm\nLIGHT_RED_BACKGROUND = 101 # xterm\nLIGHT_GREEN_BACKGROUND = 102 # xterm\nLIGHT_YELLOW_BACKGROUND = 103 # xterm\nLIGHT_BLUE_BACKGROUND = 104 # xterm\nLIGHT_MAGENTA_BACKGROUND = 105 # xterm\nLIGHT_CYAN_BACKGROUND = 106 # xterm\nWHITE_BACKGROUND = 107 # xterm\n\ndef sprint(text, *colors):\n \"\"\"Format text with color or other effects into ANSI escaped string.\"\"\"\n return \"\\33[{}m{content}\\33[{}m\".format(\";\".join([str(color) for color in colors]), RESET, content=text) if IS_ANSI_TERMINAL and colors else text\n\ndef println(text, *colors):\n \"\"\"Print text to standard output.\"\"\"\n sys.stdout.write(sprint(text, *colors) + \"\\n\")\n\ndef print_err(text, *colors):\n \"\"\"Print text to standard error.\"\"\"\n sys.stderr.write(sprint(text, *colors) + \"\\n\")\n\ndef print_log(text, *colors):\n \"\"\"Print a log message to standard error.\"\"\"\n sys.stderr.write(sprint(\"{}: {}\".format(script_name, text), *colors) + \"\\n\")\n\ndef i(message):\n \"\"\"Print a normal log message.\"\"\"\n print_log(message)\n\ndef d(message):\n \"\"\"Print a debug log message.\"\"\"\n print_log(message, BLUE)\n\ndef w(message):\n \"\"\"Print a warning log message.\"\"\"\n print_log(message, YELLOW)\n\ndef e(message, exit_code=None):\n \"\"\"Print an error log message.\"\"\"\n print_log(message, YELLOW, BOLD)\n if exit_code is not None:\n sys.exit(exit_code)\n\ndef wtf(message, exit_code=1):\n \"\"\"What a Terrible Failure!\"\"\"\n print_log(message, RED, BOLD)\n if exit_code is not None:\n sys.exit(exit_code)\n\ndef yes_or_no(message):\n ans = str(input('%s (y/N) ' % message)).lower().strip()\n return ans == 'y'\n" }, { "path": "src/you_get/util/os.py", "content": "#!/usr/bin/env python\n\nfrom platform import system\n\ndef detect_os():\n \"\"\"Detect operating system.\n \"\"\"\n\n # Inspired by:\n # https://github.com/scivision/pybashutils/blob/78b7f2b339cb03b1c37df94015098bbe462f8526/pybashutils/windows_linux_detect.py\n\n syst = system().lower()\n os = 'unknown'\n\n if 'cygwin' in syst:\n os = 'cygwin'\n elif 'darwin' in syst:\n os = 'mac'\n elif 'linux' in syst:\n os = 'linux'\n # detect WSL https://github.com/Microsoft/BashOnWindows/issues/423\n try:\n with open('/proc/version', 'r') as f:\n if 'microsoft' in f.read().lower():\n os = 'wsl'\n except: pass\n elif 'windows' in syst:\n os = 'windows'\n elif 'bsd' in syst:\n os = 'bsd'\n\n return os\n" }, { "path": "src/you_get/util/strings.py", "content": "try:\n # py 3.4\n from html import unescape as unescape_html\nexcept ImportError:\n import re\n from html.entities import entitydefs\n\n def unescape_html(string):\n '''HTML entity decode'''\n string = re.sub(r'&#[^;]+;', _sharp2uni, string)\n string = re.sub(r'&[^;]+;', lambda m: entitydefs[m.group(0)[1:-1]], string)\n return string\n\n def _sharp2uni(m):\n '''&#...; ==> unicode'''\n s = m.group(0)[2:].rstrip(';；')\n if s.startswith('x'):\n return chr(int('0'+s, 16))\n else:\n return chr(int(s))\n\nfrom .fs import legitimize\n\ndef get_filename(htmlstring):\n return legitimize(unescape_html(htmlstring))\n\ndef parameterize(string):\n return \"'%s'\" % string.replace(\"'\", r\"'\\''\")\n" }, { "path": "src/you_get/util/term.py", "content": "#!/usr/bin/env python\n\ndef get_terminal_size():\n \"\"\"Get (width, height) of the current terminal.\"\"\"\n try:\n import fcntl, termios, struct # fcntl module only available on Unix\n return struct.unpack('hh', fcntl.ioctl(1, termios.TIOCGWINSZ, '1234'))\n except:\n return (40, 80)\n" }, { "path": "src/you_get/version.py", "content": "#!/usr/bin/env python\n\nscript_name = 'you-get'\n__version__ = '0.4.1743'\n" }, { "path": "tests/test.py", "content": "#!/usr/bin/env python\n\nimport unittest\n\nfrom you_get.extractors import (\n imgur,\n magisto,\n youtube,\n missevan,\n acfun,\n bilibili,\n soundcloud,\n tiktok,\n twitter,\n miaopai\n)\n\n\nclass YouGetTests(unittest.TestCase):\n def test_imgur(self):\n imgur.download('http://imgur.com/WVLk5nD', info_only=True)\n imgur.download('https://imgur.com/we-should-have-listened-WVLk5nD', info_only=True)\n\n def test_magisto(self):\n magisto.download(\n 'http://www.magisto.com/album/video/f3x9AAQORAkfDnIFDA',\n info_only=True\n )\n\n #def test_youtube(self):\n #youtube.download(\n # 'http://www.youtube.com/watch?v=pzKerr0JIPA', info_only=True\n #)\n #youtube.download('http://youtu.be/pzKerr0JIPA', info_only=True)\n #youtube.download(\n # 'http://www.youtube.com/attribution_link?u=/watch?v%3DldAKIzq7bvs%26feature%3Dshare', # noqa\n # info_only=True\n #)\n #youtube.download(\n # 'https://www.youtube.com/watch?v=oRdxUFDoQe0', info_only=True\n #)\n\n def test_acfun(self):\n acfun.download('https://www.acfun.cn/v/ac44560432', info_only=True)\n\n #def test_bilibili(self):\n #bilibili.download('https://www.bilibili.com/video/BV1sL4y177sC', info_only=True)\n\n #def test_soundcloud(self):\n ## single song\n #soundcloud.download(\n # 'https://soundcloud.com/keiny-pham/impure-bird', info_only=True\n #)\n ## playlist\n #soundcloud.download(\n # 'https://soundcloud.com/anthony-flieger/sets/cytus', info_only=True\n #)\n\n def test_tiktok(self):\n tiktok.download('https://www.tiktok.com/@zukky_48/video/7398162058153315605', info_only=True)\n tiktok.download('https://www.tiktok.com/@/video/7398162058153315605', info_only=True)\n tiktok.download('https://t.tiktok.com/i18n/share/video/7398162058153315605/', info_only=True)\n tiktok.download('https://vt.tiktok.com/ZSYKjKt6M/', info_only=True)\n\n def test_twitter(self):\n twitter.download('https://twitter.com/elonmusk/status/1530516552084234244', info_only=True)\n twitter.download('https://x.com/elonmusk/status/1530516552084234244', info_only=True)\n\n def test_weibo(self):\n miaopai.download('https://video.weibo.com/show?fid=1034:4825403706245135', info_only=True)\n\nif __name__ == '__main__':\n unittest.main()\n" }, { "path": "tests/test_common.py", "content": "#!/usr/bin/env python\n\nimport unittest\n\nfrom you_get.common import *\n\nclass TestCommon(unittest.TestCase):\n \n def test_match1(self):\n self.assertEqual(match1('http://youtu.be/1234567890A', r'youtu.be/([^/]+)'), '1234567890A')\n self.assertEqual(match1('http://youtu.be/1234567890A', r'youtu.be/([^/]+)', r'youtu.(\\w+)'), ['1234567890A', 'be'])\n" }, { "path": "tests/test_util.py", "content": "#!/usr/bin/env python\n\nimport unittest\n\nfrom you_get.util.fs import *\n\nclass TestUtil(unittest.TestCase):\n def test_legitimize(self):\n self.assertEqual(legitimize(\"1*2\", os=\"linux\"), \"1*2\")\n self.assertEqual(legitimize(\"1*2\", os=\"mac\"), \"1*2\")\n self.assertEqual(legitimize(\"1*2\", os=\"windows\"), \"1-2\")\n self.assertEqual(legitimize(\"1*2\", os=\"wsl\"), \"1-2\")\n" }, { "path": "you-get", "content": "#!/usr/bin/env python3\nimport os, sys\n\n_srcdir = '%s/src/' % os.path.dirname(os.path.realpath(__file__))\n_filepath = os.path.dirname(sys.argv[0])\nsys.path.insert(1, os.path.join(_filepath, _srcdir))\n\nif sys.version_info[0] == 3:\n import you_get\n if __name__ == '__main__':\n you_get.main(repo_path=_filepath)\nelse: # Python 2\n from you_get.util import log\n log.e(\"[fatal] Python 3 is required!\")\n log.wtf(\"try to run this script using 'python3 you-get'.\")\n" }, { "path": "you-get.plugin.zsh", "content": "#!/usr/bin/env zsh\nalias you-get=\"noglob python3 $(dirname $0)/you-get\"\nalias you-vlc=\"noglob python3 $(dirname $0)/you-get --player vlc\"\n" } ]