Repository: MichaelYochpaz/iSubRip Branch: main Commit: 6bfc3663afea Files: 38 Total size: 203.2 KB Directory structure: gitextract_mbnetjyr/ ├── .github/ │ └── ISSUE_TEMPLATE/ │ ├── 1_issue_report.yml │ ├── 2_feature_request.yml │ ├── 3_question.yml │ └── config.yml ├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── README.md ├── example-config.toml ├── isubrip/ │ ├── __init__.py │ ├── __main__.py │ ├── cli.py │ ├── commands/ │ │ ├── __init__.py │ │ └── download.py │ ├── config.py │ ├── constants.py │ ├── data_structures.py │ ├── logger.py │ ├── scrapers/ │ │ ├── __init__.py │ │ ├── appletv_scraper.py │ │ ├── itunes_scraper.py │ │ └── scraper.py │ ├── subtitle_formats/ │ │ ├── __init__.py │ │ ├── subrip.py │ │ ├── subtitles.py │ │ └── webvtt.py │ ├── ui.py │ └── utils.py ├── pyproject.toml └── tests/ ├── .gitignore ├── __init__.py ├── benchmarks/ │ ├── __init__.py │ ├── download_benchmark.py │ └── download_benchmark_module.py ├── test_utils.py └── tools/ ├── __init__.py ├── generate_mock_data.py └── mock_loader.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/ISSUE_TEMPLATE/1_issue_report.yml ================================================ name: Bug / Issue Report description: Report a bug or an issue. title: "[Issue]: " labels: [bug] body: - type: markdown attributes: value: > **Before opening an issue, please make sure you are running the latest version of iSubRip, and that there isn't an already-existing open issue for your issue under the [issues tab](https://github.com/MichaelYochpaz/iSubRip/labels/bug).** - type: checkboxes id: check-confirmation attributes: label: Confirmations options: - label: "I have checked the issues tab, and couldn't find an existing open issue for the issue I want to report." required: true - type: dropdown id: os-type attributes: label: OS Type description: The operation system that's being used to run iSubRip. options: - Windows - MacOS - Linux validations: required: true - type: input id: python-version attributes: label: Python Version description: | The Python version that's being used to run iSubRip. Can be checked by running `python --version`. placeholder: | Example: "3.10.6" validations: required: true - type: input id: version attributes: label: Package Version description: | iSubRip's version that's being used. Can be checked by running `python -m pip show isubrip`. placeholder: | Example: "2.3.2" validations: required: true - type: textarea id: description attributes: label: Description description: | A summary of the issue. Include as much information as possible, and steps to reproduce (if they're known). Log files (see README for more information) can be attached by clicking the area to highlight it, and then dragging & dropping files in. validations: required: true - type: textarea id: output-log attributes: label: Output Log description: | iSubRip's output when the issue occurred. Please include the command that was used to run iSubRip. render: Text placeholder: | Example: isubrip https://itunes.apple.com/us/movie/can-you-hear-us-now/id1617191490 Scraping https://itunes.apple.com/us/movie/can-you-hear-us-now/id1617191490... Found movie: Can You Hear Us Now? Traceback (most recent call last): File "%appdata%\local\programs\python\python38-32\lib\runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "%appdata%\local\programs\python\python38-32\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "%appdata%\local\programs\python\python38-32\scripts\isubrip.exe\__main__.py", line 7, in File "%appdata%\local\programs\python\python38-32\lib\site-packages\isubrip\__main__.py", line 91, in main os.makedirs(current_download_path, exist_ok=True) File "%appdata%\local\programs\python\python38-32\lib\os.py", line 221, in makedirs mkdir(name, mode) OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\\%appdata%\\Local\\Temp\\iSubRip\\Can.You.Hear.Us.Now?.iT.WEB' validations: required: true - type: textarea id: config attributes: label: Config description: | The iSubRip config file you are using. **Leave empty only if there is no config file in use.** render: TOML placeholder: | Example: [downloads] folder = "C:\\Subtitles\\iTunes" languages = ["en-US", "fr-FR", "he"] zip = false [subtitles] convert-to-srt = true fix-rtl = true validations: required: false ================================================ FILE: .github/ISSUE_TEMPLATE/2_feature_request.yml ================================================ name: Feature Request description: Request a new feature or improvement to an existing one. title: "[Feature Request]: " labels: [feature-request] body: - type: markdown attributes: value: > **Before opening an issue, please make sure there isn't an already-existing issue, open or closed, for this feature request under the [issues tab](https://github.com/MichaelYochpaz/iSubRip/issues?q=label%3Afeature-request).** - type: checkboxes id: check-confirmation attributes: label: Confirmations options: - label: "I have checked the issues tab, and couldn't find an existing issue with my feature request." required: true - type: textarea id: description attributes: label: Description description: A summary of the feature you want to request. validations: required: true ================================================ FILE: .github/ISSUE_TEMPLATE/3_question.yml ================================================ name: Ask a question description: Ask a question regarding iSubRip. title: "[Question]: " labels: [question] body: - type: markdown attributes: value: | **Please use this template only for questions. For issue / bug reports and feature requests, use one of the other templates.** - type: textarea id: description attributes: label: Question description: The question you want to ask. validations: required: true ================================================ FILE: .github/ISSUE_TEMPLATE/config.yml ================================================ blank_issues_enabled: false ================================================ FILE: .gitignore ================================================ # From: https://github.com/github/gitignore/blob/main/Python.gitignore # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: # .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintainted in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ ================================================ FILE: CHANGELOG.md ================================================ # Changelog ## 2.6.8 [2025-10-14] ### Changes: * Removed Python 3.9 support, added Python 3.14 support. ### Bug Fixes: * Fixed a regression issue where the source type (e.g. "WEB") was missing from zip file names following the changes in version 2.6.7. ([Issue #100](https://github.com/MichaelYochpaz/iSubRip/issues/100)) --- ## 2.6.7 [2025-10-10] ### Added: * Updated M3U8 playlists loading to use alternative playlist URLs as fallback in case of an empty response. * Added playlist URLs to debug-level logs. ([Issue #94](https://github.com/MichaelYochpaz/iSubRip/issues/94)) --- ## 2.6.6 [2025-08-13] ### Changes: * Updated the date format in logs for unreleased content that has a release date. ([Issue #91](https://github.com/MichaelYochpaz/iSubRip/issues/91)) ### Bug Fixes: * Fixed an issue where in some cases, when converting from WebVTT to SubRip with the `subrip-alignment-conversion` setting enabled, top-aligned captions were not detected correctly. ([Issue #90](https://github.com/MichaelYochpaz/iSubRip/issues/90)) --- ## 2.6.5 [2025-06-20] ### Changes: * Added missing languages to the list of RTL languages (relevant if the `languages.fix-rtl` config setting is enabled). * Minor improvements to the download progress bar. ### Bug Fixes: * Fixed an issue where in some cases, the "Downloaded subtitles (X/Y)" progress log line would print repeatedly. ([Issue #87](https://github.com/MichaelYochpaz/iSubRip/issues/87)) --- ## 2.6.4 [2025-06-13] ### Changes: * Minor improvements to the download progress bar. ### Bug Fixes: * Fixed an issue where if there is no config file, an error is raised instead of using default settings. ([Issue #86](https://github.com/MichaelYochpaz/iSubRip/issues/86)) --- ## 2.6.3 [2025-03-17] ### Bug Fixes: * Fixed an issue where logs containing the percentage character (`%`) would raise an error. ([Issue #82](https://github.com/MichaelYochpaz/iSubRip/issues/82)) --- ## 2.6.2 [2025-02-04] ### Bug Fixes: * Fixed an issue where AppleTV API calls would fail due to changes on AppleTV requiring a missing `utsk` parameter. ([Issue #80](https://github.com/MichaelYochpaz/iSubRip/issues/80)) * Fixed an issue where iTunes URLs would not work due to iTunes no longer redirecting to AppleTV. A different method will be used now to find corresponding AppleTV URLs. Also added a retry mechanism as it appears to be a bit unreliable at times. (thanks @yonatand1230 for suggesting this method!). ([Issue #78](https://github.com/MichaelYochpaz/iSubRip/issues/78)) * Removed progress bar when where there are no matching subtitles to download (previously, it would just show 0/0 with 0% progress). --- ## 2.6.1 [2025-01-31] ### Bug Fixes: * Fixed a backwards compatibility issue in code, which would cause errors when running on Python versions lower than 3.12. ([Issue #78](https://github.com/MichaelYochpaz/iSubRip/issues/78)) --- ## 2.6.0 [2025-01-28] **The following update contains breaking changes to the config file. If you are using one, please update your config file accordingly.** ### Added: * Added a new `general.log-level` config setting, the log level of stdout (console) output. Set to `info` by default. Can be changed to `debug`, `warning`, or `error`. See the updated [example config](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml) for an example. ### Changes: * Console output has been overhauled and improved, with colorful interactive output. * Config file is now parsed and validated in a more reliable and efficient manner. Configuration errors will now be more readable and descriptive. * **Breaking config changes** - the `scrapers` config category has been updated. Settings that should apply for all scrapers are now under the `scrapers.default` category instead of straight under `scrapers`. See the updated [example config](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml) for examples. * Updated AppleTV scraper request parameters. * Minor improvements to logs. * Python 3.8 is no longer supported. Minimum supported version has been updated to 3.9. ### Bug Fixes: * Fixed an issue where if `verify-ssl` is set to `false`, and the `urllib3` package (which isn't a dependency of iSubRip) is not installed, an error could be thrown. --- ## 2.5.6 [2024-07-07] ### Bug Fixes: * Fixed an issue where the update message from version `2.5.4` to `2.5.5` would still appear after updating. ([Issue #73](https://github.com/MichaelYochpaz/iSubRip/issues/73)) --- ## 2.5.5 [2024-07-06] ### Added: * Added new `timeout` setting to the config file, for the option to change the timeout for all / specific scrapers. See the updated [example config](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml) for usage examples. ([Issue #71](https://github.com/MichaelYochpaz/iSubRip/issues/71)) ### Changes: * Default timeout for requests has been updated from 5 seconds to 10 seconds. ([Issue #71](https://github.com/MichaelYochpaz/iSubRip/issues/71)) --- ## 2.5.4 [2024-04-28] ### Bug Fixes: * Fixed an issue where if the `logs` directory does not exist, the folder isn't created, causing an error. ([Issue #67](https://github.com/MichaelYochpaz/iSubRip/issues/67)) * Fixed an issue where the summary log of successful and failed download would not account for failed downloads. ([Issue #68](https://github.com/MichaelYochpaz/iSubRip/issues/68)) --- ## 2.5.3 [2024-04-09] ### Added: * Added new `proxy` and `verify-ssl` settings to the config file, for allowing the usage of a proxy when making requests, and disabling SSL verification. See the updated [example config](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml) for usage examples. ### Changes: * `subtitles.rtl-languages` config setting is no longer supported, and its values are now hardcoded and can't be modified. ### Bug Fixes: * Fixed an issue where in some cases, `STYLE` blocks would repeat throughout the subtitles file, and cause inaccurate cue count. ([Issue #63](https://github.com/MichaelYochpaz/iSubRip/issues/63)) * Fixed an issue where the WebVTT Style blocks would have their `STYLE` tag replaced with a `REGION` tag in downloaded subtitles. * Fixed an issue where an empty playlist (with a size of 0 bytes) would be reported as a valid playlist with no matching subtitles. ([Issue #65](https://github.com/MichaelYochpaz/iSubRip/issues/65)) --- ## 2.5.2 [2024-01-06] ### Bug Fixes: * Fixed an issue where errors would not be handled gracefully, and cause an unexpected crash. ([Issue #55](https://github.com/MichaelYochpaz/iSubRip/issues/55)) --- ## 2.5.1 [2023-12-23] ### Bug Fixes: * Fixed an issue where source abbreviation was missing from file names of downloaded subtitles files. ([Issue #53](https://github.com/MichaelYochpaz/iSubRip/issues/53)) --- ## 2.5.0 [2023-12-16] ### Added: * Added logs. See the new [Logs section in the README](https://github.com/MichaelYochpaz/iSubRip#logs) for more information. * Added a new `subtitles.webvtt.subrip-alignment-conversion` config setting (which is off by default), which if set to true, will add the `{\an8}` tag at the start of lines that are annotated at the top (with the `line:0.00%` WebVTT setting) when converting to SubRip. ([Issue #35](https://github.com/MichaelYochpaz/iSubRip/issues/35)) * Implemented caching for AppleTV's storefront configuration data, which should reduce the amount of requests used when scraping multiple AppleTV URLs from the same storefront. ### Changes: * Big backend changes to the structure of the code, mostly to improve modularity and allow for easier development in the future, and improve performance. * Updated the CLI output to utilize logs and print with colors according to log-level. * Improved error handling in some cases where an invalid URL is used. ### Bug Fixes: * Fixed an issue where if a movie is a pre-order with a set release date, a message with availability date wouldn't be printed in some cases. --- ## 2.4.3 [2023-06-18] ### Bug Fixes: * Fixed an issue where some AppleTV URLs (or iTunes links that refer to such URLs) would not be matched in some cases, resulting in a "No matching scraper was found..." error. ([Issue #46](https://github.com/MichaelYochpaz/iSubRip/issues/46)) --- ## 2.4.2 [2023-06-02] ### Changes: * Improved error handling for subtitles downloads. ([Issue #44](https://github.com/MichaelYochpaz/iSubRip/issues/44)) ### Bug Fixes: * Fixed an issue where using a ZIP file, and saving to a different drive than the OS drive would fail. ([Issue #43](https://github.com/MichaelYochpaz/iSubRip/issues/43)) --- ## 2.4.1 [2023-05-25] ### Bug Fixes: * Fixed an issue where saving subtitles to a different drive than the OS drive would fail. ([Issue #41](https://github.com/MichaelYochpaz/iSubRip/issues/41)) * Fixed AppleTV URLs with multiple iTunes playlists causing an error. ([Issue #42](https://github.com/MichaelYochpaz/iSubRip/issues/42)) --- ## 2.4.0 [2023-05-23] ### Added: - iTunes links will now redirect to AppleTV and scrape metadata from there, as AppleTV has additional and more accurate metadata. - Improved error messages to be more informative and case-specific: - If a movie is a pre-order and has no available playlist, a proper error message will be printed with its release date (if available). - If trying to scrape AppleTV+ content or series (which aren't currently supported), a proper error will be printed. ### Changes: - A major refactor to the code, to make it more modular and allow for easier development of new features in the future. - Multiple changes (with some breaking changes) to the config file: - The `downloads.format` setting is deprecated, and replaced by the `subtitles.convert-to-srt` setting. - The `downloads.merge-playlists` setting is deprecated, with no replacement. If an AppleTV link has multiple playlists, they will be downloaded separately. - The `downloads.user-agent` setting is deprecated, with no replacement. The user-agent used by the scraper, will be used for downloads as well. - The `scraping` config category no longer exists, and is replaced by a `scrapers` category, which has a sub-category with settings for each scraper (for example, a `scrapers.itunes` sub-category). - Old config paths that were previously deprecated are no longer supported and will no longer work. The updated config settings can be found in the [example config](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml). ### Notes: * This release includes a major rewrite of the code, which may have introduced new bugs to some core features. If you encountered one, [please report it](https://github.com/MichaelYochpaz/iSubRip/issues/new/choose). * Minimum supported Python version bumped to 3.8. * `beautifulsoup4` and `lxml` packages are no longer required or used. --- ## 2.3.3 [2022-10-09] ### Changes: * Added release year to zip file names. ([Issue #31](https://github.com/MichaelYochpaz/iSubRip/issues/31)) * If the generated path for a zip file is already taken, a number will be appended at the end of the file's name to avoid overwriting. ([Issue #34](https://github.com/MichaelYochpaz/iSubRip/issues/34)) ### Bug Fixes: * Fixed an exception being thrown if the path to downloads folder on the config is invalid. * Fixed AppleTV URLs without a movie title not working. ([Issue #29](https://github.com/MichaelYochpaz/iSubRip/issues/29)) * Fixed issues for movies with specific characters (`/`, `:`), and Windows reserved names in their title. ([Issue #30](https://github.com/MichaelYochpaz/iSubRip/issues/30)) --- ## 2.3.2 [2022-08-06] ### Changes: * Changed config paths to the following locations: Windows: `%USERPROFILE%\.isubrip\config.json` Linux / macOS: `$HOME/.isubrip/config.json` More info under Notes (and examples on the [README](https://github.com/MichaelYochpaz/iSubRip#configuration) file). ### Bug Fixes: * Fixed an error with AppleTV links for movies released before 1970 (Epoch time). ([Issue #21](https://github.com/MichaelYochpaz/iSubRip/issues/21)) * Fixed config file not being loaded on macOS. ([Issue #22](https://github.com/MichaelYochpaz/iSubRip/issues/22)) * Fixed AppleTV scraping from the same storefront. ([Issue #24](https://github.com/MichaelYochpaz/iSubRip/issues/24)) ### Notes: * Running iSubRip with a config file in the previous locations will still work, but support for them will be dropped in the future. * `xdg` package is no longer required or used. --- ## 2.3.1 [2022-07-15] ### Changes: * Improved AppleTV scraping to utilize AppleTV's API instead of scraping HTML. ### Bug Fixes: * Fixed HTML escaped (for non-English) characters not matching AppleTV's URL RegEx. ([Issue #15](https://github.com/MichaelYochpaz/iSubRip/issues/15)) --- ## 2.3.0 [2022-06-23] ### Added: * AppleTV movie URLs are now supported. * Added a `merge-playlists` config option to treat multiple playlists that can be found on AppleTV pages as one (more info on the example config). ### Changes: * Improved subtitles parser to perserve additional WebVTT data. * The config value `user-agent` under `scraping` is now separated to 2 different values: `itunes-user-agent`, and `appletv-user-agent`. ### Bug Fixes: * Fixed movie titles with invalid Windows file-name characters (example: '?') causing a crash. ([Issue #14](https://github.com/MichaelYochpaz/iSubRip/issues/14)) * Fixed iTunes store URLs without a movie title not working. ([Issue #13](https://github.com/MichaelYochpaz/iSubRip/issues/13)) --- ## 2.2.0 [2022-04-25] ### Added: * Replaced FFmpeg usage for parsing with a native subtitles parser (downloads are much faster now). * Added a `remove-duplicates` configuration remove duplicate paragraphs. (Was previously automatically fixed by FFmpeg.) * Added `fix-rtl` and `rtl-languages` configuration to fix RTL in RTL-languaged subtitles (has to be enabled in the config). ### Changes: * FFmpeg is no longer required or used, and all FFmpeg-related settings are deprecated. ### Notes: * `fix-rtl` is off by default and has to be enabled on the config. Check the `config.toml` example file for more info. * Minimum supported Python version bumped to 3.7. --- ## 2.1.2 [2022-04-03] ### Bug Fixes: * Fixed subtitles being downloaded twice, which causes long (doubled) download times. --- ## 2.1.1 [2022-03-28] ### Bug Fixes: * Fixed a compatibility issue with Python versions that are lower than 3.10. * Fixed downloading subtitles to an archive file not working properly. * Fixed a bug where the code continues to run if subtitles download failed, as if the download was successful. --- ## 2.1.0 [2022-03-19] ### Added: * A note will be printed if a newer version is available on PyPI (can be disabled on the config). * Config will now be checked for errors before running. ### Changes: * Big improvements to scraping, which is now far more reliable. * Added release year to subtitles file names. * Config structure slightly changed. ### Notes: * If you use a user-config, it might need to be updated to match the new config structure. Example of an updated valid structure can be found [here](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml). --- ## 2.0.0 [2022-01-30] The script is now a Python package that can be installed using pip. ### Added: * Added a config file for changing configurations. (Example can be found [here](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml)) * Added an option to choose subtitles format (vtt / srt). * Added an option to choose whether to zip subtitles files or not. * Multiple links can be passed for downloading subtitles for multiple movies one after another. * Temporary files are automatically removed if the script stops unexpectedly. ### Changes: * A complete code overhaul from a single python script file to a package, while utilizing OOP and classes. * Improved scraping algorithm for faster playlist scraping. * FFmpeg will now automatically overwrite existing subtitles with the same file name. ### Bug Fixes: * Fixed a bug where in some cases, no subtitles were found since the title has HTML escaped characters, which causes bad matching when checking if a valid playlist was found. --- ## 1.0.6 [2021-07-23] ### Bug Fixes: * Fixed an issue where in some cases subtitles won't download when using `DOWNLOAD_FILTER` because of letter casing not matching. * Fixed and improved error handling, and added more descriptive error messages. ([Issue #9](https://github.com/MichaelYochpaz/iSubRip/issues/9)) --- ## 1.0.5 [2021-05-27] ### Bug Fixes: * Fixed subtitles for some movies not being found after previous release. ([Issue #8](https://github.com/MichaelYochpaz/iSubRip/issues/8)) --- ## 1.0.4 [2021-05-25] ### Bug Fixes: * Fixed the script not working after iTunes webpage data orientation slightly changed. ([Issue #6](https://github.com/MichaelYochpaz/iSubRip/issues/6) , [Issue #7](https://github.com/MichaelYochpaz/iSubRip/issues/7)) --- ## 1.0.3 [2021-04-30] ### Bug Fixes: * Fixed a bug where subtitles for suggested movies are being downloaded if movie's main playlist is not found. ([Issue #2](https://github.com/MichaelYochpaz/iSubRip/issues/2)) * Added a "cc" tag to closed-caption subtitles' filename to avoid a collision with non-cc subtitles. ([Issue #3](https://github.com/MichaelYochpaz/iSubRip/issues/3)) --- ## 1.0.2 [2021-04-15] ### Added: * Added a User-Agent for sessions to avoid being blocked. ### Changes: * `DOWNLOAD_FILTER` is no longer case-sensitive. * Added `lxml` to `requirements.txt`. ([Issue #1](https://github.com/MichaelYochpaz/iSubRip/issues/1)) ### Bug Fixes: * Fixed the script not working after iTunes webpage data orientation slightly changed. ([Issue #1](https://github.com/MichaelYochpaz/iSubRip/issues/1)) --- ## 1.0.1 [2020-12-13] ### Changes: * Improved error handling. ### Bug Fixes: * Fixed file name formatting. --- ## 1.0.0 [2020-11-02] * Initial release. ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2022 Michael Yochpaz Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ > [!CAUTION] > iSubRip is currently not working due to changes on Apple's backend. > The future of this project is currently unknown. See [#103](https://github.com/MichaelYochpaz/iSubRip/issues/103) for details. # iSubRip **iSubRip** is a Python command-line tool for scraping and downloading subtitles from AppleTV and iTunes movie pages.
Python Version PyPI Version License Monthly Downloads Total Downloads Repo Stars Issues

--- ## ✨ Features - Scrape subtitles from AppleTV and iTunes movies without needing a purchase or account. - Retrieve the expected streaming release date (if available) for unreleased movies. - Utilize asynchronous downloading to speed up the download of chunked subtitles. - Automatically convert subtitles to SubRip (SRT) format. - Fix right-to-left (RTL) alignment in RTL language subtitles automatically. - Configure settings such as download folder, preferred languages, and toggling features. ## 🚀 Quick Start ### Installation ```shell pip install isubrip ``` ### Usage ```shell isubrip [URL...] ``` (URL can be either an AppleTV or iTunes movie URL)
> [!WARNING] > iSubRip is not recommended for use as a library in other projects. > The API frequently changes, and breaking changes to the API are common, even in minor versions. > > Support will not be provided for issues arising from using this package as a library. ## 🛠 Configuration A [TOML](https://toml.io) configuration file can be created to customize various options and features. The configuration file will be searched for in one of the following paths based on your operating system: - **Windows**: `%USERPROFILE%\.isubrip\config.toml` - **Linux / macOS**: `$HOME/.isubrip/config.toml` ### Path Examples - **Windows**: `C:\Users\Michael\.isubrip\config.toml` - **Linux**: `/home/Michael/.isubrip/config.toml` - **macOS**: `/Users/Michael/.isubrip/config.toml` ### Example Configuration ```toml [downloads] folder = "C:\\Subtitles\\iTunes" languages = ["en-US", "fr-FR", "he"] zip = false [subtitles] convert-to-srt = true fix-rtl = true [subtitles.webvtt] subrip-alignment-conversion = true ``` An example config with details and explanations for all available settings can be found [here](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml). ## 📜 Logs Log files are created for each run in the following paths, depending on your operating system: **Windows**: `%USERPROFILE%\.isubrip\logs` **Linux / macOS**: `$HOME/.isubrip/logs` Log rotation (deletion of old files once a certain number of files is reached) can be configured in the configuration file using the `general.log-rotation-size` setting. The default value is `15`. For more details, see the [example configuration](https://github.com/MichaelYochpaz/iSubRip/blob/main/example-config.toml). ## 📓 Changelog The changelog for the latest, and all previous versions, can be found [here](https://github.com/MichaelYochpaz/iSubRip/blob/main/CHANGELOG.md). ## 👨🏽‍💻 Contributing This project is open-source but currently lacks the infrastructure to fully support external contributions. If you wish to contribute, please open an issue first to discuss your proposed changes to avoid working on something that might not be accepted. ## 🙏🏽 Support If you find this project helpful, please consider supporting it by: - 🌟 Starring the repository - 💖 [Sponsoring the project](https://github.com/sponsors/MichaelYochpaz) ## 📝 End User License Agreement By using iSubRip, you agree to the following terms: 1. **Disclaimer of Affiliation**: iSubRip is an independent, open-source project. It is not affiliated with, endorsed by, or in any way officially connected to Apple Inc., iTunes, or AppleTV. 2. **Educational Purpose**: This tool is developed and provided for educational and research purposes only. It demonstrates techniques for accessing and processing publicly available, unencrypted subtitle data from HLS playlists. 3. **User Responsibility and Compliance**: Any use of iSubRip is solely at the user's own risk and discretion. Users are responsible for ensuring that their use of the tool complies with all applicable laws, regulations, and terms of service of the content providers. This includes adhering to local, state, national, and international laws and regulations. 4. **Limitation of Liability**: The developers of iSubRip shall not be held responsible for any legal consequences arising from the use of this tool. This includes, but is not limited to, claims of copyright infringement, intellectual property violations, or breaches of terms of service of content providers. Users assume all risks associated with acquiring and using subtitle data through this tool. By using iSubRip, you acknowledge that you have read, understood, and agree to be bound by this agreement's terms and conditions. ## ⚖️ License This project is licensed under the MIT License. For more details, see the [LICENSE file](https://github.com/MichaelYochpaz/iSubRip/blob/main/LICENSE). ================================================ FILE: example-config.toml ================================================ # ---------------- ⚠️ IMPORTANT - READ BEFORE USING ⚠️ ---------------- # This is an example config file with all available settings and their default values (if they have one). # All settings are optional, and setting them in the config file will override their default values. # # In your config file, set only settings you wish to change from their default values. # Do NOT copy this file and use it as your config, as it will override ALL settings with the values specified here. # Use this file only as a reference to understand what different settings do, # and to decide which settings you should use in your config. # # Your config file should be saved in the following path (according to OS): # - Windows: %USERPROFILE%\.isubrip\config.toml # - Linux / macOS: $HOME/.isubrip/config.toml # --------------------------------------------------------------------- [general] # Check for updates before running, and show a note if a new version exists. # Value can be either 'true' or 'false'. check-for-updates = true # Maximum number of log files to keep in the logs folder. # Once the maximum number is reached, the oldest logs files will be deleted in rotation # until the number of files equals the maximum. log-rotation-size = 15 # Log level to use for stdout (console) output. # Value can be one of: "debug", "info", "error", "warning", "critical". log-level = "info" [downloads] # Folder to downloads files to. # The default "." value means it will download to the same folder the script ran from. # Use double backslashes in path to avoid escaping characters. Example: "C:\\Users\\\\Downloads\\" folder = "." # A list of iTunes language codes to download. # An empty array (like the one currently being used) will result in downloading all of the available subtitles. # Example: ["en-US", "fr-FR", "he"] languages = [] # Whether to overwrite existing subtitles files. # If set to false, names of existing subtitles will have a number appended to them to avoid overwriting. # Value can be either 'true' or 'false'. overwrite-existing = false # Save files into a zip archive if there is more than one matching subtitles. # Value can be either 'true' or 'false'. zip = false [subtitles] # Fix RTL for RTL languages (Arabic & Hebrew). # Value can be either 'true' or 'false'. # # NOTE: This is off by default as some subtitles use other methods to fix RTL (like writing punctuations backwards). # Using this option on these type of subtitles can break the already-fixed RTL issues. fix-rtl = false # Remove duplicate paragraphs (same text and timestamps). # Value can be either 'true' or 'false'. remove-duplicates = true # Whether to convert subtitles to SRT format. # NOTE: This can cause loss of subtitles metadata that is not supported by SRT format. convert-to-srt = false [subtitles.webvtt] # Whether to add a '{\an8}' tag to lines that are aligned at the top when converting format from WebVTT to SubRip. # Relevant only if 'subtitles.convert-to-srt' is set to 'true'. # Value can be either 'true' or 'false'. subrip-alignment-conversion = false [scrapers.default] # A subcategory to set default values for all scrapers. # These settings will be overridden by scraper-specific configuration, if set, # These settings will not apply if the scraper has a different specific default value. # Timeout in seconds for requests sent by all scrapers. timeout = 10 # User-Agent to use by default for requests sent by all scrapers. user-agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" # Proxy to use by default for requests sent by all scrapers. proxy = "http://127.0.0.1:8080" # Whether to verify SSL certificates when making requests for all scrapers. # Value can be either 'true' or 'false'. verify-ssl = true [scrapers.scraper-name] # Scraper-specific settings (set for each scraper separately). # Will override any default values previously set. # Replace 'scraper-name' with the name of the scraper to configure. # Available scrapers: itunes, appletv # Timeout in seconds for requests sent by the scraper. timeout = 10 # User-Agent to use for requests sent by the scraper. user-agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" # Proxy to use for requests sent by the scraper. proxy = "http://127.0.0.1:8080" # Whether to verify SSL certificates when making requests for the scraper. # Value can be either 'true' or 'false'. verify-ssl = true ================================================ FILE: isubrip/__init__.py ================================================ ================================================ FILE: isubrip/__main__.py ================================================ from __future__ import annotations import asyncio import logging import sys from typing import TYPE_CHECKING import httpx from pydantic import ValidationError from isubrip.cli import console from isubrip.commands.download import download from isubrip.config import Config from isubrip.constants import ( PACKAGE_NAME, PACKAGE_VERSION, data_folder_path, log_files_path, user_config_file_path, ) from isubrip.logger import logger, setup_loggers from isubrip.scrapers.scraper import Scraper, ScraperFactory from isubrip.subtitle_formats.webvtt import WebVTTCaptionBlock from isubrip.utils import ( convert_log_level, format_config_validation_error, get_model_field, raise_for_status, single_string_to_list, ) if sys.version_info >= (3, 11): import tomllib else: import tomli as tomllib if TYPE_CHECKING: from pathlib import Path log_rotation_size: int = 15 # Default size, before being updated by the config file. def main() -> None: """A wrapper for the actual main function that handles exceptions and cleanup.""" try: asyncio.run(_main()) except Exception as ex: logger.error(f"Error: {ex}") logger.debug("Debug information:", exc_info=True) exit(1) except KeyboardInterrupt: logger.debug("Keyboard interrupt detected, exiting...") exit(0) finally: if log_rotation_size > 0: handle_log_rotation(rotation_size=log_rotation_size) for scraper in ScraperFactory.get_initialized_scrapers(): logger.debug(f"Requests count for '{scraper.name}' scraper: {scraper.requests_count}") async def _main() -> None: # Assure at least one argument was passed if len(sys.argv) < 2: logger.info(f"Usage: {PACKAGE_NAME} [iTunes movie URL...]") exit(0) # Generate the data folder if it doesn't previously exist if not data_folder_path().is_dir(): data_folder_path().mkdir(parents=True) # If config file exists, parse it. Otherwise, create a config with default values if user_config_file_path().is_file(): config = parse_config(config_file_location=user_config_file_path()) else: config = Config() setup_loggers( stdout_loglevel=convert_log_level(log_level=config.general.log_level), stdout_console=console, logfile_output=True, logfile_output_path=log_files_path(), logfile_loglevel=logging.DEBUG, ) cli_args = " ".join(sys.argv[1:]) logger.debug(f"CLI Command: {PACKAGE_NAME} {cli_args}") logger.debug(f"Python version: {sys.version}") logger.debug(f"Package version: {PACKAGE_VERSION}") logger.debug(f"OS: {sys.platform}") update_settings(config=config) if config.general.check_for_updates: check_for_updates(current_package_version=PACKAGE_VERSION) try: await download( *single_string_to_list(item=sys.argv[1:]), download_path=config.downloads.folder, language_filter=config.downloads.languages, convert_to_srt=config.subtitles.convert_to_srt, overwrite_existing=config.downloads.overwrite_existing, zip=config.downloads.zip, ) finally: async_cleanup_coroutines = [] for scraper in ScraperFactory.get_initialized_scrapers(): async_cleanup_coroutines.append(scraper.async_close()) if async_cleanup_coroutines: try: await asyncio.gather(*async_cleanup_coroutines) except Exception as e: logger.warning(f"Error during async cleanup: {e}") logger.debug("Cleanup debug info:", exc_info=True) def check_for_updates(current_package_version: str) -> None: """ Check and print if a newer version of the package is available, and log accordingly. Args: current_package_version (str): The current version of the package. """ api_url = f"https://pypi.org/pypi/{PACKAGE_NAME}/json" logger.debug("Checking for package updates on PyPI...") try: response = httpx.get( url=api_url, headers={"Accept": "application/json"}, timeout=5, ) raise_for_status(response) response_data = response.json() pypi_latest_version = response_data["info"]["version"] if pypi_latest_version != current_package_version: logger.warning(f"You are currently using version '{current_package_version}' of '{PACKAGE_NAME}', " f"however version '{pypi_latest_version}' is available." f'\nConsider upgrading by running "pip install --upgrade {PACKAGE_NAME}"') else: logger.debug(f"Latest version of '{PACKAGE_NAME}' ({current_package_version}) is currently installed.") except Exception as e: logger.warning(f"Update check failed: {e}") logger.debug("Debug information:", exc_info=True) return def handle_log_rotation(rotation_size: int) -> None: """ Handle log rotation and remove old log files if needed. Args: rotation_size (int): Maximum amount of log files to keep. """ sorted_log_files = sorted(log_files_path().glob("*.log"), key=lambda file: file.stat().st_mtime, reverse=True) if len(sorted_log_files) > rotation_size: for log_file in sorted_log_files[rotation_size:]: log_file.unlink() def parse_config(config_file_location: Path) -> Config: """ Parse the configuration file and return a Config instance. Exit the program (with code 1) if an error occurs while parsing the configuration file. Args: config_file_location (Path): The location of the configuration file. Returns: Config: An instance of the Config. """ try: with config_file_location.open('rb') as file: config_data = tomllib.load(file) return Config.model_validate(config_data) except ValidationError as e: logger.error("Invalid configuration - the following errors were found in the configuration file:\n" + format_config_validation_error(exc=e) + "\nPlease update your configuration to resolve this issue.") logger.debug("Debug information:", exc_info=True) exit(1) except tomllib.TOMLDecodeError as e: logger.error(f"Error parsing config file: {e}") logger.debug("Debug information:", exc_info=True) exit(1) except Exception as e: logger.error(f"Error loading configuration: {e}") logger.debug("Debug information:", exc_info=True) exit(1) def update_settings(config: Config) -> None: """ Update settings according to config. Args: config (Config): An instance of a config to set settings according to. """ if config.general.log_level.casefold() == "debug": console.is_interactive = False Scraper.subtitles_fix_rtl = config.subtitles.fix_rtl Scraper.subtitles_remove_duplicates = config.subtitles.remove_duplicates Scraper.default_timeout = config.scrapers.default.timeout Scraper.default_user_agent = config.scrapers.default.user_agent Scraper.default_proxy = config.scrapers.default.proxy Scraper.default_verify_ssl = config.scrapers.default.verify_ssl for scraper in ScraperFactory.get_scraper_classes(): if scraper_config := get_model_field(model=config.scrapers, field=scraper.id): scraper.config = scraper_config WebVTTCaptionBlock.subrip_alignment_conversion = ( config.subtitles.webvtt.subrip_alignment_conversion ) if config.general.log_rotation_size: global log_rotation_size log_rotation_size = config.general.log_rotation_size if __name__ == "__main__": main() ================================================ FILE: isubrip/cli.py ================================================ from collections.abc import Iterator from contextlib import contextmanager from typing import Any from rich.console import Console from rich.live import Live console = Console( highlight=False, ) @contextmanager def conditional_live(renderable: Any) -> Iterator[Live | None]: """ A context manager that conditionally enables Rich's Live display based on console interactivity. When console.is_interactive is True, this behaves like Rich's Live display. When console.is_interactive is False, live updates are disabled. Args: renderable: The Rich renderable object to display in live mode. Yields: Optional[Live]: The Live display object if console is interactive, None otherwise. Example: ```python with conditional_live(progress) as live: # Your code here if live: # Optional: Check if live display is active live.update(...) ``` """ if console.is_interactive: with Live(renderable, console=console) as live: yield live else: yield None ================================================ FILE: isubrip/commands/__init__.py ================================================ ================================================ FILE: isubrip/commands/download.py ================================================ from __future__ import annotations from pathlib import Path import shutil from rich.console import Group from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn from rich.text import Text from isubrip.cli import conditional_live, console from isubrip.data_structures import ( Episode, MediaData, Movie, ScrapedMediaResponse, Season, Series, SubtitlesData, SubtitlesDownloadResults, ) from isubrip.logger import logger from isubrip.scrapers.scraper import PlaylistLoadError, Scraper, ScraperError, ScraperFactory, SubtitlesDownloadError from isubrip.ui import MinsAndSecsTimeElapsedColumn from isubrip.utils import ( TemporaryDirectory, download_subtitles_to_file, format_list, format_media_description, format_release_name, generate_non_conflicting_path, ) async def download(*urls: str, download_path: Path, language_filter: list[str] | None = None, convert_to_srt: bool = False, overwrite_existing: bool = True, zip: bool = False) -> None: """ Download subtitles from given URLs. Args: urls (list[str]): A list of URLs to download subtitles from. download_path (Path): Path to a folder where the subtitles will be downloaded to. language_filter (list[str] | None): List of specific languages to download. None for all languages (no filter). Defaults to None. convert_to_srt (bool, optional): Whether to convert the subtitles to SRT format. Defaults to False. overwrite_existing (bool, optional): Whether to overwrite existing subtitles. Defaults to True. zip (bool, optional): Whether to zip multiple subtitles. Defaults to False. """ for url in urls: try: logger.info(f"Scraping [blue]{url}[/blue]") scraper = ScraperFactory.get_scraper_instance(url=url) try: logger.debug(f"Fetching {url}") scraper_response: ScrapedMediaResponse = await scraper.get_data(url=url) except ScraperError as e: logger.error(f"Error: {e}") logger.debug("Debug information:", exc_info=True) continue media_data = scraper_response.media_data playlist_scraper = ScraperFactory.get_scraper_instance(scraper_id=scraper_response.playlist_scraper) if not media_data: logger.error(f"Error: No supported media was found for {url}.") continue for media_item in media_data: try: logger.info(f"Found {media_item.media_type}: " f"[cyan]{format_media_description(media_data=media_item)}[/cyan]") await download_media(scraper=playlist_scraper, media_item=media_item, download_path=download_path, language_filter=language_filter, convert_to_srt=convert_to_srt, overwrite_existing=overwrite_existing, zip=zip) except Exception as e: if len(media_data) > 1: logger.warning(f"Error scraping media item " f"'{format_media_description(media_data=media_item)}': {e}\n" f"Skipping to next media item...") logger.debug("Debug information:", exc_info=True) continue raise except Exception as e: logger.error(f"Error while scraping '{url}': {e}") logger.debug("Debug information:", exc_info=True) continue async def download_media(scraper: Scraper, media_item: MediaData, download_path: Path, language_filter: list[str] | None = None, convert_to_srt: bool = False, overwrite_existing: bool = True, zip: bool = False) -> None: """ Download a media item. Args: scraper (Scraper): A Scraper object to use for downloading subtitles. media_item (MediaData): A media data item to download subtitles for. download_path (Path): Path to a folder where the subtitles will be downloaded to. language_filter (list[str] | None): List of specific languages to download. None for all languages (no filter). Defaults to None. convert_to_srt (bool, optional): Whether to convert the subtitles to SRT format. Defaults to False. overwrite_existing (bool, optional): Whether to overwrite existing subtitles. Defaults to True. zip (bool, optional): Whether to zip multiple subtitles. Defaults to False. """ if isinstance(media_item, Series): for season in media_item.seasons: await download_media(media_item=season, scraper=scraper, download_path=download_path, language_filter=language_filter, convert_to_srt=convert_to_srt, overwrite_existing=overwrite_existing, zip=zip) elif isinstance(media_item, Season): for episode in media_item.episodes: logger.info(f"{format_media_description(media_data=episode, shortened=True)}:") await download_media_item(media_item=episode, scraper=scraper, download_path=download_path, language_filter=language_filter, convert_to_srt=convert_to_srt, overwrite_existing=overwrite_existing, zip=zip) elif isinstance(media_item, (Movie | Episode)): await download_media_item(media_item=media_item, scraper=scraper, download_path=download_path, language_filter=language_filter, convert_to_srt=convert_to_srt, overwrite_existing=overwrite_existing, zip=zip) async def download_media_item(scraper: Scraper, media_item: Movie | Episode, download_path: Path, language_filter: list[str] | None = None, convert_to_srt: bool = False, overwrite_existing: bool = True, zip: bool = False) -> None: """ Download subtitles for a single media item. Args: scraper (Scraper): A Scraper object to use for downloading subtitles. media_item (Movie | Episode): A movie or episode data object. download_path (Path): Path to a folder where the subtitles will be downloaded to. language_filter (list[str] | None): List of specific languages to download. None for all languages (no filter). Defaults to None. convert_to_srt (bool, optional): Whether to convert the subtitles to SRT format. Defaults to False. overwrite_existing (bool, optional): Whether to overwrite existing subtitles. Defaults to True. zip (bool, optional): Whether to zip multiple subtitles. Defaults to False. """ ex: Exception | None = None if media_item.playlist: try: results = await download_subtitles( scraper=scraper, media_data=media_item, download_path=download_path, language_filter=language_filter, convert_to_srt=convert_to_srt, overwrite_existing=overwrite_existing, zip=zip, ) success_count = len(results.successful_subtitles) failed_count = len(results.failed_subtitles) if success_count or failed_count: logger.info(f"{success_count}/{success_count + failed_count} subtitles were successfully downloaded.") else: logger.info("No matching subtitles were found.") return # noqa: TRY300 except PlaylistLoadError as e: ex = e # We get here if there is no playlist, or there is one, but it failed to load if isinstance(media_item, Movie) and media_item.preorder_availability_date: logger.info(f"[gold1]'{media_item.name}' is currently unavailable on {scraper.name}, " f"and will be available on {media_item.preorder_availability_date.strftime(r'%B %e, %Y')}.[/gold1]") else: if ex: logger.error(f"Error: {ex}") else: logger.error("Error: No valid playlist was found.") async def download_subtitles(scraper: Scraper, media_data: Movie | Episode, download_path: Path, language_filter: list[str] | None = None, convert_to_srt: bool = False, overwrite_existing: bool = True, zip: bool = False) -> SubtitlesDownloadResults: """ Download subtitles for the given media data. Args: scraper (Scraper): A Scraper object to use for downloading subtitles. media_data (Movie | Episode): A movie or episode data object. download_path (Path): Path to a folder where the subtitles will be downloaded to. language_filter (list[str] | None): List of specific languages to download. None for all languages (no filter). Defaults to None. convert_to_srt (bool, optional): Whether to convert the subtitles to SRT format. Defaults to False. overwrite_existing (bool, optional): Whether to overwrite existing subtitles. Defaults to True. zip (bool, optional): Whether to zip multiple subtitles. Defaults to False. Returns: SubtitlesDownloadResults: A SubtitlesDownloadResults object containing the results of the download. """ temp_dir_name = format_release_name( title=media_data.name if isinstance(media_data, Movie) else media_data.series_name, release_date=media_data.release_date, season_number=None if isinstance(media_data, Movie) else media_data.season_number, episode_number=None if isinstance(media_data, Movie) else media_data.episode_number, episode_name=None if isinstance(media_data, Movie) else media_data.episode_name, media_source=scraper.abbreviation, ) successful_downloads: list[SubtitlesData] = [] failed_downloads: list[SubtitlesDownloadError] = [] with TemporaryDirectory(directory_name=temp_dir_name) as temp_download_path: temp_downloads: list[Path] = [] if not media_data.playlist: raise PlaylistLoadError("No playlist was found for provided media data.") main_playlist = await scraper.load_playlist(url=media_data.playlist) # type: ignore[func-returns-value] if not main_playlist: raise PlaylistLoadError("Failed to load the main playlist.") matching_subtitles = scraper.find_matching_subtitles(main_playlist=main_playlist, # type: ignore[var-annotated] language_filter=language_filter) # If no matching subtitles were found, there's no need to continue if not matching_subtitles: return SubtitlesDownloadResults( media_data=media_data, successful_subtitles=successful_downloads, failed_subtitles=failed_downloads, is_zip=zip, ) logger.info(f"{len(matching_subtitles)} matching subtitles were found.", extra={"hide_when_interactive": True}) downloaded_subtitles: list[str] = [] progress_log = Text(f"Downloaded subtitles ({len(downloaded_subtitles)}/{len(matching_subtitles)}):") downloads_list = Text() progress_bar = Progress( SpinnerColumn(), TextColumn("[progress.description]{task.description}"), BarColumn(), TextColumn("[progress.percentage][yellow]{task.percentage:>3.0f}%[/yellow]"), TextColumn("[yellow]{task.completed}/{task.total}[/yellow]"), MinsAndSecsTimeElapsedColumn(), console=console, ) task = progress_bar.add_task("Starting download", total=len(matching_subtitles)) with conditional_live( Group(progress_log, downloads_list, Text(), progress_bar), # Empty 'Text' for line spacing ) as live: for matching_subtitles_item in matching_subtitles: language_info = scraper.format_subtitles_description( subtitles_media=matching_subtitles_item, ) if live: progress_bar.update(task, advance=1, description=f"Processing [magenta]{language_info}[/magenta]") try: subtitles_data = await scraper.download_subtitles(media_data=matching_subtitles_item, subrip_conversion=convert_to_srt) except Exception as e: if isinstance(e, SubtitlesDownloadError): failed_downloads.append(e) original_error = e.original_exc else: original_error = e logger.warning(f"Failed to download '{language_info}' subtitles: {original_error}") logger.debug("Debug information:", exc_info=original_error) continue try: temp_downloads.append(download_subtitles_to_file( media_data=media_data, subtitles_data=subtitles_data, output_path=temp_download_path, source_abbreviation=scraper.abbreviation, overwrite=overwrite_existing, )) downloaded_subtitles.append(f"• {language_info}") if live: progress_log.plain = (f"Downloaded subtitles " f"({len(downloaded_subtitles)}/{len(matching_subtitles)}):") downloads_list.plain = f"{format_list(downloaded_subtitles, width=live.console.width)}" logger.info(f"{language_info} subtitles were successfully downloaded.", extra={"hide_when_interactive": True}) successful_downloads.append(subtitles_data) except Exception as e: logger.warning(f"Failed to save '{language_info}' subtitles: {e}") logger.debug("Debug information:", exc_info=True) failed_downloads.append( SubtitlesDownloadError( language_code=subtitles_data.language_code, language_name=subtitles_data.language_name, special_type=subtitles_data.special_type, original_exc=e, ), ) if live: progress_bar.update(task, visible=False) if not zip or len(temp_downloads) == 1: for file_path in temp_downloads: if overwrite_existing: new_path = download_path / file_path.name else: new_path = generate_non_conflicting_path(file_path=download_path / file_path.name) shutil.move(src=file_path, dst=new_path) elif len(temp_downloads) > 0: zip_path = Path(shutil.make_archive( base_name=str(temp_download_path.parent / temp_download_path.name), format="zip", root_dir=temp_download_path, )) file_name = format_release_name( title=media_data.name if isinstance(media_data, Movie) else media_data.series_name, release_date=media_data.release_date, season_number=None if isinstance(media_data, Movie) else media_data.season_number, episode_number=None if isinstance(media_data, Movie) else media_data.episode_number, episode_name=None if isinstance(media_data, Movie) else media_data.episode_name, media_source=scraper.abbreviation, file_format="zip", ) if overwrite_existing: destination_path = download_path / file_name else: destination_path = generate_non_conflicting_path(file_path=download_path / file_name) shutil.move(src=zip_path, dst=destination_path) return SubtitlesDownloadResults( media_data=media_data, successful_subtitles=successful_downloads, failed_subtitles=failed_downloads, is_zip=zip, ) ================================================ FILE: isubrip/config.py ================================================ from __future__ import annotations from abc import ABC from pathlib import Path from typing import TYPE_CHECKING, Literal from pydantic import AliasGenerator, BaseModel, ConfigDict, Field, create_model, field_validator from pydantic_core import PydanticCustomError from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict, TomlConfigSettingsSource from isubrip.scrapers.scraper import DefaultScraperConfig, ScraperFactory class ConfigCategory(BaseModel, ABC): """A base class for settings categories.""" model_config = ConfigDict( extra='allow', alias_generator=AliasGenerator( validation_alias=lambda field_name: field_name.replace('_', '-'), ), ) class GeneralCategory(ConfigCategory): check_for_updates: bool = Field(default=True) verbose: bool = Field(default=False) log_level: Literal["debug", "info", "warning", "error", "critical"] = Field(default="info") log_rotation_size: int = Field(default=15) class DownloadsCategory(ConfigCategory): folder: Path = Field(default=Path.cwd().resolve()) languages: list[str] = Field(default=[]) overwrite_existing: bool = Field(default=False) zip: bool = Field(default=False) @field_validator('folder') @classmethod def assure_path_exists(cls, value: Path) -> Path: if value.exists(): if not value.is_dir(): raise PydanticCustomError( "invalid_path", "Path is not a directory.", ) else: raise PydanticCustomError( "invalid_path", "Path does not exist.") return value class WebVTTSubcategory(ConfigCategory): subrip_alignment_conversion: bool = Field(default=False) class SubtitlesCategory(ConfigCategory): fix_rtl: bool = Field(default=False) remove_duplicates: bool = Field(default=True) convert_to_srt: bool = Field(default=False) webvtt: WebVTTSubcategory = WebVTTSubcategory() class ScrapersCategory(ConfigCategory): default: DefaultScraperConfig = Field(default_factory=DefaultScraperConfig) # Resolve mypy errors as mypy doesn't support dynamic models. if TYPE_CHECKING: DynamicScrapersCategory = ScrapersCategory else: # A config model that's dynamically created based on the available scrapers and their configurations. DynamicScrapersCategory = create_model( 'DynamicScrapersCategory', __base__=ScrapersCategory, **{ scraper.id: (scraper.ScraperConfig, Field(default_factory=scraper.ScraperConfig)) for scraper in ScraperFactory.get_scraper_classes() }, # type: ignore[call-overload] ) class Config(BaseSettings): model_config = SettingsConfigDict( extra='forbid', ) general: GeneralCategory = Field(default_factory=GeneralCategory) downloads: DownloadsCategory = Field(default_factory=DownloadsCategory) subtitles: SubtitlesCategory = Field(default_factory=SubtitlesCategory) scrapers: DynamicScrapersCategory = Field(default_factory=DynamicScrapersCategory) @classmethod def settings_customise_sources( cls, settings_cls: type[BaseSettings], init_settings: PydanticBaseSettingsSource, env_settings: PydanticBaseSettingsSource, dotenv_settings: PydanticBaseSettingsSource, file_secret_settings: PydanticBaseSettingsSource, ) -> tuple[PydanticBaseSettingsSource, ...]: return ( init_settings, TomlConfigSettingsSource(settings_cls), env_settings, dotenv_settings, file_secret_settings, ) ================================================ FILE: isubrip/constants.py ================================================ from __future__ import annotations from functools import lru_cache from pathlib import Path from tempfile import gettempdir # General PACKAGE_NAME = "isubrip" PACKAGE_VERSION = "2.6.8" SCRAPER_MODULES_SUFFIX = "_scraper" USER_CONFIG_FILE_NAME = "config.toml" @lru_cache(maxsize=1) def data_folder_path() -> Path: return Path.home() / f".{PACKAGE_NAME}" @lru_cache(maxsize=1) def temp_folder_path() -> Path: return Path(gettempdir()) / PACKAGE_NAME @lru_cache(maxsize=1) def user_config_file_path() -> Path: return data_folder_path() / USER_CONFIG_FILE_NAME # Logging Paths @lru_cache(maxsize=1) def log_files_path() -> Path: return data_folder_path() / "logs" # Other WINDOWS_RESERVED_FILE_NAMES = frozenset( ["CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9"], ) RTL_LANGUAGES = frozenset(["ar", "arc", "az", "dv", "he", "ks", "ku", "fa", "ur", "yi"]) ================================================ FILE: isubrip/data_structures.py ================================================ from __future__ import annotations from abc import ABC import datetime as dt # noqa: TC003 from enum import Enum from typing import TYPE_CHECKING, Generic, Literal, NamedTuple, TypeVar import m3u8 from pydantic import BaseModel if TYPE_CHECKING: from isubrip.scrapers.scraper import SubtitlesDownloadError T = TypeVar("T") MainPlaylist = TypeVar("MainPlaylist", bound=m3u8.M3U8) PlaylistMediaItem = TypeVar("PlaylistMediaItem", bound=m3u8.Media) MediaData = TypeVar("MediaData", bound="MediaBase") class SubtitlesDownloadResults(NamedTuple): """ A named tuple containing download results. Attributes: media_data (Movie | Episode): An object containing metadata about the media the subtitles were downloaded for. successful_subtitles (list[SubtitlesData]): List of subtitles that were successfully downloaded. failed_subtitles (list[SubtitlesData]): List of subtitles that failed to download. is_zip (bool): Whether the subtitles were saved in a zip file. """ media_data: Movie | Episode successful_subtitles: list[SubtitlesData] failed_subtitles: list[SubtitlesDownloadError] is_zip: bool class SubtitlesFormat(BaseModel): """ An object containing subtitles format data. Attributes: name (str): Name of the format. file_extension (str): File extension of the format. """ name: str file_extension: str class SubtitlesFormatType(Enum): """ An Enum representing subtitles formats. Attributes: SUBRIP (SubtitlesFormat): SubRip format. WEBVTT (SubtitlesFormat): WebVTT format. """ SUBRIP = SubtitlesFormat(name="SubRip", file_extension="srt") WEBVTT = SubtitlesFormat(name="WebVTT", file_extension="vtt") class SubtitlesType(Enum): """ Subtitles special type. Attributes: CC (SubtitlesType): Closed captions. FORCED (SubtitlesType): Forced subtitles. """ CC = "CC" FORCED = "Forced" class SubtitlesData(BaseModel): """ An object containing subtitles data and metadata. Attributes: language_code (str): Language code of the language the subtitles are in. language_name (str | None, optional): Name of the language the subtitles are in. subtitles_format (SubtitlesFormatType): Format of the subtitles. content (bytes): Content of the subtitles in binary format. content_encoding (str): Encoding of subtitles content (ex. "utf-8"). special_type (SubtitlesType | None, optional): Type of the subtitles, if they're not regular. Defaults to None. """ language_code: str subtitles_format: SubtitlesFormatType content: bytes content_encoding: str language_name: str | None = None special_type: SubtitlesType | None = None class ConfigDict: str_strip_whitespace = True class MediaBase(BaseModel, ABC): """A base class for media objects.""" class Movie(MediaBase): """ An object containing movie metadata. Attributes: id (str | None, optional): ID of the movie on the service it was scraped from. Defaults to None. referrer_id (str | None, optional): ID of the movie on the original referring service. Defaults to None. name (str): Title of the movie. release_date (datetime | int | None, optional): Release date (datetime), or year (int) of the movie. Defaults to None. duration (timedelta | None, optional): Duration of the movie. Defaults to None. preorder_availability_date (datetime | None, optional): Date when the movie will be available for pre-order on the service it was scraped from. None if not a pre-order. Defaults to None. playlist (str | None, optional): Main playlist URL(s). """ media_type: Literal["movie"] = "movie" name: str release_date: dt.datetime | int id: str | None = None referrer_id: str | None = None duration: dt.timedelta | None = None preorder_availability_date: dt.datetime | None = None playlist: str | list[str] | None = None class Episode(MediaBase): """ An object containing episode metadata. Attributes: id (str | None, optional): ID of the episode on the service it was scraped from. Defaults to None. referrer_id (str | None, optional): ID of the episode on the original referring service. Defaults to None. series_name (str): Name of the series the episode is from. series_release_date (datetime | int | None, optional): Release date (datetime), or year (int) of the series. Defaults to None. season_number (int): Season number. season_name (str | None, optional): Season name. Defaults to None. episode_number (int): Episode number. episode_name (str | None, optional): Episode name. Defaults to None. episode_release_date (datetime | None): Release date of the episode. Defaults to None. episode_duration (timedelta | None, optional): Duration of the episode. Defaults to None. playlist (str | None, optional): Main playlist URL(s). """ media_type: Literal["episode"] = "episode" series_name: str season_number: int episode_number: int id: str | None = None referrer_id: str | None = None series_release_date: dt.datetime | int | None = None season_name: str | None = None release_date: dt.datetime | None = None duration: dt.timedelta | None = None episode_name: str | None = None episode_release_date: dt.datetime | None = None episode_duration: dt.timedelta | None = None playlist: str | list[str] | None = None class Season(MediaBase): """ An object containing season metadata. Attributes: id (str | None, optional): ID of the season on the service it was scraped from. Defaults to None. referrer_id (str | None, optional): ID of the season on the original referring service. Defaults to None. series_name (str): Name of the series the season is from. season_number (int): Season number. series_release_date (datetime | int | None, optional): Release date (datetime), or year (int) of the series. Defaults to None. season_name (str | None, optional): Season name. Defaults to None. season_release_date (datetime | None, optional): Release date of the season, or release year. Defaults to None. episodes (list[Episode]): A list of episode objects containing metadata about episodes of the season. """ media_type: Literal["season"] = "season" series_name: str season_number: int id: str | None = None referrer_id: str | None = None series_release_date: dt.datetime | int | None = None season_name: str | None = None season_release_date: dt.datetime | int | None = None episodes: list[Episode] = [] class Series(MediaBase): """ An object containing series metadata. Attributes: id (str | None, optional): ID of the series on the service it was scraped from. Defaults to None. series_name (str): Series name. referrer_id (str | None, optional): ID of the series on the original referring service. Defaults to None. series_release_date (datetime | int | None, optional): Release date (datetime), or year (int) of the series. Defaults to None. seasons (list[Season]): A list of season objects containing metadata about seasons of the series. """ media_type: Literal["series"] = "series" series_name: str seasons: list[Season] = [] id: str | None = None referrer_id: str | None = None series_release_date: dt.datetime | int | None = None class ScrapedMediaResponse(BaseModel, Generic[MediaData]): """ An object containing scraped media data and metadata. Attributes: media_data (list[Movie] | list[Episode] | list[Season] | list[Series]): An object containing the scraped media data. metadata_scraper (str): ID of the scraper that was used to scrape metadata. playlist_scraper (str): ID of the scraper that should be used to parse and scrape the playlist. original_data (dict): Original raw data from the API that was used to extract media's data. """ media_data: list[MediaData] metadata_scraper: str playlist_scraper: str original_data: dict ================================================ FILE: isubrip/logger.py ================================================ from __future__ import annotations import datetime as dt from functools import lru_cache import logging import re from typing import TYPE_CHECKING, ClassVar from rich.highlighter import NullHighlighter from rich.logging import RichHandler from isubrip.cli import console from isubrip.constants import ( PACKAGE_NAME, ) if TYPE_CHECKING: from pathlib import Path from rich.console import Console BBCOE_REGEX = re.compile( r"(?i)(?P\[(?P[a-z#@][^[]*?)])(?P.*)(?P\[/(?P=tag_name)])") LOG_FILE_METADATA = "[%(asctime)s | %(levelname)s | %(threadName)s | %(filename)s::%(funcName)s::%(lineno)d] " def set_logger(_logger: logging.Logger) -> None: """ Set an external logger to be used by the package. Args: _logger (logging.Logger): A logger instance to be used by the package. """ global logger logger = _logger class CustomStdoutFormatter(RichHandler): """ Custom formatter for stdout logging with Rich integration. This formatter adds color to log messages based on their level and supports hiding messages in interactive mode. """ LEVEL_COLORS: ClassVar[dict[int, str]] = { logging.ERROR: "red", logging.WARNING: "dark_orange", logging.DEBUG: "grey54", } def __init__(self, console: Console | None = None, debug_mode: bool = False) -> None: """ Initialize the stdout formatter. Args: console (Console | None, optional): Rich console instance to use for output. Defaults to None. debug_mode (bool, optional): Whether to show additional debug information. Defaults to False. """ super().__init__( console=console, show_time=debug_mode, show_level=debug_mode, show_path=debug_mode, highlighter=NullHighlighter(), markup=True, log_time_format="%H:%M:%S", rich_tracebacks=debug_mode, tracebacks_extra_lines=0, ) self._console = console def emit(self, record: logging.LogRecord) -> None: """ Emit a log record, respecting the 'hide_when_interactive' flag. Args: record (LogRecord): The log record to emit. """ # Skip emission if record is marked to be hidden in interactive mode if getattr(record, 'hide_when_interactive', False) and self._console and self._console.is_interactive: return super().emit(record) def format(self, record: logging.LogRecord) -> str: """ Format the log record with appropriate color based on level. Args: record (LogRecord): The log record to format. Returns: str: Formatted log message with Rich markup. """ # Get the message once message = record.getMessage() # Apply color based on log level using the class variable mapping if color := self.LEVEL_COLORS.get(record.levelno): record.msg = f"[{color}]{message}[/{color}]" return super().format(record) class CustomLogFileFormatter(logging.Formatter): """ Custom formatter for log files that removes Rich markup tags. """ def __init__(self) -> None: """ Initialize the formatter with metadata format but without message part. We'll append the message manually to avoid issues with special characters. """ super().__init__( fmt=LOG_FILE_METADATA, datefmt=r"%Y-%m-%d %H:%M:%S", ) @staticmethod @lru_cache(maxsize=64) def _remove_rich_markup(text: str) -> str: """ Remove Rich markup tags from text efficiently with caching. Args: text: Text containing Rich markup tags Returns: Text with Rich markup tags removed """ while match := BBCOE_REGEX.search(text): text = text[:match.start()] + match.group('content') + text[match.end():] return text def format(self, record: logging.LogRecord) -> str: """ Format the log record for file output, removing Rich markup. This implementation uses the standard formatter for the metadata part and then appends the message without formatting to avoid issues with special characters within the log message. Args: record: The log record to format Returns: Formatted log message suitable for file output """ message = record.getMessage() clean_message = self._remove_rich_markup(message) # Store the original message original_msg = record.msg original_args = record.args # Temporarily set an empty message to format just the metadata record.msg = "" record.args = None # Format the metadata part using the standard formatter metadata = super().format(record) # Restore the original message and args record.msg = original_msg record.args = original_args # Combine metadata and message without formatting the message return metadata + clean_message def setup_loggers(stdout_output: bool = True, stdout_console: Console | None = None, stdout_loglevel: int = logging.INFO, logfile_output: bool = False, logfile_output_path: Path | None = None, logfile_loglevel: int = logging.DEBUG) -> None: """ Configure loggers for both stdout and file output. Args: stdout_output (bool, optional): Whether to output logs to STDOUT. Defaults to True. stdout_console (Console | None, optional): A Rich console instance to be used for STDOUT logging. Relevant only if `stdout_output` is True. Defaults to None. stdout_loglevel (int, optional): Log level for STDOUT logger. Relevant only if `stdout_output` is True. Defaults to logging.INFO. logfile_output (bool, optional): Whether to output logs to a logfile. Defaults to True. logfile_output_path (Path | None, optional): Path to the directory where log files will be saved. Required only if `logfile_output` is True. Defaults to None. logfile_loglevel (int, optional): Log level for logfile logger. Relevant only if `logfile_output` is True. Defaults to logging.DEBUG. """ logger.handlers.clear() # Remove and reset existing handlers logger.setLevel(logging.DEBUG) if stdout_output: debug_mode = (stdout_loglevel == logging.DEBUG) stdout_handler = CustomStdoutFormatter( debug_mode=debug_mode, console=stdout_console, ) stdout_handler.setLevel(stdout_loglevel) logger.addHandler(stdout_handler) if logfile_output: if not logfile_output_path: raise ValueError("Missing required 'logfile_output_path' argument (required when 'logfile_output' is True.") if not logfile_output_path.is_dir(): logger.debug("Logs directory could not be found and will be created.") logfile_output_path.mkdir() logfile_path = logfile_output_path / f"{PACKAGE_NAME}_{dt.datetime.now().strftime(r'%Y-%m-%d_%H-%M-%S')}.log" logfile_handler = logging.FileHandler(filename=logfile_path, encoding="utf-8") logfile_handler.setLevel(logfile_loglevel) logfile_handler.setFormatter(CustomLogFileFormatter()) logger.debug(f"Log file location: '{logfile_path}'") logger.addHandler(logfile_handler) logger = logging.getLogger(PACKAGE_NAME) # Temporarily set the logger to INFO level until the config is loaded and the logger is properly set up logger.setLevel(logging.INFO) logger.addHandler(CustomStdoutFormatter(console=console)) ================================================ FILE: isubrip/scrapers/__init__.py ================================================ ================================================ FILE: isubrip/scrapers/appletv_scraper.py ================================================ from __future__ import annotations import datetime as dt from enum import Enum import fnmatch import re from typing import Any from httpx import HTTPError from isubrip.data_structures import Episode, Movie, ScrapedMediaResponse, Season, Series from isubrip.logger import logger from isubrip.scrapers.scraper import HLSScraper, ScraperError from isubrip.subtitle_formats.webvtt import WebVTTSubtitles from isubrip.utils import convert_epoch_to_datetime, parse_url_params, raise_for_status class AppleTVScraper(HLSScraper): """An Apple TV scraper.""" id = "appletv" name = "Apple TV" abbreviation = "ATV" url_regex = re.compile(r"(?i)(?Phttps?://tv\.apple\.com/(?:(?P[a-z]{2})/)?(?Pmovie|episode|season|show)/(?:(?P[\w\-%]+)/)?(?Pumc\.cmc\.[a-z\d]{23,25}))(?:\?(?P.*))?") subtitles_class = WebVTTSubtitles is_movie_scraper = True is_series_scraper = True uses_scrapers = ["itunes"] default_storefront = "US" storefronts_mapping = { "AE": "143481", "AG": "143540", "AI": "143538", "AM": "143524", "AR": "143505", "AT": "143445", "AU": "143460", "AZ": "143568", "BE": "143446", "BG": "143526", "BH": "143559", "BM": "143542", "BN": "143560", "BO": "143556", "BR": "143503", "BS": "143539", "BW": "143525", "BY": "143565", "BZ": "143555", "CA": "143455", "CH": "143459", "CL": "143483", "CO": "143501", "CR": "143495", "CV": "143580", "CY": "143557", "CZ": "143489", "DE": "143443", "DK": "143458", "DM": "143545", "DO": "143508", "EC": "143509", "EE": "143518", "EG": "143516", "ES": "143454", "FI": "143447", "FJ": "143583", "FM": "143591", "FR": "143442", "GB": "143444", "GD": "143546", "GH": "143573", "GM": "143584", "GR": "143448", "GT": "143504", "GW": "143585", "HK": "143463", "HN": "143510", "HU": "143482", "ID": "143476", "IE": "143449", "IL": "143491", "IN": "143467", "IT": "143450", "JO": "143528", "JP": "143462", "KH": "143579", "KN": "143548", "KR": "143466", "KY": "143544", "LA": "143587", "LB": "143497", "LK": "143486", "LT": "143520", "LU": "143451", "LV": "143519", "MD": "143523", "MN": "143592", "MO": "143515", "MT": "143521", "MU": "143533", "MX": "143468", "MY": "143473", "MZ": "143593", "NA": "143594", "NE": "143534", "NI": "143512", "NL": "143452", "NO": "143457", "NZ": "143461", "OM": "143562", "PA": "143485", "PE": "143507", "PH": "143474", "PL": "143478", "PT": "143453", "PY": "143513", "QA": "143498", "RU": "143469", "SA": "143479", "SE": "143456", "SG": "143464", "SI": "143499", "SK": "143496", "SV": "143506", "SZ": "143602", "TH": "143475", "TJ": "143603", "TM": "143604", "TR": "143480", "TT": "143551", "TW": "143470", "UA": "143492", "UG": "143537", "US": "143441", "VE": "143502", "VG": "143543", "VN": "143471", "ZA": "143472", "ZW": "143605", } _api_base_url = "https://tv.apple.com/api/uts/v3" _api_base_params = { "utscf": "OjAAAAAAAAA~", "caller": "web", "v": "84", "pfm": "web", } class Channel(Enum): """ An Enum representing AppleTV channels. Value represents the channel ID as used by the API. """ APPLE_TV_PLUS = "tvs.sbd.4000" DISNEY_PLUS = "tvs.sbd.1000216" ITUNES = "tvs.sbd.9001" HULU = "tvs.sbd.10000" MAX = "tvs.sbd.9050" NETFLIX = "tvs.sbd.9000" PRIME_VIDEO = "tvs.sbd.12962" STARZ = "tvs.sbd.1000308" def __init__(self, *args: Any, **kwargs: Any) -> None: super().__init__(*args, **kwargs) self._storefronts_request_params_cache: dict[str, dict[str, str]] = {} def _decide_locale(self, preferred_locales: str | list[str], default_locale: str, locales: list[str]) -> str: """ Decide which locale to use. Args: preferred_locales (str | list[str]): The preferred locales to use. default_locale (str): The default locale to use if there is no match. locales (list[str]): The locales to search in. Returns: str: The locale to use. """ if isinstance(preferred_locales, str): preferred_locales = [preferred_locales] for locale in preferred_locales: if locale in locales: return locale.replace("_", "-") if result := fnmatch.filter(locales, "en_*"): return result[0].replace("_", "-") return default_locale async def _fetch_api_data(self, storefront_id: str, endpoint: str, additional_params: dict | None = None) -> dict: """ Send a request to AppleTV's API and return the JSON response. Args: endpoint (str): The endpoint to send the request to. additional_params (dict[str, str]): Additional parameters to send with the request. Returns: dict: The JSON response. Raises: HttpError: If an HTTP error response is received. """ request_params = await self._fetch_request_params(storefront_id=storefront_id) if additional_params: request_params.update(additional_params) response = await self._client.get(url=f"{self._api_base_url}{endpoint}", params=request_params) try: raise_for_status(response) except HTTPError as e: if response.status_code == 404: raise ScraperError( "Media not found. This could indicate that the provided URL is invalid.", ) from e raise response_json: dict = response.json() response_data: dict = response_json.get("data", {}) return response_data async def _fetch_request_params(self, storefront_id: str) -> dict[str, str]: """ Fetch from the API request parameters for the given storefront ID. Uses caching with `self._storefronts_request_params_cache` for efficiency. Args: storefront_id (str): The ID of the storefront to fetch the request parameters for. Returns: dict: The request parameters for the given storefront ID. If returned from cache, a copy is returned. """ if storefront_cached_params := self._storefronts_request_params_cache.get(storefront_id): logger.debug(f"Using cached request parameters for storefront '{storefront_id}':" f"'{storefront_cached_params}'.") return storefront_cached_params.copy() configuration_data = await self._get_configuration_data(storefront_id=storefront_id) request_params: dict[str, str] = configuration_data["applicationProps"]["requiredParamsMap"]["Default"] default_locale: str = configuration_data["applicationProps"]["storefront"]["defaultLocale"] available_locales: list[str] = configuration_data["applicationProps"]["storefront"]["localesSupported"] logger.debug(f"Available locales for storefront '{storefront_id}': {available_locales}'. " f"Storefront's default locale: '{default_locale}'.") locale = self._decide_locale( preferred_locales=["en_US", "en_GB"], default_locale=default_locale, locales=available_locales, ) request_params["sf"] = storefront_id request_params["locale"] = locale logger.debug(f"Using and caching request parameters for storefront '{storefront_id}': {request_params}") self._storefronts_request_params_cache[storefront_id] = request_params.copy() return request_params async def _get_configuration_data(self, storefront_id: str) -> dict: """ Get configuration data for the given storefront ID. Args: storefront_id (str): The ID of the storefront to get the configuration data for. Returns: dict: Configuration data as returned by the API for the given storefront ID. """ logger.debug(f"Fetching configuration data for storefront '{storefront_id}'...") url = f"{self._api_base_url}/configurations" params = self._api_base_params.copy() params["sf"] = storefront_id response = await self._client.get(url=url, params=params) raise_for_status(response) logger.debug("Configuration data fetched successfully.") response_data: dict = response.json()["data"] return response_data def _map_playables_by_channel(self, playables: list[dict]) -> dict[str, dict]: """ Map playables by channel name. Args: playables (list[dict]): Playables data to map. Returns: dict: The mapped playables (in a `channel_name (str): [playables]` format). """ mapped_playables: dict = {} for playable in playables: if channel_id := playable.get("channelId"): mapped_playables.setdefault(channel_id, []).append(playable) return mapped_playables async def get_movie_data(self, storefront_id: str, movie_id: str) -> ScrapedMediaResponse[Movie]: data = await self._fetch_api_data( storefront_id=storefront_id, endpoint=f"/movies/{movie_id}", ) mapped_playables = self._map_playables_by_channel(playables=data["playables"].values()) logger.debug(f"Available channels for movie '{movie_id}': " f"{' '.join(list(mapped_playables.keys()))}") if self.Channel.ITUNES.value not in mapped_playables: if self.Channel.APPLE_TV_PLUS.value in mapped_playables: raise ScraperError("Scraping AppleTV+ content is not currently supported.") raise ScraperError("No iTunes playables could be found.") return_data = [] for playable_data in mapped_playables[self.Channel.ITUNES.value]: return_data.append(self._extract_itunes_movie_data(playable_data)) if len(return_data) > 1: logger.debug(f"{len(return_data)} iTunes playables were found for movie '{movie_id}'.") return ScrapedMediaResponse( media_data=return_data, metadata_scraper=self.id, playlist_scraper="itunes", original_data=data, ) def _extract_itunes_movie_data(self, playable_data: dict) -> Movie: """ Extract movie data from an AppleTV's API iTunes playable data. Args: playable_data (dict): The playable data from the AppleTV API. Returns: Movie: A Movie object. """ itunes_movie_id = playable_data["itunesMediaApiData"]["id"] appletv_movie_id = playable_data["canonicalId"] movie_title = playable_data["canonicalMetadata"]["movieTitle"] movie_release_date = convert_epoch_to_datetime(playable_data["canonicalMetadata"]["releaseDate"] // 1000) movie_playlists = [] movie_duration = None if offers := playable_data["itunesMediaApiData"].get("offers"): for offer in offers: if (playlist := offer.get("hlsUrl")) and offer["hlsUrl"] not in movie_playlists: movie_playlists.append(playlist) if movie_duration_int := offers[0].get("durationInMilliseconds"): movie_duration = dt.timedelta(milliseconds=movie_duration_int) if movie_expected_release_date := playable_data["itunesMediaApiData"].get("futureRentalAvailabilityDate"): movie_expected_release_date = dt.datetime.strptime(movie_expected_release_date, "%Y-%m-%d") return Movie( id=itunes_movie_id, referrer_id=appletv_movie_id, name=movie_title, release_date=movie_release_date, duration=movie_duration, preorder_availability_date=movie_expected_release_date, playlist=movie_playlists if movie_playlists else None, ) async def get_episode_data(self, storefront_id: str, episode_id: str) -> ScrapedMediaResponse[Episode]: raise NotImplementedError("Series scraping is not currently supported.") async def get_season_data(self, storefront_id: str, season_id: str, show_id: str) -> ScrapedMediaResponse[Season]: raise NotImplementedError("Series scraping is not currently supported.") async def get_show_data(self, storefront_id: str, show_id: str) -> ScrapedMediaResponse[Series]: raise NotImplementedError("Series scraping is not currently supported.") async def get_data(self, url: str) -> ScrapedMediaResponse: regex_match = self.match_url(url=url, raise_error=True) url_data = regex_match.groupdict() media_type = url_data["media_type"] if storefront_code := url_data.get("country_code"): storefront_code = storefront_code.upper() else: storefront_code = self.default_storefront media_id = url_data["media_id"] if storefront_code not in self.storefronts_mapping: raise ScraperError(f"ID mapping for storefront '{storefront_code}' could not be found.") storefront_id = self.storefronts_mapping[storefront_code] if media_type == "movie": return await self.get_movie_data(storefront_id=storefront_id, movie_id=media_id) if media_type == "episode": return await self.get_episode_data(storefront_id=storefront_id, episode_id=media_id) if media_type == "season": if (url_params := url_data.get("url_params")) and (show_id := parse_url_params(url_params).get("showId")): return await self.get_season_data(storefront_id=storefront_id, season_id=media_id, show_id=show_id) raise ScraperError("Invalid AppleTV URL: Missing 'showId' parameter.") if media_type == "show": return await self.get_show_data(storefront_id=storefront_id, show_id=media_id) raise ScraperError(f"Invalid media type '{media_type}'.") ================================================ FILE: isubrip/scrapers/itunes_scraper.py ================================================ from __future__ import annotations import asyncio import re from typing import TYPE_CHECKING, Any from isubrip.logger import logger from isubrip.scrapers.scraper import HLSScraper, ScraperError, ScraperFactory from isubrip.subtitle_formats.webvtt import WebVTTSubtitles if TYPE_CHECKING: from m3u8.model import Media from isubrip.data_structures import Movie, ScrapedMediaResponse REDIRECT_MAX_RETRIES = 5 REDIRECT_SLEEP_TIME = 2 class ItunesScraper(HLSScraper): """An iTunes movie data scraper.""" id = "itunes" name = "iTunes" abbreviation = "iT" url_regex = re.compile(r"(?i)(?Phttps?://itunes\.apple\.com/(?:(?P[a-z]{2})/)?(?Pmovie|tv-show|tv-season|show)/(?:(?P[\w\-%]+)/)?(?Pid\d{9,10}))(?:\?(?P.*))?") subtitles_class = WebVTTSubtitles is_movie_scraper = True uses_scrapers = ["appletv"] _subtitles_filters = { HLSScraper.M3U8Attribute.GROUP_ID.value: ["subtitles_ak", "subtitles_vod-ak-amt.tv.apple.com"], **HLSScraper._subtitles_filters, # noqa: SLF001 } def __init__(self, *args: Any, **kwargs: Any) -> None: super().__init__(*args, **kwargs) self._appletv_scraper = ScraperFactory.get_scraper_instance( scraper_id="appletv", raise_error=True, ) async def get_data(self, url: str) -> ScrapedMediaResponse[Movie]: """ Scrape iTunes to find info about a movie, and it's M3U8 main_playlist. Args: url (str): An iTunes store movie URL. Raises: InvalidURL: `itunes_url` is not a valid iTunes store movie URL. PageLoadError: HTML page did not load properly. HTTPError: HTTP request failed. Returns: Movie: A Movie (NamedTuple) object with movie's name, and an M3U8 object of the main_playlist if the main_playlist is found. None otherwise. """ regex_match = self.match_url(url, raise_error=True) url_data = regex_match.groupdict() country_code: str = url_data["country_code"] media_id: str = url_data["media_id"] appletv_redirect_finding_url = f"https://tv.apple.com/{country_code}/movie/{media_id}" logger.debug("Attempting to fetch redirect location from: " + appletv_redirect_finding_url) retries = 0 while True: response = await self._client.get(url=appletv_redirect_finding_url, follow_redirects=False) if response.status_code != 301 and retries < REDIRECT_MAX_RETRIES: retries += 1 logger.debug(f"AppleTV redirect URL not found (Response code: {response.status_code})," f" retrying... ({retries}/{REDIRECT_MAX_RETRIES})") await asyncio.sleep(REDIRECT_SLEEP_TIME) continue break redirect_location = response.headers.get("Location") if response.status_code != 301 or not redirect_location: raise ScraperError(f"AppleTV redirect URL not found (Response code: {response.status_code}).") # Add 'https:' if redirect_location starts with '//' if redirect_location.startswith('//'): redirect_location = "https:" + redirect_location logger.debug(f"Redirect URL: {redirect_location}") if not self._appletv_scraper.match_url(redirect_location): raise ScraperError("Redirect URL is not a valid AppleTV URL.") return await self._appletv_scraper.get_data(url=redirect_location) @staticmethod def parse_language_name(media_data: Media) -> str | None: name: str | None = media_data.name if name: return name.replace(' (forced)', '').strip() return None ================================================ FILE: isubrip/scrapers/scraper.py ================================================ from __future__ import annotations from abc import ABC, abstractmethod import asyncio from enum import Enum import importlib import inspect from pathlib import Path import re import sys from typing import TYPE_CHECKING, Any, ClassVar, Literal, TypeVar, overload import httpx import m3u8 from pydantic import AliasGenerator, BaseModel, ConfigDict, Field, create_model from isubrip.constants import PACKAGE_NAME, SCRAPER_MODULES_SUFFIX from isubrip.data_structures import ( MainPlaylist, PlaylistMediaItem, ScrapedMediaResponse, SubtitlesData, SubtitlesFormatType, SubtitlesType, ) from isubrip.logger import logger from isubrip.utils import ( SingletonMeta, format_subtitles_description, get_model_field, merge_dict_values, return_first_valid, single_string_to_list, ) if TYPE_CHECKING: from isubrip.subtitle_formats.subtitles import Subtitles ScraperT = TypeVar("ScraperT", bound="Scraper") class ScraperConfigBase(BaseModel, ABC): """ A Pydantic BaseModel for base class for scraper's configuration classes. Also serves for setting default configuration settings for all scrapers. Attributes: timeout (int | float): Timeout to use when making requests. user_agent (st): User agent to use when making requests. proxy (str | None): Proxy to use when making requests. verify_ssl (bool): Whether to verify SSL certificates. """ model_config = ConfigDict( extra='forbid', alias_generator=AliasGenerator( validation_alias=lambda field_name: field_name.replace('_', '-'), ), ) timeout: int | float | None = Field(default=None) user_agent: str | None = Field(default=None) proxy: str | None = Field(default=None) verify_ssl: bool | None = Field(default=None) class DefaultScraperConfig(ScraperConfigBase): """ A Pydantic BaseModel for scraper's configuration classes. Also serves as a default configuration for all scrapers. Attributes: timeout (int | float): Timeout to use when making requests. user_agent (st): User agent to use when making requests. proxy (str | None): Proxy to use when making requests. verify_ssl (bool): Whether to verify SSL certificates. """ timeout: int | float = Field(default=10) user_agent: str = Field( default="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36", # noqa: E501 ) proxy: str | None = Field(default=None) verify_ssl: bool = Field(default=True) class ScraperConfigSubcategory(BaseModel, ABC): """A Pydantic BaseModel for a scraper's configuration subcategory (which can be set under 'ScraperConfig').""" model_config = ConfigDict( extra='forbid', alias_generator=AliasGenerator( validation_alias=lambda field_name: field_name.replace('_', '-'), ), ) class Scraper(ABC, metaclass=SingletonMeta): """ A base class for scrapers. Attributes: default_user_agent (str): [Class Attribute] Default user agent to use if no other user agent is specified when making requests. default_proxy (str | None): [Class Attribute] Default proxy to use when making requests. default_verify_ssl (bool): [Class Attribute] Whether to verify SSL certificates by default. subtitles_fix_rtl (bool): [Class Attribute] Whether to fix RTL from downloaded subtitles. subtitles_remove_duplicates (bool): [Class Attribute] Whether to remove duplicate lines from downloaded subtitles. id (str): [Class Attribute] ID of the scraper (must be unique). name (str): [Class Attribute] Name of the scraper. abbreviation (str): [Class Attribute] Abbreviation of the scraper. url_regex (re.Pattern | list[re.Pattern]): [Class Attribute] A RegEx pattern to find URLs matching the service. subtitles_class (type[Subtitles]): [Class Attribute] Class of the subtitles format returned by the scraper. is_movie_scraper (bool): [Class Attribute] Whether the scraper is for movies. is_series_scraper (bool): [Class Attribute] Whether the scraper is for series. uses_scrapers (list[str]): [Class Attribute] A list of IDs for other scraper classes that this scraper uses. This assures that the config data for the other scrapers is passed as well. config (ScraperConfig | None): [Class Attribute] A ScraperConfig instance for the scraper, containing configurations. _session (httpx.Client): A synchronous HTTP client session. _async_session (httpx.AsyncClient): An asynchronous HTTP client session. Notes: Each scraper implements its own `ScraperConfig` class (which can be overridden and updated), inheriting from `ScraperConfigBase`, which sets configurable options for the scraper. """ class ScraperConfig(ScraperConfigBase): """A class representing scraper's configuration settings. Can be overridden to create a custom configuration with overridden default values, and additional settings.""" default_timeout: ClassVar[int | float] = 10 default_user_agent: ClassVar[str] = httpx._client.USER_AGENT # noqa: SLF001 default_proxy: ClassVar[str | None] = None default_verify_ssl: ClassVar[bool] = True subtitles_fix_rtl: ClassVar[bool] = False subtitles_remove_duplicates: ClassVar[bool] = True id: ClassVar[str] name: ClassVar[str] abbreviation: ClassVar[str] url_regex: ClassVar[re.Pattern | list[re.Pattern]] subtitles_class: ClassVar[type[Subtitles]] is_movie_scraper: ClassVar[bool] = False is_series_scraper: ClassVar[bool] = False uses_scrapers: ClassVar[list[str]] = [] config: ClassVar[ScraperConfig | None] = None def __init__(self, timeout: int | float | None = None, user_agent: str | None = None, proxy: str | None = None, verify_ssl: bool | None = None): """ Initialize a Scraper object. Args: timeout (int | float | None, optional): A timeout to use when making requests. Defaults to None. user_agent (str | None, optional): A user agent to use when making requests. Defaults to None. proxy (str | None, optional): A proxy to use when making requests. Defaults to None. verify_ssl (bool | None, optional): Whether to verify SSL certificates. Defaults to None. """ self._timeout = return_first_valid(timeout, get_model_field(model=self.config, field='timeout'), self.default_timeout, raise_error=True) self._user_agent = return_first_valid(user_agent, get_model_field(model=self.config, field='user_agent'), self.default_user_agent, raise_error=True) self._proxy = return_first_valid(proxy, get_model_field(model=self.config, field='proxy'), self.default_proxy) self._verify_ssl = return_first_valid(verify_ssl, get_model_field(model=self.config, field='verify_ssl'), self.default_verify_ssl, raise_error=True) if self._timeout != self.default_timeout: logger.debug(f"Initializing '{self.name}' scraper with custom timeout: '{self._timeout}'.") if self._user_agent != self.default_user_agent: logger.debug(f"Initializing '{self.name}' scraper with custom user-agent: '{self._user_agent}'.") if self._proxy != self.default_proxy: logger.debug(f"Initializing '{self.name}' scraper with proxy: '{self._proxy}'.") if self._verify_ssl != self.default_verify_ssl: logger.debug(f"Initializing '{self.name}' scraper with SSL verification set to: '{self._verify_ssl}'.") self._requests_counter = 0 clients_params: dict[str, Any] = { "headers": {"User-Agent": self._user_agent}, "verify": self._verify_ssl, "proxy": self._proxy, "timeout": float(self._timeout), } self._client = httpx.AsyncClient( **clients_params, event_hooks={ "request": [self._async_increment_requests_counter], }, ) # Update session settings according to configurations self._client.headers.update({"User-Agent": self._user_agent}) def _increment_requests_counter(self, request: httpx.Request) -> None: # noqa: ARG002 self._requests_counter += 1 async def _async_increment_requests_counter(self, request: httpx.Request) -> None: # noqa: ARG002 self._requests_counter += 1 @property def requests_count(self) -> int: return self._requests_counter @classmethod @overload def match_url(cls, url: str, raise_error: Literal[True] = ...) -> re.Match: ... @classmethod @overload def match_url(cls, url: str, raise_error: Literal[False] = ...) -> re.Match | None: ... @classmethod def match_url(cls, url: str, raise_error: bool = False) -> re.Match | None: """ Checks if a URL matches scraper's url regex. Args: url (str): A URL to check against the regex. raise_error (bool, optional): Whether to raise an error instead of returning None if the URL doesn't match. Returns: re.Match | None: A Match object if the URL matches the regex, None otherwise (if raise_error is False). Raises: ValueError: If the URL doesn't match the regex and raise_error is True. """ if isinstance(cls.url_regex, re.Pattern) and (match_result := re.fullmatch(pattern=cls.url_regex, string=url)): return match_result if isinstance(cls.url_regex, list): for url_regex_item in cls.url_regex: if result := re.fullmatch(pattern=url_regex_item, string=url): return result if raise_error: raise ValueError(f"URL '{url}' doesn't match the URL regex of {cls.name}.") return None async def async_close(self) -> None: await self._client.aclose() @abstractmethod async def get_data(self, url: str) -> ScrapedMediaResponse: """ Scrape media information about the media on a URL. Args: url (str): A URL to get media information about. Returns: ScrapedMediaResponse: A ScrapedMediaResponse object containing scraped media information. """ @abstractmethod async def download_subtitles(self, media_data: PlaylistMediaItem, subrip_conversion: bool = False) -> SubtitlesData: """ Download subtitles from a media object. Args: media_data (PlaylistMediaItem): A media object to download subtitles from. subrip_conversion (bool, optional): Whether to convert the subtitles to SubRip format. Defaults to False. Returns: SubtitlesData: A SubtitlesData object containing downloaded subtitles. Raises: SubtitlesDownloadError: If the subtitles failed to download. """ @abstractmethod def find_matching_media(self, main_playlist: MainPlaylist, filters: dict[str, str | list[str]] | None = None) -> list: """ Find media items that match the given filters in the main playlist (or all media items if no filters are given). Args: main_playlist (MainPlaylist): Main playlist to search for media items in. filters (dict[str, str | list[str]] | None, optional): A dictionary of filters to match media items against. Defaults to None. Returns: list: A list of media items that match the given filters. """ @abstractmethod def find_matching_subtitles(self, main_playlist: MainPlaylist, language_filter: list[str] | None = None) -> list[PlaylistMediaItem]: """ Find subtitles that match the given language filter in the main playlist. Args: main_playlist (MainPlaylist): Main playlist to search for subtitles in. language_filter (list[str] | None, optional): A list of language codes to filter subtitles by. Defaults to None. Returns: list[PlaylistMediaItem]: A list of subtitles media objects that match the given language filter. """ @abstractmethod async def load_playlist(self, url: str | list[str], headers: dict | None = None) -> MainPlaylist | None: """ Load a playlist from a URL to a representing object. Multiple URLs can be given, in which case the first one that loads successfully will be returned. Args: url (str | list[str]): URL of the M3U8 playlist to load. Can also be a list of URLs (for redundancy). headers (dict | None, optional): A dictionary of headers to use when making the request. Defaults to None (results in using session's configured headers). Returns: MainPlaylist | None: A playlist object (matching the type), or None if the playlist couldn't be loaded. """ @staticmethod @abstractmethod def detect_subtitles_type(subtitles_media: PlaylistMediaItem) -> SubtitlesType | None: """ Detect the subtitles type (Closed Captions, Forced, etc.) from a media object. Args: subtitles_media (PlaylistMediaItem): Subtitles media object to detect the type of. Returns: SubtitlesType | None: The type of the subtitles, None for regular subtitles. """ @classmethod @abstractmethod def format_subtitles_description(cls, subtitles_media: PlaylistMediaItem) -> str: """ Format a description of the subtitles media object. Args: subtitles_media (PlaylistMediaItem): Subtitles media object to format the description of. Returns: str: A formatted description of the subtitles media object. Raises: ValueError: If minimal required data is missing from the media object. """ class HLSScraper(Scraper, ABC): """A base class for HLS (m3u8) scrapers.""" class M3U8Attribute(Enum): """ An enum representing all possible M3U8 attributes. Names / Keys represent M3U8 Media object attributes (should be converted to lowercase), and values represent the name of the key for config usage. """ ASSOC_LANGUAGE = "assoc-language" AUTOSELECT = "autoselect" CHARACTERISTICS = "characteristics" CHANNELS = "channels" DEFAULT = "default" FORCED = "forced" GROUP_ID = "group-id" INSTREAM_ID = "instream-id" LANGUAGE = "language" NAME = "name" STABLE_RENDITION_ID = "stable-rendition-id" TYPE = "type" default_playlist_filters: ClassVar[dict[str, str | list[str] | None] | None] = None _subtitles_filters: dict[str, str | list[str]] = { M3U8Attribute.TYPE.value: "SUBTITLES", } # Resolve mypy errors as mypy doesn't support dynamic models. if TYPE_CHECKING: PlaylistFiltersSubcategory = ScraperConfigSubcategory else: PlaylistFiltersSubcategory = create_model( "PlaylistFiltersSubcategory", __base__=ScraperConfigSubcategory, **{ m3u8_attribute.value: (str | list[str] | None, Field(default=None)) for m3u8_attribute in M3U8Attribute }, # type: ignore[call-overload] ) class ScraperConfig(Scraper.ScraperConfig): playlist_filters: HLSScraper.PlaylistFiltersSubcategory = Field( # type: ignore[valid-type] default_factory=lambda: HLSScraper.PlaylistFiltersSubcategory(), ) def __init__(self, playlist_filters: dict[str, str | list[str] | None] | None = None, *args: Any, **kwargs: Any) -> None: super().__init__(*args, **kwargs) self._playlist_filters = return_first_valid(playlist_filters, get_model_field(model=self.config, field='playlist_filters', convert_to_dict=True, exclude_none=True), self.default_playlist_filters) if self._playlist_filters: logger.debug(f"Scraper '{self.name}' initialized with playlist filters: {self._playlist_filters}.") @staticmethod def parse_language_name(media_data: m3u8.Media) -> str | None: """ Parse the language name from an M3U8 Media object. Can be overridden in subclasses for normalization. Args: media_data (m3u8.Media): Media object to parse the language name from. Returns: str | None: The language name if found, None otherwise. """ name: str | None = media_data.name return name async def load_playlist(self, url: str | list[str], headers: dict[str, str] | None = None) -> m3u8.M3U8 | None: _headers = headers or self._client.headers result: m3u8.M3U8 | None = None urls = single_string_to_list(item=url) response: httpx.Response | None = None for idx, url_item in enumerate(urls): try: logger.debug(f"Loading M3U8 playlist from {url_item} ({idx + 1} of {len(urls)})") response = await self._client.get(url=url_item, headers=_headers, timeout=5) if not response.text: logger.debug("Received an empty response for the playlist.") continue except Exception as e: logger.debug(f"Failed to load playlist: {e}") continue if not response: raise PlaylistLoadError("Failed to load playlists from server.") result = m3u8.loads(content=response.text, uri=url_item) break return result @staticmethod def detect_subtitles_type(subtitles_media: m3u8.Media) -> SubtitlesType | None: """ Detect the subtitles type (Closed Captions, Forced, etc.) from an M3U8 Media object. Args: subtitles_media (m3u8.Media): Subtitles Media object to detect the type of. Returns: SubtitlesType | None: The type of the subtitles, None for regular subtitles. """ if subtitles_media.forced == "YES": return SubtitlesType.FORCED if subtitles_media.characteristics is not None and "public.accessibility" in subtitles_media.characteristics: return SubtitlesType.CC return None async def download_subtitles(self, media_data: m3u8.Media, subrip_conversion: bool = False) -> SubtitlesData: try: playlist_m3u8 = await self.load_playlist(url=media_data.absolute_uri) if playlist_m3u8 is None: raise PlaylistLoadError("Could not load subtitles M3U8 playlist.") # noqa: TRY301 if not media_data.language: raise ValueError("Language code not found in media data.") # noqa: TRY301 downloaded_segments = await self.download_segments(playlist=playlist_m3u8) subtitles = self.subtitles_class(data=downloaded_segments[0], language_code=media_data.language) if len(downloaded_segments) > 1: for segment_data in downloaded_segments[1:]: segment_subtitles_obj = self.subtitles_class(data=segment_data, language_code=media_data.language) subtitles.append_subtitles(segment_subtitles_obj) subtitles.polish( fix_rtl=self.subtitles_fix_rtl, remove_duplicates=self.subtitles_remove_duplicates, ) if subrip_conversion: subtitles_format = SubtitlesFormatType.SUBRIP content = subtitles.to_srt().dump() else: subtitles_format = SubtitlesFormatType.WEBVTT content = subtitles.dump() return SubtitlesData( language_code=media_data.language, language_name=self.parse_language_name(media_data=media_data), subtitles_format=subtitles_format, content=content, content_encoding=subtitles.encoding, special_type=self.detect_subtitles_type(subtitles_media=media_data), ) except Exception as e: raise SubtitlesDownloadError( language_code=media_data.language, language_name=self.parse_language_name(media_data=media_data), special_type=self.detect_subtitles_type(subtitles_media=media_data), original_exc=e, ) from e async def download_segments(self, playlist: m3u8.M3U8) -> list[bytes]: responses = await asyncio.gather( *[ self._client.get(url=segment.absolute_uri) for segment in playlist.segments ], ) responses_data = [] for result in responses: try: result.raise_for_status() responses_data.append(result.content) except Exception as e: raise DownloadError("One of the subtitles segments failed to download.") from e return responses_data def find_matching_media(self, main_playlist: m3u8.M3U8, filters: dict[str, str | list[str]] | None = None) -> list[m3u8.Media]: results: list[m3u8.Media] = [] playlist_filters: dict[str, str | list[str]] | None if self._playlist_filters: # Merge filtering dictionaries into a single dictionary playlist_filters = merge_dict_values( *[dict_item for dict_item in (filters, self._playlist_filters) if dict_item is not None], ) else: playlist_filters = filters for media in main_playlist.media: if not playlist_filters: results.append(media) continue is_valid = True for filter_name, filter_value in playlist_filters.items(): # Skip filter if its value is None if filter_value is None: continue try: filter_name_enum = HLSScraper.M3U8Attribute(filter_name) attribute_value = getattr(media, filter_name_enum.name.lower(), None) if (attribute_value is None) or ( isinstance(filter_value, list) and attribute_value.casefold() not in (x.casefold() for x in filter_value) ) or ( isinstance(filter_value, str) and filter_value.casefold() != attribute_value.casefold() ): is_valid = False break except Exception: is_valid = False if is_valid: results.append(media) return results def find_matching_subtitles(self, main_playlist: m3u8.M3U8, language_filter: list[str] | None = None) -> list[m3u8.Media]: _filters = self._subtitles_filters if language_filter: _filters[self.M3U8Attribute.LANGUAGE.value] = language_filter return self.find_matching_media(main_playlist=main_playlist, filters=_filters) @classmethod def format_subtitles_description(cls, subtitles_media: m3u8.Media) -> str: return format_subtitles_description( language_code=subtitles_media.language, language_name=cls.parse_language_name(media_data=subtitles_media), special_type=cls.detect_subtitles_type(subtitles_media=subtitles_media), ) class ScraperFactory: _scraper_classes_cache: list[type[Scraper]] | None = None _scraper_instances_cache: dict[type[Scraper], Scraper] = {} _currently_initializing: list[type[Scraper]] = [] # Used to prevent infinite recursion @classmethod def get_initialized_scrapers(cls) -> list[Scraper]: """ Get a list of all previously initialized scrapers. Returns: list[Scraper]: A list of initialized scrapers. """ return list(cls._scraper_instances_cache.values()) @classmethod def get_scraper_classes(cls) -> list[type[Scraper]]: """ Find all scraper classes in the scrapers directory. Returns: list[Scraper]: A Scraper subclass. """ if cls._scraper_classes_cache is not None: return cls._scraper_classes_cache cls._scraper_classes_cache = [] scraper_modules_paths = Path(__file__).parent.glob(f"*{SCRAPER_MODULES_SUFFIX}.py") for scraper_module_path in scraper_modules_paths: sys.path.append(str(scraper_module_path)) module = importlib.import_module(f"{PACKAGE_NAME}.scrapers.{scraper_module_path.stem}") # Find all 'Scraper' subclasses for _, obj in inspect.getmembers(module, predicate=lambda x: inspect.isclass(x) and issubclass(x, Scraper)): # Skip object if it's an abstract or imported from another module if not inspect.isabstract(obj) and obj.__module__ == module.__name__: cls._scraper_classes_cache.append(obj) return cls._scraper_classes_cache @classmethod def _get_scraper_instance(cls, scraper_class: type[ScraperT], kwargs: dict | None = None) -> ScraperT: """ Initialize and return a scraper instance. Args: scraper_class (type[ScraperT]): A scraper class to initialize. kwargs (dict | None, optional): A dictionary containing parameters to pass to the scraper's constructor. Defaults to None. Returns: Scraper: An instance of the given scraper class. """ logger.debug(f"Initializing '{scraper_class.name}' scraper...") kwargs = kwargs or {} if scraper_class not in cls._scraper_instances_cache: logger.debug(f"'{scraper_class.name}' scraper not found in cache, creating a new instance...") if scraper_class in cls._currently_initializing: raise ScraperError(f"'{scraper_class.name}' scraper is already being initialized.\n" f"Make sure there are no circular dependencies between scrapers.") cls._currently_initializing.append(scraper_class) cls._scraper_instances_cache[scraper_class] = scraper_class(**kwargs) cls._currently_initializing.remove(scraper_class) else: logger.debug(f"Cached '{scraper_class.name}' scraper instance found and will be used.") return cls._scraper_instances_cache[scraper_class] # type: ignore[return-value] @classmethod @overload def get_scraper_instance(cls, scraper_class: type[ScraperT], scraper_id: str | None = ..., url: str | None = ..., kwargs: dict | None = ..., raise_error: Literal[True] = ...) -> ScraperT: ... @classmethod @overload def get_scraper_instance(cls, scraper_class: type[ScraperT], scraper_id: str | None = ..., url: str | None = ..., kwargs: dict | None = ..., raise_error: Literal[False] = ...) -> ScraperT | None: ... @classmethod @overload def get_scraper_instance(cls, scraper_class: None = ..., scraper_id: str | None = ..., url: str | None = ..., kwargs: dict | None = ..., raise_error: Literal[True] = ...) -> Scraper: ... @classmethod @overload def get_scraper_instance(cls, scraper_class: None = ..., scraper_id: str | None = ..., url: str | None = ..., kwargs: dict | None = ..., raise_error: Literal[False] = ...) -> Scraper | None: ... @classmethod def get_scraper_instance(cls, scraper_class: type[Scraper] | None = None, scraper_id: str | None = None, url: str | None = None, kwargs: dict | None = None, raise_error: bool = True) -> Scraper | None: """ Find, initialize and return a scraper that matches the given URL or ID. Args: scraper_class (type[ScraperT] | None, optional): A scraper class to initialize. Defaults to None. scraper_id (str | None, optional): ID of a scraper to initialize. Defaults to None. url (str | None, optional): A URL to match a scraper for to initialize. Defaults to None. kwargs (dict | None, optional): A dictionary containing parameters to pass to the scraper's constructor. Defaults to None. raise_error (bool, optional): Whether to raise an error if no scraper was found. Defaults to False. Returns: ScraperT | Scraper | None: An instance of a scraper that matches the given URL or ID, None otherwise (if raise_error is False). Raises: ValueError: If no scraper was found and 'raise_error' is True. """ if not any((scraper_class, scraper_id, url)): raise ValueError("At least one of: 'scraper_class', 'scraper_id', or 'url' must be provided.") if scraper_class: return cls._get_scraper_instance( scraper_class=scraper_class, kwargs=kwargs, ) if scraper_id: logger.debug(f"Searching for a scraper object with ID '{scraper_id}'...") for scraper in cls.get_scraper_classes(): if scraper.id == scraper_id: return cls._get_scraper_instance( scraper_class=scraper, kwargs=kwargs, ) elif url: logger.debug(f"Searching for a scraper object that matches URL '{url}'...") for scraper in cls.get_scraper_classes(): if scraper.match_url(url) is not None: return cls._get_scraper_instance( scraper_class=scraper, kwargs=kwargs, ) error_message = "No matching scraper was found." if raise_error: raise ValueError(error_message) logger.debug(error_message) return None class ScraperError(Exception): pass class DownloadError(ScraperError): pass class PlaylistLoadError(ScraperError): pass class SubtitlesDownloadError(ScraperError): def __init__(self, language_code: str | None, language_name: str | None = None, special_type: SubtitlesType | None = None, original_exc: Exception | None = None, *args: Any, **kwargs: dict[str, Any]): """ Initialize a SubtitlesDownloadError instance. Args: language_code (str | None, optional): Language code of the subtitles that failed to download. language_name (str | None, optional): Language name of the subtitles that failed to download. special_type (SubtitlesType | None, optional): Type of the subtitles that failed to download. original_exc (Exception | None, optional): The original exception that caused the error. """ super().__init__(*args, **kwargs) self.language_code = language_code self.language_name = language_name self.special_type = special_type self.original_exc = original_exc ================================================ FILE: isubrip/subtitle_formats/__init__.py ================================================ ================================================ FILE: isubrip/subtitle_formats/subrip.py ================================================ from __future__ import annotations from typing import Any from isubrip.data_structures import SubtitlesFormatType from isubrip.subtitle_formats.subtitles import Subtitles, SubtitlesCaptionBlock class SubRipCaptionBlock(SubtitlesCaptionBlock): """A subtitles caption block based on the SUBRIP format.""" def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and \ self.start_time == other.start_time and self.end_time == other.end_time and self.payload == other.payload def __str__(self) -> str: result_str = "" time_format = "%H:%M:%S,%f" result_str += f"{self.start_time.strftime(time_format)[:-3]} --> {self.end_time.strftime(time_format)[:-3]}\n" result_str += f"{self.payload}" return result_str def to_srt(self) -> SubRipCaptionBlock: return self class SubRipSubtitles(Subtitles[SubRipCaptionBlock]): """An object representing a SubRip subtitles file.""" format = SubtitlesFormatType.SUBRIP def _dumps(self) -> str: subtitles_str = "" for i, block in enumerate(iterable=self.blocks, start=1): subtitles_str += f"{i}\n{str(block)}\n\n" return subtitles_str.rstrip('\n') def _loads(self, data: str) -> None: raise NotImplementedError("SubRip subtitles loading is not supported.") ================================================ FILE: isubrip/subtitle_formats/subtitles.py ================================================ from __future__ import annotations from abc import ABC, abstractmethod from copy import deepcopy from datetime import time from typing import TYPE_CHECKING, Any, ClassVar, Generic, TypeVar from isubrip.constants import RTL_LANGUAGES from isubrip.logger import logger if TYPE_CHECKING: from isubrip.data_structures import SubtitlesFormatType from isubrip.subtitle_formats.subrip import SubRipCaptionBlock, SubRipSubtitles RTL_CONTROL_CHARS = ('\u200e', '\u200f', '\u202a', '\u202b', '\u202c', '\u202d', '\u202e') RTL_CHAR = '\u202b' SubtitlesT = TypeVar('SubtitlesT', bound='Subtitles') SubtitlesBlockT = TypeVar('SubtitlesBlockT', bound='SubtitlesBlock') class SubtitlesBlock(ABC): """ Abstract base class for subtitles blocks. Attributes: modified (bool): Whether the block has been modified. """ def __init__(self) -> None: self.modified: bool = False @abstractmethod def __copy__(self) -> SubtitlesBlock: """Create a copy of the block.""" @abstractmethod def __eq__(self, other: Any) -> bool: """Check if two objects are equal.""" @abstractmethod def __str__(self) -> str: """Return a string representation of the block.""" class SubtitlesCaptionBlock(SubtitlesBlock, ABC): """ A base class for subtitles caption blocks. Attributes: start_time (time): Start timestamp of the caption block. end_time (time): End timestamp of the caption block. payload (str): Caption block's payload. """ def __init__(self, start_time: time, end_time: time, payload: str): """ Initialize a new SubtitlesCaptionBlock object. Args: start_time: Start timestamp of the caption block. end_time: End timestamp of the caption block. payload: Caption block's payload. """ super().__init__() self.start_time = start_time self.end_time = end_time self.payload = payload def __copy__(self) -> SubtitlesCaptionBlock: copy = self.__class__(self.start_time, self.end_time, self.payload) copy.modified = self.modified return copy def fix_rtl(self) -> None: """Fix payload's text direction to RTL.""" previous_payload = self.payload # Remove previous RTL-related formatting for char in RTL_CONTROL_CHARS: self.payload = self.payload.replace(char, '') # Add RLM char at the start of every line self.payload = RTL_CHAR + self.payload.replace("\n", f"\n{RTL_CHAR}") if self.payload != previous_payload: self.modified = True @abstractmethod def to_srt(self) -> SubRipCaptionBlock: """ Convert WebVTT caption block to SRT caption block. Returns: SubRipCaptionBlock: The caption block in SRT format. """ ... class Subtitles(Generic[SubtitlesBlockT], ABC): """ An object representing subtitles, made out of blocks. Attributes: _modified (bool): Whether the subtitles have been modified. format (SubtitlesFormatType): [Class Attribute] Format of the subtitles (contains name and file extension). language_code (str): Language code of the subtitles. blocks (list[SubtitlesBlock]): A list of subtitles blocks that make up the subtitles. encoding (str): Encoding of the subtitles. raw_data (bytes | None): Raw data of the subtitles. """ format: ClassVar[SubtitlesFormatType] def __init__(self, data: bytes | None, language_code: str, encoding: str = "utf-8"): """ Initialize a new Subtitles object. Args: data (bytes | None): Raw data of the subtitles. language_code (str): Language code of the subtitles. encoding (str, optional): Encoding of the subtitles. Defaults to "utf-8". """ self._modified = False self.raw_data = None self.blocks: list[SubtitlesBlockT] = [] self.language_code = language_code self.encoding = encoding if data: self.raw_data = data self._load(data=data) def __add__(self: SubtitlesT, obj: SubtitlesBlockT | SubtitlesT) -> SubtitlesT: """ Add a new subtitles block, or append blocks from another subtitles object. Args: obj (SubtitlesBlock | Subtitles): A subtitles block or another subtitles object. Returns: Subtitles: The current subtitles object. """ if isinstance(obj, SubtitlesBlock): self.add_blocks(obj) elif isinstance(obj, self.__class__): self.append_subtitles(obj) else: logger.warning(f"Cannot add object of type '{type(obj)}' to '{type(self)}' object. Skipping...") return self def __copy__(self: SubtitlesT) -> SubtitlesT: """Create a copy of the subtitles object.""" copy = self.__class__(data=None, language_code=self.language_code, encoding=self.encoding) copy.raw_data = self.raw_data copy.blocks = [block.__copy__() for block in self.blocks] copy._modified = self.modified() # noqa: SLF001 return copy def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and self.blocks == other.blocks def __str__(self) -> str: return self.dumps() def _dump(self) -> bytes: """ Dump subtitles object to bytes representing the subtitles. Returns: bytes: The subtitles in a bytes object. """ return self._dumps().encode(encoding=self.encoding) @abstractmethod def _dumps(self) -> str: """ Dump subtitles object to a string representing the subtitles. Returns: str: The subtitles in a string format. """ ... def _load(self, data: bytes) -> None: """ Load and parse subtitles data from bytes. Args: data (bytes): Subtitles data to load. """ parsed_data = data.decode(encoding=self.encoding) self._loads(data=parsed_data) @abstractmethod def _loads(self, data: str) -> None: """ Load and parse subtitles data from a string. Args: data (bytes): Subtitles data to load. """ ... def dump(self) -> bytes: """ Dump subtitles to a bytes object representing the subtitles. Returns the original raw subtitles data if they have not been modified, and raw data is available. Returns: bytes: The subtitles in a bytes object. """ if self.raw_data is not None and not self.modified(): logger.debug("Returning original raw data as subtitles have not been modified.") return self.raw_data return self._dump() def dumps(self) -> str: """ Dump subtitles to a string representing the subtitles. Returns the original raw subtitles data if they have not been modified, and raw data is available. Returns: """ if self.raw_data is not None and not self.modified(): logger.debug("Returning original raw data (decoded) as subtitles have not been modified.") return self.raw_data.decode(encoding=self.encoding) return self._dumps() def add_blocks(self: SubtitlesT, blocks: SubtitlesBlockT | list[SubtitlesBlockT], set_modified: bool = True) -> SubtitlesT: """ Add a new subtitles block to current subtitles. Args: blocks (SubtitlesBlock | list[SubtitlesBlock]): A block object or a list of block objects to append. set_modified (bool, optional): Whether to set the subtitles as modified. Defaults to True. Returns: Subtitles: The current subtitles object. """ if isinstance(blocks, list): if not blocks: return self self.blocks.extend(blocks) else: self.blocks.append(blocks) if set_modified: self._modified = True return self def append_subtitles(self: SubtitlesT, subtitles: SubtitlesT) -> SubtitlesT: """ Append subtitles to an existing subtitles object. Args: subtitles (Subtitles): Subtitles object to append to current subtitles. Returns: Subtitles: The current subtitles object. """ if subtitles.blocks: self.add_blocks(deepcopy(subtitles.blocks)) if subtitles.modified(): self._modified = True return self def polish(self: SubtitlesT, fix_rtl: bool = False, remove_duplicates: bool = True, ) -> SubtitlesT: """ Apply various fixes to subtitles. Args: fix_rtl (bool, optional): Whether to fix text direction of RTL languages. Defaults to False. remove_duplicates (bool, optional): Whether to remove duplicate captions. Defaults to True. Returns: Subtitles: The current subtitles object. """ fix_rtl = (fix_rtl and self.language_code.split('-')[0] in RTL_LANGUAGES) if not any(( fix_rtl, remove_duplicates, )): return self previous_block: SubtitlesBlockT | None = None for block in self.blocks: if fix_rtl: block.fix_rtl() if remove_duplicates and previous_block is not None and block == previous_block: self.blocks.remove(previous_block) self._modified = True previous_block = block return self def modified(self) -> bool: """ Check if the subtitles have been modified (by checking if any of its blocks have been modified). Returns: bool: True if the subtitles have been modified, False otherwise. """ return self._modified or any(block.modified for block in self.blocks) def to_srt(self) -> SubRipSubtitles: """ Convert subtitles to SRT format. Returns: SubRipSubtitles: The subtitles in SRT format. """ from isubrip.subtitle_formats.subrip import SubRipSubtitles subrip_subtitles = SubRipSubtitles( data=None, language_code=self.language_code, encoding=self.encoding, ) subrip_blocks = [block.to_srt() for block in self.blocks if isinstance(block, SubtitlesCaptionBlock)] subrip_subtitles.add_blocks(subrip_blocks) return subrip_subtitles def split_timestamp(timestamp: str) -> tuple[time, time]: """ Split a subtitles timestamp into start and end. Args: timestamp (str): A subtitles timestamp. For example: "00:00:00.000 --> 00:00:00.000" Returns: tuple(time, time): A tuple containing start and end times as a datetime object. """ # Support ',' character in timestamp's milliseconds (used in SubRip format). timestamp = timestamp.replace(',', '.') start_time, end_time = timestamp.split(" --> ") return time.fromisoformat(start_time), time.fromisoformat(end_time) ================================================ FILE: isubrip/subtitle_formats/webvtt.py ================================================ from __future__ import annotations from abc import ABCMeta from copy import deepcopy import re from typing import TYPE_CHECKING, Any, ClassVar from isubrip.data_structures import SubtitlesFormatType from isubrip.subtitle_formats.subrip import SubRipCaptionBlock from isubrip.subtitle_formats.subtitles import RTL_CHAR, Subtitles, SubtitlesBlock, SubtitlesCaptionBlock from isubrip.utils import split_subtitles_timestamp if TYPE_CHECKING: from datetime import time # WebVTT Documentation: # https://www.w3.org/TR/webvtt1/#cues # https://developer.mozilla.org/en-US/docs/Web/API/WebVTT_API#webvtt_cues bottom_line_alignment_regex = re.compile(r"line:0+(?:\.0+)?%") class WebVTTBlock(SubtitlesBlock, metaclass=ABCMeta): """ Abstract base class for WEBVTT cue blocks. """ is_caption_block: bool = False class WebVTTCaptionBlock(SubtitlesCaptionBlock, WebVTTBlock): """An object representing a WebVTT caption block.""" subrip_alignment_conversion: ClassVar[bool] = False is_caption_block: bool = True def __init__(self, start_time: time, end_time: time, payload: str, settings: str = "", identifier: str = ""): """ Initialize a new object representing a WebVTT caption block. Args: start_time (time): Cue start time. end_time (time): Cue end time. settings (str): Cue settings. payload (str): Cue payload. """ super().__init__(start_time=start_time, end_time=end_time, payload=payload) self.identifier = identifier self.settings = settings def __copy__(self) -> WebVTTCaptionBlock: copy = self.__class__(start_time=self.start_time, end_time=self.end_time, payload=self.payload, settings=self.settings, identifier=self.identifier) copy.modified = self.modified return copy def to_srt(self) -> SubRipCaptionBlock: # Add a {\an8} tag at the start of the payload if the `line` setting is set to 0% if self.subrip_alignment_conversion and WEBVTT_BOTTOM_LINE_ALIGNMENT_REGEX.search(self.settings): # If the payload starts with an RTL control char, add the tag after it if self.payload.startswith(RTL_CHAR): payload = RTL_CHAR + WEBVTT_ALIGN_TOP_TAG + self.payload[len(RTL_CHAR):] else: payload = WEBVTT_ALIGN_TOP_TAG + self.payload else: payload = self.payload return SubRipCaptionBlock(start_time=self.start_time, end_time=self.end_time, payload=payload) def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and \ self.start_time == other.start_time and self.end_time == other.end_time and self.payload == other.payload def __str__(self) -> str: result_str = "" time_format = "%H:%M:%S.%f" # Add identifier (if it exists) if self.identifier: result_str += f"{self.identifier}\n" result_str += f"{self.start_time.strftime(time_format)[:-3]} --> {self.end_time.strftime(time_format)[:-3]}" if self.settings: result_str += f" {self.settings}" result_str += f"\n{self.payload}" return result_str class WebVTTCommentBlock(WebVTTBlock): """An object representing a WebVTT comment block.""" header = "NOTE" def __init__(self, payload: str, inline: bool = False) -> None: """ Initialize a new object representing a WebVTT comment block. Args: payload (str): Comment payload. """ super().__init__() self.payload = payload self.inline = inline def __copy__(self) -> WebVTTCommentBlock: copy = self.__class__(payload=self.payload, inline=self.inline) copy.modified = self.modified return copy def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and self.inline == other.inline and self.payload == other.payload def __str__(self) -> str: if self.inline: return f"{self.header} {self.payload}" if self.payload: return f"{self.header}\n{self.payload}" return self.header class WebVTTStyleBlock(WebVTTBlock): """An object representing a WebVTT style block.""" header = "STYLE" def __init__(self, payload: str) -> None: """ Initialize a new object representing a WebVTT style block. Args: payload (str): Style payload. """ super().__init__() self.payload = payload def __copy__(self) -> WebVTTStyleBlock: copy = self.__class__(payload=self.payload) copy.modified = self.modified return copy def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and self.payload == other.payload def __str__(self) -> str: return f"{self.header}\n{self.payload}" class WebVTTRegionBlock(WebVTTBlock): """An object representing a WebVTT region block.""" header = "REGION" def __init__(self, payload: str) -> None: """ Initialize a new object representing a WebVTT region block. Args: payload (str): Region payload. """ super().__init__() self.payload = payload def __copy__(self) -> WebVTTRegionBlock: copy = self.__class__(payload=self.payload) copy.modified = self.modified return copy def __eq__(self, other: Any) -> bool: return isinstance(other, type(self)) and self.payload == other.payload def __str__(self) -> str: return f"{self.header} {self.payload}" class WebVTTSubtitles(Subtitles[WebVTTBlock]): """An object representing a WebVTT subtitles file.""" format = SubtitlesFormatType.WEBVTT def _dumps(self) -> str: """ Dump subtitles to a string representing the subtitles in a WebVTT format. Returns: str: The subtitles in a string using a WebVTT format. """ subtitles_str = "WEBVTT\n\n" for block in self.blocks: subtitles_str += str(block) + "\n\n" return subtitles_str.rstrip('\n') def _loads(self, data: str) -> None: """ Load and parse WebVTT subtitles data from a string. Args: data (bytes): Subtitles data to load. """ prev_line: str = "" lines_iterator = iter(data.splitlines()) for line in lines_iterator: # If the line is a timestamp if caption_block_regex := re.match(WEBVTT_CAPTION_BLOCK_REGEX, line): # If previous line wasn't empty, add it as an identifier if prev_line: caption_identifier = prev_line else: caption_identifier = "" caption_timestamps = split_subtitles_timestamp(caption_block_regex.group(1)) caption_settings = caption_block_regex.group(2) caption_payload = "" for additional_line in lines_iterator: if not additional_line: line = additional_line break caption_payload += additional_line + "\n" caption_payload = caption_payload.rstrip("\n") self.blocks.append(WebVTTCaptionBlock( identifier=caption_identifier, start_time=caption_timestamps[0], end_time=caption_timestamps[1], settings=caption_settings, payload=caption_payload)) elif comment_block_regex := re.match(WEBVTT_COMMENT_HEADER_REGEX, line): comment_payload = "" inline = False if comment_block_regex.group(1) is not None: comment_payload += comment_block_regex.group(1) + "\n" inline = True for additional_line in lines_iterator: if not additional_line: line = additional_line break comment_payload += additional_line + "\n" self.blocks.append(WebVTTCommentBlock(comment_payload.rstrip("\n"), inline=inline)) elif line.rstrip(' \t') == WebVTTRegionBlock.header: region_payload = "" for additional_line in lines_iterator: if not additional_line: line = additional_line break region_payload += additional_line + "\n" self.blocks.append(WebVTTRegionBlock(region_payload.rstrip("\n"))) elif line.rstrip(' \t') == WebVTTStyleBlock.header: style_payload = "" for additional_line in lines_iterator: if not additional_line: line = additional_line break style_payload += additional_line + "\n" self.blocks.append(WebVTTStyleBlock(style_payload.rstrip("\n"))) prev_line = line def append_subtitles(self: WebVTTSubtitles, subtitles: WebVTTSubtitles) -> WebVTTSubtitles: if subtitles.blocks: subtitles_copy = deepcopy(subtitles) # Remove head blocks from the subtitles that will be appended subtitles_copy.remove_head_blocks() self.add_blocks(subtitles_copy.blocks) if subtitles_copy.modified(): self._modified = True return self def remove_head_blocks(self) -> None: """ Remove all head blocks (Style / Region) from the subtitles. NOTE: Comment blocks are removed as well if they are before the first caption block (since they're probably related to the head blocks). """ for block in self.blocks: if isinstance(block, WebVTTCaptionBlock): break if isinstance(block, WebVTTCommentBlock | WebVTTStyleBlock | WebVTTRegionBlock): self.blocks.remove(block) # --- Constants --- WEBVTT_PERCENTAGE_REGEX = r"\d{1,3}(?:\.\d+)?%" WEBVTT_CAPTION_TIMINGS_REGEX = \ r"(?:[0-5]\d:)?[0-5]\d:[0-5]\d[\.,]\d{3}[ \t]+-->[ \t]+(?:[0-5]\d:)?[0-5]\d:[0-5]\d[\.,]\d{3}" WEBVTT_CAPTION_SETTING_ALIGNMENT_REGEX = r"align:(?:start|center|middle|end|left|right)" WEBVTT_CAPTION_SETTING_LINE_REGEX = rf"line:(?:{WEBVTT_PERCENTAGE_REGEX}|-?\d+%)(?:,(?:start|center|middle|end))?" WEBVTT_CAPTION_SETTING_POSITION_REGEX = rf"position:{WEBVTT_PERCENTAGE_REGEX}(?:,(?:start|center|middle|end))?" WEBVTT_CAPTION_SETTING_REGION_REGEX = r"region:(?:(?!(?:-->)|\t)\S)+" WEBVTT_CAPTION_SETTING_SIZE_REGEX = rf"size:{WEBVTT_PERCENTAGE_REGEX}" WEBVTT_CAPTION_SETTING_VERTICAL_REGEX = r"vertical:(?:lr|rl)" WEBVTT_CAPTION_SETTINGS_REGEX = ("(?:" f"(?:{WEBVTT_CAPTION_SETTING_ALIGNMENT_REGEX})|" f"(?:{WEBVTT_CAPTION_SETTING_LINE_REGEX})|" f"(?:{WEBVTT_CAPTION_SETTING_POSITION_REGEX})|" f"(?:{WEBVTT_CAPTION_SETTING_REGION_REGEX})|" f"(?:{WEBVTT_CAPTION_SETTING_SIZE_REGEX})|" f"(?:{WEBVTT_CAPTION_SETTING_VERTICAL_REGEX})|" f"(?:[ \t]+)" ")*") WEBVTT_CAPTION_BLOCK_REGEX = re.compile(rf"^({WEBVTT_CAPTION_TIMINGS_REGEX})[ \t]*({WEBVTT_CAPTION_SETTINGS_REGEX})?") WEBVTT_COMMENT_HEADER_REGEX = re.compile(rf"^{WebVTTCommentBlock.header}(?:$|[ \t])(.+)?") WEBVTT_BOTTOM_LINE_ALIGNMENT_REGEX = re.compile(r"line:0+(?:\.0+)?%") WEBVTT_ALIGN_TOP_TAG = r"{\an8}" ================================================ FILE: isubrip/ui.py ================================================ from __future__ import annotations import math from typing import TYPE_CHECKING from rich.progress import TimeElapsedColumn from rich.text import Text if TYPE_CHECKING: from rich.progress import Task class MinsAndSecsTimeElapsedColumn(TimeElapsedColumn): """Renders time elapsed in minutes and seconds.""" def render(self, task: Task) -> Text: """Show time elapsed.""" elapsed = task.finished_time if task.finished else task.elapsed if elapsed is None: return Text("-:--", style="progress.elapsed") minutes, seconds = divmod(math.ceil(elapsed), 60) return Text(f"{minutes:02d}:{seconds:02d}", style="progress.elapsed") ================================================ FILE: isubrip/utils.py ================================================ from __future__ import annotations from abc import ABCMeta import datetime as dt from functools import lru_cache import logging from pathlib import Path import re import secrets import shutil import sys from typing import TYPE_CHECKING, Any, Literal, cast, overload from wcwidth import wcswidth from isubrip.constants import WINDOWS_RESERVED_FILE_NAMES, temp_folder_path from isubrip.data_structures import ( Episode, MediaBase, Movie, Season, Series, SubtitlesData, SubtitlesFormatType, SubtitlesType, T, ) from isubrip.logger import logger if TYPE_CHECKING: from os import PathLike from types import TracebackType import httpx from pydantic import BaseModel, ValidationError class SingletonMeta(ABCMeta): """ A metaclass that implements the Singleton pattern. When a class using this metaclass is initialized, it will return the same instance every time. """ _instances: dict[object, object] = {} def __call__(cls, *args: Any, **kwargs: Any) -> object: if cls._instances.get(cls) is None: cls._instances[cls] = super().__call__(*args, **kwargs) return cls._instances[cls] class TemporaryDirectory: """ A context manager for creating and managing a temporary directory. Args: directory_name (str | None, optional): Name of the directory to generate. If not specified, a random string will be generated. Defaults to None. """ def __init__(self, directory_name: str | None = None): if directory_name: self.directory_name = sanitize_path_segment(directory_name) else: self.directory_name = secrets.token_hex(5) self.path = temp_folder_path() / self.directory_name def __enter__(self) -> Path: """Create the temporary directory and return its path.""" if self.path.is_dir(): logger.debug(f"Temporary directory '{self.path}' already exists. " f"Emptying directory from all contents...") shutil.rmtree(self.path) self.path.mkdir(parents=True) logger.debug(f"Temporary directory has been generated: '{self.path}'") return self.path def __exit__(self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None) -> None: """Clean up the temporary directory.""" self.cleanup() def cleanup(self) -> None: """Remove the temporary directory.""" if not self.path.exists(): return logger.debug(f"Removing temporary directory: '{self.path}'") try: shutil.rmtree(self.path) except Exception as e: logger.warning(f"Failed to remove temporary directory '{self.path}': {e}") def convert_epoch_to_datetime(epoch_timestamp: int) -> dt.datetime: """ Convert an epoch timestamp to a datetime object. Args: epoch_timestamp (int): Epoch timestamp. Returns: datetime: A datetime object representing the timestamp. """ if epoch_timestamp >= 0: return dt.datetime.fromtimestamp(epoch_timestamp) return dt.datetime(1970, 1, 1) + dt.timedelta(seconds=epoch_timestamp) def convert_log_level(log_level: str) -> int: """ Convert a log level string to a logging level. Args: log_level (str): Log level string. Returns: int: Logging level. Raises: ValueError: If the log level is invalid. """ log_level_upper = log_level.upper() if log_level_upper not in ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'): raise ValueError(f"Invalid log level: {log_level}") return cast("int", getattr(logging, log_level_upper)) def download_subtitles_to_file(media_data: Movie | Episode, subtitles_data: SubtitlesData, output_path: str | PathLike, source_abbreviation: str | None = None, overwrite: bool = False) -> Path: """ Download subtitles to a file. Args: media_data (Movie | Episode): An object containing media data. subtitles_data (SubtitlesData): A SubtitlesData object containing subtitles data. output_path (str | PathLike): Path to the output folder. source_abbreviation (str | None, optional): Abbreviation of the source the subtitles are downloaded from. Defaults to None. overwrite (bool, optional): Whether to overwrite files if they already exist. Defaults to True. Returns: Path: Path to the downloaded subtitles file. Raises: ValueError: If the path in `output_path` does not exist. """ output_path = Path(output_path) if not output_path.is_dir(): raise ValueError(f"Invalid path: {output_path}") if isinstance(media_data, Movie): file_name = format_release_name(title=media_data.name, release_date=media_data.release_date, media_source=source_abbreviation, language_code=subtitles_data.language_code, subtitles_type=subtitles_data.special_type, file_format=subtitles_data.subtitles_format) else: # isinstance(media_data, Episode): file_name = format_release_name(title=media_data.series_name, release_date=media_data.release_date, season_number=media_data.season_number, episode_number=media_data.episode_number, episode_name=media_data.episode_name, media_source=source_abbreviation, language_code=subtitles_data.language_code, subtitles_type=subtitles_data.special_type, file_format=subtitles_data.subtitles_format) file_path = output_path / file_name if file_path.exists() and not overwrite: file_path = generate_non_conflicting_path(file_path=file_path) with file_path.open('wb') as f: f.write(subtitles_data.content) return file_path def format_config_validation_error(exc: ValidationError) -> str: """ Format a Pydantic ValidationError into a human-readable string. Args: exc (ValidationError): The ValidationError instance containing validation errors. Returns: str: A formatted string describing the validation errors, including the location, type, value, and error messages for each invalid field. """ validation_errors = exc.errors() error_str = "" consolidated_errors: dict[str, dict[str, Any]] = {} for validation_error in validation_errors: value: Any = validation_error['input'] value_type: str = type(value).__name__ location: list[str] = [str(item) for item in validation_error['loc']] error_msg: str = validation_error['msg'] # When the expected type is a union, Pydantic returns several errors for each type, # with the type being the last item in the location list if ( isinstance(location[-1], str) and (location[-1].endswith(']') or location[-1] in ('str', 'int', 'float', 'bool')) ): location.pop() if len(location) > 1: location_str = ".".join(location) else: location_str = location[0] if location_str in consolidated_errors: consolidated_errors[location_str]['errors'].append(error_msg) else: consolidated_errors[location_str] = {} consolidated_errors[location_str]['info'] = { "value": value, "type": value_type, } consolidated_errors[location_str]['errors'] = [error_msg] for error_loc, error_data in consolidated_errors.items(): error_type = error_data['info']['type'] error_value = error_data['info']['value'] error_str += f"'{error_loc}' (type: '{error_type}', value: '{error_value}'):\n" for error in error_data['errors']: error_str += f" {error}\n" return error_str def format_list(items: list[str], width: int = 80) -> str: """ Format a list of strings into a grid-like display with dynamic column widths. The function automatically calculates the optimal number of columns based on the maximum item width and the desired total width. It properly handles Unicode characters by using their display width. Args: items (list[str]): List of strings to format width (int, optional): Maximum width of the output in characters. Defaults to 80. Returns: str: A formatted string with items arranged in columns Example: >>> items = ["Item 1", "Long Item 2", "Item 3", "Item 4"] >>> print(format_list(items, width=40)) Item 1 Long Item 2 Item 3 Item 4 """ if not items: return "" # Calculate true display width for each item and add spacing item_widths = [(s, wcswidth(s)) for s in items] column_width = max(width for _, width in item_widths) + 4 # Add spacing between columns columns = max(1, width // column_width) # At least one column # Build rows with proper spacing rows = [] for i in range(0, len(item_widths), columns): row_items = item_widths[i:i + columns] cols = [] for text, text_width in row_items: padding = " " * (column_width - text_width) cols.append(f"{text}{padding}") rows.append("".join(cols).rstrip()) return "\n".join(rows) def format_media_description(media_data: MediaBase, shortened: bool = False) -> str: """ Generate a short description string of a media object. Args: media_data (MediaBase): An object containing media data. shortened (bool, optional): Whether to generate a shortened description. Defaults to False. Returns: str: A short description string of the media object. """ if isinstance(media_data, Movie): release_year = ( media_data.release_date.year if isinstance(media_data.release_date, dt.datetime) else media_data.release_date ) description_str = f"{media_data.name.strip()} [{release_year}]" if media_data.id: description_str += f" (ID: {media_data.id})" return description_str if isinstance(media_data, Series): description_str = f"{media_data.series_name.strip()}" if media_data.series_release_date: if isinstance(media_data.series_release_date, dt.datetime): description_str += f" [{media_data.series_release_date.year}]" else: description_str += f" [{media_data.series_release_date}]" if media_data.id: description_str += f" (ID: {media_data.id})" return description_str if isinstance(media_data, Season): description_str = "" if not shortened: description_str = f"{media_data.series_name.strip()} - " description_str += f"Season {media_data.season_number}" if media_data.season_name: description_str += f" - {media_data.season_name.strip()}" if media_data.id: description_str += f" (ID: {media_data.id})" return description_str if isinstance(media_data, Episode): description_str = "" if not shortened: description_str = f"{media_data.series_name.strip()} - " description_str += f"S{media_data.season_number:02d}E{media_data.episode_number:02d}" if media_data.episode_name: description_str += f" - {media_data.episode_name.strip()}" if media_data.id: description_str += f" (ID: {media_data.id})" return description_str raise ValueError(f"Unsupported media type: '{type(media_data)}'") def format_release_name(title: str, release_date: dt.datetime | int | None = None, season_number: int | None = None, episode_number: int | None = None, episode_name: str | None = None, media_source: str | None = None, source_type: str | None = "WEB", additional_info: str | list[str] | None = None, language_code: str | None = None, subtitles_type: SubtitlesType | None = None, file_format: str | SubtitlesFormatType | None = None) -> str: """ Format a release name. Args: title (str): Media title. release_date (int | None, optional): Release date (datetime), or year (int) of the media. Defaults to None. season_number (int | None, optional): Season number. Defaults to None. episode_number (int | None, optional): Episode number. Defaults to None. episode_name (str | None, optional): Episode name. Defaults to None. media_source (str | None, optional): Media source name (full or abbreviation). Defaults to None. source_type(str | None, optional): General source type (WEB, BluRay, etc.). Defaults to None. additional_info (list[str] | str | None, optional): Additional info to add to the file name. Defaults to None. language_code (str | None, optional): Language code. Defaults to None. subtitles_type (SubtitlesType | None, optional): Subtitles type. Defaults to None. file_format (SubtitlesFormat | str | None, optional): File format to use. Defaults to None. Returns: str: Generated file name. """ file_name = slugify_title(title=title, separator=".") if release_date is not None: if isinstance(release_date, dt.datetime): release_year = release_date.year else: release_year = release_date file_name += f".{release_year}" if season_number is not None and episode_number is not None: file_name += f".S{season_number:02}E{episode_number:02}" if episode_name is not None: file_name += f".{slugify_title(title=episode_name, separator='.')}" if media_source is not None: file_name += f".{media_source}" if source_type is not None: file_name += f".{source_type}" if additional_info is not None: if isinstance(additional_info, list | tuple): additional_info = '.'.join(additional_info) file_name += f".{additional_info}" if language_code is not None: file_name += f".{language_code}" if subtitles_type is not None: file_name += f".{subtitles_type.value.lower()}" sanitized_basename = sanitize_path_segment(file_name) if file_format is not None: if isinstance(file_format, SubtitlesFormatType): file_format_str = file_format.value.file_extension else: file_format_str = file_format.lstrip('.') return f"{sanitized_basename}.{file_format_str}" return sanitized_basename @lru_cache def format_subtitles_description(language_code: str | None = None, language_name: str | None = None, special_type: SubtitlesType | None = None) -> str: """ Format a subtitles description using its attributes. Args: language_code (str | None, optional): Language code. Defaults to None. language_name (str | None, optional): Language name. Defaults to None. special_type (SubtitlesType | None, optional): Subtitles type. Defaults to None. Returns: str: Formatted subtitles description. Raises: ValueError: If neither `language_code` nor `language_name` is provided. """ if language_name and language_code: language_str = f"{language_name} ({language_code})" elif result := (language_name or language_code): language_str = result else: raise ValueError("Either 'language_code' or 'language_name' must be provided.") if special_type: language_str += f" [{special_type.value}]" return language_str def get_model_field(model: BaseModel | None, field: str, convert_to_dict: bool = False, **kwargs: Any) -> Any: """ Get a field from a Pydantic model. Args: model (BaseModel | None): A Pydantic model. field (str): Field name to retrieve. convert_to_dict (bool, optional): Whether to convert the field value to a dictionary. Defaults to False. **kwargs: Additional keyword arguments to pass to the serialization method (`model_dump`). Relevant only if `convert_to_dict` is True, and `field_value` is a Pydantic model. Returns: Any: The field value. """ if model and hasattr(model, field): field_value = getattr(model, field) if convert_to_dict and hasattr(field_value, 'model_dump'): return field_value.model_dump(**kwargs) return field_value return None def generate_non_conflicting_path(file_path: Path, has_extension: bool = True) -> Path: """ Generate a non-conflicting path for a file. If the file already exists, a number will be added to the end of the file name. Args: file_path (Path): Path to a file. has_extension (bool, optional): Whether the name of the file includes file extension. Defaults to True. Returns: Path: A non-conflicting file path. """ if isinstance(file_path, str): file_path = Path(file_path) if not file_path.exists(): return file_path i = 1 while True: if has_extension: new_file_path = file_path.parent / f"{file_path.stem}-{i}{file_path.suffix}" else: new_file_path = file_path.parent / f"{file_path}-{i}" if not new_file_path.exists(): return new_file_path i += 1 def merge_dict_values(*dictionaries: dict) -> dict: """ A function for merging the values of multiple dictionaries using the same keys. If a key already exists, the value will be added to a list of values mapped to that key. Examples: merge_dict_values({'a': 1, 'b': 3}, {'a': 2, 'b': 4}) -> {'a': [1, 2], 'b': [3, 4]} merge_dict_values({'a': 1, 'b': 2}, {'a': 1, 'b': [2, 3]}) -> {'a': 1, 'b': [2, 3]} Note: This function support only merging of lists or single items (no tuples or other iterables), and without any nesting (lists within lists). Args: *dictionaries (dict): Dictionaries to merge. Returns: dict: A merged dictionary. """ _dictionaries: list[dict] = [d for d in dictionaries if d] if len(_dictionaries) == 0: return {} if len(_dictionaries) == 1: return _dictionaries[0] result: dict = {} for _dict in _dictionaries: for key, value in _dict.items(): if key in result: if isinstance(result[key], list): if isinstance(value, list): result[key].extend(value) else: result[key].append(value) else: if isinstance(value, list): result[key] = [result[key], *value] else: result[key] = [result[key], value] else: result[key] = value return result def raise_for_status(response: httpx.Response) -> None: """ Raise an exception if the response status code is invalid. Uses 'response.raise_for_status()' internally, with additional logging. Args: response (httpx.Response): A response object. """ truncation_threshold = 1500 if not response.is_error: return if len(response.text) > truncation_threshold: # Truncate the response as in some cases there could be an unexpected long HTML response response_text = response.text[:truncation_threshold].rstrip() + " " else: response_text = response.text logger.debug(f"Response status code: {response.status_code}") if response.headers.get('Content-Type'): logger.debug(f"Response type: {response.headers['Content-Type']}") logger.debug(f"Response text: {response_text}") response.raise_for_status() def parse_url_params(url_params: str) -> dict: """ Parse GET parameters from a URL to a dictionary. Args: url_params (str): URL parameters. (e.g. 'param1=value1¶m2=value2') Returns: dict: A dictionary containing the URL parameters. """ url_params = url_params.split('?')[-1].rstrip('&') params_list = url_params.split('&') if len(params_list) == 0 or \ (len(params_list) == 1 and '=' not in params_list[0]): return {} return {key: value for key, value in (param.split('=') for param in params_list)} @overload def return_first_valid(*values: T | None, raise_error: Literal[True] = ...) -> T: ... @overload def return_first_valid(*values: T | None, raise_error: Literal[False] = ...) -> T | None: ... def return_first_valid(*values: T | None, raise_error: bool = False) -> T | None: """ Return the first non-None value from a list of values. Args: *values (T): Values to check. raise_error (bool, optional): Whether to raise an error if all values are None. Defaults to False. Returns: T | None: The first non-None value, or None if all values are None and `raise_error` is False. Raises: ValueError: If all values are None and `raise_error` is True. """ for value in values: if value is not None: return value if raise_error: raise ValueError("No valid value found.") return None def single_string_to_list(item: str | list[str]) -> list[str]: """ Convert a single string to a list containing the string. If None is passed, an empty list will be returned. Args: item (str | list[str]): A string or a list of strings. Returns: list[str]: A list containing the string, or an empty list if None was passed. """ if item is None: return [] if isinstance(item, list): return item return [item] def split_subtitles_timestamp(timestamp: str) -> tuple[dt.time, dt.time]: """ Split a subtitles timestamp into start and end. Args: timestamp (str): A subtitles timestamp. For example: "00:00:00.000 --> 00:00:00.000" Returns: tuple(time, time): A tuple containing start and end times as a datetime object. """ # Support ',' character in timestamp's milliseconds (used in SubRip format). timestamp = timestamp.replace(',', '.') start_time, end_time = timestamp.split(" --> ") return dt.time.fromisoformat(start_time), dt.time.fromisoformat(end_time) _REMOVED_CHARS_SLUG = str.maketrans("", "", '<>():"?*') _REMOVED_CHARS_FS = str.maketrans("", "", '<>:"/\\|?*') _REPLACE_WITH_SEPARATOR = [" - ", " & ", ",", ":", "|", "/", " "] @lru_cache def slugify_title(title: str, separator: str = " ") -> str: """ Normalize a title into a slug-like string for creating name components. This function is for normalization, not for filesystem safety. Args: title (str): A media title. separator (str, optional): A separator to use between words. Defaults to " ". Returns: str: A slugified title. """ title = title.strip() title = title.replace("…", "...") # Replace multi-character sequences first for item in _REPLACE_WITH_SEPARATOR: if separator == " " and item in (" & ", " - "): continue if item == " & ": title = title.replace(item, f"{separator}&{separator}") else: title = title.replace(item, separator) # Remove invalid characters for a slug title = title.translate(_REMOVED_CHARS_SLUG) # Replace multiple separators with a single one if separator: title = re.sub(f"[{re.escape(separator)}]+", separator, title) return title @lru_cache def sanitize_path_segment(segment: str, platform: str | None = None) -> str: """ Sanitize a file or directory name (path segment) for a given OS. Args: segment (str): The path segment to sanitize (file or directory name). platform (str | None, optional): Target platform ('win32', 'linux', 'darwin'). Defaults to 'sys.platform' (current platform). Returns: str: A sanitized path segment safe for use on the target filesystem. """ if platform is None: platform = sys.platform # Remove characters illegal on all filesystems segment = segment.translate(_REMOVED_CHARS_FS) if platform == "win32": # On Windows, remove trailing dots and spaces segment = segment.rstrip(". ") # Handle reserved device names (match first segment before dot) base_name, _, _ = segment.partition('.') if base_name.upper() in WINDOWS_RESERVED_FILE_NAMES: segment = f"_{segment}" # Ensure the segment is not empty if not segment: return "_" return segment ================================================ FILE: pyproject.toml ================================================ [project] name = "isubrip" version = "2.6.8" description = "A Python package for scraping and downloading subtitles from AppleTV / iTunes movie pages." authors = [ {name = "Michael Yochpaz"} ] readme = "README.md" keywords = [ "iTunes", "AppleTV", "movies", "subtitles", "scrape", "scraper", "download", "m3u8" ] classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: End Users/Desktop", "Operating System :: Microsoft :: Windows", "Operating System :: MacOS", "Operating System :: POSIX :: Linux", "Topic :: Utilities", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", ] requires-python = ">= 3.10" dependencies = [ "httpx[http2]>=0.28.1", "m3u8>=6.0.0", "pydantic>=2.12.0", "pydantic-settings>=2.11.0", "pygments>=2.19.1", # Used by 'rich'. Specified here as version 2.18 appears to cause issues. "rich>=14.2.0", "tomli>=2.3.0", "wcwidth>=0.2.14", ] [project.urls] Homepage = "https://github.com/MichaelYochpaz/iSubRip" Repository = "https://github.com/MichaelYochpaz/iSubRip" Issues = "https://github.com/MichaelYochpaz/iSubRip/issues" Changelog = "https://github.com/MichaelYochpaz/iSubRip/blob/main/CHANGELOG.md" [project.scripts] isubrip = "isubrip.__main__:main" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.uv] dev-dependencies = [ "mypy>=1.14.1", "pyperf>=2.9.0", "pytest>=8.4.2", "ruff>=0.9.3", ] [tool.mypy] check_untyped_defs = true disallow_untyped_defs = true explicit_package_bases = true ignore_missing_imports = true python_version = "3.10" warn_return_any = true exclude = ["tests/mock_data"] plugins = ["pydantic.mypy"] [tool.ruff] line-length = 120 target-version = "py310" [tool.ruff.lint] select = [ "ARG", "ASYNC", "B", "C4", "COM", "E", "F", "FA", "I", "INP", "ISC", "N", "PIE", "PGH", "PT", "PTH", "Q", "RSE", "RET", "RUF", "S", "SIM", "SLF", "T20", "TCH", "TID", "TRY", "UP", ] ignore = [ "C416", "Q000", "RUF010", "RUF012", "SIM108", "TD002", "TD003", "TRY003", ] unfixable = ["ARG"] [tool.ruff.lint.per-file-ignores] "tests/*.py" = ["S101"] [tool.ruff.lint.flake8-tidy-imports] ban-relative-imports = "all" [tool.ruff.lint.flake8-quotes] docstring-quotes = "double" [tool.ruff.lint.isort] force-sort-within-sections = true [tool.ruff.lint.pyupgrade] keep-runtime-typing = true ================================================ FILE: tests/.gitignore ================================================ mock_data/ ================================================ FILE: tests/__init__.py ================================================ ================================================ FILE: tests/benchmarks/__init__.py ================================================ ================================================ FILE: tests/benchmarks/download_benchmark.py ================================================ import pyperf from tests.benchmarks.download_benchmark_module import benchmark # Note: Run this script with the `-o` flag to output results to a file. def main() -> None: runner = pyperf.Runner( processes=5, values=10, warmups=1, ) runner.bench_async_func( name="Download Benchmark", func=benchmark, ) if __name__ == "__main__": main() ================================================ FILE: tests/benchmarks/download_benchmark_module.py ================================================ import asyncio import logging from pathlib import Path from unittest.mock import patch from isubrip.cli import console from isubrip.commands.download import download from isubrip.logger import logger, setup_loggers from isubrip.utils import TemporaryDirectory from tests.tools.mock_loader import MockLoader async def benchmark() -> None: setup_loggers( stdout_loglevel=logging.INFO, stdout_console=console, logfile_output=False, ) url = "https://tv.apple.com/il/movie/interstellar/umc.cmc.1vrwat5k1ucm5k42q97ioqyq3" mock_data_path = Path("tests/mock_data/appletv/il/umc.cmc.1vrwat5k1ucm5k42q97ioqyq3") mock_loader = MockLoader(mock_data_path, logger=logger) with TemporaryDirectory() as temp_dir: logger.info(f"Temporary directory created at: {temp_dir}") with patch("httpx.AsyncClient.send", side_effect=mock_loader.mock_send_handler): await download(url, download_path=temp_dir) if __name__ == "__main__": asyncio.run(benchmark()) ================================================ FILE: tests/test_utils.py ================================================ import datetime as dt import pytest from isubrip.data_structures import Episode, Movie, Season, Series, SubtitlesFormatType, SubtitlesType from isubrip.utils import ( format_media_description, format_release_name, sanitize_path_segment, slugify_title, ) class TestSlugifyTitle: @pytest.mark.parametrize( ("input_title", "separator", "expected_output"), [ ("The Lord of the Rings: The Fellowship of the Ring", ".", "The.Lord.of.the.Rings.The.Fellowship.of.the.Ring"), ("The Lord of the Rings: The Fellowship of the Ring", " ", "The Lord of the Rings The Fellowship of the Ring"), ("Once Upon a Time... in Hollywood", ".", "Once.Upon.a.Time.in.Hollywood"), ("Once Upon a Time... in Hollywood", " ", "Once Upon a Time... in Hollywood"), ("Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb?", ".", "Dr.Strangelove.or.How.I.Learned.to.Stop.Worrying.and.Love.the.Bomb"), ("Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb?", " ", "Dr. Strangelove or How I Learned to Stop Worrying and Love the Bomb"), ("Mission: Impossible - The Final Reckoning", ".", "Mission.Impossible.The.Final.Reckoning"), ("Mission: Impossible - The Final Reckoning", " ", "Mission Impossible - The Final Reckoning"), ("Deadpool & Wolverine", ".", "Deadpool.&.Wolverine"), ("Deadpool & Wolverine", " ", "Deadpool & Wolverine"), ("50/50", ".", "50.50"), ("50/50", " ", "50 50"), ("50 / 50", ".", "50.50"), ("50 / 50", " ", "50 50"), ("V/H/S", ".", "V.H.S"), ("V/H/S", " ", "V H S"), ("What If...?", ".", "What.If."), ("What If...?", " ", "What If..."), ], ) def test_slugify_title(self, input_title: str, separator: str, expected_output: str) -> None: assert slugify_title(title=input_title, separator=separator) == expected_output class TestSanitizePath: @pytest.mark.parametrize( ("name", "expected_name"), [ ("A/B/C", "ABC"), ("AC", "ABC"), ('A"B"C', "ABC"), ("A:B:C", "ABC"), ("A|B|C", "ABC"), ("A?B?C", "ABC"), ("A*B*C", "ABC"), ], ) def test_sanitize_common_illegal_chars(self, name: str, expected_name: str) -> None: assert sanitize_path_segment(name) == expected_name @pytest.mark.parametrize( ("name", "expected_unix", "expected_windows"), [ ("name.", "name.", "name"), ("name..", "name..", "name"), ("name ", "name ", "name"), (" name", " name", " name"), ("name. ", "name. ", "name"), ("COM1", "COM1", "_COM1"), ("Con.Air", "Con.Air", "_Con.Air"), ("aux.txt", "aux.txt", "_aux.txt"), ("", "_", "_"), ], ) def test_sanitize_platform_specific(self, name: str, expected_unix: str, expected_windows: str) -> None: # Test Unix-like behavior assert sanitize_path_segment(name, platform='linux') == expected_unix # Test Windows behavior assert sanitize_path_segment(name, platform='win32') == expected_windows class TestFormatMediaDescription: def test_movie_with_datetime_and_id(self) -> None: movie = Movie(name="Inception", release_date=dt.datetime(2010, 7, 16), id="ID123") assert format_media_description(media_data=movie) == "Inception [2010] (ID: ID123)" def test_series_with_year_no_id(self) -> None: series = Series(series_name="The Office", series_release_date=2005) assert format_media_description(media_data=series) == "The Office [2005]" def test_season_full_with_name_and_id(self) -> None: season = Season(series_name="True Detective", series_release_date=2024, season_number=4, season_name="Night Country", id="ID321") assert format_media_description(media_data=season) == "True Detective - Season 4 - Night Country (ID: ID321)" def test_season_full_with_name_and_id_and_extra_spaces(self) -> None: season = Season(series_name=" True Detective ", series_release_date=2024, season_number=4, season_name=" Night Country ", id="ID321") assert format_media_description(media_data=season) == "True Detective - Season 4 - Night Country (ID: ID321)" def test_season_shortened_no_id(self) -> None: season = Season(series_name="Stranger Things", season_number=3) assert format_media_description(media_data=season, shortened=True) == "Season 3" def test_episode_full_with_name_and_id(self) -> None: ep = Episode(series_name="Breaking Bad", season_number=5, episode_number=14, episode_name="Ozymandias", id="ID111") assert format_media_description(media_data=ep) == "Breaking Bad - S05E14 - Ozymandias (ID: ID111)" def test_episode_shortened_no_id(self) -> None: ep = Episode(series_name="Breaking Bad", season_number=5, episode_number=14, episode_name="Ozymandias", id="ID111") assert format_media_description(media_data=ep, shortened=True) == "S05E14 - Ozymandias (ID: ID111)" class TestFormatReleaseName: def test_movie_with_source_and_web_default(self) -> None: assert format_release_name( title="Interstellar", release_date=2014, media_source="iT", ) == "Interstellar.2014.iT.WEB" def test_movie_with_source_type_none(self) -> None: assert format_release_name( title="Interstellar", release_date=2014, media_source="iT", source_type=None, ) == "Interstellar.2014.iT" def test_episode_with_source_web(self) -> None: assert format_release_name( title="Breaking Bad", season_number=5, episode_number=14, media_source="iT", ) == "Breaking.Bad.S05E14.iT.WEB" def test_episode_with_name_included(self) -> None: assert format_release_name( title="Breaking Bad", season_number=1, episode_number=1, episode_name="Pilot", media_source="iT", ) == "Breaking.Bad.S01E01.Pilot.iT.WEB" def test_additional_info_language_and_subtitles_type_and_format_enum(self) -> None: assert format_release_name( title="Interstellar", release_date=2014, media_source="iT", additional_info=["HDR", "DV"], language_code="en", subtitles_type=SubtitlesType.FORCED, file_format=SubtitlesFormatType.SUBRIP, ) == "Interstellar.2014.iT.WEB.HDR.DV.en.forced.srt" def test_movie_zip_with_source_and_web(self) -> None: assert format_release_name( title="Interstellar", release_date=2014, media_source="iT", file_format="zip", ) == "Interstellar.2014.iT.WEB.zip" ================================================ FILE: tests/tools/__init__.py ================================================ ================================================ FILE: tests/tools/generate_mock_data.py ================================================ from __future__ import annotations from abc import ABC, abstractmethod import argparse import asyncio import hashlib import inspect import json import logging from pathlib import Path import shutil import typing from typing import TYPE_CHECKING, Any, ClassVar from isubrip.cli import console from isubrip.logger import logger, setup_loggers from isubrip.scrapers.scraper import HLSScraper, PlaylistLoadError, Scraper, ScraperFactory if TYPE_CHECKING: import httpx import m3u8 from isubrip.scrapers.appletv_scraper import AppleTVScraper setup_loggers( stdout_loglevel=logging.DEBUG, stdout_console=console, logfile_output=False, ) MOCK_GENERATOR_MAPPING: dict[str, type[MockDataGenerator]] = {} class MockDataGenerator(ABC): """Abstract base class for mock data generators.""" MOCK_DATA_ROOT: ClassVar[Path] = Path(__file__).parent.parent / "mock_data" def __init_subclass__(cls, **kwargs: Any) -> None: """Automatically register mock data generator subclasses.""" super().__init_subclass__(**kwargs) if not inspect.isabstract(cls): type_hints = typing.get_type_hints(cls) scraper_class = type_hints.get("scraper") if scraper_class and hasattr(scraper_class, "id"): MOCK_GENERATOR_MAPPING[scraper_class.id] = cls else: logger.warning( f"MockDataGenerator subclass '{cls.__name__}' does not have a valid scraper class defined.") def __init__(self, scraper: Scraper): self.scraper = scraper @abstractmethod def output_path(self, url: str) -> Path: """ Generate a relative path for the output file, based on the URL. Args: url: The URL to get the output directory for. Returns: A relative Path object for the output file. """ async def generate(self, url: str, languages: list[str] | None = None, force: bool = False) -> None: """ Generate mock data for a given URL. Args: url: The URL to generate mock data for. languages: A list of languages to download subtitles for. force: If True, delete existing mock data before generating new data. """ output_dir = self.MOCK_DATA_ROOT / self.output_path(url) manifest_path = output_dir / "manifest.json" if output_dir.is_dir() and manifest_path.is_file(): if force: logger.info(f"Removing existing mock data at: {output_dir}") shutil.rmtree(output_dir) else: logger.error(f"Mock data already exists at: {output_dir}. Use the --force flag to overwrite.") return output_dir.mkdir(parents=True, exist_ok=True) manifest: dict[str, str] = {} # Define a hook to save responses async def save_response_hook(response: httpx.Response) -> None: await response.aread() url = str(response.request.url) if url in manifest: logger.debug(f"URL already processed, skipping: {url}") return logger.info(f"Intercepted response from: {url}") filename = hashlib.sha256(url.encode('utf-8')).hexdigest() file_path = output_dir / filename file_path.write_bytes(response.content) manifest[url] = filename logger.debug(f"Saved response to: {file_path}") try: # Attach the hook to the scraper's HTTP client self.scraper._client.event_hooks['response'].append(save_response_hook) # noqa: SLF001 logger.info("Fetching media data...") scraped_data = await self.scraper.get_data(url=url) if not scraped_data or not scraped_data.media_data: logger.error(f"Could not retrieve media data for {url}") return media_item = scraped_data.media_data[0] playlist_url = getattr(media_item, 'playlist', None) if not playlist_url: logger.error(f"No playlist URL found in scraped data for {url}") return main_playlist_url = playlist_url[0] if isinstance(playlist_url, list) else playlist_url logger.info(f"Loading main playlist from: {main_playlist_url}") main_playlist: m3u8.M3U8 | None = await self.scraper.load_playlist(url=main_playlist_url) if not main_playlist: logger.error(f"Could not load main playlist from {main_playlist_url}") return logger.info(f"Searching for subtitle playlists (Languages: {languages or 'all'})...") subtitle_media_items: list[m3u8.Media] = self.scraper.find_matching_subtitles( main_playlist, language_filter=languages) if not subtitle_media_items: logger.warning(f"No matching subtitles found for languages: {languages or 'all'}") return for sub_media in subtitle_media_items: try: logger.info(f"Loading subtitle playlist: {sub_media.absolute_uri}") subtitle_playlist: m3u8.M3U8 | None = await self.scraper.load_playlist( url=sub_media.absolute_uri) if subtitle_playlist and subtitle_playlist.segments: logger.info(f"Downloading {len(subtitle_playlist.segments)} segments...") if isinstance(self.scraper, HLSScraper): await self.scraper.download_segments(subtitle_playlist) except PlaylistLoadError as e: logger.error(f"Failed to load subtitle playlist {sub_media.absolute_uri}: {e}") except Exception: logger.error(f"Unexpected error processing playlist {sub_media.absolute_uri}", exc_info=True) except Exception as e: logger.error(f"An error occurred during data generation: {e}", exc_info=True) finally: if manifest: manifest_path = output_dir / "manifest.json" manifest_path.write_text(json.dumps(manifest, indent=4, sort_keys=True), encoding="utf-8") logger.info(f"Successfully generated {len(manifest)} mock data entries.") logger.info(f"Manifest written to: {manifest_path.resolve()}") class AppleTVMockDataGenerator(MockDataGenerator): """A mock data generator for the Apple TV scraper.""" scraper: AppleTVScraper def output_path(self, url: str) -> Path: url_regex_match: dict[str, Any] = self.scraper.match_url(url=url, raise_error=True).groupdict() media_id: str = url_regex_match["media_id"] storefront: str = url_regex_match["country_code"] return Path(self.scraper.id) / storefront / media_id async def main() -> None: parser = argparse.ArgumentParser( description="Generate mock HTTP response data for iSubRip testing. " "This script runs a scraper against a live URL and saves all " "HTTP responses to be used in offline tests.", ) parser.add_argument( "-u", "--url", required=True, metavar="URL", help="The full URL of the media item to fetch data for.", ) parser.add_argument( "-l", "--languages", nargs="*", default=None, help="Optional: language code(s) to filter subtitles (e.g., 'en', 'es'). Fetches all if not set.", ) parser.add_argument( "-f", "--force", action="store_true", help="Force regeneration of mock data, deleting any existing data.", ) args = parser.parse_args() scraper = ScraperFactory.get_scraper_instance(url=args.url, raise_error=True) if not scraper: return if generator_class := MOCK_GENERATOR_MAPPING.get(scraper.id): generator = generator_class(scraper=scraper) logger.info(f"Starting mock data generation for URL: {args.url}") await generator.generate(url=args.url, languages=args.languages, force=args.force) logger.info("Mock data generation process complete.") else: logger.error(f"No mock data generator found for scraper: {scraper.id}") await scraper.async_close() if __name__ == "__main__": asyncio.run(main()) ================================================ FILE: tests/tools/mock_loader.py ================================================ from __future__ import annotations import json import logging from pathlib import Path from typing import Any import httpx class MockLoader: """ An asynchronous mock loader for HTTP requests, designed to load mock data from a specified directory. """ def __init__(self, mock_data_dir: Path | str, logger: logging.Logger | None = None): self.mock_data_dir = Path(mock_data_dir) self.logger = logger or logging.getLogger(__name__) self._manifest: dict[str, Path] = {} self.logger.info(f"Mock data initialized with using the data on: {self.mock_data_dir}") manifest_paths = list(self.mock_data_dir.rglob("manifest.json")) if not manifest_paths: raise FileNotFoundError(f"No manifest file was found in {self.mock_data_dir}.") for manifest_path in manifest_paths: self.logger.info(f"Loading manifest from {manifest_path}...") try: manifest_data = json.loads(manifest_path.read_text(encoding="utf-8")) for url, filename in manifest_data.items(): self._manifest[url] = manifest_path.parent / filename self.logger.info(f"Manifest {manifest_path} loaded successfully.") except json.JSONDecodeError: self.logger.exception(f"Failed to decode manifest file: {manifest_path}") except Exception: self.logger.exception(f"Failed to load manifest file: {manifest_path}", exc_info=True) if not self._manifest: raise ValueError("No valid mock data found in any manifest files.") self.logger.info(f"Loaded {len(self._manifest)} mock data entries from {len(manifest_paths)} manifests.") async def mock_send_handler(self, request: httpx.Request, *args: Any, **kwargs: Any) -> httpx.Response: # noqa: ARG002 """An async handler to be used as a side_effect for a patched httpx.AsyncClient.send.""" url_str = str(request.url) if url_str not in self._manifest: raise KeyError(f"Mock data not found for URL: {url_str}") response_path = self._manifest[url_str] response_content = response_path.read_bytes() return httpx.Response( status_code=200, content=response_content, request=request, )