Repository: martin68/apt-smart Branch: master Commit: 870876dd6ba5 Files: 35 Total size: 219.2 KB Directory structure: gitextract_79k4xdni/ ├── .github/ │ └── workflows/ │ └── python-ci.yml ├── .gitignore ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README-zh-cn.rst ├── README.rst ├── apt_smart/ │ ├── __init__.py │ ├── backends/ │ │ ├── __init__.py │ │ ├── debian.py │ │ ├── linuxmint.py │ │ └── ubuntu.py │ ├── cli.py │ ├── http.py │ ├── releases.py │ └── tests.py ├── constraints.txt ├── docs/ │ ├── api.rst │ ├── changelog.rst │ ├── conf.py │ ├── index.rst │ └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── scripts/ │ ├── check-code-style.sh │ ├── collect-full-coverage.sh │ └── install-on-travis.sh ├── setup.cfg ├── setup.py ├── test_custom_mirrors.txt └── tox.ini ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/workflows/python-ci.yml ================================================ name: Python CI on: push: branches-ignore: - '/^[0-9]/' pull_request: branches-ignore: - '/^[0-9]/' jobs: test: runs-on: ubuntu-20.04 strategy: matrix: python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "pypy3.10"] steps: - name: Checkout repository uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: bash scripts/install-on-travis.sh - name: Run checks run: make check - name: Run full coverage run: make full-coverage - name: Upload coverage to Coveralls if: success() uses: coverallsapp/github-action@v2 with: github-token: ${{ secrets.GITHUB_TOKEN }} ================================================ FILE: .gitignore ================================================ .cache/ .coverage .coverage* .directory *.egg-info/ *.pyc ================================================ FILE: .travis.yml ================================================ sudo: true language: python python: - "2.7" - "3.4" - "3.5" - "3.6" - "3.7" - "3.8" - "pypy" install: - scripts/install-on-travis.sh script: - make check - make full-coverage after_success: - coveralls branches: except: - /^[0-9]/ ================================================ FILE: CHANGELOG.rst ================================================ Changelog ========= The purpose of this document is to list all of the notable changes to this project. The format was inspired by `Keep a Changelog`_. This project adheres to `semantic versioning`_. .. contents:: :local: .. _Keep a Changelog: http://keepachangelog.com/ .. _semantic versioning: http://semver.org/ `Release 7.1.3`_ (2020-5-31) ----------------------------- - Support 'mirror://' scheme: https://github.com/martin68/apt-smart/issues/3 - Update releases.py bundled Releases by running 'make releases', related https://github.com/martin68/apt-smart/issues/4 - In python2 decode() default encoding is ascii, causing https://github.com/martin68/apt-smart/issues/5 , specify utf-8 - Fix current_mirror in linuxmint's ubuntu mode, causing -U -c 'mirror_url' changed linuxmint's mirror_url instead of ubuntu's .. _Release 7.1.3: https://github.com/martin68/apt-smart/compare/7.1.3...7.1.3 `Release 7.1.2`_ (2019-11-28) ----------------------------- - Support Python 3.8 - Add `-C --codename` flag to create chroot with a distribution codename. - Blacklist BASE_URL mirror if matches blacklist pattern, this helps when BASE_URL (official) mirror is the only up-to-date one and you find it so slow that you'd like to blacklist it. .. _Release 7.1.2: https://github.com/martin68/apt-smart/compare/7.1.1...7.1.2 `Release 7.1.1`_ (2019-11-04) ----------------------------- - For Linux Mint, backup official-package-repositories.list to backup_dir: backup_by_apt-smart - In Readme, add install commands for Linux Mint and a note about run with sudo .. _Release 7.1.1: https://github.com/martin68/apt-smart/compare/7.1.0...7.1.1 `Release 7.1.0`_ (2019-11-01) ----------------------------- - Add support for Linux Mint - Add -U, --ubuntu to opt in Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. --auto-change-mirror --ubuntu will auto-change Linux Mint's upstream Ubuntu mirror .. _Release 7.1.0: https://github.com/martin68/apt-smart/compare/7.0.7...7.1.0 `Release 7.0.7`_ (2019-9-30) ---------------------------- - Fix install_sources_list() for Python 3 - Fix --change-mirror - fix Travis CI  `io.UnsupportedOperation:fileno` error by changing the way to run test cases containing smart_update() - More test cases .. _Release 7.0.7: https://github.com/martin68/apt-smart/compare/7.0.6...7.0.7 `Release 7.0.6`_ (2019-9-25) ---------------------------- - Readme & help about proxy setting - Deal with the [options] in sources.list by stripping it from sources.list, and then get it back when generating new sources.list , fix https://github.com/jblakeman/apt-select/issues/54 - Add a warning: custom mirror file's path and filename must NOT contain whitespace - Add -R, --create-chroot=local_dir_absolute_path to create chroot with the best mirror in a local directory with absolute_path - More test cases .. _Release 7.0.6: https://github.com/martin68/apt-smart/compare/7.0.5...7.0.6 `Release 7.0.5`_ (2019-9-21) ---------------------------- - Add -F, --file-to-read=local_file_absolute_path (path and filename must NOT contain whitespace) to read a local absolute path file containing custom mirror URLs (one URL per line) to add custom mirrors to rank. So now you can use e.g. `-l -F ~/mirrors.txt` options to add some custom mirrors to rank with mirrors in official mirror list. - Updated BUNDLED_RELEASES in releases.py .. _Release 7.0.5: https://github.com/martin68/apt-smart/compare/7.0.4...7.0.5 `Release 7.0.4`_ (2019-9-20) ---------------------------- - Fix error on EOL release - Check OLD_RELEASES_URL's MirrorStatus to confirm if it is EOL, to fix https://github.com/xolox/python-apt-mirror-updater/issues/9 .. _Release 7.0.4: https://github.com/martin68/apt-smart/compare/7.0.3...7.0.4 `Release 7.0.3`_ (2019-9-19) ---------------------------- - Fix `--url-char-len` option to specify the length of chars in mirrors' URL to display when using `--list-mirrors`, so that now you can use e.g. `-l -L 29` options to narrow down the table of ranked mirrors when you want to paste it to somewhere the table displayed badly. .. _Release 7.0.3: https://github.com/martin68/apt-smart/compare/7.0.2...7.0.3 `Release 7.0.2`_ (2019-9-19) ---------------------------- - Add :attr:url_char_len to specify the length of chars in mirrors' URL to display when using `--list-mirrors`, so that now you can use e.g. `-l -L 29` options to narrow down the table of ranked mirrors when you want to paste it to somewhere the table displayed badly. .. _Release 7.0.2: https://github.com/martin68/apt-smart/compare/7.0,1...7.0.2 `Release 7.0.1`_ (2019-9-18) ---------------------------- - Better output format when use `--list-mirrors` .. _Release 7.0.1: https://github.com/martin68/apt-smart/compare/7.0...7.0.1 `Release 7.0`_ (2019-9-15) --------------------------- - Rename the project and module to ``apt-smart`` - For Ubuntu, new mirrors discovery mechanism: at first it queries ``MIRROR_SELECTION_URL``, and ``MIRRORS_URL`` as fallback. - For Debian, new mirrors discovery mechanism: get mirrors within the country which the user is in. - New mechanism of determining whether a mirror is up-to-date: download the InRelease file and parse the ``Date`` value in it. - New and more robust ``distribution_codename`` using APT sources.list - Enable retry when ``fetch_url`` is timeout for bad connections. - Drop Python 2.6 support and add Python 3.7 - Drop max_mirrors limit since we can smartly get mirrors within the user's country. .. _Release 7.0: https://github.com/martin68/apt-smart/compare/6.1...7.0 `Release 6.1`_ (2018-10-19) --------------------------- - Bug fix for Ubuntu keyring selection that prevented ``ubuntu-archive-removed-keys.gpg`` from being used. - Bug fix for ``coerce_release()`` when given a release number. - Moved pathnames of Debian and Ubuntu keyring files to constants. - Added logging to enable debugging of keyring selection process. - Added proper tests for keyring selection and release coercion. .. _Release 6.1: https://github.com/xolox/python-apt-mirror-updater/compare/6.0...6.1 `Release 6.0`_ (2018-10-14) --------------------------- Enable the creation of Ubuntu <= 12.04 chroots on Ubuntu >= 17.04 hosts by working around (what I am convinced is) a bug in ``debootstrap`` which picks the wrong keyring when setting up chroots of old releases. For more information refer to issue `#8`_. I've bumped the major version number for this release because the highly specific ``apt_smart.eol`` module changed into the much more generic ``apt_smart.releases`` module. Also the ``release_label`` property was removed. .. _Release 6.0: https://github.com/xolox/python-apt-mirror-updater/compare/5.2...6.0 .. _#8: https://github.com/xolox/python-apt-mirror-updater/issues/8 `Release 5.2`_ (2018-10-08) --------------------------- Use `mirrors.ubuntu.com/mirrors.txt`_ without placing our full trust in it like older versions of ``apt-smart`` did 😇. Feedback in issue `#6`_ suggested that `mirrors.ubuntu.com/mirrors.txt`_ is working properly (again) and should be preferred over scraping Launchpad. However I prefer for ``apt-smart`` to be a reliable "do what I mean" program and `mirrors.ubuntu.com/mirrors.txt`_ has proven to be unreliable in the past (see the discussion in `#6`_). As a compromise I've changed the Ubuntu mirror discovery as follows: 1. Discover Ubuntu mirrors on Launchpad. 2. Try to discover mirrors using `mirrors.ubuntu.com/mirrors.txt`_ and iff successful, narrow down the list produced in step 1 based on the URLs reported in step 2. 3. Rank the discovered / narrowed down mirrors and pick the best one. The reason why I've decided to add this additional complexity is because it has bothered me in the past that Ubuntu mirror discovery was slow and this does help a lot. Also, why not use a service provided by Ubuntu to speed things up? Unrelated to the use of `mirrors.ubuntu.com/mirrors.txt`_ I've also bumped the ``executor`` requirement (twice) in order to pull in upstream improvements discussed in `executor issue #10`_ and `executor issue #15`_. .. _Release 5.2: https://github.com/xolox/python-apt-mirror-updater/compare/5.1...5.2 .. _mirrors.ubuntu.com/mirrors.txt: http://mirrors.ubuntu.com/mirrors.txt .. _#6: https://github.com/xolox/python-apt-mirror-updater/issues/6 .. _executor issue #10: https://github.com/xolox/python-executor/issues/10 .. _executor issue #15: https://github.com/xolox/python-executor/issues/15 `Release 5.1`_ (2018-06-22) --------------------------- Work on release 5.1 started with the intention of publishing a 5.0.2 bug fix release for the EOL detection of Debian LTS releases reported in `#5`_, however unrelated changes were required to stabilize the test suite. This explains how 5.0.2 became 5.1 😇. When I started working on resolving the issue reported in `#5`_ it had been quite a while since the previous release (233 days) and so some technical debt had accumulated in the project, causing the test suite to break. Most significantly, Travis CI switched their workers from Ubuntu 12.04 to 14.04. Here's a detailed overview of changes: - Bug fix for EOL detection of Debian LTS releases (reported in `#5`_). - Bug fix for trivial string matching issue in test suite (caused by a naively written test). - Bug fix for recursive ``repr()`` calls potentially causing infinite recursion, depending on logging level (see e.g. build 395421319_). - Updated bundled EOL dates based on distro-info-data available in Ubuntu 18.04. - Added this changelog to the documentation, including a link in the readme. - Make sure the ``test_gather_eol_dates`` test method runs on Travis CI (by installing the distro-info-data_ package). This exposed a Python 3 incompatibility (in build 395410569_) that has since been resolved. - Include documentation in source distributions (``MANIFEST.in``). - Silence flake8 complaining about bogus D402 issues. - Add license='MIT' key to ``setup.py`` script. - Bumped copyright to 2018. .. _Release 5.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0.1...5.1 .. _#5: https://github.com/xolox/python-apt-mirror-updater/issues/5 .. _395421319: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319 .. _distro-info-data: https://packages.ubuntu.com/distro-info-data .. _395410569: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395410569 `Release 5.0.1`_ (2017-11-01) ----------------------------- Bug fix release for invalid enumeration value (oops). .. _Release 5.0.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0...5.0.1 `Release 5.0`_ (2017-11-01) --------------------------- Reliable end of life (EOL) detection. Recently I ran into the issue that the logic to check whether a release is EOL (that works by checking if the security mirror serves a ``Release.gpg`` file for the release) failed on me. More specifically the following URL existed at the time of writing (2017-11-01) even though Ubuntu 12.04 went EOL back in April: http://security.ubuntu.com/ubuntu/dists/precise/Release.gpg At the same time issue `#1`_ and pull request `#2`_ were also indications that the EOL detection was fragile and error prone. This potential fragility had bugged me ever since publishing `apt-smart` and this week I finally finished a more robust and deterministic EOL detection scheme. This release includes pull requests `#2`_ and `#4`_, fixing issues `#1`_ and `#3`_. Here's a detailed overview of changes: - Addition: Allow optional arguments to ``apt-get update`` (`#3`_, `#4`_). - I simplified and improved the feature requested in issue `#3`_ and implemented in pull request `#4`_ by switching from an optional list argument to 'star-args' and applying the same calling convention to ``smart_update()`` as well. - This is backwards incompatible with the implementation in pull request `#4`_ (which I merged into the ``dev`` branch but never published to PyPI) and it's also technically backwards incompatible in the sense that keyword arguments could previously be given to ``smart_update()`` as positional arguments. This explains why I'm bumping the major version number. - Bug fix for incorrect marking of EOL when HTTP connections fail (`#2`_). - Refactoring: Apply timeout handling to HTTP response bodies. - Refactoring: Distinguish 404 from other HTTP errors: - This change enhances ``validate_mirror()`` by making a distinction between a confirmed HTTP 404 response versus other error conditions which may be of a more transient nature. - The goal of this change is to preserve the semantics requested in issue `#1`_ and implemented in pull request `#2`_ without needing the additional HTTP request performed by ``can_connect_to_mirror()``. - Because ``validate_mirror()`` previously returned a boolean but now returns an enumeration member this change is technically backwards incompatible, then again ``validate_mirror()`` isn't specifically intended for callers because it concerns internal logic of apt-smart. I'm nevertheless bumping the major version number. - Refactoring: Improve HTTP request exception handling: - 404 responses and timeouts are no longer subject to retrying. - The exception ``apt_smart.http.NotFoundError`` is now raised on HTTP 404 responses. Other unexpected HTTP response codes raise ``apt_smart.http.InvalidResponseError``. - The specific distinction between 404 and !200 was made because the 404 response has become significant in checking for EOL status. .. _Release 5.0: https://github.com/xolox/python-apt-mirror-updater/compare/4.0...5.0 .. _#1: https://github.com/xolox/python-apt-mirror-updater/issues/1 .. _#2: https://github.com/xolox/python-apt-mirror-updater/pull/2 .. _#3: https://github.com/xolox/python-apt-mirror-updater/issues/3 .. _#4: https://github.com/xolox/python-apt-mirror-updater/pull/4 `Release 4.0`_ (2017-06-14) --------------------------- Robust validation of available mirrors (backwards incompatible). .. _Release 4.0: https://github.com/xolox/python-apt-mirror-updater/compare/3.1...4.0 `Release 3.1`_ (2017-06-13) --------------------------- Made mirror comparison more robust. .. _Release 3.1: https://github.com/xolox/python-apt-mirror-updater/compare/3.0...3.1 `Release 3.0`_ (2017-06-13) --------------------------- Added Debian archive support (with old releases): - Addition: Added Debian archive support (old releases). - Improvement: Don't bother validating archive / old-releases mirror. - Refactoring: Moved URLs to backend specific modules. .. _Release 3.0: https://github.com/xolox/python-apt-mirror-updater/compare/2.1...3.0 `Release 2.1`_ (2017-06-12) --------------------------- Restored Python 3 compatibility, improved robustness: - Improvement: Make the ``is_available`` and ``is_updating`` properties of the ``CandidateMirror`` class more robust. - Bug fix: I suck at Unicode in Python (most people do :-p). - Cleanup: Remove unused import from test suite. .. _Release 2.1: https://github.com/xolox/python-apt-mirror-updater/compare/2.0...2.1 `Release 2.0`_ (2017-06-11) --------------------------- Generation of ``sources.list`` files and chroot creation. Detailed overview of changes: - Addition: Added a simple ``debootstrap`` wrapper. - Addition: Programmatic /etc/apt/sources.list generation - Bug fix for ``check_suite_available()``. - Bug fix: Never apply Ubuntu's old release handling to Debian. - Bug fix: Never remove ``/var/lib/apt/lists/lock file``. - Improvement: Enable stable mirror selection - Improvement: Make it possible to override distributor ID and codename - Improvement: Render interactive spinner during mirror ranking. - Refactoring: Generalize AptMirrorUpdater initializer (backwards incompatible!) - Refactoring: Generalize backend module loading - Refactoring: Modularize ``/etc/apt/sources.list`` writing. .. _Release 2.0: https://github.com/xolox/python-apt-mirror-updater/compare/1.0...2.0 `Release 1.0`_ (2017-06-08) --------------------------- Improved Ubuntu mirror discovery, added an automated test suite, and more. The bump to version 1.0 isn't so much intended to communicate that this is now mature software, it's just that I made several backwards incompatible changes in order to improve the modularity of the code base, make it easier to develop automated tests, maintain platform support, etc :-). A more detailed overview of (significant) changes: - Improved Ubuntu mirror discovery (by scraping Launchpad instead). - Extracted mirror discovery to separate (backend specific) modules. - Extracted HTTP handling to a separate module. - Enable Control-C to interrupt concurrent connection tests. - Expose limit in Python API and command line interface and make limit optional by passing 0. - Bug fix for Python 3 incompatibility: Stop using ``sys.maxint`` :-). .. _Release 1.0: https://github.com/xolox/python-apt-mirror-updater/compare/0.3.1...1.0 `Release 0.3.1`_ (2016-06-29) ----------------------------- Avoid 'nested' smart updates (the old code worked fine but gave confusing output and performed more work than necessary, which bothered me :-). .. _Release 0.3.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.3...0.3.1 `Release 0.3`_ (2016-06-29) --------------------------- Make smart update understand EOL suites .. _Release 0.3: https://github.com/xolox/python-apt-mirror-updater/compare/0.2...0.3 `Release 0.2`_ (2016-06-29) --------------------------- Bug fix: Replace ``security.ubuntu.com`` as well. .. _Release 0.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.2...0.2 `Release 0.1.2`_ (2016-06-29) ----------------------------- Bug fix: Explicitly terminate multiprocessing pool. .. _Release 0.1.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.1...0.1.2 `Release 0.1.1`_ (2016-03-10) ----------------------------- Initial release (added ``MANIFEST.in``). .. _Release 0.1.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.1...0.1.1 `Release 0.1`_ (2016-03-10) --------------------------- Initial commit. .. _Release 0.1: https://github.com/xolox/python-apt-mirror-updater/tree/0.1 ================================================ FILE: LICENSE.txt ================================================ Copyright (c) 2019 martin68 2018 Peter Odding Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: MANIFEST.in ================================================ include *.rst include *.txt graft docs ================================================ FILE: Makefile ================================================ # Makefile for the 'apt-smart' package. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io PACKAGE_NAME = apt-smart WORKON_HOME ?= $(HOME)/.virtualenvs VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME) PATH := $(VIRTUAL_ENV)/bin:$(PATH) MAKE := $(MAKE) --no-print-directory SHELL = bash default: @echo "Makefile for $(PACKAGE_NAME)" @echo @echo 'Usage:' @echo @echo ' make install install the package in a virtual environment' @echo ' make reset recreate the virtual environment' @echo ' make check check coding style (PEP-8, PEP-257)' @echo ' make test run the test suite, report coverage' @echo ' make tox run the tests on all Python versions' @echo ' make eol update apt_smart.eol module' @echo ' make readme update usage in readme' @echo ' make docs update documentation using Sphinx' @echo ' make publish publish changes to GitHub/PyPI' @echo ' make clean cleanup all temporary files' @echo install: @test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)" @test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --quiet "$(VIRTUAL_ENV)" @test -x "$(VIRTUAL_ENV)/bin/pip" || easy_install pip @test -x "$(VIRTUAL_ENV)/bin/pip" || pip install --quiet pip @pip install --quiet --requirement=requirements.txt @"$(VIRTUAL_ENV)/bin/pip" install --upgrade pip setuptools wheel @pip install --quiet flufl.enum @pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true @pip install --quiet --no-deps --ignore-installed . reset: $(MAKE) clean rm -Rf "$(VIRTUAL_ENV)" $(MAKE) install check: install @scripts/check-code-style.sh test: install @pip install --quiet --requirement=requirements-tests.txt @py.test --cov @coverage html tox: install @pip install --quiet tox && tox # The following makefile target isn't documented on purpose, I don't want # people to execute this without them knowing very well what they're doing. full-coverage: install @pip install --quiet --requirement=requirements-tests.txt @scripts/collect-full-coverage.sh @coverage html cog: install @echo Installing cog ... @pip install --quiet cogapp releases: cog @cog.py -r apt_smart/releases.py readme: cog @cog.py -r README*.rst docs: releases readme @pip install --quiet sphinx @cd docs && sphinx-build -nb html -d build/doctrees . build/html publish: install git push origin && git push --tags origin $(MAKE) clean pip install --quiet twine wheel python setup.py sdist bdist_wheel twine upload dist/* $(MAKE) clean clean: @rm -Rf *.egg .cache .coverage .coverage.* .tox build dist docs/build htmlcov @find -depth -type d -name __pycache__ -exec rm -Rf {} \; @find -type f -name '*.pyc' -delete .PHONY: default install reset check test tox readme docs publish clean ================================================ FILE: README-zh-cn.rst ================================================ apt-smart: 智能的 Debian/Ubuntu/Linux Mint 镜像源自动选择工具 ================================================================= .. image:: https://travis-ci.org/martin68/apt-smart.svg?branch=master :target: https://travis-ci.org/martin68/apt-smart .. image:: https://coveralls.io/repos/martin68/apt-smart/badge.svg?branch=master :target: https://coveralls.io/r/martin68/apt-smart?branch=master `apt-smart` 提供健壮的 Debian_ 、 Ubuntu_ 和 `Linux Mint`_ apt-get_ 镜像源 (也称软件源) 自动选择。 它能智能发现镜像源、排序镜像源并且自动切换，以及实现健壮的包列表更新 (参见 features_). 目前在 Python 2.7, 3.4, 3.5, 3.6, 3.7, 3.8 和 PyPy 测试通过 (尽管test coverage目前还很低，参见 status_). .. contents:: :local: Why? -------- 作为 `apt-mirror-updater `_ 的继承者, 为你寻找最好镜像源的过程中 `apt-smart` 的智能、速度、准确性和健壮性方面都有提升和改进 (参见 changelog_)。 并且有计划增加反向代理模式——在设置好之后你就可以忘掉它，它在后台运行不需要root权限，在任何时候都指向最好的镜像源。 其他发行版如 Linux Mint（已完成！）， ROS等的支持也在计划之中。 .. _features: Features -------- **智能发现可用的镜像源** 通过查询 `Debian mirror list `_ 或 `Ubuntu mirror list1 `_ 或 `Ubuntu mirror list2 `_ 或 `Linux Mint mirror list `_ (自动选择镜像源列表)来 自动查找 Debian_ 、 Ubuntu_ 和 `Linux Mint`_ 镜像源。它能够智能地获取用户所在国家的镜像源。 **智能排序可用的镜像源** 可用镜像源按照如下方式排序：带宽、是否更新及时（up-to-date），并且排除了正在更新的镜像源 (参见 `issues with mirror updates`_)。 例如使用 `--list-mirrors` 参数将会有类似输出： .. code-block:: sh ----------------------------------------------------------------------------------------------------- | Rank | Mirror URL | Available? | Updating? | Last updated | Bandwidth | ----------------------------------------------------------------------------------------------------- | 1 | http://archive.ubuntu.com/ubuntu | Yes | No | Up to date | 16.95 KB/s | | 2 | http://mirrors.cqu.edu.cn/ubuntu | Yes | No | 3 hours behind | 427.43 KB/s | | 3 | http://mirrors.nju.edu.cn/ubuntu | Yes | No | 5 hours behind | 643.27 KB/s | | 4 | http://mirrors.tuna.tsinghua.e...| Yes | No | 5 hours behind | 440.09 KB/s | | 5 | http://mirrors.cn99.com/ubuntu | Yes | No | 13 hours behind | 2.64 MB/s | | 6 | http://mirrors.huaweicloud.com...| Yes | No | 13 hours behind | 532.01 KB/s | | 7 | http://mirrors.dgut.edu.cn/ubuntu| Yes | No | 13 hours behind | 328.25 KB/s | | 8 | http://mirrors.aliyun.com/ubuntu | Yes | No | 23 hours behind | 1.06 MB/s | | 9 | http://ftp.sjtu.edu.cn/ubuntu | Yes | No | 23 hours behind | 647.2 KB/s | | 10 | http://mirrors.yun-idc.com/ubuntu| Yes | No | 23 hours behind | 526.6 KB/s | | 11 | http://mirror.lzu.edu.cn/ubuntu | Yes | No | 23 hours behind | 210.99 KB/s | | 12 | http://mirrors.ustc.edu.cn/ubuntu| Yes | Yes | 8 hours behind | 455.02 KB/s | | 13 | http://mirrors.sohu.com/ubuntu | No | No | Unknown | 90.28 bytes/s | ----------------------------------------------------------------------------------------------------- **自动切换镜像源** 设置在 ``/etc/apt/sources.list`` 的镜像源可以用一条很简单的命令更改。你可以让它自动选择镜像源或者由你指定。 **健壮的包列表更新** 好几个 apt-get_ 的子命令在更新的过程中可能会失败 (参见 `issues with mirror updates`_) ，而`apt-smart` 通过 wrap ``apt-get update`` 可以在检测到错误时重试，并且在检测到当前镜像源在“更新中“时自动切换至另一个镜像 (因为曾经出现过“更新中“的状态持续很长时间，这有时是不可接受的，特别是在自动化维护的时候) .. _status: Status ------ 尽管设置了自动测试，但项目还处于早期状态，所以: .. warning:: `apt-smart` 弄坏你的系统可别怪我没警告过你哦! 但碰上最糟糕的情况恐怕也只有 ``/etc/apt/sources.list`` 损坏了吧。程序在做任何更改前都会自动备份一次， 所以应该能手动恢复。 Installation ------------ `apt-smart` 可以在 PyPI_ 上找到，所以安装应该很简单（把下面命令全部一次性粘贴到终端窗口）: .. code-block:: sh sudo apt update sudo apt install python-pip python-setuptools python-wheel -y # 不询问直接安装python-pip等依赖 pip install --user apt-smart # --user参数表示安装到per user site-packages directory echo "export PATH=\$(python -c 'import site; print(site.USER_BASE + \"/bin\")'):\$PATH" >> ~/.bashrc source ~/.bashrc # 设置per user site-packages directory到PATH 安装 Python 包有几种方法 (例如 `per user site-packages directory`_, 或 `virtual environments`_ 或 安装到系统全局) 在这里不详细展开叙述。 如果 apt-smart 有新版本发布了， 你可以通过如下命令升级： .. code-block:: sh pip install --user apt-smart --upgrade **注意** ： apt-smart 是个 APT 的助手工具，而 **不是** apt/apt-get 命令的替代，所以通常 apt-smart 不应该用 ``sudo`` 或以 ``su`` 运行， 如果 apt-smart 需要 root 最高权限以继续（例如更改 sources.list），它会让用户输入密码。 Usage ----- 使用 `apt-smart` 有两种方法: 作为命令行工具 ``apt-smart`` 以及作为 Python API. 作为 Python API 的详细信息请参考文档—— `Read the Docs`_. 其命令行接口如下所示： .. contents:: :local: .. A DRY solution to avoid duplication of the `apt-smart --help' text: .. .. [[[cog .. from humanfriendly.usage import inject_usage .. inject_usage('apt_smart.cli') .. ]]] **Usage:** `apt-smart [OPTIONS]` The apt-smart program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating. **Supported options:** .. csv-table:: :header: Option, Description :widths: 30, 70 "``-r``, ``--remote-host=SSH_ALIAS``","Operate on a remote system instead of the local system. The ``SSH_ALIAS`` argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access." "``-f``, ``--find-current-mirror``","Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output." "``-F``, ``--file-to-read=local_file_absolute_path``","Read a local absolute path (path and filename must NOT contain whitespace) file containing custom mirror URLs (one URL per line) to add custom mirrors to rank." "``-b``, ``--find-best-mirror``","Discover available mirrors, rank them, select the best one and report its URL on standard output." "``-l``, ``--list-mirrors``",List available (ranked) mirrors on the terminal in a human readable format. "``-L``, ``--url-char-len=int``","An integer to specify the length of chars in mirrors' URL to display when using ``--list-mirrors``, default is 34" "``-c``, ``--change-mirror=MIRROR_URL``",Update /etc/apt/sources.list to use the given ``MIRROR_URL``. "``-a``, ``--auto-change-mirror``","Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror." "``-u``, ``--update``, ``--update-package-lists``","Update the package lists using ""apt-get update"", retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated." "``-U``, ``--ubuntu``","Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. ``--auto-change-mirror`` ``--ubuntu`` will auto-change Linux Mint's upstream Ubuntu mirror" "``-x``, ``--exclude=PATTERN``","Add a pattern to the mirror selection blacklist. ``PATTERN`` is expected to be a shell pattern (containing wild cards like ""?"" and ""\*"") that is matched against the full URL of each mirror." "``-v``, ``--verbose``",Increase logging verbosity (can be repeated). "``-V``, ``--version``",Show version number and Python version. "``-R``, ``--create-chroot=local_dir_absolute_path``",Create chroot with the best mirror in a local directory with absolute_path "``-q``, ``--quiet``",Decrease logging verbosity (can be repeated). "``-h``, ``--help``"," Show this message and exit. Note: since apt-smart uses `urlopen` method in The Python Standard Library, you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type: export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080' These will not persist however (no longer active after you close the terminal), so you may wish to add the line to your ~/.bashrc" .. [[[end]]] .. _issues with mirror updates: Issues with mirror updates -------------------------- 最常见的 ``apt-get update`` 错误是 'hash sum mismatch' (参见 `Debian bug #624122`_)。 当错误产生的时候，一个名为 ``Archive-Update-in-Progress-*`` 的文件有时会出现 该镜像源的首页 (参见 `Debian bug #110837`_). 这个状态有时会持续很长时间。 My working theory about these 'hash sum mismatch' errors is that they are caused by the fact that mirror updates aren't atomic, apparently causing ``apt-get update`` to download a package list whose datafiles aren't consistent with each other. If this assumption proves to be correct (and also assuming that different mirrors are updated at different times :-) then the command ``apt-smart --update-package-lists`` should work around this annoying failure mode (by automatically switching to a different mirror when 'hash sum mismatch' errors are encountered). Publishing `apt-smart` to the world is my attempt to contribute to this situation instead of complaining in bug trackers (see above) where no robust and automated solution is emerging (at the time of writing). Who knows, maybe some day these issues will be resolved by moving logic similar to what I've implemented here into ``apt-get`` itself. Of course it would also help if mirror updates were atomic... Contact ------- The latest version of `apt-smart` is available on PyPI_ and GitHub_. The documentation is hosted on `Read the Docs`_ and includes a changelog_. For bug reports please create an issue on GitHub_. License ------- This software is licensed under the `MIT license`_. © 2020 martin68 © 2018 Peter Odding. .. External references: .. _apt-get: https://en.wikipedia.org/wiki/Advanced_Packaging_Tool .. _at work: http://www.paylogic.com/ .. _changelog: https://apt-smart.readthedocs.io/en/latest/changelog.html .. _Debian bug #110837: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=110837 .. _Debian bug #624122: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624122 .. _Debian: https://en.wikipedia.org/wiki/Debian .. _documentation: https://apt-smart.readthedocs.io .. _GitHub: https://github.com/martin68/apt-smart .. _Linux Mint: https://linuxmint.com .. _MIT license: http://en.wikipedia.org/wiki/MIT_License .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ .. _PyPI: https://pypi.python.org/pypi/apt-smart .. _Read the Docs: https://apt-smart.readthedocs.io .. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system) .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ ================================================ FILE: README.rst ================================================ apt-smart: Smart, automated Debian/Ubuntu/Linux Mint mirror selection ===================================================================== .. image:: https://travis-ci.org/martin68/apt-smart.svg?branch=master :target: https://travis-ci.org/martin68/apt-smart .. image:: https://coveralls.io/repos/martin68/apt-smart/badge.svg?branch=master :target: https://coveralls.io/r/martin68/apt-smart?branch=master `简体中文 `_ The `apt-smart` package automates robust apt-get_ mirror (a.k.a Repositories, Sources) selection for Debian_ , Ubuntu_ and `Linux Mint`_ by enabling smart discovery of available mirrors, smart ranking of available mirrors, automatic switching between mirrors and robust package list updating (see features_). It's currently tested on Python 2.7, 3.4, 3.5, 3.6, 3.7, 3.8 and PyPy (although test coverage is still rather low, see status_). .. contents:: :local: Why? -------- As a successor of `apt-mirror-updater `_, `apt-smart` has many improvements in intelligence, speed, accuracy and robustness (see changelog_) when offering the best mirror for you. It has a plan to optionally be a set-and-forget smart daemon: running in the background as a reverse proxy always redirecting to the best mirror without root privilege. It also has a plan to support other distros like: Linux Mint (Done!) , ROS... .. _features: Features -------- **Smart discovery of available mirrors** Debian_ , Ubuntu_ and `Linux Mint`_ mirrors are discovered automatically by querying the `Debian mirror list `_ or the `Ubuntu mirror list1 `_ or the `Ubuntu mirror list2 `_ or the `Linux Mint mirror list `_ (the applicable mirror list is automatically selected based on the current platform). It can smartly get mirrors within the country which the user is in. **Smart ranking of available mirrors** Discovered mirrors are ranked by bandwidth (to pick the fastest mirror) and whether they're up-to-date and excluded if they're being updated (see `issues with mirror updates`_). e.g. with `--list-mirrors` flag it would output like this: .. code-block:: sh ----------------------------------------------------------------------------------------------------- | Rank | Mirror URL | Available? | Updating? | Last updated | Bandwidth | ----------------------------------------------------------------------------------------------------- | 1 | http://archive.ubuntu.com/ubuntu | Yes | No | Up to date | 16.95 KB/s | | 2 | http://mirrors.cqu.edu.cn/ubuntu | Yes | No | 3 hours behind | 427.43 KB/s | | 3 | http://mirrors.nju.edu.cn/ubuntu | Yes | No | 5 hours behind | 643.27 KB/s | | 4 | http://mirrors.tuna.tsinghua.e...| Yes | No | 5 hours behind | 440.09 KB/s | | 5 | http://mirrors.cn99.com/ubuntu | Yes | No | 13 hours behind | 2.64 MB/s | | 6 | http://mirrors.huaweicloud.com...| Yes | No | 13 hours behind | 532.01 KB/s | | 7 | http://mirrors.dgut.edu.cn/ubuntu| Yes | No | 13 hours behind | 328.25 KB/s | | 8 | http://mirrors.aliyun.com/ubuntu | Yes | No | 23 hours behind | 1.06 MB/s | | 9 | http://ftp.sjtu.edu.cn/ubuntu | Yes | No | 23 hours behind | 647.2 KB/s | | 10 | http://mirrors.yun-idc.com/ubuntu| Yes | No | 23 hours behind | 526.6 KB/s | | 11 | http://mirror.lzu.edu.cn/ubuntu | Yes | No | 23 hours behind | 210.99 KB/s | | 12 | http://mirrors.ustc.edu.cn/ubuntu| Yes | Yes | 8 hours behind | 455.02 KB/s | | 13 | http://mirrors.sohu.com/ubuntu | No | No | Unknown | 90.28 bytes/s | ----------------------------------------------------------------------------------------------------- **Automatic switching between mirrors** The main mirror configured in ``/etc/apt/sources.list`` can be changed with a single command. The new (to be configured) mirror can be selected automatically or configured explicitly by the user. **Robust package list updating** Several apt-get_ subcommands can fail if the current mirror is being updated (see `issues with mirror updates`_) and `apt-smart` tries to work around this by wrapping ``apt-get update`` to retry on failures and automatically switch to a different mirror when it looks like the current mirror is being updated (because I've seen such updates take more than 15 minutes and it's not always acceptable to wait for so long, especially in automated solutions). .. _status: Status ------ On the one hand the `apt-smart` package was developed based on quite a few years of experience in using apt-get_ on Debian_ and Ubuntu_ systems. On the other hand the Python package itself is relatively new: it was developed and published in Sep 2019. As such: .. warning:: Until `apt-smart` has been rigorously tested I consider it a proof of concept (beta software) so if it corrupts your system you can't complain that you weren't warned! The worst that can happen (assuming you trust my judgement ;-) is that ``/etc/apt/sources.list`` is corrupted however a backup copy is made before any changes are applied, so I don't see how this can result in irreversible corruption. I'm working on an automated test suite but at the moment I'm still a bit fuzzy on how to create representative tests for the error handling code paths (also, writing a decent test suite requires a significant chunk of time :-). Installation ------------ The `apt-smart` package is available on PyPI_ which means installation should be as simple as (paste all below commands together into terminal): .. code-block:: sh sudo apt update sudo apt install python-pip python-setuptools python-wheel -y # install python-pip and so on without asking pip install --user apt-smart # --user flag means install to per user site-packages directory(see below) echo "export PATH=\$(python -c 'import site; print(site.USER_BASE + \"/bin\")'):\$PATH" >> ~/.bashrc source ~/.bashrc # set per user site-packages directory to PATH There's actually a multitude of ways to install Python packages (e.g. the `per user site-packages directory`_, `virtual environments`_ or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-). If a new version of apt-smart has been released, you can upgrade it via: .. code-block:: sh pip install --user apt-smart --upgrade **Note**. ``apt-smart`` is a *helper* for the ``apt`` tool. It is **NOT** a *replacement* for ``apt`` (or for ``apt-get``). So, ``apt-smart`` should *not* be run *instead* of either of those commands. Nor should ``apt-smart`` be run with ``sudo`` or via ``su``; if ``apt-smart`` happens to need root privilege in order for it to continue (in order that it may, for example, change ``sources.list``), then it will prompt for a password. Usage ----- There are two ways to use the `apt-smart` package: As the command line program ``apt-smart`` and as a Python API. For details about the Python API please refer to the API documentation available on `Read the Docs`_. The command line interface is described below. .. contents:: :local: .. A DRY solution to avoid duplication of the `apt-smart --help' text: .. .. [[[cog .. from humanfriendly.usage import inject_usage .. inject_usage('apt_smart.cli') .. ]]] **Usage:** `apt-smart [OPTIONS]` The apt-smart program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating. **Supported options:** .. csv-table:: :header: Option, Description :widths: 30, 70 "``-r``, ``--remote-host=SSH_ALIAS``","Operate on a remote system instead of the local system. The ``SSH_ALIAS`` argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access." "``-f``, ``--find-current-mirror``","Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output." "``-F``, ``--file-to-read=local_file_absolute_path``","Read a local absolute path (path and filename must NOT contain whitespace) file containing custom mirror URLs (one URL per line) to add custom mirrors to rank." "``-b``, ``--find-best-mirror``","Discover available mirrors, rank them, select the best one and report its URL on standard output." "``-l``, ``--list-mirrors``",List available (ranked) mirrors on the terminal in a human readable format. "``-L``, ``--url-char-len=int``","An integer to specify the length of chars in mirrors' URL to display when using ``--list-mirrors``, default is 34" "``-c``, ``--change-mirror=MIRROR_URL``",Update /etc/apt/sources.list to use the given ``MIRROR_URL``. "``-a``, ``--auto-change-mirror``","Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror." "``-u``, ``--update``, ``--update-package-lists``","Update the package lists using ""apt-get update"", retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated." "``-U``, ``--ubuntu``","Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. ``--auto-change-mirror`` ``--ubuntu`` will auto-change Linux Mint's upstream Ubuntu mirror" "``-x``, ``--exclude=PATTERN``","Add a pattern to the mirror selection blacklist. ``PATTERN`` is expected to be a shell pattern (containing wild cards like ""?"" and ""\*"") that is matched against the full URL of each mirror." "``-v``, ``--verbose``",Increase logging verbosity (can be repeated). "``-V``, ``--version``",Show version number and Python version. "``-R``, ``--create-chroot=local_dir_absolute_path``",Create chroot with the best mirror in a local directory with absolute_path "``-q``, ``--quiet``",Decrease logging verbosity (can be repeated). "``-h``, ``--help``"," Show this message and exit. Note: since apt-smart uses `urlopen` method in The Python Standard Library, you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type: export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080' These will not persist however (no longer active after you close the terminal), so you may wish to add the line to your ~/.bashrc" .. [[[end]]] .. _issues with mirror updates: Issues with mirror updates -------------------------- The most frequent failure that we run into is ``apt-get update`` crapping out with 'hash sum mismatch' errors (see also `Debian bug #624122`_). When this happens a file called ``Archive-Update-in-Progress-*`` can sometimes be found on the index page of the mirror that is being used (see also `Debian bug #110837`_). I've seen these situations last for more than 15 minutes. My working theory about these 'hash sum mismatch' errors is that they are caused by the fact that mirror updates aren't atomic, apparently causing ``apt-get update`` to download a package list whose datafiles aren't consistent with each other. If this assumption proves to be correct (and also assuming that different mirrors are updated at different times :-) then the command ``apt-smart --update-package-lists`` should work around this annoying failure mode (by automatically switching to a different mirror when 'hash sum mismatch' errors are encountered). Publishing `apt-smart` to the world is my attempt to contribute to this situation instead of complaining in bug trackers (see above) where no robust and automated solution is emerging (at the time of writing). Who knows, maybe some day these issues will be resolved by moving logic similar to what I've implemented here into ``apt-get`` itself. Of course it would also help if mirror updates were atomic... Contact ------- The latest version of `apt-smart` is available on PyPI_ and GitHub_. The documentation is hosted on `Read the Docs`_ and includes a changelog_. For bug reports please create an issue on GitHub_. License ------- This software is licensed under the `MIT license`_. © 2020 martin68 © 2018 Peter Odding. .. External references: .. _apt-get: https://en.wikipedia.org/wiki/Advanced_Packaging_Tool .. _at work: http://www.paylogic.com/ .. _changelog: https://apt-smart.readthedocs.io/en/latest/changelog.html .. _Debian bug #110837: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=110837 .. _Debian bug #624122: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624122 .. _Debian: https://en.wikipedia.org/wiki/Debian .. _documentation: https://apt-smart.readthedocs.io .. _GitHub: https://github.com/martin68/apt-smart .. _Linux Mint: https://linuxmint.com .. _MIT license: http://en.wikipedia.org/wiki/MIT_License .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ .. _PyPI: https://pypi.python.org/pypi/apt-smart .. _Read the Docs: https://apt-smart.readthedocs.io .. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system) .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ ================================================ FILE: apt_smart/__init__.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: May 31, 2020 # URL: https://apt-smart.readthedocs.io """ Automated, robust ``apt-get`` mirror selection for Debian and Ubuntu. The main entry point for this module is the :class:`AptMirrorUpdater` class, so if you don't know where to start that would be a good place :-). You can also take a look at the source code of the :mod:`apt_smart.cli` module for an example that uses the :class:`AptMirrorUpdater` class. """ # Standard library modules. import fnmatch import logging import os import sys import time import calendar # Python 2.x / 3.x compatibility. try: from enum import Enum except ImportError: from flufl.enum import Enum # External dependencies. from capturer import CaptureOutput from executor.contexts import ChangeRootContext, LocalContext from humanfriendly import AutomaticSpinner, Timer, compact, format_timespan, pluralize try: from property_manager3 import ( PropertyManager, cached_property, key_property, lazy_property, mutable_property, set_property, ) except ImportError: from property_manager import ( PropertyManager, cached_property, key_property, lazy_property, mutable_property, set_property, ) from six import text_type from six.moves.urllib.parse import urlparse # Modules included in our package. from apt_smart.http import NotFoundError, fetch_concurrent, fetch_url, get_default_concurrency from apt_smart.releases import coerce_release from apt_smart.releases import discover_releases # Semi-standard module versioning. __version__ = '7.1.3' SOURCES_LIST_ENCODING = 'UTF-8' """The text encoding of :attr:`main_sources_list` (a string).""" MAX_MIRRORS = 50 """A sane default value for :attr:`AptMirrorUpdater.max_mirrors`.""" URL_CHAR_LEN = 34 """A default value for :attr:`AptMirrorUpdater.url_char_len`.""" LAST_UPDATED_DEFAULT = 60 * 60 * 24 * 7 * 4 """A default, pessimistic :attr:`~CandidateMirror.last_updated` value (a number).""" # Initialize a logger for this module. logger = logging.getLogger(__name__) class AptMirrorUpdater(PropertyManager): """Python API for the `apt-smart` program.""" repr_properties = ( 'architecture', 'backend', 'blacklist', 'concurrency', 'context', 'distribution_codename', 'distributor_id', 'max_mirrors', 'old_releases_url', 'security_url', ) """ Override the list of properties included in :func:`repr()` output (a tuple of strings). The :class:`~property_manager.PropertyManager` superclass defines a :class:`~property_manager.PropertyManager.__repr__()` method that includes the values of computed properties in its output. In the case of `apt-smart` this behavior would trigger external command execution and (lots of) HTTP calls, sometimes with unintended side effects, namely `infinite recursion`_. By setting :attr:`repr_properties` to a list of "safe" properties this problematic behavior can be avoided. .. _infinite recursion: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319 """ @mutable_property def architecture(self): """ The name of the Debian package architecture (a string like 'i386' or 'amd64'). The package architecture is used to detect whether `Debian LTS`_ status applies to the given system (the Debian LTS team supports a specific subset of package architectures). .. _Debian LTS: https://wiki.debian.org/LTS """ value = self.context.capture('dpkg', '--print-architecture') set_property(self, 'architecture', value) return value @cached_property def available_mirrors(self): """A list of :class:`CandidateMirror` objects (ordered from best to worst)""" mirrors = set() if self.release_is_eol: logger.warning("Skipping mirror discovery because %s is EOL.", self.release) else: if self.read_custom_mirror_file: mirrors.update(self.read_custom_mirror_file) logger.info("Custom mirrors added from file:") for mirror in mirrors: logger.info(mirror.mirror_url) logger.info("Adding BASE_URL mirror:") if self.distributor_id == 'debian': # For Debian, base_url typically is not in MIRRORS_URL, # add it explicitly base_url_prefix = self.backend.BASE_URL.split('/dists/codename-updates/Release')[0] mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self)) elif self.distributor_id == 'ubuntu': # For Ubuntu, base_url is not in MIRRORS_URL for # some countries e.g. US (found it in Travis CI), add it explicitly base_url_prefix = self.backend.BASE_URL.split('/dists/codename-security/Release')[0] mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self)) elif self.distributor_id == 'linuxmint': # For Linux Mint, base_url typically is not in MIRRORS_URL, # add it explicitly base_url_prefix = self.backend.BASE_URL.split('/dists/codename/Release')[0] mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self)) logger.info(base_url_prefix) for candidate in self.backend.discover_mirrors(): if any(fnmatch.fnmatch(candidate.mirror_url, pattern) for pattern in self.blacklist)\ and normalize_mirror_url(candidate.mirror_url) != base_url_prefix: logger.warning("Ignoring blacklisted mirror %s.", candidate.mirror_url) else: candidate.updater = self mirrors.add(candidate) # We make an attempt to incorporate the system's current mirror in # the candidates but we don't propagate failures while doing so. try: # Gotcha: We should never include the system's current mirror in # the candidates when we're bootstrapping a chroot for a different # platform. # if self.distributor_id == self.context.distributor_id: # We don't need to check this since # 60850cc2 commint (Reimplement more robust :attr:`distribution_codename` using APT sources.list) # already using :attr:`context` and self.context.distributor_id has issue: # https://github.com/xolox/python-executor/issues/17 mirrors.add(CandidateMirror(mirror_url=self.current_mirror, updater=self)) except Exception as e: logger.warning("Failed to add current mirror to set of available mirrors! (%s)", e) # Sort the mirrors based on the currently available information. return sorted(mirrors, key=lambda c: c.sort_key, reverse=True) @cached_property def backend(self): """ The backend module whose name matches :attr:`distributor_id`. :raises: :exc:`~exceptions.EnvironmentError` when no matching backend module is available. """ logger.debug("Checking whether %s platform is supported ..", self.distributor_id.capitalize()) module_path = "%s.backends.%s" % (__name__, self.distributor_id) try: __import__(module_path) except ImportError: msg = "%s platform is unsupported! (only Debian and Ubuntu are supported)" raise EnvironmentError(msg % self.distributor_id.capitalize()) else: return sys.modules[module_path] @cached_property def best_mirror(self): """ The URL of the first mirror in :attr:`ranked_mirrors` (a string). This is a shortcut for using :attr:`ranked_mirrors` to select the best mirror from :attr:`available_mirrors`, falling back to the old releases URL when :attr:`release_is_eol` is :data:`True`. """ logger.debug("Selecting best %s mirror ..", self.distributor_id.capitalize()) if self.release_is_eol: logger.info("%s is EOL, using %s.", self.release, self.old_releases_url) return self.old_releases_url else: return self.ranked_mirrors[0].mirror_url @cached_property def blacklist(self): """ A set of strings with :mod:`fnmatch` patterns (defaults to an empty set). When :attr:`available_mirrors` encounters a mirror whose URL matches one of the patterns in :attr:`blacklist` the mirror will be ignored. """ return set() @mutable_property def concurrency(self): """ The number of concurrent HTTP connections allowed while ranking mirrors (a number). The value of this property defaults to the value computed by :func:`.get_default_concurrency()`. """ return get_default_concurrency() @mutable_property(cached=True) def context(self): """ An execution context created using :mod:`executor.contexts`. The value of this property defaults to a :class:`~executor.contexts.LocalContext` object. """ return LocalContext() @mutable_property(cached=True) def current_mirror(self): """ The URL of the main mirror in use in :attr:`main_sources_list` (a string). The :attr:`current_mirror` property's value is computed using :func:`find_current_mirror()`, but can be changed and cached by :func:`distribution_codename` for Linux Mint's Ubuntu Mode. """ if self.ubuntu_mode and self.distribution_codename: # :func:`distribution_codename` will set current_mirror return self.current_mirror else: logger.debug("Parsing %s to find current mirror of %s ..", self.main_sources_list, self.context) return find_current_mirror(self.get_sources_list()) @mutable_property def distribution_codename_old(self): """ deprecated: The distribution codename (a lowercase string like 'trusty' or 'xenial'). This relies on :mod:`executor` which is not robust to detect codename when neither /etc/lsb-release nor lsb_release command are available, e.g. the official Debian docker image (see https://github.com/xolox/python-executor/issues/17 ) The value of this property defaults to the value of the :attr:`executor.contexts.AbstractContext.distribution_codename` property which is the right choice 99% of the time. """ return self.context.distribution_codename @mutable_property(cached=True) def distribution_codename(self): """ The distribution codename (a lowercase string like 'trusty' or 'xenial') The value of this property is determined using APT sources.list and should be more robust. Similar to :func:`find_current_mirror` but return token[2] instead. Also refer code of :func:`coerce_release`. """ for line in self.get_sources_list().splitlines(): # The first token should be `deb' or `deb-src', the second token is # the mirror's URL, the third token is the `distribution' and any # further tokens are `components'. tokens = line.split() if (len(tokens) >= 4 and tokens[0] in ('deb', 'deb-src') and tokens[1].startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/')) and 'main' in tokens[3:]): matches = [release for release in discover_releases() if tokens[2].lower() in release.codename.lower()] if len(matches) != 1: continue if self.ubuntu_mode and matches[0].distributor_id == 'linuxmint': self.current_mirror = tokens[1] continue if self.ubuntu_mode: logging.info("In Ubuntu Mode, pretend to be %s" % coerce_release(tokens[2])) return tokens[2] raise EnvironmentError("Failed to determine the distribution codename using apt's package resource list!") @mutable_property(cached=True) def distributor_id(self): """ The distributor ID (a lowercase string like 'debian' or 'ubuntu'). The default value of this property is based on the :attr:`~apt_smart.releases.Release.distributor_id` property of :attr:`release` (which in turn is based on :attr:`distribution_codename`). Because Debian and Ubuntu code names are unambiguous this means that in practice you can provide a value for :attr:`distribution_codename` and omit :attr:`distributor_id` and everything should be fine. """ return self.release.distributor_id @cached_property def main_sources_list(self): """ The absolute pathname of the list of configured APT data sources (a string). For new version of Linux Mint, main_sources_list is: /etc/apt/sources.list.d/official-package-repositories.list """ if self.context.exists('/etc/apt/sources.list.d/official-package-repositories.list'): logger.debug("/etc/apt/sources.list.d/official-package-repositories.list exists,\ use it as main_sources_list instead of /etc/apt/sources.list") return '/etc/apt/sources.list.d/official-package-repositories.list' else: return '/etc/apt/sources.list' @mutable_property def max_mirrors(self): """Limits the number of mirrors to rank (a number, defaults to :data:`MAX_MIRRORS`).""" return MAX_MIRRORS @mutable_property def url_char_len(self): """ The length of chars in mirrors' URL to display(a number, defaults to :data:`URL_CHAR_LEN`) Specify the length of chars in mirrors' URL to display when using --list-mirrors """ return URL_CHAR_LEN @mutable_property def ubuntu_mode(self): """ For Linux Mint, deal with upstream Ubuntu mirror instead of Linux Mint mirror if True Default is False, can be set True via -U, --ubuntu flag """ return False @mutable_property def old_releases_url(self): """The URL of the mirror that serves old releases for this :attr:`backend` (a string).""" return self.backend.OLD_RELEASES_URL @mutable_property def base_url(self): """The actual official base URL according to :data:`.BASE_URL`""" return self.backend.BASE_URL.replace('codename', self.distribution_codename) @mutable_property def base_last_updated(self): """ The Unix timestamp to determine which mirrors are up-to-date (an int) The value of this property is gotten from :attr:`base_url`'s update date as minuend """ @cached_property def ranked_mirrors(self): """ A list of :class:`CandidateMirror` objects (ordered from best to worst). The value of this property is computed by concurrently testing the mirrors in :attr:`available_mirrors` for the following details: - availability (:attr:`~CandidateMirror.is_available`) - connection speed (:attr:`~CandidateMirror.bandwidth`) - update status (:attr:`~CandidateMirror.is_updating`) The number of mirrors to test is limited to :attr:`max_mirrors` and you can change the number of simultaneous HTTP connections allowed by setting :attr:`concurrency`. """ timer = Timer() # Sort the candidates based on the currently available information # (and transform the input argument into a list in the process). mirrors = sorted(self.available_mirrors, key=lambda c: c.sort_key, reverse=True) # Limit the number of candidates to a reasonable number? # NO, we don't need to now since the backends.debian can smartly get mirrors within a country. # Without max_mirrors limit we can fix errors within United States (Travis CI reported) where # where we can get 80+ mirrors. If limit applies, base_url mirror may be deleted, then error occurs. """ if self.max_mirrors and len(mirrors) > self.max_mirrors: mirrors = mirrors[:self.max_mirrors] """ # Prepare the Release.gpg URLs to fetch. mapping = dict((c.release_gpg_url, c) for c in mirrors) num_mirrors = pluralize(len(mapping), "mirror") logger.info("Checking %s for availability and performance ..", num_mirrors) # Concurrently fetch the Release.gpg files. with AutomaticSpinner(label="Checking mirrors"): for url, data, elapsed_time in fetch_concurrent(mapping.keys(), concurrency=self.concurrency): candidate = mapping[url] candidate.release_gpg_contents = data candidate.release_gpg_latency = elapsed_time logger.info("Start retrieving :attr:`base_last_updated` using is_available") self.base_last_updated = 0 if self.release_is_eol: self.base_last_updated = int(time.time()) logger.warning("%s is EOL, so using time.time() as :attr:`base_last_updated`: %i", self.release, self.base_last_updated) elif mapping[self.base_url].is_available: logger.debug(":attr:`base_last_updated`: %i", self.base_last_updated) # base_url 's contents are up-to-date naturally,so set its last_updated 0 mapping[self.base_url].last_updated = 0 else: # base_url not available, use time at the moment as base_last_updated. self.base_last_updated = int(time.time()) logger.warning("%s is not available, so using time.time() as :attr:`base_last_updated`: %i", self.base_url, self.base_last_updated) # Concurrently check for Archive-Update-in-Progress markers. update_mapping = dict((c.archive_update_in_progress_url, c) for c in mirrors if c.is_available) logger.info("Checking %s for Archive-Update-in-Progress marker ..", pluralize(len(update_mapping), "mirror")) with AutomaticSpinner(label="Checking mirrors"): for url, data, elapsed_time in fetch_concurrent(update_mapping.keys(), concurrency=self.concurrency): update_mapping[url].is_updating = data is not None # Sanity check our results. mirrors = list(mapping.values()) logger.info("Finished checking %s (took %s).", num_mirrors, timer) if not any(c.is_available for c in mirrors): raise Exception("It looks like all %s are unavailable!" % num_mirrors) if all(c.is_updating for c in mirrors): logger.warning("It looks like all %s are being updated?!", num_mirrors) # blacklist BASE_URL mirror if matches blacklist pattern if any(fnmatch.fnmatch(mapping[self.base_url].mirror_url, pattern) for pattern in self.blacklist): logger.warning("Ignoring blacklisted BASE_URL mirror %s.", mapping[self.base_url].mirror_url) mirrors.remove(mapping[self.base_url]) return sorted(mirrors, key=lambda c: c.sort_key, reverse=True) @cached_property def release(self): """A :class:`.Release` object corresponding to :attr:`distributor_id` and :attr:`distribution_codename`.""" return coerce_release(self.distribution_codename) @cached_property def release_is_eol(self): """ :data:`True` if the release is EOL (end of life), :data:`False` otherwise. There are three ways in which the value of this property can be computed: - When available, the first of the following EOL dates will be compared against the current date to determine whether the release is EOL: - If the :attr:`backend` module contains a ``get_eol_date()`` function (only the :mod:`~apt_smart.backends.debian` module does at the time of writing) then it is called and if it returns a number, this number is the EOL date for the release. This function was added to enable apt-smart backend modules to override the default EOL dates, more specifically to respect the `Debian LTS`_ release schedule (see also `issue #5`_). - Otherwise the :attr:`~apt_smart.releases.Release.eol_date` of :attr:`release` is used. - As a fall back :func:`validate_mirror()` is used to check whether :attr:`security_url` results in :data:`MirrorStatus.MAYBE_EOL`. .. _Debian LTS: https://wiki.debian.org/LTS .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 """ release_is_eol = None logger.debug("Checking whether %s is EOL ..", self.release) # Check if the backend provides custom EOL dates. if hasattr(self.backend, 'get_eol_date'): eol_date = self.backend.get_eol_date(self) if eol_date: release_is_eol = (time.time() >= eol_date) source = "custom EOL dates" # Check if the bundled data contains an applicable EOL date. if release_is_eol is None and self.release.eol_date: release_is_eol = self.release.is_eol source = "known EOL dates" # Validate the security mirror as a fall back. if release_is_eol is None: release_is_eol = (self.validate_mirror(self.security_url) == MirrorStatus.MAYBE_EOL) source = "security mirror" if release_is_eol and self.distributor_id == 'linuxmint': logger.info( "%s seems EOL (based on %s), but for Linux Mint no OLD_RELEASES_URL, so act as not EOL.", self.release, source, ) release_is_eol = False return release_is_eol if release_is_eol: # Still need to check due to # https://github.com/xolox/python-apt-mirror-updater/issues/9 logger.info("%s seems EOL, checking %s MirrorStatus to confirm.", self.release, self.old_releases_url) release_is_eol = (self.validate_mirror(self.old_releases_url) == MirrorStatus.AVAILABLE) if not release_is_eol: source = "%s is not available" % self.old_releases_url logger.info( "%s is %s (based on %s).", self.release, "EOL" if release_is_eol else "supported", source, ) return release_is_eol @mutable_property def security_url(self): """The URL of the mirror that serves security updates for this :attr:`backend` (a string).""" return self.backend.SECURITY_URL @cached_property def stable_mirror(self): """ A mirror URL that is stable for the given execution context (a string). The value of this property defaults to the value of :attr:`current_mirror`, however if the current mirror can't be determined or is deemed inappropriate by :func:`validate_mirror()` then :attr:`best_mirror` will be used instead. This provides a stable mirror selection algorithm which is useful because switching mirrors causes ``apt-get update`` to unconditionally download all package lists and this takes a lot of time so should it be avoided when unnecessary. """ if self.release_is_eol: logger.debug("%s is EOL, falling back to %s.", self.release, self.old_releases_url) return self.old_releases_url else: try: logger.debug("Trying to select current mirror as stable mirror ..") return self.current_mirror except Exception: logger.debug("Failed to determine current mirror, selecting best mirror instead ..") return self.best_mirror @cached_property def validated_mirrors(self): """Dictionary of validated mirrors (used by :func:`validate_mirror()`).""" return {} @mutable_property def custom_mirror_file_path(self): """The local custom mirror file's absolute path, can be set by `-F` flag""" return None @cached_property def read_custom_mirror_file(self): """ Read a file containing custom mirror URLs (one URL per line) to add custom mirrors to rank. :param file_to_read: The local file's absolute path :returns: A set of mirrors read from file """ if self.custom_mirror_file_path is None: return {} else: logger.info("The file path you input is %s", self.custom_mirror_file_path) mirrors = set() with open(self.custom_mirror_file_path) as f: for line in f: if line.strip().startswith(('http://', 'https://', 'ftp://')): mirrors.add(CandidateMirror(mirror_url=line.strip(), updater=self)) return mirrors def change_mirror(self, new_mirror=None, update=True): """ Change the main mirror in use in :attr:`main_sources_list`. :param new_mirror: The URL of the new mirror (a string, defaults to :attr:`best_mirror`). :param update: Whether an ``apt-get update`` should be run after changing the mirror (a boolean, defaults to :data:`True`). """ timer = Timer() # Default to the best available mirror. if new_mirror: logger.info("Changing mirror of %s to %s ..", self.context, new_mirror) else: logger.info("Changing mirror of %s to best available mirror ..", self.context) new_mirror = self.best_mirror logger.info("Selected mirror: %s", new_mirror) # Parse /etc/apt/sources.list to replace the old mirror with the new one. sources_list = self.get_sources_list() mirrors_to_replace = [normalize_mirror_url(self.current_mirror)] if self.release_is_eol: # When a release goes EOL the security updates mirrors stop # serving that release as well, so we need to remove them. logger.debug("Replacing %s URLs as well ..", self.security_url) mirrors_to_replace.append(normalize_mirror_url(self.security_url)) else: logger.debug("Not replacing %s URLs.", self.security_url) lines = sources_list.splitlines() sources_list_options = self.get_sources_list_options for i, line in enumerate(lines): # The first token should be `deb' or `deb-src', the second token is # the mirror's URL, the third token is the `distribution' and any # further tokens are `components'. tokens = line.split() if (len(tokens) >= 4 and tokens[0] in ('deb', 'deb-src') and normalize_mirror_url(tokens[1]) in mirrors_to_replace): tokens[1] = new_mirror if i in sources_list_options: tokens.insert(1, '[' + sources_list_options[i] + ']') # Get the [options] back lines[i] = u' '.join(tokens) # Install the modified package resource list. self.install_sources_list(u'\n'.join(lines)) # Clear (relevant) cached properties. del self.current_mirror # Make sure previous package lists are removed. self.clear_package_lists() # Make sure the package lists are up to date. if update: self.smart_update(switch_mirrors=False) logger.info("Finished changing mirror of %s in %s.", self.context, timer) def clear_package_lists(self): """Clear the package list cache by removing the files under ``/var/lib/apt/lists``.""" timer = Timer() logger.info("Clearing package list cache of %s ..", self.context) self.context.execute( # We use an ugly but necessary find | xargs pipeline here because # find's -delete option implies -depth which negates -prune. Sigh. 'find /var/lib/apt/lists -type f -name lock -prune -o -type f -print0 | xargs -0 rm -f', sudo=True, ) logger.info("Successfully cleared package list cache of %s in %s.", self.context, timer) def create_chroot(self, directory, codename=None, arch=None): """ Bootstrap a basic Debian or Ubuntu system using debootstrap_. :param directory: The pathname of the target directory (a string). :param codename: The codename of the target (a string). :param arch: The target architecture (a string or :data:`None`). :returns: A :class:`~executor.contexts.ChangeRootContext` object. If `directory` already exists and isn't empty then it is assumed that the chroot has already been created and debootstrap_ won't be run. Before this method returns it changes :attr:`context` to the chroot. .. _debootstrap: https://manpages.debian.org/debootstrap """ logger.debug("Checking if chroot already exists (%s) ..", directory) if self.context.exists(directory) and self.context.list_entries(directory): logger.info("The chroot already exists, skipping initialization.") first_run = False else: # Ensure the `debootstrap' program is installed. if not self.context.find_program('debootstrap'): logger.info("Installing `debootstrap' program ..") self.context.execute('apt-get', 'install', '--yes', 'debootstrap', sudo=True) # Use the `debootstrap' program to create the chroot. timer = Timer() debootstrap_command = ['debootstrap'] if arch: debootstrap_command.append('--arch=%s' % arch) release_chroot = None keyring_chroot = '' codename_chroot = '' best_mirror_chroot = None generate_sources_list_chroot = None if codename and codename != self.distribution_codename: updater_chroot = AptMirrorUpdater() updater_chroot.distribution_codename = codename if updater_chroot.distributor_id == 'linuxmint': msg = "It seems no sense to create chroot of Linux Mint, " \ "please specify a codename of Ubuntu or Debian " \ "to create chroot." raise ValueError(msg) if not self.context.exists(updater_chroot.release.keyring_file): if updater_chroot.distributor_id == 'ubuntu': self.context.execute('apt-get', 'install', '--yes', 'ubuntu-keyring', sudo=True) elif updater_chroot.distributor_id == 'debian': self.context.execute('apt-get', 'install', '--yes', 'debian-archive-keyring', sudo=True) release_chroot = updater_chroot.release keyring_chroot = updater_chroot.release.keyring_file codename_chroot = codename best_mirror_chroot = updater_chroot.best_mirror else: if self.distributor_id == 'linuxmint': msg = "It seems no sense to create chroot of Linux Mint, " \ "please use -C to specify a codename of Ubuntu or Debian " \ "to create chroot." raise ValueError(msg) release_chroot = self.release keyring_chroot = self.release.keyring_file codename_chroot = self.distribution_codename best_mirror_chroot = self.best_mirror logger.info("Creating %s chroot in %s ..", release_chroot, directory) debootstrap_command.append('--keyring=%s' % keyring_chroot) debootstrap_command.append(codename_chroot) debootstrap_command.append(directory) debootstrap_command.append(best_mirror_chroot) self.context.execute(*debootstrap_command, sudo=True) logger.info("Took %s to create %s chroot.", timer, release_chroot) first_run = True # Switch the execution context to the chroot and reset the locale (to # avoid locale warnings emitted by post-installation scripts run by # `apt-get install'). self.context = ChangeRootContext( chroot=directory, environment=dict(LC_ALL='C'), ) # Clear the values of cached properties that can be # invalidated by switching the execution context. del self.current_mirror del self.stable_mirror if codename and codename != self.distribution_codename: updater_chroot.context = self.context del updater_chroot.current_mirror del updater_chroot.stable_mirror generate_sources_list_chroot = updater_chroot.generate_sources_list() else: generate_sources_list_chroot = self.generate_sources_list() # The following initialization only needs to happen on the first # run, but it requires the context to be set to the chroot. if first_run: # Make sure the `lsb_release' program is available. It is # my experience that this package cannot be installed using # `debootstrap --include=lsb-release', it specifically # needs to be installed afterwards. self.context.execute('apt-get', 'install', '--yes', 'lsb-release', sudo=True) # Cleanup downloaded *.deb archives. self.context.execute('apt-get', 'clean', sudo=True) # Install a suitable /etc/apt/sources.list file. The logic behind # generate_sources_list() depends on the `lsb_release' program. logger.debug("sources.list for chroot generated:") logger.debug(generate_sources_list_chroot) self.install_sources_list(generate_sources_list_chroot) # Make sure the package lists are up to date. self.smart_update() return self.context def dumb_update(self, *args): """ Update the system's package lists (by running ``apt-get update``). :param args: Command line arguments to ``apt-get update`` (zero or more strings). The :func:`dumb_update()` method doesn't do any error handling or retrying, if that's what you're looking for then you need :func:`smart_update()` instead. """ timer = Timer() logger.info("Updating package lists of %s ..", self.context) self.context.execute('apt-get', 'update', *args, sudo=True) logger.info("Finished updating package lists of %s in %s.", self.context, timer) def generate_sources_list(self, **options): """ Generate the contents of ``/etc/apt/sources.list``. If no `mirror_url` keyword argument is given then :attr:`stable_mirror` is used as a default. Please refer to the documentation of the Debian (:func:`apt_smart.backends.debian.generate_sources_list()`) and Ubuntu (:func:`apt_smart.backends.ubuntu.generate_sources_list()`) backend implementations of this method for details on argument handling and the return value. """ if options.get('mirror_url') is None: options['mirror_url'] = self.stable_mirror options.setdefault('codename', self.distribution_codename) return self.backend.generate_sources_list(**options) @mutable_property def get_sources_list_options(self): """ Get the contents of [options] in :attr:`main_sources_list`. [options] can be set into sources.list, e.g. deb [arch=amd64] http://mymirror/ubuntu bionic main restricted see details at https://manpages.debian.org/jessie/apt/sources.list.5.en.html The [options] is often not considered and breaks parsing in many projects, see https://github.com/jblakeman/apt-select/issues/54 We begin to deal with the [options] by stripping it from sources.list, and then get it back when generating new sources.list """ def get_sources_list(self): """ Get the contents of :attr:`main_sources_list`. :returns: A Unicode string. This code currently assumes that the ``sources.list`` file is encoded using :data:`SOURCES_LIST_ENCODING`. I'm not actually sure if this is correct because I haven't been able to find a formal specification! Feedback is welcome :-). This code strips [options] from sources.list, stores it in :attr:`get_sources_list_options` """ contents = self.context.read_file(self.main_sources_list) contents = contents.decode(SOURCES_LIST_ENCODING) sources_list_options = {} contents_raw = [] # stripped contents without options for i, line in enumerate(contents.splitlines()): if line.find('[') > 0: # found '[' and not starts with '[' startswith_deb = line.split('[')[0] temp = line.split('[')[1] sources_list_options[i] = temp.split(']')[0] startswith_http = temp.split(']')[1] contents_raw.append(startswith_deb + startswith_http) elif line.find('[') == -1: # not found contents_raw.append(line) self.get_sources_list_options = sources_list_options return '\n'.join(contents_raw) def ignore_mirror(self, pattern): """ Add a pattern to the mirror discovery :attr:`blacklist`. :param pattern: A shell pattern (containing wild cards like ``?`` and ``*``) that is matched against the full URL of each mirror. When a pattern is added to the blacklist any previously cached values of :attr:`available_mirrors`, :attr:`best_mirror`, :attr:`ranked_mirrors` and :attr:`stable_mirror` are cleared. This makes sure that mirrors blacklisted after mirror discovery has already run are ignored. """ # Update the blacklist. logger.info("Adding pattern to mirror discovery blacklist: %s", pattern) self.blacklist.add(pattern) # Clear (relevant) cached properties. del self.available_mirrors del self.best_mirror del self.ranked_mirrors del self.stable_mirror def install_sources_list(self, contents): """ Install a new ``/etc/apt/sources.list`` file. :param contents: The new contents of the sources list (a Unicode string). You can generate a suitable value using the :func:`generate_sources_list()` method. """ if isinstance(contents, text_type): contents = contents.encode(SOURCES_LIST_ENCODING) logger.info("Installing new %s ..", self.main_sources_list) with self.context: # Write the sources.list contents to a temporary file. We make sure # the file always ends in a newline to adhere to UNIX conventions. temporary_file = '/tmp/apt-smart-sources-list-%i.txt' % os.getpid() contents_to_write = contents.rstrip() + b'\n' self.context.write_file(temporary_file, contents_to_write) # Make sure the temporary file is cleaned up when we're done with it. self.context.cleanup('rm', '--force', temporary_file) # Make a backup copy of /etc/apt/sources.list in case shit hits the fan? if self.context.exists(self.main_sources_list): dirname, basename = os.path.split(self.main_sources_list) if basename == 'official-package-repositories.list': backup_dir = os.path.join(dirname, 'backup_by_apt-smart') # Backup to dir for Linux Mint if not self.context.exists(backup_dir): self.context.execute('mkdir', backup_dir, sudo=True) backup_copy = '%s.backup.%i' % (os.path.join(backup_dir, basename), time.time()) else: backup_copy = '%s.backup.%i' % (self.main_sources_list, time.time()) logger.info("Backing up contents of %s to %s ..", self.main_sources_list, backup_copy) self.context.execute('cp', self.main_sources_list, backup_copy, sudo=True) # Move the temporary file into place without changing ownership and permissions. self.context.execute( 'cp', '--no-preserve=mode,ownership', temporary_file, self.main_sources_list, sudo=True, ) def smart_update(self, *args, **kw): """ Update the system's package lists (switching mirrors if necessary). :param args: Command line arguments to ``apt-get update`` (zero or more strings). :param max_attempts: The maximum number of attempts at successfully updating the system's package lists (an integer, defaults to 10). :param switch_mirrors: :data:`True` if we're allowed to switch mirrors on 'hash sum mismatch' errors, :data:`False` otherwise. :raises: If updating of the package lists fails 10 consecutive times (`max_attempts`) an exception is raised. While :func:`dumb_update()` simply runs ``apt-get update`` the :func:`smart_update()` function works quite differently: - First the system's package lists are updated using :func:`dumb_update()`. If this is successful we're done. - If the update fails we check the command's output for the phrase 'hash sum mismatch'. If we find this phrase we assume that the current mirror is faulty and switch to another one. - Failing ``apt-get update`` runs are retried up to `max_attempts`. """ backoff_time = 10 max_attempts = kw.get('max_attempts', 10) switch_mirrors = kw.get('switch_mirrors', True) for i in range(1, max_attempts + 1): with CaptureOutput() as session: try: self.dumb_update(*args) return except Exception: if i < max_attempts: output = session.get_text() # Check for EOL releases. This somewhat peculiar way of # checking is meant to ignore 404 responses from # `secondary package mirrors' like PPAs. If the output # of `apt-get update' implies that the release is EOL # we need to verify our assumption. if any(self.current_mirror in line and u'404' in line.split() for line in output.splitlines()): logger.warning("%s may be EOL, checking ..", self.release) if self.release_is_eol: if switch_mirrors: logger.warning("Switching to old releases mirror because %s is EOL ..", self.release) self.change_mirror(self.old_releases_url, update=False) continue else: raise Exception(compact(""" Failed to update package lists because it looks like the current release (%s) is end of life but I'm not allowed to switch mirrors! (there's no point in retrying so I'm not going to) """, self.distribution_codename)) # Check for `hash sum mismatch' errors. if switch_mirrors and u'hash sum mismatch' in output.lower(): logger.warning("Detected 'hash sum mismatch' failure, switching to other mirror ..") self.ignore_mirror(self.current_mirror) self.change_mirror(update=False) else: logger.warning("Retrying after `apt-get update' failed (%i/%i) ..", i, max_attempts) # Deal with unidentified (but hopefully transient) failures by retrying but backing off # to give the environment (network connection, mirror state, etc.) time to stabilize. logger.info("Sleeping for %s before retrying update ..", format_timespan(backoff_time)) time.sleep(backoff_time) if backoff_time <= 120: backoff_time *= 2 else: backoff_time += backoff_time / 3 raise Exception("Failed to update package lists %i consecutive times?!" % max_attempts) def validate_mirror(self, mirror_url): """ Make sure a mirror serves :attr:`distribution_codename`. :param mirror_url: The base URL of the mirror (a string). :returns: One of the values in the :class:`MirrorStatus` enumeration. """ mirror_url = normalize_mirror_url(mirror_url) key = (mirror_url, self.distribution_codename) value = self.validated_mirrors.get(key) if value is None: logger.info("Checking if %s is available on %s ..", self.release, mirror_url) # Try to download the Release.gpg file, in the assumption that # this file should always exist and is more or less guaranteed # to be relatively small. try: mirror = CandidateMirror(mirror_url=mirror_url, updater=self) mirror.release_gpg_contents = fetch_url(mirror.release_gpg_url, retry=False) value = (MirrorStatus.AVAILABLE if mirror.is_available else MirrorStatus.UNAVAILABLE) except NotFoundError: # When the mirror is serving 404 responses it can be an # indication that the release has gone end of life. In any # case the mirror is unavailable. value = MirrorStatus.MAYBE_EOL except Exception: # When we get an unspecified error that is not a 404 # response we conclude that the mirror is unavailable. value = MirrorStatus.UNAVAILABLE # Cache the mirror status that we just determined. self.validated_mirrors[key] = value return value class CandidateMirror(PropertyManager): """A candidate mirror groups a mirror URL with its availability and performance metrics.""" @mutable_property def bandwidth(self): """ The bytes per second achieved while fetching :attr:`release_gpg_url` (a number or :data:`None`). The value of this property is computed based on the values of :attr:`release_gpg_contents` and :attr:`release_gpg_latency`. """ if self.release_gpg_contents and self.release_gpg_latency: return len(self.release_gpg_contents) / self.release_gpg_latency @lazy_property def archive_update_in_progress_url(self): """ The URL of the file whose existence indicates that the mirror is being updated (a string). The value of this property is computed based on the value of :attr:`mirror_url`. """ return '%s/Archive-Update-in-Progress-%s' % ( self.mirror_url, urlparse(self.mirror_url).netloc, ) @key_property def mirror_url(self): """The base URL of the mirror (a string).""" @mirror_url.setter def mirror_url(self, value): """Normalize the mirror URL when set.""" set_property(self, 'mirror_url', normalize_mirror_url(value)) @mutable_property def is_available(self): """ :data:`True` if :attr:`release_gpg_contents` contains the expected data, :data:`False` otherwise. The value of this property is computed by checking whether :attr:`release_gpg_contents` contains the expected data. This may seem like a rather obscure way of validating a mirror, but it was specifically chosen to detect all sorts of ways in which mirrors can be broken: - Webservers with a broken configuration that return an error page for all URLs. - Mirrors whose domain name registration has expired, where the domain is now being squatted and returns HTTP 200 OK responses for all URLs (whether they "exist" or not). """ value = False if self.release_gpg_contents: value = b'Date:' in self.release_gpg_contents if not value: logger.debug("Missing data, considering mirror unavailable (%s).", self.release_gpg_url) else: # Get all data following "Date: " date_string_raw = self.release_gpg_contents.decode('utf-8').split("Date: ", 1) if len(date_string_raw) == 2: # split succussfully using "Date: " # Get only date string like "Sun, 25 Aug 2019 23:35:36 UTC", drop other data date_string = date_string_raw[1].split("\n")[0] if date_string.endswith("UTC"): # Convert it into UNIX timestamp last_updated_time = calendar.timegm(time.strptime(date_string, "%a, %d %b %Y %H:%M:%S %Z")) if self.updater.base_last_updated == 0: # First time launch this method, must be base_url self.updater.base_last_updated = last_updated_time logger.debug("base_last_updated: %i", self.updater.base_last_updated) else: # if last_updated is 0 means this mirror is up-to-date self.last_updated = self.updater.base_last_updated - last_updated_time logger.debug("last_updated: %i", self.last_updated) else: logger.debug("Not UTC? Correct me. " + date_string) logger.debug("Looks good, %s is_available return True", self.release_gpg_url) else: # split fails because lacking "Date: " logger.debug("Missing Date, considering mirror unavailable (%s).", self.release_gpg_url) value = False set_property(self, 'is_available', value) return value @mutable_property def is_updating(self): """:data:`True` if the mirror is being updated, :data:`False` otherwise.""" @mutable_property def last_updated(self): """The time in seconds since the most recent mirror update (a number or :data:`None`).""" @mutable_property def release_gpg_contents(self): """ The contents downloaded from :attr:`release_gpg_url` (a string or :data:`None`). By downloading the file available at :attr:`release_gpg_url` and setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency` you enable the :attr:`bandwidth` and :attr:`is_available` properties to be computed. """ @mutable_property def release_gpg_latency(self): """ The time it took to download :attr:`release_gpg_url` (a number or :data:`None`). By downloading the file available at :attr:`release_gpg_url` and setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency` you enable the :attr:`bandwidth` and :attr:`is_available` properties to be computed. """ @mutable_property def release_gpg_url(self): """ The URL of the ``Release`` file that will be used to test the mirror (a string or :data:`None`). The value of this property is based on :attr:`mirror_url` and the :attr:`~AptMirrorUpdater.distribution_codename` property of the :attr:`updater` object. """ if self.updater and self.updater.distribution_codename: if self.updater.distributor_id == 'ubuntu': return '%s/dists/%s-security/Release' % ( self.mirror_url, self.updater.distribution_codename, ) elif self.updater.distributor_id == 'debian': return '%s/dists/%s-updates/Release' % ( self.mirror_url, self.updater.distribution_codename, ) elif self.updater.distributor_id == 'linuxmint': return '%s/dists/%s/Release' % ( self.mirror_url, self.updater.distribution_codename, ) @mutable_property def sort_key(self): """ A tuple that can be used to sort the mirror by its availability/performance metrics. The tuple created by this property contains four numbers in the following order: 1. The number 1 when :attr:`is_available` is :data:`True` or the number 0 when :attr:`is_available` is :data:`False` (because most importantly a mirror must be available). 2. The number 0 when :attr:`is_updating` is :data:`True` or the number 1 when :attr:`is_updating` is :data:`False` (because being updated at this very moment is *bad*). 3. The negated value of :attr:`last_updated` (because the lower :attr:`last_updated` is, the better). If :attr:`last_updated` is :data:`None` then :data:`LAST_UPDATED_DEFAULT` is used instead. 4. The value of :attr:`bandwidth` (because the higher :attr:`bandwidth` is, the better). By sorting :class:`CandidateMirror` objects on these tuples in ascending order, the last mirror in the sorted results will be the "most suitable mirror" (given the available information). """ return (int(self.is_available), int(not self.is_updating), -(self.last_updated if self.last_updated is not None else LAST_UPDATED_DEFAULT), self.bandwidth or 0) @mutable_property(repr=False) def updater(self): """A reference to the :class:`AptMirrorUpdater` object that created the candidate.""" class MirrorStatus(Enum): """Enumeration for mirror statuses determined by :func:`AptMirrorUpdater.validate_mirror()`.""" AVAILABLE = 1 """The mirror is accepting connections and serving the expected content.""" MAYBE_EOL = 2 """The mirror is serving HTTP 404 "Not Found" responses instead of the expected content.""" UNAVAILABLE = 3 """The mirror is not accepting connections or not serving the expected content.""" def find_current_mirror(sources_list): """ Find the URL of the main mirror that is currently in use by ``apt-get``. :param sources_list: The contents of apt's package resource list, e.g. the contents of :attr:`main_sources_list` (a string). :returns: The URL of the main mirror in use (a string). :raises: If the main mirror can't be determined :exc:`~exceptions.EnvironmentError` is raised. The main mirror is determined by looking for the first ``deb`` or ``deb-src`` directive in apt's package resource list whose URL uses the HTTP or FTP scheme and whose components contain ``main``. """ for line in sources_list.splitlines(): # The first token should be `deb' or `deb-src', the second token is # the mirror's URL, the third token is the `distribution' and any # further tokens are `components'. tokens = line.split() if (len(tokens) >= 4 and tokens[0] in ('deb', 'deb-src') and tokens[1].startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/')) and 'main' in tokens[3:]): return tokens[1] raise EnvironmentError("Failed to determine current mirror in apt's package resource list!") def mirrors_are_equal(a, b): """ Check whether two mirror URLS are equal. :param a: The first mirror URL (a string). :param b: The second mirror URL (a string). :returns: :data:`True` if the mirror URLs are equal, :data:`False` otherwise. """ return normalize_mirror_url(a) == normalize_mirror_url(b) def normalize_mirror_url(url): """ Normalize a mirror URL so it can be compared using string equality comparison. :param url: The mirror URL to normalize (a string). :returns: The normalized mirror URL (a string). """ return url.rstrip('/') ================================================ FILE: apt_smart/backends/__init__.py ================================================ """Namespace for platform specific mirror discovery in :mod:`apt_smart`.""" ================================================ FILE: apt_smart/backends/debian.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """ Discovery of Debian package archive mirrors. Here are references to some of the material that I've needed to consult while working on this module: - `Notes about sources.list on the Debian wiki `_ - `The Debian backports webpages `_ - `Documentation about the "proposed-updates" mechanism `_ """ # Standard library modules. import logging import json # External dependencies. import six from bs4 import BeautifulSoup from humanfriendly import Timer, format, pluralize # Modules included in our package. from apt_smart import CandidateMirror, mirrors_are_equal from apt_smart.http import fetch_url LTS_ARCHITECTURES = ('i386', 'amd64', 'armel', 'armhf') """The names of the architectures supported by the Debian LTS team (a tuple of strings).""" LTS_RELEASES = { 'jessie': 1593468000, # 2020-06-30 'stretch': 1656540000, # 2022-06-30 } """ A dictionary with `Debian LTS`_ releases and their EOL dates. This is needed because distro-info-data_ doesn't contain information about Debian LTS releases but nevertheless ``archive.debian.org`` doesn't adopt a release until the LTS status expires (this was originally reported in `issue #5`_). .. _Debian LTS: https://wiki.debian.org/LTS .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 """ MIRRORS_URL = 'https://www.debian.org/mirror/list' """The URL of the HTML page listing all primary Debian mirrors (a string).""" SECURITY_URL = 'http://security.debian.org/' """The base URL of the Debian mirror with security updates (a string).""" OLD_RELEASES_URL = 'http://archive.debian.org/debian-archive/debian/' """The URL where EOL (end of life) Debian releases are hosted (a string).""" BASE_URL = 'http://ftp.debian.org/debian/dists/codename-updates/Release' """The URL where official repo treated as base are hosted (a string). The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated` to determine which mirrors are up-to-date""" DEFAULT_SUITES = 'release', 'security', 'updates' """A tuple of strings with the Debian suites that are enabled by default.""" VALID_COMPONENTS = 'main', 'contrib', 'non-free' """A tuple of strings with the names of the components available in the Debian package repositories.""" VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed-updates' """A tuple of strings with the names of the suites available in the Debian package repositories.""" # Initialize a logger for this module. logger = logging.getLogger(__name__) def discover_mirrors(): """ Discover available Debian mirrors by querying :data:`MIRRORS_URL`. :returns: A set of :class:`.CandidateMirror` objects that have their :attr:`~.CandidateMirror.mirror_url` property set. :raises: If no mirrors are discovered an exception is raised. An example run: >>> from apt_smart.backends.debian import discover_mirrors >>> from pprint import pprint >>> pprint(discover_mirrors()) set([CandidateMirror(mirror_url='http://ftp.at.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.au.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.be.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.bg.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.br.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.by.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.ca.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.ch.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.cn.debian.org/debian/'), CandidateMirror(mirror_url='http://ftp.cz.debian.org/debian/'), ...]) """ timer = Timer() logger.info("Discovering Debian mirrors at %s ..", MIRRORS_URL) # Find which country the user is in to get mirrors in that country try: url = 'https://ipapi.co/json' response = fetch_url(url, timeout=2) # On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5, # so decode it to str if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country_name'] logger.info("Found your location: %s by %s", country, url) except Exception: url = 'http://ip-api.com/json' response = fetch_url(url, timeout=5) if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country'] logger.info("Found your location: %s by %s", country, url) data = fetch_url(MIRRORS_URL, timeout=20, retry=True) soup = BeautifulSoup(data, 'html.parser') tables = soup.findAll('table') flag = False # flag is True when find the row's text is that country mirrors = set() if not tables: raise Exception("Failed to locate element in Debian mirror page! (%s)" % MIRRORS_URL) else: for row in tables[1].findAll("tr"): # tables[1] organises mirrors by country. if flag: if not row.a: # End of mirrors located in that country break else: mirrors.add(CandidateMirror(mirror_url=row.a['href'])) if row.get_text() == country: flag = True if len(mirrors) < 3: # Too few, add tables[0] which contains Primary Debian mirror sites all around the world. mirrors.add(CandidateMirror(mirror_url=a['href']) for a in tables[0].findAll('a', href=True)) if not mirrors: raise Exception("Failed to discover any Debian mirrors! (using %s)" % MIRRORS_URL) logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Debian mirror"), timer) return mirrors def generate_sources_list(mirror_url, codename, suites=DEFAULT_SUITES, components=VALID_COMPONENTS, enable_sources=False): """ Generate the contents of ``/etc/apt/sources.list`` for a Debian system. :param mirror_url: The base URL of the mirror (a string). :param codename: The codename of a Debian release (a string like 'wheezy' or 'jessie') or a Debian release class (a string like 'stable', 'testing', etc). :param suites: An iterable of strings (defaults to :data:`DEFAULT_SUITES`, refer to :data:`VALID_SUITES` for details). :param components: An iterable of strings (refer to :data:`VALID_COMPONENTS` for details). :param enable_sources: :data:`True` to include ``deb-src`` entries, :data:`False` to omit them. :returns: The suggested contents of ``/etc/apt/sources.list`` (a string). """ # Validate the suites. invalid_suites = [s for s in suites if s not in VALID_SUITES] if invalid_suites: msg = "Invalid Debian suite(s) given! (%s)" raise ValueError(msg % invalid_suites) # Validate the components. invalid_components = [c for c in components if c not in VALID_COMPONENTS] if invalid_components: msg = "Invalid Debian component(s) given! (%s)" raise ValueError(msg % invalid_components) # Generate the /etc/apt/sources.list file contents. lines = [] directives = ('deb', 'deb-src') if enable_sources else ('deb',) for suite in suites: for directive in directives: lines.append(format( '{directive} {mirror} {suite} {components}', directive=directive, mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL) else (SECURITY_URL if suite == 'security' else mirror_url)), suite=(codename if suite == 'release' else ( ('%s/updates' % codename if suite == 'security' else codename + '-' + suite))), components=' '.join(components), )) return '\n'.join(lines) def get_eol_date(updater): """ Override the EOL date for `Debian LTS`_ releases. :param updater: The :class:`~apt_smart.AptMirrorUpdater` object. :returns: The overridden EOL date (a number) or :data:`None`. """ if updater.architecture in LTS_ARCHITECTURES: return LTS_RELEASES.get(updater.distribution_codename) ================================================ FILE: apt_smart/backends/linuxmint.py ================================================ # Automated, robust apt-get mirror selection for Debian ,Ubuntu and Linux Mint. # # Author: martin68 and Peter Odding # Last Change: October 29, 2019 # URL: https://apt-smart.readthedocs.io """Discovery of Linux Mint package archive mirrors.""" # Standard library modules. import json import logging # External dependencies. import six from bs4 import BeautifulSoup from humanfriendly import Timer, pluralize # Modules included in our package. from apt_smart import CandidateMirror from apt_smart.http import fetch_url MIRRORS_URL = 'https://linuxmint.com/mirrors.php' """The URL of the HTML page listing official Linux Mint mirrors (a string).""" SECURITY_URL = 'http://security.ubuntu.com/ubuntu' """The URL where Ubuntu ( Linux Mint's codebase )security updates are hosted (a string).""" BASE_URL = 'http://packages.linuxmint.com/dists/codename/Release' """The URL where official repo treated as base are hosted (a string). The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated` to determine which mirrors are up-to-date""" DEFAULT_SUITES = 'release', 'updates', 'backports', 'security' """A tuple of strings with the Linux Mint suites that are enabled by default.""" VALID_COMPONENTS = 'main', 'restricted', 'universe', 'multiverse' """A tuple of strings with the names of the components available in the Linux Mint package repositories.""" VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed' """ A tuple of strings with the names of the suites available in the Linux Mint package repositories. The actual name of the 'release' suite is the codename of the relevant Linux Mint release, while the names of the other suites are formed by concatenating the codename with the suite name (separated by a dash). As an example to make things more concrete, Ubuntu 16.04 has the following five suites available: ``xenial`` (this is the release suite), ``xenial-security``, ``xenial-updates``, ``xenial-backports`` and ``xenial-proposed``. """ # Initialize a logger for this module. logger = logging.getLogger(__name__) def discover_mirrors(): """ Discover available Linux Mint mirrors. :returns: A set of :class:`.CandidateMirror` objects that have their :attr:`~.CandidateMirror.mirror_url` property set and may have the :attr:`~.CandidateMirror.last_updated` property set. :raises: If no mirrors are discovered an exception is raised. This queries :data:`MIRRORS_URL`to discover available Linux Mint mirrors. Here's an example run: >>> from apt_smart.backends.linuxmint import discover_mirrors >>> from pprint import pprint >>> pprint(discover_mirrors()) set([CandidateMirror(mirror_url='http://mirrors.cqu.edu.cn/linuxmint/'), CandidateMirror(mirror_url='http://mirrors.hust.edu.cn/linuxmint/'), CandidateMirror(mirror_url='http://mirrors.shu.edu.cn/linuxmint/'), CandidateMirror(mirror_url='https://mirrors.tuna.tsinghua.edu.cn/linuxmint/'), CandidateMirror(mirror_url='http://mirrors.ustc.edu.cn/linuxmint/'), CandidateMirror(mirror_url='http://mirrors.zju.edu.cn/linuxmint/'), ...]) """ timer = Timer() mirrors = set() logger.info("Discovering Linux Mint mirrors at %s ..", MIRRORS_URL) # Find which country the user is in to get mirrors in that country try: url = 'https://ipapi.co/json' response = fetch_url(url, timeout=2) # On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5, # so decode it to str if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country_name'] logger.info("Found your location: %s by %s", country, url) except Exception: url = 'http://ip-api.com/json' response = fetch_url(url, timeout=5) if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country'] logger.info("Found your location: %s by %s", country, url) if country == 'United States': country = 'USA' try: data = fetch_url(MIRRORS_URL, timeout=15) except Exception: logger.warning("Time out, try again") data = fetch_url(MIRRORS_URL, timeout=70, retry=True) soup = BeautifulSoup(data, 'html.parser') tables = soup.findAll('table') if not tables: raise Exception("Failed to locate
element in Linux Mint mirror page! (%s)" % MIRRORS_URL) else: while len(mirrors) < 3: for row in tables[2].findAll("tr"): if country in row.get_text(): for data in row.findAll("td"): data_text = data.get_text() # Check if the link looks like a mirror URL. if data_text.startswith(('http://', 'https://')): mirrors.add(CandidateMirror(mirror_url=data_text)) if country == 'Worldwide': break # break while loop when already set to 'Worldwide' if len(mirrors) < 3: logging.info("Too few mirrors found in your country, get more Worldwide mirrors.") country = 'Worldwide' if not mirrors: raise Exception("Failed to discover any Linux Mint mirrors! (using %s)" % MIRRORS_URL) logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Linux Mint mirror"), timer) return mirrors ================================================ FILE: apt_smart/backends/ubuntu.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """Discovery of Ubuntu package archive mirrors.""" # Standard library modules. import json import logging # External dependencies. import six from bs4 import BeautifulSoup, UnicodeDammit from humanfriendly import Timer, format, pluralize # Modules included in our package. from apt_smart import CandidateMirror, mirrors_are_equal from apt_smart.http import fetch_url MIRRORS_URL = 'https://launchpad.net/ubuntu/+archivemirrors' """The URL of the HTML page listing official Ubuntu mirrors (a string).""" MIRROR_SELECTION_URL = 'http://mirrors.ubuntu.com/mirrors.txt' """The URL of a plain text listing of "geographically suitable" mirror URLs (a string).""" OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/' """The URL where EOL (end of life) Ubuntu releases are hosted (a string).""" SECURITY_URL = 'http://security.ubuntu.com/ubuntu' """The URL where Ubuntu security updates are hosted (a string).""" BASE_URL = 'http://archive.ubuntu.com/ubuntu/dists/codename-security/Release' """The URL where official repo treated as base are hosted (a string). The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated` to determine which mirrors are up-to-date""" DEFAULT_SUITES = 'release', 'updates', 'backports', 'security' """A tuple of strings with the Ubuntu suites that are enabled by default.""" VALID_COMPONENTS = 'main', 'restricted', 'universe', 'multiverse' """A tuple of strings with the names of the components available in the Ubuntu package repositories.""" VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed' """ A tuple of strings with the names of the suites available in the Ubuntu package repositories. The actual name of the 'release' suite is the codename of the relevant Ubuntu release, while the names of the other suites are formed by concatenating the codename with the suite name (separated by a dash). As an example to make things more concrete, Ubuntu 16.04 has the following five suites available: ``xenial`` (this is the release suite), ``xenial-security``, ``xenial-updates``, ``xenial-backports`` and ``xenial-proposed``. """ # Initialize a logger for this module. logger = logging.getLogger(__name__) def discover_mirrors_old(): """ Discover available Ubuntu mirrors. (fallback) :returns: A set of :class:`.CandidateMirror` objects that have their :attr:`~.CandidateMirror.mirror_url` property set and may have the :attr:`~.CandidateMirror.last_updated` property set. :raises: If no mirrors are discovered an exception is raised. This queries :data:`MIRRORS_URL`to discover available Ubuntu mirrors. Here's an example run: >>> from apt_smart.backends.ubuntu import discover_mirrors_old >>> from pprint import pprint >>> pprint(discover_mirrors_old()) set([CandidateMirror(mirror_url='http://archive.ubuntu.com/ubuntu/'), CandidateMirror(mirror_url='http://ftp.nluug.nl/os/Linux/distr/ubuntu/'), CandidateMirror(mirror_url='http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/'), CandidateMirror(mirror_url='http://ftp.tudelft.nl/archive.ubuntu.com/'), CandidateMirror(mirror_url='http://mirror.1000mbps.com/ubuntu/'), CandidateMirror(mirror_url='http://mirror.amsiohosting.net/archive.ubuntu.com/'), CandidateMirror(mirror_url='http://mirror.i3d.net/pub/ubuntu/'), CandidateMirror(mirror_url='http://mirror.nforce.com/pub/linux/ubuntu/'), CandidateMirror(mirror_url='http://mirror.nl.leaseweb.net/ubuntu/'), CandidateMirror(mirror_url='http://mirror.transip.net/ubuntu/ubuntu/'), ...]) It may be super-slow somewhere ( with 100Mbps fibre though ) in the world to access launchpad.net (see below), so we have to no longer rely on MIRRORS_URL . time curl -o/dev/null 'https://launchpad.net/ubuntu/+archivemirrors' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 263k 100 263k 0 0 5316 0 0:00:50 0:00:50 --:--:-- 6398 real 0m50.869s user 0m0.045s sys 0m0.039s But it can be a fallback when MIRROR_SELECTION_URL is down. """ mirrors = set() logger.info("Discovering Ubuntu mirrors at %s ..", MIRRORS_URL) # Find which country the user is in to get mirrors in that country try: url = 'https://ipapi.co/json' response = fetch_url(url, timeout=2) # On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5, # so decode it to str if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country_name'] logger.info("Found your location: %s by %s", country, url) except Exception: url = 'http://ip-api.com/json' response = fetch_url(url, timeout=5) if isinstance(response, six.binary_type): response = response.decode('utf-8') data = json.loads(response) country = data['country'] logger.info("Found your location: %s by %s", country, url) data = fetch_url(MIRRORS_URL, timeout=70, retry=True) soup = BeautifulSoup(data, 'html.parser') tables = soup.findAll('table') flag = False # flag is True when find the row's text is that country if not tables: raise Exception("Failed to locate
element in Ubuntu mirror page! (%s)" % MIRRORS_URL) else: for row in tables[0].findAll("tr"): if flag: if not row.a: # End of mirrors located in that country break else: for a in row.findAll('a', href=True): # Check if the link looks like a mirror URL. if a['href'].startswith(('http://', 'https://')): mirrors.add(CandidateMirror(mirror_url=a['href'])) if row.th and row.th.get_text() == country: flag = True if not mirrors: raise Exception("Failed to discover any Ubuntu mirrors! (using %s)" % MIRRORS_URL) return mirrors def discover_mirrors(): """ Discover available Ubuntu mirrors. :returns: A set of :class:`.CandidateMirror` objects that have their :attr:`~.CandidateMirror.mirror_url` property set and may have the :attr:`~.CandidateMirror.last_updated` property set. :raises: If no mirrors are discovered an exception is raised. This only queries :data:`MIRROR_SELECTION_URL` to discover available Ubuntu mirrors. Here's an example run: >>> from apt_smart.backends.ubuntu import discover_mirrors >>> from pprint import pprint >>> pprint(discover_mirrors()) """ timer = Timer() mirrors = set() mirrors = discover_mirror_selection() if not mirrors: logger.warning("Failed to discover any Ubuntu mirrors! (using %s)" % MIRROR_SELECTION_URL) logger.info("Trying to use %s as fallback" % MIRRORS_URL) mirrors = discover_mirrors_old() elif len(mirrors) < 2: logger.warning("Too few mirrors, trying to use %s to find more" % MIRRORS_URL) mirrors |= discover_mirrors_old() # add mirrors from discover_mirrors_old() logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Ubuntu mirror"), timer) return mirrors def discover_mirror_selection(): """Discover "geographically suitable" Ubuntu mirrors.""" timer = Timer() logger.info("Identifying fast Ubuntu mirrors using %s ..", MIRROR_SELECTION_URL) data = fetch_url(MIRROR_SELECTION_URL, timeout=3, retry=True, max_attempts=5) # shorter timeout with more retries is good for unstable connections to MIRROR_SELECTION_URL dammit = UnicodeDammit(data) mirrors = set( CandidateMirror(mirror_url=mirror_url.strip()) for mirror_url in dammit.unicode_markup.splitlines() if mirror_url and not mirror_url.isspace() and mirror_url.startswith(('http://', 'https://')) ) logger.debug("Found %s in %s.", pluralize(len(mirrors), "fast Ubuntu mirror"), timer) return mirrors def generate_sources_list(mirror_url, codename, suites=DEFAULT_SUITES, components=VALID_COMPONENTS, enable_sources=False): """ Generate the contents of ``/etc/apt/sources.list`` for an Ubuntu system. :param mirror_url: The base URL of the mirror (a string). :param codename: The codename of the Ubuntu release (a string like 'trusty' or 'xenial'). :param suites: An iterable of strings (defaults to :data:`DEFAULT_SUITES`, refer to :data:`VALID_SUITES` for details). :param components: An iterable of strings (refer to :data:`VALID_COMPONENTS` for details). :param enable_sources: :data:`True` to include ``deb-src`` entries, :data:`False` to omit them. :returns: The suggested contents of ``/etc/apt/sources.list`` (a string). """ # Validate the suites. invalid_suites = [s for s in suites if s not in VALID_SUITES] if invalid_suites: msg = "Invalid Ubuntu suite(s) given! (%s)" raise ValueError(msg % invalid_suites) # Validate the components. invalid_components = [c for c in components if c not in VALID_COMPONENTS] if invalid_components: msg = "Invalid Ubuntu component(s) given! (%s)" raise ValueError(msg % invalid_components) # Generate the /etc/apt/sources.list file contents. lines = [] directives = ('deb', 'deb-src') if enable_sources else ('deb',) for suite in suites: for directive in directives: lines.append(format( '{directive} {mirror} {suite} {components}', directive=directive, mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL) else (SECURITY_URL if suite == 'security' else mirror_url)), suite=(codename if suite == 'release' else codename + '-' + suite), components=' '.join(components), )) return '\n'.join(lines) ================================================ FILE: apt_smart/cli.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """ Usage: apt-smart [OPTIONS] The apt-smart program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating. Supported options: -r, --remote-host=SSH_ALIAS Operate on a remote system instead of the local system. The SSH_ALIAS argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access. -f, --find-current-mirror Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output. -F, --file-to-read=local_file_absolute_path Read a local absolute path (path and filename must NOT contain whitespace) file containing custom mirror URLs (one URL per line) to add custom mirrors to rank. -b, --find-best-mirror Discover available mirrors, rank them, select the best one and report its URL on standard output. -l, --list-mirrors List available (ranked) mirrors on the terminal in a human readable format. -L, --url-char-len=int An integer to specify the length of chars in mirrors' URL to display when using --list-mirrors, default is 34 -c, --change-mirror=MIRROR_URL Update /etc/apt/sources.list to use the given MIRROR_URL. -a, --auto-change-mirror Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror. -u, --update, --update-package-lists Update the package lists using `apt-get update', retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated. -U, --ubuntu Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. --auto-change-mirror --ubuntu will auto-change Linux Mint's upstream Ubuntu mirror -x, --exclude=PATTERN Add a pattern to the mirror selection blacklist. PATTERN is expected to be a shell pattern (containing wild cards like `?' and `*') that is matched against the full URL of each mirror. -v, --verbose Increase logging verbosity (can be repeated). -V, --version Show version number and Python version. -R, --create-chroot=local_dir_absolute_path Create chroot with the best mirror in a local directory with absolute_path -C, --codename=codename Must use with -R , create chroot with a codename of Ubuntu or Debian, e.g. bionic, buster -q, --quiet Decrease logging verbosity (can be repeated). -h, --help Show this message and exit. Note: since apt-smart uses `urlopen` method in The Python Standard Library, you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type: export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080' These will not persist however (no longer active after you close the terminal), so you may wish to add the line to your ~/.bashrc """ # Standard library modules. import functools import getopt import logging import sys import os # External dependencies. import coloredlogs from executor.contexts import LocalContext, RemoteContext from humanfriendly import format_size, format_table, format_timespan from humanfriendly.terminal import connected_to_terminal, output, usage, warning # Modules included in our package. from apt_smart import MAX_MIRRORS, URL_CHAR_LEN, AptMirrorUpdater from apt_smart import __version__ as updater_version # Initialize a logger for this module. logger = logging.getLogger(__name__) def main(): """Command line interface for the ``apt-smart`` program.""" # Initialize logging to the terminal and system log. coloredlogs.install(syslog=True) # Command line option defaults. context = LocalContext() updater = AptMirrorUpdater(context=context) limit = MAX_MIRRORS url_char_len = URL_CHAR_LEN ubuntu_mode = False chroot_path = '' codename = '' actions = [] # Parse the command line arguments. try: options, arguments = getopt.getopt(sys.argv[1:], 'r:fF:blL:c:auUx:m:vVR:C:qh', [ 'remote-host=', 'find-current-mirror', 'find-best-mirror', 'file-to-read=', 'list-mirrors', 'url-char-len=', 'change-mirror=', 'auto-change-mirror', 'update', 'update-package-lists', 'ubuntu', 'exclude=', 'max=', 'verbose', 'version', 'create-chroot=', 'codename=', 'quiet', 'help', ]) for option, value in options: if option in ('-r', '--remote-host'): if actions: msg = "The %s option should be the first option given on the command line!" raise Exception(msg % option) context = RemoteContext(value) updater = AptMirrorUpdater(context=context) elif option in ('-f', '--find-current-mirror'): actions.append(functools.partial(report_current_mirror, updater)) elif option in ('-F', '--file-to-read'): updater.custom_mirror_file_path = value elif option in ('-b', '--find-best-mirror'): actions.append(functools.partial(report_best_mirror, updater)) elif option in ('-l', '--list-mirrors'): actions.append(functools.partial(report_available_mirrors, updater)) elif option in ('-L', '--url-char-len'): url_char_len = int(value) elif option in ('-c', '--change-mirror'): if value.strip().startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/')): actions.append(functools.partial(updater.change_mirror, value)) else: raise Exception("\'%s\' is not a valid mirror URL" % value) elif option in ('-a', '--auto-change-mirror'): actions.append(updater.change_mirror) elif option in ('-u', '--update', '--update-package-lists'): actions.append(updater.smart_update) elif option in ('-U', '--ubuntu'): ubuntu_mode = True elif option in ('-x', '--exclude'): actions.insert(0, functools.partial(updater.ignore_mirror, value)) elif option in ('-m', '--max'): limit = int(value) elif option in ('-v', '--verbose'): coloredlogs.increase_verbosity() elif option in ('-V', '--version'): output("Version: %s on Python %i.%i", updater_version, sys.version_info[0], sys.version_info[1]) return elif option in ('-C', '--codename'): codename = value elif option in ('-R', '--create-chroot'): chroot_path = value elif option in ('-q', '--quiet'): coloredlogs.decrease_verbosity() elif option in ('-h', '--help'): usage(__doc__) return else: assert False, "Unhandled option!" if codename and not chroot_path: assert False, "--codename must be used with valid -R to specify chroot path" if chroot_path: actions.append(functools.partial(updater.create_chroot, chroot_path, codename=codename)) if not actions: usage(__doc__) return # Propagate options to the Python API. updater.max_mirrors = limit updater.url_char_len = url_char_len updater.ubuntu_mode = ubuntu_mode except Exception as e: warning("Error: Failed to parse command line arguments! (%s)" % e) sys.exit(1) # Perform the requested action(s). try: for callback in actions: callback() except Exception: logger.exception("Encountered unexpected exception! Aborting ..") sys.exit(1) def report_current_mirror(updater): """Print the URL of the currently configured ``apt-get`` mirror.""" output(updater.current_mirror) def report_best_mirror(updater): """Print the URL of the "best" mirror.""" output(updater.best_mirror) def report_available_mirrors(updater): """Print the available mirrors to the terminal (in a human friendly format).""" if connected_to_terminal() or os.getenv('TRAVIS') == 'true': # make Travis CI test this code # https://docs.travis-ci.com/user/environment-variables/#default-environment-variables have_bandwidth = any(c.bandwidth for c in updater.ranked_mirrors) have_last_updated = any(c.last_updated is not None for c in updater.ranked_mirrors) column_names = ["Rank", "Mirror URL", "Available?", "Updating?"] if have_last_updated: column_names.append("Last updated") if have_bandwidth: column_names.append("Bandwidth") data = [] long_mirror_urls = {} if os.getenv('TRAVIS') == 'true' and updater.url_char_len < 50: updater.url_char_len = 50 for i, candidate in enumerate(updater.ranked_mirrors, start=1): if len(candidate.mirror_url) <= updater.url_char_len: stripped_mirror_url = candidate.mirror_url else: # the mirror_url is too long, strip it stripped_mirror_url = candidate.mirror_url[:updater.url_char_len - 3] stripped_mirror_url = stripped_mirror_url + "..." long_mirror_urls[i] = candidate.mirror_url # store it, output as full afterwards row = [i, stripped_mirror_url, "Yes" if candidate.is_available else "No", "Yes" if candidate.is_updating else "No"] if have_last_updated: row.append("Up to date" if candidate.last_updated == 0 else ( "%s behind" % format_timespan(candidate.last_updated, max_units=1) if candidate.last_updated else "Unknown" )) if have_bandwidth: row.append("%s/s" % format_size(round(candidate.bandwidth, 0)) if candidate.bandwidth else "Unknown") data.append(row) output(format_table(data, column_names=column_names)) if long_mirror_urls: output(u"Full URLs which are too long to be shown in above table:") for key in long_mirror_urls: output(u"%i: %s", key, long_mirror_urls[key]) else: output(u"\n".join( candidate.mirror_url for candidate in updater.ranked_mirrors if candidate.is_available and not candidate.is_updating )) ================================================ FILE: apt_smart/http.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """Simple, robust and concurrent HTTP requests (designed for one very narrow use case).""" # Standard library modules. import logging import multiprocessing import signal # External dependencies. from humanfriendly import Timer, format_size from six.moves.urllib.request import urlopen from stopit import SignalTimeout # , TimeoutException # Initialize a logger for this module. logger = logging.getLogger(__name__) # Stop the `stopit' logger from logging tracebacks. logging.getLogger('stopit').setLevel(logging.ERROR) def fetch_url(url, timeout=10, retry=False, max_attempts=3): """ Fetch a URL, optionally retrying on failure. :param url: The URL to fetch (a string). :param timeout: The maximum time in seconds that's allowed to pass before the request is aborted (a number, defaults to 10 seconds). :param retry: Whether to retry on failure (defaults to :data:`False`). :param max_attempts: The maximum number of attempts when retrying is enabled (an integer, defaults to three). :returns: The response body (a byte string). :raises: Any of the following exceptions can be raised: - :exc:`NotFoundError` when the URL returns a 404 status code. - :exc:`InvalidResponseError` when the URL returns a status code that isn't 200. - :exc:`stopit.TimeoutException` when the request takes longer than `timeout` seconds (refer to the `stopit documentation `_ for details). - Any exception raised by Python's standard library in the last attempt (assuming all attempts raise an exception). """ timer = Timer() logger.debug("Fetching %s ..", url) for i in range(1, max_attempts + 1): try: with SignalTimeout(timeout, swallow_exc=False): response = urlopen(url) status_code = response.getcode() if status_code != 200: exc_type = (NotFoundError if status_code == 404 else InvalidResponseError) raise exc_type("URL returned unexpected status code %s! (%s)" % (status_code, url)) response_body = response.read() logger.debug("Took %s to fetch %s.", timer, url) return response_body except NotFoundError: # We never retry 404 responses but retry timeouts. raise except Exception as e: if retry and i < max_attempts: logger.warning("Failed to fetch %s, retrying (%i/%i, error was: %s)", url, i, max_attempts, e) else: raise def fetch_concurrent(urls, concurrency=None): """ Fetch the given URLs concurrently using :mod:`multiprocessing`. :param urls: An iterable of URLs (strings). :param concurrency: Override the concurrency (an integer, defaults to the value computed by :func:`get_default_concurrency()`). :returns: A list of tuples like those returned by :func:`fetch_worker()`. """ if concurrency is None: concurrency = get_default_concurrency() pool = multiprocessing.Pool(concurrency) try: return pool.map(fetch_worker, urls, chunksize=1) finally: pool.terminate() def get_default_concurrency(): """ Get the default concurrency for :func:`fetch_concurrent()`. :returns: A positive integer number. """ return max(4, multiprocessing.cpu_count() * 2) def fetch_worker(url): """ Fetch the given URL for :func:`fetch_concurrent()`. :param url: The URL to fetch (a string). :returns: A tuple of three values: 1. The URL that was fetched (a string). 2. The data that was fetched (a string or :data:`None`). 3. The number of seconds it took to fetch the URL (a number). """ # Ignore Control-C instead of raising KeyboardInterrupt because (due to a # quirk in multiprocessing) this can cause the parent and child processes # to get into a deadlock kind of state where only Control-Z will get you # your precious terminal back; super annoying IMHO. signal.signal(signal.SIGINT, signal.SIG_IGN) timer = Timer() try: data = fetch_url(url, retry=False) except Exception as e: logger.debug("Failed to fetch %s! (%s)", url, e) data = None else: kbps = format_size(round(len(data) / timer.elapsed_time, 2)) logger.debug("Downloaded %s at %s per second.", url, kbps) return url, data, timer.elapsed_time class InvalidResponseError(Exception): """Raised by :func:`fetch_url()` when a URL returns a status code that isn't 200.""" class NotFoundError(InvalidResponseError): """Raised by :func:`fetch_url()` when a URL returns a 404 status code.""" ================================================ FILE: apt_smart/releases.py ================================================ # Easy to use metadata on Debian and Ubuntu releases. # # Author: martin68 and Peter Odding # Last Change: May 31, 2020 # URL: https://apt-smart.readthedocs.io """ Easy to use metadata on Debian and Ubuntu releases. This module started out with the purpose of reliable `end of life`_ (EOL) detection for Debian and Ubuntu releases based on data provided by the distro-info-data_ package. Since then the need arose to access more of the available metadata and so the ``eol`` module became the ``releases`` module. Debian and Ubuntu releases have an EOL date that marks the end of support for each release. At that date the release stops receiving further (security) updates and some time after package mirrors stop serving the release. The distro-info-data_ package contains CSV files with metadata about Debian and Ubuntu releases. This module parses those CSV files to make this metadata available in Python. This enables `apt-smart` to make an informed decision about the following questions: 1. Is a given Debian or Ubuntu release expected to be available on mirrors or will it only be available in the archive of old releases? 2. Is the signing key of a given Ubuntu release expected to be included in the main keyring (:data:`UBUNTU_KEYRING_CURRENT`) or should the keyring with removed keys (:data:`UBUNTU_KEYRING_REMOVED`) be used? To make it possible to run `apt-smart` without direct access to the CSV files, a copy of the relevant information has been embedded in the source code. .. _end of life: https://en.wikipedia.org/wiki/End-of-life_(product) .. _distro-info-data: https://packages.debian.org/distro-info-data """ # Standard library modules. import csv import datetime import decimal import glob import logging import numbers import os # External dependencies. import six from executor import execute from humanfriendly.decorators import cached try: from property_manager3 import PropertyManager, key_property, lazy_property, required_property, writable_property except ImportError: from property_manager import PropertyManager, key_property, lazy_property, required_property, writable_property from six import string_types from itertools import product DISTRO_INFO_DIRECTORY = '/usr/share/distro-info' """The pathname of the directory with CSV files containing release metadata (a string).""" DEBIAN_KEYRING_CURRENT = '/usr/share/keyrings/debian-archive-keyring.gpg' """The pathname of the main Debian keyring file (a string).""" UBUNTU_KEYRING_CURRENT = '/usr/share/keyrings/ubuntu-archive-keyring.gpg' """The pathname of the main Ubuntu keyring file (a string).""" UBUNTU_KEYRING_REMOVED = '/usr/share/keyrings/ubuntu-archive-removed-keys.gpg' """The pathname of the Ubuntu keyring file with removed keys (a string).""" # Public identifiers that require documentation. __all__ = ( 'DISTRO_INFO_DIRECTORY', 'DEBIAN_KEYRING_CURRENT', 'UBUNTU_KEYRING_CURRENT', 'UBUNTU_KEYRING_REMOVED', 'Release', 'coerce_release', 'discover_releases', 'ubuntu_keyring_updated', ) # Initialize a logger. logger = logging.getLogger(__name__) def coerce_release(value): """ Try to coerce the given value to a Debian or Ubuntu release. :param value: The value to coerce (a number, a string or a :class:`Release` object). :returns: A :class:`Release` object. :raises: :exc:`~exceptions.ValueError` when the given value cannot be coerced to a known release. The following values can be coerced: - Numbers and numbers formatted as strings match :attr:`Release.version`. - Strings match :attr:`Release.codename` (case insensitive). .. warning:: Don't use floating point numbers like 10.04 because their actual value will be something like 10.039999999999999147 which won't match the intended release. """ # Release objects pass through untouched. if isinstance(value, Release): return value # Numbers and version strings are matched against release versions. if isinstance(value, numbers.Number) or is_version_string(value): typed_value = decimal.Decimal(value) matches = [release for release in discover_releases() if release.version == typed_value] if len(matches) != 1: msg = "The number %s doesn't match a known Debian or Ubuntu or Linux Mint release!" raise ValueError(msg % value) return matches[0] # Other strings are matched against release code names. matches = [release for release in discover_releases() if value.lower() in release.codename.lower()] if len(matches) != 1: msg = "The string %r doesn't match a known Debian or Ubuntu or Linux Mint release!" raise ValueError(msg % value) return matches[0] @cached def discover_releases(): """ Discover known Debian and Ubuntu releases. :returns: A list of discovered :class:`Release` objects sorted by :attr:`~Release.distributor_id` and :attr:`~Release.version`. The first time this function is called it will try to parse the CSV files in ``/usr/share/distro-info`` and merge any releases it finds with the releases embedded into the source code of this module. The result is cached and returned each time the function is called. It's not a problem if the ``/usr/share/distro-info`` directory doesn't exist or doesn't contain any ``*.csv`` files (it won't cause a warning or error). Of course in this case only the embedded releases will be returned. """ # Discover the known releases on the first call to discover_releases(). # First we check the CSV files on the system where apt-smart # is running, because those files may be more up-to-date than the # bundled information is. result = set() for filename in glob.glob(os.path.join(DISTRO_INFO_DIRECTORY, '*.csv')): for release in parse_csv_file(filename): result.add(release) # Add the releases bundled with apt-smart to the result # without causing duplicate entries (due to the use of a set and key # properties). result.update(BUNDLED_RELEASES) # Sort the releases by distributor ID and version / series. return sorted(result, key=lambda r: (r.distributor_id, r.version or 0, r.series)) def table_to_2d(table_tag): # https://stackoverflow.com/a/48451104 rowspans = [] # track pending rowspans rows = table_tag.find_all('tr') # first scan, see how many columns we need colcount = 0 for r, row in enumerate(rows): cells = row.find_all(['td', 'th'], recursive=False) # count columns (including spanned). # add active rowspans from preceding rows # we *ignore* the colspan value on the last cell, to prevent # creating 'phantom' columns with no actual cells, only extended # colspans. This is achieved by hardcoding the last cell width as 1. # a colspan of 0 means "fill until the end" but can really only apply # to the last cell; ignore it elsewhere. colcount = max( colcount, sum(int(c.get('colspan', 1)) or 1 for c in cells[:-1]) + len(cells[-1:]) + len(rowspans)) # update rowspan bookkeeping; 0 is a span to the bottom. rowspans += [int(c.get('rowspan', 1)) or len(rows) - r for c in cells] rowspans = [s - 1 for s in rowspans if s > 1] # it doesn't matter if there are still rowspan numbers 'active'; no extra # rows to show in the table means the larger than 1 rowspan numbers in the # last table row are ignored. # build an empty matrix for all possible cells table = [[None] * colcount for row in rows] # fill matrix from row data rowspans = {} # track pending rowspans, column number mapping to count for row, row_elem in enumerate(rows): span_offset = 0 # how many columns are skipped due to row and colspans for col, cell in enumerate(row_elem.find_all(['td', 'th'], recursive=False)): # adjust for preceding row and colspans col += span_offset while rowspans.get(col, 0): span_offset += 1 col += 1 # fill table data rowspan = rowspans[col] = int(cell.get('rowspan', 1)) or len(rows) - row colspan = int(cell.get('colspan', 1)) or colcount - col # next column is offset by the colspan span_offset += colspan - 1 value = cell.get_text() for drow, dcol in product(range(rowspan), range(colspan)): try: table[row + drow][col + dcol] = value rowspans[col + dcol] = rowspan except IndexError: # rowspan or colspan outside the confines of the table pass # update rowspan bookkeeping rowspans = {c: s - 1 for c, s in rowspans.items() if s > 1} return table def discover_linuxmint_releases(array_2d): d = {} # a dict to map table head to column number head = array_2d[0] for i, data in enumerate(head): d[data] = i last = None for entry in array_2d[1:-1]: if entry[d['Codename\n']] == last: continue # skip same codename entry last = entry[d['Codename\n']] yield Release( codename=parse_data_wiki(entry[d['Codename\n']]), compatible_repository=( parse_data_wiki(entry[d['Compatible repository\n']]).split('(')[1].split(' ')[0].lower() if entry[d['Compatible repository\n']].find('(') > 0 else parse_data_wiki(entry[d['Compatible repository\n']]) ), is_lts=('Yes' in entry[d['LTS?\n']]), created_date=parse_date_wiki(entry[d['Release date\n']]), distributor_id='linuxmint', eol_date=parse_date_wiki(entry[d['Support End\n']]), extended_eol_date=parse_date_wiki(entry[d['Support End\n']]), release_date=parse_date_wiki(entry[d['Release date\n']]), series=parse_data_wiki(entry[d['Codename\n']]).lower(), version=parse_version_wiki(entry[d['Version\n']]) if entry[d['Version\n']] else None, ) def is_version_string(value): """Check whether the given value is a string containing a positive number.""" try: return isinstance(value, string_types) and float(value) > 0 except Exception: return False def parse_csv_file(filename): """ Parse a CSV file in the format of the ``/usr/share/distro-info/*.csv`` files. :param filename: The pathname of the CSV file (a string). :returns: A generator of :class:`Release` objects. """ # We import this here to avoid a circular import. from apt_smart.backends.debian import LTS_RELEASES basename, extension = os.path.splitext(os.path.basename(filename)) distributor_id = basename.lower() with open(filename) as handle: for entry in csv.DictReader(handle): yield Release( codename=entry['codename'], is_lts=( entry['series'] in LTS_RELEASES if distributor_id == 'debian' else ( 'LTS' in entry['version'] if distributor_id == 'ubuntu' else ( # Neither Debian nor Ubuntu, let's not assume anything... False ) ) ), created_date=parse_date(entry['created']), distributor_id=distributor_id, eol_date=parse_date(entry['eol']), extended_eol_date=( # Special handling for Debian LTS releases. datetime.datetime.fromtimestamp(LTS_RELEASES[entry['series']]).date() if distributor_id == 'debian' and entry['series'] in LTS_RELEASES # Ubuntu LTS releases are defined by the CSV file. else parse_date(entry.get('eol-server')) ), release_date=parse_date(entry['release']), series=entry['series'], version=parse_version(entry['version']) if entry['version'] else None, ) def parse_data_wiki(value): r"""Strip a string such as ``19 December 2018 [18]\n`` to ``19 December 2018``""" if six.PY2: return value.encode("utf8").split('[')[0].strip() else: return value.split('[')[0].strip() def parse_date(value): """Convert a ``YYYY-MM-DD`` string to a :class:`datetime.date` object.""" return datetime.datetime.strptime(value, '%Y-%m-%d').date() if value else None def parse_date_wiki(value): r"""Convert a string such as ``19 December 2018`` ``August 02, 2019\n`` to a :class:`datetime.date` object.""" value = parse_data_wiki(value) if value == 'Unknown': value = '30 April 2008' if len(value) < 15: if not value[:1].isdigit(): # Only Month Year value = '30 ' + value if len(value) < 5: # Only Year value = '30 April ' + value try: return datetime.datetime.strptime(value, '%d %B %Y').date() if value else None except ValueError: return datetime.datetime.strptime(value, '%B %d, %Y').date() if value else None def parse_version(value): """Convert a version string to a floating point number.""" for token in value.split(): try: return decimal.Decimal(token) except ValueError: pass msg = "Failed to convert version string to number! (%r)" raise ValueError(msg % value) def parse_version_wiki(value): """Convert a version string (got from wiki page) to a floating point number.""" value = parse_data_wiki(value) value = value.split(": ")[1] for token in value.split(): try: return decimal.Decimal(token) except ValueError: pass msg = "Failed to convert version string to number! (%r)" raise ValueError(msg % value) @cached def ubuntu_keyring_updated(): """ Detect update `#1363482`_ to the ``ubuntu-keyring`` package. :returns: :data:`True` when version ``2016.10.27`` or newer is installed, :data:`False` when an older version is installed. This function checks if the changes discussed in Launchpad bug `#1363482`_ apply to the current system using the ``dpkg-query --show`` and ``dpkg --compare-versions`` commands. For more details refer to `issue #8`_. .. _#1363482: https://bugs.launchpad.net/ubuntu/+source/ubuntu-keyring/+bug/1363482 .. _issue #8: https://github.com/xolox/python-apt-mirror-updater/issues/8 """ # Use external commands to check the installed version of the package. version = execute('dpkg-query', '--show', '--showformat=${Version}', 'ubuntu-keyring', capture=True) logger.debug("Detected ubuntu-keyring package version: %s", version) updated = execute('dpkg', '--compare-versions', version, '>=', '2016.10.27', check=False, silent=True) logger.debug("Does Launchpad bug #1363482 apply? %s", updated) return updated class Release(PropertyManager): """Data class for metadata on Debian and Ubuntu releases.""" @key_property def codename(self): """The long version of :attr:`series` (a string).""" @writable_property def compatible_repository(self): """For Linux Mint, compatible which Ubuntu version's repository""" @required_property def created_date(self): """The date on which the release was created (a :class:`~datetime.date` object).""" @key_property def distributor_id(self): """The name of the distributor (a string like ``debian`` or ``ubuntu`` or ``linuxmint``).""" @writable_property def eol_date(self): """The date on which the desktop release stops being supported (a :class:`~datetime.date` object).""" @writable_property def extended_eol_date(self): """The date on which the server release stops being supported (a :class:`~datetime.date` object).""" @lazy_property def is_eol(self): """Whether the release has reached its end-of-life date (a boolean or :data:`None`).""" eol_date = self.extended_eol_date or self.eol_date if eol_date: return datetime.date.today() >= eol_date else: return False @writable_property def is_lts(self): """Whether a release is a long term support release (a boolean).""" @writable_property def release_date(self): """The date on which the release was published (a :class:`~datetime.date` object).""" @key_property def series(self): """The short version of :attr:`codename` (a string).""" @writable_property def version(self): """ The version number of the release (a :class:`~decimal.Decimal` number). This property has a :class:`~decimal.Decimal` value to enable proper sorting based on numeric comparison. """ @lazy_property def keyring_file(self): """ The pathname of the keyring with signing keys for this release (a string). This property exists to work around a bug in ``debootstrap`` which may use the wrong keyring to create Ubuntu chroots, for more details refer to :func:`ubuntu_keyring_updated()`. """ filename = None reason = None logger.debug("Selecting keyring file for %s ..", self) if self.distributor_id == 'debian': filename = DEBIAN_KEYRING_CURRENT reason = "only known keyring" elif self.distributor_id == 'ubuntu': if ubuntu_keyring_updated(): if self.version > decimal.Decimal('12.04'): filename = UBUNTU_KEYRING_CURRENT reason = "new keyring package / new release" else: filename = UBUNTU_KEYRING_REMOVED reason = "new keyring package / old release" else: filename = UBUNTU_KEYRING_CURRENT reason = "old keyring package" else: msg = "Unsupported distributor ID! (%s)" raise EnvironmentError(msg % self.distributor_id) logger.debug("Using %s (reason: %s).", filename, reason) return filename def __str__(self): """ Render a human friendly representation of a :class:`Release` object. The result will be something like this: - Debian 9 (stretch) - Ubuntu 18.04 (bionic) """ label = [self.distributor_id.capitalize()] if self.version: label.append(str(self.version)) label.append("(%s)" % self.series) return " ".join(label) # [[[cog # # import cog # import decimal # from bs4 import BeautifulSoup # from apt_smart.releases import discover_releases, discover_linuxmint_releases, table_to_2d # from apt_smart.http import fetch_url # # set_of_releases = set() # url = 'https://en.wikipedia.org/wiki/Linux_Mint_version_history' # response = fetch_url(url, timeout=15, retry=True) # soup = BeautifulSoup(response, 'html.parser') # tables = soup.findAll('table') # for release in discover_linuxmint_releases(table_to_2d(tables[1])): # set_of_releases.add(release) # set_of_releases.update(discover_releases()) # indent = " " * 4 # cog.out("\nBUNDLED_RELEASES = [\n") # for release in sorted(set_of_releases, key=lambda r: (r.distributor_id, r.version or 0, r.series)): # cog.out(indent + "Release(\n") # for name in release.find_properties(cached=False): # value = getattr(release, name) # if value is not None: # if isinstance(value, decimal.Decimal): # # It seems weirdly inconsistency to me that this is needed # # for decimal.Decimal() but not for datetime.date() but I # # guess the simple explanation is that repr() output simply # # isn't guaranteed to be accepted by eval(). # value = "decimal." + repr(value) # else: # value = repr(value) # cog.out(indent * 2 + name + "=" + value + ",\n") # cog.out(indent + "),\n") # cog.out("]\n\n") # # ]]] BUNDLED_RELEASES = [ Release( codename='Experimental', created_date=datetime.date(1993, 8, 16), distributor_id='debian', is_lts=False, series='experimental', ), Release( codename='Sid', created_date=datetime.date(1993, 8, 16), distributor_id='debian', is_lts=False, series='sid', ), Release( codename='Buzz', created_date=datetime.date(1993, 8, 16), distributor_id='debian', eol_date=datetime.date(1997, 6, 5), is_lts=False, release_date=datetime.date(1996, 6, 17), series='buzz', version=decimal.Decimal('1.1'), ), Release( codename='Rex', created_date=datetime.date(1996, 6, 17), distributor_id='debian', eol_date=datetime.date(1998, 6, 5), is_lts=False, release_date=datetime.date(1996, 12, 12), series='rex', version=decimal.Decimal('1.2'), ), Release( codename='Bo', created_date=datetime.date(1996, 12, 12), distributor_id='debian', eol_date=datetime.date(1999, 3, 9), is_lts=False, release_date=datetime.date(1997, 6, 5), series='bo', version=decimal.Decimal('1.3'), ), Release( codename='Hamm', created_date=datetime.date(1997, 6, 5), distributor_id='debian', eol_date=datetime.date(2000, 3, 9), is_lts=False, release_date=datetime.date(1998, 7, 24), series='hamm', version=decimal.Decimal('2.0'), ), Release( codename='Slink', created_date=datetime.date(1998, 7, 24), distributor_id='debian', eol_date=datetime.date(2000, 10, 30), is_lts=False, release_date=datetime.date(1999, 3, 9), series='slink', version=decimal.Decimal('2.1'), ), Release( codename='Potato', created_date=datetime.date(1999, 3, 9), distributor_id='debian', eol_date=datetime.date(2003, 7, 30), is_lts=False, release_date=datetime.date(2000, 8, 15), series='potato', version=decimal.Decimal('2.2'), ), Release( codename='Woody', created_date=datetime.date(2000, 8, 15), distributor_id='debian', eol_date=datetime.date(2006, 6, 30), is_lts=False, release_date=datetime.date(2002, 7, 19), series='woody', version=decimal.Decimal('3.0'), ), Release( codename='Sarge', created_date=datetime.date(2002, 7, 19), distributor_id='debian', eol_date=datetime.date(2008, 3, 30), is_lts=False, release_date=datetime.date(2005, 6, 6), series='sarge', version=decimal.Decimal('3.1'), ), Release( codename='Etch', created_date=datetime.date(2005, 6, 6), distributor_id='debian', eol_date=datetime.date(2010, 2, 15), is_lts=False, release_date=datetime.date(2007, 4, 8), series='etch', version=decimal.Decimal('4.0'), ), Release( codename='Lenny', created_date=datetime.date(2007, 4, 8), distributor_id='debian', eol_date=datetime.date(2012, 2, 6), is_lts=False, release_date=datetime.date(2009, 2, 14), series='lenny', version=decimal.Decimal('5.0'), ), Release( codename='Squeeze', created_date=datetime.date(2009, 2, 14), distributor_id='debian', eol_date=datetime.date(2014, 5, 31), is_lts=False, release_date=datetime.date(2011, 2, 6), series='squeeze', version=decimal.Decimal('6.0'), ), Release( codename='Wheezy', created_date=datetime.date(2011, 2, 6), distributor_id='debian', eol_date=datetime.date(2016, 4, 26), is_lts=False, release_date=datetime.date(2013, 5, 4), series='wheezy', version=decimal.Decimal('7'), ), Release( codename='Jessie', created_date=datetime.date(2013, 5, 4), distributor_id='debian', eol_date=datetime.date(2018, 6, 6), extended_eol_date=datetime.date(2020, 6, 30), is_lts=True, release_date=datetime.date(2015, 4, 25), series='jessie', version=decimal.Decimal('8'), ), Release( codename='Stretch', created_date=datetime.date(2015, 4, 25), distributor_id='debian', eol_date=datetime.date(2020, 7, 6), extended_eol_date=datetime.date(2022, 6, 30), is_lts=True, release_date=datetime.date(2017, 6, 17), series='stretch', version=decimal.Decimal('9'), ), Release( codename='Buster', created_date=datetime.date(2017, 6, 17), distributor_id='debian', is_lts=False, release_date=datetime.date(2019, 7, 6), series='buster', version=decimal.Decimal('10'), ), Release( codename='Bullseye', created_date=datetime.date(2019, 7, 6), distributor_id='debian', is_lts=False, series='bullseye', version=decimal.Decimal('11'), ), Release( codename='Bookworm', created_date=datetime.date(2021, 8, 1), distributor_id='debian', is_lts=False, series='bookworm', version=decimal.Decimal('12'), ), Release( codename='Ada', compatible_repository='Kubuntu 6.06', created_date=datetime.date(2006, 8, 27), distributor_id='linuxmint', eol_date=datetime.date(2008, 4, 30), extended_eol_date=datetime.date(2008, 4, 30), is_lts=False, release_date=datetime.date(2006, 8, 27), series='ada', version=decimal.Decimal('1.0'), ), Release( codename='Barbara', compatible_repository='edgy', created_date=datetime.date(2006, 11, 13), distributor_id='linuxmint', eol_date=datetime.date(2008, 4, 30), extended_eol_date=datetime.date(2008, 4, 30), is_lts=False, release_date=datetime.date(2006, 11, 13), series='barbara', version=decimal.Decimal('2.0'), ), Release( codename='Bea', compatible_repository='edgy', created_date=datetime.date(2006, 12, 20), distributor_id='linuxmint', eol_date=datetime.date(2008, 4, 30), extended_eol_date=datetime.date(2008, 4, 30), is_lts=False, release_date=datetime.date(2006, 12, 20), series='bea', version=decimal.Decimal('2.1'), ), Release( codename='Bianca', compatible_repository='edgy', created_date=datetime.date(2007, 2, 20), distributor_id='linuxmint', eol_date=datetime.date(2008, 4, 30), extended_eol_date=datetime.date(2008, 4, 30), is_lts=False, release_date=datetime.date(2007, 2, 20), series='bianca', version=decimal.Decimal('2.2'), ), Release( codename='Cassandra', compatible_repository='feisty', created_date=datetime.date(2007, 5, 30), distributor_id='linuxmint', eol_date=datetime.date(2008, 10, 30), extended_eol_date=datetime.date(2008, 10, 30), is_lts=False, release_date=datetime.date(2007, 5, 30), series='cassandra', version=decimal.Decimal('3.0'), ), Release( codename='Celena', compatible_repository='feisty', created_date=datetime.date(2007, 9, 24), distributor_id='linuxmint', eol_date=datetime.date(2008, 10, 30), extended_eol_date=datetime.date(2008, 10, 30), is_lts=False, release_date=datetime.date(2007, 9, 24), series='celena', version=decimal.Decimal('3.1'), ), Release( codename='Daryna', compatible_repository='gutsy', created_date=datetime.date(2007, 10, 15), distributor_id='linuxmint', eol_date=datetime.date(2009, 4, 30), extended_eol_date=datetime.date(2009, 4, 30), is_lts=False, release_date=datetime.date(2007, 10, 15), series='daryna', version=decimal.Decimal('4.0'), ), Release( codename='Elyssa', compatible_repository='hardy', created_date=datetime.date(2008, 6, 8), distributor_id='linuxmint', eol_date=datetime.date(2011, 4, 30), extended_eol_date=datetime.date(2011, 4, 30), is_lts=True, release_date=datetime.date(2008, 6, 8), series='elyssa', version=decimal.Decimal('5'), ), Release( codename='Felicia', compatible_repository='intrepid', created_date=datetime.date(2008, 12, 15), distributor_id='linuxmint', eol_date=datetime.date(2010, 4, 30), extended_eol_date=datetime.date(2010, 4, 30), is_lts=False, release_date=datetime.date(2008, 12, 15), series='felicia', version=decimal.Decimal('6'), ), Release( codename='Gloria', compatible_repository='jaunty', created_date=datetime.date(2009, 5, 26), distributor_id='linuxmint', eol_date=datetime.date(2010, 10, 30), extended_eol_date=datetime.date(2010, 10, 30), is_lts=False, release_date=datetime.date(2009, 5, 26), series='gloria', version=decimal.Decimal('7'), ), Release( codename='Helena', compatible_repository='karmic', created_date=datetime.date(2009, 11, 28), distributor_id='linuxmint', eol_date=datetime.date(2011, 4, 30), extended_eol_date=datetime.date(2011, 4, 30), is_lts=False, release_date=datetime.date(2009, 11, 28), series='helena', version=decimal.Decimal('8'), ), Release( codename='Isadora', compatible_repository='lucid', created_date=datetime.date(2010, 5, 18), distributor_id='linuxmint', eol_date=datetime.date(2013, 4, 30), extended_eol_date=datetime.date(2013, 4, 30), is_lts=True, release_date=datetime.date(2010, 5, 18), series='isadora', version=decimal.Decimal('9'), ), Release( codename='Julia', compatible_repository='maverick', created_date=datetime.date(2010, 11, 12), distributor_id='linuxmint', eol_date=datetime.date(2012, 4, 30), extended_eol_date=datetime.date(2012, 4, 30), is_lts=False, release_date=datetime.date(2010, 11, 12), series='julia', version=decimal.Decimal('10'), ), Release( codename='Katya', compatible_repository='natty', created_date=datetime.date(2011, 5, 26), distributor_id='linuxmint', eol_date=datetime.date(2012, 10, 30), extended_eol_date=datetime.date(2012, 10, 30), is_lts=False, release_date=datetime.date(2011, 5, 26), series='katya', version=decimal.Decimal('11'), ), Release( codename='Lisa', compatible_repository='oneiric', created_date=datetime.date(2011, 11, 26), distributor_id='linuxmint', eol_date=datetime.date(2013, 4, 30), extended_eol_date=datetime.date(2013, 4, 30), is_lts=False, release_date=datetime.date(2011, 11, 26), series='lisa', version=decimal.Decimal('12'), ), Release( codename='Maya', compatible_repository='precise', created_date=datetime.date(2012, 5, 23), distributor_id='linuxmint', eol_date=datetime.date(2017, 4, 30), extended_eol_date=datetime.date(2017, 4, 30), is_lts=True, release_date=datetime.date(2012, 5, 23), series='maya', version=decimal.Decimal('13'), ), Release( codename='Nadia', compatible_repository='quantal', created_date=datetime.date(2012, 11, 20), distributor_id='linuxmint', eol_date=datetime.date(2014, 5, 30), extended_eol_date=datetime.date(2014, 5, 30), is_lts=False, release_date=datetime.date(2012, 11, 20), series='nadia', version=decimal.Decimal('14'), ), Release( codename='Olivia', compatible_repository='raring', created_date=datetime.date(2013, 5, 29), distributor_id='linuxmint', eol_date=datetime.date(2014, 1, 30), extended_eol_date=datetime.date(2014, 1, 30), is_lts=False, release_date=datetime.date(2013, 5, 29), series='olivia', version=decimal.Decimal('15'), ), Release( codename='Petra', compatible_repository='saucy', created_date=datetime.date(2013, 11, 30), distributor_id='linuxmint', eol_date=datetime.date(2014, 7, 30), extended_eol_date=datetime.date(2014, 7, 30), is_lts=False, release_date=datetime.date(2013, 11, 30), series='petra', version=decimal.Decimal('16'), ), Release( codename='Qiana', compatible_repository='trusty', created_date=datetime.date(2014, 5, 31), distributor_id='linuxmint', eol_date=datetime.date(2019, 4, 30), extended_eol_date=datetime.date(2019, 4, 30), is_lts=True, release_date=datetime.date(2014, 5, 31), series='qiana', version=decimal.Decimal('17'), ), Release( codename='Rebecca', compatible_repository='trusty', created_date=datetime.date(2014, 11, 29), distributor_id='linuxmint', eol_date=datetime.date(2019, 4, 30), extended_eol_date=datetime.date(2019, 4, 30), is_lts=True, release_date=datetime.date(2014, 11, 29), series='rebecca', version=decimal.Decimal('17.1'), ), Release( codename='Rafaela', compatible_repository='trusty', created_date=datetime.date(2015, 6, 30), distributor_id='linuxmint', eol_date=datetime.date(2019, 4, 30), extended_eol_date=datetime.date(2019, 4, 30), is_lts=True, release_date=datetime.date(2015, 6, 30), series='rafaela', version=decimal.Decimal('17.2'), ), Release( codename='Rosa', compatible_repository='trusty', created_date=datetime.date(2015, 12, 4), distributor_id='linuxmint', eol_date=datetime.date(2019, 4, 30), extended_eol_date=datetime.date(2019, 4, 30), is_lts=True, release_date=datetime.date(2015, 12, 4), series='rosa', version=decimal.Decimal('17.3'), ), Release( codename='Sarah', compatible_repository='xenial', created_date=datetime.date(2016, 6, 30), distributor_id='linuxmint', eol_date=datetime.date(2021, 4, 30), extended_eol_date=datetime.date(2021, 4, 30), is_lts=True, release_date=datetime.date(2016, 6, 30), series='sarah', version=decimal.Decimal('18'), ), Release( codename='Serena', compatible_repository='xenial', created_date=datetime.date(2016, 12, 16), distributor_id='linuxmint', eol_date=datetime.date(2021, 4, 30), extended_eol_date=datetime.date(2021, 4, 30), is_lts=True, release_date=datetime.date(2016, 12, 16), series='serena', version=decimal.Decimal('18.1'), ), Release( codename='Sonya', compatible_repository='xenial', created_date=datetime.date(2017, 7, 2), distributor_id='linuxmint', eol_date=datetime.date(2021, 4, 30), extended_eol_date=datetime.date(2021, 4, 30), is_lts=True, release_date=datetime.date(2017, 7, 2), series='sonya', version=decimal.Decimal('18.2'), ), Release( codename='Sylvia', compatible_repository='xenial', created_date=datetime.date(2017, 11, 27), distributor_id='linuxmint', eol_date=datetime.date(2021, 4, 30), extended_eol_date=datetime.date(2021, 4, 30), is_lts=True, release_date=datetime.date(2017, 11, 27), series='sylvia', version=decimal.Decimal('18.3'), ), Release( codename='Tara', compatible_repository='bionic', created_date=datetime.date(2018, 6, 29), distributor_id='linuxmint', eol_date=datetime.date(2023, 4, 30), extended_eol_date=datetime.date(2023, 4, 30), is_lts=True, release_date=datetime.date(2018, 6, 29), series='tara', version=decimal.Decimal('19'), ), Release( codename='Tessa', compatible_repository='bionic', created_date=datetime.date(2018, 12, 19), distributor_id='linuxmint', eol_date=datetime.date(2023, 4, 30), extended_eol_date=datetime.date(2023, 4, 30), is_lts=True, release_date=datetime.date(2018, 12, 19), series='tessa', version=decimal.Decimal('19.1'), ), Release( codename='Tina', compatible_repository='bionic', created_date=datetime.date(2019, 8, 2), distributor_id='linuxmint', eol_date=datetime.date(2023, 4, 30), extended_eol_date=datetime.date(2023, 4, 30), is_lts=True, release_date=datetime.date(2019, 8, 2), series='tina', version=decimal.Decimal('19.2'), ), Release( codename='Tricia', compatible_repository='bionic', created_date=datetime.date(2019, 12, 18), distributor_id='linuxmint', eol_date=datetime.date(2023, 4, 30), extended_eol_date=datetime.date(2023, 4, 30), is_lts=True, release_date=datetime.date(2019, 12, 18), series='tricia', version=decimal.Decimal('19.3'), ), Release( codename='Ulyana', compatible_repository='focal', created_date=datetime.date(2020, 6, 27), distributor_id='linuxmint', eol_date=datetime.date(2025, 4, 30), extended_eol_date=datetime.date(2025, 4, 30), is_lts=True, release_date=datetime.date(2020, 6, 27), series='ulyana', version=decimal.Decimal('20'), ), Release( codename='Warty Warthog', created_date=datetime.date(2004, 3, 5), distributor_id='ubuntu', eol_date=datetime.date(2006, 4, 30), is_lts=False, release_date=datetime.date(2004, 10, 20), series='warty', version=decimal.Decimal('4.10'), ), Release( codename='Hoary Hedgehog', created_date=datetime.date(2004, 10, 20), distributor_id='ubuntu', eol_date=datetime.date(2006, 10, 31), is_lts=False, release_date=datetime.date(2005, 4, 8), series='hoary', version=decimal.Decimal('5.04'), ), Release( codename='Breezy Badger', created_date=datetime.date(2005, 4, 8), distributor_id='ubuntu', eol_date=datetime.date(2007, 4, 13), is_lts=False, release_date=datetime.date(2005, 10, 12), series='breezy', version=decimal.Decimal('5.10'), ), Release( codename='Dapper Drake', created_date=datetime.date(2005, 10, 12), distributor_id='ubuntu', eol_date=datetime.date(2009, 7, 14), extended_eol_date=datetime.date(2011, 6, 1), is_lts=True, release_date=datetime.date(2006, 6, 1), series='dapper', version=decimal.Decimal('6.06'), ), Release( codename='Edgy Eft', created_date=datetime.date(2006, 6, 1), distributor_id='ubuntu', eol_date=datetime.date(2008, 4, 25), is_lts=False, release_date=datetime.date(2006, 10, 26), series='edgy', version=decimal.Decimal('6.10'), ), Release( codename='Feisty Fawn', created_date=datetime.date(2006, 10, 26), distributor_id='ubuntu', eol_date=datetime.date(2008, 10, 19), is_lts=False, release_date=datetime.date(2007, 4, 19), series='feisty', version=decimal.Decimal('7.04'), ), Release( codename='Gutsy Gibbon', created_date=datetime.date(2007, 4, 19), distributor_id='ubuntu', eol_date=datetime.date(2009, 4, 18), is_lts=False, release_date=datetime.date(2007, 10, 18), series='gutsy', version=decimal.Decimal('7.10'), ), Release( codename='Hardy Heron', created_date=datetime.date(2007, 10, 18), distributor_id='ubuntu', eol_date=datetime.date(2011, 5, 12), extended_eol_date=datetime.date(2013, 5, 9), is_lts=True, release_date=datetime.date(2008, 4, 24), series='hardy', version=decimal.Decimal('8.04'), ), Release( codename='Intrepid Ibex', created_date=datetime.date(2008, 4, 24), distributor_id='ubuntu', eol_date=datetime.date(2010, 4, 30), is_lts=False, release_date=datetime.date(2008, 10, 30), series='intrepid', version=decimal.Decimal('8.10'), ), Release( codename='Jaunty Jackalope', created_date=datetime.date(2008, 10, 30), distributor_id='ubuntu', eol_date=datetime.date(2010, 10, 23), is_lts=False, release_date=datetime.date(2009, 4, 23), series='jaunty', version=decimal.Decimal('9.04'), ), Release( codename='Karmic Koala', created_date=datetime.date(2009, 4, 23), distributor_id='ubuntu', eol_date=datetime.date(2011, 4, 29), is_lts=False, release_date=datetime.date(2009, 10, 29), series='karmic', version=decimal.Decimal('9.10'), ), Release( codename='Lucid Lynx', created_date=datetime.date(2009, 10, 29), distributor_id='ubuntu', eol_date=datetime.date(2013, 5, 9), extended_eol_date=datetime.date(2015, 4, 29), is_lts=True, release_date=datetime.date(2010, 4, 29), series='lucid', version=decimal.Decimal('10.04'), ), Release( codename='Maverick Meerkat', created_date=datetime.date(2010, 4, 29), distributor_id='ubuntu', eol_date=datetime.date(2012, 4, 10), is_lts=False, release_date=datetime.date(2010, 10, 10), series='maverick', version=decimal.Decimal('10.10'), ), Release( codename='Natty Narwhal', created_date=datetime.date(2010, 10, 10), distributor_id='ubuntu', eol_date=datetime.date(2012, 10, 28), is_lts=False, release_date=datetime.date(2011, 4, 28), series='natty', version=decimal.Decimal('11.04'), ), Release( codename='Oneiric Ocelot', created_date=datetime.date(2011, 4, 28), distributor_id='ubuntu', eol_date=datetime.date(2013, 5, 9), is_lts=False, release_date=datetime.date(2011, 10, 13), series='oneiric', version=decimal.Decimal('11.10'), ), Release( codename='Precise Pangolin', created_date=datetime.date(2011, 10, 13), distributor_id='ubuntu', eol_date=datetime.date(2017, 4, 26), extended_eol_date=datetime.date(2017, 4, 26), is_lts=True, release_date=datetime.date(2012, 4, 26), series='precise', version=decimal.Decimal('12.04'), ), Release( codename='Quantal Quetzal', created_date=datetime.date(2012, 4, 26), distributor_id='ubuntu', eol_date=datetime.date(2014, 5, 16), is_lts=False, release_date=datetime.date(2012, 10, 18), series='quantal', version=decimal.Decimal('12.10'), ), Release( codename='Raring Ringtail', created_date=datetime.date(2012, 10, 18), distributor_id='ubuntu', eol_date=datetime.date(2014, 1, 27), is_lts=False, release_date=datetime.date(2013, 4, 25), series='raring', version=decimal.Decimal('13.04'), ), Release( codename='Saucy Salamander', created_date=datetime.date(2013, 4, 25), distributor_id='ubuntu', eol_date=datetime.date(2014, 7, 17), is_lts=False, release_date=datetime.date(2013, 10, 17), series='saucy', version=decimal.Decimal('13.10'), ), Release( codename='Trusty Tahr', created_date=datetime.date(2013, 10, 17), distributor_id='ubuntu', eol_date=datetime.date(2019, 4, 25), extended_eol_date=datetime.date(2019, 4, 25), is_lts=True, release_date=datetime.date(2014, 4, 17), series='trusty', version=decimal.Decimal('14.04'), ), Release( codename='Utopic Unicorn', created_date=datetime.date(2014, 4, 17), distributor_id='ubuntu', eol_date=datetime.date(2015, 7, 23), is_lts=False, release_date=datetime.date(2014, 10, 23), series='utopic', version=decimal.Decimal('14.10'), ), Release( codename='Vivid Vervet', created_date=datetime.date(2014, 10, 23), distributor_id='ubuntu', eol_date=datetime.date(2016, 1, 23), is_lts=False, release_date=datetime.date(2015, 4, 23), series='vivid', version=decimal.Decimal('15.04'), ), Release( codename='Wily Werewolf', created_date=datetime.date(2015, 4, 23), distributor_id='ubuntu', eol_date=datetime.date(2016, 7, 22), is_lts=False, release_date=datetime.date(2015, 10, 22), series='wily', version=decimal.Decimal('15.10'), ), Release( codename='Xenial Xerus', created_date=datetime.date(2015, 10, 22), distributor_id='ubuntu', eol_date=datetime.date(2021, 4, 21), extended_eol_date=datetime.date(2021, 4, 21), is_lts=True, release_date=datetime.date(2016, 4, 21), series='xenial', version=decimal.Decimal('16.04'), ), Release( codename='Yakkety Yak', created_date=datetime.date(2016, 4, 21), distributor_id='ubuntu', eol_date=datetime.date(2017, 7, 20), is_lts=False, release_date=datetime.date(2016, 10, 13), series='yakkety', version=decimal.Decimal('16.10'), ), Release( codename='Zesty Zapus', created_date=datetime.date(2016, 10, 13), distributor_id='ubuntu', eol_date=datetime.date(2018, 1, 13), is_lts=False, release_date=datetime.date(2017, 4, 13), series='zesty', version=decimal.Decimal('17.04'), ), Release( codename='Artful Aardvark', created_date=datetime.date(2017, 4, 13), distributor_id='ubuntu', eol_date=datetime.date(2018, 7, 19), is_lts=False, release_date=datetime.date(2017, 10, 19), series='artful', version=decimal.Decimal('17.10'), ), Release( codename='Bionic Beaver', created_date=datetime.date(2017, 10, 19), distributor_id='ubuntu', eol_date=datetime.date(2023, 4, 26), extended_eol_date=datetime.date(2023, 4, 26), is_lts=True, release_date=datetime.date(2018, 4, 26), series='bionic', version=decimal.Decimal('18.04'), ), Release( codename='Cosmic Cuttlefish', created_date=datetime.date(2018, 4, 26), distributor_id='ubuntu', eol_date=datetime.date(2019, 7, 18), is_lts=False, release_date=datetime.date(2018, 10, 18), series='cosmic', version=decimal.Decimal('18.10'), ), Release( codename='Disco Dingo', created_date=datetime.date(2018, 10, 18), distributor_id='ubuntu', eol_date=datetime.date(2020, 1, 18), is_lts=False, release_date=datetime.date(2019, 4, 18), series='disco', version=decimal.Decimal('19.04'), ), Release( codename='Eoan Ermine', created_date=datetime.date(2019, 4, 18), distributor_id='ubuntu', eol_date=datetime.date(2020, 7, 17), is_lts=False, release_date=datetime.date(2019, 10, 17), series='eoan', version=decimal.Decimal('19.10'), ), Release( codename='Focal Fossa', created_date=datetime.date(2019, 10, 17), distributor_id='ubuntu', eol_date=datetime.date(2025, 4, 23), extended_eol_date=datetime.date(2025, 4, 23), is_lts=True, release_date=datetime.date(2020, 4, 23), series='focal', version=decimal.Decimal('20.04'), ), Release( codename='Groovy Gorilla', created_date=datetime.date(2020, 4, 23), distributor_id='ubuntu', eol_date=datetime.date(2021, 7, 22), is_lts=False, release_date=datetime.date(2020, 10, 22), series='groovy', version=decimal.Decimal('20.10'), ), ] # [[[end]]] ================================================ FILE: apt_smart/tests.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """Test suite for the ``apt-smart`` package.""" # Standard library modules. import decimal import logging import os import time # External dependencies. from executor import execute from humanfriendly.testing import TestCase, run_cli # from humanfriendly.text import split # Modules included in our package. from apt_smart import AptMirrorUpdater, normalize_mirror_url, MirrorStatus from apt_smart.cli import main from apt_smart.releases import ( DEBIAN_KEYRING_CURRENT, UBUNTU_KEYRING_CURRENT, UBUNTU_KEYRING_REMOVED, coerce_release, discover_releases, ubuntu_keyring_updated, ) # Initialize a logger for this module. logger = logging.getLogger(__name__) class AptMirrorUpdaterTestCase(TestCase): """:mod:`unittest` compatible container for the :mod:`apt_smart` test suite.""" def test_debian_mirror_discovery(self): """Test the discovery of Debian mirror URLs.""" from apt_smart.backends.debian import discover_mirrors mirrors = discover_mirrors() assert len(mirrors) > 6 for candidate in mirrors: check_debian_mirror(candidate.mirror_url) def test_ubuntu_mirror_discovery(self): """Test the discovery of Ubuntu mirror URLs.""" from apt_smart.backends.ubuntu import discover_mirrors mirrors = discover_mirrors() assert len(mirrors) > 10 for candidate in mirrors: check_ubuntu_mirror(candidate.mirror_url) def test_ubuntu_mirror_discovery_old(self): """Test fallback the discovery of Ubuntu mirror URLs using launchpad.net.""" from apt_smart.backends.ubuntu import discover_mirrors_old mirrors = discover_mirrors_old() assert len(mirrors) > 10 for candidate in mirrors: check_ubuntu_mirror(candidate.mirror_url) def test_linuxmint_mirror_discovery(self): """Test the discovery of Linux Mint mirror URLs.""" from apt_smart.backends.linuxmint import discover_mirrors mirrors = discover_mirrors() assert len(mirrors) > 10 for candidate in mirrors: check_ubuntu_mirror(candidate.mirror_url) def test_adaptive_mirror_discovery(self): """Test the discovery of mirrors for the current type of system.""" updater = AptMirrorUpdater() assert len(updater.available_mirrors) > 10 for candidate in updater.available_mirrors: check_mirror_url(candidate.mirror_url) def test_mirror_ranking(self): """Test the ranking of discovered mirrors.""" updater = AptMirrorUpdater() # Make sure that multiple discovered mirrors are available. assert sum(m.is_available for m in updater.ranked_mirrors) > 9 def test_best_mirror_selection(self): """Test the selection of a "best" mirror.""" updater = AptMirrorUpdater() check_mirror_url(updater.best_mirror) def test_current_mirror_discovery(self): """Test that the current mirror can be extracted from ``/etc/apt/sources.list``.""" exit_code, output = run_cli(main, '--find-current-mirror') assert exit_code == 0 check_mirror_url(output.strip()) def test_create_chroot(self): """Test create chroot""" if os.getuid() != 0: return self.skipTest("root privileges required to opt in") updater = AptMirrorUpdater() updater.create_chroot('/test_chroot') assert 'Filename:' in updater.context.capture('apt-cache', 'show', 'python') def test_create_chroot_with_codename(self): """Test create chroot with codename""" if os.getuid() != 0: return self.skipTest("root privileges required to opt in") updater = AptMirrorUpdater() updater.create_chroot('/test_chroot_stretch', 'stretch') assert 'Filename:' in updater.context.capture('apt-cache', 'show', 'python') def test_change_mirror(self): """Test change mirror""" if os.getuid() != 0: return self.skipTest("root privileges required to opt in") updater = AptMirrorUpdater() updater.change_mirror() assert have_package_lists() def test_report_available_mirrors(self): """Test that print the available mirrors to the terminal.""" exit_code, output = run_cli(main, '--list-mirrors') assert exit_code == 0 def test_report_available_mirrors_more(self): """Test that print the available mirrors to the terminal.""" test_custom_mirrors = os.path.join(os.getcwd(), 'test_custom_mirrors.txt') exit_code, output = run_cli(main, '--list-mirrors', '--file-to-read', test_custom_mirrors, '--exclude', '*edu*', '--url-char-len', '51') assert exit_code == 0 def test_dumb_update(self): """Test that our dumb ``apt-get update`` wrapper works.""" if os.getuid() != 0: return self.skipTest("root privileges required to opt in") updater = AptMirrorUpdater() # Remove all existing package lists. updater.clear_package_lists() # Verify that package lists aren't available. assert not have_package_lists() # Run `apt-get update' to download the package lists. updater.dumb_update() # Verify that package lists are again available. assert have_package_lists() def test_smart_update(self): """ Test that our smart ``apt-get update`` wrapper works. Currently this test simply ensures coverage of the happy path. Ideally it will evolve to test the handled edge cases as well. """ if os.getuid() != 0: return self.skipTest("root privileges required to opt in") updater = AptMirrorUpdater() # Remove all existing package lists. updater.clear_package_lists() # Verify that package lists aren't available. assert not have_package_lists() # Run `apt-get update' to download the package lists. updater.smart_update() # Verify that package lists are again available. assert have_package_lists() def test_discover_releases(self): """Test that release discovery works properly.""" releases = discover_releases() # Check that a reasonable number of Debian and Ubuntu releases was discovered. assert len([r for r in releases if r.distributor_id == 'debian']) > 10 assert len([r for r in releases if r.distributor_id == 'ubuntu']) > 10 assert len([r for r in releases if r.distributor_id == 'linuxmint']) > 10 # Check that LTS releases of Debian as well as Ubuntu were discovered. assert any(r.distributor_id == 'debian' and r.is_lts for r in releases) assert any(r.distributor_id == 'ubuntu' and r.is_lts for r in releases) assert any(r.distributor_id == 'linuxmint' and r.is_lts for r in releases) # Sanity check against duplicate releases. assert sum(r.series == 'bionic' for r in releases) == 1 assert sum(r.series == 'jessie' for r in releases) == 1 assert sum(r.series == 'tina' for r in releases) == 1 # Sanity check some known LTS releases. assert any(r.series == 'bionic' and r.is_lts for r in releases) assert any(r.series == 'stretch' and r.is_lts for r in releases) assert any(r.series == 'tina' and r.is_lts for r in releases) def test_discover_linuxmint_releases(self): """Test that release discovery works properly.""" import decimal from bs4 import BeautifulSoup from apt_smart.releases import discover_linuxmint_releases, table_to_2d from apt_smart.http import fetch_url from humanfriendly.terminal import output url = 'https://en.wikipedia.org/wiki/Linux_Mint_version_history' response = fetch_url(url, timeout=15, retry=True) soup = BeautifulSoup(response, 'html.parser') indent = " " * 4 releases = set() tables = soup.findAll('table') if not tables: raise Exception("Failed to locate
element in page %s" % url) else: output("\nBUNDLED_RELEASES = [\n") for release in discover_linuxmint_releases(table_to_2d(tables[1])): releases.add(release) output(indent + "Release(\n") for name in release.find_properties(cached=False): value = getattr(release, name) if value is not None: if isinstance(value, decimal.Decimal): # It seems weirdly inconsistency to me that this is needed # for decimal.Decimal() but not for datetime.date() but I # guess the simple explanation is that repr() output simply # isn't guaranteed to be accepted by eval(). value = "decimal." + repr(value) else: value = repr(value) output(indent * 2 + name + "=" + value + ",\n") output(indent + "),\n") output("]\n\n") # Check that a reasonable number of Debian and Ubuntu releases was discovered. assert len(releases) > 10 assert len([r for r in releases if r.distributor_id == 'linuxmint']) > 10 # Check that LTS releases of Debian as well as Ubuntu were discovered. assert any(r.distributor_id == 'linuxmint' and r.is_lts for r in releases) # Sanity check against duplicate releases. assert sum(r.series == 'tina' for r in releases) == 1 # Sanity check some known LTS releases. assert any(r.series == 'tina' and r.is_lts for r in releases) def test_coerce_release(self): """Test the coercion of release objects.""" # Test coercion of short code names. assert coerce_release('lucid').version == decimal.Decimal('10.04') assert coerce_release('tina').version == decimal.Decimal('19.2') assert coerce_release('woody').distributor_id == 'debian' assert coerce_release('tina').distributor_id == 'linuxmint' # Test coercion of version numbers. assert coerce_release('10.04').series == 'lucid' assert coerce_release('19.2').series == 'tina' def test_keyring_selection(self): """Make sure keyring selection works as intended.""" # Check Debian keyring selection. lenny = coerce_release('lenny') assert lenny.keyring_file == DEBIAN_KEYRING_CURRENT # Check Ubuntu <= 12.04 keyring selection. precise = coerce_release('precise') if ubuntu_keyring_updated(): assert precise.keyring_file == UBUNTU_KEYRING_REMOVED else: assert precise.keyring_file == UBUNTU_KEYRING_CURRENT # Check Ubuntu > 12.04 keyring selection. bionic = coerce_release('bionic') assert bionic.keyring_file == UBUNTU_KEYRING_CURRENT def test_debian_lts_eol_date(self): """ Regression test for `issue #5`_. .. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5 """ updater = AptMirrorUpdater( distributor_id='debian', distribution_codename='jessie', architecture='amd64', ) eol_expected = (time.time() >= 1593468000) assert updater.release_is_eol == eol_expected def test_trusty_eol(self): """ Test 'trusty' for `issue #9`_. .. _issue #9: https://github.com/xolox/python-apt-mirror-updater/issues/9 """ updater = AptMirrorUpdater( distributor_id='ubuntu', distribution_codename='trusty', architecture='amd64', ) assert updater.release_is_eol == (updater.validate_mirror(updater.old_releases_url) == MirrorStatus.AVAILABLE) def have_package_lists(): """ Check if apt's package lists are available. :returns: :data:`True` when package lists are available, :data:`False` otherwise. This function checks that the output of ``apt-cache show python`` contains a ``Filename: ...`` key/value pair which indicates that apt knows where to download the package archive that installs the ``python`` package. """ return 'Filename:' in execute('apt-cache', 'show', 'python', check=False, capture=True) def check_mirror_url(url): """Check whether the given URL looks like a Debian or Ubuntu mirror URL.""" if not (is_debian_mirror(url) or is_ubuntu_mirror(url)): msg = "Invalid mirror URL! (%r)" raise AssertionError(msg % url) def check_debian_mirror(url): """Ensure the given URL looks like a Debian mirror URL.""" if not is_debian_mirror(url): msg = "Invalid Debian mirror URL! (%r)" raise AssertionError(msg % url) def check_ubuntu_mirror(url): """Ensure the given URL looks like a Ubuntu mirror URL.""" if not is_ubuntu_mirror(url): msg = "Invalid Ubuntu mirror URL! (%r)" raise AssertionError(msg % url) def is_debian_mirror(url): """Check whether the given URL looks like a Debian mirror URL.""" url = normalize_mirror_url(url) if has_compatible_scheme(url): # Found a mirror ( http:// mirror.cs.unm.edu/archive ) ends with archive # and no 'ubuntu' in it, which break Travis CI checks, so make it return True return True # components = split(url, '/') # return components[-1] == 'debian' def is_ubuntu_mirror(url): """Check whether the given URL looks like a Ubuntu mirror URL.""" url = normalize_mirror_url(url) if has_compatible_scheme(url): # This function previously performed much more specific checks but in # 2018 the test suite started encountering a number of legitimate # mirror URLs that no longer passed the checks. As such this function # was dumbed down until nothing much remained :-P. # Found a mirror ( http:// mirror.cs.unm.edu/archive ) ends with archive # and no 'ubuntu' in it, which break Travis CI checks, so make it return True return True # return 'ubuntu' in url.lower() def has_compatible_scheme(url): """Check whether the given URL uses a scheme compatible with and intended to be used by apt.""" return url.startswith(('http://', 'https://')) ================================================ FILE: constraints.txt ================================================ # This is a pip constraints file that is used to preserve Python 2.6 # compatibility (on Travis CI). Why I'm still doing that in 2018 is # a good question, maybe simply to prove that I can :-P. # flake8 3.0.0 drops explicit support for Python 2.6: # http://flake8.pycqa.org/en/latest/release-notes/3.0.0.html flake8 < 3.0.0 ; python_version < '2.7' # flake8-docstrings 1.0.0 switches from pep257 to pydocstyle and I haven't been # able to find a combination of versions of flake8-docstrings and pydocstyle # that actually works on Python 2.6. Here's the changelog: # https://gitlab.com/pycqa/flake8-docstrings/blob/master/HISTORY.rst flake8-docstrings < 1.0.0 ; python_version < '2.7' # pyflakes 2.0.0 drops Python 2.6 compatibility: # https://github.com/PyCQA/pyflakes/blob/master/NEWS.txt pyflakes < 2.0.0 ; python_version < '2.7' # pytest 3.3 drops Python 2.6 compatibility: # https://docs.pytest.org/en/latest/changelog.html#pytest-3-3-0-2017-11-23 pytest < 3.3 ; python_version < '2.7' # pycparser < 2.19 drops Python 2.6 compatibility: # https://github.com/eliben/pycparser/blob/master/CHANGES pycparser < 2.19 ; python_version < '2.7' ================================================ FILE: docs/api.rst ================================================ API documentation ================= The following documentation is based on the source code of version |release| of the `apt-smart` package. The following modules are available: .. contents:: :local: :mod:`apt_smart` ------------------------- .. automodule:: apt_smart :members: :mod:`apt_smart.backends.debian` ----------------------------------------- .. automodule:: apt_smart.backends.debian :members: :mod:`apt_smart.backends.ubuntu` ----------------------------------------- .. automodule:: apt_smart.backends.ubuntu :members: :mod:`apt_smart.cli` ----------------------------- .. automodule:: apt_smart.cli :members: :mod:`apt_smart.http` ------------------------------ .. automodule:: apt_smart.http :members: :mod:`apt_smart.releases` ---------------------------------- .. automodule:: apt_smart.releases :members: ================================================ FILE: docs/changelog.rst ================================================ .. include:: ../CHANGELOG.rst ================================================ FILE: docs/conf.py ================================================ # Automated, robust apt-get mirror selection for Debian and Ubuntu. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """Sphinx documentation configuration for the `apt-smart` package.""" import os import sys # Add the 'apt-smart' source distribution's root directory to the module path. sys.path.insert(0, os.path.abspath(os.pardir)) # -- General configuration ----------------------------------------------------- # Sphinx extension module names. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'humanfriendly.sphinx', ] # Sort members by the source order instead of alphabetically. autodoc_member_order = 'bysource' # Paths that contain templates, relative to this directory. templates_path = ['templates'] # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = 'apt-smart' copyright = '2019 martin68, 2018 Peter Odding' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # Find the package version and make it the release. from apt_smart import __version__ as updater_version # noqa # The short X.Y version. version = '.'.join(updater_version.split('.')[:2]) # The full version, including alpha/beta/rc tags. release = updater_version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. language = 'en' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['build'] # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # Refer to the Python standard library. # From: http://twistedmatrix.com/trac/ticket/4582. intersphinx_mapping = dict( python2=('https://docs.python.org/2', None), python3=('https://docs.python.org/3', None), executor=('https://executor.readthedocs.io/en/latest/', None), propertymanager=('https://property-manager.readthedocs.io/en/latest/', None), ) # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'nature' # Output file base name for HTML help builder. htmlhelp_basename = 'aptmirrorupdaterdoc' ================================================ FILE: docs/index.rst ================================================ apt-smart: Smart, automated Debian/Ubuntu/Linux Mint mirror selection ======================================================================== Welcome to the documentation of `apt-smart` version |release|! Source code: https://github.com/martin68/apt-smart The following sections are available: .. contents:: :local: User documentation ------------------ The readme is the best place to start reading, it's targeted at all users and documents the command line interface: .. toctree:: readme.rst API documentation ----------------- The following API documentation is automatically generated from the source code: .. toctree:: api.rst Change log ---------- The change log lists notable changes to the project: .. toctree:: changelog.rst ================================================ FILE: docs/readme.rst ================================================ .. include:: ../README.rst ================================================ FILE: requirements-checks.txt ================================================ # Python packages required to run `make check'. flake8 >= 2.6.0 flake8-docstrings >= 0.2.8 pyflakes >= 1.2.3 ================================================ FILE: requirements-tests.txt ================================================ pytest >= 2.6.1 pytest-cov >= 2.2.1 ================================================ FILE: requirements-travis.txt ================================================ --requirement=requirements-checks.txt --requirement=requirements-tests.txt --requirement=requirements.txt coveralls virtualenv setuptools ================================================ FILE: requirements.txt ================================================ # Installation requirements for apt-smart. wheel >= 0.26.0 beautifulsoup4 >= 4.4.1 capturer >= 2.1.1 coloredlogs >= 5.0 executor >= 21.1.1 humanfriendly >= 4.17 property-manager >= 1.3; python_version < '3.3' property-manager3; python_version >= '3.3' six >= 1.10.0 stopit >= 1.1.1 ================================================ FILE: scripts/check-code-style.sh ================================================ #!/bin/bash -e # I value automated code style checks that break my Travis CI builds but I also # value compatibility with Python 2.6, however recently it seems that flake8 # has dropped Python 2.6 compatibility. That's only fair, but now I need to # work around it, hence this trivial script :-). # Python 2.7 also has an issue with flake8 now, so disable it for Python 2.7 if python -c 'import sys; sys.exit(0 if sys.version_info[:2] > (2, 7) else 1)'; then echo "Updating installation of flake8 .." >&2 pip install --upgrade --quiet --requirement=requirements-checks.txt flake8 else echo "Skipping code style checks on Python 2.7 .." >&2 fi ================================================ FILE: scripts/collect-full-coverage.sh ================================================ #!/bin/bash -e # This shell script is used by the makefile and Travis CI to run the # apt-smart test suite as root, allowing it to make changes # to the system that's running the test suite (one of my laptops # or a Travis CI worker). # Run the test suite with root privileges. sudo $(which py.test) --cov # Restore the ownership of the coverage data. sudo chown --reference="$PWD" --recursive $PWD # Update the HTML coverage overview. coverage html ================================================ FILE: scripts/install-on-travis.sh ================================================ #!/bin/bash -e # Install the required Python packages. pip install --requirement=requirements-travis.txt # For github actions python 3.12, install within virtualenv. /home/runner/.virtualenvs/apt-smart/bin/pip install --upgrade pip setuptools wheel # Install the project itself, making sure that potential character encoding # and/or decoding errors in the setup script are caught as soon as possible. LC_ALL=C pip install . # Let apt-get, dpkg and related tools know that we want the following # commands to be 100% automated (no interactive prompts). export DEBIAN_FRONTEND=noninteractive # Update apt-get's package lists. sudo -E apt-get update -qq # Make sure the /usr/share/distro-info/*.csv files are available, # this enables the test_gather_eol_dates() test. sudo -E apt-get install --yes distro-info-data ================================================ FILE: setup.cfg ================================================ # Enable universal wheels because `apt-smart' # is pure Python and works on Python 2 and 3 alike. [wheel] universal=1 ================================================ FILE: setup.py ================================================ #!/usr/bin/env python # Setup script for the `apt-smart' package. # # Author: martin68 and Peter Odding # Last Change: September 15, 2019 # URL: https://apt-smart.readthedocs.io """ Setup script for the `apt-smart` package. **python setup.py install** Install from the working directory into the current Python environment. **python setup.py sdist** Build a source distribution archive. **python setup.py bdist_wheel** Build a wheel distribution archive. """ # Standard library modules. import codecs import os import re import sys # De-facto standard solution for Python packaging. from setuptools import find_packages, setup def get_contents(*args): """Get the contents of a file relative to the source distribution directory.""" with codecs.open(get_absolute_path(*args), 'r', 'UTF-8') as handle: return handle.read() def get_version(*args): """Extract the version number from a Python module.""" contents = get_contents(*args) metadata = dict(re.findall('__([a-z]+)__ = [\'"]([^\'"]+)', contents)) return metadata['version'] def get_install_requires(): """Get the conditional dependencies for source distributions.""" install_requires = get_requirements('requirements.txt') if 'bdist_wheel' not in sys.argv: if sys.version_info[:2] == (2, 6): # flufl.enum 4.1 drops Python 2.6 compatibility. install_requires.append('flufl.enum >= 4.0.1, < 4.1') elif sys.version_info[:2] < (3, 4): install_requires.append('flufl.enum >= 4.0.1') return sorted(install_requires) def get_extras_require(): """Get the conditional dependencies for wheel distributions.""" extras_require = {} if have_environment_marker_support(): # flufl.enum 4.1 drops Python 2.6 compatibility. extras_require[':python_version == "2.6"'] = ['flufl.enum >= 4.0.1, < 4.1'] expression = ':%s' % ' or '.join([ 'python_version == "2.6"', 'python_version == "2.7"', 'python_version == "3.0"', 'python_version == "3.1"', 'python_version == "3.2"', 'python_version == "3.3"', ]) extras_require[expression] = ['flufl.enum >= 4.0.1'] return extras_require def get_requirements(*args): """Get requirements from pip requirement files.""" requirements = set() with open(get_absolute_path(*args)) as handle: for line in handle: # Strip comments. line = re.sub(r'^#.*|\s#.*', '', line) # Ignore empty lines if line and not line.isspace(): requirements.add(re.sub(r'\s+', '', line)) return sorted(requirements) def get_absolute_path(*args): """Transform relative pathnames into absolute pathnames.""" return os.path.join(os.path.dirname(os.path.abspath(__file__)), *args) def have_environment_marker_support(): """ Check whether setuptools has support for PEP-426 environment marker support. Based on the ``setup.py`` script of the ``pytest`` package: https://bitbucket.org/pytest-dev/pytest/src/default/setup.py """ try: from pkg_resources import parse_version from setuptools import __version__ return parse_version(__version__) >= parse_version('0.7.2') except Exception: return False setup( name='apt-smart', version=get_version('apt_smart', '__init__.py'), description="Automated, robust apt-get mirror selection for Debian and Ubuntu", long_description=get_contents('README.rst'), url='https://apt-smart.readthedocs.io', author='martin68', author_email=' ', license='MIT', packages=find_packages(), install_requires=get_install_requires(), extras_require=get_extras_require(), entry_points=dict(console_scripts=[ 'apt-smart = apt_smart.cli:main', ]), classifiers=[ 'Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'Intended Audience :: Information Technology', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: MIT License', 'Natural Language :: English', 'Operating System :: POSIX :: Linux', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Topic :: Software Development', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: System :: Shells', 'Topic :: System :: System Shells', 'Topic :: System :: Systems Administration', 'Topic :: Terminals', 'Topic :: Utilities', ]) ================================================ FILE: test_custom_mirrors.txt ================================================ http://mirrors.layeronline.com/ubuntu/ http://mirror.csclub.uwaterloo.ca/ubuntu/ http://muug.ca/mirror/ubuntu/ ================================================ FILE: tox.ini ================================================ [tox] envlist = py27, py34, py35, py36, py37, pypy [testenv] deps = -rrequirements-tests.txt commands = py.test {posargs} [pytest] addopts = --capture=no --verbose python_files = apt_smart/tests.py [flake8] exclude = .tox ignore = D211,D400,D401,D402,W503 max-line-length = 120