Showing preview only (232K chars total). Download the full file or copy to clipboard to get everything.
Repository: martin68/apt-smart
Branch: master
Commit: 870876dd6ba5
Files: 35
Total size: 219.2 KB
Directory structure:
gitextract_79k4xdni/
├── .github/
│ └── workflows/
│ └── python-ci.yml
├── .gitignore
├── .travis.yml
├── CHANGELOG.rst
├── LICENSE.txt
├── MANIFEST.in
├── Makefile
├── README-zh-cn.rst
├── README.rst
├── apt_smart/
│ ├── __init__.py
│ ├── backends/
│ │ ├── __init__.py
│ │ ├── debian.py
│ │ ├── linuxmint.py
│ │ └── ubuntu.py
│ ├── cli.py
│ ├── http.py
│ ├── releases.py
│ └── tests.py
├── constraints.txt
├── docs/
│ ├── api.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── index.rst
│ └── readme.rst
├── requirements-checks.txt
├── requirements-tests.txt
├── requirements-travis.txt
├── requirements.txt
├── scripts/
│ ├── check-code-style.sh
│ ├── collect-full-coverage.sh
│ └── install-on-travis.sh
├── setup.cfg
├── setup.py
├── test_custom_mirrors.txt
└── tox.ini
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/python-ci.yml
================================================
name: Python CI
on:
push:
branches-ignore:
- '/^[0-9]/'
pull_request:
branches-ignore:
- '/^[0-9]/'
jobs:
test:
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "pypy3.10"]
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: bash scripts/install-on-travis.sh
- name: Run checks
run: make check
- name: Run full coverage
run: make full-coverage
- name: Upload coverage to Coveralls
if: success()
uses: coverallsapp/github-action@v2
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
================================================
FILE: .gitignore
================================================
.cache/
.coverage
.coverage*
.directory
*.egg-info/
*.pyc
================================================
FILE: .travis.yml
================================================
sudo: true
language: python
python:
- "2.7"
- "3.4"
- "3.5"
- "3.6"
- "3.7"
- "3.8"
- "pypy"
install:
- scripts/install-on-travis.sh
script:
- make check
- make full-coverage
after_success:
- coveralls
branches:
except:
- /^[0-9]/
================================================
FILE: CHANGELOG.rst
================================================
Changelog
=========
The purpose of this document is to list all of the notable changes to this
project. The format was inspired by `Keep a Changelog`_. This project adheres
to `semantic versioning`_.
.. contents::
:local:
.. _Keep a Changelog: http://keepachangelog.com/
.. _semantic versioning: http://semver.org/
`Release 7.1.3`_ (2020-5-31)
-----------------------------
- Support 'mirror://' scheme: https://github.com/martin68/apt-smart/issues/3
- Update releases.py bundled Releases by running 'make releases', related https://github.com/martin68/apt-smart/issues/4
- In python2 decode() default encoding is ascii, causing https://github.com/martin68/apt-smart/issues/5 , specify utf-8
- Fix current_mirror in linuxmint's ubuntu mode, causing -U -c 'mirror_url' changed linuxmint's mirror_url instead of ubuntu's
.. _Release 7.1.3: https://github.com/martin68/apt-smart/compare/7.1.3...7.1.3
`Release 7.1.2`_ (2019-11-28)
-----------------------------
- Support Python 3.8
- Add `-C --codename` flag to create chroot with a distribution codename.
- Blacklist BASE_URL mirror if matches blacklist pattern, this helps when BASE_URL (official) mirror is the only up-to-date one and you find it so slow that you'd like to blacklist it.
.. _Release 7.1.2: https://github.com/martin68/apt-smart/compare/7.1.1...7.1.2
`Release 7.1.1`_ (2019-11-04)
-----------------------------
- For Linux Mint, backup official-package-repositories.list to backup_dir: backup_by_apt-smart
- In Readme, add install commands for Linux Mint and a note about run with sudo
.. _Release 7.1.1: https://github.com/martin68/apt-smart/compare/7.1.0...7.1.1
`Release 7.1.0`_ (2019-11-01)
-----------------------------
- Add support for Linux Mint
- Add -U, --ubuntu to opt in Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. --auto-change-mirror --ubuntu will auto-change Linux Mint's upstream Ubuntu mirror
.. _Release 7.1.0: https://github.com/martin68/apt-smart/compare/7.0.7...7.1.0
`Release 7.0.7`_ (2019-9-30)
----------------------------
- Fix install_sources_list() for Python 3
- Fix --change-mirror
- fix Travis CI `io.UnsupportedOperation:fileno` error by changing the way to run test cases containing smart_update()
- More test cases
.. _Release 7.0.7: https://github.com/martin68/apt-smart/compare/7.0.6...7.0.7
`Release 7.0.6`_ (2019-9-25)
----------------------------
- Readme & help about proxy setting
- Deal with the [options] in sources.list by stripping it from sources.list, and then get it back when generating new sources.list ,
fix https://github.com/jblakeman/apt-select/issues/54
- Add a warning: custom mirror file's path and filename must NOT contain whitespace
- Add -R, --create-chroot=local_dir_absolute_path to create chroot with the best mirror in a local directory with absolute_path
- More test cases
.. _Release 7.0.6: https://github.com/martin68/apt-smart/compare/7.0.5...7.0.6
`Release 7.0.5`_ (2019-9-21)
----------------------------
- Add -F, --file-to-read=local_file_absolute_path (path and filename must NOT contain whitespace) to read a local absolute path file containing custom mirror URLs (one URL per line) to add custom mirrors to rank.
So now you can use e.g. `-l -F ~/mirrors.txt` options to add some custom mirrors to rank with mirrors in official mirror list.
- Updated BUNDLED_RELEASES in releases.py
.. _Release 7.0.5: https://github.com/martin68/apt-smart/compare/7.0.4...7.0.5
`Release 7.0.4`_ (2019-9-20)
----------------------------
- Fix error on EOL release
- Check OLD_RELEASES_URL's MirrorStatus to confirm if it is EOL, to fix https://github.com/xolox/python-apt-mirror-updater/issues/9
.. _Release 7.0.4: https://github.com/martin68/apt-smart/compare/7.0.3...7.0.4
`Release 7.0.3`_ (2019-9-19)
----------------------------
- Fix `--url-char-len` option to specify the length of chars in mirrors' URL to display when using `--list-mirrors`,
so that now you can use e.g. `-l -L 29` options to narrow down the table of ranked mirrors when you want to paste it to somewhere the table displayed badly.
.. _Release 7.0.3: https://github.com/martin68/apt-smart/compare/7.0.2...7.0.3
`Release 7.0.2`_ (2019-9-19)
----------------------------
- Add :attr:url_char_len to specify the length of chars in mirrors' URL to display when using `--list-mirrors`,
so that now you can use e.g. `-l -L 29` options to narrow down the table of ranked mirrors when you want to paste it to somewhere the table displayed badly.
.. _Release 7.0.2: https://github.com/martin68/apt-smart/compare/7.0,1...7.0.2
`Release 7.0.1`_ (2019-9-18)
----------------------------
- Better output format when use `--list-mirrors`
.. _Release 7.0.1: https://github.com/martin68/apt-smart/compare/7.0...7.0.1
`Release 7.0`_ (2019-9-15)
---------------------------
- Rename the project and module to ``apt-smart``
- For Ubuntu, new mirrors discovery mechanism: at first it queries ``MIRROR_SELECTION_URL``, and ``MIRRORS_URL`` as fallback.
- For Debian, new mirrors discovery mechanism: get mirrors within the country which the user is in.
- New mechanism of determining whether a mirror is up-to-date: download the InRelease file and parse the ``Date`` value in it.
- New and more robust ``distribution_codename`` using APT sources.list
- Enable retry when ``fetch_url`` is timeout for bad connections.
- Drop Python 2.6 support and add Python 3.7
- Drop max_mirrors limit since we can smartly get mirrors within the user's country.
.. _Release 7.0: https://github.com/martin68/apt-smart/compare/6.1...7.0
`Release 6.1`_ (2018-10-19)
---------------------------
- Bug fix for Ubuntu keyring selection that prevented
``ubuntu-archive-removed-keys.gpg`` from being used.
- Bug fix for ``coerce_release()`` when given a release number.
- Moved pathnames of Debian and Ubuntu keyring files to constants.
- Added logging to enable debugging of keyring selection process.
- Added proper tests for keyring selection and release coercion.
.. _Release 6.1: https://github.com/xolox/python-apt-mirror-updater/compare/6.0...6.1
`Release 6.0`_ (2018-10-14)
---------------------------
Enable the creation of Ubuntu <= 12.04 chroots on Ubuntu >= 17.04 hosts by
working around (what I am convinced is) a bug in ``debootstrap`` which picks
the wrong keyring when setting up chroots of old releases. For more information
refer to issue `#8`_.
I've bumped the major version number for this release because the highly
specific ``apt_smart.eol`` module changed into the much more generic
``apt_smart.releases`` module. Also the ``release_label`` property was
removed.
.. _Release 6.0: https://github.com/xolox/python-apt-mirror-updater/compare/5.2...6.0
.. _#8: https://github.com/xolox/python-apt-mirror-updater/issues/8
`Release 5.2`_ (2018-10-08)
---------------------------
Use `mirrors.ubuntu.com/mirrors.txt`_ without placing our full trust in it like
older versions of ``apt-smart`` did 😇.
Feedback in issue `#6`_ suggested that `mirrors.ubuntu.com/mirrors.txt`_ is
working properly (again) and should be preferred over scraping Launchpad.
However I prefer for ``apt-smart`` to be a reliable "do what I mean"
program and `mirrors.ubuntu.com/mirrors.txt`_ has proven to be unreliable in
the past (see the discussion in `#6`_). As a compromise I've changed the Ubuntu
mirror discovery as follows:
1. Discover Ubuntu mirrors on Launchpad.
2. Try to discover mirrors using `mirrors.ubuntu.com/mirrors.txt`_ and iff
successful, narrow down the list produced in step 1 based on the URLs
reported in step 2.
3. Rank the discovered / narrowed down mirrors and pick the best one.
The reason why I've decided to add this additional complexity is because it has
bothered me in the past that Ubuntu mirror discovery was slow and this does
help a lot. Also, why not use a service provided by Ubuntu to speed things up?
Unrelated to the use of `mirrors.ubuntu.com/mirrors.txt`_ I've also bumped the
``executor`` requirement (twice) in order to pull in upstream improvements
discussed in `executor issue #10`_ and `executor issue #15`_.
.. _Release 5.2: https://github.com/xolox/python-apt-mirror-updater/compare/5.1...5.2
.. _mirrors.ubuntu.com/mirrors.txt: http://mirrors.ubuntu.com/mirrors.txt
.. _#6: https://github.com/xolox/python-apt-mirror-updater/issues/6
.. _executor issue #10: https://github.com/xolox/python-executor/issues/10
.. _executor issue #15: https://github.com/xolox/python-executor/issues/15
`Release 5.1`_ (2018-06-22)
---------------------------
Work on release 5.1 started with the intention of publishing a 5.0.2 bug fix
release for the EOL detection of Debian LTS releases reported in `#5`_, however
unrelated changes were required to stabilize the test suite. This explains how
5.0.2 became 5.1 😇.
When I started working on resolving the issue reported in `#5`_ it had been
quite a while since the previous release (233 days) and so some technical debt
had accumulated in the project, causing the test suite to break. Most
significantly, Travis CI switched their workers from Ubuntu 12.04 to 14.04.
Here's a detailed overview of changes:
- Bug fix for EOL detection of Debian LTS releases (reported in `#5`_).
- Bug fix for trivial string matching issue in test suite (caused by a naively
written test).
- Bug fix for recursive ``repr()`` calls potentially causing infinite
recursion, depending on logging level (see e.g. build 395421319_).
- Updated bundled EOL dates based on distro-info-data available in Ubuntu 18.04.
- Added this changelog to the documentation, including a link in the readme.
- Make sure the ``test_gather_eol_dates`` test method runs on Travis CI (by
installing the distro-info-data_ package). This exposed a Python 3
incompatibility (in build 395410569_) that has since been resolved.
- Include documentation in source distributions (``MANIFEST.in``).
- Silence flake8 complaining about bogus D402 issues.
- Add license='MIT' key to ``setup.py`` script.
- Bumped copyright to 2018.
.. _Release 5.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0.1...5.1
.. _#5: https://github.com/xolox/python-apt-mirror-updater/issues/5
.. _395421319: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319
.. _distro-info-data: https://packages.ubuntu.com/distro-info-data
.. _395410569: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395410569
`Release 5.0.1`_ (2017-11-01)
-----------------------------
Bug fix release for invalid enumeration value (oops).
.. _Release 5.0.1: https://github.com/xolox/python-apt-mirror-updater/compare/5.0...5.0.1
`Release 5.0`_ (2017-11-01)
---------------------------
Reliable end of life (EOL) detection.
Recently I ran into the issue that the logic to check whether a release is EOL
(that works by checking if the security mirror serves a ``Release.gpg`` file
for the release) failed on me. More specifically the following URL existed at
the time of writing (2017-11-01) even though Ubuntu 12.04 went EOL back in
April:
http://security.ubuntu.com/ubuntu/dists/precise/Release.gpg
At the same time issue `#1`_ and pull request `#2`_ were also indications that
the EOL detection was fragile and error prone. This potential fragility had
bugged me ever since publishing `apt-smart` and this week I finally
finished a more robust and deterministic EOL detection scheme.
This release includes pull requests `#2`_ and `#4`_, fixing issues `#1`_ and
`#3`_. Here's a detailed overview of changes:
- Addition: Allow optional arguments to ``apt-get update`` (`#3`_, `#4`_).
- I simplified and improved the feature requested in issue `#3`_ and
implemented in pull request `#4`_ by switching from an optional list
argument to 'star-args' and applying the same calling convention to
``smart_update()`` as well.
- This is backwards incompatible with the implementation in pull request
`#4`_ (which I merged into the ``dev`` branch but never published to PyPI)
and it's also technically backwards incompatible in the sense that keyword
arguments could previously be given to ``smart_update()`` as positional
arguments. This explains why I'm bumping the major version number.
- Bug fix for incorrect marking of EOL when HTTP connections fail (`#2`_).
- Refactoring: Apply timeout handling to HTTP response bodies.
- Refactoring: Distinguish 404 from other HTTP errors:
- This change enhances ``validate_mirror()`` by making a distinction between
a confirmed HTTP 404 response versus other error conditions which may be of
a more transient nature.
- The goal of this change is to preserve the semantics requested in issue
`#1`_ and implemented in pull request `#2`_ without needing the additional
HTTP request performed by ``can_connect_to_mirror()``.
- Because ``validate_mirror()`` previously returned a boolean but now returns
an enumeration member this change is technically backwards incompatible,
then again ``validate_mirror()`` isn't specifically intended for callers
because it concerns internal logic of apt-smart. I'm nevertheless
bumping the major version number.
- Refactoring: Improve HTTP request exception handling:
- 404 responses and timeouts are no longer subject to retrying.
- The exception ``apt_smart.http.NotFoundError`` is now raised on
HTTP 404 responses. Other unexpected HTTP response codes raise
``apt_smart.http.InvalidResponseError``.
- The specific distinction between 404 and !200 was made because the 404
response has become significant in checking for EOL status.
.. _Release 5.0: https://github.com/xolox/python-apt-mirror-updater/compare/4.0...5.0
.. _#1: https://github.com/xolox/python-apt-mirror-updater/issues/1
.. _#2: https://github.com/xolox/python-apt-mirror-updater/pull/2
.. _#3: https://github.com/xolox/python-apt-mirror-updater/issues/3
.. _#4: https://github.com/xolox/python-apt-mirror-updater/pull/4
`Release 4.0`_ (2017-06-14)
---------------------------
Robust validation of available mirrors (backwards incompatible).
.. _Release 4.0: https://github.com/xolox/python-apt-mirror-updater/compare/3.1...4.0
`Release 3.1`_ (2017-06-13)
---------------------------
Made mirror comparison more robust.
.. _Release 3.1: https://github.com/xolox/python-apt-mirror-updater/compare/3.0...3.1
`Release 3.0`_ (2017-06-13)
---------------------------
Added Debian archive support (with old releases):
- Addition: Added Debian archive support (old releases).
- Improvement: Don't bother validating archive / old-releases mirror.
- Refactoring: Moved URLs to backend specific modules.
.. _Release 3.0: https://github.com/xolox/python-apt-mirror-updater/compare/2.1...3.0
`Release 2.1`_ (2017-06-12)
---------------------------
Restored Python 3 compatibility, improved robustness:
- Improvement: Make the ``is_available`` and ``is_updating`` properties of the
``CandidateMirror`` class more robust.
- Bug fix: I suck at Unicode in Python (most people do :-p).
- Cleanup: Remove unused import from test suite.
.. _Release 2.1: https://github.com/xolox/python-apt-mirror-updater/compare/2.0...2.1
`Release 2.0`_ (2017-06-11)
---------------------------
Generation of ``sources.list`` files and chroot creation.
Detailed overview of changes:
- Addition: Added a simple ``debootstrap`` wrapper.
- Addition: Programmatic /etc/apt/sources.list generation
- Bug fix for ``check_suite_available()``.
- Bug fix: Never apply Ubuntu's old release handling to Debian.
- Bug fix: Never remove ``/var/lib/apt/lists/lock file``.
- Improvement: Enable stable mirror selection
- Improvement: Make it possible to override distributor ID and codename
- Improvement: Render interactive spinner during mirror ranking.
- Refactoring: Generalize AptMirrorUpdater initializer (backwards incompatible!)
- Refactoring: Generalize backend module loading
- Refactoring: Modularize ``/etc/apt/sources.list`` writing.
.. _Release 2.0: https://github.com/xolox/python-apt-mirror-updater/compare/1.0...2.0
`Release 1.0`_ (2017-06-08)
---------------------------
Improved Ubuntu mirror discovery, added an automated test suite, and more.
The bump to version 1.0 isn't so much intended to communicate that this
is now mature software, it's just that I made several backwards
incompatible changes in order to improve the modularity of the code
base, make it easier to develop automated tests, maintain platform
support, etc :-).
A more detailed overview of (significant) changes:
- Improved Ubuntu mirror discovery (by scraping Launchpad instead).
- Extracted mirror discovery to separate (backend specific) modules.
- Extracted HTTP handling to a separate module.
- Enable Control-C to interrupt concurrent connection tests.
- Expose limit in Python API and command line interface and make limit optional by passing 0.
- Bug fix for Python 3 incompatibility: Stop using ``sys.maxint`` :-).
.. _Release 1.0: https://github.com/xolox/python-apt-mirror-updater/compare/0.3.1...1.0
`Release 0.3.1`_ (2016-06-29)
-----------------------------
Avoid 'nested' smart updates (the old code worked fine but gave confusing
output and performed more work than necessary, which bothered me :-).
.. _Release 0.3.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.3...0.3.1
`Release 0.3`_ (2016-06-29)
---------------------------
Make smart update understand EOL suites
.. _Release 0.3: https://github.com/xolox/python-apt-mirror-updater/compare/0.2...0.3
`Release 0.2`_ (2016-06-29)
---------------------------
Bug fix: Replace ``security.ubuntu.com`` as well.
.. _Release 0.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.2...0.2
`Release 0.1.2`_ (2016-06-29)
-----------------------------
Bug fix: Explicitly terminate multiprocessing pool.
.. _Release 0.1.2: https://github.com/xolox/python-apt-mirror-updater/compare/0.1.1...0.1.2
`Release 0.1.1`_ (2016-03-10)
-----------------------------
Initial release (added ``MANIFEST.in``).
.. _Release 0.1.1: https://github.com/xolox/python-apt-mirror-updater/compare/0.1...0.1.1
`Release 0.1`_ (2016-03-10)
---------------------------
Initial commit.
.. _Release 0.1: https://github.com/xolox/python-apt-mirror-updater/tree/0.1
================================================
FILE: LICENSE.txt
================================================
Copyright (c) 2019 martin68 2018 Peter Odding
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: MANIFEST.in
================================================
include *.rst
include *.txt
graft docs
================================================
FILE: Makefile
================================================
# Makefile for the 'apt-smart' package.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
PACKAGE_NAME = apt-smart
WORKON_HOME ?= $(HOME)/.virtualenvs
VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME)
PATH := $(VIRTUAL_ENV)/bin:$(PATH)
MAKE := $(MAKE) --no-print-directory
SHELL = bash
default:
@echo "Makefile for $(PACKAGE_NAME)"
@echo
@echo 'Usage:'
@echo
@echo ' make install install the package in a virtual environment'
@echo ' make reset recreate the virtual environment'
@echo ' make check check coding style (PEP-8, PEP-257)'
@echo ' make test run the test suite, report coverage'
@echo ' make tox run the tests on all Python versions'
@echo ' make eol update apt_smart.eol module'
@echo ' make readme update usage in readme'
@echo ' make docs update documentation using Sphinx'
@echo ' make publish publish changes to GitHub/PyPI'
@echo ' make clean cleanup all temporary files'
@echo
install:
@test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)"
@test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --quiet "$(VIRTUAL_ENV)"
@test -x "$(VIRTUAL_ENV)/bin/pip" || easy_install pip
@test -x "$(VIRTUAL_ENV)/bin/pip" || pip install --quiet pip
@pip install --quiet --requirement=requirements.txt
@"$(VIRTUAL_ENV)/bin/pip" install --upgrade pip setuptools wheel
@pip install --quiet flufl.enum
@pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true
@pip install --quiet --no-deps --ignore-installed .
reset:
$(MAKE) clean
rm -Rf "$(VIRTUAL_ENV)"
$(MAKE) install
check: install
@scripts/check-code-style.sh
test: install
@pip install --quiet --requirement=requirements-tests.txt
@py.test --cov
@coverage html
tox: install
@pip install --quiet tox && tox
# The following makefile target isn't documented on purpose, I don't want
# people to execute this without them knowing very well what they're doing.
full-coverage: install
@pip install --quiet --requirement=requirements-tests.txt
@scripts/collect-full-coverage.sh
@coverage html
cog: install
@echo Installing cog ...
@pip install --quiet cogapp
releases: cog
@cog.py -r apt_smart/releases.py
readme: cog
@cog.py -r README*.rst
docs: releases readme
@pip install --quiet sphinx
@cd docs && sphinx-build -nb html -d build/doctrees . build/html
publish: install
git push origin && git push --tags origin
$(MAKE) clean
pip install --quiet twine wheel
python setup.py sdist bdist_wheel
twine upload dist/*
$(MAKE) clean
clean:
@rm -Rf *.egg .cache .coverage .coverage.* .tox build dist docs/build htmlcov
@find -depth -type d -name __pycache__ -exec rm -Rf {} \;
@find -type f -name '*.pyc' -delete
.PHONY: default install reset check test tox readme docs publish clean
================================================
FILE: README-zh-cn.rst
================================================
apt-smart: 智能的 Debian/Ubuntu/Linux Mint 镜像源自动选择工具
=================================================================
.. image:: https://travis-ci.org/martin68/apt-smart.svg?branch=master
:target: https://travis-ci.org/martin68/apt-smart
.. image:: https://coveralls.io/repos/martin68/apt-smart/badge.svg?branch=master
:target: https://coveralls.io/r/martin68/apt-smart?branch=master
`apt-smart` 提供健壮的 Debian_ 、 Ubuntu_ 和 `Linux Mint`_ apt-get_ 镜像源 (也称软件源) 自动选择。
它能智能发现镜像源、排序镜像源并且自动切换，以及实现健壮的包列表更新 (参见 features_). 目前在 Python 2.7, 3.4, 3.5,
3.6, 3.7, 3.8 和 PyPy 测试通过 (尽管test coverage目前还很低，参见 status_).
.. contents::
:local:
Why?
--------
作为 `apt-mirror-updater <https://github.com/xolox/python-apt-mirror-updater>`_ 的继承者,
为你寻找最好镜像源的过程中 `apt-smart` 的智能、速度、准确性和健壮性方面都有提升和改进 (参见 changelog_)。
并且有计划增加反向代理模式——在设置好之后你就可以忘掉它，它在后台运行不需要root权限，在任何时候都指向最好的镜像源。
其他发行版如 Linux Mint（已完成！）， ROS等的支持也在计划之中。
.. _features:
Features
--------
**智能发现可用的镜像源**
通过查询 `Debian mirror list <https://www.debian.org/mirror/list>`_ 或 `Ubuntu
mirror list1 <http://mirrors.ubuntu.com/mirrors.txt>`_ 或 `Ubuntu
mirror list2 <https://launchpad.net/ubuntu/+archivemirrors>`_ 或 `Linux Mint mirror list <https://linuxmint.com/mirrors.php>`_ (自动选择镜像源列表)来
自动查找 Debian_ 、 Ubuntu_ 和 `Linux Mint`_ 镜像源。它能够智能地获取用户所在国家的镜像源。
**智能排序可用的镜像源**
可用镜像源按照如下方式排序：带宽、是否更新及时（up-to-date），并且排除了正在更新的镜像源 (参见 `issues with mirror updates`_)。
例如使用 `--list-mirrors` 参数将会有类似输出：
.. code-block:: sh
-----------------------------------------------------------------------------------------------------
| Rank | Mirror URL | Available? | Updating? | Last updated | Bandwidth |
-----------------------------------------------------------------------------------------------------
| 1 | http://archive.ubuntu.com/ubuntu | Yes | No | Up to date | 16.95 KB/s |
| 2 | http://mirrors.cqu.edu.cn/ubuntu | Yes | No | 3 hours behind | 427.43 KB/s |
| 3 | http://mirrors.nju.edu.cn/ubuntu | Yes | No | 5 hours behind | 643.27 KB/s |
| 4 | http://mirrors.tuna.tsinghua.e...| Yes | No | 5 hours behind | 440.09 KB/s |
| 5 | http://mirrors.cn99.com/ubuntu | Yes | No | 13 hours behind | 2.64 MB/s |
| 6 | http://mirrors.huaweicloud.com...| Yes | No | 13 hours behind | 532.01 KB/s |
| 7 | http://mirrors.dgut.edu.cn/ubuntu| Yes | No | 13 hours behind | 328.25 KB/s |
| 8 | http://mirrors.aliyun.com/ubuntu | Yes | No | 23 hours behind | 1.06 MB/s |
| 9 | http://ftp.sjtu.edu.cn/ubuntu | Yes | No | 23 hours behind | 647.2 KB/s |
| 10 | http://mirrors.yun-idc.com/ubuntu| Yes | No | 23 hours behind | 526.6 KB/s |
| 11 | http://mirror.lzu.edu.cn/ubuntu | Yes | No | 23 hours behind | 210.99 KB/s |
| 12 | http://mirrors.ustc.edu.cn/ubuntu| Yes | Yes | 8 hours behind | 455.02 KB/s |
| 13 | http://mirrors.sohu.com/ubuntu | No | No | Unknown | 90.28 bytes/s |
-----------------------------------------------------------------------------------------------------
**自动切换镜像源**
设置在 ``/etc/apt/sources.list`` 的镜像源可以用一条很简单的命令更改。你可以让它自动选择镜像源或者由你指定。
**健壮的包列表更新**
好几个 apt-get_ 的子命令在更新的过程中可能会失败 (参见 `issues with mirror updates`_) ，而`apt-smart` 通过
wrap ``apt-get update`` 可以在检测到错误时重试，并且在检测到当前镜像源在“更新中“时自动切换至另一个镜像
(因为曾经出现过“更新中“的状态持续很长时间，这有时是不可接受的，特别是在自动化维护的时候)
.. _status:
Status
------
尽管设置了自动测试，但项目还处于早期状态，所以:
.. warning:: `apt-smart` 弄坏你的系统可别怪我没警告过你哦! 但碰上最糟糕的情况恐怕也只有
``/etc/apt/sources.list`` 损坏了吧。程序在做任何更改前都会自动备份一次，
所以应该能手动恢复。
Installation
------------
`apt-smart` 可以在 PyPI_ 上找到，所以安装应该很简单（把下面命令全部一次性粘贴到终端窗口）:
.. code-block:: sh
sudo apt update
sudo apt install python-pip python-setuptools python-wheel -y # 不询问直接安装python-pip等依赖
pip install --user apt-smart # --user参数表示安装到per user site-packages directory
echo "export PATH=\$(python -c 'import site; print(site.USER_BASE + \"/bin\")'):\$PATH" >> ~/.bashrc
source ~/.bashrc # 设置per user site-packages directory到PATH
安装 Python 包有几种方法 (例如 `per user site-packages directory`_, 或 `virtual environments`_ 或 安装到系统全局)
在这里不详细展开叙述。
如果 apt-smart 有新版本发布了， 你可以通过如下命令升级：
.. code-block:: sh
pip install --user apt-smart --upgrade
**注意** ： apt-smart 是个 APT 的助手工具，而 **不是** apt/apt-get 命令的替代，所以通常 apt-smart 不应该用 ``sudo`` 或以 ``su`` 运行，
如果 apt-smart 需要 root 最高权限以继续（例如更改 sources.list），它会让用户输入密码。
Usage
-----
使用 `apt-smart` 有两种方法: 作为命令行工具 ``apt-smart`` 以及作为 Python API.
作为 Python API 的详细信息请参考文档—— `Read the Docs`_.
其命令行接口如下所示：
.. contents::
:local:
.. A DRY solution to avoid duplication of the `apt-smart --help' text:
..
.. [[[cog
.. from humanfriendly.usage import inject_usage
.. inject_usage('apt_smart.cli')
.. ]]]
**Usage:** `apt-smart [OPTIONS]`
The apt-smart program automates robust apt-get mirror selection for
Debian and Ubuntu by enabling discovery of available mirrors, ranking of
available mirrors, automatic switching between mirrors and robust package list
updating.
**Supported options:**
.. csv-table::
:header: Option, Description
:widths: 30, 70
"``-r``, ``--remote-host=SSH_ALIAS``","Operate on a remote system instead of the local system. The ``SSH_ALIAS``
argument gives the SSH alias of the remote host. It is assumed that the
remote account has root privileges or password-less sudo access."
"``-f``, ``--find-current-mirror``","Determine the main mirror that is currently configured in
/etc/apt/sources.list and report its URL on standard output."
"``-F``, ``--file-to-read=local_file_absolute_path``","Read a local absolute path (path and filename must NOT contain whitespace) file
containing custom mirror URLs (one URL per line) to add custom mirrors to rank."
"``-b``, ``--find-best-mirror``","Discover available mirrors, rank them, select the best one and report its
URL on standard output."
"``-l``, ``--list-mirrors``",List available (ranked) mirrors on the terminal in a human readable format.
"``-L``, ``--url-char-len=int``","An integer to specify the length of chars in mirrors' URL to display when
using ``--list-mirrors``, default is 34"
"``-c``, ``--change-mirror=MIRROR_URL``",Update /etc/apt/sources.list to use the given ``MIRROR_URL``.
"``-a``, ``--auto-change-mirror``","Discover available mirrors, rank the mirrors by connection speed and update
status and update /etc/apt/sources.list to use the best available mirror."
"``-u``, ``--update``, ``--update-package-lists``","Update the package lists using ""apt-get update"", retrying on failure and
automatically switch to a different mirror when it looks like the current
mirror is being updated."
"``-U``, ``--ubuntu``","Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror.
e.g. ``--auto-change-mirror`` ``--ubuntu`` will auto-change Linux Mint's upstream Ubuntu mirror"
"``-x``, ``--exclude=PATTERN``","Add a pattern to the mirror selection blacklist. ``PATTERN`` is expected to be
a shell pattern (containing wild cards like ""?"" and ""\*"") that is matched
against the full URL of each mirror."
"``-v``, ``--verbose``",Increase logging verbosity (can be repeated).
"``-V``, ``--version``",Show version number and Python version.
"``-R``, ``--create-chroot=local_dir_absolute_path``",Create chroot with the best mirror in a local directory with absolute_path
"``-q``, ``--quiet``",Decrease logging verbosity (can be repeated).
"``-h``, ``--help``"," Show this message and exit.
Note: since apt-smart uses `urlopen` method in The Python Standard Library,
you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type:
export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080'
These will not persist however (no longer active after you close the terminal),
so you may wish to add the line to your ~/.bashrc"
.. [[[end]]]
.. _issues with mirror updates:
Issues with mirror updates
--------------------------
最常见的 ``apt-get update`` 错误是 'hash sum mismatch' (参见 `Debian bug #624122`_)。
当错误产生的时候，一个名为 ``Archive-Update-in-Progress-*`` 的文件有时会出现
该镜像源的首页 (参见 `Debian bug #110837`_). 这个状态有时会持续很长时间。
My working theory about these 'hash sum mismatch' errors is that they are
caused by the fact that mirror updates aren't atomic, apparently causing
``apt-get update`` to download a package list whose datafiles aren't consistent
with each other. If this assumption proves to be correct (and also assuming
that different mirrors are updated at different times :-) then the command
``apt-smart --update-package-lists`` should work around this annoying
failure mode (by automatically switching to a different mirror when 'hash sum
mismatch' errors are encountered).
Publishing `apt-smart` to the world is my attempt to contribute to
this situation instead of complaining in bug trackers (see above) where no
robust and automated solution is emerging (at the time of writing). Who knows,
maybe some day these issues will be resolved by moving logic similar to what
I've implemented here into ``apt-get`` itself. Of course it would also help if
mirror updates were atomic...
Contact
-------
The latest version of `apt-smart` is available on PyPI_ and GitHub_.
The documentation is hosted on `Read the Docs`_ and includes a changelog_. For
bug reports please create an issue on GitHub_.
License
-------
This software is licensed under the `MIT license`_.
© 2020 martin68
© 2018 Peter Odding.
.. External references:
.. _apt-get: https://en.wikipedia.org/wiki/Advanced_Packaging_Tool
.. _at work: http://www.paylogic.com/
.. _changelog: https://apt-smart.readthedocs.io/en/latest/changelog.html
.. _Debian bug #110837: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=110837
.. _Debian bug #624122: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624122
.. _Debian: https://en.wikipedia.org/wiki/Debian
.. _documentation: https://apt-smart.readthedocs.io
.. _GitHub: https://github.com/martin68/apt-smart
.. _Linux Mint: https://linuxmint.com
.. _MIT license: http://en.wikipedia.org/wiki/MIT_License
.. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/
.. _PyPI: https://pypi.python.org/pypi/apt-smart
.. _Read the Docs: https://apt-smart.readthedocs.io
.. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system)
.. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/
================================================
FILE: README.rst
================================================
apt-smart: Smart, automated Debian/Ubuntu/Linux Mint mirror selection
=====================================================================
.. image:: https://travis-ci.org/martin68/apt-smart.svg?branch=master
:target: https://travis-ci.org/martin68/apt-smart
.. image:: https://coveralls.io/repos/martin68/apt-smart/badge.svg?branch=master
:target: https://coveralls.io/r/martin68/apt-smart?branch=master
`简体中文 <https://github.com/martin68/apt-smart/blob/master/README-zh-cn.rst>`_
The `apt-smart` package automates robust apt-get_ mirror (a.k.a Repositories, Sources) selection for
Debian_ , Ubuntu_ and `Linux Mint`_ by enabling smart discovery of available mirrors, smart ranking of
available mirrors, automatic switching between mirrors and robust package list
updating (see features_). It's currently tested on Python 2.7, 3.4, 3.5,
3.6, 3.7, 3.8 and PyPy (although test coverage is still rather low, see status_).
.. contents::
:local:
Why?
--------
As a successor of `apt-mirror-updater <https://github.com/xolox/python-apt-mirror-updater>`_,
`apt-smart` has many improvements in intelligence, speed, accuracy and robustness (see changelog_) when offering the best mirror for you.
It has a plan to optionally be a set-and-forget smart daemon: running in the background as a reverse proxy
always redirecting to the best mirror without root privilege. It also has a plan to support other distros like: Linux Mint (Done!) , ROS...
.. _features:
Features
--------
**Smart discovery of available mirrors**
Debian_ , Ubuntu_ and `Linux Mint`_ mirrors are discovered automatically by querying the
`Debian mirror list <https://www.debian.org/mirror/list>`_ or the `Ubuntu
mirror list1 <http://mirrors.ubuntu.com/mirrors.txt>`_ or the `Ubuntu
mirror list2 <https://launchpad.net/ubuntu/+archivemirrors>`_ or the `Linux Mint mirror list <https://linuxmint.com/mirrors.php>`_ (the applicable
mirror list is automatically selected based on the current platform).
It can smartly get mirrors within the country which the user is in.
**Smart ranking of available mirrors**
Discovered mirrors are ranked by bandwidth (to pick the fastest mirror) and whether they're up-to-date and
excluded if they're being updated (see `issues with mirror updates`_). e.g. with `--list-mirrors` flag it would output like this:
.. code-block:: sh
-----------------------------------------------------------------------------------------------------
| Rank | Mirror URL | Available? | Updating? | Last updated | Bandwidth |
-----------------------------------------------------------------------------------------------------
| 1 | http://archive.ubuntu.com/ubuntu | Yes | No | Up to date | 16.95 KB/s |
| 2 | http://mirrors.cqu.edu.cn/ubuntu | Yes | No | 3 hours behind | 427.43 KB/s |
| 3 | http://mirrors.nju.edu.cn/ubuntu | Yes | No | 5 hours behind | 643.27 KB/s |
| 4 | http://mirrors.tuna.tsinghua.e...| Yes | No | 5 hours behind | 440.09 KB/s |
| 5 | http://mirrors.cn99.com/ubuntu | Yes | No | 13 hours behind | 2.64 MB/s |
| 6 | http://mirrors.huaweicloud.com...| Yes | No | 13 hours behind | 532.01 KB/s |
| 7 | http://mirrors.dgut.edu.cn/ubuntu| Yes | No | 13 hours behind | 328.25 KB/s |
| 8 | http://mirrors.aliyun.com/ubuntu | Yes | No | 23 hours behind | 1.06 MB/s |
| 9 | http://ftp.sjtu.edu.cn/ubuntu | Yes | No | 23 hours behind | 647.2 KB/s |
| 10 | http://mirrors.yun-idc.com/ubuntu| Yes | No | 23 hours behind | 526.6 KB/s |
| 11 | http://mirror.lzu.edu.cn/ubuntu | Yes | No | 23 hours behind | 210.99 KB/s |
| 12 | http://mirrors.ustc.edu.cn/ubuntu| Yes | Yes | 8 hours behind | 455.02 KB/s |
| 13 | http://mirrors.sohu.com/ubuntu | No | No | Unknown | 90.28 bytes/s |
-----------------------------------------------------------------------------------------------------
**Automatic switching between mirrors**
The main mirror configured in ``/etc/apt/sources.list`` can be changed with a
single command. The new (to be configured) mirror can be selected
automatically or configured explicitly by the user.
**Robust package list updating**
Several apt-get_ subcommands can fail if the current mirror is being updated
(see `issues with mirror updates`_) and `apt-smart` tries to work
around this by wrapping ``apt-get update`` to retry on failures and
automatically switch to a different mirror when it looks like the current
mirror is being updated (because I've seen such updates take more than 15
minutes and it's not always acceptable to wait for so long, especially in
automated solutions).
.. _status:
Status
------
On the one hand the `apt-smart` package was developed based on quite a
few years of experience in using apt-get_ on Debian_ and Ubuntu_ systems. On the
other hand the Python package itself is relatively new: it was developed and
published in Sep 2019. As such:
.. warning:: Until `apt-smart` has been rigorously tested I consider
it a proof of concept (beta software) so if it corrupts your
system you can't complain that you weren't warned! The worst that can happen
(assuming you trust my judgement ;-) is that
``/etc/apt/sources.list`` is corrupted however a backup copy is
made before any changes are applied, so I don't see how this can
result in irreversible corruption.
I'm working on an automated test suite but at the moment I'm still a bit fuzzy
on how to create representative tests for the error handling code paths (also,
writing a decent test suite requires a significant chunk of time :-).
Installation
------------
The `apt-smart` package is available on PyPI_ which means installation
should be as simple as (paste all below commands together into terminal):
.. code-block:: sh
sudo apt update
sudo apt install python-pip python-setuptools python-wheel -y # install python-pip and so on without asking
pip install --user apt-smart # --user flag means install to per user site-packages directory(see below)
echo "export PATH=\$(python -c 'import site; print(site.USER_BASE + \"/bin\")'):\$PATH" >> ~/.bashrc
source ~/.bashrc # set per user site-packages directory to PATH
There's actually a multitude of ways to install Python packages (e.g. the `per
user site-packages directory`_, `virtual environments`_ or just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions ;-).
If a new version of apt-smart has been released, you can upgrade it via:
.. code-block:: sh
pip install --user apt-smart --upgrade
**Note**. ``apt-smart`` is a *helper* for the ``apt`` tool. It is **NOT** a
*replacement* for ``apt`` (or for ``apt-get``). So, ``apt-smart`` should
*not* be run *instead* of either of those commands. Nor should
``apt-smart`` be run with ``sudo`` or via ``su``; if ``apt-smart``
happens to need root privilege in order for it to continue (in order
that it may, for example, change ``sources.list``), then it will prompt
for a password.
Usage
-----
There are two ways to use the `apt-smart` package: As the command line
program ``apt-smart`` and as a Python API. For details about the
Python API please refer to the API documentation available on `Read the Docs`_.
The command line interface is described below.
.. contents::
:local:
.. A DRY solution to avoid duplication of the `apt-smart --help' text:
..
.. [[[cog
.. from humanfriendly.usage import inject_usage
.. inject_usage('apt_smart.cli')
.. ]]]
**Usage:** `apt-smart [OPTIONS]`
The apt-smart program automates robust apt-get mirror selection for
Debian and Ubuntu by enabling discovery of available mirrors, ranking of
available mirrors, automatic switching between mirrors and robust package list
updating.
**Supported options:**
.. csv-table::
:header: Option, Description
:widths: 30, 70
"``-r``, ``--remote-host=SSH_ALIAS``","Operate on a remote system instead of the local system. The ``SSH_ALIAS``
argument gives the SSH alias of the remote host. It is assumed that the
remote account has root privileges or password-less sudo access."
"``-f``, ``--find-current-mirror``","Determine the main mirror that is currently configured in
/etc/apt/sources.list and report its URL on standard output."
"``-F``, ``--file-to-read=local_file_absolute_path``","Read a local absolute path (path and filename must NOT contain whitespace) file
containing custom mirror URLs (one URL per line) to add custom mirrors to rank."
"``-b``, ``--find-best-mirror``","Discover available mirrors, rank them, select the best one and report its
URL on standard output."
"``-l``, ``--list-mirrors``",List available (ranked) mirrors on the terminal in a human readable format.
"``-L``, ``--url-char-len=int``","An integer to specify the length of chars in mirrors' URL to display when
using ``--list-mirrors``, default is 34"
"``-c``, ``--change-mirror=MIRROR_URL``",Update /etc/apt/sources.list to use the given ``MIRROR_URL``.
"``-a``, ``--auto-change-mirror``","Discover available mirrors, rank the mirrors by connection speed and update
status and update /etc/apt/sources.list to use the best available mirror."
"``-u``, ``--update``, ``--update-package-lists``","Update the package lists using ""apt-get update"", retrying on failure and
automatically switch to a different mirror when it looks like the current
mirror is being updated."
"``-U``, ``--ubuntu``","Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror.
e.g. ``--auto-change-mirror`` ``--ubuntu`` will auto-change Linux Mint's upstream Ubuntu mirror"
"``-x``, ``--exclude=PATTERN``","Add a pattern to the mirror selection blacklist. ``PATTERN`` is expected to be
a shell pattern (containing wild cards like ""?"" and ""\*"") that is matched
against the full URL of each mirror."
"``-v``, ``--verbose``",Increase logging verbosity (can be repeated).
"``-V``, ``--version``",Show version number and Python version.
"``-R``, ``--create-chroot=local_dir_absolute_path``",Create chroot with the best mirror in a local directory with absolute_path
"``-q``, ``--quiet``",Decrease logging verbosity (can be repeated).
"``-h``, ``--help``"," Show this message and exit.
Note: since apt-smart uses `urlopen` method in The Python Standard Library,
you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type:
export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080'
These will not persist however (no longer active after you close the terminal),
so you may wish to add the line to your ~/.bashrc"
.. [[[end]]]
.. _issues with mirror updates:
Issues with mirror updates
--------------------------
The most frequent failure that we run into is ``apt-get update`` crapping out
with 'hash sum mismatch' errors (see also `Debian bug #624122`_). When this
happens a file called ``Archive-Update-in-Progress-*`` can sometimes be found
on the index page of the mirror that is being used (see also `Debian bug
#110837`_). I've seen these situations last for more than 15 minutes.
My working theory about these 'hash sum mismatch' errors is that they are
caused by the fact that mirror updates aren't atomic, apparently causing
``apt-get update`` to download a package list whose datafiles aren't consistent
with each other. If this assumption proves to be correct (and also assuming
that different mirrors are updated at different times :-) then the command
``apt-smart --update-package-lists`` should work around this annoying
failure mode (by automatically switching to a different mirror when 'hash sum
mismatch' errors are encountered).
Publishing `apt-smart` to the world is my attempt to contribute to
this situation instead of complaining in bug trackers (see above) where no
robust and automated solution is emerging (at the time of writing). Who knows,
maybe some day these issues will be resolved by moving logic similar to what
I've implemented here into ``apt-get`` itself. Of course it would also help if
mirror updates were atomic...
Contact
-------
The latest version of `apt-smart` is available on PyPI_ and GitHub_.
The documentation is hosted on `Read the Docs`_ and includes a changelog_. For
bug reports please create an issue on GitHub_.
License
-------
This software is licensed under the `MIT license`_.
© 2020 martin68
© 2018 Peter Odding.
.. External references:
.. _apt-get: https://en.wikipedia.org/wiki/Advanced_Packaging_Tool
.. _at work: http://www.paylogic.com/
.. _changelog: https://apt-smart.readthedocs.io/en/latest/changelog.html
.. _Debian bug #110837: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=110837
.. _Debian bug #624122: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=624122
.. _Debian: https://en.wikipedia.org/wiki/Debian
.. _documentation: https://apt-smart.readthedocs.io
.. _GitHub: https://github.com/martin68/apt-smart
.. _Linux Mint: https://linuxmint.com
.. _MIT license: http://en.wikipedia.org/wiki/MIT_License
.. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/
.. _PyPI: https://pypi.python.org/pypi/apt-smart
.. _Read the Docs: https://apt-smart.readthedocs.io
.. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system)
.. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/
================================================
FILE: apt_smart/__init__.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: May 31, 2020
# URL: https://apt-smart.readthedocs.io
"""
Automated, robust ``apt-get`` mirror selection for Debian and Ubuntu.
The main entry point for this module is the :class:`AptMirrorUpdater` class, so
if you don't know where to start that would be a good place :-). You can also
take a look at the source code of the :mod:`apt_smart.cli` module for
an example that uses the :class:`AptMirrorUpdater` class.
"""
# Standard library modules.
import fnmatch
import logging
import os
import sys
import time
import calendar
# Python 2.x / 3.x compatibility.
try:
from enum import Enum
except ImportError:
from flufl.enum import Enum
# External dependencies.
from capturer import CaptureOutput
from executor.contexts import ChangeRootContext, LocalContext
from humanfriendly import AutomaticSpinner, Timer, compact, format_timespan, pluralize
try:
from property_manager3 import (
PropertyManager,
cached_property,
key_property,
lazy_property,
mutable_property,
set_property,
)
except ImportError:
from property_manager import (
PropertyManager,
cached_property,
key_property,
lazy_property,
mutable_property,
set_property,
)
from six import text_type
from six.moves.urllib.parse import urlparse
# Modules included in our package.
from apt_smart.http import NotFoundError, fetch_concurrent, fetch_url, get_default_concurrency
from apt_smart.releases import coerce_release
from apt_smart.releases import discover_releases
# Semi-standard module versioning.
__version__ = '7.1.3'
SOURCES_LIST_ENCODING = 'UTF-8'
"""The text encoding of :attr:`main_sources_list` (a string)."""
MAX_MIRRORS = 50
"""A sane default value for :attr:`AptMirrorUpdater.max_mirrors`."""
URL_CHAR_LEN = 34
"""A default value for :attr:`AptMirrorUpdater.url_char_len`."""
LAST_UPDATED_DEFAULT = 60 * 60 * 24 * 7 * 4
"""A default, pessimistic :attr:`~CandidateMirror.last_updated` value (a number)."""
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
class AptMirrorUpdater(PropertyManager):
"""Python API for the `apt-smart` program."""
repr_properties = (
'architecture',
'backend',
'blacklist',
'concurrency',
'context',
'distribution_codename',
'distributor_id',
'max_mirrors',
'old_releases_url',
'security_url',
)
"""
Override the list of properties included in :func:`repr()` output (a tuple of strings).
The :class:`~property_manager.PropertyManager` superclass defines a
:class:`~property_manager.PropertyManager.__repr__()` method that includes
the values of computed properties in its output.
In the case of `apt-smart` this behavior would trigger external
command execution and (lots of) HTTP calls, sometimes with unintended side
effects, namely `infinite recursion`_.
By setting :attr:`repr_properties` to a list of "safe" properties this
problematic behavior can be avoided.
.. _infinite recursion: https://travis-ci.org/xolox/python-apt-mirror-updater/jobs/395421319
"""
@mutable_property
def architecture(self):
"""
The name of the Debian package architecture (a string like 'i386' or 'amd64').
The package architecture is used to detect whether `Debian LTS`_ status
applies to the given system (the Debian LTS team supports a specific
subset of package architectures).
.. _Debian LTS: https://wiki.debian.org/LTS
"""
value = self.context.capture('dpkg', '--print-architecture')
set_property(self, 'architecture', value)
return value
@cached_property
def available_mirrors(self):
"""A list of :class:`CandidateMirror` objects (ordered from best to worst)"""
mirrors = set()
if self.release_is_eol:
logger.warning("Skipping mirror discovery because %s is EOL.", self.release)
else:
if self.read_custom_mirror_file:
mirrors.update(self.read_custom_mirror_file)
logger.info("Custom mirrors added from file:")
for mirror in mirrors:
logger.info(mirror.mirror_url)
logger.info("Adding BASE_URL mirror:")
if self.distributor_id == 'debian': # For Debian, base_url typically is not in MIRRORS_URL,
# add it explicitly
base_url_prefix = self.backend.BASE_URL.split('/dists/codename-updates/Release')[0]
mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self))
elif self.distributor_id == 'ubuntu': # For Ubuntu, base_url is not in MIRRORS_URL for
# some countries e.g. US (found it in Travis CI), add it explicitly
base_url_prefix = self.backend.BASE_URL.split('/dists/codename-security/Release')[0]
mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self))
elif self.distributor_id == 'linuxmint': # For Linux Mint, base_url typically is not in MIRRORS_URL,
# add it explicitly
base_url_prefix = self.backend.BASE_URL.split('/dists/codename/Release')[0]
mirrors.add(CandidateMirror(mirror_url=base_url_prefix, updater=self))
logger.info(base_url_prefix)
for candidate in self.backend.discover_mirrors():
if any(fnmatch.fnmatch(candidate.mirror_url, pattern) for pattern in self.blacklist)\
and normalize_mirror_url(candidate.mirror_url) != base_url_prefix:
logger.warning("Ignoring blacklisted mirror %s.", candidate.mirror_url)
else:
candidate.updater = self
mirrors.add(candidate)
# We make an attempt to incorporate the system's current mirror in
# the candidates but we don't propagate failures while doing so.
try:
# Gotcha: We should never include the system's current mirror in
# the candidates when we're bootstrapping a chroot for a different
# platform.
# if self.distributor_id == self.context.distributor_id: # We don't need to check this since
# 60850cc2 commint (Reimplement more robust :attr:`distribution_codename` using APT sources.list)
# already using :attr:`context` and self.context.distributor_id has issue:
# https://github.com/xolox/python-executor/issues/17
mirrors.add(CandidateMirror(mirror_url=self.current_mirror, updater=self))
except Exception as e:
logger.warning("Failed to add current mirror to set of available mirrors! (%s)", e)
# Sort the mirrors based on the currently available information.
return sorted(mirrors, key=lambda c: c.sort_key, reverse=True)
@cached_property
def backend(self):
"""
The backend module whose name matches :attr:`distributor_id`.
:raises: :exc:`~exceptions.EnvironmentError` when no matching backend
module is available.
"""
logger.debug("Checking whether %s platform is supported ..", self.distributor_id.capitalize())
module_path = "%s.backends.%s" % (__name__, self.distributor_id)
try:
__import__(module_path)
except ImportError:
msg = "%s platform is unsupported! (only Debian and Ubuntu are supported)"
raise EnvironmentError(msg % self.distributor_id.capitalize())
else:
return sys.modules[module_path]
@cached_property
def best_mirror(self):
"""
The URL of the first mirror in :attr:`ranked_mirrors` (a string).
This is a shortcut for using :attr:`ranked_mirrors` to select the
best mirror from :attr:`available_mirrors`, falling back to the
old releases URL when :attr:`release_is_eol` is :data:`True`.
"""
logger.debug("Selecting best %s mirror ..", self.distributor_id.capitalize())
if self.release_is_eol:
logger.info("%s is EOL, using %s.", self.release, self.old_releases_url)
return self.old_releases_url
else:
return self.ranked_mirrors[0].mirror_url
@cached_property
def blacklist(self):
"""
A set of strings with :mod:`fnmatch` patterns (defaults to an empty set).
When :attr:`available_mirrors` encounters a mirror whose URL matches
one of the patterns in :attr:`blacklist` the mirror will be ignored.
"""
return set()
@mutable_property
def concurrency(self):
"""
The number of concurrent HTTP connections allowed while ranking mirrors (a number).
The value of this property defaults to the value computed by
:func:`.get_default_concurrency()`.
"""
return get_default_concurrency()
@mutable_property(cached=True)
def context(self):
"""
An execution context created using :mod:`executor.contexts`.
The value of this property defaults to a
:class:`~executor.contexts.LocalContext` object.
"""
return LocalContext()
@mutable_property(cached=True)
def current_mirror(self):
"""
The URL of the main mirror in use in :attr:`main_sources_list` (a string).
The :attr:`current_mirror` property's value is computed using
:func:`find_current_mirror()`, but can be changed and cached by :func:`distribution_codename`
for Linux Mint's Ubuntu Mode.
"""
if self.ubuntu_mode and self.distribution_codename: # :func:`distribution_codename` will set current_mirror
return self.current_mirror
else:
logger.debug("Parsing %s to find current mirror of %s ..", self.main_sources_list, self.context)
return find_current_mirror(self.get_sources_list())
@mutable_property
def distribution_codename_old(self):
"""
deprecated: The distribution codename (a lowercase string like 'trusty' or 'xenial').
This relies on :mod:`executor` which is not robust to detect codename when
neither /etc/lsb-release nor lsb_release command are available, e.g. the official
Debian docker image (see https://github.com/xolox/python-executor/issues/17 )
The value of this property defaults to the value of the
:attr:`executor.contexts.AbstractContext.distribution_codename`
property which is the right choice 99% of the time.
"""
return self.context.distribution_codename
@mutable_property(cached=True)
def distribution_codename(self):
"""
The distribution codename (a lowercase string like 'trusty' or 'xenial')
The value of this property is determined using APT sources.list and should be more robust.
Similar to :func:`find_current_mirror` but return token[2] instead.
Also refer code of :func:`coerce_release`.
"""
for line in self.get_sources_list().splitlines():
# The first token should be `deb' or `deb-src', the second token is
# the mirror's URL, the third token is the `distribution' and any
# further tokens are `components'.
tokens = line.split()
if (len(tokens) >= 4
and tokens[0] in ('deb', 'deb-src')
and tokens[1].startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/'))
and 'main' in tokens[3:]):
matches = [release for release in discover_releases() if tokens[2].lower() in release.codename.lower()]
if len(matches) != 1:
continue
if self.ubuntu_mode and matches[0].distributor_id == 'linuxmint':
self.current_mirror = tokens[1]
continue
if self.ubuntu_mode:
logging.info("In Ubuntu Mode, pretend to be %s" % coerce_release(tokens[2]))
return tokens[2]
raise EnvironmentError("Failed to determine the distribution codename using apt's package resource list!")
@mutable_property(cached=True)
def distributor_id(self):
"""
The distributor ID (a lowercase string like 'debian' or 'ubuntu').
The default value of this property is based on the
:attr:`~apt_smart.releases.Release.distributor_id` property of
:attr:`release` (which in turn is based on :attr:`distribution_codename`).
Because Debian and Ubuntu code names are unambiguous this means that in
practice you can provide a value for :attr:`distribution_codename` and
omit :attr:`distributor_id` and everything should be fine.
"""
return self.release.distributor_id
@cached_property
def main_sources_list(self):
"""
The absolute pathname of the list of configured APT data sources (a string).
For new version of Linux Mint, main_sources_list is:
/etc/apt/sources.list.d/official-package-repositories.list
"""
if self.context.exists('/etc/apt/sources.list.d/official-package-repositories.list'):
logger.debug("/etc/apt/sources.list.d/official-package-repositories.list exists,\
use it as main_sources_list instead of /etc/apt/sources.list")
return '/etc/apt/sources.list.d/official-package-repositories.list'
else:
return '/etc/apt/sources.list'
@mutable_property
def max_mirrors(self):
"""Limits the number of mirrors to rank (a number, defaults to :data:`MAX_MIRRORS`)."""
return MAX_MIRRORS
@mutable_property
def url_char_len(self):
"""
The length of chars in mirrors' URL to display(a number, defaults to :data:`URL_CHAR_LEN`)
Specify the length of chars in mirrors' URL to display when using --list-mirrors
"""
return URL_CHAR_LEN
@mutable_property
def ubuntu_mode(self):
"""
For Linux Mint, deal with upstream Ubuntu mirror instead of Linux Mint mirror if True
Default is False, can be set True via -U, --ubuntu flag
"""
return False
@mutable_property
def old_releases_url(self):
"""The URL of the mirror that serves old releases for this :attr:`backend` (a string)."""
return self.backend.OLD_RELEASES_URL
@mutable_property
def base_url(self):
"""The actual official base URL according to :data:`.BASE_URL`"""
return self.backend.BASE_URL.replace('codename', self.distribution_codename)
@mutable_property
def base_last_updated(self):
"""
The Unix timestamp to determine which mirrors are up-to-date (an int)
The value of this property is gotten from :attr:`base_url`'s update date as minuend
"""
@cached_property
def ranked_mirrors(self):
"""
A list of :class:`CandidateMirror` objects (ordered from best to worst).
The value of this property is computed by concurrently testing the
mirrors in :attr:`available_mirrors` for the following details:
- availability (:attr:`~CandidateMirror.is_available`)
- connection speed (:attr:`~CandidateMirror.bandwidth`)
- update status (:attr:`~CandidateMirror.is_updating`)
The number of mirrors to test is limited to :attr:`max_mirrors` and you
can change the number of simultaneous HTTP connections allowed by
setting :attr:`concurrency`.
"""
timer = Timer()
# Sort the candidates based on the currently available information
# (and transform the input argument into a list in the process).
mirrors = sorted(self.available_mirrors, key=lambda c: c.sort_key, reverse=True)
# Limit the number of candidates to a reasonable number?
# NO, we don't need to now since the backends.debian can smartly get mirrors within a country.
# Without max_mirrors limit we can fix errors within United States (Travis CI reported) where
# where we can get 80+ mirrors. If limit applies, base_url mirror may be deleted, then error occurs.
"""
if self.max_mirrors and len(mirrors) > self.max_mirrors:
mirrors = mirrors[:self.max_mirrors]
"""
# Prepare the Release.gpg URLs to fetch.
mapping = dict((c.release_gpg_url, c) for c in mirrors)
num_mirrors = pluralize(len(mapping), "mirror")
logger.info("Checking %s for availability and performance ..", num_mirrors)
# Concurrently fetch the Release.gpg files.
with AutomaticSpinner(label="Checking mirrors"):
for url, data, elapsed_time in fetch_concurrent(mapping.keys(), concurrency=self.concurrency):
candidate = mapping[url]
candidate.release_gpg_contents = data
candidate.release_gpg_latency = elapsed_time
logger.info("Start retrieving :attr:`base_last_updated` using is_available")
self.base_last_updated = 0
if self.release_is_eol:
self.base_last_updated = int(time.time())
logger.warning("%s is EOL, so using time.time() as :attr:`base_last_updated`: %i",
self.release, self.base_last_updated)
elif mapping[self.base_url].is_available:
logger.debug(":attr:`base_last_updated`: %i", self.base_last_updated)
# base_url 's contents are up-to-date naturally,so set its last_updated 0
mapping[self.base_url].last_updated = 0
else: # base_url not available, use time at the moment as base_last_updated.
self.base_last_updated = int(time.time())
logger.warning("%s is not available, so using time.time() as :attr:`base_last_updated`: %i",
self.base_url, self.base_last_updated)
# Concurrently check for Archive-Update-in-Progress markers.
update_mapping = dict((c.archive_update_in_progress_url, c) for c in mirrors if c.is_available)
logger.info("Checking %s for Archive-Update-in-Progress marker ..",
pluralize(len(update_mapping), "mirror"))
with AutomaticSpinner(label="Checking mirrors"):
for url, data, elapsed_time in fetch_concurrent(update_mapping.keys(), concurrency=self.concurrency):
update_mapping[url].is_updating = data is not None
# Sanity check our results.
mirrors = list(mapping.values())
logger.info("Finished checking %s (took %s).", num_mirrors, timer)
if not any(c.is_available for c in mirrors):
raise Exception("It looks like all %s are unavailable!" % num_mirrors)
if all(c.is_updating for c in mirrors):
logger.warning("It looks like all %s are being updated?!", num_mirrors)
# blacklist BASE_URL mirror if matches blacklist pattern
if any(fnmatch.fnmatch(mapping[self.base_url].mirror_url, pattern) for pattern in self.blacklist):
logger.warning("Ignoring blacklisted BASE_URL mirror %s.", mapping[self.base_url].mirror_url)
mirrors.remove(mapping[self.base_url])
return sorted(mirrors, key=lambda c: c.sort_key, reverse=True)
@cached_property
def release(self):
"""A :class:`.Release` object corresponding to :attr:`distributor_id` and :attr:`distribution_codename`."""
return coerce_release(self.distribution_codename)
@cached_property
def release_is_eol(self):
"""
:data:`True` if the release is EOL (end of life), :data:`False` otherwise.
There are three ways in which the value of this property can be computed:
- When available, the first of the following EOL dates will be compared
against the current date to determine whether the release is EOL:
- If the :attr:`backend` module contains a ``get_eol_date()``
function (only the :mod:`~apt_smart.backends.debian`
module does at the time of writing) then it is called and if it
returns a number, this number is the EOL date for the release.
This function was added to enable apt-smart backend
modules to override the default EOL dates, more specifically to
respect the `Debian LTS`_ release schedule (see also `issue #5`_).
- Otherwise the :attr:`~apt_smart.releases.Release.eol_date`
of :attr:`release` is used.
- As a fall back :func:`validate_mirror()` is used to check whether
:attr:`security_url` results in :data:`MirrorStatus.MAYBE_EOL`.
.. _Debian LTS: https://wiki.debian.org/LTS
.. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5
"""
release_is_eol = None
logger.debug("Checking whether %s is EOL ..", self.release)
# Check if the backend provides custom EOL dates.
if hasattr(self.backend, 'get_eol_date'):
eol_date = self.backend.get_eol_date(self)
if eol_date:
release_is_eol = (time.time() >= eol_date)
source = "custom EOL dates"
# Check if the bundled data contains an applicable EOL date.
if release_is_eol is None and self.release.eol_date:
release_is_eol = self.release.is_eol
source = "known EOL dates"
# Validate the security mirror as a fall back.
if release_is_eol is None:
release_is_eol = (self.validate_mirror(self.security_url) == MirrorStatus.MAYBE_EOL)
source = "security mirror"
if release_is_eol and self.distributor_id == 'linuxmint':
logger.info(
"%s seems EOL (based on %s), but for Linux Mint no OLD_RELEASES_URL, so act as not EOL.",
self.release, source,
)
release_is_eol = False
return release_is_eol
if release_is_eol: # Still need to check due to
# https://github.com/xolox/python-apt-mirror-updater/issues/9
logger.info("%s seems EOL, checking %s MirrorStatus to confirm.", self.release, self.old_releases_url)
release_is_eol = (self.validate_mirror(self.old_releases_url) == MirrorStatus.AVAILABLE)
if not release_is_eol:
source = "%s is not available" % self.old_releases_url
logger.info(
"%s is %s (based on %s).", self.release,
"EOL" if release_is_eol else "supported", source,
)
return release_is_eol
@mutable_property
def security_url(self):
"""The URL of the mirror that serves security updates for this :attr:`backend` (a string)."""
return self.backend.SECURITY_URL
@cached_property
def stable_mirror(self):
"""
A mirror URL that is stable for the given execution context (a string).
The value of this property defaults to the value of
:attr:`current_mirror`, however if the current mirror can't be
determined or is deemed inappropriate by :func:`validate_mirror()`
then :attr:`best_mirror` will be used instead.
This provides a stable mirror selection algorithm which is useful
because switching mirrors causes ``apt-get update`` to unconditionally
download all package lists and this takes a lot of time so should it be
avoided when unnecessary.
"""
if self.release_is_eol:
logger.debug("%s is EOL, falling back to %s.", self.release, self.old_releases_url)
return self.old_releases_url
else:
try:
logger.debug("Trying to select current mirror as stable mirror ..")
return self.current_mirror
except Exception:
logger.debug("Failed to determine current mirror, selecting best mirror instead ..")
return self.best_mirror
@cached_property
def validated_mirrors(self):
"""Dictionary of validated mirrors (used by :func:`validate_mirror()`)."""
return {}
@mutable_property
def custom_mirror_file_path(self):
"""The local custom mirror file's absolute path, can be set by `-F` flag"""
return None
@cached_property
def read_custom_mirror_file(self):
"""
Read a file containing custom mirror URLs (one URL per line) to add custom mirrors to rank.
:param file_to_read: The local file's absolute path
:returns: A set of mirrors read from file
"""
if self.custom_mirror_file_path is None:
return {}
else:
logger.info("The file path you input is %s", self.custom_mirror_file_path)
mirrors = set()
with open(self.custom_mirror_file_path) as f:
for line in f:
if line.strip().startswith(('http://', 'https://', 'ftp://')):
mirrors.add(CandidateMirror(mirror_url=line.strip(), updater=self))
return mirrors
def change_mirror(self, new_mirror=None, update=True):
"""
Change the main mirror in use in :attr:`main_sources_list`.
:param new_mirror: The URL of the new mirror (a string, defaults to
:attr:`best_mirror`).
:param update: Whether an ``apt-get update`` should be run after
changing the mirror (a boolean, defaults to
:data:`True`).
"""
timer = Timer()
# Default to the best available mirror.
if new_mirror:
logger.info("Changing mirror of %s to %s ..", self.context, new_mirror)
else:
logger.info("Changing mirror of %s to best available mirror ..", self.context)
new_mirror = self.best_mirror
logger.info("Selected mirror: %s", new_mirror)
# Parse /etc/apt/sources.list to replace the old mirror with the new one.
sources_list = self.get_sources_list()
mirrors_to_replace = [normalize_mirror_url(self.current_mirror)]
if self.release_is_eol:
# When a release goes EOL the security updates mirrors stop
# serving that release as well, so we need to remove them.
logger.debug("Replacing %s URLs as well ..", self.security_url)
mirrors_to_replace.append(normalize_mirror_url(self.security_url))
else:
logger.debug("Not replacing %s URLs.", self.security_url)
lines = sources_list.splitlines()
sources_list_options = self.get_sources_list_options
for i, line in enumerate(lines):
# The first token should be `deb' or `deb-src', the second token is
# the mirror's URL, the third token is the `distribution' and any
# further tokens are `components'.
tokens = line.split()
if (len(tokens) >= 4
and tokens[0] in ('deb', 'deb-src')
and normalize_mirror_url(tokens[1]) in mirrors_to_replace):
tokens[1] = new_mirror
if i in sources_list_options:
tokens.insert(1, '[' + sources_list_options[i] + ']') # Get the [options] back
lines[i] = u' '.join(tokens)
# Install the modified package resource list.
self.install_sources_list(u'\n'.join(lines))
# Clear (relevant) cached properties.
del self.current_mirror
# Make sure previous package lists are removed.
self.clear_package_lists()
# Make sure the package lists are up to date.
if update:
self.smart_update(switch_mirrors=False)
logger.info("Finished changing mirror of %s in %s.", self.context, timer)
def clear_package_lists(self):
"""Clear the package list cache by removing the files under ``/var/lib/apt/lists``."""
timer = Timer()
logger.info("Clearing package list cache of %s ..", self.context)
self.context.execute(
# We use an ugly but necessary find | xargs pipeline here because
# find's -delete option implies -depth which negates -prune. Sigh.
'find /var/lib/apt/lists -type f -name lock -prune -o -type f -print0 | xargs -0 rm -f',
sudo=True,
)
logger.info("Successfully cleared package list cache of %s in %s.", self.context, timer)
def create_chroot(self, directory, codename=None, arch=None):
"""
Bootstrap a basic Debian or Ubuntu system using debootstrap_.
:param directory: The pathname of the target directory (a string).
:param codename: The codename of the target (a string).
:param arch: The target architecture (a string or :data:`None`).
:returns: A :class:`~executor.contexts.ChangeRootContext` object.
If `directory` already exists and isn't empty then it is assumed that
the chroot has already been created and debootstrap_ won't be run.
Before this method returns it changes :attr:`context` to the chroot.
.. _debootstrap: https://manpages.debian.org/debootstrap
"""
logger.debug("Checking if chroot already exists (%s) ..", directory)
if self.context.exists(directory) and self.context.list_entries(directory):
logger.info("The chroot already exists, skipping initialization.")
first_run = False
else:
# Ensure the `debootstrap' program is installed.
if not self.context.find_program('debootstrap'):
logger.info("Installing `debootstrap' program ..")
self.context.execute('apt-get', 'install', '--yes', 'debootstrap', sudo=True)
# Use the `debootstrap' program to create the chroot.
timer = Timer()
debootstrap_command = ['debootstrap']
if arch:
debootstrap_command.append('--arch=%s' % arch)
release_chroot = None
keyring_chroot = ''
codename_chroot = ''
best_mirror_chroot = None
generate_sources_list_chroot = None
if codename and codename != self.distribution_codename:
updater_chroot = AptMirrorUpdater()
updater_chroot.distribution_codename = codename
if updater_chroot.distributor_id == 'linuxmint':
msg = "It seems no sense to create chroot of Linux Mint, " \
"please specify a codename of Ubuntu or Debian " \
"to create chroot."
raise ValueError(msg)
if not self.context.exists(updater_chroot.release.keyring_file):
if updater_chroot.distributor_id == 'ubuntu':
self.context.execute('apt-get', 'install', '--yes', 'ubuntu-keyring', sudo=True)
elif updater_chroot.distributor_id == 'debian':
self.context.execute('apt-get', 'install', '--yes', 'debian-archive-keyring', sudo=True)
release_chroot = updater_chroot.release
keyring_chroot = updater_chroot.release.keyring_file
codename_chroot = codename
best_mirror_chroot = updater_chroot.best_mirror
else:
if self.distributor_id == 'linuxmint':
msg = "It seems no sense to create chroot of Linux Mint, " \
"please use -C to specify a codename of Ubuntu or Debian " \
"to create chroot."
raise ValueError(msg)
release_chroot = self.release
keyring_chroot = self.release.keyring_file
codename_chroot = self.distribution_codename
best_mirror_chroot = self.best_mirror
logger.info("Creating %s chroot in %s ..", release_chroot, directory)
debootstrap_command.append('--keyring=%s' % keyring_chroot)
debootstrap_command.append(codename_chroot)
debootstrap_command.append(directory)
debootstrap_command.append(best_mirror_chroot)
self.context.execute(*debootstrap_command, sudo=True)
logger.info("Took %s to create %s chroot.", timer, release_chroot)
first_run = True
# Switch the execution context to the chroot and reset the locale (to
# avoid locale warnings emitted by post-installation scripts run by
# `apt-get install').
self.context = ChangeRootContext(
chroot=directory,
environment=dict(LC_ALL='C'),
)
# Clear the values of cached properties that can be
# invalidated by switching the execution context.
del self.current_mirror
del self.stable_mirror
if codename and codename != self.distribution_codename:
updater_chroot.context = self.context
del updater_chroot.current_mirror
del updater_chroot.stable_mirror
generate_sources_list_chroot = updater_chroot.generate_sources_list()
else:
generate_sources_list_chroot = self.generate_sources_list()
# The following initialization only needs to happen on the first
# run, but it requires the context to be set to the chroot.
if first_run:
# Make sure the `lsb_release' program is available. It is
# my experience that this package cannot be installed using
# `debootstrap --include=lsb-release', it specifically
# needs to be installed afterwards.
self.context.execute('apt-get', 'install', '--yes', 'lsb-release', sudo=True)
# Cleanup downloaded *.deb archives.
self.context.execute('apt-get', 'clean', sudo=True)
# Install a suitable /etc/apt/sources.list file. The logic behind
# generate_sources_list() depends on the `lsb_release' program.
logger.debug("sources.list for chroot generated:")
logger.debug(generate_sources_list_chroot)
self.install_sources_list(generate_sources_list_chroot)
# Make sure the package lists are up to date.
self.smart_update()
return self.context
def dumb_update(self, *args):
"""
Update the system's package lists (by running ``apt-get update``).
:param args: Command line arguments to ``apt-get update`` (zero or more strings).
The :func:`dumb_update()` method doesn't do any error handling or
retrying, if that's what you're looking for then you need
:func:`smart_update()` instead.
"""
timer = Timer()
logger.info("Updating package lists of %s ..", self.context)
self.context.execute('apt-get', 'update', *args, sudo=True)
logger.info("Finished updating package lists of %s in %s.", self.context, timer)
def generate_sources_list(self, **options):
"""
Generate the contents of ``/etc/apt/sources.list``.
If no `mirror_url` keyword argument is given then :attr:`stable_mirror`
is used as a default.
Please refer to the documentation of the Debian
(:func:`apt_smart.backends.debian.generate_sources_list()`)
and Ubuntu (:func:`apt_smart.backends.ubuntu.generate_sources_list()`)
backend implementations of this method for details on argument handling
and the return value.
"""
if options.get('mirror_url') is None:
options['mirror_url'] = self.stable_mirror
options.setdefault('codename', self.distribution_codename)
return self.backend.generate_sources_list(**options)
@mutable_property
def get_sources_list_options(self):
"""
Get the contents of [options] in :attr:`main_sources_list`.
[options] can be set into sources.list, e.g.
deb [arch=amd64] http://mymirror/ubuntu bionic main restricted
see details at
https://manpages.debian.org/jessie/apt/sources.list.5.en.html
The [options] is often not considered and breaks parsing in many projects, see
https://github.com/jblakeman/apt-select/issues/54
We begin to deal with the [options] by stripping it from sources.list,
and then get it back when generating new sources.list
"""
def get_sources_list(self):
"""
Get the contents of :attr:`main_sources_list`.
:returns: A Unicode string.
This code currently assumes that the ``sources.list`` file is encoded
using :data:`SOURCES_LIST_ENCODING`. I'm not actually sure if this is
correct because I haven't been able to find a formal specification!
Feedback is welcome :-).
This code strips [options] from sources.list, stores it in :attr:`get_sources_list_options`
"""
contents = self.context.read_file(self.main_sources_list)
contents = contents.decode(SOURCES_LIST_ENCODING)
sources_list_options = {}
contents_raw = [] # stripped contents without options
for i, line in enumerate(contents.splitlines()):
if line.find('[') > 0: # found '[' and not starts with '['
startswith_deb = line.split('[')[0]
temp = line.split('[')[1]
sources_list_options[i] = temp.split(']')[0]
startswith_http = temp.split(']')[1]
contents_raw.append(startswith_deb + startswith_http)
elif line.find('[') == -1: # not found
contents_raw.append(line)
self.get_sources_list_options = sources_list_options
return '\n'.join(contents_raw)
def ignore_mirror(self, pattern):
"""
Add a pattern to the mirror discovery :attr:`blacklist`.
:param pattern: A shell pattern (containing wild cards like ``?`` and
``*``) that is matched against the full URL of each
mirror.
When a pattern is added to the blacklist any previously cached values
of :attr:`available_mirrors`, :attr:`best_mirror`, :attr:`ranked_mirrors`
and :attr:`stable_mirror` are cleared. This makes sure that mirrors
blacklisted after mirror discovery has already run are ignored.
"""
# Update the blacklist.
logger.info("Adding pattern to mirror discovery blacklist: %s", pattern)
self.blacklist.add(pattern)
# Clear (relevant) cached properties.
del self.available_mirrors
del self.best_mirror
del self.ranked_mirrors
del self.stable_mirror
def install_sources_list(self, contents):
"""
Install a new ``/etc/apt/sources.list`` file.
:param contents: The new contents of the sources list (a Unicode
string). You can generate a suitable value using
the :func:`generate_sources_list()` method.
"""
if isinstance(contents, text_type):
contents = contents.encode(SOURCES_LIST_ENCODING)
logger.info("Installing new %s ..", self.main_sources_list)
with self.context:
# Write the sources.list contents to a temporary file. We make sure
# the file always ends in a newline to adhere to UNIX conventions.
temporary_file = '/tmp/apt-smart-sources-list-%i.txt' % os.getpid()
contents_to_write = contents.rstrip() + b'\n'
self.context.write_file(temporary_file, contents_to_write)
# Make sure the temporary file is cleaned up when we're done with it.
self.context.cleanup('rm', '--force', temporary_file)
# Make a backup copy of /etc/apt/sources.list in case shit hits the fan?
if self.context.exists(self.main_sources_list):
dirname, basename = os.path.split(self.main_sources_list)
if basename == 'official-package-repositories.list':
backup_dir = os.path.join(dirname, 'backup_by_apt-smart') # Backup to dir for Linux Mint
if not self.context.exists(backup_dir):
self.context.execute('mkdir', backup_dir, sudo=True)
backup_copy = '%s.backup.%i' % (os.path.join(backup_dir, basename), time.time())
else:
backup_copy = '%s.backup.%i' % (self.main_sources_list, time.time())
logger.info("Backing up contents of %s to %s ..", self.main_sources_list, backup_copy)
self.context.execute('cp', self.main_sources_list, backup_copy, sudo=True)
# Move the temporary file into place without changing ownership and permissions.
self.context.execute(
'cp', '--no-preserve=mode,ownership',
temporary_file, self.main_sources_list,
sudo=True,
)
def smart_update(self, *args, **kw):
"""
Update the system's package lists (switching mirrors if necessary).
:param args: Command line arguments to ``apt-get update`` (zero or more strings).
:param max_attempts: The maximum number of attempts at successfully
updating the system's package lists (an integer,
defaults to 10).
:param switch_mirrors: :data:`True` if we're allowed to switch mirrors
on 'hash sum mismatch' errors, :data:`False`
otherwise.
:raises: If updating of the package lists fails 10 consecutive times
(`max_attempts`) an exception is raised.
While :func:`dumb_update()` simply runs ``apt-get update`` the
:func:`smart_update()` function works quite differently:
- First the system's package lists are updated using
:func:`dumb_update()`. If this is successful we're done.
- If the update fails we check the command's output for the phrase
'hash sum mismatch'. If we find this phrase we assume that the
current mirror is faulty and switch to another one.
- Failing ``apt-get update`` runs are retried up to `max_attempts`.
"""
backoff_time = 10
max_attempts = kw.get('max_attempts', 10)
switch_mirrors = kw.get('switch_mirrors', True)
for i in range(1, max_attempts + 1):
with CaptureOutput() as session:
try:
self.dumb_update(*args)
return
except Exception:
if i < max_attempts:
output = session.get_text()
# Check for EOL releases. This somewhat peculiar way of
# checking is meant to ignore 404 responses from
# `secondary package mirrors' like PPAs. If the output
# of `apt-get update' implies that the release is EOL
# we need to verify our assumption.
if any(self.current_mirror in line and u'404' in line.split() for line in output.splitlines()):
logger.warning("%s may be EOL, checking ..", self.release)
if self.release_is_eol:
if switch_mirrors:
logger.warning("Switching to old releases mirror because %s is EOL ..",
self.release)
self.change_mirror(self.old_releases_url, update=False)
continue
else:
raise Exception(compact("""
Failed to update package lists because it looks like
the current release (%s) is end of life but I'm not
allowed to switch mirrors! (there's no point in
retrying so I'm not going to)
""", self.distribution_codename))
# Check for `hash sum mismatch' errors.
if switch_mirrors and u'hash sum mismatch' in output.lower():
logger.warning("Detected 'hash sum mismatch' failure, switching to other mirror ..")
self.ignore_mirror(self.current_mirror)
self.change_mirror(update=False)
else:
logger.warning("Retrying after `apt-get update' failed (%i/%i) ..", i, max_attempts)
# Deal with unidentified (but hopefully transient) failures by retrying but backing off
# to give the environment (network connection, mirror state, etc.) time to stabilize.
logger.info("Sleeping for %s before retrying update ..", format_timespan(backoff_time))
time.sleep(backoff_time)
if backoff_time <= 120:
backoff_time *= 2
else:
backoff_time += backoff_time / 3
raise Exception("Failed to update package lists %i consecutive times?!" % max_attempts)
def validate_mirror(self, mirror_url):
"""
Make sure a mirror serves :attr:`distribution_codename`.
:param mirror_url: The base URL of the mirror (a string).
:returns: One of the values in the :class:`MirrorStatus` enumeration.
"""
mirror_url = normalize_mirror_url(mirror_url)
key = (mirror_url, self.distribution_codename)
value = self.validated_mirrors.get(key)
if value is None:
logger.info("Checking if %s is available on %s ..", self.release, mirror_url)
# Try to download the Release.gpg file, in the assumption that
# this file should always exist and is more or less guaranteed
# to be relatively small.
try:
mirror = CandidateMirror(mirror_url=mirror_url, updater=self)
mirror.release_gpg_contents = fetch_url(mirror.release_gpg_url, retry=False)
value = (MirrorStatus.AVAILABLE if mirror.is_available else MirrorStatus.UNAVAILABLE)
except NotFoundError:
# When the mirror is serving 404 responses it can be an
# indication that the release has gone end of life. In any
# case the mirror is unavailable.
value = MirrorStatus.MAYBE_EOL
except Exception:
# When we get an unspecified error that is not a 404
# response we conclude that the mirror is unavailable.
value = MirrorStatus.UNAVAILABLE
# Cache the mirror status that we just determined.
self.validated_mirrors[key] = value
return value
class CandidateMirror(PropertyManager):
"""A candidate mirror groups a mirror URL with its availability and performance metrics."""
@mutable_property
def bandwidth(self):
"""
The bytes per second achieved while fetching :attr:`release_gpg_url` (a number or :data:`None`).
The value of this property is computed based on the values of
:attr:`release_gpg_contents` and :attr:`release_gpg_latency`.
"""
if self.release_gpg_contents and self.release_gpg_latency:
return len(self.release_gpg_contents) / self.release_gpg_latency
@lazy_property
def archive_update_in_progress_url(self):
"""
The URL of the file whose existence indicates that the mirror is being updated (a string).
The value of this property is computed based on the value of
:attr:`mirror_url`.
"""
return '%s/Archive-Update-in-Progress-%s' % (
self.mirror_url, urlparse(self.mirror_url).netloc,
)
@key_property
def mirror_url(self):
"""The base URL of the mirror (a string)."""
@mirror_url.setter
def mirror_url(self, value):
"""Normalize the mirror URL when set."""
set_property(self, 'mirror_url', normalize_mirror_url(value))
@mutable_property
def is_available(self):
"""
:data:`True` if :attr:`release_gpg_contents` contains the expected data, :data:`False` otherwise.
The value of this property is computed by checking whether
:attr:`release_gpg_contents` contains the expected data.
This may seem like a rather obscure way of
validating a mirror, but it was specifically chosen to detect
all sorts of ways in which mirrors can be broken:
- Webservers with a broken configuration that return an error page for
all URLs.
- Mirrors whose domain name registration has expired, where the domain
is now being squatted and returns HTTP 200 OK responses for all URLs
(whether they "exist" or not).
"""
value = False
if self.release_gpg_contents:
value = b'Date:' in self.release_gpg_contents
if not value:
logger.debug("Missing data, considering mirror unavailable (%s).", self.release_gpg_url)
else:
# Get all data following "Date: "
date_string_raw = self.release_gpg_contents.decode('utf-8').split("Date: ", 1)
if len(date_string_raw) == 2: # split succussfully using "Date: "
# Get only date string like "Sun, 25 Aug 2019 23:35:36 UTC", drop other data
date_string = date_string_raw[1].split("\n")[0]
if date_string.endswith("UTC"):
# Convert it into UNIX timestamp
last_updated_time = calendar.timegm(time.strptime(date_string, "%a, %d %b %Y %H:%M:%S %Z"))
if self.updater.base_last_updated == 0: # First time launch this method, must be base_url
self.updater.base_last_updated = last_updated_time
logger.debug("base_last_updated: %i", self.updater.base_last_updated)
else:
# if last_updated is 0 means this mirror is up-to-date
self.last_updated = self.updater.base_last_updated - last_updated_time
logger.debug("last_updated: %i", self.last_updated)
else:
logger.debug("Not UTC? Correct me. " + date_string)
logger.debug("Looks good, %s is_available return True", self.release_gpg_url)
else: # split fails because lacking "Date: "
logger.debug("Missing Date, considering mirror unavailable (%s).", self.release_gpg_url)
value = False
set_property(self, 'is_available', value)
return value
@mutable_property
def is_updating(self):
""":data:`True` if the mirror is being updated, :data:`False` otherwise."""
@mutable_property
def last_updated(self):
"""The time in seconds since the most recent mirror update (a number or :data:`None`)."""
@mutable_property
def release_gpg_contents(self):
"""
The contents downloaded from :attr:`release_gpg_url` (a string or :data:`None`).
By downloading the file available at :attr:`release_gpg_url` and
setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency`
you enable the :attr:`bandwidth` and :attr:`is_available` properties to
be computed.
"""
@mutable_property
def release_gpg_latency(self):
"""
The time it took to download :attr:`release_gpg_url` (a number or :data:`None`).
By downloading the file available at :attr:`release_gpg_url` and
setting :attr:`release_gpg_contents` and :attr:`release_gpg_latency`
you enable the :attr:`bandwidth` and :attr:`is_available` properties to
be computed.
"""
@mutable_property
def release_gpg_url(self):
"""
The URL of the ``Release`` file that will be used to test the mirror (a string or :data:`None`).
The value of this property is based on :attr:`mirror_url` and the
:attr:`~AptMirrorUpdater.distribution_codename` property of the
:attr:`updater` object.
"""
if self.updater and self.updater.distribution_codename:
if self.updater.distributor_id == 'ubuntu':
return '%s/dists/%s-security/Release' % (
self.mirror_url, self.updater.distribution_codename,
)
elif self.updater.distributor_id == 'debian':
return '%s/dists/%s-updates/Release' % (
self.mirror_url, self.updater.distribution_codename,
)
elif self.updater.distributor_id == 'linuxmint':
return '%s/dists/%s/Release' % (
self.mirror_url, self.updater.distribution_codename,
)
@mutable_property
def sort_key(self):
"""
A tuple that can be used to sort the mirror by its availability/performance metrics.
The tuple created by this property contains four numbers in the following order:
1. The number 1 when :attr:`is_available` is :data:`True` or
the number 0 when :attr:`is_available` is :data:`False`
(because most importantly a mirror must be available).
2. The number 0 when :attr:`is_updating` is :data:`True` or
the number 1 when :attr:`is_updating` is :data:`False`
(because being updated at this very moment is *bad*).
3. The negated value of :attr:`last_updated` (because the
lower :attr:`last_updated` is, the better). If :attr:`last_updated`
is :data:`None` then :data:`LAST_UPDATED_DEFAULT` is used instead.
4. The value of :attr:`bandwidth` (because the higher
:attr:`bandwidth` is, the better).
By sorting :class:`CandidateMirror` objects on these tuples in
ascending order, the last mirror in the sorted results will be the
"most suitable mirror" (given the available information).
"""
return (int(self.is_available),
int(not self.is_updating),
-(self.last_updated if self.last_updated is not None else LAST_UPDATED_DEFAULT),
self.bandwidth or 0)
@mutable_property(repr=False)
def updater(self):
"""A reference to the :class:`AptMirrorUpdater` object that created the candidate."""
class MirrorStatus(Enum):
"""Enumeration for mirror statuses determined by :func:`AptMirrorUpdater.validate_mirror()`."""
AVAILABLE = 1
"""The mirror is accepting connections and serving the expected content."""
MAYBE_EOL = 2
"""The mirror is serving HTTP 404 "Not Found" responses instead of the expected content."""
UNAVAILABLE = 3
"""The mirror is not accepting connections or not serving the expected content."""
def find_current_mirror(sources_list):
"""
Find the URL of the main mirror that is currently in use by ``apt-get``.
:param sources_list: The contents of apt's package resource list, e.g. the
contents of :attr:`main_sources_list` (a string).
:returns: The URL of the main mirror in use (a string).
:raises: If the main mirror can't be determined
:exc:`~exceptions.EnvironmentError` is raised.
The main mirror is determined by looking for the first ``deb`` or
``deb-src`` directive in apt's package resource list whose URL uses the
HTTP or FTP scheme and whose components contain ``main``.
"""
for line in sources_list.splitlines():
# The first token should be `deb' or `deb-src', the second token is
# the mirror's URL, the third token is the `distribution' and any
# further tokens are `components'.
tokens = line.split()
if (len(tokens) >= 4
and tokens[0] in ('deb', 'deb-src')
and tokens[1].startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/'))
and 'main' in tokens[3:]):
return tokens[1]
raise EnvironmentError("Failed to determine current mirror in apt's package resource list!")
def mirrors_are_equal(a, b):
"""
Check whether two mirror URLS are equal.
:param a: The first mirror URL (a string).
:param b: The second mirror URL (a string).
:returns: :data:`True` if the mirror URLs are equal,
:data:`False` otherwise.
"""
return normalize_mirror_url(a) == normalize_mirror_url(b)
def normalize_mirror_url(url):
"""
Normalize a mirror URL so it can be compared using string equality comparison.
:param url: The mirror URL to normalize (a string).
:returns: The normalized mirror URL (a string).
"""
return url.rstrip('/')
================================================
FILE: apt_smart/backends/__init__.py
================================================
"""Namespace for platform specific mirror discovery in :mod:`apt_smart`."""
================================================
FILE: apt_smart/backends/debian.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
"""
Discovery of Debian package archive mirrors.
Here are references to some of the material that I've needed to consult while
working on this module:
- `Notes about sources.list on the Debian wiki <https://wiki.debian.org/SourcesList>`_
- `The Debian backports webpages <https://backports.debian.org/Instructions/>`_
- `Documentation about the "proposed-updates" mechanism <https://www.debian.org/releases/proposed-updates.html>`_
"""
# Standard library modules.
import logging
import json
# External dependencies.
import six
from bs4 import BeautifulSoup
from humanfriendly import Timer, format, pluralize
# Modules included in our package.
from apt_smart import CandidateMirror, mirrors_are_equal
from apt_smart.http import fetch_url
LTS_ARCHITECTURES = ('i386', 'amd64', 'armel', 'armhf')
"""The names of the architectures supported by the Debian LTS team (a tuple of strings)."""
LTS_RELEASES = {
'jessie': 1593468000, # 2020-06-30
'stretch': 1656540000, # 2022-06-30
}
"""
A dictionary with `Debian LTS`_ releases and their EOL dates.
This is needed because distro-info-data_ doesn't contain information
about Debian LTS releases but nevertheless ``archive.debian.org``
doesn't adopt a release until the LTS status expires (this was
originally reported in `issue #5`_).
.. _Debian LTS: https://wiki.debian.org/LTS
.. _issue #5: https://github.com/xolox/python-apt-mirror-updater/issues/5
"""
MIRRORS_URL = 'https://www.debian.org/mirror/list'
"""The URL of the HTML page listing all primary Debian mirrors (a string)."""
SECURITY_URL = 'http://security.debian.org/'
"""The base URL of the Debian mirror with security updates (a string)."""
OLD_RELEASES_URL = 'http://archive.debian.org/debian-archive/debian/'
"""The URL where EOL (end of life) Debian releases are hosted (a string)."""
BASE_URL = 'http://ftp.debian.org/debian/dists/codename-updates/Release'
"""The URL where official repo treated as base are hosted (a string).
The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated`
to determine which mirrors are up-to-date"""
DEFAULT_SUITES = 'release', 'security', 'updates'
"""A tuple of strings with the Debian suites that are enabled by default."""
VALID_COMPONENTS = 'main', 'contrib', 'non-free'
"""A tuple of strings with the names of the components available in the Debian package repositories."""
VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed-updates'
"""A tuple of strings with the names of the suites available in the Debian package repositories."""
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
def discover_mirrors():
"""
Discover available Debian mirrors by querying :data:`MIRRORS_URL`.
:returns: A set of :class:`.CandidateMirror` objects that have their
:attr:`~.CandidateMirror.mirror_url` property set.
:raises: If no mirrors are discovered an exception is raised.
An example run:
>>> from apt_smart.backends.debian import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://ftp.at.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.au.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.be.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.bg.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.br.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.by.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.ca.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.ch.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.cn.debian.org/debian/'),
CandidateMirror(mirror_url='http://ftp.cz.debian.org/debian/'),
...])
"""
timer = Timer()
logger.info("Discovering Debian mirrors at %s ..", MIRRORS_URL)
# Find which country the user is in to get mirrors in that country
try:
url = 'https://ipapi.co/json'
response = fetch_url(url, timeout=2)
# On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5,
# so decode it to str
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country_name']
logger.info("Found your location: %s by %s", country, url)
except Exception:
url = 'http://ip-api.com/json'
response = fetch_url(url, timeout=5)
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country']
logger.info("Found your location: %s by %s", country, url)
data = fetch_url(MIRRORS_URL, timeout=20, retry=True)
soup = BeautifulSoup(data, 'html.parser')
tables = soup.findAll('table')
flag = False # flag is True when find the row's text is that country
mirrors = set()
if not tables:
raise Exception("Failed to locate <table> element in Debian mirror page! (%s)" % MIRRORS_URL)
else:
for row in tables[1].findAll("tr"): # tables[1] organises mirrors by country.
if flag:
if not row.a: # End of mirrors located in that country
break
else:
mirrors.add(CandidateMirror(mirror_url=row.a['href']))
if row.get_text() == country:
flag = True
if len(mirrors) < 3: # Too few, add tables[0] which contains Primary Debian mirror sites all around the world.
mirrors.add(CandidateMirror(mirror_url=a['href']) for a in tables[0].findAll('a', href=True))
if not mirrors:
raise Exception("Failed to discover any Debian mirrors! (using %s)" % MIRRORS_URL)
logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Debian mirror"), timer)
return mirrors
def generate_sources_list(mirror_url, codename,
suites=DEFAULT_SUITES,
components=VALID_COMPONENTS,
enable_sources=False):
"""
Generate the contents of ``/etc/apt/sources.list`` for a Debian system.
:param mirror_url: The base URL of the mirror (a string).
:param codename: The codename of a Debian release (a string like 'wheezy'
or 'jessie') or a Debian release class (a string like
'stable', 'testing', etc).
:param suites: An iterable of strings (defaults to
:data:`DEFAULT_SUITES`, refer to
:data:`VALID_SUITES` for details).
:param components: An iterable of strings (refer to
:data:`VALID_COMPONENTS` for details).
:param enable_sources: :data:`True` to include ``deb-src`` entries,
:data:`False` to omit them.
:returns: The suggested contents of ``/etc/apt/sources.list`` (a string).
"""
# Validate the suites.
invalid_suites = [s for s in suites if s not in VALID_SUITES]
if invalid_suites:
msg = "Invalid Debian suite(s) given! (%s)"
raise ValueError(msg % invalid_suites)
# Validate the components.
invalid_components = [c for c in components if c not in VALID_COMPONENTS]
if invalid_components:
msg = "Invalid Debian component(s) given! (%s)"
raise ValueError(msg % invalid_components)
# Generate the /etc/apt/sources.list file contents.
lines = []
directives = ('deb', 'deb-src') if enable_sources else ('deb',)
for suite in suites:
for directive in directives:
lines.append(format(
'{directive} {mirror} {suite} {components}', directive=directive,
mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL)
else (SECURITY_URL if suite == 'security' else mirror_url)),
suite=(codename if suite == 'release' else (
('%s/updates' % codename if suite == 'security'
else codename + '-' + suite))),
components=' '.join(components),
))
return '\n'.join(lines)
def get_eol_date(updater):
"""
Override the EOL date for `Debian LTS`_ releases.
:param updater: The :class:`~apt_smart.AptMirrorUpdater` object.
:returns: The overridden EOL date (a number) or :data:`None`.
"""
if updater.architecture in LTS_ARCHITECTURES:
return LTS_RELEASES.get(updater.distribution_codename)
================================================
FILE: apt_smart/backends/linuxmint.py
================================================
# Automated, robust apt-get mirror selection for Debian ,Ubuntu and Linux Mint.
#
# Author: martin68 and Peter Odding
# Last Change: October 29, 2019
# URL: https://apt-smart.readthedocs.io
"""Discovery of Linux Mint package archive mirrors."""
# Standard library modules.
import json
import logging
# External dependencies.
import six
from bs4 import BeautifulSoup
from humanfriendly import Timer, pluralize
# Modules included in our package.
from apt_smart import CandidateMirror
from apt_smart.http import fetch_url
MIRRORS_URL = 'https://linuxmint.com/mirrors.php'
"""The URL of the HTML page listing official Linux Mint mirrors (a string)."""
SECURITY_URL = 'http://security.ubuntu.com/ubuntu'
"""The URL where Ubuntu ( Linux Mint's codebase )security updates are hosted (a string)."""
BASE_URL = 'http://packages.linuxmint.com/dists/codename/Release'
"""The URL where official repo treated as base are hosted (a string).
The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated`
to determine which mirrors are up-to-date"""
DEFAULT_SUITES = 'release', 'updates', 'backports', 'security'
"""A tuple of strings with the Linux Mint suites that are enabled by default."""
VALID_COMPONENTS = 'main', 'restricted', 'universe', 'multiverse'
"""A tuple of strings with the names of the components available in the Linux Mint package repositories."""
VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed'
"""
A tuple of strings with the names of the suites available in the Linux Mint package
repositories.
The actual name of the 'release' suite is the codename of the relevant Linux Mint
release, while the names of the other suites are formed by concatenating the
codename with the suite name (separated by a dash).
As an example to make things more concrete, Ubuntu 16.04 has the following five
suites available: ``xenial`` (this is the release suite), ``xenial-security``,
``xenial-updates``, ``xenial-backports`` and ``xenial-proposed``.
"""
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
def discover_mirrors():
"""
Discover available Linux Mint mirrors.
:returns: A set of :class:`.CandidateMirror` objects that have their
:attr:`~.CandidateMirror.mirror_url` property set and may have
the :attr:`~.CandidateMirror.last_updated` property set.
:raises: If no mirrors are discovered an exception is raised.
This queries :data:`MIRRORS_URL`to discover available Linux Mint mirrors.
Here's an example run:
>>> from apt_smart.backends.linuxmint import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://mirrors.cqu.edu.cn/linuxmint/'),
CandidateMirror(mirror_url='http://mirrors.hust.edu.cn/linuxmint/'),
CandidateMirror(mirror_url='http://mirrors.shu.edu.cn/linuxmint/'),
CandidateMirror(mirror_url='https://mirrors.tuna.tsinghua.edu.cn/linuxmint/'),
CandidateMirror(mirror_url='http://mirrors.ustc.edu.cn/linuxmint/'),
CandidateMirror(mirror_url='http://mirrors.zju.edu.cn/linuxmint/'),
...])
"""
timer = Timer()
mirrors = set()
logger.info("Discovering Linux Mint mirrors at %s ..", MIRRORS_URL)
# Find which country the user is in to get mirrors in that country
try:
url = 'https://ipapi.co/json'
response = fetch_url(url, timeout=2)
# On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5,
# so decode it to str
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country_name']
logger.info("Found your location: %s by %s", country, url)
except Exception:
url = 'http://ip-api.com/json'
response = fetch_url(url, timeout=5)
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country']
logger.info("Found your location: %s by %s", country, url)
if country == 'United States':
country = 'USA'
try:
data = fetch_url(MIRRORS_URL, timeout=15)
except Exception:
logger.warning("Time out, try again")
data = fetch_url(MIRRORS_URL, timeout=70, retry=True)
soup = BeautifulSoup(data, 'html.parser')
tables = soup.findAll('table')
if not tables:
raise Exception("Failed to locate <table> element in Linux Mint mirror page! (%s)" % MIRRORS_URL)
else:
while len(mirrors) < 3:
for row in tables[2].findAll("tr"):
if country in row.get_text():
for data in row.findAll("td"):
data_text = data.get_text()
# Check if the link looks like a mirror URL.
if data_text.startswith(('http://', 'https://')):
mirrors.add(CandidateMirror(mirror_url=data_text))
if country == 'Worldwide':
break # break while loop when already set to 'Worldwide'
if len(mirrors) < 3:
logging.info("Too few mirrors found in your country, get more Worldwide mirrors.")
country = 'Worldwide'
if not mirrors:
raise Exception("Failed to discover any Linux Mint mirrors! (using %s)" % MIRRORS_URL)
logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Linux Mint mirror"), timer)
return mirrors
================================================
FILE: apt_smart/backends/ubuntu.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
"""Discovery of Ubuntu package archive mirrors."""
# Standard library modules.
import json
import logging
# External dependencies.
import six
from bs4 import BeautifulSoup, UnicodeDammit
from humanfriendly import Timer, format, pluralize
# Modules included in our package.
from apt_smart import CandidateMirror, mirrors_are_equal
from apt_smart.http import fetch_url
MIRRORS_URL = 'https://launchpad.net/ubuntu/+archivemirrors'
"""The URL of the HTML page listing official Ubuntu mirrors (a string)."""
MIRROR_SELECTION_URL = 'http://mirrors.ubuntu.com/mirrors.txt'
"""The URL of a plain text listing of "geographically suitable" mirror URLs (a string)."""
OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/'
"""The URL where EOL (end of life) Ubuntu releases are hosted (a string)."""
SECURITY_URL = 'http://security.ubuntu.com/ubuntu'
"""The URL where Ubuntu security updates are hosted (a string)."""
BASE_URL = 'http://archive.ubuntu.com/ubuntu/dists/codename-security/Release'
"""The URL where official repo treated as base are hosted (a string).
The Release file contains `Date:` which can be gotten as :attr:`.base_last_updated`
to determine which mirrors are up-to-date"""
DEFAULT_SUITES = 'release', 'updates', 'backports', 'security'
"""A tuple of strings with the Ubuntu suites that are enabled by default."""
VALID_COMPONENTS = 'main', 'restricted', 'universe', 'multiverse'
"""A tuple of strings with the names of the components available in the Ubuntu package repositories."""
VALID_SUITES = 'release', 'security', 'updates', 'backports', 'proposed'
"""
A tuple of strings with the names of the suites available in the Ubuntu package
repositories.
The actual name of the 'release' suite is the codename of the relevant Ubuntu
release, while the names of the other suites are formed by concatenating the
codename with the suite name (separated by a dash).
As an example to make things more concrete, Ubuntu 16.04 has the following five
suites available: ``xenial`` (this is the release suite), ``xenial-security``,
``xenial-updates``, ``xenial-backports`` and ``xenial-proposed``.
"""
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
def discover_mirrors_old():
"""
Discover available Ubuntu mirrors. (fallback)
:returns: A set of :class:`.CandidateMirror` objects that have their
:attr:`~.CandidateMirror.mirror_url` property set and may have
the :attr:`~.CandidateMirror.last_updated` property set.
:raises: If no mirrors are discovered an exception is raised.
This queries :data:`MIRRORS_URL`to discover available Ubuntu mirrors.
Here's an example run:
>>> from apt_smart.backends.ubuntu import discover_mirrors_old
>>> from pprint import pprint
>>> pprint(discover_mirrors_old())
set([CandidateMirror(mirror_url='http://archive.ubuntu.com/ubuntu/'),
CandidateMirror(mirror_url='http://ftp.nluug.nl/os/Linux/distr/ubuntu/'),
CandidateMirror(mirror_url='http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/'),
CandidateMirror(mirror_url='http://ftp.tudelft.nl/archive.ubuntu.com/'),
CandidateMirror(mirror_url='http://mirror.1000mbps.com/ubuntu/'),
CandidateMirror(mirror_url='http://mirror.amsiohosting.net/archive.ubuntu.com/'),
CandidateMirror(mirror_url='http://mirror.i3d.net/pub/ubuntu/'),
CandidateMirror(mirror_url='http://mirror.nforce.com/pub/linux/ubuntu/'),
CandidateMirror(mirror_url='http://mirror.nl.leaseweb.net/ubuntu/'),
CandidateMirror(mirror_url='http://mirror.transip.net/ubuntu/ubuntu/'),
...])
It may be super-slow somewhere ( with 100Mbps fibre though ) in the world to access launchpad.net (see below),
so we have to no longer rely on MIRRORS_URL .
time curl -o/dev/null 'https://launchpad.net/ubuntu/+archivemirrors'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 263k 100 263k 0 0 5316 0 0:00:50 0:00:50 --:--:-- 6398
real 0m50.869s
user 0m0.045s
sys 0m0.039s
But it can be a fallback when MIRROR_SELECTION_URL is down.
"""
mirrors = set()
logger.info("Discovering Ubuntu mirrors at %s ..", MIRRORS_URL)
# Find which country the user is in to get mirrors in that country
try:
url = 'https://ipapi.co/json'
response = fetch_url(url, timeout=2)
# On py3 response is bytes and json.loads throws TypeError in py3.4 and 3.5,
# so decode it to str
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country_name']
logger.info("Found your location: %s by %s", country, url)
except Exception:
url = 'http://ip-api.com/json'
response = fetch_url(url, timeout=5)
if isinstance(response, six.binary_type):
response = response.decode('utf-8')
data = json.loads(response)
country = data['country']
logger.info("Found your location: %s by %s", country, url)
data = fetch_url(MIRRORS_URL, timeout=70, retry=True)
soup = BeautifulSoup(data, 'html.parser')
tables = soup.findAll('table')
flag = False # flag is True when find the row's text is that country
if not tables:
raise Exception("Failed to locate <table> element in Ubuntu mirror page! (%s)" % MIRRORS_URL)
else:
for row in tables[0].findAll("tr"):
if flag:
if not row.a: # End of mirrors located in that country
break
else:
for a in row.findAll('a', href=True):
# Check if the link looks like a mirror URL.
if a['href'].startswith(('http://', 'https://')):
mirrors.add(CandidateMirror(mirror_url=a['href']))
if row.th and row.th.get_text() == country:
flag = True
if not mirrors:
raise Exception("Failed to discover any Ubuntu mirrors! (using %s)" % MIRRORS_URL)
return mirrors
def discover_mirrors():
"""
Discover available Ubuntu mirrors.
:returns: A set of :class:`.CandidateMirror` objects that have their
:attr:`~.CandidateMirror.mirror_url` property set and may have
the :attr:`~.CandidateMirror.last_updated` property set.
:raises: If no mirrors are discovered an exception is raised.
This only queries :data:`MIRROR_SELECTION_URL` to
discover available Ubuntu mirrors. Here's an example run:
>>> from apt_smart.backends.ubuntu import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
"""
timer = Timer()
mirrors = set()
mirrors = discover_mirror_selection()
if not mirrors:
logger.warning("Failed to discover any Ubuntu mirrors! (using %s)" % MIRROR_SELECTION_URL)
logger.info("Trying to use %s as fallback" % MIRRORS_URL)
mirrors = discover_mirrors_old()
elif len(mirrors) < 2:
logger.warning("Too few mirrors, trying to use %s to find more" % MIRRORS_URL)
mirrors |= discover_mirrors_old() # add mirrors from discover_mirrors_old()
logger.info("Discovered %s in %s.", pluralize(len(mirrors), "Ubuntu mirror"), timer)
return mirrors
def discover_mirror_selection():
"""Discover "geographically suitable" Ubuntu mirrors."""
timer = Timer()
logger.info("Identifying fast Ubuntu mirrors using %s ..", MIRROR_SELECTION_URL)
data = fetch_url(MIRROR_SELECTION_URL, timeout=3, retry=True, max_attempts=5)
# shorter timeout with more retries is good for unstable connections to MIRROR_SELECTION_URL
dammit = UnicodeDammit(data)
mirrors = set(
CandidateMirror(mirror_url=mirror_url.strip())
for mirror_url in dammit.unicode_markup.splitlines()
if mirror_url and not mirror_url.isspace() and mirror_url.startswith(('http://', 'https://'))
)
logger.debug("Found %s in %s.", pluralize(len(mirrors), "fast Ubuntu mirror"), timer)
return mirrors
def generate_sources_list(mirror_url, codename,
suites=DEFAULT_SUITES,
components=VALID_COMPONENTS,
enable_sources=False):
"""
Generate the contents of ``/etc/apt/sources.list`` for an Ubuntu system.
:param mirror_url: The base URL of the mirror (a string).
:param codename: The codename of the Ubuntu release (a string like 'trusty' or 'xenial').
:param suites: An iterable of strings (defaults to :data:`DEFAULT_SUITES`,
refer to :data:`VALID_SUITES` for details).
:param components: An iterable of strings (refer to
:data:`VALID_COMPONENTS` for details).
:param enable_sources: :data:`True` to include ``deb-src`` entries,
:data:`False` to omit them.
:returns: The suggested contents of ``/etc/apt/sources.list`` (a string).
"""
# Validate the suites.
invalid_suites = [s for s in suites if s not in VALID_SUITES]
if invalid_suites:
msg = "Invalid Ubuntu suite(s) given! (%s)"
raise ValueError(msg % invalid_suites)
# Validate the components.
invalid_components = [c for c in components if c not in VALID_COMPONENTS]
if invalid_components:
msg = "Invalid Ubuntu component(s) given! (%s)"
raise ValueError(msg % invalid_components)
# Generate the /etc/apt/sources.list file contents.
lines = []
directives = ('deb', 'deb-src') if enable_sources else ('deb',)
for suite in suites:
for directive in directives:
lines.append(format(
'{directive} {mirror} {suite} {components}', directive=directive,
mirror=(OLD_RELEASES_URL if mirrors_are_equal(mirror_url, OLD_RELEASES_URL)
else (SECURITY_URL if suite == 'security' else mirror_url)),
suite=(codename if suite == 'release' else codename + '-' + suite),
components=' '.join(components),
))
return '\n'.join(lines)
================================================
FILE: apt_smart/cli.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
"""
Usage: apt-smart [OPTIONS]
The apt-smart program automates robust apt-get mirror selection for
Debian and Ubuntu by enabling discovery of available mirrors, ranking of
available mirrors, automatic switching between mirrors and robust package list
updating.
Supported options:
-r, --remote-host=SSH_ALIAS
Operate on a remote system instead of the local system. The SSH_ALIAS
argument gives the SSH alias of the remote host. It is assumed that the
remote account has root privileges or password-less sudo access.
-f, --find-current-mirror
Determine the main mirror that is currently configured in
/etc/apt/sources.list and report its URL on standard output.
-F, --file-to-read=local_file_absolute_path
Read a local absolute path (path and filename must NOT contain whitespace) file
containing custom mirror URLs (one URL per line) to add custom mirrors to rank.
-b, --find-best-mirror
Discover available mirrors, rank them, select the best one and report its
URL on standard output.
-l, --list-mirrors
List available (ranked) mirrors on the terminal in a human readable format.
-L, --url-char-len=int
An integer to specify the length of chars in mirrors' URL to display when
using --list-mirrors, default is 34
-c, --change-mirror=MIRROR_URL
Update /etc/apt/sources.list to use the given MIRROR_URL.
-a, --auto-change-mirror
Discover available mirrors, rank the mirrors by connection speed and update
status and update /etc/apt/sources.list to use the best available mirror.
-u, --update, --update-package-lists
Update the package lists using `apt-get update', retrying on failure and
automatically switch to a different mirror when it looks like the current
mirror is being updated.
-U, --ubuntu
Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror.
e.g. --auto-change-mirror --ubuntu will auto-change Linux Mint's upstream Ubuntu mirror
-x, --exclude=PATTERN
Add a pattern to the mirror selection blacklist. PATTERN is expected to be
a shell pattern (containing wild cards like `?' and `*') that is matched
against the full URL of each mirror.
-v, --verbose
Increase logging verbosity (can be repeated).
-V, --version
Show version number and Python version.
-R, --create-chroot=local_dir_absolute_path
Create chroot with the best mirror in a local directory with absolute_path
-C, --codename=codename
Must use with -R , create chroot with a codename of Ubuntu or Debian, e.g. bionic, buster
-q, --quiet
Decrease logging verbosity (can be repeated).
-h, --help
Show this message and exit.
Note: since apt-smart uses `urlopen` method in The Python Standard Library,
you can set Environment Variables to make apt-smart connect via HTTP proxy, e.g. in terminal type:
export {http,https,ftp}_proxy='http://user:password@myproxy.com:1080'
These will not persist however (no longer active after you close the terminal),
so you may wish to add the line to your ~/.bashrc
"""
# Standard library modules.
import functools
import getopt
import logging
import sys
import os
# External dependencies.
import coloredlogs
from executor.contexts import LocalContext, RemoteContext
from humanfriendly import format_size, format_table, format_timespan
from humanfriendly.terminal import connected_to_terminal, output, usage, warning
# Modules included in our package.
from apt_smart import MAX_MIRRORS, URL_CHAR_LEN, AptMirrorUpdater
from apt_smart import __version__ as updater_version
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
def main():
"""Command line interface for the ``apt-smart`` program."""
# Initialize logging to the terminal and system log.
coloredlogs.install(syslog=True)
# Command line option defaults.
context = LocalContext()
updater = AptMirrorUpdater(context=context)
limit = MAX_MIRRORS
url_char_len = URL_CHAR_LEN
ubuntu_mode = False
chroot_path = ''
codename = ''
actions = []
# Parse the command line arguments.
try:
options, arguments = getopt.getopt(sys.argv[1:], 'r:fF:blL:c:auUx:m:vVR:C:qh', [
'remote-host=', 'find-current-mirror', 'find-best-mirror', 'file-to-read=',
'list-mirrors', 'url-char-len=', 'change-mirror=', 'auto-change-mirror', 'update',
'update-package-lists', 'ubuntu', 'exclude=', 'max=', 'verbose', 'version', 'create-chroot=',
'codename=', 'quiet', 'help',
])
for option, value in options:
if option in ('-r', '--remote-host'):
if actions:
msg = "The %s option should be the first option given on the command line!"
raise Exception(msg % option)
context = RemoteContext(value)
updater = AptMirrorUpdater(context=context)
elif option in ('-f', '--find-current-mirror'):
actions.append(functools.partial(report_current_mirror, updater))
elif option in ('-F', '--file-to-read'):
updater.custom_mirror_file_path = value
elif option in ('-b', '--find-best-mirror'):
actions.append(functools.partial(report_best_mirror, updater))
elif option in ('-l', '--list-mirrors'):
actions.append(functools.partial(report_available_mirrors, updater))
elif option in ('-L', '--url-char-len'):
url_char_len = int(value)
elif option in ('-c', '--change-mirror'):
if value.strip().startswith(('http://', 'https://', 'ftp://', 'mirror://', 'mirror+file:/')):
actions.append(functools.partial(updater.change_mirror, value))
else:
raise Exception("\'%s\' is not a valid mirror URL" % value)
elif option in ('-a', '--auto-change-mirror'):
actions.append(updater.change_mirror)
elif option in ('-u', '--update', '--update-package-lists'):
actions.append(updater.smart_update)
elif option in ('-U', '--ubuntu'):
ubuntu_mode = True
elif option in ('-x', '--exclude'):
actions.insert(0, functools.partial(updater.ignore_mirror, value))
elif option in ('-m', '--max'):
limit = int(value)
elif option in ('-v', '--verbose'):
coloredlogs.increase_verbosity()
elif option in ('-V', '--version'):
output("Version: %s on Python %i.%i", updater_version, sys.version_info[0], sys.version_info[1])
return
elif option in ('-C', '--codename'):
codename = value
elif option in ('-R', '--create-chroot'):
chroot_path = value
elif option in ('-q', '--quiet'):
coloredlogs.decrease_verbosity()
elif option in ('-h', '--help'):
usage(__doc__)
return
else:
assert False, "Unhandled option!"
if codename and not chroot_path:
assert False, "--codename must be used with valid -R to specify chroot path"
if chroot_path:
actions.append(functools.partial(updater.create_chroot, chroot_path, codename=codename))
if not actions:
usage(__doc__)
return
# Propagate options to the Python API.
updater.max_mirrors = limit
updater.url_char_len = url_char_len
updater.ubuntu_mode = ubuntu_mode
except Exception as e:
warning("Error: Failed to parse command line arguments! (%s)" % e)
sys.exit(1)
# Perform the requested action(s).
try:
for callback in actions:
callback()
except Exception:
logger.exception("Encountered unexpected exception! Aborting ..")
sys.exit(1)
def report_current_mirror(updater):
"""Print the URL of the currently configured ``apt-get`` mirror."""
output(updater.current_mirror)
def report_best_mirror(updater):
"""Print the URL of the "best" mirror."""
output(updater.best_mirror)
def report_available_mirrors(updater):
"""Print the available mirrors to the terminal (in a human friendly format)."""
if connected_to_terminal() or os.getenv('TRAVIS') == 'true': # make Travis CI test this code
# https://docs.travis-ci.com/user/environment-variables/#default-environment-variables
have_bandwidth = any(c.bandwidth for c in updater.ranked_mirrors)
have_last_updated = any(c.last_updated is not None for c in updater.ranked_mirrors)
column_names = ["Rank", "Mirror URL", "Available?", "Updating?"]
if have_last_updated:
column_names.append("Last updated")
if have_bandwidth:
column_names.append("Bandwidth")
data = []
long_mirror_urls = {}
if os.getenv('TRAVIS') == 'true' and updater.url_char_len < 50:
updater.url_char_len = 50
for i, candidate in enumerate(updater.ranked_mirrors, start=1):
if len(candidate.mirror_url) <= updater.url_char_len:
stripped_mirror_url = candidate.mirror_url
else: # the mirror_url is too long, strip it
stripped_mirror_url = candidate.mirror_url[:updater.url_char_len - 3]
stripped_mirror_url = stripped_mirror_url + "..."
long_mirror_urls[i] = candidate.mirror_url # store it, output as full afterwards
row = [i, stripped_mirror_url,
"Yes" if candidate.is_available else "No",
"Yes" if candidate.is_updating else "No"]
if have_last_updated:
row.append("Up to date" if candidate.last_updated == 0 else (
"%s behind" % format_timespan(candidate.last_updated, max_units=1)
if candidate.last_updated else "Unknown"
))
if have_bandwidth:
row.append("%s/s" % format_size(round(candidate.bandwidth, 0))
if candidate.bandwidth else "Unknown")
data.append(row)
output(format_table(data, column_names=column_names))
if long_mirror_urls:
output(u"Full URLs which are too long to be shown in above table:")
for key in long_mirror_urls:
output(u"%i: %s", key, long_mirror_urls[key])
else:
output(u"\n".join(
candidate.mirror_url for candidate in updater.ranked_mirrors
if candidate.is_available and not candidate.is_updating
))
================================================
FILE: apt_smart/http.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
"""Simple, robust and concurrent HTTP requests (designed for one very narrow use case)."""
# Standard library modules.
import logging
import multiprocessing
import signal
# External dependencies.
from humanfriendly import Timer, format_size
from six.moves.urllib.request import urlopen
from stopit import SignalTimeout # , TimeoutException
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
# Stop the `stopit' logger from logging tracebacks.
logging.getLogger('stopit').setLevel(logging.ERROR)
def fetch_url(url, timeout=10, retry=False, max_attempts=3):
"""
Fetch a URL, optionally retrying on failure.
:param url: The URL to fetch (a string).
:param timeout: The maximum time in seconds that's allowed to pass before
the request is aborted (a number, defaults to 10 seconds).
:param retry: Whether to retry on failure (defaults to :data:`False`).
:param max_attempts: The maximum number of attempts when retrying is
enabled (an integer, defaults to three).
:returns: The response body (a byte string).
:raises: Any of the following exceptions can be raised:
- :exc:`NotFoundError` when the URL returns a 404 status code.
- :exc:`InvalidResponseError` when the URL returns a status code
that isn't 200.
- :exc:`stopit.TimeoutException` when the request takes longer
than `timeout` seconds (refer to the `stopit documentation
<https://pypi.python.org/pypi/stopit>`_ for details).
- Any exception raised by Python's standard library in the last
attempt (assuming all attempts raise an exception).
"""
timer = Timer()
logger.debug("Fetching %s ..", url)
for i in range(1, max_attempts + 1):
try:
with SignalTimeout(timeout, swallow_exc=False):
response = urlopen(url)
status_code = response.getcode()
if status_code != 200:
exc_type = (NotFoundError if status_code == 404 else InvalidResponseError)
raise exc_type("URL returned unexpected status code %s! (%s)" % (status_code, url))
response_body = response.read()
logger.debug("Took %s to fetch %s.", timer, url)
return response_body
except NotFoundError:
# We never retry 404 responses but retry timeouts.
raise
except Exception as e:
if retry and i < max_attempts:
logger.warning("Failed to fetch %s, retrying (%i/%i, error was: %s)", url, i, max_attempts, e)
else:
raise
def fetch_concurrent(urls, concurrency=None):
"""
Fetch the given URLs concurrently using :mod:`multiprocessing`.
:param urls: An iterable of URLs (strings).
:param concurrency: Override the concurrency (an integer, defaults to the
value computed by :func:`get_default_concurrency()`).
:returns: A list of tuples like those returned by :func:`fetch_worker()`.
"""
if concurrency is None:
concurrency = get_default_concurrency()
pool = multiprocessing.Pool(concurrency)
try:
return pool.map(fetch_worker, urls, chunksize=1)
finally:
pool.terminate()
def get_default_concurrency():
"""
Get the default concurrency for :func:`fetch_concurrent()`.
:returns: A positive integer number.
"""
return max(4, multiprocessing.cpu_count() * 2)
def fetch_worker(url):
"""
Fetch the given URL for :func:`fetch_concurrent()`.
:param url: The URL to fetch (a string).
:returns: A tuple of three values:
1. The URL that was fetched (a string).
2. The data that was fetched (a string or :data:`None`).
3. The number of seconds it took to fetch the URL (a number).
"""
# Ignore Control-C instead of raising KeyboardInterrupt because (due to a
# quirk in multiprocessing) this can cause the parent and child processes
# to get into a deadlock kind of state where only Control-Z will get you
# your precious terminal back; super annoying IMHO.
signal.signal(signal.SIGINT, signal.SIG_IGN)
timer = Timer()
try:
data = fetch_url(url, retry=False)
except Exception as e:
logger.debug("Failed to fetch %s! (%s)", url, e)
data = None
else:
kbps = format_size(round(len(data) / timer.elapsed_time, 2))
logger.debug("Downloaded %s at %s per second.", url, kbps)
return url, data, timer.elapsed_time
class InvalidResponseError(Exception):
"""Raised by :func:`fetch_url()` when a URL returns a status code that isn't 200."""
class NotFoundError(InvalidResponseError):
"""Raised by :func:`fetch_url()` when a URL returns a 404 status code."""
================================================
FILE: apt_smart/releases.py
================================================
# Easy to use metadata on Debian and Ubuntu releases.
#
# Author: martin68 and Peter Odding
# Last Change: May 31, 2020
# URL: https://apt-smart.readthedocs.io
"""
Easy to use metadata on Debian and Ubuntu releases.
This module started out with the purpose of reliable `end of life`_ (EOL)
detection for Debian and Ubuntu releases based on data provided by the
distro-info-data_ package. Since then the need arose to access more of the
available metadata and so the ``eol`` module became the ``releases`` module.
Debian and Ubuntu releases have an EOL date that marks the end of support for
each release. At that date the release stops receiving further (security)
updates and some time after package mirrors stop serving the release.
The distro-info-data_ package contains CSV files with metadata about Debian and
Ubuntu releases. This module parses those CSV files to make this metadata
available in Python. This enables `apt-smart` to make an informed
decision about the following questions:
1. Is a given Debian or Ubuntu release expected to be available on mirrors or
will it only be available in the archive of old releases?
2. Is the signing key of a given Ubuntu release expected to be included in the
main keyring (:data:`UBUNTU_KEYRING_CURRENT`) or should the keyring with
removed keys (:data:`UBUNTU_KEYRING_REMOVED`) be used?
To make it possible to run `apt-smart` without direct access to the
CSV files, a copy of the relevant information has been embedded in the source
code.
.. _end of life: https://en.wikipedia.org/wiki/End-of-life_(product)
.. _distro-info-data: https://packages.debian.org/distro-info-data
"""
# Standard library modules.
import csv
import datetime
import decimal
import glob
import logging
import numbers
import os
# External dependencies.
import six
from executor import execute
from humanfriendly.decorators import cached
try:
from property_manager3 import PropertyManager, key_property, lazy_property, required_property, writable_property
except ImportError:
from property_manager import PropertyManager, key_property, lazy_property, required_property, writable_property
from six import string_types
from itertools import product
DISTRO_INFO_DIRECTORY = '/usr/share/distro-info'
"""The pathname of the directory with CSV files containing release metadata (a string)."""
DEBIAN_KEYRING_CURRENT = '/usr/share/keyrings/debian-archive-keyring.gpg'
"""The pathname of the main Debian keyring file (a string)."""
UBUNTU_KEYRING_CURRENT = '/usr/share/keyrings/ubuntu-archive-keyring.gpg'
"""The pathname of the main Ubuntu keyring file (a string)."""
UBUNTU_KEYRING_REMOVED = '/usr/share/keyrings/ubuntu-archive-removed-keys.gpg'
"""The pathname of the Ubuntu keyring file with removed keys (a string)."""
# Public identifiers that require documentation.
__all__ = (
'DISTRO_INFO_DIRECTORY',
'DEBIAN_KEYRING_CURRENT',
'UBUNTU_KEYRING_CURRENT',
'UBUNTU_KEYRING_REMOVED',
'Release',
'coerce_release',
'discover_releases',
'ubuntu_keyring_updated',
)
# Initialize a logger.
logger = logging.getLogger(__name__)
def coerce_release(value):
"""
Try to coerce the given value to a Debian or Ubuntu release.
:param value: The value to coerce (a number, a string or a :class:`Release` object).
:returns: A :class:`Release` object.
:raises: :exc:`~exceptions.ValueError` when the given value cannot be coerced to a known release.
The following values can be coerced:
- Numbers and numbers formatted as strings match :attr:`Release.version`.
- Strings match :attr:`Release.codename` (case insensitive).
.. warning:: Don't use floating point numbers like 10.04 because their
actual value will be something like 10.039999999999999147
which won't match the intended release.
"""
# Release objects pass through untouched.
if isinstance(value, Release):
return value
# Numbers and version strings are matched against release versions.
if isinstance(value, numbers.Number) or is_version_string(value):
typed_value = decimal.Decimal(value)
matches = [release for release in discover_releases() if release.version == typed_value]
if len(matches) != 1:
msg = "The number %s doesn't match a known Debian or Ubuntu or Linux Mint release!"
raise ValueError(msg % value)
return matches[0]
# Other strings are matched against release code names.
matches = [release for release in discover_releases() if value.lower() in release.codename.lower()]
if len(matches) != 1:
msg = "The string %r doesn't match a known Debian or Ubuntu or Linux Mint release!"
raise ValueError(msg % value)
return matches[0]
@cached
def discover_releases():
"""
Discover known Debian and Ubuntu releases.
:returns: A list of discovered :class:`Release` objects sorted by
:attr:`~Release.distributor_id` and :attr:`~Release.version`.
The first time this function is called it will try to parse the CSV files
in ``/usr/share/distro-info`` and merge any releases it finds with the
releases embedded into the source code of this module. The result is cached
and returned each time the function is called. It's not a problem if the
``/usr/share/distro-info`` directory doesn't exist or doesn't contain any
``*.csv`` files (it won't cause a warning or error). Of course in this case
only the embedded releases will be returned.
"""
# Discover the known releases on the first call to discover_releases().
# First we check the CSV files on the system where apt-smart
# is running, because those files may be more up-to-date than the
# bundled information is.
result = set()
for filename in glob.glob(os.path.join(DISTRO_INFO_DIRECTORY, '*.csv')):
for release in parse_csv_file(filename):
result.add(release)
# Add the releases bundled with apt-smart to the result
# without causing duplicate entries (due to the use of a set and key
# properties).
result.update(BUNDLED_RELEASES)
# Sort the releases by distributor ID and version / series.
return sorted(result, key=lambda r: (r.distributor_id, r.version or 0, r.series))
def table_to_2d(table_tag): # https://stackoverflow.com/a/48451104
rowspans = [] # track pending rowspans
rows = table_tag.find_all('tr')
# first scan, see how many columns we need
colcount = 0
for r, row in enumerate(rows):
cells = row.find_all(['td', 'th'], recursive=False)
# count columns (including spanned).
# add active rowspans from preceding rows
# we *ignore* the colspan value on the last cell, to prevent
# creating 'phantom' columns with no actual cells, only extended
# colspans. This is achieved by hardcoding the last cell width as 1.
# a colspan of 0 means "fill until the end" but can really only apply
# to the last cell; ignore it elsewhere.
colcount = max(
colcount,
sum(int(c.get('colspan', 1)) or 1 for c in cells[:-1]) + len(cells[-1:]) + len(rowspans))
# update rowspan bookkeeping; 0 is a span to the bottom.
rowspans += [int(c.get('rowspan', 1)) or len(rows) - r for c in cells]
rowspans = [s - 1 for s in rowspans if s > 1]
# it doesn't matter if there are still rowspan numbers 'active'; no extra
# rows to show in the table means the larger than 1 rowspan numbers in the
# last table row are ignored.
# build an empty matrix for all possible cells
table = [[None] * colcount for row in rows]
# fill matrix from row data
rowspans = {} # track pending rowspans, column number mapping to count
for row, row_elem in enumerate(rows):
span_offset = 0 # how many columns are skipped due to row and colspans
for col, cell in enumerate(row_elem.find_all(['td', 'th'], recursive=False)):
# adjust for preceding row and colspans
col += span_offset
while rowspans.get(col, 0):
span_offset += 1
col += 1
# fill table data
rowspan = rowspans[col] = int(cell.get('rowspan', 1)) or len(rows) - row
colspan = int(cell.get('colspan', 1)) or colcount - col
# next column is offset by the colspan
span_offset += colspan - 1
value = cell.get_text()
for drow, dcol in product(range(rowspan), range(colspan)):
try:
table[row + drow][col + dcol] = value
rowspans[col + dcol] = rowspan
except IndexError:
# rowspan or colspan outside the confines of the table
pass
# update rowspan bookkeeping
rowspans = {c: s - 1 for c, s in rowspans.items() if s > 1}
return table
def discover_linuxmint_releases(array_2d):
d = {} # a dict to map table head to column number
head = array_2d[0]
for i, data in enumerate(head):
d[data] = i
last = None
for entry in array_2d[1:-1]:
if entry[d['Codename\n']] == last:
continue # skip same codename entry
last = entry[d['Codename\n']]
yield Release(
codename=parse_data_wiki(entry[d['Codename\n']]),
compatible_repository=(
parse_data_wiki(entry[d['Compatible repository\n']]).split('(')[1].split(' ')[0].lower()
if entry[d['Compatible repository\n']].find('(') > 0
else parse_data_wiki(entry[d['Compatible repository\n']])
),
is_lts=('Yes' in entry[d['LTS?\n']]),
created_date=parse_date_wiki(entry[d['Release date\n']]),
distributor_id='linuxmint',
eol_date=parse_date_wiki(entry[d['Support End\n']]),
extended_eol_date=parse_date_wiki(entry[d['Support End\n']]),
release_date=parse_date_wiki(entry[d['Release date\n']]),
series=parse_data_wiki(entry[d['Codename\n']]).lower(),
version=parse_version_wiki(entry[d['Version\n']]) if entry[d['Version\n']] else None,
)
def is_version_string(value):
"""Check whether the given value is a string containing a positive number."""
try:
return isinstance(value, string_types) and float(value) > 0
except Exception:
return False
def parse_csv_file(filename):
"""
Parse a CSV file in the format of the ``/usr/share/distro-info/*.csv`` files.
:param filename: The pathname of the CSV file (a string).
:returns: A generator of :class:`Release` objects.
"""
# We import this here to avoid a circular import.
from apt_smart.backends.debian import LTS_RELEASES
basename, extension = os.path.splitext(os.path.basename(filename))
distributor_id = basename.lower()
with open(filename) as handle:
for entry in csv.DictReader(handle):
yield Release(
codename=entry['codename'],
is_lts=(
entry['series'] in LTS_RELEASES if distributor_id == 'debian' else (
'LTS' in entry['version'] if distributor_id == 'ubuntu' else (
# Neither Debian nor Ubuntu, let's not assume anything...
False
)
)
),
created_date=parse_date(entry['created']),
distributor_id=distributor_id,
eol_date=parse_date(entry['eol']),
extended_eol_date=(
# Special handling for Debian LTS releases.
datetime.datetime.fromtimestamp(LTS_RELEASES[entry['series']]).date()
if distributor_id == 'debian' and entry['series'] in LTS_RELEASES
# Ubuntu LTS releases are defined by the CSV file.
else parse_date(entry.get('eol-server'))
),
release_date=parse_date(entry['release']),
series=entry['series'],
version=parse_version(entry['version']) if entry['version'] else None,
)
def parse_data_wiki(value):
r"""Strip a string such as ``19 December 2018 [18]\n`` to ``19 December 2018``"""
if six.PY2:
return value.encode("utf8").split('[')[0].strip()
else:
return value.split('[')[0].strip()
def parse_date(value):
"""Convert a ``YYYY-MM-DD`` string to a :class:`datetime.date` object."""
return datetime.datetime.strptime(value, '%Y-%m-%d').date() if value else None
def parse_date_wiki(value):
r"""Convert a string such as ``19 December 2018`` ``August 02, 2019\n`` to a :class:`datetime.date` object."""
value = parse_data_wiki(value)
if value == 'Unknown':
value = '30 April 2008'
if len(value) < 15:
if not value[:1].isdigit(): # Only Month Year
value = '30 ' + value
if len(value) < 5: # Only Year
value = '30 April ' + value
try:
return datetime.datetime.strptime(value, '%d %B %Y').date() if value else None
except ValueError:
return datetime.datetime.strptime(value, '%B %d, %Y').date() if value else None
def parse_version(value):
"""Convert a version string to a floating point number."""
for token in value.split():
try:
return decimal.Decimal(token)
except ValueError:
pass
msg = "Failed to convert version string to number! (%r)"
raise ValueError(msg % value)
def parse_version_wiki(value):
"""Convert a version string (got from wiki page) to a floating point number."""
value = parse_data_wiki(value)
value = value.split(": ")[1]
for token in value.split():
try:
return decimal.Decimal(token)
except ValueError:
pass
msg = "Failed to convert version string to number! (%r)"
raise ValueError(msg % value)
@cached
def ubuntu_keyring_updated():
"""
Detect update `#1363482`_ to the ``ubuntu-keyring`` package.
:returns: :data:`True` when version ``2016.10.27`` or newer is installed,
:data:`False` when an older version is installed.
This function checks if the changes discussed in Launchpad bug `#1363482`_
apply to the current system using the ``dpkg-query --show`` and ``dpkg
--compare-versions`` commands. For more details refer to `issue #8`_.
.. _#1363482: https://bugs.launchpad.net/ubuntu/+source/ubuntu-keyring/+bug/1363482
.. _issue #8: https://github.com/xolox/python-apt-mirror-updater/issues/8
"""
# Use external commands to check the installed version of the package.
version = execute('dpkg-query', '--show', '--showformat=${Version}', 'ubuntu-keyring', capture=True)
logger.debug("Detected ubuntu-keyring package version: %s", version)
updated = execute('dpkg', '--compare-versions', version, '>=', '2016.10.27', check=False, silent=True)
logger.debug("Does Launchpad bug #1363482 apply? %s", updated)
return updated
class Release(PropertyManager):
"""Data class for metadata on Debian and Ubuntu releases."""
@key_property
def codename(self):
"""The long version of :attr:`series` (a string)."""
@writable_property
def compatible_repository(self):
"""For Linux Mint, compatible which Ubuntu version's repository"""
@required_property
def created_date(self):
"""The date on which the release was created (a :class:`~datetime.date` object)."""
@key_property
def distributor_id(self):
"""The name of the distributor (a string like ``debian`` or ``ubuntu`` or ``linuxmint``)."""
@writable_property
def eol_date(self):
"""The date on which the desktop release stops being supported (a :class:`~datetime.date` object)."""
@writable_property
def extended_eol_date(self):
"""The date on which the server release stops being supported (a :class:`~datetime.date` object)."""
@lazy_property
def is_eol(self):
"""Whether the release has reached its end-of-life date (a boolean or :data:`None`)."""
eol_date = self.extended_eol_date or self.eol_date
if eol_date:
return datetime.date.today() >= eol_date
else:
return False
@writable_property
def is_lts(self):
"""Whether a release is a long term support release (a boolean)."""
@writable_property
def release_date(self):
"""The date on which the release was published (a :class:`~datetime.date` object)."""
@key_property
def series(self):
"""The short version of :attr:`codename` (a string)."""
@writable_property
def version(self):
"""
The version number of the release (a :class:`~decimal.Decimal` number).
This property has a :class:`~decimal.Decimal` value to enable proper
sorting based on numeric comparison.
"""
@lazy_property
def keyring_file(self):
"""
The pathname of the keyring with signing keys for this release (a string).
This property exists to work around a bug in ``debootstrap`` which may
use the wrong keyring to create Ubuntu chroots, for more details refer
to :func:`ubuntu_keyring_updated()`.
"""
filename = None
reason = None
logger.debug("Selecting keyring file for %s ..", self)
if self.distributor_id == 'debian':
filename = DEBIAN_KEYRING_CURRENT
reason = "only known keyring"
elif self.distributor_id == 'ubuntu':
if ubuntu_keyring_updated():
if self.version > decimal.Decimal('12.04'):
filename = UBUNTU_KEYRING_CURRENT
reason = "new keyring package / new release"
else:
filename = UBUNTU_KEYRING_REMOVED
reason = "new keyring package / old release"
else:
filename = UBUNTU_KEYRING_CURRENT
reason = "old keyring package"
else:
msg = "Unsupported distributor ID! (%s)"
raise EnvironmentError(msg % self.distributor_id)
logger.debug("Using %s (reason: %s).", filename, reason)
return filename
def __str__(self):
"""
Render a human friendly representation of a :class:`Release` object.
The result will be something like this:
- Debian 9 (stretch)
- Ubuntu 18.04 (bionic)
"""
label = [self.distributor_id.capitalize()]
if self.version:
label.append(str(self.version))
label.append("(%s)" % self.series)
return " ".join(label)
# [[[cog
#
# import cog
# import decimal
# from bs4 import BeautifulSoup
# from apt_smart.releases import discover_releases, discover_linuxmint_releases, table_to_2d
# from apt_smart.http import fetch_url
#
# set_of_releases = set()
# url = 'https://en.wikipedia.org/wiki/Linux_Mint_version_history'
# response = fetch_url(url, timeout=15, retry=True)
# soup = BeautifulSoup(response, 'html.parser')
# tables = soup.findAll('table')
# for release in discover_linuxmint_releases(table_to_2d(tables[1])):
# set_of_releases.add(release)
# set_of_releases.update(discover_releases())
# indent = " " * 4
# cog.out("\nBUNDLED_RELEASES = [\n")
# for release in sorted(set_of_releases, key=lambda r: (r.distributor_id, r.version or 0, r.series)):
# cog.out(indent + "Release(\n")
# for name in release.find_properties(cached=False):
# value = getattr(release, name)
# if value is not None:
# if isinstance(value, decimal.Decimal):
# # It seems weirdly inconsistency to me that this is needed
# # for decimal.Decimal() but not for datetime.date() but I
# # guess the simple explanation is that repr() output simply
# # isn't guaranteed to be accepted by eval().
# value = "decimal." + repr(value)
# else:
# value = repr(value)
# cog.out(indent * 2 + name + "=" + value + ",\n")
# cog.out(indent + "),\n")
# cog.out("]\n\n")
#
# ]]]
BUNDLED_RELEASES = [
Release(
codename='Experimental',
created_date=datetime.date(1993, 8, 16),
distributor_id='debian',
is_lts=False,
series='experimental',
),
Release(
codename='Sid',
created_date=datetime.date(1993, 8, 16),
distributor_id='debian',
is_lts=False,
series='sid',
),
Release(
codename='Buzz',
created_date=datetime.date(1993, 8, 16),
distributor_id='debian',
eol_date=datetime.date(1997, 6, 5),
is_lts=False,
release_date=datetime.date(1996, 6, 17),
series='buzz',
version=decimal.Decimal('1.1'),
),
Release(
codename='Rex',
created_date=datetime.date(1996, 6, 17),
distributor_id='debian',
eol_date=datetime.date(1998, 6, 5),
is_lts=False,
release_date=datetime.date(1996, 12, 12),
series='rex',
version=decimal.Decimal('1.2'),
),
Release(
codename='Bo',
created_date=datetime.date(1996, 12, 12),
distributor_id='debian',
eol_date=datetime.date(1999, 3, 9),
is_lts=False,
release_date=datetime.date(1997, 6, 5),
series='bo',
version=decimal.Decimal('1.3'),
),
Release(
codename='Hamm',
created_date=datetime.date(1997, 6, 5),
distributor_id='debian',
eol_date=datetime.date(2000, 3, 9),
is_lts=False,
release_date=datetime.date(1998, 7, 24),
series='hamm',
version=decimal.Decimal('2.0'),
),
Release(
codename='Slink',
created_date=datetime.date(1998, 7, 24),
distributor_id='debian',
eol_date=datetime.date(2000, 10, 30),
is_lts=False,
release_date=datetime.date(1999, 3, 9),
series='slink',
version=decimal.Decimal('2.1'),
),
Release(
codename='Potato',
created_date=datetime.date(1999, 3, 9),
distributor_id='debian',
eol_date=datetime.date(2003, 7, 30),
is_lts=False,
release_date=datetime.date(2000, 8, 15),
series='potato',
version=decimal.Decimal('2.2'),
),
Release(
codename='Woody',
created_date=datetime.date(2000, 8, 15),
distributor_id='debian',
eol_date=datetime.date(2006, 6, 30),
is_lts=False,
release_date=datetime.date(2002, 7, 19),
series='woody',
version=decimal.Decimal('3.0'),
),
Release(
codename='Sarge',
created_date=datetime.date(2002, 7, 19),
distributor_id='debian',
eol_date=datetime.date(2008, 3, 30),
is_lts=False,
release_date=datetime.date(2005, 6, 6),
series='sarge',
version=decimal.Decimal('3.1'),
),
Release(
codename='Etch',
created_date=datetime.date(2005, 6, 6),
distributor_id='debian',
eol_date=datetime.date(2010, 2, 15),
is_lts=False,
release_date=datetime.date(2007, 4, 8),
series='etch',
version=decimal.Decimal('4.0'),
),
Release(
codename='Lenny',
created_date=datetime.date(2007, 4, 8),
distributor_id='debian',
eol_date=datetime.date(2012, 2, 6),
is_lts=False,
release_date=datetime.date(2009, 2, 14),
series='lenny',
version=decimal.Decimal('5.0'),
),
Release(
codename='Squeeze',
created_date=datetime.date(2009, 2, 14),
distributor_id='debian',
eol_date=datetime.date(2014, 5, 31),
is_lts=False,
release_date=datetime.date(2011, 2, 6),
series='squeeze',
version=decimal.Decimal('6.0'),
),
Release(
codename='Wheezy',
created_date=datetime.date(2011, 2, 6),
distributor_id='debian',
eol_date=datetime.date(2016, 4, 26),
is_lts=False,
release_date=datetime.date(2013, 5, 4),
series='wheezy',
version=decimal.Decimal('7'),
),
Release(
codename='Jessie',
created_date=datetime.date(2013, 5, 4),
distributor_id='debian',
eol_date=datetime.date(2018, 6, 6),
extended_eol_date=datetime.date(2020, 6, 30),
is_lts=True,
release_date=datetime.date(2015, 4, 25),
series='jessie',
version=decimal.Decimal('8'),
),
Release(
codename='Stretch',
created_date=datetime.date(2015, 4, 25),
distributor_id='debian',
eol_date=datetime.date(2020, 7, 6),
extended_eol_date=datetime.date(2022, 6, 30),
is_lts=True,
release_date=datetime.date(2017, 6, 17),
series='stretch',
version=decimal.Decimal('9'),
),
Release(
codename='Buster',
created_date=datetime.date(2017, 6, 17),
distributor_id='debian',
is_lts=False,
release_date=datetime.date(2019, 7, 6),
series='buster',
version=decimal.Decimal('10'),
),
Release(
codename='Bullseye',
created_date=datetime.date(2019, 7, 6),
distributor_id='debian',
is_lts=False,
series='bullseye',
version=decimal.Decimal('11'),
),
Release(
codename='Bookworm',
created_date=datetime.date(2021, 8, 1),
distributor_id='debian',
is_lts=False,
series='bookworm',
version=decimal.Decimal('12'),
),
Release(
codename='Ada',
compatible_repository='Kubuntu 6.06',
created_date=datetime.date(2006, 8, 27),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 4, 30),
extended_eol_date=datetime.date(2008, 4, 30),
is_lts=False,
release_date=datetime.date(2006, 8, 27),
series='ada',
version=decimal.Decimal('1.0'),
),
Release(
codename='Barbara',
compatible_repository='edgy',
created_date=datetime.date(2006, 11, 13),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 4, 30),
extended_eol_date=datetime.date(2008, 4, 30),
is_lts=False,
release_date=datetime.date(2006, 11, 13),
series='barbara',
version=decimal.Decimal('2.0'),
),
Release(
codename='Bea',
compatible_repository='edgy',
created_date=datetime.date(2006, 12, 20),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 4, 30),
extended_eol_date=datetime.date(2008, 4, 30),
is_lts=False,
release_date=datetime.date(2006, 12, 20),
series='bea',
version=decimal.Decimal('2.1'),
),
Release(
codename='Bianca',
compatible_repository='edgy',
created_date=datetime.date(2007, 2, 20),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 4, 30),
extended_eol_date=datetime.date(2008, 4, 30),
is_lts=False,
release_date=datetime.date(2007, 2, 20),
series='bianca',
version=decimal.Decimal('2.2'),
),
Release(
codename='Cassandra',
compatible_repository='feisty',
created_date=datetime.date(2007, 5, 30),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 10, 30),
extended_eol_date=datetime.date(2008, 10, 30),
is_lts=False,
release_date=datetime.date(2007, 5, 30),
series='cassandra',
version=decimal.Decimal('3.0'),
),
Release(
codename='Celena',
compatible_repository='feisty',
created_date=datetime.date(2007, 9, 24),
distributor_id='linuxmint',
eol_date=datetime.date(2008, 10, 30),
extended_eol_date=datetime.date(2008, 10, 30),
is_lts=False,
release_date=datetime.date(2007, 9, 24),
series='celena',
version=decimal.Decimal('3.1'),
),
Release(
codename='Daryna',
compatible_repository='gutsy',
created_date=datetime.date(2007, 10, 15),
distributor_id='linuxmint',
eol_date=datetime.date(2009, 4, 30),
extended_eol_date=datetime.date(2009, 4, 30),
is_lts=False,
release_date=datetime.date(2007, 10, 15),
series='daryna',
version=decimal.Decimal('4.0'),
),
Release(
codename='Elyssa',
compatible_repository='hardy',
created_date=datetime.date(2008, 6, 8),
distributor_id='linuxmint',
eol_date=datetime.date(2011, 4, 30),
extended_eol_date=datetime.date(2011, 4, 30),
is_lts=True,
release_date=datetime.date(2008, 6, 8),
series='elyssa',
version=decimal.Decimal('5'),
),
Release(
codename='Felicia',
compatible_repository='intrepid',
created_date=datetime.date(2008, 12, 15),
distributor_id='linuxmint',
eol_date=datetime.date(2010, 4, 30),
extended_eol_date=datetime.date(2010, 4, 30),
is_lts=False,
release_date=datetime.date(2008, 12, 15),
series='felicia',
version=decimal.Decimal('6'),
),
Release(
codename='Gloria',
compatible_repository='jaunty',
created_date=datetime.date(2009, 5, 26),
distributor_id='linuxmint',
eol_date=datetime.date(2010, 10, 30),
extended_eol_date=datetime.date(2010, 10, 30),
is_lts=False,
release_date=datetime.date(2009, 5, 26),
series='gloria',
version=decimal.Decimal('7'),
),
Release(
codename='Helena',
compatible_repository='karmic',
created_date=datetime.date(2009, 11, 28),
distributor_id='linuxmint',
eol_date=datetime.date(2011, 4, 30),
extended_eol_date=datetime.date(2011, 4, 30),
is_lts=False,
release_date=datetime.date(2009, 11, 28),
series='helena',
version=decimal.Decimal('8'),
),
Release(
codename='Isadora',
compatible_repository='lucid',
created_date=datetime.date(2010, 5, 18),
distributor_id='linuxmint',
eol_date=datetime.date(2013, 4, 30),
extended_eol_date=datetime.date(2013, 4, 30),
is_lts=True,
release_date=datetime.date(2010, 5, 18),
series='isadora',
version=decimal.Decimal('9'),
),
Release(
codename='Julia',
compatible_repository='maverick',
created_date=datetime.date(2010, 11, 12),
distributor_id='linuxmint',
eol_date=datetime.date(2012, 4, 30),
extended_eol_date=datetime.date(2012, 4, 30),
is_lts=False,
release_date=datetime.date(2010, 11, 12),
series='julia',
version=decimal.Decimal('10'),
),
Release(
codename='Katya',
compatible_repository='natty',
created_date=datetime.date(2011, 5, 26),
distributor_id='linuxmint',
eol_date=datetime.date(2012, 10, 30),
extended_eol_date=datetime.date(2012, 10, 30),
is_lts=False,
release_date=datetime.date(2011, 5, 26),
series='katya',
version=decimal.Decimal('11'),
),
Release(
codename='Lisa',
compatible_repository='oneiric',
created_date=datetime.date(2011, 11, 26),
distributor_id='linuxmint',
eol_date=datetime.date(2013, 4, 30),
extended_eol_date=datetime.date(2013, 4, 30),
is_lts=False,
release_date=datetime.date(2011, 11, 26),
series='lisa',
version=decimal.Decimal('12'),
),
Release(
codename='Maya',
compatible_repository='precise',
created_date=datetime.date(2012, 5, 23),
distributor_id='linuxmint',
eol_date=datetime.date(2017, 4, 30),
extended_eol_date=datetime.date(2017, 4, 30),
is_lts=True,
release_date=datetime.date(2012, 5, 23),
series='maya',
version=decimal.Decimal('13'),
),
Release(
codename='Nadia',
compatible_repository='quantal',
created_date=datetime.date(2012, 11, 20),
distributor_id='linuxmint',
eol_date=datetime.date(2014, 5, 30),
extended_eol_date=datetime.date(2014, 5, 30),
is_lts=False,
release_date=datetime.date(2012, 11, 20),
series='nadia',
version=decimal.Decimal('14'),
),
Release(
codename='Olivia',
compatible_repository='raring',
created_date=datetime.date(2013, 5, 29),
distributor_id='linuxmint',
eol_date=datetime.date(2014, 1, 30),
extended_eol_date=datetime.date(2014, 1, 30),
is_lts=False,
release_date=datetime.date(2013, 5, 29),
series='olivia',
version=decimal.Decimal('15'),
),
Release(
codename='Petra',
compatible_repository='saucy',
created_date=datetime.date(2013, 11, 30),
distributor_id='linuxmint',
eol_date=datetime.date(2014, 7, 30),
extended_eol_date=datetime.date(2014, 7, 30),
is_lts=False,
release_date=datetime.date(2013, 11, 30),
series='petra',
version=decimal.Decimal('16'),
),
Release(
codename='Qiana',
compatible_repository='trusty',
created_date=datetime.date(2014, 5, 31),
distributor_id='linuxmint',
eol_date=datetime.date(2019, 4, 30),
extended_eol_date=datetime.date(2019, 4, 30),
is_lts=True,
release_date=datetime.date(2014, 5, 31),
series='qiana',
version=decimal.Decimal('17'),
),
Release(
codename='Rebecca',
compatible_repository='trusty',
created_date=datetime.date(2014, 11, 29),
distributor_id='linuxmint',
eol_date=datetime.date(2019, 4, 30),
extended_eol_date=datetime.date(2019, 4, 30),
is_lts=True,
release_date=datetime.date(2014, 11, 29),
series='rebecca',
version=decimal.Decimal('17.1'),
),
Release(
codename='Rafaela',
compatible_repository='trusty',
created_date=datetime.date(2015, 6, 30),
distributor_id='linuxmint',
eol_date=datetime.date(2019, 4, 30),
extended_eol_date=datetime.date(2019, 4, 30),
is_lts=True,
release_date=datetime.date(2015, 6, 30),
series='rafaela',
version=decimal.Decimal('17.2'),
),
Release(
codename='Rosa',
compatible_repository='trusty',
created_date=datetime.date(2015, 12, 4),
distributor_id='linuxmint',
eol_date=datetime.date(2019, 4, 30),
extended_eol_date=datetime.date(2019, 4, 30),
is_lts=True,
release_date=datetime.date(2015, 12, 4),
series='rosa',
version=decimal.Decimal('17.3'),
),
Release(
codename='Sarah',
compatible_repository='xenial',
created_date=datetime.date(2016, 6, 30),
distributor_id='linuxmint',
eol_date=datetime.date(2021, 4, 30),
extended_eol_date=datetime.date(2021, 4, 30),
is_lts=True,
release_date=datetime.date(2016, 6, 30),
series='sarah',
version=decimal.Decimal('18'),
),
Release(
codename='Serena',
compatible_repository='xenial',
created_date=datetime.date(2016, 12, 16),
distributor_id='linuxmint',
eol_date=datetime.date(2021, 4, 30),
extended_eol_date=datetime.date(2021, 4, 30),
is_lts=True,
release_date=datetime.date(2016, 12, 16),
series='serena',
version=decimal.Decimal('18.1'),
),
Release(
codename='Sonya',
compatible_repository='xenial',
created_date=datetime.date(2017, 7, 2),
distributor_id='linuxmint',
eol_date=datetime.date(2021, 4, 30),
extended_eol_date=datetime.date(2021, 4, 30),
is_lts=True,
release_date=datetime.date(2017, 7, 2),
series='sonya',
version=decimal.Decimal('18.2'),
),
Release(
codename='Sylvia',
compatible_repository='xenial',
created_date=datetime.date(2017, 11, 27),
distributor_id='linuxmint',
eol_date=datetime.date(2021, 4, 30),
extended_eol_date=datetime.date(2021, 4, 30),
is_lts=True,
release_date=datetime.date(2017, 11, 27),
series='sylvia',
version=decimal.Decimal('18.3'),
),
Release(
codename='Tara',
compatible_repository='bionic',
created_date=datetime.date(2018, 6, 29),
distributor_id='linuxmint',
eol_date=datetime.date(2023, 4, 30),
extended_eol_date=datetime.date(2023, 4, 30),
is_lts=True,
release_date=datetime.date(2018, 6, 29),
series='tara',
version=decimal.Decimal('19'),
),
Release(
codename='Tessa',
compatible_repository='bionic',
created_date=datetime.date(2018, 12, 19),
distributor_id='linuxmint',
eol_date=datetime.date(2023, 4, 30),
extended_eol_date=datetime.date(2023, 4, 30),
is_lts=True,
release_date=datetime.date(2018, 12, 19),
series='tessa',
version=decimal.Decimal('19.1'),
),
Release(
codename='Tina',
compatible_repository='bionic',
created_date=datetime.date(2019, 8, 2),
distributor_id='linuxmint',
eol_date=datetime.date(2023, 4, 30),
extended_eol_date=datetime.date(2023, 4, 30),
is_lts=True,
release_date=datetime.date(2019, 8, 2),
series='tina',
version=decimal.Decimal('19.2'),
),
Release(
codename='Tricia',
compatible_repository='bionic',
created_date=datetime.date(2019, 12, 18),
distributor_id='linuxmint',
eol_date=datetime.date(2023, 4, 30),
extended_eol_date=datetime.date(2023, 4, 30),
is_lts=True,
release_date=datetime.date(2019, 12, 18),
series='tricia',
version=decimal.Decimal('19.3'),
),
Release(
codename='Ulyana',
compatible_repository='focal',
created_date=datetime.date(2020, 6, 27),
distributor_id='linuxmint',
eol_date=datetime.date(2025, 4, 30),
extended_eol_date=datetime.date(2025, 4, 30),
is_lts=True,
release_date=datetime.date(2020, 6, 27),
series='ulyana',
version=decimal.Decimal('20'),
),
Release(
codename='Warty Warthog',
created_date=datetime.date(2004, 3, 5),
distributor_id='ubuntu',
eol_date=datetime.date(2006, 4, 30),
is_lts=False,
release_date=datetime.date(2004, 10, 20),
series='warty',
version=decimal.Decimal('4.10'),
),
Release(
codename='Hoary Hedgehog',
created_date=datetime.date(2004, 10, 20),
distributor_id='ubuntu',
eol_date=datetime.date(2006, 10, 31),
is_lts=False,
release_date=datetime.date(2005, 4, 8),
series='hoary',
version=decimal.Decimal('5.04'),
),
Release(
codename='Breezy Badger',
created_date=datetime.date(2005, 4, 8),
distributor_id='ubuntu',
eol_date=datetime.date(2007, 4, 13),
is_lts=False,
release_date=datetime.date(2005, 10, 12),
series='breezy',
version=decimal.Decimal('5.10'),
),
Release(
codename='Dapper Drake',
created_date=datetime.date(2005, 10, 12),
distributor_id='ubuntu',
eol_date=datetime.date(2009, 7, 14),
extended_eol_date=datetime.date(2011, 6, 1),
is_lts=True,
release_date=datetime.date(2006, 6, 1),
series='dapper',
version=decimal.Decimal('6.06'),
),
Release(
codename='Edgy Eft',
created_date=datetime.date(2006, 6, 1),
distributor_id='ubuntu',
eol_date=datetime.date(2008, 4, 25),
is_lts=False,
release_date=datetime.date(2006, 10, 26),
series='edgy',
version=decimal.Decimal('6.10'),
),
Release(
codename='Feisty Fawn',
created_date=datetime.date(2006, 10, 26),
distributor_id='ubuntu',
eol_date=datetime.date(2008, 10, 19),
is_lts=False,
release_date=datetime.date(2007, 4, 19),
series='feisty',
version=decimal.Decimal('7.04'),
),
Release(
codename='Gutsy Gibbon',
created_date=datetime.date(2007, 4, 19),
distributor_id='ubuntu',
eol_date=datetime.date(2009, 4, 18),
is_lts=False,
release_date=datetime.date(2007, 10, 18),
series='gutsy',
version=decimal.Decimal('7.10'),
),
Release(
codename='Hardy Heron',
created_date=datetime.date(2007, 10, 18),
distributor_id='ubuntu',
eol_date=datetime.date(2011, 5, 12),
extended_eol_date=datetime.date(2013, 5, 9),
is_lts=True,
release_date=datetime.date(2008, 4, 24),
series='hardy',
version=decimal.Decimal('8.04'),
),
Release(
codename='Intrepid Ibex',
created_date=datetime.date(2008, 4, 24),
distributor_id='ubuntu',
eol_date=datetime.date(2010, 4, 30),
is_lts=False,
release_date=datetime.date(2008, 10, 30),
series='intrepid',
version=decimal.Decimal('8.10'),
),
Release(
codename='Jaunty Jackalope',
created_date=datetime.date(2008, 10, 30),
distributor_id='ubuntu',
eol_date=datetime.date(2010, 10, 23),
is_lts=False,
release_date=datetime.date(2009, 4, 23),
series='jaunty',
version=decimal.Decimal('9.04'),
),
Release(
codename='Karmic Koala',
created_date=datetime.date(2009, 4, 23),
distributor_id='ubuntu',
eol_date=datetime.date(2011, 4, 29),
is_lts=False,
release_date=datetime.date(2009, 10, 29),
series='karmic',
version=decimal.Decimal('9.10'),
),
Release(
codename='Lucid Lynx',
created_date=datetime.date(2009, 10, 29),
distributor_id='ubuntu',
eol_date=datetime.date(2013, 5, 9),
extended_eol_date=datetime.date(2015, 4, 29),
is_lts=True,
release_date=datetime.date(2010, 4, 29),
series='lucid',
version=decimal.Decimal('10.04'),
),
Release(
codename='Maverick Meerkat',
created_date=datetime.date(2010, 4, 29),
distributor_id='ubuntu',
eol_date=datetime.date(2012, 4, 10),
is_lts=False,
release_date=datetime.date(2010, 10, 10),
series='maverick',
version=decimal.Decimal('10.10'),
),
Release(
codename='Natty Narwhal',
created_date=datetime.date(2010, 10, 10),
distributor_id='ubuntu',
eol_date=datetime.date(2012, 10, 28),
is_lts=False,
release_date=datetime.date(2011, 4, 28),
series='natty',
version=decimal.Decimal('11.04'),
),
Release(
codename='Oneiric Ocelot',
created_date=datetime.date(2011, 4, 28),
distributor_id='ubuntu',
eol_date=datetime.date(2013, 5, 9),
is_lts=False,
release_date=datetime.date(2011, 10, 13),
series='oneiric',
version=decimal.Decimal('11.10'),
),
Release(
codename='Precise Pangolin',
created_date=datetime.date(2011, 10, 13),
distributor_id='ubuntu',
eol_date=datetime.date(2017, 4, 26),
extended_eol_date=datetime.date(2017, 4, 26),
is_lts=True,
release_date=datetime.date(2012, 4, 26),
series='precise',
version=decimal.Decimal('12.04'),
),
Release(
codename='Quantal Quetzal',
created_date=datetime.date(2012, 4, 26),
distributor_id='ubuntu',
eol_date=datetime.date(2014, 5, 16),
is_lts=False,
release_date=datetime.date(2012, 10, 18),
series='quantal',
version=decimal.Decimal('12.10'),
),
Release(
codename='Raring Ringtail',
created_date=datetime.date(2012, 10, 18),
distributor_id='ubuntu',
eol_date=datetime.date(2014, 1, 27),
is_lts=False,
release_date=datetime.date(2013, 4, 25),
series='raring',
version=decimal.Decimal('13.04'),
),
Release(
codename='Saucy Salamander',
created_date=datetime.date(2013, 4, 25),
distributor_id='ubuntu',
eol_date=datetime.date(2014, 7, 17),
is_lts=False,
release_date=datetime.date(2013, 10, 17),
series='saucy',
version=decimal.Decimal('13.10'),
),
Release(
codename='Trusty Tahr',
created_date=datetime.date(2013, 10, 17),
distributor_id='ubuntu',
eol_date=datetime.date(2019, 4, 25),
extended_eol_date=datetime.date(2019, 4, 25),
is_lts=True,
release_date=datetime.date(2014, 4, 17),
series='trusty',
version=decimal.Decimal('14.04'),
),
Release(
codename='Utopic Unicorn',
created_date=datetime.date(2014, 4, 17),
distributor_id='ubuntu',
eol_date=datetime.date(2015, 7, 23),
is_lts=False,
release_date=datetime.date(2014, 10, 23),
series='utopic',
version=decimal.Decimal('14.10'),
),
Release(
codename='Vivid Vervet',
created_date=datetime.date(2014, 10, 23),
distributor_id='ubuntu',
eol_date=datetime.date(2016, 1, 23),
is_lts=False,
release_date=datetime.date(2015, 4, 23),
series='vivid',
version=decimal.Decimal('15.04'),
),
Release(
codename='Wily Werewolf',
created_date=datetime.date(2015, 4, 23),
distributor_id='ubuntu',
eol_date=datetime.date(2016, 7, 22),
is_lts=False,
release_date=datetime.date(2015, 10, 22),
series='wily',
version=decimal.Decimal('15.10'),
),
Release(
codename='Xenial Xerus',
created_date=datetime.date(2015, 10, 22),
distributor_id='ubuntu',
eol_date=datetime.date(2021, 4, 21),
extended_eol_date=datetime.date(2021, 4, 21),
is_lts=True,
release_date=datetime.date(2016, 4, 21),
series='xenial',
version=decimal.Decimal('16.04'),
),
Release(
codename='Yakkety Yak',
created_date=datetime.date(2016, 4, 21),
distributor_id='ubuntu',
eol_date=datetime.date(2017, 7, 20),
is_lts=False,
release_date=datetime.date(2016, 10, 13),
series='yakkety',
version=decimal.Decimal('16.10'),
),
Release(
codename='Zesty Zapus',
created_date=datetime.date(2016, 10, 13),
distributor_id='ubuntu',
eol_date=datetime.date(2018, 1, 13),
is_lts=False,
release_date=datetime.date(2017, 4, 13),
series='zesty',
version=decimal.Decimal('17.04'),
),
Release(
codename='Artful Aardvark',
created_date=datetime.date(2017, 4, 13),
distributor_id='ubuntu',
eol_date=datetime.date(2018, 7, 19),
is_lts=False,
release_date=datetime.date(2017, 10, 19),
series='artful',
version=decimal.Decimal('17.10'),
),
Release(
codename='Bionic Beaver',
created_date=datetime.date(2017, 10, 19),
distributor_id='ubuntu',
eol_date=datetime.date(2023, 4, 26),
extended_eol_date=datetime.date(2023, 4, 26),
is_lts=True,
release_date=datetime.date(2018, 4, 26),
series='bionic',
version=decimal.Decimal('18.04'),
),
Release(
codename='Cosmic Cuttlefish',
created_date=datetime.date(2018, 4, 26),
distributor_id='ubuntu',
eol_date=datetime.date(2019, 7, 18),
is_lts=False,
release_date=datetime.date(2018, 10, 18),
series='cosmic',
version=decimal.Decimal('18.10'),
),
Release(
codename='Disco Dingo',
created_date=datetime.date(2018, 10, 18),
distributor_id='ubuntu',
eol_date=datetime.date(2020, 1, 18),
is_lts=False,
release_date=datetime.date(2019, 4, 18),
series='disco',
version=decimal.Decimal('19.04'),
),
Release(
codename='Eoan Ermine',
created_date=datetime.date(2019, 4, 18),
distributor_id='ubuntu',
eol_date=datetime.date(2020, 7, 17),
is_lts=False,
release_date=datetime.date(2019, 10, 17),
series='eoan',
version=decimal.Decimal('19.10'),
),
Release(
codename='Focal Fossa',
created_date=datetime.date(2019, 10, 17),
distributor_id='ubuntu',
eol_date=datetime.date(2025, 4, 23),
extended_eol_date=datetime.date(2025, 4, 23),
is_lts=True,
release_date=datetime.date(2020, 4, 23),
series='focal',
version=decimal.Decimal('20.04'),
),
Release(
codename='Groovy Gorilla',
created_date=datetime.date(2020, 4, 23),
distributor_id='ubuntu',
eol_date=datetime.date(2021, 7, 22),
is_lts=False,
release_date=datetime.date(2020, 10, 22),
series='groovy',
version=decimal.Decimal('20.10'),
),
]
# [[[end]]]
================================================
FILE: apt_smart/tests.py
================================================
# Automated, robust apt-get mirror selection for Debian and Ubuntu.
#
# Author: martin68 and Peter Odding
# Last Change: September 15, 2019
# URL: https://apt-smart.readthedocs.io
"""Test suite for the ``apt-smart`` package."""
# Standard library modules.
import decimal
import logging
import os
import time
# External dependencies.
from executor import execute
from humanfriendly.testing import TestCase, run_cli
# from humanfriendly.text import split
# Modules included in our package.
from apt_smart import AptMirrorUpdater, normalize_mirror_url, MirrorStatus
from apt_smart.cli import main
from apt_smart.releases import (
DEBIAN_KEYRING_CURRENT,
UBUNTU_KEYRING_CURRENT,
UBUNTU_KEYRING_REMOVED,
coerce_release,
discover_releases,
ubuntu_keyring_updated,
)
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
class AptMirrorUpdaterTestCase(TestCase):
""":mod:`unittest` compatible container for the :mod:`apt_smart` test suite."""
def test_debian_mirror_discovery(self):
"""Test the discovery of Debian mirror URLs."""
from apt_smart.backends.debian import discover_mirrors
mirrors = discover_mirrors()
assert len(mirrors) > 6
for candidate in mirrors:
check_debian_mirror(candidate.mirror_url)
def test_ubuntu_mirror_discovery(self):
"""Test the discovery of Ubuntu mirror URLs."""
from apt_smart.backends.ubuntu import discover_mirrors
mirrors = discover_mirrors()
assert len(mirrors) > 10
for candidate in mirrors:
check_ubuntu_mirror(candidate.mirror_url)
def test_ubuntu_mirr
gitextract_79k4xdni/ ├── .github/ │ └── workflows/ │ └── python-ci.yml ├── .gitignore ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README-zh-cn.rst ├── README.rst ├── apt_smart/ │ ├── __init__.py │ ├── backends/ │ │ ├── __init__.py │ │ ├── debian.py │ │ ├── linuxmint.py │ │ └── ubuntu.py │ ├── cli.py │ ├── http.py │ ├── releases.py │ └── tests.py ├── constraints.txt ├── docs/ │ ├── api.rst │ ├── changelog.rst │ ├── conf.py │ ├── index.rst │ └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── scripts/ │ ├── check-code-style.sh │ ├── collect-full-coverage.sh │ └── install-on-travis.sh ├── setup.cfg ├── setup.py ├── test_custom_mirrors.txt └── tox.ini
SYMBOL INDEX (135 symbols across 9 files)
FILE: apt_smart/__init__.py
class AptMirrorUpdater (line 79) | class AptMirrorUpdater(PropertyManager):
method architecture (line 113) | def architecture(self):
method available_mirrors (line 128) | def available_mirrors(self):
method backend (line 177) | def backend(self):
method best_mirror (line 195) | def best_mirror(self):
method blacklist (line 211) | def blacklist(self):
method concurrency (line 221) | def concurrency(self):
method context (line 231) | def context(self):
method current_mirror (line 241) | def current_mirror(self):
method distribution_codename_old (line 256) | def distribution_codename_old(self):
method distribution_codename (line 271) | def distribution_codename(self):
method distributor_id (line 301) | def distributor_id(self):
method main_sources_list (line 316) | def main_sources_list(self):
method max_mirrors (line 331) | def max_mirrors(self):
method url_char_len (line 336) | def url_char_len(self):
method ubuntu_mode (line 345) | def ubuntu_mode(self):
method old_releases_url (line 354) | def old_releases_url(self):
method base_url (line 359) | def base_url(self):
method base_last_updated (line 364) | def base_last_updated(self):
method ranked_mirrors (line 372) | def ranked_mirrors(self):
method release (line 445) | def release(self):
method release_is_eol (line 450) | def release_is_eol(self):
method security_url (line 513) | def security_url(self):
method stable_mirror (line 518) | def stable_mirror(self):
method validated_mirrors (line 544) | def validated_mirrors(self):
method custom_mirror_file_path (line 549) | def custom_mirror_file_path(self):
method read_custom_mirror_file (line 554) | def read_custom_mirror_file(self):
method change_mirror (line 573) | def change_mirror(self, new_mirror=None, update=True):
method clear_package_lists (line 626) | def clear_package_lists(self):
method create_chroot (line 638) | def create_chroot(self, directory, codename=None, arch=None):
method dumb_update (line 745) | def dumb_update(self, *args):
method generate_sources_list (line 760) | def generate_sources_list(self, **options):
method get_sources_list_options (line 779) | def get_sources_list_options(self):
method get_sources_list (line 793) | def get_sources_list(self):
method ignore_mirror (line 821) | def ignore_mirror(self, pattern):
method install_sources_list (line 843) | def install_sources_list(self, contents):
method smart_update (line 881) | def smart_update(self, *args, **kw):
method validate_mirror (line 953) | def validate_mirror(self, mirror_url):
class CandidateMirror (line 987) | class CandidateMirror(PropertyManager):
method bandwidth (line 992) | def bandwidth(self):
method archive_update_in_progress_url (line 1003) | def archive_update_in_progress_url(self):
method mirror_url (line 1015) | def mirror_url(self):
method mirror_url (line 1019) | def mirror_url(self, value):
method is_available (line 1024) | def is_available(self):
method is_updating (line 1072) | def is_updating(self):
method last_updated (line 1076) | def last_updated(self):
method release_gpg_contents (line 1080) | def release_gpg_contents(self):
method release_gpg_latency (line 1091) | def release_gpg_latency(self):
method release_gpg_url (line 1102) | def release_gpg_url(self):
method sort_key (line 1125) | def sort_key(self):
method updater (line 1153) | def updater(self):
class MirrorStatus (line 1157) | class MirrorStatus(Enum):
function find_current_mirror (line 1171) | def find_current_mirror(sources_list):
function mirrors_are_equal (line 1198) | def mirrors_are_equal(a, b):
function normalize_mirror_url (line 1210) | def normalize_mirror_url(url):
FILE: apt_smart/backends/debian.py
function discover_mirrors (line 78) | def discover_mirrors():
function generate_sources_list (line 150) | def generate_sources_list(mirror_url, codename,
function get_eol_date (line 197) | def get_eol_date(updater):
FILE: apt_smart/backends/linuxmint.py
function discover_mirrors (line 57) | def discover_mirrors():
FILE: apt_smart/backends/ubuntu.py
function discover_mirrors_old (line 63) | def discover_mirrors_old():
function discover_mirrors (line 150) | def discover_mirrors():
function discover_mirror_selection (line 180) | def discover_mirror_selection():
function generate_sources_list (line 196) | def generate_sources_list(mirror_url, codename,
FILE: apt_smart/cli.py
function main (line 125) | def main():
function report_current_mirror (line 217) | def report_current_mirror(updater):
function report_best_mirror (line 222) | def report_best_mirror(updater):
function report_available_mirrors (line 227) | def report_available_mirrors(updater):
FILE: apt_smart/http.py
function fetch_url (line 26) | def fetch_url(url, timeout=10, retry=False, max_attempts=3):
function fetch_concurrent (line 71) | def fetch_concurrent(urls, concurrency=None):
function get_default_concurrency (line 89) | def get_default_concurrency():
function fetch_worker (line 98) | def fetch_worker(url):
class InvalidResponseError (line 126) | class InvalidResponseError(Exception):
class NotFoundError (line 131) | class NotFoundError(InvalidResponseError):
FILE: apt_smart/releases.py
function coerce_release (line 88) | def coerce_release(value):
function discover_releases (line 125) | def discover_releases():
function table_to_2d (line 156) | def table_to_2d(table_tag): # https://stackoverflow.com/a/48451104
function discover_linuxmint_releases (line 216) | def discover_linuxmint_releases(array_2d):
function is_version_string (line 245) | def is_version_string(value):
function parse_csv_file (line 253) | def parse_csv_file(filename):
function parse_data_wiki (line 292) | def parse_data_wiki(value):
function parse_date (line 300) | def parse_date(value):
function parse_date_wiki (line 305) | def parse_date_wiki(value):
function parse_version (line 321) | def parse_version(value):
function parse_version_wiki (line 332) | def parse_version_wiki(value):
function ubuntu_keyring_updated (line 346) | def ubuntu_keyring_updated():
class Release (line 368) | class Release(PropertyManager):
method codename (line 373) | def codename(self):
method compatible_repository (line 377) | def compatible_repository(self):
method created_date (line 381) | def created_date(self):
method distributor_id (line 385) | def distributor_id(self):
method eol_date (line 389) | def eol_date(self):
method extended_eol_date (line 393) | def extended_eol_date(self):
method is_eol (line 397) | def is_eol(self):
method is_lts (line 406) | def is_lts(self):
method release_date (line 410) | def release_date(self):
method series (line 414) | def series(self):
method version (line 418) | def version(self):
method keyring_file (line 427) | def keyring_file(self):
method __str__ (line 458) | def __str__(self):
FILE: apt_smart/tests.py
class AptMirrorUpdaterTestCase (line 36) | class AptMirrorUpdaterTestCase(TestCase):
method test_debian_mirror_discovery (line 40) | def test_debian_mirror_discovery(self):
method test_ubuntu_mirror_discovery (line 48) | def test_ubuntu_mirror_discovery(self):
method test_ubuntu_mirror_discovery_old (line 56) | def test_ubuntu_mirror_discovery_old(self):
method test_linuxmint_mirror_discovery (line 64) | def test_linuxmint_mirror_discovery(self):
method test_adaptive_mirror_discovery (line 72) | def test_adaptive_mirror_discovery(self):
method test_mirror_ranking (line 79) | def test_mirror_ranking(self):
method test_best_mirror_selection (line 85) | def test_best_mirror_selection(self):
method test_current_mirror_discovery (line 90) | def test_current_mirror_discovery(self):
method test_create_chroot (line 96) | def test_create_chroot(self):
method test_create_chroot_with_codename (line 104) | def test_create_chroot_with_codename(self):
method test_change_mirror (line 112) | def test_change_mirror(self):
method test_report_available_mirrors (line 120) | def test_report_available_mirrors(self):
method test_report_available_mirrors_more (line 125) | def test_report_available_mirrors_more(self):
method test_dumb_update (line 132) | def test_dumb_update(self):
method test_smart_update (line 146) | def test_smart_update(self):
method test_discover_releases (line 165) | def test_discover_releases(self):
method test_discover_linuxmint_releases (line 185) | def test_discover_linuxmint_releases(self):
method test_coerce_release (line 231) | def test_coerce_release(self):
method test_keyring_selection (line 242) | def test_keyring_selection(self):
method test_debian_lts_eol_date (line 257) | def test_debian_lts_eol_date(self):
method test_trusty_eol (line 271) | def test_trusty_eol(self):
function have_package_lists (line 285) | def have_package_lists():
function check_mirror_url (line 299) | def check_mirror_url(url):
function check_debian_mirror (line 306) | def check_debian_mirror(url):
function check_ubuntu_mirror (line 313) | def check_ubuntu_mirror(url):
function is_debian_mirror (line 320) | def is_debian_mirror(url):
function is_ubuntu_mirror (line 331) | def is_ubuntu_mirror(url):
function has_compatible_scheme (line 345) | def has_compatible_scheme(url):
FILE: setup.py
function get_contents (line 32) | def get_contents(*args):
function get_version (line 38) | def get_version(*args):
function get_install_requires (line 45) | def get_install_requires():
function get_extras_require (line 57) | def get_extras_require():
function get_requirements (line 75) | def get_requirements(*args):
function get_absolute_path (line 88) | def get_absolute_path(*args):
function have_environment_marker_support (line 93) | def have_environment_marker_support():
Condensed preview — 35 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (235K chars).
[
{
"path": ".github/workflows/python-ci.yml",
"chars": 887,
"preview": "name: Python CI\n\non:\n push:\n branches-ignore:\n - '/^[0-9]/'\n pull_request:\n branches-ignore:\n - '/^[0-"
},
{
"path": ".gitignore",
"chars": 57,
"preview": ".cache/\n.coverage\n.coverage*\n.directory\n*.egg-info/\n*.pyc"
},
{
"path": ".travis.yml",
"chars": 259,
"preview": "sudo: true\nlanguage: python\npython:\n - \"2.7\"\n - \"3.4\"\n - \"3.5\"\n - \"3.6\"\n - \"3.7\"\n - \"3.8\"\n - \"pypy\"\ninstall:\n - "
},
{
"path": "CHANGELOG.rst",
"chars": 18260,
"preview": "Changelog\n=========\n\nThe purpose of this document is to list all of the notable changes to this\nproject. The format was "
},
{
"path": "LICENSE.txt",
"chars": 1070,
"preview": "Copyright (c) 2019 martin68 2018 Peter Odding\n\nPermission is hereby granted, free of charge, to any person obtaining\na c"
},
{
"path": "MANIFEST.in",
"chars": 39,
"preview": "include *.rst\ninclude *.txt\ngraft docs\n"
},
{
"path": "Makefile",
"chars": 2831,
"preview": "# Makefile for the 'apt-smart' package.\n#\n# Author: martin68 and Peter Odding\n# Last Change: September 15, 2019\n# URL: h"
},
{
"path": "README-zh-cn.rst",
"chars": 10670,
"preview": "apt-smart: 智能的 Debian/Ubuntu/Linux Mint 镜像源自动选择工具\n=================================================================\n\n.. "
},
{
"path": "README.rst",
"chars": 13832,
"preview": "apt-smart: Smart, automated Debian/Ubuntu/Linux Mint mirror selection\n=================================================="
},
{
"path": "apt_smart/__init__.py",
"chars": 57493,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "apt_smart/backends/__init__.py",
"chars": 76,
"preview": "\"\"\"Namespace for platform specific mirror discovery in :mod:`apt_smart`.\"\"\"\n"
},
{
"path": "apt_smart/backends/debian.py",
"chars": 8766,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "apt_smart/backends/linuxmint.py",
"chars": 5580,
"preview": "# Automated, robust apt-get mirror selection for Debian ,Ubuntu and Linux Mint.\n#\n# Author: martin68 and Peter Odding\n# "
},
{
"path": "apt_smart/backends/ubuntu.py",
"chars": 10456,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "apt_smart/cli.py",
"chars": 10956,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "apt_smart/http.py",
"chars": 5082,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "apt_smart/releases.py",
"chars": 49769,
"preview": "# Easy to use metadata on Debian and Ubuntu releases.\n#\n# Author: martin68 and Peter Odding\n# Last Change: May 31, 2020\n"
},
{
"path": "apt_smart/tests.py",
"chars": 14714,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "constraints.txt",
"chars": 1147,
"preview": "# This is a pip constraints file that is used to preserve Python 2.6\n# compatibility (on Travis CI). Why I'm still doing"
},
{
"path": "docs/api.rst",
"chars": 859,
"preview": "API documentation\n=================\n\nThe following documentation is based on the source code of version |release| of\nthe"
},
{
"path": "docs/changelog.rst",
"chars": 30,
"preview": ".. include:: ../CHANGELOG.rst\n"
},
{
"path": "docs/conf.py",
"chars": 2644,
"preview": "# Automated, robust apt-get mirror selection for Debian and Ubuntu.\n#\n# Author: martin68 and Peter Odding\n# Last Change:"
},
{
"path": "docs/index.rst",
"chars": 760,
"preview": "apt-smart: Smart, automated Debian/Ubuntu/Linux Mint mirror selection\n=================================================="
},
{
"path": "docs/readme.rst",
"chars": 27,
"preview": ".. include:: ../README.rst\n"
},
{
"path": "requirements-checks.txt",
"chars": 109,
"preview": "# Python packages required to run `make check'.\nflake8 >= 2.6.0\nflake8-docstrings >= 0.2.8\npyflakes >= 1.2.3\n"
},
{
"path": "requirements-tests.txt",
"chars": 36,
"preview": "pytest >= 2.6.1\npytest-cov >= 2.2.1\n"
},
{
"path": "requirements-travis.txt",
"chars": 138,
"preview": "--requirement=requirements-checks.txt\n--requirement=requirements-tests.txt\n--requirement=requirements.txt\ncoveralls\nvirt"
},
{
"path": "requirements.txt",
"chars": 283,
"preview": "# Installation requirements for apt-smart.\n\nwheel >= 0.26.0\nbeautifulsoup4 >= 4.4.1\ncapturer >= 2.1.1\ncoloredlogs >= 5.0"
},
{
"path": "scripts/check-code-style.sh",
"chars": 653,
"preview": "#!/bin/bash -e\n\n# I value automated code style checks that break my Travis CI builds but I also\n# value compatibility wi"
},
{
"path": "scripts/collect-full-coverage.sh",
"chars": 454,
"preview": "#!/bin/bash -e\n\n# This shell script is used by the makefile and Travis CI to run the\n# apt-smart test suite as root, all"
},
{
"path": "scripts/install-on-travis.sh",
"chars": 820,
"preview": "#!/bin/bash -e\n\n# Install the required Python packages.\npip install --requirement=requirements-travis.txt\n\n# For github "
},
{
"path": "setup.cfg",
"chars": 119,
"preview": "# Enable universal wheels because `apt-smart'\n# is pure Python and works on Python 2 and 3 alike.\n\n[wheel]\nuniversal=1\n"
},
{
"path": "setup.py",
"chars": 5146,
"preview": "#!/usr/bin/env python\n\n# Setup script for the `apt-smart' package.\n#\n# Author: martin68 and Peter Odding\n# Last Change: "
},
{
"path": "test_custom_mirrors.txt",
"chars": 111,
"preview": "http://mirrors.layeronline.com/ubuntu/\nhttp://mirror.csclub.uwaterloo.ca/ubuntu/\nhttp://muug.ca/mirror/ubuntu/\n"
},
{
"path": "tox.ini",
"chars": 281,
"preview": "[tox]\nenvlist = py27, py34, py35, py36, py37, pypy\n\n[testenv]\ndeps = -rrequirements-tests.txt\ncommands = py.test {posarg"
}
]
About this extraction
This page contains the full source code of the martin68/apt-smart GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 35 files (219.2 KB), approximately 55.1k tokens, and a symbol index with 135 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.