jblakeman / apt-select Goto Github PK

View Code? Open in Web Editor NEW

329.0 12.0 44.0 248 KB

Ubuntu Archive Mirror reporting tool for apt sources configuration.

License: MIT License

Shell 5.49% Python 94.51%

apt-select's Introduction

apt-select

Find a fast, up-to-date Ubuntu Archive Mirror.

Features

Tests latency to mirrors in a given country's mirror list at mirrors.ubuntu.com.
- 3 requests are sent to each mirror, minumum round trip time being used for rank.
Reports latency, status, and bandwidth capacity of the fastest mirrors in a ranked list.
- Status and bandwidth are scraped from launchpad.
Generates sources.list file using new mirror.
- New mirror can be chosen from a list or selected automatically using the top ranked mirror (default).

Installation

Target most recent release:

pip install apt-select

or:

pip3 install apt-select

Target project master branch:

pip install git+https://github.com/jblakeman/apt-select.git

or:

git clone https://github.com/jblakeman/apt-select
python apt-select/setup.py install

Invocation

$ apt-select --help
usage: apt-select [-h] [-C [COUNTRY]] [-t [NUMBER]] [-m [STATUS] | -p]
                  [-c | -l]

Find the fastest Ubuntu apt mirrors.
Generate new sources.list file.

optional arguments:
  -h, --help            show this help message and exit
  -C [COUNTRY], --country [COUNTRY]
                        specify a country to test its list of mirrors
                        used to match country list file names found at mirrors.ubuntu.com
                        COUNTRY should follow ISO 3166-1 alpha-2 format
                        default: US
  -t [NUMBER], --top-number [NUMBER]
                        specify number of mirrors to return
                        default: 1
  -m [STATUS], --min-status [STATUS]
                        return mirrors with minimum status
                        choices:
                           up-to-date
                           one-day-behind
                           two-days-behind
                           one-week-behind
                           unknown
                        default: up-to-date
  -p, --ping-only       rank mirror(s) by latency only, disregard status(es)
                        cannot be used with -m/--min-status
  -c, --choose          choose mirror from a list
                        requires -t/--top-num NUMBER where NUMBER > 1
  -l, --list            print list of mirrors only, don't generate file
                        cannot be used with -c/--choose

The exit code is 0 on success, 1 on error, and 4 if sources.list already has the chosen
mirror and a new one was not generated.

Examples

Get the top mirror from the United Kingdom to generate a new sources.list::

apt-select --country GB

Choose from the top 3 mirrors, including those last updated a week ago::

apt-select -c -t 3 -m one-week-behind

Find the top 10 mirrors, output latency info only, and don't generate new sources.list::

apt-select -t 10 -p -l

After new sources.list is generated in current working directory, backup and replace to update apt::

sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup && \
sudo mv sources.list /etc/apt/

Supported URI Types

Currently, http, https and ftp are supported.

/etc/apt/sources.list should contain sources in the following format::

[deb|deb-src] [http|https|ftp]://mirror.example.com/path [component1] [component2] [...]

Dependencies

Python HTML parser, BeautifulSoup.

HTTP Requests library, requests.

apt-select's People

Contributors

Stargazers

Watchers

apt-select's Issues

`AttributeError: 'Sources' object has no attribute 'APT_FILE'`

After running apt-select to get a mirror I get the following error (full log of my actions):

# apt-select --country NL -t 25 --choose
Getting list of mirrors...done.
Testing latency to mirror(s)
[24/24] 100%
Getting list of launchpad URLs...done.
Looking up 24 status(es)
[15/24] 62%
1. mirror.nl.leaseweb.net
    Latency: 12.88 ms
    Org:     LeaseWeb B.V.
    Status:  Up to date
    Speed:   4 Gbps
2. nl3.archive.ubuntu.com
    Latency: 13.04 ms
    Org:     PCextreme B.V.
    Status:  Up to date
    Speed:   2 Gbps
3. ftp.snt.utwente.nl
    Latency: 13.90 ms
    Org:     University of Twente, The Netherlands
    Status:  Up to date
    Speed:   10 Gbps
4. mirrors.xtom.nl
    Latency: 14.35 ms
    Org:     xTom
    Status:  Up to date
    Speed:   10 Gbps
5. nl.archive.ubuntu.com (current)
    Latency: 15.14 ms
    Org:     BIT B.V.
    Status:  Up to date
    Speed:   10 Gbps
6. mirror.nl.datapacket.com
    Latency: 15.86 ms
    Org:     datapacket.com
    Status:  Up to date
    Speed:   20 Gbps
7. ubuntu.mirror.true.nl
    Latency: 16.89 ms
    Org:     True BV
    Status:  Up to date
    Speed:   1 Gbps
8. mirror.1000mbps.com
    Latency: 17.45 ms
    Org:     1000Mbps.com
    Status:  Up to date
    Speed:   10 Gbps
9. mirror.nforce.com
    Latency: 17.46 ms
    Org:     NFOrce Entertainment B.V.
    Status:  Up to date
    Speed:   10 Gbps
10. mirror.dataone.nl
    Latency: 17.55 ms
    Org:     Dataone Datacenters
    Status:  Up to date
    Speed:   4 Gbps
11. mirror.transip.net
    Latency: 17.57 ms
    Org:     TransIP B.V.
    Status:  Up to date
    Speed:   4 Gbps
12. mirror.serverius.net
    Latency: 18.71 ms
    Org:     Serverius
    Status:  Up to date
    Speed:   10 Gbps
13. mirror.amsiohosting.net
    Latency: 18.77 ms
    Org:     Amsio B.V.
    Status:  Up to date
    Speed:   2 Gbps
14. mirror.previder.nl
    Latency: 19.31 ms
    Org:     Previder B.V.
    Status:  Up to date
    Speed:   1 Gbps
15. mirror.i3d.net
    Latency: 21.29 ms
    Org:     i3D.net
    Status:  Up to date
    Speed:   10 Gbps
Choose a mirror (1 - 15)
'q' to quit 3
Selecting mirror http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/ ...
Traceback (most recent call last):
  File "/usr/local/bin/apt-select", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/apt_select/__main__.py", line 245, in main
    apt_select()
  File "/usr/local/lib/python3.6/dist-packages/apt_select/__main__.py", line 227, in apt_select
    'apt': sources.APT_FILE
AttributeError: 'Sources' object has no attribute 'APT_FILE'

Traceback on Ubuntu 17.10

I installed apt-select using the following:

git clone https://github.com/jblakeman/apt-select.git sudo python apt-select/setup.py install

I don't see any blatant errors when installing. When I attempt to invoke

apt-select

I get the attached trace back. I can transcribe it here if needed.

Add Cross-Platform/Distribution Compatibility

Currently, apt-select must be run on an Ubuntu operating system. However, we could just check the platform and only allow for file generation if run on an Ubuntu OS while still offering full reporting functionality when run on other platforms or distributions.

Status lookups hang

If the --top-number argument >= the number of mirrors that were successfully tested for latency, and the number of successful [1] mirror info lookups <= the number of mirrors successfully tested, the status lookups will hang indefinitely.

dependencies instructions incorrect

small typo. your instructions indicate to install beautifulsoup using

apt-get install python-bf4

but it should be

apt-get install python bs4 (bs as in beautifulsoup)

Linux Mint support

# lsb_release -a
No LSB modules are available.
Distributor ID:	Linuxmint
Description:	Linux Mint 20
Release:	20
Codename:	ulyana

# apt-select --help
Error setting system information:
	Linuxmint distributions are not supported. Ubuntu is required.

Would be nice to make it work on Linux Mint - not for the Mint-specific repos, just for the Ubuntu ones.

Error when runing apt-select

Getting list of mirrors ... done.
Testing 6 mirror(s) [5/6] 83%Could not resolve hostname
[Errno -3] Temporary failure in name resolution
Traceback (most recent call last):
  File "apt-select.py", line 177, in <module>
    lowest = ping.minRTT()
  File "/home/tibi/work/apt-select/mirrors.py", line 49, in minRTT
    rtt = self.__tcpPing()
  File "/home/tibi/work/apt-select/mirrors.py", line 36, in __tcpPing
    s.connect((self.addr, port))
AttributeError: RoundTrip instance has no attribute 'addr'

Cannot operate through a proxy

Looks like a well written tool, but not designed to deal with a proxy use case? I.e. assumes open internet access?

With my http_proxy env variables set, it failed to work

$ apt-select
Getting list of mirrors...done.
Testing latency to mirror(s)
[0/3] 0%    connection to ubuntu.mirror.neology.co.za: timed out
[1/3] 33%   connection to ftp.leg.uct.ac.za: timed out
[2/3] 66%   connection to archive.ubuntu.com: timed out
[3/3] 100%
Cannot connect to any mirrors in ['http://ubuntu.mirror.neology.co.za/ubuntu/', 'http://ftp.leg.uct.ac.za/ubuntu/', 'http://archive.ubuntu.com/ubuntu/']

the problem is that in mirrors.py, the __tcp_ping() function cannot work via an HTTP proxy.

So this would entail quite a bit of rework.

detect if using a proxy
skip or implement an alternate measure of RTT through HTTP proxy
simplest may be to skip RTT testing if we use a proxy and rank purely based on bandwidth tests?

(If I find some spare time, I'll try hack at it and submit a PR)

Ranked list is not output for choice when Launchpad isn't readable

If the choose flag is enabled and status lookups are aborted (likely because the request timed out to launchpad.net) a selection list is not populated after reverting to latency rankings.

$ apt-select -t 5 -c
Getting list of mirrors...done.
Testing latency to mirror(s)
[72/72] 100%
Getting list of launchpad URLs...https://launchpad.net/ubuntu/+archivemirrors: <urlopen error timed out>
Unable to retrieve list of launchpad sites
Reverting to latency only
Choose a mirror (1 - 0)
'q' to quit

Running with no arguments fails

$ ./apt-select.py
Getting list of mirrors ...
done.
Testing 76 mirror(s) [76/76] 100%
Traceback (most recent call last):
File "./apt-select.py", line 296, in
if not repo:
NameError: name 'repo' is not defined

"Error finding current repositories " - no source file generated

arthur@machine:~/local/apt-select$ ./apt-select.py 
[snip]
Choose a mirror from the list (1 - 5) 1
Error finding current repositories

Remove mirror

I'm one of the maintainers of the @nextcloud VM and we use apt-select in our script to adapt the mirrors to the location our users are in.

Though I got a report that http://ftp.availo.se/ubuntu doesn't fetch (or even contains) all packages and therefore the installation fails.

So, please remove it.

show scheme in output

apt-select now successfully parses https urls, but it doesn't show in the output if a mirror is http or https.

please add the scheme to the output

parsing sources.list with https fails

Hello, parsing with "https" mirrors in the sources.list fails, taking out the 's' turning it to "http" will fix it.

add verbose flag

with the option -v it should show more info, which url it is parsing during processing

Please add PyPi package

Hey,

Super useful project. I would be very happy to have it listed in PyPi to make it slightly easier to install.

Thanks!

Skipping file generation - Connection refused

Hello,

is there a way that incorrect responses from a mirror can be ignored? Otherwise the further execution in a GitLab pipeline aborts.

RUN apt-select --country DE

Getting list of mirrors...done.
Testing latency to mirror(s)
connection to ftp.rz.tu-bs.de: [Errno 111] Connection refused
[45/45] 100%
Getting list of launchpad URLs...done.
Looking up 1 status(es)
Traceback (most recent call last):
File "/root/.local/bin/apt-select", line 8, in
sys.exit(main())
^^^^^^
File "/root/.local/lib/python3.12/site-packages/apt_select/main.py", line 245, in main
apt_select()
File "/root/.local/lib/python3.12/site-packages/apt_select/main.py", line 175, in apt_select
sources.set_current_archives()
File "/root/.local/lib/python3.12/site-packages/apt_select/apt.py", line 161, in set_current_archives
raise SourcesFileError((
apt_select.apt.SourcesFileError: Error finding current main URI in /etc/apt/sources.list
Skipping file generation

Regards
nowrap

Use HTTP and FTP connections for latency testing

Currently, TCP sockets are used for testing latency. One benefit of this approach is that it's fast. Drawbacks are that it's a less accurate representation of the speed of an HTTP or FTP connection, which are used when updating and upgrading via apt-get and aptitude. Also, testing when connecting through a proxy is problematic. See #45.

The proposed solution is to use httplib and ftplib for connection time based round trip tests, similar to how TCP sockets are currently used. I've briefly had a chance to test these libraries, and a potential drawback is that the first two connection attempts seem to have a much higher round trip time than the third and after. So unless the number of connection tests (3) per mirror are increased (slower), it would be harder to get enough information to reduce the noise that network variation can introduce.

Any feedback on this proposal would be appreciated.

Measure actual download speed

For the top few results it would be very useful to measure the actual download speed. The latency tells
you the server is close and most likely has good routing, but it could still be taxed to near it's maximum capacity.

To illustrate a simple test using the mirrors fetched from http://mirrors.ubuntu.com/mirrors.txt
A list of mirrors, followed by the latency (5 pings) and the download performance as outputted by curl:

mirror: http://mirrors.noction.com/ubuntu/archive/ mirrors.noction.com : 11.75 13.36 14.32 14.36 15.04
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  15.8M      0  0:00:03  0:00:03 --:--:-- 15.8M

mirror: http://osmirror.rug.nl/ubuntu/ osmirror.rug.nl : - - - - -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  32.8M      0  0:00:01  0:00:01 --:--:-- 32.8M

mirror: http://mirror.dataone.nl/ubuntu-archive/ mirror.dataone.nl : 12.74 13.81 17.42 83.72 14.00
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  24.0M      0  0:00:02  0:00:02 --:--:-- 24.0M

mirror: http://mirror.nl.leaseweb.net/ubuntu/ mirror.nl.leaseweb.net : 12.49 11.28 11.18 10.95 14.03
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  32.6M      0  0:00:01  0:00:01 --:--:-- 32.6M

mirror: http://mirrors.nl.eu.kernel.org/ubuntu/ mirrors.nl.eu.kernel.org : 195.34 189.46 192.50 190.03 189.61
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  7304k      0  0:00:07  0:00:07 --:--:-- 9196k

mirror: http://ubuntu.mirror.cambrium.nl/ubuntu/ ubuntu.mirror.cambrium.nl : 12.07 11.71 13.25 11.19 13.21
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  23.7M      0  0:00:02  0:00:02 --:--:-- 23.7M

mirror: http://ftp.tudelft.nl/archive.ubuntu.com/ ftp.tudelft.nl : 19.16 11.97 13.18 13.60 13.69
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  24.3M      0  0:00:02  0:00:02 --:--:-- 24.3M

mirror: http://ubuntu.mirror.true.nl/ubuntu/ ubuntu.mirror.true.nl : 17.14 13.80 15.07 22.22 9.96
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  33.4M      0  0:00:01  0:00:01 --:--:-- 33.4M

mirror: http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/ ftp.snt.utwente.nl : 15.19 22.67 15.82 15.47 17.84
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  20.6M      0  0:00:02  0:00:02 --:--:-- 20.6M

mirror: http://ftp.nluug.nl/os/Linux/distr/ubuntu/ ftp.nluug.nl : 15.86 11.20 10.38 74.28 77.86
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  23.9M      0  0:00:02  0:00:02 --:--:-- 23.9M

mirror: http://mirror.i3d.net/pub/ubuntu/ mirror.i3d.net : 20.47 18.63 16.63 14.35 12.19
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  20.3M      0  0:00:02  0:00:02 --:--:-- 20.3M

mirror: http://nl3.archive.ubuntu.com/ubuntu/ nl3.archive.ubuntu.com : 21.67 19.40 18.24 19.69 19.01
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  31.8M      0  0:00:01  0:00:01 --:--:-- 31.8M

mirror: http://nl.archive.ubuntu.com/ubuntu/ nl.archive.ubuntu.com : 12.72 14.37 14.39 15.06 21.63
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  12.9M      0  0:00:04  0:00:04 --:--:-- 12.9M

mirror: http://mirror.amsiohosting.net/archive.ubuntu.com/ mirror.amsiohosting.net : 14.44 14.44 13.70 12.83 14.47
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  23.6M      0  0:00:02  0:00:02 --:--:-- 23.6M

mirror: http://mirror.1000mbps.com/ubuntu/ mirror.1000mbps.com : 13.05 11.30 11.43 11.41 15.09
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  33.6M      0  0:00:01  0:00:01 --:--:-- 33.7M

mirror: http://mirror.nforce.com/pub/linux/ubuntu/ mirror.nforce.com : 12.91 18.86 93.86 72.64 35.41
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  32.7M      0  0:00:01  0:00:01 --:--:-- 32.7M

mirror: http://mirror.transip.net/ubuntu/ubuntu/ mirror.transip.net : 14.20 16.50 11.64 14.43 24.58
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  17.4M      0  0:00:03  0:00:03 --:--:-- 17.4M

mirror: http://archive.ubuntu.com/ubuntu/ archive.ubuntu.com : 19.13 19.80 22.77 24.00 21.61
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.0M  100 54.0M    0     0  7752k      0  0:00:07  0:00:07 --:--:-- 10.6M

As you can see, several of the mirrors have really low latency but don't offer fast downloads.
For example, the nl3.archive.ubuntu.com server has a latency varying from 19.01 to 21.67 which would mean it doesn't end up at the top of the list while the download speed of 31.8MB/s is far more important to me.

Similarly, mirrors.noction.com only downloads with 15.8MB/s with more favorable latencies (varying between 11.75 and 15.04ms).

for mirror in $(curl http://mirrors.ubuntu.com/mirrors.txt); do
    echo
    echo -n "mirror: $mirror "
    host=$(echo "$mirror" | awk -F '/' '{print $3}')
    fping -C 5 -q "$host"
    curl -o /dev/null "${mirror}/dists/xenial/main/installer-amd64/current/images/netboot/mini.iso"
done

Option to search all mirrors or multiple mirrors

In europe, the fastest mirror must not be in your country, so it would be great, if you could select more than one Country mirror

apt-select fail to parse sources.list with a specific arch

In my sources.list I select [arch=amd64] which breaks apt-select parsing. if I remove [arch=amd64] it works fine.

For instance:

    deb [arch=amd64] http://mymirror/ubuntu trusty main restricted

The problem is in __confirm_apt_source_uri

Breaks with FTP apt sources

Greetings,

Just tried a run on a freshly upgraded Ubuntu 14.10. After installing the python3-bs4 package, I run the program directly, and receive the following output:

#python3 ./apt-select.py -l
Getting list of mirrors ... done.
Testing 66 mirror(s) [66/66] 100%
Traceback (most recent call last):
    File "./apt-select.py", line 277, in <module>
        repo_name = match(r'http://([\w\.\-]+)/', repo[0]).group(1)
 AttributeError: 'NoneType' object has no attribute 'group'

My python is quite weak, but it looks as if the tool is having a hard time parsing my sources.list file. I think this is because the regex on line 277 expects the file to start with an http:// deb source, but my file is mostly FTPs. If that line is changed as:

repo_name = match(r'(http://|ftp://)([\w\.\-]+)/', repo[0]).group(1)

..then all works as expected.

Allow specifying different mirrors.txt

For me the hardcoded URL currently only lists 5 mirrors. However I see there are many different lists, for example http://mirrors.ubuntu.com/VA.txt with 183 mirrors which found several faster ones than the default. Now I don't know exactly what that VA list is - but anyway could be nice to be able to pass an arbitrary URL with a list on the command line.

option to only select https

If you are searching for the fastest mirror with https, it would be handy if you could add the desired scheme on the commandline as option

UnicodeEncodeError when listing mirror info with unicode chars

I got the following trace dump after running apt-select --country CA -t 10 -m two-days-behind -l:

Traceback (most recent call last):
  File "/usr/local/bin/apt-select", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/apt_select/__main__.py", line 236, in main
    apt_select()
  File "/usr/local/lib/python3.5/dist-packages/apt_select/__main__.py", line 188, in apt_select
    print_status(info, rank)
  File "/usr/local/lib/python3.5/dist-packages/apt_select/__main__.py", line 82, in print_status
    'speed': info['Speed']
UnicodeEncodeError: 'ascii' codec can't encode character '\u0142' in position 73: ordinal not in range(128)

This happens when it tries to encode the name of a mirror hosted by Rafał Rzeczkowski (note the unicode character in the 5th position).
Seems related to #4; perhaps you missed a few places where it is still encoding/decoding in ascii rather than utf-8.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 5: ordinal not in range(128)

Selecting mirror http://www.mirrorservice.org/sites/archive.ubuntu.com/ubuntu/ ...
Traceback (most recent call last):
File "/www/myenv/bin/apt-select", line 11, in
sys.exit(main())
File "/www/myenv/local/lib/python2.7/site-packages/apt_select/main.py", line 244, in main
apt_select()
File "/www/myenv/local/lib/python2.7/site-packages/apt_select/main.py", line 233, in apt_select
sources.generate_new_config(work_dir, new_mirror)
File "/www/myenv/local/lib/python2.7/site-packages/apt_select/apt.py", line 177, in generate_new_config
self.__set_config_lines(new_mirror)
File "/www/myenv/local/lib/python2.7/site-packages/apt_select/apt.py", line 173, in __set_config_lines
self._lines = self._lines.replace(url, new_mirror)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 5: ordinal not in range(128)

Apt-select should do a bandwidth test

The correlation between bandwidth and latency is not as great as you might think. In many cases in the past I found that the Heanet server in Ireland gives me more bandwidth than the servers within the Netherlands (I'm based in Amsterdam). Even though the latency is a bit (5ms) higher, the effective bandwidth can be about 5x more :)

Invalid country. latam is not in ISO 3166-1 alpha-2 format

i have tried to run a nextcloud automated script. but everytime i choose my location (argentina) i got the error and the script fails

Invalid country. latam is not in ISO 3166-1 alpha-2 format.

Even if i try to modify the keyboard layout and the timezone, it ended with the legend
Invalid country. latam is not in ISO 3166-1 alpha-2 format

mirrors.py has problem

superthin@myubuntu:~/github/apt-select$ ./apt-select.py -c -t 3 -m one-week-behind
Getting list of mirrors ... done.
Testing 3 mirror(s) [3/3] 100%
Looking up status(es) [0/3] 0%Traceback (most recent call last):
File "./apt-select.py", line 226, in
flag_status
File "/home/superthin/github/apt-select/mirrors.py", line 108, in getInfo
if "unknown" in status:
TypeError: argument of type 'NoneType' is not iterable

My Ubuntu version/ edition is Ubuntu 14.04 LTS.

error finding current repositories

I got the above error message.

My current sources.list has only this line:

deb http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty main

Fails to consider `mirror` scheme

With (newer ?) ubuntu release, one can use a mirror protocol that use directly the mirrors from mirrors.ubuntu.com

Tihs means lines in sources.list looks like this:

deb mirror://mirrors.ubuntu.com/mirrors.txt [...]

When this is set in sources.list, then apt-select bails out with the error:

Traceback (most recent call last):
  File "/home/solstice/.local/bin/apt-select", line 11, in <module>
    sys.exit(main())
  File "/home/solstice/.local/lib/python2.7/site-packages/apt_select/__main__.py", line 244, in main
    apt_select()
  File "/home/solstice/.local/lib/python2.7/site-packages/apt_select/__main__.py", line 175, in apt_select
    sources.set_current_archives()
  File "/home/solstice/.local/lib/python2.7/site-packages/apt_select/apt.py", line 164, in set_current_archives
    self.skip_gen_msg)
apt_select.apt.SourcesFileError: Error finding current main URI in /etc/apt/sources.list
Skipping file generation

Option with empty argument causing TypeError

Arguments -t/--top-number and -m/--min-status fall back to default values when the options are not present on the command line. However, when they are passed without a following argument, None is stored and causes the following error:

TypeError: unsupported operand type for long(): 'NoneType'

Use the const keyword argument to argparse.add_argument, storing the option's default value to handle those cases.

TypeError: not enough arguments for format string

# apt-select --help
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/apt_select/apt_system.py", line 33, in AptSystem
    dist, codename = get_release.__func__()
  File "/usr/local/lib/python3.5/dist-packages/apt_select/apt_system.py", line 21, in get_release
    for s in check_output(["lsb_release", "-ics"]).split()]
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 693, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'lsb_release'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/apt-select", line 11, in <module>
    load_entry_point('apt-select==1.0.3', 'console_scripts', 'apt-select')()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2649, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2303, in load
    return self.resolve()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2309, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/apt_select/__main__.py", line 8, in <module>
    from apt_select.mirrors import Mirrors
  File "/usr/local/lib/python3.5/dist-packages/apt_select/mirrors.py", line 11, in <module>
    from apt_select.apt_system import AptSystem
  File "/usr/local/lib/python3.5/dist-packages/apt_select/apt_system.py", line 14, in <module>
    class AptSystem(object):
  File "/usr/local/lib/python3.5/dist-packages/apt_select/apt_system.py", line 35, in AptSystem
    raise ValueError("%s\n%s" % _not_ubuntu)
TypeError: not enough arguments for format string

apt-select : Command not recognized

So i have recently installed pop os on my laptop, i'm trying to setup fastest mirrors so i downloaded this program using
pip3 install apt-select
and it got installed without any issues but when i try to run it in terminal, it says
apt-select: command not found
what am i doing wrong,
i've also tried export PATH=”$PATH:/usr/local/bin/python” but no luck running it.

NameError: name 'info_size' is not defined

Just FYI... I cloned master (f7179f1), ran the script, and got this error:

# apt-select.py
Getting list of mirrors ... done.
Testing 73 mirror(s) [73/73] 100%
Traceback (most recent call last):
  File "/root/bin/apt-select.py", line 245, in <module>
    if info_size == 0:
NameError: name 'info_size' is not defined

mirrors.txt only has a single mirror to choose from

http://mirrors.ubuntu.com/mirrors.txt only contains a single mirror:
http://archive.ubuntu.com/ubuntu/

Please update the script to check the current country's file, which contain much more mirrors, for example:
Brazil - http://mirrors.ubuntu.com/BR.txt
USA - http://mirrors.ubuntu.com/US.txt
France - http://mirrors.ubuntu.com/FR.txt

Has this been abandoned?

@knatten @jblakeman has this been abandoned?

Needs CI and Unit Testing

We should add integration tests for all primary success cases, mocking socket connections (latency testing), Launchpad mirror page responses (status, bandwidth, info tests), stdin for user interaction, /etc/apt/sources.list reads and successful file generation (builtins.open).

UnicodeDecodeError - apt-select.py, line 163

Traceback (most recent call last):
File "./apt-select.py", line 163, in
data = d.getInfo()
File "/home/jirka99/Stažené/netselect/apt-select-master/mirrors.py", line 113, in getInfo
launch_html = launch_html.read().decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 8093: ordinal not in range(128)

2 consecutive err's in mirrors.py

$ apt-select/apt-select.py -t 10 -l
Getting list of mirrors ... done.
Testing 10 mirror(s) [10/10] 100%
Looking up status(es) [0/9] 0%Traceback (most recent call last):
  File "apt-select/apt-select.py", line 217, in <module>
    flag_status
  File "/home/gudzenkov/scripts/apt-select/mirrors.py", line 121, in getInfo
    if "unknown" in status:
TypeError: argument of type 'NoneType' is not iterable```

$ apt-select/apt-select.py -t 10 -l
Getting list of mirrors ... done.
Testing 10 mirror(s) [10/10] 100%
Looking up status(es) [0/3] 0%Traceback (most recent call last):
  File "apt-select/apt-select.py", line 217, in <module>
    flag_status
  File "/home/gudzenkov/scripts/apt-select/mirrors.py", line 106, in getInfo
    launch_html = urlopen(archive)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error timed out>

Mirrors list from mirrors.ubuntu.com/mirrors.txt :

http://kartolo.sby.datautama.net.id/ubuntu/
http://mirror.poliwangi.ac.id/ubuntu/
http://kebo.pens.ac.id/ubuntu/
http://jaran.undip.ac.id/ubuntu/
http://kambing.ui.ac.id/ubuntu/
http://suro.ubaya.ac.id/ubuntu/
http://buaya.klas.or.id/ubuntu/
http://mirror.unej.ac.id/ubuntu/
http://mirror.kavalinux.com/ubuntu/
http://archive.ubuntu.com/ubuntu/

Mirror's status, Indonesia:

Kelompok Linux Arek Suroboyo http ftp rsync 1 Gbps One week behind
University of Indonesia http ftp rsync 1 Gbps One week behind
Datautama-Net-ID http rsync 1 Gbps Last update unknown
Politeknik Elektronika Negeri Surabaya http ftp rsync 1 Gbps One week behind
Universitas Jember http ftp rsync 1 Gbps Up to date
UPT Pusat Komputer Undip http ftp rsync 100 Mbps One week behind
kavalinux http 100 Mbps Up to date
Universitas Surabaya http ftp rsync 100 Mbps Last update unknown
State Polytechnic of Banyuwangi http 10 Mbps Last update unknown

ImportError from module renaming

Module was renamed, and now cannot be imported. Import name must be changed.

ImportError: No module named util_funcs

Ranked list output can be out of order

When Launchpad lookups are being received from a queue, the new ranked list is build irrespective of latency results, and the output is out of order. The list should be sorted again after lookups are completed.

No instructions for uninstall provided

There are instructions to install, but not even the setup.py --help provides instructions to uninstall.

I suggest this should be in the README?

Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)

possibly operator error ... this is what I did

cd ~/src/github.com/jblakeman/apt-select
python setup.py build
sudo python setup.py install

pip install apt-select

Requirement already satisfied: apt-select in /usr/local/lib/python2.7/dist-packages/apt_select-1.0.3-py2.7.egg
Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages/requests-2.13.0-py2.7.egg (from apt-select)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python2.7/dist-packages/beautifulsoup4-4.5.3-py2.7.egg (from apt-select)


sudo apt-select   

Getting list of mirrors...done.
Testing latency to mirror(s)
	connection to mirror.tocici.com: timed out
[79/79] 100%
Getting list of launchpad URLs...done.
Looking up 1 status(es)
1. mirror.imbrandon.com
    Latency: 24.12 ms
    Org:     www.imbrandon.com
    Status:  Up to date
    Speed:   100 Mbps
Selecting mirror http://mirror.imbrandon.com/ubuntu/ ...
New config file saved to /home/stens/src/github.com/jblakeman/apt-select/sources.list

OK so now I have the freshly generated config so install it per instructions

sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup &&   sudo mv sources.list /etc/apt/

so now do an update

sudo apt-get update && sudo apt-get upgrade

Get:1 http://mirror.imbrandon.com/ubuntu yakkety InRelease
Err:1 http://mirror.imbrandon.com/ubuntu yakkety InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:2 http://mirror.imbrandon.com/ubuntu yakkety-updates InRelease
Err:2 http://mirror.imbrandon.com/ubuntu yakkety-updates InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:3 http://mirror.imbrandon.com/ubuntu yakkety-backports InRelease
Err:3 http://mirror.imbrandon.com/ubuntu yakkety-backports InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:4 http://mirror.imbrandon.com/ubuntu yakkety-security InRelease
Err:4 http://mirror.imbrandon.com/ubuntu yakkety-security InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Hit:5 https://apt.dockerproject.org/repo ubuntu-yakkety InRelease
Fetched 6,232 B in 0s (14.1 kB/s)
Reading package lists... Done
E: Failed to fetch http://mirror.imbrandon.com/ubuntu/dists/yakkety/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://mirror.imbrandon.com/ubuntu/dists/yakkety-updates/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://mirror.imbrandon.com/ubuntu/dists/yakkety-backports/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://mirror.imbrandon.com/ubuntu/dists/yakkety-security/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Sources (restricted/source/Sources) is configured multiple times in /etc/apt/sources.list:2 and /etc/apt/sources.list:7

this is on a new install of ubuntu 16.10

Provide more detailed environment usage instructions

Thanks this is great.. went from 400KB/s using default setup then 600KB/s using mirrors:// to 43MB/s using your script... this should be included in Ubuntu by default, the fact that ubuntu doesnt speed test mirrors is crazy.

THANKS!

Only suggestion: Short of moving the entire program into one script(so you could wget it and just delete it) you could give install/uninstall instructions using venv
new Ubuntu install would be...

cd ~
sudo apt-get install python3-venv
python3 -m venv apt-select
source apt-select/bin/activate
pip install apt-select
apt-select
deactivate
sudo chown root:root sources.list
sudo chmod 644 sources.list
sudo mv /etc/apt/sources.list /etc/apt/sources.list.orig
sudo mv sources.list /etc/apt/

to uninstall:
rm -rf ~/apt-select

Support for Ubuntu 24.04

(venv) ➜  apt-select git:(master) ✗ apt-select
WARNING: no country code provided. defaulting to US.
Getting list of mirrors...done.
Testing latency to mirror(s)
[91/91] 100%
Getting list of launchpad URLs...done.
Looking up 1 status(es)
Traceback (most recent call last):
  File "/home/adeshssd/.local/bin/apt-select", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/apt_select-2.2.1-py3.12.egg/apt_select/__main__.py", line 245, in main
  File "/usr/local/lib/python3.12/dist-packages/apt_select-2.2.1-py3.12.egg/apt_select/__main__.py", line 175, in apt_select
  File "/usr/local/lib/python3.12/dist-packages/apt_select-2.2.1-py3.12.egg/apt_select/apt.py", line 161, in set_current_archives
apt_select.apt.SourcesFileError: Error finding current main URI in /etc/apt/sources.list

(venv) ➜  apt-select git:(master) ✗ cat /etc/apt/sources.list
# Ubuntu sources have moved to /etc/apt/sources.list.d/ubuntu.sources

Ubuntu 24.04 has moved to DEB822-STYLE for sources.