linux-repository-mirror

Run linux repository mirrors with docker/kubernetes

• This implementation is intended to be used in a small organization and cannot be deployed as a public mirror due to the performance reasons.
• Furthermore, because current configuration has been adjusted for @yamaoka-kitaguchi-lab local environment, if you want to deploy this elsewhere, you'll need to read through all the files and modify them accordingly.

How to use: configure your linux to fetch from the local mirror

As of August 10th, 2020, following seven linux mirrors are available - see Mirroring details for further information.

ArchLinux

• http://mirror.intra.net.ict.e.titech.ac.jp/pacman/archlinux
• http://pacman.mirror.intra.net.ict.e.titech.ac.jp/archlinux

Add the below lines to the beginning of your /etc/pacman.d/mirrorlist and then run pacman -Syy for force syncing.

## Tokyo Tech
Server = http://pacman.mirror.intra.net.ict.e.titech.ac.jp/archlinux/$repo/os/$arch

Manjaro

• http://mirror.intra.net.ict.e.titech.ac.jp/pacman/manjaro
• http://pacman.mirror.intra.net.ict.e.titech.ac.jp/manjaro

Add the below lines to the beginning of your /etc/pacman.d/mirrorlist and then run pacman -Syy for force syncing.

## Tokyo Tech
Server = http://pacman.mirror.intra.net.ict.e.titech.ac.jp/manjaro/$repo/os/$arch

Ubuntu

For both server and desktop editions of AMD64 and i386 architecture. Only available in 20.04 LTS (focal) and 18.04 LTS (bionic).

• http://mirror.intra.net.ict.e.titech.ac.jp/apt/ubuntu
• http://apt.mirror.intra.net.ict.e.titech.ac.jp/ubuntu

Use the following command to replace the repository URLs from Japanese official archive to the local mirror.

% sed -i.bak -e "s%jp.archive.ubuntu.com%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list

Debian

Testing and stable release of 11 (bullseye), 10 (buster), and 9 (strech) are available.

• http://mirror.intra.net.ict.e.titech.ac.jp/apt/debian
• http://apt.mirror.intra.net.ict.e.titech.ac.jp/debian

Use the following command to replace the repository URLs from Japanese official archive to the local mirror.

% sed -i.bak -e "s%ftp.jp.debian.org%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list

Raspbian

• http://mirror.intra.net.ict.e.titech.ac.jp/apt/raspbian
• http://apt.mirror.intra.net.ict.e.titech.ac.jp/raspbian

Use the following command to replace the repository URLs from the official archive to the local mirror.

% sed -i.bak -e "s%raspbian.raspberrypi.org%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list

Cumulus

This repository is for lab's external router FS N5850-48S6Q and is not intended to be used on any other machine.

• http://mirror.intra.net.ict.e.titech.ac.jp/apt/cumulus
• http://apt.mirror.intra.net.ict.e.titech.ac.jp/cumulus

Use the following command to replace the repository URLs from the official archive to the local mirror - also refer to the official documentation as appropriate.

% sed -i.bak -e "s%repo3.cumulusnetworks.com/repo%apt.mirror.intra.net.ict.e.titech.ac.jp/cumulus%g" /etc/apt/sources.list

Gentoo

• http://mirror.intra.net.ict.e.titech.ac.jp/portage/gentoo
• http://portage.mirror.intra.net.ict.e.titech.ac.jp/gentoo

Check and follow the instructions of official handbook.

How to deploy

There are two ways to deploy; both ways are very simple and easy, but if you're not familiar with kubernetes, deploying with docker-compose is recommended.

NOTE: You need at least 2.5TB of free disk space to deploy all mirrors. The initial synchronization may take a few days because of the bandwidth limitation on upstream servers.

Deploy using docker-compose with systemd

If you don't have docker-compose (1.26.0 or higher), get it first. All commands must be executed as root (use sudo).

% make install-compose

Then deploy mirrors on your system with:

% make install

It's done. All mirrors are automatically synchronized and kept up to date by systemd/Timers. Of course, you can initiate git updates or mirror synchronization anytime you want. The following commands may help you:

% make update   # Update git repository and rebuild docker images
% make sync     # Sync all mirrors with their upstream servers
% make publish  # Publish mirrors (start Web server)
% make log      # Show syncing logs

Type make uninstall to unregister automatic mirror syncing, and type make destroy to purge all of local repositories.

Deploy using helm3

Prior to the deployment, install and setup microk8s, then prepare PVs. Check hint if necessary.

Add chart repository.

% microk8s helm3 repo add linux-repository-mirror https://yamaoka-kitaguchi-lab.github.io/linux-repository-mirror/chart
% microk8s helm3 repo list
NAME                    URL
linux-repository-mirror https://yamaoka-kitaguchi-lab.github.io/linux-repository-mirror/chart

Instead of the above commands, you can clone this repository and move directory to chart.

% git clone --depth 1 https://github.com/yamaoka-kitaguchi-lab/linux-repository-mirror
% cd linux-repository-mirror/chart

Deploy mirrors. All mirrors are automatically synchronized and kept up to date by CronJob.

% microk8s helm3 install linux-repository-mirror linux-repository-mirror
% microk8s kubectl get all,pv,pvc --namespace default

Mirroring details

Each repository size is as of June 20, 2020. ALL means that the local mirror is a complete replication of the upper server.

Hints and tips

When you decided to make a new mirror somewhere

Why don't you set the lab mirror as your upstream server. You can download packages without annoying bandwidth limitation. This is especially useful for the initial synchronization, which requires a lot of data transfer.

Coming soon

1. Add FreeBSD repository mirror (Of course BSD is not Linux, though) (#1)
2. Allow rsync client to pull packages from this local mirror (#2)

Known issues

1. ArchLinux repository has not been updated due to the upper mirror failure (#4)
2. I18n support of Debian has been freezed due to the upper mirror failure (#5)
3. Horizontal Pod Autoscaler doesn't work with microk8s (#3)

Versioning

Releases are managed in CalVer scheme and it is independent of chart versioning. For the versions available, see the tags on this repository.

License

This project is licensed under the MIT License - see the LICENSE file for details.