Git Product home page Git Product logo

charm-apt-mirror's Introduction

Apt Mirror

Description

A small tool that provides ability to mirror any parts (or even all) of Debian and Ubuntu GNU/Linux distributions or any other apt sources which typically are provided by open source developers.

Usage

Deployment

The charm can be deployed using Juju:

juju deploy cs:apt-mirror

The charm can handle arbitrary set of upstream DEB sources via setting mirror-list. Example below shows a bundle with this charm configured to mirror multiple Ubuntu series (Bionic and Focal) in a single repository and expose this repository via NGINX. Additionally PPAs and external repositories can be mirrored.

series: bionic
machines:
  '0':
    series: bionic
services:
  nginx:
    charm: cs:~majduk/nginx
    expose: true
    num_units: 1
    to:
    - '0'
  apt-mirror:
    charm: cs:~majduk/apt-mirror
    expose: true
    num_units: 1
    options:
      mirror-list: |-
        deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
        deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
        deb http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse
        deb http://security.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
        deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse
        deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
        deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
        deb http://security.ubuntu.com/ubuntu focal-security main restricted universe multiverse
    to:
    - '0'
relations:
- - nginx
  - apt-mirror

The repository needs to be exposed over HTTP using some web server. In the example above, Nginx charmed HTTP server is used to expose the repository.

Repository consumption

The clients can be pointed to the repository by simply updating the /etc/apt/sources.list file to point to the repository, for example repo.example.com:

deb http://repo.example.com/archive.ubuntu.com/ubuntu focal main restricted universe multiverse
deb http://repo.example.com/archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
deb http://repo.example.com/archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
deb http://repo.example.com/security.ubuntu.com/ubuntu focal-security main restricted universe multiverse

Additional notes:

  • The repository can be consumed by units deploed by MAAS. Please refer to MAAS documentation for the detailed MAAS configuration options.
  • The repository can be consumed by Juju deployed models. Please refer to the discourse on offline deployment strategies and Juju documentation for more details. In order to use the repository from the Juju model level, simplistic approach is just to use Juju model apt-mirror config option and install-sources config options for the charms being used.

Repository management

The repository exposes single, selected snapshot to the clients. After the repository is deployed, it is necessary to pull the upstream packages to the repository:

juju run-action apt-mirror/0 synchronize

When the action execution completes (depending on the configured mirror list and available bandwidth, it can take a considerable amount of time), snapshot can be created:

juju run-action --wait apt-mirror/0 create-snapshot

Output of the action contains the created snapshot name, for example snapshot-20210329092856.

This snapshot can in turn be published to be used by the clients:

juju run-action --wait apt-mirror/0 publish-snapshot name=snapshot-20210329092856

Full list of available snapshots can be obtained by running:

juju run-action --wait apt-mirror/0 list-snapshots

Currently published snapshot is shown in juju status.

Repository can be synchronized with the upstream multiple times and multiple snapshots can be created. It's possible to expose any arbitrary snapshot, making it possible to fine tune the packages available to the repository cilents.

The repository allows also specifying a Cron job via cron-schedule option, to regularily, automatically sync to the upstream to make sure the repository tracks upstream at a certain delay. To expose the latest packages to the clients, snapshot still needs to be created and published.

Developing

Create a virtual environment and activate it

make dev-environment
source .venv/bin/activate

Testing

Run lint tests:

make lint

Run unit tests:

make unittests

charm-apt-mirror's People

Contributors

agileshaw avatar chanchiwai-ray avatar esunar avatar gleland avatar majduk avatar pjack avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

charm-apt-mirror's Issues

Some docs needed

Heya!

I'm trying out the charm but its not clear from the README how to:

  • mirror more than a single series (e.g. bionic + focal) within the same config "mirror-list"
  • How to configure other models apt-mirror to use the charm

Great work!

juju-no-proxy exceptions are not supported by the charm

The use-case is to run in a proxied environment with internally accessilbe apt-mirror source. Charm currently only supports proxy either fully on or off based on the model defaults and it is not possible to configure proxy exceptions as it doesn't support juju-no-proxy.

Snapshot creation fails if directory structures within /var/spool/apt-mirror/mirror attempt to create the same symlink twice

I've encountered this on a cloud where we're using charm-apt-mirror:

2021-09-09 22:55:38 INFO juju-log /var/spool/apt-mirror/mirror/<REDACTED>/apt-mirror/archive.ubuntu.com/ubuntu/pool -> /var/spool/apt-mirror/snapshot-20210909225538//archive.ubu
[...]
2021-09-09 22:57:51 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 201, in <module>
    main(AptMirrorCharm)
  File "/var/lib/juju/agents/unit-apt-mirror-0/charm/venv/ops/main.py", line 402, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-apt-mirror-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-apt-mirror-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-apt-mirror-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-apt-mirror-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 117, in _on_create_snapshot_action
    os.symlink(src_pool, dst_pool)
FileExistsError: [Errno 17] File exists: '/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu/pool' -> '/var/spool/apt-mirror/snapshot-20210909225538/archive.ubuntu.com/ubuntu/pool'

This is in an environment where we have one central mirror pulling from upstream, and several secondary mirrors pulling from the central mirror. It appears that one of the secondary mirrors was at one point pulling from upstream and had a /var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu directory present in addition to the /var/spool/apt-mirror/mirror/<REDACTED>/apt-mirror/archive.ubuntu.com/ubuntu directory.

It may be necessary to clean up old repository directories in some way to avoid this conflict.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.