Cachi2 is a CLI tool that pre-fetches your project's dependencies to aid in making your build process hermetic.
To see if we support your package manager(s), please check the package managers section.
The primary intended use of Cachi2's outputs is for network-isolated container builds (see usage).
Please note that Cachi2 is rather picky, aiming to:
- encourage or enforce best practices
- enforce building from source - no pre-built artifacts, such as Python wheels
- never execute arbitrary code - looking at you setup.py (discouraged)
- keep the implementation simple
To play nicely with Cachi2, the build process for your project must be
-
Defined - Cachi2 only fetches dependencies that are explicitly declared - typically in a lockfile generated by your package manager.
-
Reproducible - Cachi2 will refuse to fetch a dependency if it's not pinned to an exact version. This goes for transitive dependencies as well (and ties to the Defined point). Most package managers pin all dependencies automatically in lockfiles.
-
Secure - Even with a lockfile, your build is not truly safe from supply chain attacks (such as dependency confusion) unless you verify the checksums of all dependencies. If your package manager supports specifying the expected checksums, we strongly encourage you to make use of them.
⚠ Cachi2 will verify checksums if present, but doesn't require them by default. This may change in the future.
In return, Cachi2 will help make your build
- Auditable - by generating a manifest of all the dependencies that go into your build.
The ability to achieve the goals depends on the hermeticity of the build process. Ideally, you should try to isolate the build from both the internet and the underlying host system to avoid implicit dependencies, irreproducible behavior and whole hosts of other issues. Cachi2 itself is not a hermetic build system. We suggest you take advantage of existing technologies - such as containers - to achieve isolation (see usage).
We do not distribute Cachi2 as a standalone package as of now.
To install Cachi2 for local development, see the development section.
quay.io/containerbuildsystem/cachi2:latest
The container is re-built automatically on every merge to the main branch.
You may wish to set up an alias to make local usage more convenient:
alias cachi2='podman run --rm -ti -v "$PWD:$PWD:z" -w "$PWD" quay.io/containerbuildsystem/cachi2:latest'
Note that the alias mounts the current working directory - the container will have access to files in that directory and nowhere else.
cachi2 fetch-deps \
--source ./my-repo \
--output ./cachi2-output \
gomod
The fetch-deps
command fetches your project's dependencies and stores them on your disk. You can then use these
outputs to, say, build a container image.
See docs/usage.md for a more detailed, practical (cough) example of Cachi2 usage.
You might also like to check out cachi2 --help
and the --help
texts of the available subcommands.
Set up a virtual environment that has everything you will need for development:
make venv
source venv/bin/activate
This installs the Cachi2 CLI in editable mode, which means changes to the source code will reflect in the behavior of the CLI without the need for reinstalling.
You may need to install the following dependencies before creating the virtualenv:
dnf install python3.9 python3-virtualenv
The CLI also depends on the following non-Python dependencies:
dnf install golang-bin git
You should now have everything needed to try out the CLI or hack on the code in vim your favorite
editor.
Cachi2's codebase conforms to standards enforced by a collection of formatters, linters and other code checkers:
- black (with a line-length of 100) for consistent formatting
- isort to keep imports sorted
- flake8 to (de-)lint the code and
politelyask for docstrings - mypy for type-checking. Please include type annotations for new code.
- pytest to run unit tests and report coverage stats. Please aim for (near) full coverage of new code.
Options for all the tools are configured in pyproject.toml and tox.ini.
Run all the checks that your pull request will be subjected to:
make test
We try to keep error messages friendly and actionable.
- If there is a known solution, the error message should politely suggest the solution.
- Include a link to the documentation when suitable.
- If there is no known solution, suggest where to look for help.
- If retrying is a possible solution, suggest retrying and where to look for help if the issue persists.
The error classes aim to encourage these guidelines. See the errors.py module.
Run all unit tests (but no other checks):
make test-unit
For finer control over which tests get executed, e.g. to run all tests in a specific file, activate the virtualenv and run:
tox -e python3.9 -- tests/unit/test_cli.py
Even better, run it stepwise (exit on first failure, re-start from the failed test next time):
tox -e python3.9 -- tests/unit/test_cli.py --stepwise
You can also run a single test class or a single test method:
tox -e python3.9 -- tests/unit/test_cli.py::TestGenerateEnv
tox -e python3.9 -- tests/unit/test_cli.py::TestGenerateEnv::test_invalid_format
tox -e python3.9 -- tests/unit/extras/test_envfile.py::test_cannot_determine_format
In short, tox passes all arguments to the right of --
directly to pytest.
Build Cachi2 image and run all integration tests (but no other checks):
make test-integration
To run integration-tests with custom image, specify the CACHI2_IMAGE environment variable. Examples:
CACHI2_IMAGE=quay.io/containerbuildsystem/cachi2:{tag} tox -e integration
CACHI2_IMAGE=localhost/cachi2-${USER}:latest tox -e integration
Similarly to unit tests, for finer control over which tests get executed, e.g. to run only 1 specific test case, execute:
CACHI2_IMAGE=localhost/cachi2-${USER}:latest tox -e integration -- tests/integration/test_package_managers.py::test_packages[gomod_without_deps]
Supported:
Planned:
- npm
- yarn
- rubygems
Based on the supported package managers in the original Cachito.
The gomod package manager works by parsing the go.mod file present in the source
repository to determine which dependencies to download. Cachi2 does not parse this file on its own - rather, we rely on
the go
command to download and list the required dependencies.
From go 1.17 onward, the go.mod file includes all the transitively required dependencies of your application - see the section about Pruned module graphs in the 1.17 changelog. In previous go versions, the go.mod file included only direct dependencies. Cachi2 does support downloading and listing all transitive dependencies for earlier versions thanks to Go's backwards compatibility2. Note that using go >= 1.17 in your project has the added benefit of downloading fewer dependencies (as noted in the changelog), in some cases drastically so.
See docs/gomod.md for more details.
https://pip.pypa.io/en/stable/
Cachi2 supports pip by parsing requirements.txt files present in the source repository and downloading the declared dependencies.
The files must be lockfiles, i.e. declare all the transitive dependencies and pin them to specific versions. Generating such a lockfile is best done using tools like pip-compile.
Note that, per the Cachi2 goals, we download only source distributions. This means pip will need to rebuild all the dependencies from source, which makes the build process more complex than you might expect.
See docs/pip.md for more details.
Cachi2 was derived (but is not a direct fork) from Cachito and is still in early development phase.
Footnotes
-
Cachi2 expects to use a specific version of the
go
command when downloading dependencies. This is the version installed in the cachi2 container. We do not guarantee correctness if you run Cachi2 locally (outside the container) with a different Go version. You are free to use a different version to build your project. ↩ -
The
go
command promises to be backwards compatible with previous versions. If your go.mod file specifies the intended go version, Cachi2 should handle it appropriately. If your go version is higher than what Cachi2 uses, there is a good chance it will be compatible regardless, as long as the dependency resolution did not change between the two versions. For example, dependency resolution did change in go 1.18 but not in go 1.19. ↩ ↩2