conda-incubator / conda-docker Goto Github PK
View Code? Open in Web Editor NEWCreate minimal docker images from conda environments
License: BSD 3-Clause "New" or "Revised" License
Create minimal docker images from conda environments
License: BSD 3-Clause "New" or "Revised" License
Awesome and exciting project :D <3
Does this support pip packages as well? Or do they all need to be in conda? Not sure how the layering would work..
Currently, it has this comment:
# FIXME: this should reall be check_output(), but chroot or fakechroot is
# giving some weird segfault after the install command completes ¯\_(ツ)_/¯
subprocess.call(
[
"fakechroot",
"chroot",
new_root,
"/_conda.exe",
"install",
"--offline",
"--file",
"/opt/conda/pkgs/env.txt",
"-y",
"--prefix",
"/opt/conda",
],
env=env,
cwd=host_conda_opt,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
Suggestions:
Was able to reproduce this issue:
digest = get_blob(image, "sha256:2de88731a44c549d7de6b752d0ae11907850b8363e6d1e2ca862d1fe6a14a4e8", token)
returned 400requests.get(f"https://registry-1.docker.io/v2/{image}/blobs/sha256:2de88731a44c549d7de6b752d0ae11907850b8363e6d1e2ca862d1fe6a14a4e8", headers = {"Authorization": f"Bearer {token}"})
returned 200So the easiest workaround is to have requests as a dependency. I'm sure there is something that is not being set properly with urllib.
Hi,
this sounds really useful for many users who are using conda successfully but having trouble packaging their conda envs easily into docker or singularity containers.
My question is about the status and future of this project. Is this a working prototype ? Or mature already, or just a test? How does it compare to the experimental conda-pack ?
Thanks for your efforts
Colin
I've tried a number of different ways and I'm having trouble installing the package via conda-forge due to package conflicts.
conda create -n conda_docker -c conda-forge --no-default-packages python=3.8 conda-docker
That returns the following:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package python conflicts for:
conda-docker -> conda -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*']
python=3.8
conda-docker -> python
Thanks!
On Arch Linux the example:
conda docker build -b frolvlad/alpine-glibc:latest -i example-image:123456 -o demo.tar numpy numba flask
fails with the error:
Executing transaction: done
Rolling back transaction: done
Exception('No compatible shell found!')
()
/.../sbin/chroot.fakechroot: line 147: 602419 Segmentation fault (core dumped) env -u FAKECHROOT_BASE_ORIG FAKECHROOT_CMD_ORIG= LD_LIBRARY_PATH="$fakechroot_chroot_paths" FAKECHROOT_BASE="$fakechroot_chroot_base" "$fakechroot_chroot_chroot" "${@:1:$(($fakechroot_chroot_n - 1))}" "$fakechroot_chroot_final_newroot" "${@:$(($fakechroot_chroot_n + 1))}"
INFO:conda_docker.conda:building conda environment took 20.671 [s]
INFO:conda_docker.conda:adding conda environment in package layers
Traceback (most recent call last):
File "/.../bin/conda-docker", line 11, in <module>
load_entry_point('conda-docker', 'console_scripts', 'conda-docker')()
File "/.../conda_docker/cli.py", line 121, in main
cli(args)
File "/.../conda_docker/cli.py", line 26, in cli
args.func(args)
File "/.../conda_docker/cli.py", line 114, in handle_conda_build
layering_strategy=args.layering_strategy,
File "/.../conda_docker/conda.py", line 635, in build_docker_environment
layering_strategy=layering_strategy,
File "/.../conda_docker/conda.py", line 582, in add_conda_layers
image, hostpath, arcpath=arcpath, filter=filter, records=records
File "/.../conda_docker/conda.py", line 548, in add_conda_package_layers
with open(meta_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpi3vqxxt8/opt/conda/conda-meta/_libgcc_mutex-0.1-conda_forge.json'
This is because PATH is being propagated from the host and doesn't include /bin
(in this alpine image the shell is /bin/bash
). Appending /bin
to PATH
gets me slightly further (before hitting an unrelated segfault that I'm still debugging).
the minimal example fail on my computer:
> conda docker build -b frolvlad/alpine-glibc:latest \ (dockertest)
-i example-image:123456 \
-o demo.tar \
numpy numba flask
INFO:conda_docker.conda:solving conda environment
Dry run. Exiting.
INFO:conda_docker.conda:solving conda environment took 20.188 [s]
INFO:conda_docker.conda:loading repodata
INFO:conda_docker.conda:loading repodata took 14.458 [s]
Traceback (most recent call last):
File "/home/resolvent/miniforge3/envs/dockertest/bin/conda-docker", line 10, in <module>
sys.exit(main())
^^^^^^
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/cli.py", line 125, in main
cli(args)
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/cli.py", line 26, in cli
args.func(args)
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/cli.py", line 96, in handle_conda_build
precs = find_precs(
^^^^^^^^^^^
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/conda.py", line 255, in find_precs
precs = precs_from_package_specs(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/conda.py", line 222, in precs_from_package_specs
raise e
^^^^^^^
File "/home/resolvent/miniforge3/envs/dockertest/lib/python3.11/site-packages/conda_docker/conda.py", line 217, in precs_from_package_specs
pkg_repodata = repodatas[channel]["packages"][fn]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 'ca-certificates-2022.12.7-ha878542_0.tar.bz2'
looking into repodatas[channel]["packages"]
I see that it has:
...
ca-certificates-2022.6.15-ha878542_0.tar.bz2
ca-certificates-2022.6.15.1-ha878542_0.tar.bz2
ca-certificates-2022.6.15.2-ha878542_0.tar.bz2
ca-certificates-2022.9.14-ha878542_0.tar.bz2
ca-certificates-2022.9.24-ha878542_0.tar.bz2
cabal-2.4.1.0-h2ccb070_0.tar.bz2
...
but https://anaconda.org/conda-forge/ca-certificates/files has 2022.12.7
How can this be??
@scopatz looking to use this in conda-store and need to latest PR. Could you create a release of conda-docker?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.