mamba-org / micromamba-docker Goto Github PK
View Code? Open in Web Editor NEWRapid builds of small Conda-based containers using micromamba.
Home Page: https://micromamba-docker.readthedocs.io/
License: Apache License 2.0
Rapid builds of small Conda-based containers using micromamba.
Home Page: https://micromamba-docker.readthedocs.io/
License: Apache License 2.0
$ docker run -it --rm -v $(pwd):/tmp mambaorg/micromamba:0.21.2 micromamba list
info ****************** Backtrace Start ******************
debug Loading configuration
trace Compute configurable 'create_base'
trace Compute configurable 'no_env'
trace Compute configurable 'no_rc'
trace Compute configurable 'rc_files'
trace Compute configurable 'root_prefix'
trace Get RC files configuration from locations up to HomeDir
trace Configuration not found at '/home/mambauser/.mambarc'
trace Configuration not found at '/home/mambauser/.condarc'
trace Configuration not found at '/home/mambauser/.conda/condarc.d'
trace Configuration not found at '/home/mambauser/.conda/condarc'
trace Configuration not found at '/home/mambauser/.conda/.condarc'
trace Configuration not found at '/opt/conda/.mambarc'
trace Configuration not found at '/opt/conda/condarc.d'
trace Configuration not found at '/opt/conda/condarc'
trace Configuration not found at '/opt/conda/.condarc'
trace Configuration not found at '/var/lib/conda/.mambarc'
trace Configuration not found at '/var/lib/conda/condarc.d/'
trace Configuration not found at '/var/lib/conda/condarc'
trace Configuration not found at '/var/lib/conda/.condarc'
trace Configuration not found at '/etc/conda/.mambarc'
trace Configuration not found at '/etc/conda/condarc.d/'
trace Configuration not found at '/etc/conda/condarc'
trace Configuration not found at '/etc/conda/.condarc'
trace Update configurable 'no_env'
trace Compute configurable 'file_specs'
trace Compute configurable 'spec_file_env_name'
trace Compute configurable 'env_name'
trace Compute configurable 'envs_dirs'
trace Compute configurable 'use_target_prefix_fallback'
trace Compute configurable 'target_prefix'
trace Get RC files configuration from locations up to TargetPrefix
trace Configuration not found at '/opt/conda/.mambarc'
trace Configuration not found at '/opt/conda/condarc.d'
trace Configuration not found at '/opt/conda/condarc'
trace Configuration not found at '/opt/conda/.condarc'
trace Configuration not found at '/home/mambauser/.mambarc'
trace Configuration not found at '/home/mambauser/.condarc'
trace Configuration not found at '/home/mambauser/.conda/condarc.d'
trace Configuration not found at '/home/mambauser/.conda/condarc'
trace Configuration not found at '/home/mambauser/.conda/.condarc'
trace Configuration not found at '/opt/conda/.mambarc'
trace Configuration not found at '/opt/conda/condarc.d'
trace Configuration not found at '/opt/conda/condarc'
trace Configuration not found at '/opt/conda/.condarc'
trace Configuration not found at '/var/lib/conda/.mambarc'
trace Configuration not found at '/var/lib/conda/condarc.d/'
trace Configuration not found at '/var/lib/conda/condarc'
trace Configuration not found at '/var/lib/conda/.condarc'
trace Configuration not found at '/etc/conda/.mambarc'
trace Configuration not found at '/etc/conda/condarc.d/'
trace Configuration not found at '/etc/conda/condarc'
trace Configuration not found at '/etc/conda/.condarc'
trace Update configurable 'no_env'
trace Compute configurable 'target_prefix_checks'
error Expected environment not found at prefix: /opt/conda
critical Aborting.
info ****************** Backtrace End ********************
$
It seems like in 0.14.0, $HOME
is set to "/home/micromamba"
, while still using the default root
user to execute the Dockerfile. This results in confusing behavior, where tilde expansion does not match the actual user running the Dockerfile.
(I ran into this because I need to set ~/.ssh/known_hosts
, so that I can pip install
a library from Github.)
% docker run --rm -it --platform linux/amd64 mambaorg/micromamba:0.14.0
(base) root@23e0e1238de6:/# whoami
root
(base) root@23e0e1238de6:/# echo $HOME
/home/micromamba
(base) root@23e0e1238de6:/# cd ~
(base) root@23e0e1238de6:~# pwd
/home/micromamba
On 0.13.1, $HOME
and ~
both map to /root
, which is what I would expect.
$HOME
environment variable in the micromamba Dockerfile?ARG VERSION=0.13.1
is out of date? Latest commit is: Line 6 in 5a3e6ad
Currently only the amd64 images get tested. All images should get tested before pushing.
So, I'm facing an issue in which I'm struggling to permanently activate Micromamba in a zsh shell.
I'm pulling the Micromamba Docker Hub image and installing Oh My Zsh afterwards using this method:
https://github.com/deluan/zsh-in-docker
Following the advice from micromamba --help
regarding permanently initialising the shell, I can get things to work temporarily.
I can use micromamba activate
/ micromamba deactivate
, and my environment is correctly found with micromamba list
.
However, if I then open up a new terminal in VS Code, things begin to fall apart. I need to repeat my actions for activate
and deactivate
to become available, and my environment is no longer correctly found with list
.
I imagine this is somewhat related to MAMBA_ROOT_PREFIX
but I'm too much of a n0ob to Linux & Docker to figure it out.
Any friendly words of advice here? Should I be pulling from the Micromamba base image at all if I intend to work with zsh, or should I build my own image from Debian-Slim like in your Dockerfile?
Much obliged for the great work.
I'm currently getting ERROR Could not read environment 'dependencies' as list of strings from /tmp/environment.yml
with the following Dockerfile
FROM mambaorg/micromamba:0.8.2
# create the conda environment and install cea...
# && remove Python Bytecode files & Static Libraries to save 400MB in space
COPY environment.yml /tmp/environment.yml
RUN micromamba install -y -n cea -f /tmp/environment.yml \
&& rm /opt/conda/pkgs/cache/* \
&& find /opt/conda/ -follow -type f -name '*.a' -delete \
&& find /opt/conda/ -follow -type f -name '*.pyc' -delete
& environment.yml
name: cea
channels:
- conda-forge
- defaults
- dlr-sc
- pythonocc
- oce
dependencies:
- affine=2.3.0
- attrs=19.3.0
- backcall=0.2.0
- beautifulsoup4=4.9.1
- blas=1.0
- bleach=3.1.5
- blosc=1.20.0
- boost-cpp=1.72.0
- branca=0.4.1
- brotlipy=0.7.0
- bs4=4.9.1
- bzip2=1.0.8
- ca-certificates=2020.6.20
- certifi=2020.6.20
- cffi=1.14.1
- cfitsio=3.470
- chardet=3.0.4
- click=7.1.2
- click-plugins=1.1.1
- cligj=0.5.0
- colorama=0.4.3
- cryptography=3.0
- curl=7.71.1
- cvxopt=1.2.5
- cycler=0.10.0
- decorator=4.4.2
- defusedxml=0.6.0
- deprecated=1.2.10
- descartes=1.1.0
- double-conversion=3.1.5
- dsdp=5.8
- eigen=3.3.7
- entrypoints=0.3
- esda=2.3.1
- expat=2.2.9
- fastcache=1.1.0
- ffmpeg=4.3
- fftw=3.3.8
- fiona=1.8.9.post2
- folium=0.11.0
- fontconfig=2.13.1
- freeimage=3.18.0
- freetype=2.10.2
- freexl=1.0.5
- future=0.18.2
- gdal=2.4.4
- geopandas=0.8.1
- geos=3.8.1
- geotiff=1.5.1
- gettext=0.19.8.1
- gevent=20.6.2
- gevent-websocket=0.10.1
- giddy=2.3.3
- gl2ps=1.4.2
- glew=2.1.0
- glib=2.65.0
- glpk=4.65
- greenlet=0.4.16
- gsl=2.6
- hdf4=4.2.13
- hdf5=1.10.5
- icc_rt=2019.0.0
- icu=64.2
- idna=2.10
- importlib-metadata=1.7.0
- importlib_metadata=1.7.0
- inequality=1.0.0
- intel-openmp=2020.1
- ipdb=0.13.0
- ipykernel=5.3.4
- ipython=7.16.1
- ipython_genutils=0.2.0
- ipywidgets=7.5.1
- jedi=0.17.2
- jinja2=2.11.2
- joblib=0.16.0
- jpeg=9d
- jsoncpp=1.8.4
- jsonschema=3.2.0
- jupyter
- kealib=1.4.13
- kiwisolver=1.2.0
- krb5=1.17.1
- libblas=3.8.0
- libcblas=3.8.0
- libclang=9.0.1
- libcurl=7.71.1
- libffi=3.2.1
- libgdal=2.4.4
- libiconv=1.15
- libkml=1.3.0
- liblapack=3.8.0
- libllvm9=9.0.1
- libnetcdf=4.7.4
- libogg=1.3.2
- libpng=1.6.37
- libpq=12.3
- libpysal=4.3.0
- libsodium=1.0.17
- libspatialindex=1.9.3
- libspatialite=4.3.0a
- libssh2=1.9.0
- libtheora=1.1.1
- libtiff=4.1.0
- libwebp-base=1.1.0
- libxml2=2.9.10
- llvmlite=0.33.0
- loguru=0.5.0
- lz4-c=1.9.2
- m2w64-expat=2.1.1
- m2w64-gcc-libgfortran=5.3.0
- m2w64-gcc-libs=5.3.0
- m2w64-gcc-libs-core=5.3.0
- m2w64-gettext=0.19.7
- m2w64-gmp=6.1.0
- m2w64-libiconv=1.14
- m2w64-libwinpthread-git=5.0.0.4634.697f757
- m2w64-xz=5.2.2
- mapclassify=2.3.0
- markupsafe=1.1.1
- matplotlib=3.3.0
- matplotlib-base=3.3.0
- mgwr=2.1.1
- mistune=0.8.4
- mkl=2020.1
- mkl-service=2.3.0
- mock=4.0.2
- mpmath=1.1.0
- msys2-conda-epoch=20160418
- munch=2.5.0
- nbconvert=5.6.1
- nbformat=5.0.7
- networkx=2.4
- nose=1.3.7
- notebook=6.0.3
- numba=0.50.1
- numexpr=2.7.1
- numpy=1.19.1
- occt=7.4.0
- olefile=0.46
- openjpeg=2.3.1
- openssl=1.1.1g
- osmnx=0.15.1
- packaging=20.4
- palettable=3.3.0
- pandas=1.0.5
- pandoc=2.10.1
- pandocfilters=1.4.2
- parso=0.7.1
- patsy=0.5.1
- pcre=8.44
- pickleshare=0.7.5
- pillow=7.2.0
- pip=20.1.1
- pointpats=2.1.0
- poppler=0.67.0
- poppler-data=0.4.9
- postgresql=12.3
- proj=7.0.0
- prometheus_client=0.8.0
- prompt-toolkit=3.0.5
- pugixml=1.10
- pycparser=2.20
- pygments=2.6.1
- pyopenssl=19.1.0
- pyparsing=2.4.7
- pyproj=2.6.1.post1
- pyqt=5.12.3
- pyrsistent=0.16.0
- pysal=2.1.0
- pysocks=1.7.1
- pytables=3.6.1
- python=3.7.8
- python-dateutil=2.8.1
- python_abi=3.7
- pythonocc-core=7.4.0
- pytz=2020.1
- pywin32=227
- pywinpty=0.5.7
- pyzmq=19.0.1
- qt=5.12.5
- quantecon=0.4.8
- rasterio=1.1.5
- rasterstats=0.14.0
- requests=2.24.0
- rtree=0.9.4
- scikit-learn=0.23.1
- scipy=1.5.0
- seaborn=0.10.1
- seaborn-base=0.10.1
- segregation=1.3.0
- send2trash=1.5.0
- setuptools=49.2.0
- shapely=1.7.0
- simplejson=3.17.2
- six=1.15.0
- snuggs=1.4.7
- soupsieve=2.0.1
- spaghetti=1.5.0
- spglm=1.0.7
- spint=1.0.6
- splot=1.1.3
- spreg=1.1.1
- spvcm=0.3.0
- sqlite=3.32.3
- statsmodels=0.11.1
- sympy=1.6.1
- tbb=2020.1
- terminado=0.8.3
- testpath=0.4.4
- threadpoolctl=2.1.0
- tiledb=2.0.6
- tk=8.6.10
- tobler=0.3.1
- tornado=6.0.4
- tqdm=4.48.0
- traitlets=4.3.3
- urllib3=1.25.10
- utfcpp=3.1.1
- vc=14.1
- vs2015_runtime=14.16.27012
- vtk=8.2.0
- wcwidth=0.2.5
- webencodings=0.5.1
- wheel=0.34.2
- widgetsnbextension=3.5.1
- win32_setctime=1.0.1
- win_inet_pton=1.1.0
- wincertstore=0.2
- winpty=0.4.3
- wntr=0.2.2.1
- wrapt=1.12.1
- xerces-c=3.2.2
- xz=5.2.5
- zeromq=4.3.2
- zipp=3.1.0
- zlib=1.2.11
- zope.event=4.4
- zope.interface=5.1.0
- zstd=1.4.5
- pip:
- alabaster==0.7.12
- aniso8601==8.0.0
- arviz==0.9.0
- babel==2.8.0
# - cea-workflow-utils==1.0.0
- cftime==1.2.1
- cityenergyanalyst==3.8.0a0
- colorlover==0.3.0
- cufflinks==0.17.3
- deap==1.3.1
- docutils==0.16
- ephem==3.7.7.1
- et-xmlfile==1.0.1
- fastprogress==0.2.3
- flask==1.1.2
- flask-restplus==0.13.0
- flask-socketio==4.3.1
- h5py==2.10.0
- imagesize==1.2.0
- itsdangerous==1.1.0
- jdcal==1.4.1
- keyring==21.2.1
- lxml==4.5.2
- netcdf4==1.5.4
- openpyxl==3.0.4
- pkginfo==1.5.0.1
- plotly==4.9.0
- psutil==5.7.2
- py4design-cea==0.30
- pycollada==0.7.1
- pymc3==3.9.2
- pyqt5-sip==4.19.18
- pyqtwebengine==5.12.1
- pyshp==2.1.0
- python-engineio==3.13.1
- python-socketio==4.6.0
- pywin32-ctypes==0.2.0
- pyyaml==5.3.1
- readme-renderer==26.0
- requests-toolbelt==0.9.1
- retrying==1.3.3
- rfc3986==1.4.0
- salib==1.3.11
- snowballstemmer==2.0.0
- staticmap==0.5.4
- theano==1.0.4
- timezonefinder==4.4.0
- twine==3.2.0
- typing-extensions==3.7.4.2
- utm==0.5.0
- werkzeug==1.0.1
- xarray==0.16.0
- xlrd==1.2.0
- xlwt==1.3.0
prefix: C:\Users\darthoma\Miniconda2\envs\cea-3.8.0a1
Full error message:
=> [internal] load build definition from Dockerfile.micromamba 0.0s
=> => transferring dockerfile: 48B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/mambaorg/micromamba:0.8.2 1.7s
=> [auth] mambaorg/micromamba:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 44.54kB 0.1s
=> [ 1/11] FROM docker.io/mambaorg/micromamba:0.8.2@sha256:5eae5f08305d921145942fd31b12f4060fd4fc7ef5cd35 0.0s
=> [ 5/11] COPY ./docker/environment.yml /tmp/environment.yml
=> ERROR [ 6/11] RUN micromamba install -y -n cea -f /tmp/environment.yml && rm /opt/conda/pkgs/cache 0.4s
------
> [ 6/11] RUN micromamba install -y -n cea -f /tmp/environment.yml && rm /opt/conda/pkgs/cache/* && find /opt/conda/ -follow -type f -name '*.a' -delete && find /opt/conda/ -follow -type f -name '*.pyc' -delete:
#11 0.355 ERROR Could not read environment 'dependencies' as list of strings from /tmp/environment.yml
------
executor failed running [/bin/bash -c micromamba install -y -n cea -f /tmp/environment.yml && rm /opt/conda/pkgs/cache/* && find /opt/conda/ -follow -type f -name '*.a' -delete && find /opt/conda/ -follow -type f -name '*.pyc' -delete]: exit code: 1
Also tried:
name: cea
channels:
- conda-forge
- defaults
dependencies:
- cvxopt
- deap
- docutils
- ephem
- flask
- flask-restplus
- flask-socketio
- gevent
- h5py
- ipykernel
- ipython
- jupyter
- lxml
- openpyxl
- plotly
- salib
- sphinx
- sympy
- theano
- utm
- wntr
- xlrd
- xlwt
- descartes
- fiona
- folium
- gdal
- geopandas
- geopy
- geos
- hdf4
- hdf5
- jinja2
- json-c
- llvmlite
- matplotlib
- networkx
- numba
- numpy
- osmnet
- osmnx
- pandana
- pandas
- pcre
- pip
- proj
- psutil
- pyproj
- pysal<2.3.0
- python>=3.7
- pythonocc-core
- pytz
- pyyaml
- scikit-learn
- scipy
- seaborn
- setuptools
- shapely
- urbanaccess
- yaml
- timezonefinder
- twine
- pymc3
- python-cufflinks
- gevent-websocket
- pip:
- opencv-contrib-python
- staticmap
- py4design-cea
Please let me know if you see anything obvious as am very interested in using micromamba
to slim down this docker
image instead of a multistage build! Thanks for creating / maintaining it, seems like a really nice alternative to miniconda
here!
As noted in issue #7, the auto builds are not using the current version of micromamba. The micromamba version needs to be in the environment variable VERSION when the image build command is called.
I am using mambaorg/micromamba:0.8.2
to run CI using Drone.
Here is my reducted Dockerfile:
FROM mambaorg/micromamba:0.8.2 as base
COPY ./resources/env.yml .
RUN cat env.yml
RUN micromamba install -y -n base -f env.yml && \
rm /opt/conda/pkgs/cache/*
COPY ./resources/requirements.txt .
RUN pip install -r requirements.txt
FROM {registry}/research-base:base as test
COPY ./resources/test_requirements.txt .
RUN pip install -r test_requirements.txt
COPY setup.cfg .
When I create the docker image and run it locally, micromamba is working and all runs smoothly. In the pipeline, however, when I run it like that:
- name: run tests
image: {registry}/research-base:test
commands:
- which python
- python --version
- which micromamba
- micromamba list
- python -m flake8 ./SE
- python -m pytest ./tests
Micromamba is seen, and python is the one installed via Yaml, BUT returns error on mcromamba list
28 | ERROR Prefix and name arguments are empty.
When I removed it, it tried running tests, but cant find dependencies.
When I tried adding this command:
- eval "$(./bin/micromamba shell hook -s bash)"
- micromamba activate
it returned this:
10 | The following argument was not expected: activate
11 | Run with --help for more information.
PS: Some additional info on docker:
Client: Docker Engine - Community
--
6 | Version: 19.03.13
7 | API version: 1.40
8 | Go version: go1.13.15
9 | Git commit: 4484c46
10 | Built: Wed Sep 16 16:58:04 2020
11 | OS/Arch: linux/amd64
12 | Experimental: false
13 |
14 | Server: Docker Engine - Community
15 | Engine:
16 | Version: 19.03.13
17 | API version: 1.40 (minimum version 1.12)
18 | Go version: go1.13.15
19 | Git commit: 4484c46
20 | Built: Wed Sep 16 17:04:43 2020
21 | OS/Arch: linux/amd64
22 | Experimental: false
23 | containerd:
24 | Version: v1.3.7
25 | GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
26 | runc:
27 | Version: 1.0.0-rc10
28 | GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
29 | docker-init:
30 | Version: 0.18.0
31 | GitCommit: fec3683
When using micromamba
in a container, I copy some local data to a container and try to change permissions and get the following error. This occurs during the # update permissions on sql documents
section below, when I'm trying to get permissions set correctly on a volume.
> [3/7] RUN mkdir /sql_documents && chmod a=rwx,g+s /sql_documents && setfacl -d -m u::rwx /sql_documents && setfacl -d -m g::rwx /sql_documents && setfacl -d -m o::rwx /sql_documents && chown --recursive micromamba:micromamba /sql_documents:
mamba-org/mamba#7 0.453 chown: invalid user: ‘micromamba:micromamba’
And here's my Dockerfile.
FROM mambaorg/micromamba:latest AS micromamba
EXPOSE 8000
USER root
RUN apt-get update && apt-get upgrade -y && apt-get install -y acl
# update permissions on sql documents
RUN mkdir /sql_documents && \
chmod a=rwx,g+s /sql_documents && \
setfacl -d -m u::rwx /sql_documents && \
setfacl -d -m g::rwx /sql_documents && \
setfacl -d -m o::rwx /sql_documents && \
chown --recursive micromamba:micromamba /sql_documents
USER micromamba
# copy fled directory to docker and install libraries & fled
COPY --chown=micromamba:micromamba . /fled
# install requirements into base environment
RUN micromamba install -y -n base -f /fled/env.yaml
# activate base environment
ARG MAMBA_DOCKERFILE_ACTIVATE=1
# install fled via pip
RUN pip install /fled
# cleanup
RUN micromamba clean --all --yes
CMD ["uvicorn", "fled.main:app", "--host", "0.0.0.0", "--port", "8000"]
This works with 0.19.0 but not 0.19.1.
I looked at the diffs between 0.19.0 and 0.19.1 and nothing stood out. I also looked at the micromamba/Changelog.md
and didn't notice anything.
I think it should now be easy to also build a docker image for ARM64 Linux.
Basically the same process, just using a different URL for micromamba (linux-aarch64) and base image.
Hi there,
Thanks for creating Micromamba and Micromamba Docker!
I followed the instructions to create a Dockerfile and no matter which version I use (0.21.2, 0.21.1, 0.19.0) I get:
error No prefix found at: /opt/conda/envs/<my-env-name>
critical Aborting.
when running micromamba install -y -f /tmp/env.yml
. I also get this when I try to create an environment interactively.
Likewise, if I run a plain micromamba list
interactively, I get:
error Expected environment not found at prefix: /opt/conda
critical Aborting.
I guess I must be doing something fundamentally wrong. Could someone please point the mistake out?
Many thanks,
Andreas
As I remarked in #119 (comment), neither tags of the form 0.20.0
nor git-abc1234
are sufficient to uniquely specify an image. (I feel like our current recommendation of using digests is correct, but overkill.) One possible solution would be to use combined tags like 0.20.0-git-abc1234
. However that solution masks another underlying problem, namely that the current GH Action fails to run the update.sh
script, so the readme and ARG VERSION
have the wrong version numbers. If this were fixed, then an upstream version change would trigger a commit here, and therefore the git-style tags would become unique.
I think it's better in this case to solve the underlying problem and run update.sh
on GH. Unfortunately this involves setting up git credentials in the Action. I'm not yet experienced enough on GH Actions to comprehend all the security implications... (I'm sure it's not that hard, but it probably deserves diligence from someone with more experience than I have, since this isn't a good case for e.g. copy-pasting from a random blog. 😂)
Reproduce with:
docker pull mambaorg/micromamba:0.7.14
docker run mambaorg/micromamba:0.7.14 mamba --version
# 0.7.13
Hi,
I've run into a problem where I need to be able to execute commands via docker exec
or similar, however unless the image is entered into interactively the conda env is not activated.
Example:
docker run -it --rm --name test -d mambaorg/micromamba
docker exec test micromamba install
This results in the following:
info ****************** Backtrace Start ******************
debug Loading configuration
trace Compute configurable 'create_base'
trace Compute configurable 'no_env'
trace Compute configurable 'no_rc'
trace Compute configurable 'rc_files'
trace Compute configurable 'root_prefix'
trace Get RC files configuration from locations up to HomeDir
trace Configuration not found at '/home/mambauser/.mambarc'
trace Configuration not found at '/home/mambauser/.condarc'
trace Configuration not found at '/home/mambauser/.conda/condarc.d'
trace Configuration not found at '/home/mambauser/.conda/condarc'
trace Configuration not found at '/home/mambauser/.conda/.condarc'
trace Configuration not found at '/opt/conda/.mambarc'
trace Configuration not found at '/opt/conda/condarc.d'
trace Configuration not found at '/opt/conda/condarc'
trace Configuration not found at '/opt/conda/.condarc'
trace Configuration not found at '/var/lib/conda/.mambarc'
trace Configuration not found at '/var/lib/conda/condarc.d/'
trace Configuration not found at '/var/lib/conda/condarc'
trace Configuration not found at '/var/lib/conda/.condarc'
trace Configuration not found at '/etc/conda/.mambarc'
trace Configuration not found at '/etc/conda/condarc.d/'
trace Configuration not found at '/etc/conda/condarc'
trace Configuration not found at '/etc/conda/.condarc'
trace Update configurable 'no_env'
trace Compute configurable 'file_specs'
trace Compute configurable 'spec_file_env_name'
trace Compute configurable 'env_name'
trace Compute configurable 'envs_dirs'
trace Compute configurable 'use_target_prefix_fallback'
trace Compute configurable 'target_prefix'
trace Get RC files configuration from locations up to TargetPrefix
trace Configuration not found at '/home/mambauser/.mambarc'
trace Configuration not found at '/home/mambauser/.condarc'
trace Configuration not found at '/home/mambauser/.conda/condarc.d'
trace Configuration not found at '/home/mambauser/.conda/condarc'
trace Configuration not found at '/home/mambauser/.conda/.condarc'
trace Configuration not found at '/opt/conda/.mambarc'
trace Configuration not found at '/opt/conda/condarc.d'
trace Configuration not found at '/opt/conda/condarc'
trace Configuration not found at '/opt/conda/.condarc'
trace Configuration not found at '/var/lib/conda/.mambarc'
trace Configuration not found at '/var/lib/conda/condarc.d/'
trace Configuration not found at '/var/lib/conda/condarc'
trace Configuration not found at '/var/lib/conda/.condarc'
trace Configuration not found at '/etc/conda/.mambarc'
trace Configuration not found at '/etc/conda/condarc.d/'
trace Configuration not found at '/etc/conda/condarc'
trace Configuration not found at '/etc/conda/.condarc'
trace Update configurable 'no_env'
trace Compute configurable 'target_prefix_checks'
error No target prefix specified
critical Aborting.
info ****************** Backtrace End ********************
Was wondering if this could be added to the micromamba image or any ideas of how to enable it?
Currently the test suite runs on the version of micromamba in ARG VERSION
. This is rarely the version that will be used by the build action. Use some of the code in check_version.py
to automate setting of ARG VERSION
with in the test suite.
In the spirit of #61, I wonder if we should eliminate the init
and activate
commands for the root
and micromamba
users in the Dockerfile.
Even though we activate by default in our ENTRYPOINT
, this feels to me like it could be a bit extreme.
A middle ground might be (in the context of #69)
echo "source _activate_current_env.sh" >> ~/.bashrc
At least this way it would work in case MAMBA_ROOT_PREFIX
changes.
What's the best way to use this image in multi-stage builds?
I'm interested in applications where the first stage is to create an environment using micromamba
, and then in the secod stage, with a different base image, that environment is copied to the new image. For example if the second stage needs to be a nvidia/cuda
based image, etc. Like the following
FROM mamabaorg/micromamba AS micromamba
# Create env here
FROM new_base
COPY --from=micromamba ???
# set some env vars, bashrc?
It seems like /bin/conda
and /bin/mamba
, which were available in 0.13.1, are no longer present in 0.14.0?
$ docker run --rm -it --platform linux/amd64 mambaorg/micromamba:0.14.0
(base) root@bc1eb1302a18:/# which -a micromamba
/bin/micromamba
(base) root@bc1eb1302a18:/# which -a conda
(base) root@bc1eb1302a18:/# which -a mamba
I tried testing with micromamba 0.14.1, but it seems like that's not available on Docker Hub: https://hub.docker.com/r/mambaorg/micromamba/tags
There are many different ways to run a program under from a docker image or during an image build. We need to test that a micromamba installed program can be found when run in each of these ways. In the examples below, we will use python as the example micromamba installed program:
docker run micromamba-derived python
docker run micromamba-derived --entrypoint python
# does this really need to work? Probably not.RUN python -c "import os; os.system('touch foobar')"
RUN ["python", "-c", "import os; os.system('touch foobar')"]
CMD python -c "print('hello')"
CMD ["/opt/conda/bin/python", "-c", "print('hello')"]
An automated test suite needs to be made to test that the micromamba docker image makes installed packages accessible in all of these contexts. I was hoping to make use of container-structure-test, but it always overrides the entrypoint setting, which makes it unfit for this type of testing. BATS might be a good option.
Hi there,
thanks for this great container! I'd like to mention, that cleaning via mamba after the installation saves additional memory:
rm /opt/conda/pkgs/cache/* && \
mamba clean -ay
Does cleaning makes the deletion of the pkgs cache optional? A high-level interface call should be preferred, right?
Eventually this should be added to the build instructions (as people tend to copy/paste/adopt things).
I think we need to run update.sh
In order to be able to reproduce image builds, one really should render the environment specification into a lockfile before building, and then use the lockfile (and not the environment specification) during build to set up the environment with micromamba.
I think it would suffice to add a section to "Advanced Usages".
Will need to update the test suite to run on all base images.
Hi guys,
I was looking for alternative docker to python3.8+ that runs on osx-arm64 and I've tried your package only to get this error:
The bug seems to appeared in 0.15.3
This command fails on macos m1:
docker run -it --rm mambaorg/micromamba:0.17.0 /bin/bash
/bin/entrypoint.sh: line 16: /bin/micromamba: No such file or directory
docker run -it --rm mambaorg/micromamba:0.15.3 /bin/bash -c 'micromamba --version '
/bin/entrypoint.sh: line 16: /bin/micromamba: No such file or directory
But the following working just fine
docker run -it --rm mambaorg/micromamba:0.15.2 /bin/bash -c 'micromamba --version '
0.15.2
docker run -it --rm mambaorg/micromamba:0.15.0 /bin/bash -c 'micromamba --version '
0.15.0
docker run -it --rm mambaorg/micromamba:0.14.0 /bin/bash -c 'micromamba --version '
0.14.0
The documentation of the mambaorg/micromamba recommends the use of $MAMBA_USER
to avoid hardcoding the mamba user (which is nice).
However, this seems not to work when using buildah rather than Docker: see e.g. this build failure:
For now, I am working around it by hardcoding mambauser which does the job for now.
So nothing urgent :-)
Context: buildah is a convenient tool to build docker images in CI jobs as it does not require
special privileges, like e.g. docker-in-docker does.
See #101 for an example use case and an initial solution.
Hello,
sorry if this is the wrong place to ask this, but I am have a python project that I am trying to dockerize via mamba. This makes building my image much much faster, so before the question: thank you very much for making this!!!
My project needs to do some git calls, so I need git available inside the container. I tried adding the following to my Dockerfile:
FROM mambaorg/micromamba:0.15.2
USER root
RUN apt update && apt install -y git-all
USER micromamba
COPY --chown=micromamba:micromamba environment.yml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml
WORKDIR /app
However, this makes the container really really big. The install of git claims:
1 upgraded, 403 newly installed, 0 to remove and 1 not upgraded.
Need to get 195 MB of archives.
After this operation, 903 MB of additional disk space will be used.
Do you want to continue? [Y/n] Abort
is there anyway to get this smaller? I can do just apt install -y git
rather than git-all
but eveen that is 104 MB
Sorry for the beginner question, I am still learning docker, so there might be something obvious I have overlooked.
When I use a python command in an ENTRYPOINT
layer at the end on my Dockerfile, I get bash: line 1: python: command not found
I would love to hear your insight if I've mistaken somewhere (or if the issue isn't related to Mamba) :)
This is my Dockerfile.
FROM mambaorg/micromamba:0.19.1
WORKDIR /app
ENV LISTEN_PORT 80
COPY --chown=micromamba:micromamba base-env.yaml base-env.yaml
RUN micromamba install -y -f base-env.yaml && \
micromamba clean --all --yes
# (otherwise python will not be found)
ARG MAMBA_DOCKERFILE_ACTIVATE=1
COPY . .
WORKDIR ./src
ENTRYPOINT python -m service.start
Need to investigate: https://github.com/krallin/tini
I am using micromamba with a large set of both conda and pip dependencies. When running docker build
, the micromamba install
step fails to install some dependencies, but the docker build
continues without error, leading to an image missing a number of dependencies.
I'm using Docker Desktop on Mac, and I noticed that increasing the memory resources for Docker from the default of 2.0GB up to 2.5GB results in successful build, so it seems like insufficient memory is causing the micromamba install
failure.
The main issue is that the docker build
completes successfully even though micromamba failed to install all the packages, resulting in a broken image. Ideally, micromamba would cause the build to fail.
My dockerfile:
# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 mambaorg/micromamba:0.13.1
COPY environment-lock.yml /environment-lock.yml
RUN --mount=type=cache,target=/opt/conda/pkgs --mount=type=cache,target=/root/.cache \
micromamba install -y --file /environment-lock.yml
The environment-lock.yml is very large, and includes both conda and pip dependencies. I tried paring it down, but it was hard to reproduce the failure with a more minimal list. Let me know if you want me to share the full list.
With --progress plain
, I see the following output from docker build
while micromamba is in the middle of installing pip dependencies specified in environment-lock.yml (this is after the conda deps were installed):
[...]
#9 441.2 Collecting tensorboard-plugin-wit==1.8.0
#9 441.2 Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
#9 442.7 Collecting tensorflow==2.5.0
#9 442.7 Downloading tensorflow-2.5.0-cp38-cp38-manylinux2010_x86_64.whl (454.4 MB)
#9 466.3 /tmp/mambafVPPFiskWCz: line 4: 2363 Killed /opt/conda/bin/pip install absl-py==0.11.0 apache-beam==2.30.0 [...a bunch of other pip packages...] xarray==0.16.2 --no-input
#9 DONE 466.7s
With some random poking, I found an example /tmp/mamba* file that looked like this:
lib/python3.8/site-packages/ansiwrap/__init__.py
lib/python3.8/site-packages/ansiwrap/ansistate.py
lib/python3.8/site-packages/ansiwrap/core.py
eval "$('/bin/micromamba' 'shell' 'hook' '-s' 'bash' '-p' '/opt/conda')"
It seems like the "Line 4: ... Killed" is referring to the final line in the script mentioned above. I think the problem is that the system is terminating the command on line 4 due to running out of memory, but this failure isn't propagated to the RUN micromamba install ...
, so the docker build
succeeds.
I think including set -e
or using exec
in this script would cause the error to propagate to micromamba.
Some instances of people saying they needed to increase the memory limits for docker build:
A quick (not necessarily accurate) skim of my docker image ls
shows a whole lot of superfluous tags generated by the test suite:
run-exec-form
run-shell-form
different-user
new-user
cmd-shell-form
cmd-exec-form
cli-invocations
mamba-1322-workaround
multi-env
We should probably be tagging these as micromamba:test-run-exec-form
, etc.
I'd submit a PR to fix this, but there are already enough merge conflicts in my current PRs. 😝
I'd like to find a way to keep the documentation on dockerhub in sync with the README.md on the main branch of the git repo.
Hey @wholtz , when do you think there will be a release to Docker Hub with a Debian Bullseye based image?
Thanks in advance!
Constraints:
There will be three steps to implementing this change:
Hello! Happy holidays! I just ran into some issues with the 0.19.1 micromamba image on Docker Hub. I'm getting the error "command pip not found" when I mamba-install dependencies and then try to pip install a local package copied to the Docker container.
RUN micromamba install -y -n base -f /fled/env.yaml && \
pip install /fled && \
micromamba clean --all --yes
where env.yaml
is:
name: base
channels:
- conda-forge
dependencies:
- python=3.9
- sqlmodel
- fastapi
- pandas
- networkx
- uvicorn
- pip
- beartype
- neo4j-python-driver
Which results in:
I also tested 0.19.0 and get some linking warnings with several packages like libpng.
When I go back to 0.18.2 everything works great. I looked and 0.19.1 is maybe pre-release or something, since it's not marked as a release on the mamba repo? I was looking at the wrong tag.
Just wanted to raise this issue in case other people run into it.
Thanks for mamba and micromamba! They are wonderful tools!
Hi!
I get a permissions error when trying to use docker volumes under mcOS with mambaorg/micromamba
base image.
Here is a simple dockerfile:
FROM mambaorg/micromamba:latest
WORKDIR /usr/app
VOLUME ["/usr/app/logs"]
RUN micromamba install -y -n base -c conda-forge \
python \
requests && \
micromamba clean --all --yes
COPY src/. .
I build the image and run it with
docker run -it -v test-logs:/usr/app/logs test-image /bin/bash
Then I try to create a file in /usr/app/logs
:
(base) micromamba@b09922ab3d15:/usr/app$ cd logs
(base) micromamba@b09922ab3d15:/usr/app/logs$ touch foo
touch: cannot touch 'foo': Permission denied
I'm pretty sure this happens because of the custom micromamba
user having issues with macOS file permissions. Other Docker images using root
user don't have such issues.
I had to make the experience that an image build where we had based from (tagged) 0.19.1 started to fail later, because the image on dockerhub with that tag changed.
I haven't checked whether you have now already checks in place, but in case not, I would like to encourage you to never "move" tags of versions, with "latest" being the exception.
Since size is obviously a concern for this image, I was wondering whether you are interested in replacing debian:buster-slim
with bitnami/minideb:buster
which is even smaller. I ran a quick local test replacing the $BASE_IMAGE
and it seems to work just fine but it's hard for me to imagine all the implications.
I'm happy to create a PR if that's desired.
Whenever micromamba shell hook -s bash
is executed and $MAMBA_ROOT_PREFIX/etc/profile.d/mamba.sh
exists, then the output seems to be mamba.sh
. The standard mamba.sh
initialization script defines only the mamba()
shell function, leaving conda()
and micromamba()
undefined.
The entrypoint.sh
for this image calls
eval "$(/bin/micromamba shell hook -s bash)"
micromamba activate "$ENV_NAME"
If mamba.sh
exists (for example if the user installs mamba
), then the micromamba
command above refers to the binary instead of the shell function, it results in the following error message:
The following arguments were not expected: base activate
Run with --help for more information.
To reproduce, use either of these Dockerfiles:
FROM mambaorg/micromamba:0.18.2
RUN micromamba install -c conda-forge --prefix=$MAMBA_ROOT_PREFIX mamba
FROM mambaorg/micromamba:0.18.2
RUN mkdir -p /opt/conda/etc/profile.d/ \
&& echo "echo Ran mamba.sh" > /opt/conda/etc/profile.d/mamba.sh
and then run
docker build . -t micromamba-docker-issue-57
docker run --rm -it micromamba-docker-issue-57
and you will see the output
Ran mamba.sh
The following arguments were not expected: base activate
Run with --help for more information.
It seems to me like the proper fix should probably be upstream in micromamba to ensure that micromamba()
, mamba()
, and conda()
are always defined in the shell init. For example, maybe it should ignore mamba.sh
, or use mamba.sh
and append the definition of micromamba()
?
Hi @wholtz
I have completed the github action pipeline now, and we build & push (automatically) the freshest releases to mambaorg/micromamba
now.
Is it OK for you if we change the reference in the readme? I made you a maintainer of that image as well.
The github action checks for a new release every 6 hours and pushes one if there is a release.
What do you see as the best way to make micromamba available to a non-root user? Due to the defined environment variable BASH_ENV=/root/.bashrc
, I see permission errors. A simple way could be to make /root/.bashrc
readable for everyone but that seems wrong.
Also, I guess I would need to activate the conda base environment the same way as was done for the root user. Do any other pitfalls come to your mind?
Version: 0.15.2
Since the user is micromamba
, but the default user executing COPY/ADD in Docker is root
, files that get copied over cannot be changed or modified by user micromamba
.
Failing example:
ADD script.sh /home/micromamba/script.sh
RUN chmod +x /home/micromamba/script.sh
Error: changing permissions of '/home/micromamba/script.sh': Operation not permitted
Explanation: script.sh
is owned by root
, user micromamba
cannot change root
files.
The problem and a solution is detailed here: moby/moby#6119 (comment)
ADD --chown=micromamba:micromamba script.sh /home/micromamba/script.sh
I am leaving this here for others to see, took me a while to figure out.
It might be good to add env.yaml
using chown, so users of this image have a example to go with.
It seems that micromamba
returns an exit code 1
on
micromamba list foo
(i.e. for an non-existing package), which is inconsistent with conda
Ref mamba-org/provision-with-micromamba#41
Could we also publish an image based on ubuntu:20.04
(focal)?
The test suite now calls check_version.py
. The dev docs need to indicate this requirement. Should also come up with an easy and standard way to install the deps.
I'm building Docker images based on this that I would like to use in nextflow pipelines.
Nextflow runs commands with a shebang
#!/bin/bash -ue
which is out of my control.
Unfortunately, in /opt/conda/etc/profile.d/mamba.sh
the PS1
variable is undefined. This causes an error due to the -u
flag and aborts my entire pipeline. Do you see that the micromamba image could at least set PS1
even without a value? Otherwise I can, of course, do so in my derived images.
It seems that this is done anyway at the end of the script:
if [ -z "${CONDA_SHLVL+x}" ]; then
\export CONDA_SHLVL=0
# In dev-mode MAMBA_EXE is python.exe and on Windows
# it is in a different relative location to condabin.
if [ -n "${_CE_CONDA+x}" ] && [ -n "${WINDIR+x}" ]; then
PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
else
PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
fi
\export PATH
# We're not allowing PS1 to be unbound. It must at least be set.
# However, we're not exporting it, which can cause problems when starting a second shell
# via a first shell (i.e. starting zsh from bash).
if [ -z "${PS1+x}" ]; then
PS1=
fi
fi
I'd prefer to get rid of this line which prepends $MAMBA_ROOT_PREFIX/bin
to PATH
.
Here are a few reasons why I think that line is a bad idea:
MAMBA_ROOT_PREFIX
is changed by the user. Then the default is still on the path, but the new value is not.SHELL
or ENTRYPOINT
, see #60.)Does this make sense? Is there some use case I'm missing?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.