conda / conda-pack Goto Github PK

View Code? Open in Web Editor NEW

477.0 477.0 85.0 923 KB

Package conda environments for redistribution

Home Page: https://conda.github.io/conda-pack/

License: BSD 3-Clause "New" or "Revised" License

Python 96.94% Shell 1.55% Batchfile 1.51%

conda-pack's People

Contributors

Stargazers

Watchers

Forkers

rootopia jcrist jakirkham mrocklin radovankavicky gapdata nehaljwani kashp sodre dsludwig mingwandroid mandeep speleo3 zjiaksmc soapy1 chrisburr xhochy bbbbb38438 amangarg96 masseeh johanjeppsson mattf-apache cjh1 trendelkampschroer argilla-io callumjhays xinggui007 sed-inria-sophia nunofernandes-plight jarnorfb guziy saraedum lunarlanding global-localhost global19 global19-atlassian-net siboehm ehmoussi jaimergp vvzen classicvalues guilhermebs rigzba21 lzh111222334 jtilly kenodegard rafiyr abolla22 calebho python-repository-hub scdub rtanti isabella232 t0nych3n danieljmv01 trellixvulnteam dholth dbast jjhelmus samkenxstream arpitjain799 morotti e8035669 agomezh-nav linuxpham aifimmunology andysham patricklabatut alex-salcido wh1isper bakiaglilit krishanbhasin math411 jharlow-intel mwies-cldr pgwhalen melund kirill888 aryarm farhakousar1601 ayurchuk

conda-pack's Issues

Getting a "Not a conda environment: new_env/bin/activate"

I tried to export the following env with conda-pack:

name: GIS
channels:
  - conda-forge
  - defaults
dependencies:
  - conda-forge/label/dev::rasterio
  - fiona
  - shapely
  - pip:
    - conda-pack

I executed the following commands to accomplish this:

conda env create --name GIS environment.yml
conda activate GIS
conda-pack
conda deactivate
mkdir new_env
tar -xf GIS.tar.gz -C new_env
source activate new_env/bin/activate

but I'm getting,

Not a conda environment: ~/computer_one/new_env/bin/activate

Am I doing something wrong here?

I know that that the env is not very conventional, this is part of a demonstration about the power of conda environments where we want to show how to get a dev package, get packages from multiple channels, packages from pip, and re-distribute the env with conda-pack.

Incompatible/buggy tar file created under windows when hardlinks encountered

Hi,

It seems there's a bug in conda-pack tarfile when creating
archives under windows containing hardlinked files.

Bug creates errors and exit code 2 when unpacking using gnu tar
and failure to untar the whole archive when unpacking using 7zip.

The final gnu tar error is:

tar: Exiting with failure status due to previous errors

The intermediate gnu tar errors are of the form:

tar: Skipping to next header

OS: Windows 10 as archiver

Versions:

Conda: 4.6.14
Python: Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
conda-pack 0.3.1

Steps to reproduce:

Install miniconda under windows

conda create --name testenv
conda install -c conda-forge conda-pack
conda update --all
conda activate testenv
conda install sphinx
conda deactivate
conda pack -n testenv -o testenv.tgz

Then unpack testenv.tgz with gnu tar under Linux.

The files that cause the error are hardlinked.
For example these files in envs/testenv/Scripts:

74,752 sphinx-apidoc.exe
74,752 sphinx-autogen.exe
74,752 sphinx-build.exe
74,752 sphinx-quickstart.exe

There are a number of reports of similar errors in python tarfile, it
seems. It might be good to warn of probs with hardlinked files using
tar formats under Windows.

Thanks for coda-pack!

===Rich

Updated 12may19

How to replicate your R example using SparkR and not Sparkly R

Hello,

I followed the steps outlined here : https://conda.github.io/conda-pack/spark.html for distributing a Conda r env in spark and it worked! I am attempting to replicate the same with SparkR but not being able to.

I have completed the following:
conda create -p /home/ubuntu/r_env --copy -y -q r-essentials -c r
conda install conda-pack

conda activate example
conda pack -o environment.tar.gz -d ./environment

and I am unsure if the following spark session is correct:

library(SparkR)
sparkR.session(master = "yarn",sparkConfig = list(
spark.yarn.dist.archives="environment.tar.gz#environment",
spark.num.executors="3",
spark.executor.cores="3",
spark.executor.memory="6G",
spark.r.command = "./environment/bin/Rscript",
spark.yarn.appMasterEnv.R_HOME="./environment/lib/R",
spark.yarn.appMasterEnv.RHOME = "./environment",
spark.yarn.appMasterEnv.R_SHARE_DIR = "./environment/lib/R/share",
spark.yarn.appMasterEnv.R_INCLUDE_DIR = "./environment/lib/R/include",
spark.yarn.executorEnv.R_HOME="./environment/lib/R",
spark.yarn.executorEnv.RHOME = "./environment",
spark.yarn.executorEnv.R_SHARE_DIR = "./environment/lib/R/share",
spark.yarn.executorEnv.R_INCLUDE_DIR = "./environment/lib/R/include"
))

Will the conda-unpack not replace the #!/usr/bin/env python in executables in an environment?

I have an environment which has gunicorn for eg. installed in it. When I run conda-unpack after extracting the packed environment the gunicorn executable still has "#!/usr/bin/env python" as a shebang line. Shouldn't it be "#!/usr/bin/env <env_path>/bin/python" ? Should we always run the executables by activating the environment? What if I want to fork the gunicorn process from within another program. Is there a way to programmatically activate an environment?

`Expected hostname at index` Error

When I try to submit hello world example from the documentation, I get the following error.

(py35) hussainsultan@cluster-07d2-m:~$ skein application submit helloworld-skein.yaml
Error: Failed to submit application, exception:
Expected hostname at index 7: file://:0/home/hussainsultan/payload.txt

Ignore editable packages (expert mode)

I have a setting where I want to pack an environment with editable packages. For these package, I have a dedicated mechanism to restore them on environment recreation. Would it be possible to add a flag to the packing mechanism that doesn't bundle them?

conda-unpack.exe does not work for 32-bit windows

When packing a 32-bit windows environment, I am unable to run conda-unpack.exe, which I assume is due to it being a 64-bit executable. The packed environment itself seems to work fine.

I had a look and came up with something like this, but I am unsure if this is the way to go, and how to ensure that I don't break anything. Any suggestions?

CondaPackError: Files managed by conda were found to have been deleted/overwritten

Following issue #78 - this reproduces an error

We are running on OS :

openSUSE Leap 42.3

setting up environment with

conda create -n test-pack  -y -c conda-forge   python=3.6
conda activate test-pack
conda install -c conda-forge --force-reinstall -y pandas-profiling

We get an error when running:

conda pack
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

    statsmodels='0.10.0'

Here is the results of conda list --export:

(test-pack) ds@linux-vnlc:~> conda list --export
This file may be used to create an environment using:
$ conda create --name --file
platform: linux-64
_libgcc_mutex=0.1=main
asn1crypto=0.24.0=py36_1003
astroid=2.2.5=py36_0
astropy=3.2.1=py36h516909a_0
atomicwrites=1.3.0=py_0
attrs=19.1.0=py_0
bleach=3.1.0=py_0
ca-certificates=2019.6.16=hecc5488_0
certifi=2019.6.16=py36_1
cffi=1.12.3=py36h8022711_0
chardet=3.0.4=py36_1003
confuse=1.0.0=py_0
cryptography=2.7=py36h72c5cf5_0
cycler=0.10.0=py_1
dbus=1.13.6=he372182_0
decorator=4.4.0=py_0
defusedxml=0.5.0=py_1
entrypoints=0.3=py36_1000
expat=2.2.5=he1b5a44_1003
fontconfig=2.13.1=h86ecdb6_1001
freetype=2.10.0=he983fc9_0
gettext=0.19.8.1=hc5be6a0_1002
glib=2.58.3=h6f030ca_1002
gst-plugins-base=1.14.5=h0935bb2_0
gstreamer=1.14.5=h36ae1b5_0
htmlmin=0.1.12=py_1
icu=64.2=he1b5a44_0
idna=2.8=py36_1000
importlib_metadata=0.18=py36_0
ipython_genutils=0.2.0=py_1
isort=4.3.21=py36_0
jinja2=2.10.1=py_0
jpeg=9c=h14c3975_1001
jsonschema=3.0.1=py36_0
jupyter_client=5.3.1=py_0
jupyter_core=4.4.0=py_0
kiwisolver=1.1.0=py36hc9558a2_0
lazy-object-proxy=1.4.1=py36h516909a_0
libblas=3.8.0=10_openblas
libcblas=3.8.0=10_openblas
libffi=3.2.1=he1b5a44_1006
libgcc-ng=9.1.0=hdf63c60_0
libgfortran-ng=7.3.0=hdf63c60_0
libiconv=1.15=h516909a_1005
liblapack=3.8.0=10_openblas
libopenblas=0.3.6=h6e990d7_5
libpng=1.6.37=hed695b0_0
libsodium=1.0.17=h516909a_0
libstdcxx-ng=9.1.0=hdf63c60_0
libuuid=2.32.1=h14c3975_1000
libxcb=1.13=h14c3975_1002
libxml2=2.9.9=hee79883_2
llvmlite=0.29.0=py36hfd453ef_1
markupsafe=1.1.1=py36h14c3975_0
matplotlib=3.1.1=py36_1
matplotlib-base=3.1.1=py36he7580a8_1
mccabe=0.6.1=py_1
missingno=0.4.2=py_0
mistune=0.8.4=py36h14c3975_1000
more-itertools=7.1.0=py_0
nbconvert=5.5.0=py_0
nbformat=4.4.0=py_1
ncurses=6.1=hf484d3e_1002
numba=0.45.0=py36hb3f55d8_0
numpy=1.16.4=py36h95a1406_0
openblas=0.3.6=h6e990d7_5
openssl=1.1.1c=h516909a_0
packaging=19.0=py_0
pandas=0.25.0=py36hb3f55d8_0
pandas-profiling=2.3.0=py_0
pandoc=2.7.3=0
pandocfilters=1.4.2=py_1
patsy=0.5.1=py_0
pcre=8.41=hf484d3e_1003
phik=0.9.8=py_0
pip=19.2.1=py36_0
pluggy=0.12.0=py_0
psutil=5.6.3=py36h516909a_0
pthread-stubs=0.4=h14c3975_1001
py=1.8.0=py_0
pycparser=2.19=py36_1
pygments=2.4.2=py_0
pylint=2.3.1=py36_0
pyopenssl=19.0.0=py36_0
pyparsing=2.4.1.1=py_0
pyqt=5.9.2=py36hcca6a23_0
pyrsistent=0.15.4=py36h516909a_0
pysocks=1.7.0=py36_0
pytest=5.0.1=py36_1
pytest-arraydiff=0.3=py_0
pytest-astropy=0.5.0=py_0
pytest-doctestplus=0.3.0=py_0
pytest-openfiles=0.3.1=py_0
pytest-pylint=0.14.0=py_0
pytest-remotedata=0.3.1=py_0
pytest-runner=5.1=py_0
python=3.6.7=h357f687_1005
python-dateutil=2.8.0=py_0
pytz=2019.1=py_0
pyyaml=5.1.1=py36h516909a_0
pyzmq=18.0.2=py36h1768529_2
qt=5.9.7=h0c104cb_3
readline=8.0=hf8c457e_0
requests=2.22.0=py36_1
scipy=1.3.0=py36h921218d_1
seaborn=0.9.0=py_1
setuptools=41.0.1=py36_0
sip=4.19.8=py36hf484d3e_1000
six=1.12.0=py36_1000
sqlite=3.29.0=hcee41ef_0
statsmodels=0.10.1=py36hc1659b7_0
testpath=0.4.2=py_1001
tk=8.6.9=hed695b0_1002
tornado=6.0.3=py36h516909a_0
traitlets=4.3.2=py36_1000
typed-ast=1.4.0=py36h516909a_0
urllib3=1.25.3=py36_0
wcwidth=0.1.7=py_1
webencodings=0.5.1=py_1
wheel=0.33.4=py36_0
wrapt=1.11.2=py36h516909a_0
xorg-libxau=1.0.9=h14c3975_0
xorg-libxdmcp=1.1.3=h516909a_0
xz=5.2.4=h14c3975_1001
yaml=0.1.7=h14c3975_1001
zeromq=4.3.2=he1b5a44_2
zipp=0.5.1=py_0
zlib=1.2.11=h516909a_1005

conda-pack fails for this environment spec

Got the following exception for the spec given below even though none of the packages listed in the exception (jinja2) are installed/overwritten by pip at all.

Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- jinja2=u'2.10'

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

Environment config:

channels:
  - conda-forge
  - defaults
dependencies:
  - attrs=18.2.0=py_0
  - blas=1.1=openblas
  - ca-certificates=2018.10.15=ha4d7672_0
  - certifi=2018.10.15=py27_1000
  - click=7.0=py_0
  - cloudpickle=0.2.1=py27_2
  - conda-pack=0.3.0=py_0
  - docopt=0.6.2=py_1
  - fastavro=0.21.13=py27h470a237_0
  - filechunkio=1.8=py_2
  - flask=1.0.2=py_2
  - gunicorn=19.9.0=py27_1000
  - ipykernel=4.10.0=py27_1
  - ipython=5.8.0=py27_0
  - itsdangerous=1.1.0=py_0
  - jinja2=2.10=py_1
  - jsonschema=3.0.0a3=py27_1000
  - jupyter_client=5.2.3=py_1
  - jupyter_core=4.4.0=py_0
  - krb5=1.16.2=hbb41f41_0
  - libgfortran=3.0.0=1
  - libsodium=1.0.16=h470a237_1
  - libxgboost=0.80=hfc679d8_1
  - markupsafe=1.1.0=py27h470a237_0
  - mkl_fft=1.0.6=py27_0
  - mkl_random=1.0.2=py27_0
  - mock=2.0.0=py27_1000
  - numpy=1.13.3=py27_blas_openblas_201
  - openblas=0.2.20=8
  - openssl=1.0.2p=h470a237_1
  - pandas=0.23.4=py27hf8a1672_0
  - pip=10.0.1=py27_0
  - psutil=5.4.8=py27h470a237_0
  - py-xgboost=0.80=py27hfc679d8_1
  - pyrsistent=0.14.5=py27h470a237_1
  - python=2.7.15=h33da82c_4
  - python-hdfs=2.1.0=py_1
  - pyyaml=3.13=py27h470a237_1
  - pyzmq=17.1.2=py27hae99301_1
  - requests=2.20.1=py27_1000
  - retrying=1.3.3=py_2
  - scikit-learn=0.19.1=py27_blas_openblas_201
  - scipy=1.1.0=py27_blas_openblash7943236_201
  - simplejson=3.16.1=py27h470a237_0
  - sqlalchemy=1.2.14=py27h470a237_0
  - thrift=0.11.0=py27hfc679d8_1
  - xgboost=0.80=py27hfc679d8_1
  - zeromq=4.2.5=hfc679d8_6
  - _py-xgboost-mutex=2.0=cpu_0
  - asn1crypto=0.24.0=py27_0
  - backports=1.0=py27_1
  - backports.functools_lru_cache=1.5=py27_1
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0rc1=py27_0
  - backports_abc=0.5=py27_0
  - bleach=1.5.0=py27_0
  - boto=2.49.0=py27_0
  - boto3=1.9.35=py27_0
  - botocore=1.12.35=py27_0
  - bz2file=0.98=py27_1
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - cryptography=2.3.1=py27hc365091_0
  - cycler=0.10.0=py27hc7354d3_0
  - dbus=1.13.2=h714fa37_1
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - enum34=1.1.6=py27_1
  - expat=2.2.6=he6710b0_0
  - fontconfig=2.13.0=h9420a91_0
  - freetype=2.9.1=h8a8886c_1
  - funcsigs=1.0.2=py27h83f16ab_0
  - functools32=3.2.3.2=py27_1
  - futures=3.2.0=py27_0
  - gensim=1.0.1=np113py27_0
  - glib=2.56.2=hd408876_0
  - gst-plugins-base=1.14.0=hbbd80ab_1
  - gstreamer=1.14.0=hb453b48_1
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=0.9999999=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.7=py27_0
  - intel-openmp=2019.0=118
  - ipaddress=1.0.22=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - jmespath=0.9.3=py27_0
  - jpeg=9b=h024ee3a_2
  - keras=2.1.3=py27_0
  - kiwisolver=1.0.1=py27hf484d3e_0
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.35=hbc83047_0
  - libprotobuf=3.6.1=hd408876_0
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - libuuid=1.0.3=h1bed415_2
  - libxcb=1.13=h1bed415_1
  - libxml2=2.9.8=h26e45fe_1
  - linecache2=1.0.0=py27_0
  - markdown=3.0.1=py27_0
  - matplotlib=2.2.2=py27hb69df0a_2
  - mkl=2019.0=118
  - ncurses=6.1=hf484d3e_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.1=py27_0
  - pbr=5.1.0=py27_0
  - pcre=8.42=h439df22_0
  - pexpect=4.6.0=py27_0
  - pickleshare=0.7.5=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.6.1=py27he6710b0_0
  - ptyprocess=0.6.0=py27_0
  - pycparser=2.19=py27_0
  - pygments=2.2.0=py27h4a8b6f5_0
  - pykerberos=1.1.14=py27h2de3caf_2
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.3.0=py27_0
  - pyqt=5.9.2=py27h22d08a2_1
  - pysocks=1.6.8=py27_0
  - python-dateutil=2.7.5=py27_0
  - pytz=2018.7=py27_0
  - qt=5.9.6=h8703b6f_2
  - readline=7.0=h7b6447c_5
  - requests-kerberos=0.12.0=py27_0
  - s3transfer=0.1.13=py27_0
  - scandir=1.9.0=py27h14c3975_0
  - seaborn=0.8.1=py27_0
  - setuptools=40.5.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - sip=4.19.13=py27he6710b0_0
  - six=1.11.0=py27_1
  - smart_open=1.7.1=py27_0
  - sqlite=3.25.2=h7b6447c_0
  - statsmodels=0.9.0=py27h035aef0_0
  - subprocess32=3.5.3=py27h7b6447c_0
  - tensorflow=1.3.0=0
  - tensorflow-base=1.3.0=py27h0dbb4d0_1
  - tensorflow-tensorboard=1.5.1=py27hf484d3e_1
  - termcolor=1.1.0=py27_1
  - tk=8.6.8=hbc83047_0
  - tornado=5.1.1=py27h7b6447c_0
  - tqdm=4.26.0=py27h28b3542_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.23=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - werkzeug=0.14.1=py27_0
  - wheel=0.32.2=py27_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zlib=1.2.11=ha838bed_2
  - pip:
    - hdfs==2.1.0
    - state-machine==0.2.10

add the new unpack env to Conda env list

When doing conda env list, we cannot see the new unpack environnement,
can you add a way to add it ?

Windows: site-packages marked as "editable package"

I created a new environment to test this out on my Windows machine, and for some reason it's saying that site-packages is an editable package which is preventing me from using conda-pack:

(my_env) C:\Users\me>conda pack --zip-symlinks --format zip
Collecting packages...
CondaPackError: Cannot pack an environment with editable packages
installed (e.g. from `python setup.py develop` or
 `pip install -e`). Editable packages found:

- c:\users\me\appdata\local\conda\conda\envs\my_env\lib\site-packages

Overwrite existing archive

Looking at the help string I couldn't immediately tell if there was a way to overwrite the existing archive if one exists. When constructing a pack I find myself removing the old version, then constructing a new one in two steps

Packing fails despite excluding files

If you trim down your environment by e.g. removing all static library archives, conda-pack is unable to package this environment. This is also the case when repacking an environment that previously went through conda-pack but where exclusions where used in packing the environment.

Reproducible example:

conda create -p $(pwd)/test-env conda-pack python=3.7
conda activate $(pwd)/test-env
conda-pack --exclude '*.a' -o env.tar.gz
mkdir other-env
cd other-env
source bin/activate
conda-pack --exclude '*.a' -o env.tar.gz

which leads to the following error:

Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- bzip2='1.0.8'
- readline='8.0'
- python='3.7.3'
- libcxxabi='8.0.0'
- libcxx='8.0.0'
- xz='5.2.4'
- zlib='1.2.11'
- tk='8.6.9'
- libffi='3.2.1'
- ncurses='6.1'
- sqlite='3.29.0'

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

A possible solution would be to do the integrity checks on the call to pack and not during loading the environment. @jcrist would that be an acceptable solution for you?

Windows support

This currently only supports linux/osx/*nix-likes. Some windows specific behaviors have been added, but the tests currently fail on windows (partly due to / vs \ handling).

What is the motivation behind the tarball structure

Currently you have to unpack the tarball using:

$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env

while the more common structure is to pack things into a subdirectory, as:

$ tar -xzf my_env.tar.gz
$ cd my_env

What are the motivation for the current format?

For my use case, when I want to distribute the tarball to users, the second option is better. I can of course repack it, but it would be better if conda-pack itself did it. Perhaps there can be an option how the tarball should be structured?

Difference to `constructor`

What's the difference between conda-pack and constructor? When would you use one and when the other? What are the pros and cons of each?

Verbose setting

Add a verbose setting to log all archived files and their paths.

windows PermissionError matplotlib pdf

I am trying to pack an environment contaning matplotlib. For some reason it throws a permission error pointing to a pdf although I have full read/write/exec access to the whole folder.

PermissionError: [Errno 13] Permission denied: 'C:\\Users\\hy.amanieu\\AppData\\Local\\Continuum\\anaconda3\\pkgs\\matplotlib-3.1.0-py37hc8f65d3_0\\Lib/site-packages/matplotlib/mpl-data/images/back.pdf'

I could however pack the environment when going through python and import conda_pack.
OS is windows 10.

Allow conda-unpack to be implicit

It would be very useful if a conda environment newly deployed from a conda-pack archive could "know" whether conda-unpack has been run, and if not, run it automatically on the first invocation of python from the environment.

This makes the system generally easier to use, and makes some specific use cases much easier. My particular use case is PySpark jobs that want to select from a library of packed Conda environments (rather than expect the needed environment to be pre-deployed to every node in the cluster). PySpark already knows how to declare an archive file to be automatically expanded in each job environment, and allows declaring the python executable to be on a path that may be within such an expanded archive. But the need to invoke conda-unpack after expanding the archive is not within its purview. Calling conda-unpack explicitly in user-written code is okay for the single "driver" node, but very clumsy for the many "worker" nodes, which are not expected to run additional setup code. The only obvious solution is to hack the approach that I am proposing here, e.g. by setting the python executable path to a script that does a first-time call to conda-unpack and then calls python.

Requiring a single user to run conda-unpack explicitly is perfectly reasonable, but when it becomes an infrastructure issue, I would argue it is much better to do mandatory cleanup actions (like conda-unpack) automatically.

However, if the project team feels the user must explicitly make the decision to run conda-unpack, please allow an option to conda-pack that will enable this behavior of automatic unpack at the first execution. Thanks.

CondaPackError : Files managed by conda were found to have been deleted/overwritten

Hi, I have seen some of conda-pack updates regarding this issues and I have upgraded my conda-pack to the latest version but the problem is still there.

A bit background story:
Basically I have two environments which actually if I tried to clone (or move using .yml and spec-list.txt), it throws error because the package probably are obsolete/deprecated/not supported anymore because it was created on Jan 2018. So I try to use conda-pack to see if it can wrap the envs.

_CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

pandas='0.22.0'
pip='9.0.3'
util-linux='2.21'

This is usually due to pip uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using conda list, and fix the environment by ensuring
only one version of each package is installed (conda preferred)._

When running conda list on those packages:
_# Name Version Build Channel
geopandas 0.4.0
pandas 0.22.0 py36_0 conda-for_ge

# Name Version Build Channel
pip 9.0.3 py36_0 conda-forge

# Name Version Build Channel
util-linux 2.21 0

Anyone could help on these?

PS: updating the package is not an option because it might impact the apps/code that run inside those envs.

Apply techniques for making smaller conda environments

How possible would it be to migrate some of the techniques described in https://jcrist.github.io/conda-docker-tips.html to reduce the size of the tarball. I suppose it would be best to make a command line option for this if it doesn't already exist.

`conda clean -afy
Remove select file types: .js.map, .pyc, .a, and bokeh's unminified .js files.

conda-unpack failed

Steps to reproduce on the new machine:

$ mkdir antlr37
$ tar xzf antlr37.tar.gz -C antlr37
$ source antlr37/bin/activate
(antlr37) $ conda-unpack
Traceback (most recent call last):
  File "/users/certik/scratch1/pack/antlr37/bin/conda-unpack", line 408, in <module>
    placeholder, mode=mode)
  File "/users/certik/scratch1/pack/antlr37/bin/conda-unpack", line 66, in update_prefix
    with open(path, 'rb+') as fh:
OSError: [Errno 26] Text file busy: '/users/certik/scratch1/pack/antlr37/bin/python3.7'

I used the latest conda-pack available on conda forge.

I didn't install Python nor conda on the target machine.

jupyter kernel.json file change

In the env packed by conda-pack,
I found a the kernel.json file changed to

{
 "argv": [
  "/opt/anaconda1anaconda2anaconda3/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

Is /opt/anaconda1anaconda2anaconda3/bin/python hard coded?
Is there a way for user to set it manually?

Handle `.pth` files

Things like python setup.py develop install with a special link file and make an entry in easy-install.pth. Need to special case the handling of these files to either:

pull in the required files as necessary
error if easy-install.pth isn't empty

I'm not sure which behavior is intuitive here.

Conda pack can't pack an environment that doesn't have python

For example an r environment created as say

conda create -n renv r-base

Ensure roundtrip test would have caught unwritable binaries error in #47

The testing infrastructure failed to catch the bug reported in #47.

When looking at the tests, it seems they do not actually test relocating the environment, they only test individual parts of the package as specified in the tests here: https://github.com/conda/conda-pack/tree/60a814326c1200998362e856040d1a9d6b98d7dc/conda_pack/tests

As such, I think the tests should be updated to test that actually relocating the environment works, on Linux, Mac and Windows.

Add tests

Support other shells

Currently only posix and cmd shells are supported by conda-pack but the full list of conda supported shells is:

Compare files from `conda-meta` with files in environment directory to catch manual uninstalls

The following environment when packed doesn't resolve to the right version of packages in the final unpacked environment. Spacy depends on a different version of thinc specified in the environment file creating a conflict and hence finally being resolved by pip (thinc : 6.10.3). But the unpacked environment still has thinc : 6.5.2. I have packed the conda env with unmanaged=True. Can someone help explain this behaviour?

channels:
 - conda-forge
 - defaults
dependencies:
 - thinc=6.5.2=py27_0
 - pip:
   - spacy==2.0.9

Please add an option to not modify PS1

Regular conda has this, it'd be very nice if the activate/deactivate scripts could either be modified on creation in the redistributable archive or otherwise respect an environment variable or something along those lines.

Thank you.

Install conda in a temporary directory if missing

CondaCreator component of knit had a nice feature: it installed Miniconda if it was not available in $PATH. One could argue that installing conda is easy and that the user would probably want to do it anyway, but I think it is a reasonable and user-friendly default behaviour despite that.

Would you like to see this implemented in conda-pack?

[Feature Request] Pack only what is NEEDED

User story:

I am a C/C++ developer and I use a conda environment to get my dependencies (mostly libraries). My code generates several binaries and shared libraries. Now, when I want to run my binary in production, it's not efficient to create the entire environment again, because:

My binary could just be depending on a certain number of shared libraries, not all. For example, I might be using the unzip conda package to unzip something, but I don't need the unzip binary for my final binary. Another example is I might use the gxx_linux-64 package to compile my binary, but I don't need the compiler during the execution of the binary. I just need libstdcxx-ng.
My binary could be linking to some libraries statically, so they are not needed at runtime. Example: googletest/gmock/googlebenchmark for tests

Sure, a recommended way for the user would be to create a conda package every time and then conda install it. But creating a conda package on the dev machine and then deploying it in production would require additional time to scp the binary to a channel, and then index it.

Another way would be to maintain an environment.yaml file and use it to deploy the binary. But that suffers from the same problem of sifting through your requirements to see what is actually NEEDED.

A simple way to go about this would be to recursively look at the NEEDED ELF section in the binary/shared libraries that I created and pack only those packages which provided the shared libraries I depend on (from the conda env prefix). Conda build has a function for the reverse lookup: https://github.com/conda/conda-build/blob/3.10.9/conda_build/inspect_pkg.py#L35-L43

The cli might look something like:

conda-pack /path/to/build/folder/containing/my/project/binaries/or/libraries --needed-only

This would pack all the requirements really needed, and during conda-unpack, in addition to what it already does right now, it would also invoke patchelf on my binaries/shared libraries to fix the RPATHs to point to the new location.

libblas.so FileNotFoundError

I'm trying to conda-pack an R environment:

conda create -y -n r-brd-env --copy r-sparklyr r-essentials r-changepoint r-zoo r-dplyr r-aws.s3 r-mixtools r-timedate r-data.table r-h2o r-tidyr r-rcpproll r-lubridate r-foreach r-domc conda-pack -c conda-forge

conda activate r-brd-env

conda-pack -o r_env.zip -d ./r_env.zip/

Collecting packages...
Packing environment at '/home/centos/miniconda/envs/r-brd-env' to 'r_env.zip'
[############                            ] | 32% Completed |  1min  4.3s
Traceback (most recent call last):
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/cli.py", line 135, in main
    pack(name=args.name,
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/core.py", line 462, in pack
    return env.pack(output=output, format=format, arcroot=arcroot,
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/core.py", line 333, in pack
    packer.add(f)
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/core.py", line 929, in add
    self.archive.add(file.source, file.target)
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/formats.py", line 207, in add
    self._add(source, target)
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/site-packages/conda_pack/formats.py", line 291, in _add
    self.archive.write(source, target)
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/zipfile.py", line 1722, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "/home/centos/miniconda/envs/r-brd-env/lib/python3.8/zipfile.py", line 509, in from_file
    st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/centos/miniconda/pkgs/libblas-3.8.0-14_openblas/lib/libblas.so'

# packages in environment at /home/centos/miniconda/envs/r-brd-env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_r-mutex                  1.0.1               anacondar_1    conda-forge
attrs                     19.3.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
binutils_impl_linux-64    2.31.1               h7fc9f1b_2    conda-forge
binutils_linux-64         2.31.1              h6176602_12    conda-forge
bleach                    3.1.0                      py_0    conda-forge
bwidget                   1.9.13                        0    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
cairo                     1.16.0            hfb77d84_1002    conda-forge
certifi                   2019.9.11                py38_0    conda-forge
conda-pack                0.4.0                      py_0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
entrypoints               0.3                   py38_1000    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
fribidi                   1.0.5             h516909a_1002    conda-forge
gcc_impl_linux-64         7.3.0                habb00fd_2    conda-forge
gcc_linux-64              7.3.0               h553295d_12    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
gfortran_impl_linux-64    7.3.0                hdf63c60_2    conda-forge
gfortran_linux-64         7.3.0               h553295d_12    conda-forge
glib                      2.58.3          py38h6f030ca_1002    conda-forge
graphite2                 1.3.13            hf484d3e_1000    conda-forge
gsl                       2.5                  h294904e_1    conda-forge
gxx_impl_linux-64         7.3.0                hdf63c60_2    conda-forge
gxx_linux-64              7.3.0               h553295d_12    conda-forge
harfbuzz                  2.4.0                h9f30f68_3    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
importlib_metadata        0.23                     py38_0    conda-forge
ipykernel                 5.1.3            py38h5ca1d4c_0    conda-forge
ipython                   7.9.0            py38h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.15.1                   py38_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
jsonschema                3.1.1                    py38_0    conda-forge
jupyter_client            5.3.3                    py38_1    conda-forge
jupyter_core              4.5.0                      py_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0    defaults
libgfortran-ng            7.3.0                hdf63c60_2    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_3    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libsodium                 1.0.17               h516909a_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0    defaults
libtiff                   4.1.0                hfc65ed5_0    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
make                      4.2.1             h14c3975_2004    conda-forge
markupsafe                1.1.1            py38h516909a_0    conda-forge
mistune                   0.8.4           py38h516909a_1000    conda-forge
more-itertools            7.2.0                      py_0    conda-forge
nbconvert                 5.6.1                    py38_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
notebook                  6.0.1                    py38_0    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
pandoc                    2.7.3                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
pango                     1.42.4               ha030887_1    conda-forge
parso                     0.5.1                      py_0    conda-forge
pcre                      8.43                 he1b5a44_0    conda-forge
pexpect                   4.7.0                    py38_0    conda-forge
pickleshare               0.7.5                 py38_1000    conda-forge
pip                       19.3.1                   py38_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.10                     py_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pyrsistent                0.15.5           py38h516909a_0    conda-forge
python                    3.8.0                h357f687_3    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pyzmq                     18.1.1           py38h1768529_0    conda-forge
r-askpass                 1.1               r36hcdcec82_1    conda-forge
r-assertthat              0.2.1             r36h6115d3f_1    conda-forge
r-aws.s3                  0.3.20          r36h6115d3f_1002    conda-forge
r-aws.signature           0.5.2             r36h6115d3f_0    conda-forge
r-backports               1.1.5             r36hcdcec82_0    conda-forge
r-base                    3.6.1                hba50c9b_5    conda-forge
r-base64enc               0.1_3           r36hcdcec82_1003    conda-forge
r-bitops                  1.0_6           r36hcdcec82_1003    conda-forge
r-boot                    1.3_23            r36h6115d3f_2    conda-forge
r-broom                   0.5.2             r36h6115d3f_1    conda-forge
r-callr                   3.3.2             r36h6115d3f_0    conda-forge
r-caret                   6.0_84            r36hcdcec82_1    conda-forge
r-cellranger              1.1.0           r36h6115d3f_1002    conda-forge
r-changepoint             2.2.2           r36h516909a_1003    conda-forge
r-class                   7.3_15          r36hcdcec82_1001    conda-forge
r-cli                     1.1.0             r36h6115d3f_2    conda-forge
r-clipr                   0.7.0             r36h6115d3f_0    conda-forge
r-cluster                 2.1.0             r36h9bbef5b_2    conda-forge
r-codetools               0.2_16          r36h6115d3f_1001    conda-forge
r-colorspace              1.4_1             r36hcdcec82_1    conda-forge
r-config                  0.3             r36h6115d3f_1002    conda-forge
r-crayon                  1.3.4           r36h6115d3f_1002    conda-forge
r-curl                    4.2               r36hcdcec82_0    conda-forge
r-data.table              1.12.6            r36hcdcec82_0    conda-forge
r-dbi                     1.0.0           r36h6115d3f_1002    conda-forge
r-dbplyr                  1.4.2             r36h6115d3f_1    conda-forge
r-digest                  0.6.22            r36h0357c0b_0    conda-forge
r-domc                    1.3.6                     r36_0    conda-forge
r-dplyr                   0.8.3             r36h0357c0b_3    conda-forge
r-ellipsis                0.3.0             r36hcdcec82_0    conda-forge
r-essentials              3.6                    r36_2001    conda-forge
r-evaluate                0.14              r36h6115d3f_1    conda-forge
r-fansi                   0.4.0           r36hcdcec82_1001    conda-forge
r-fastmap                 1.0.1             r36h0357c0b_0    conda-forge
r-forcats                 0.4.0             r36h6115d3f_1    conda-forge
r-foreach                 1.4.7             r36h6115d3f_0    conda-forge
r-foreign                 0.8_72            r36hcdcec82_0    conda-forge
r-forge                   0.2.0             r36h6115d3f_1    conda-forge
r-formatr                 1.7               r36h6115d3f_1    conda-forge
r-fs                      1.3.1             r36h0357c0b_1    conda-forge
r-generics                0.0.2           r36h6115d3f_1002    conda-forge
r-ggplot2                 3.2.1             r36h6115d3f_0    conda-forge
r-gistr                   0.4.2           r36h6115d3f_1002    conda-forge
r-glmnet                  2.0_18            r36h9bbef5b_2    conda-forge
r-glue                    1.3.1             r36hcdcec82_1    conda-forge
r-gower                   0.2.1             r36hcdcec82_1    conda-forge
r-gtable                  0.3.0             r36h6115d3f_2    conda-forge
r-h2o                     3.26.0.2          r36h6115d3f_0    defaults
r-haven                   2.2.0             r36hde08347_0    conda-forge
r-hexbin                  1.28.0            r36h9bbef5b_0    conda-forge
r-highr                   0.8               r36h6115d3f_1    conda-forge
r-hms                     0.5.2             r36h6115d3f_0    conda-forge
r-htmltools               0.4.0             r36h0357c0b_0    conda-forge
r-htmlwidgets             1.5.1             r36h6115d3f_0    conda-forge
r-httpuv                  1.5.2             r36h0357c0b_1    conda-forge
r-httr                    1.4.1             r36h6115d3f_1    conda-forge
r-ipred                   0.9_9             r36hcdcec82_1    conda-forge
r-irdisplay               0.7                    r36_1001    conda-forge
r-irkernel                1.0.2             r36h6115d3f_2    conda-forge
r-iterators               1.0.12            r36h6115d3f_0    conda-forge
r-jsonlite                1.6             r36hcdcec82_1001    conda-forge
r-kernsmooth              2.23_15         r36h9bbef5b_1004    conda-forge
r-knitr                   1.26              r36h6115d3f_0    conda-forge
r-labeling                0.3             r36h6115d3f_1002    conda-forge
r-later                   1.0.0             r36h0357c0b_0    conda-forge
r-lattice                 0.20_38         r36hcdcec82_1002    conda-forge
r-lava                    1.6.6             r36h6115d3f_0    conda-forge
r-lazyeval                0.2.2             r36hcdcec82_1    conda-forge
r-lifecycle               0.1.0             r36h6115d3f_0    conda-forge
r-lubridate               1.7.4           r36h0357c0b_1002    conda-forge
r-magrittr                1.5             r36h6115d3f_1002    conda-forge
r-maps                    3.3.0           r36hcdcec82_1003    conda-forge
r-markdown                1.1               r36hcdcec82_0    conda-forge
r-mass                    7.3_51.4          r36hcdcec82_1    conda-forge
r-matrix                  1.2_17            r36hcdcec82_1    conda-forge
r-mgcv                    1.8_29            r36hcdcec82_0    conda-forge
r-mime                    0.7               r36hcdcec82_1    conda-forge
r-mixtools                1.1.0           r36hcdcec82_1003    conda-forge
r-modelmetrics            1.2.2             r36h0357c0b_1    conda-forge
r-modelr                  0.1.5             r36h6115d3f_0    conda-forge
r-munsell                 0.5.0           r36h6115d3f_1002    conda-forge
r-nlme                    3.1_141           r36h9bbef5b_1    conda-forge
r-nnet                    7.3_12          r36hcdcec82_1003    conda-forge
r-numderiv                2016.8_1.1        r36h6115d3f_1    conda-forge
r-openssl                 1.4.1             r36h9c8475f_0    conda-forge
r-pbdzmq                  0.3_3           r36h559a7a4_1002    conda-forge
r-pillar                  1.4.2             r36h6115d3f_2    conda-forge
r-pkgconfig               2.0.3             r36h6115d3f_0    conda-forge
r-plogr                   0.2.0           r36h6115d3f_1002    conda-forge
r-plyr                    1.8.4           r36h0357c0b_1003    conda-forge
r-prettyunits             1.0.2           r36h6115d3f_1002    conda-forge
r-processx                3.4.1             r36hcdcec82_0    conda-forge
r-prodlim                 2019.10.13        r36h0357c0b_0    conda-forge
r-progress                1.2.2             r36h6115d3f_1    conda-forge
r-promises                1.1.0             r36h0357c0b_0    conda-forge
r-pryr                    0.1.4           r36h0357c0b_1003    conda-forge
r-ps                      1.3.0           r36hcdcec82_1001    conda-forge
r-purrr                   0.3.3             r36hcdcec82_0    conda-forge
r-quantmod                0.4_15            r36h6115d3f_1    conda-forge
r-r2d3                    0.2.3           r36h6115d3f_1001    conda-forge
r-r6                      2.4.1             r36h6115d3f_0    conda-forge
r-randomforest            4.6_14          r36h9bbef5b_1002    conda-forge
r-rappdirs                0.3.1           r36hcdcec82_1003    conda-forge
r-rbokeh                  0.5.0           r36h6115d3f_1002    conda-forge
r-rcolorbrewer            1.1_2           r36h6115d3f_1002    conda-forge
r-rcpp                    1.0.3             r36h0357c0b_0    conda-forge
r-rcpproll                0.3.0           r36h0357c0b_1001    conda-forge
r-rcurl                   1.95_4.12         r36hcdcec82_1    conda-forge
r-readr                   1.3.1           r36h0357c0b_1002    conda-forge
r-readxl                  1.3.1             r36h0357c0b_2    conda-forge
r-recipes                 0.1.7             r36h6115d3f_0    conda-forge
r-recommended             3.6                    r36_1003    conda-forge
r-rematch                 1.0.1           r36h6115d3f_1002    conda-forge
r-repr                    1.0.1             r36h6115d3f_1    conda-forge
r-reprex                  0.3.0             r36h6115d3f_1    conda-forge
r-reshape2                1.4.3           r36h0357c0b_1004    conda-forge
r-rlang                   0.4.1             r36hcdcec82_0    conda-forge
r-rmarkdown               1.17              r36h6115d3f_0    conda-forge
r-rpart                   4.1_15            r36hcdcec82_1    conda-forge
r-rprojroot               1.3_2           r36h6115d3f_1002    conda-forge
r-rstudioapi              0.10              r36h6115d3f_2    conda-forge
r-rvest                   0.3.5             r36h6115d3f_0    conda-forge
r-scales                  1.0.0           r36h0357c0b_1002    conda-forge
r-segmented               1.0_0             r36h6115d3f_1    conda-forge
r-selectr                 0.4_1           r36h6115d3f_1001    conda-forge
r-shiny                   1.4.0             r36h6115d3f_0    conda-forge
r-sourcetools             0.1.7           r36he1b5a44_1001    conda-forge
r-sparklyr                1.0.4             r36h6115d3f_0    conda-forge
r-spatial                 7.3_11          r36hcdcec82_1003    conda-forge
r-squarem                 2017.10_1       r36h6115d3f_1002    conda-forge
r-stringi                 1.4.3             r36h0e574ca_3    conda-forge
r-stringr                 1.4.0             r36h6115d3f_1    conda-forge
r-survival                2.44_1.1          r36hcdcec82_1    conda-forge
r-sys                     3.3               r36hcdcec82_0    conda-forge
r-tibble                  2.1.3             r36hcdcec82_1    conda-forge
r-tidyr                   1.0.0             r36h0357c0b_0    conda-forge
r-tidyselect              0.2.5           r36h0357c0b_1001    conda-forge
r-tidyverse               1.2.1           r36h6115d3f_1002    conda-forge
r-timedate                3043.102        r36h6115d3f_1001    conda-forge
r-tinytex                 0.17              r36h6115d3f_0    conda-forge
r-ttr                     0.23_5            r36hcdcec82_0    conda-forge
r-utf8                    1.1.4           r36hcdcec82_1001    conda-forge
r-uuid                    0.1_2           r36hcdcec82_1002    conda-forge
r-vctrs                   0.2.0             r36hcdcec82_1    conda-forge
r-viridislite             0.3.0           r36h6115d3f_1002    conda-forge
r-whisker                 0.4               r36h6115d3f_0    conda-forge
r-withr                   2.1.2           r36h6115d3f_1001    conda-forge
r-xfun                    0.11              r36h6115d3f_0    conda-forge
r-xml2                    1.2.2             r36h0357c0b_0    conda-forge
r-xtable                  1.8_4             r36h6115d3f_2    conda-forge
r-xts                     0.11_2            r36hcdcec82_1    conda-forge
r-yaml                    2.2.0           r36hcdcec82_1002    conda-forge
r-zeallot                 0.1.0           r36h6115d3f_1001    conda-forge
r-zoo                     1.8_6             r36hcdcec82_1    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
sed                       4.7               h1bed415_1000    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.6.0                   py38_1    conda-forge
six                       1.13.0                   py38_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
terminado                 0.8.3                    py38_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
tk                        8.6.9             hed695b0_1003    conda-forge
tktable                   2.10                 h555a92e_3    conda-forge
tornado                   6.0.3            py38h516909a_0    conda-forge
traitlets                 4.3.3                    py38_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.6                   py38_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zeromq                    4.3.2                he1b5a44_2    conda-forge
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.3                h3b9ef0a_0    conda-forge

Not sure how to debug or resolve this. Any suggestions?

conda unpack script not created in Windows

If I install a new (2019-7) version of Anaconda in Windows, then create a new environment (based off jupyterlab), then try and pack this environment, a tar.gz file is created, but it contains no conda-unpack script.

Just to expand, to assist with debugging, the following will produce a tar.gz file that DOES contain an unpack script on Windows:

conda create -n testenv python -y
conda pack -n testenv

but this will produce a tar.gz file that DOES NOT contain an unpack script on Windows:

conda create -n testlab jupyterlab -y
conda pack -n testlab

Here is the output from conda info:

    active environment : jupyterlab
    active env location : C:\Users\ben\Anaconda3\envs\jupyterlab
            shell level : 1
       user config file : C:\Users\ben\.condarc
 populated config files : 
          conda version : 4.7.11
    conda-build version : 3.18.8
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : C:\Users\ben\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\ben\Anaconda3\pkgs
                          C:\Users\ben\.conda\pkgs
                          C:\Users\ben\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\ben\Anaconda3\envs
                          C:\Users\ben\.conda\envs
                          C:\Users\ben\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False

Update: After some more investigation, I believe the problem is that the tar file reaches 2Gb, and then I think it stops storing files.

Pack claims conda files were overridden with pip, but pip was not used

I am trying to create a conda-pack for this environment:

name: tomopy-short-course
channels:
  - conda-forge
  - defaults
dependencies:
  - conda
  - dxchange
  - jupyter
  - python 3*
  - tomopy 1.5*

I used the following command to create a conda-pack:

conda pack -n tomopy-short-course
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- attrs='19.1.0'
- backcall='0.1.0'
- bleach='3.1.0'
- cloudpickle='1.2.1'
- colorama='0.4.1'
- cycler='0.10.0'
- dask-core='2.0.0'
- decorator='4.4.0'
- defusedxml='0.5.0'
- dxfile='0.5'
- edffile='5.0.0'
- ipython_genutils='0.2.0'
- ipywidgets='7.5.0'
- jinja2='2.10.1'
- jupyter='1.0.0'
- jupyter_client='5.2.4'
- jupyter_console='6.0.0'
- jupyter_core='4.4.0'
- nbconvert='5.5.0'
- nbformat='4.4.0'
- networkx='2.3'
- olefile='0.46'
- pandocfilters='1.4.2'
- parso='0.5.0'
- prometheus_client='0.7.1'
- prompt_toolkit='2.0.9'
- pygments='2.4.2'
- pyparsing='2.4.0'
- python-dateutil='2.8.0'
- qtconsole='4.5.1'
- send2trash='1.5.0'
- spefile='1.6'
- testpath='0.4.2'
- tomopy='1.5.1'
- toolz='0.9.0'
- tqdm='4.32.2'
- wcwidth='0.1.7'
- webencodings='0.5.1'

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

I get this error message claiming that these packages were conflicted. None of these packages are duplicated or installed with pip, so I have no idea why I'm getting this error.

Here is information about my conda installation.

active environment : base
    active env location : C:\Users\dchin\Miniconda3
            shell level : 1
       user config file : C:\Users\dchin\.condarc
 populated config files : C:\Users\dchin\.condarc
          conda version : 4.7.5
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages :
       base environment : C:\Users\dchin\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : C:\Users\dchin\Miniconda3\pkgs
                          C:\Users\dchin\.conda\pkgs
                          C:\Users\dchin\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\dchin\Miniconda3\envs
                          C:\Users\dchin\.conda\envs
                          C:\Users\dchin\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.5 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False

Tests fail on OSX due to "_conda_pack_activate:.:38: no such file or directory: deactivate`

The tests fail on OSX for me. I'm running them in zsh with py.test conda_pack/tests. The failures are all because conda hasn't been loaded in my test setting and thus deactivate from main conda is not available.

Example error is:

>           assert out == 'Done\n'
E           AssertionError: assert '_conda_pack_...ivate\nDone\n' == 'Done\n'
E             - _conda_pack_activate:.:38: no such file or directory: deactivate
E               Done

conda_pack/tests/test_core.py:281: AssertionError

Outside of tests conda-pack, is usable though.

Handle prefix replacement

Currently we don't handle prefixes (shebangs and other absolute paths). For the most part this is fine, but will break for some libraries or executables.

Conda maintains an index of prefixes to replace on install for each package at CONDAROOT/pkgs/some_package/info/has_prefix. If the package has a has_prefix file, this contains information on the files that need prefixes replaced, the prefix placeholder, and the replacement type (binary or text). Note that after install the placeholder is already replaced, so we'd have to have some logic to figure out what the new search string is.

Code to do the replacement in conda: https://github.com/conda/conda/blob/master/conda/core/portability.py

Ideally packing would then:

Find all files to pack and archive as is (same as currently done)
Find all files that need prefixes replaced
Generate a script that does the prefix replacement after unpacking. This would probably be a python script that's constant on all archives, and a file that includes the information needed to do the prefix replacement.

After unpacking, the user would then run $ENVDIR/bin/python $ENVDIR/fix_prefixes.py (or maybe make that file executable so it's just $ENVDIR/fix_prefixes.py).

Running '<env_path>/bin/python <envpath>/bin/conda-unpack' fails

As suggested on the conda-pack documentation , running conda-unpack without activating the environment (/tmp/XXXXXXX/bin/python /tmp/XXXXXXX/bin/conda-unpack) fails with the following error:

Traceback (most recent call last):
File "/tmp/XXXXXXX/bin/conda-unpack", line 409, in
placeholder, mode=mode)
File "/tmp/XXXXXXX/bin/conda-unpack", line 66, in update_prefix
with open(path, 'rb+') as fh:
OSError: [Errno 26] Text file busy: '/tmp/XXXXXXX/bin/python3.6'

PS: /tmp/XXXXXXX/bin/python is soft linked to /tmp/XXXXXXX/bin/python3.6

Qt setup not restored correctly

I've been using conda-pack on Windows 10 with great results. Such a useful tool! I did run into an issue when trying to unpack and use Spyder from a previous environment. The qt.conf file had the old paths in it even after I ran conda-unpack on the new machine. I kept getting errors trying to launch Spyder until I set QT_PLUGIN_PATH to the new location of the environment.

For example, the host machine's environment was packed up from C:\Users\durden\AppData\Local\Continuum\miniconda2\envs\myenv. Then, I unpacked everything on a new machine and was testing the install in a different path (because I don't want the path hard-coded to my username). So, I had to do the following to get Spyder working after unpacking:

set QT_PLUGIN_PATH = C:\Python\myenv\Library\plugins

Should conda-pack do this automatically when calling unpack? Not sure where the responsibility for this should be, but thought I'd open it up for discussion so others can see what the temporary fix is.

Add support for 'xz' compression

Another common tarball format uses xz compression (wikipedia. Right now xz is not recognized by conda-pack. Is this a trivial change that I could implement? Or is it unlikely that support can be added? Could someone point me to the right places in the code?

Support other archive formats

Currently only .zip is supported. While this works for my needs, "identical" zipfiles are not bytewise identical, as the creation time is stored in the file as metadata (even if the files are identical). To allow for hashing of packed environments it might be nice to support other formats like tar.gz or tar.bz2.

Properly handle root environment

The root environment is different than sub-environments in a couple key ways:

The path contains extra directories like the package cache, conda-bld directory, etc...
The environment contains conda itself, and may contain packages that depend on it (these are forbidden in sub-environments)

A few options for handling the root environment:

Special case the handling to directories/files that shouldn't be packed, and packages that depend on conda
Error nicely and tell the user to create a sub-environment

I'm not sure which would be more expected. The second is a lot easier to implement.

Windows error messages: how to help?

I realize conda-pack claims that it doesn't support Windows, however I noticed enough on_win entries in the code that I thought I'd give it a go. I have latest conda-pack installed from source, which I believe means that thanks to #12 and #48 should have a reasonable chance of working. However I get the errors below:

> conda pack -n test -o test-deps.zip
Collecting packages...
CondaPackError: Failed to determine path to environment: 'test'. This may be due to conda not being on your PATH. The full error is below:

b''

I can tell you conda is definitely on my path:

>where conda
C:\Users\USERNAME\AppData\Local\Continuum\anaconda3\Library\bin\conda.bat
C:\Users\USERNAME\AppData\Local\Continuum\anaconda3\Scripts\conda.exe

>conda -V
conda 4.5.11

One possibility: USERNAME has a space in it. That is, unfortunately, unavoidable.

Packing an environment including `conda` results in duplicate files in the archive

If an environment containing conda is packaged, the resulting archive contains two different activate.bat, I suppose one from conda, the other from conda-pack.

Is this unavoidable?
I noticed this behaviour only when decompressing the archive. It would be great to be warned during the packaging process.

What I am trying to achieve

I need to deploy a Python environment to a production machine with no Internet connection. Having conda in the environment would allow me to quickly update a package using a custom offline channel, e.g. on an USB stick.

Steps to reproduce

conda create --name test-env conda

conda pack --name test-env

tar -tzf test-env.tar.gz | sort | uniq --count --repeated
    2 Scripts/activate.bat

Output of `conda info`

     active environment : packaging
    active env location : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\envs\packaging
            shell level : 1
       user config file : C:\Users\angelo.peronio\.condarc
 populated config files : C:\Users\angelo.peronio\.condarc
          conda version : 4.7.11
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : __cuda=10.2
       base environment : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\pkgs
                          C:\Users\angelo.peronio\.conda\pkgs
                          C:\Users\angelo.peronio\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\angelo.peronio\AppData\Local\Continuum\miniconda3\envs
                          C:\Users\angelo.peronio\.conda\envs
                          C:\Users\angelo.peronio\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False

fail to pack env in Mac caused by paython.app and anaconda-anvigator

when I run command: conda pack -n ml_env -o out_ml_env.tar.gz
I got the following error message:
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

python.app=u'2'
anaconda-navigator=u'1.9.7'

Python.app are depend on many package, cannot delete it from conda.

conda-unpack relies on how it is invoked for prefix-patching

# Create a test env
$ conda create -yp test2 python openssl pip
$ pip install git+https://github.com/conda/conda-pack.git

# Note the prefix
$ grep ^prefix= $CONDA_PREFIX/lib/pkgconfig/libcrypto.pc
prefix=/tmp/wani.1531021259/test2

# Pack this env
$ conda-pack -p $PWD/test2
Collecting packages...
Packing environment at '/tmp/wani.1531021259/test2' to 'test2.tar.gz'
[########################################] | 100% Completed |  6.9s

# Unpack it to test3
$ mkdir test3
$ tar -xf test2.tar.gz -C test3
$ ./test3/bin/conda-unpack 
$ grep ^prefix= test3/lib/pkgconfig/libcrypto.pc
prefix=./test3 <---- Incorrect

# Unpack it to test4
$ mkdir test4
$ tar -xf test2.tar.gz -C test4
$ $PWD/test4/bin/conda-unpack <--- use abs path
$ grep ^prefix= test4/lib/pkgconfig/libcrypto.pc
prefix=/tmp/wani.1531021259/test4 <----- Correct

conda create with python2.7 base still creates python3.6 environment by default

If one downloads Miniconda2-latest-MacOSX-x86_64.sh and runs it, you get as expected a python2.7 base environment. Go ahead and activate it, then invoke conda create with a few packages, but without specifying a version of python. It builds a python3.6 environment.

I could not find documentation one way or the other, but I did expect it to give me another python2.7 environment, especially since I had activated the base environment before invoking conda create.

Question about conda-meta exclusion

I notice that I can create a non-base conda environment that includes conda>4.5, pack this environment, and later unpack it without issue.

conda create -n pack-base python=3 conda pip

# normal conda output removed for brevity ...

conda pack -n pack-base --arcroot pack-base
Collecting packages...
Packing environment at '/Users/parente/miniconda3/envs/pack-base' to 'pack-base.tar.gz'
[########################################] | 100% Completed |  5.0s

# on another host
tar xzf pack-base.tar.gz -C tmp
source /tmp/pack-base/bin/activate
conda unpack

I notice that conda-meta is not included by default in the packed environment, which means conda list (and maybe other conda commands?) does not function after unpacking.

conda list

EnvironmentLocationNotFound: Not a conda environment: /tmp/pack-base

However, if I manually add the conda-meta from the original environment to the tarball so that it's available later, conda seems to work properly after unpacking. So much so that I can list, install, uninstall, etc. packages in that unpacked environment.

Could you explain why conda-meta is not included in the tarball generated by conda pack? I'm guessing there's a good reason, and that if I add conda and conda-meta to my packed envs, I'll hit some surprise down the line that I've not discovered yet.

I ask because I have an interest in packing and distributing environments in which conda itself functions.

Windows: conda-pack fails for packages with pip installed entry points

I'm using conda-pack 0.3.1 on Windows 10 pro.

I created a conda environment which has a python package with entry points to scripts and conda-packed this environment.
When I untar it, run the conda-unpack.exe file and launch the script (to which the python package is pointing), it gives the following error:

Fatal error in launcher: Unable to create process using '"c:\envs\testing-package-cp\python.exe" "C:\Users\aman.garg\Downloads\testing-package\Scripts\bar.exe" '

Here, the path c:\envs\testing-package-cp\python.exe is pointing to python of the conda environment from which the tar was created

This issue occurs only on Windows and not on Linux or MacOS.