Compare files from `conda-meta` with files in environment directory to catch manual uninstalls about conda-pack HOT 6 CLOSED

Hmmm, this is a bug, but it can be avoided by not specifying thinc in dependencies (since the pip installation of spacy will overwrite the existing thinc install). Why are you using spacy from pip instead of from conda-forge?

Basically, what's happening here is:

• conda-env builds a new environment and first installs all the conda packages. This pulls in thinc=6.5.2.
• conda-env then installs pip packages, which results in pip uninstalling thinc=6.5.2 and replacing it with thinc=6.10.3 from pypi. But since pip did the uninstall, the metadata for thinc is still in the conda-meta directory (pip doesn't play 100% well with conda).
• conda-pack packages the environment by reading in the metadata from the conda-meta directory, which still includes the information for thinc=6.5.2. It then crawls the directory and finds thinc=6.10.3 from pypi and also includes that.

We should filter the files found from collecting conda-meta with those actually in the environment directory to catch issues of pip installs breaking stuff.

However, this can be avoided by having your pip install never uninstall packages (in this case, remove thinc from dependencies).

from conda-pack.

Thanks @jcrist for the explanation. Can I raise a PR with the fix for this? Agreed that this can be avoided by proactively pruning out conflicts from the environment.yaml itself, but with the environment already being created should the conda-pack fail fast for now in such case until the fix has been made to stop the user from packing the environment with incorrect set of packages?

from conda-pack.

Hmmm, this is a bit trickier than I originally thought. The main issue here is that manual changes to the environment may not change the total set of files, and instead just be small edits/replacement of individual files. In this case looking for missing conda files wouldn't catch these issues, as the sets would be identical. Instead we should check overwrites either by:

• Hashing the file and comparing to the hash in the conda metadata. This is further complicated because older versions of the metadata don't include file hashes.
• Looking at the hardlink status of the file to see if it still points to the package cache (only would work for files that are hardlinked, but would be faster than hashing).

This requires a larger refactor than I expected it would. You are welcome to work on it if you want, but it's not an easy problem. I'll probably have more time to work on this late next week.

from conda-pack.

Sure @jcrist . I realize this is a complicated problem. I'll wait for you to work upon it.

from conda-pack.

This should be fixed in #35.

from conda-pack.

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!

from conda-pack.