I have a particular python script that uses subprocess to create child processes that execute another script. What is the recommended way of passing down the info about the shiv's site directory down to the child?

As an example, suppose I made a shiv like this:

$ shiv requests -o foo.shiv

I then make a parent.py that creates a child process:

import os
import subprocess
import sys

print("Parent sys.executable:", sys.executable)
print("Parent sys.path:", sys.path)
print("Parent sys.argv", sys.argv)
print("Parent PYTHONPATH", os.environ.get("PYTHONPATH"))

child = subprocess.run([
    sys.executable,
    'child.py',
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)

print(child)

And then I make a child.py that makes use of requests, like so:

import requests

print(requests)

When I run it, I get output that looks like this:

$ ./foo.shiv parent.py
Parent sys.executable: /Users/kedo/.pyenv/versions/3.6.5/bin/python
Parent sys.path: ['./foo.shiv', '/Users/kedo/.pyenv/versions/3.6.5/lib/python36.zip', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/lib-dynload', '/Users/kedo/.local/lib/python3.6/site-packages', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages', '/Users/kedo/Workspace/shiv/src', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/importlib_resources-0.8-py3.6.egg', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/click-6.7-py3.6.egg', '/Users/kedo/.pyenv/versions/3.6.5/lib/python3.6/site-packages/wheel-0.31.1-py3.6.egg', '/Users/kedo/.shiv/foo_0550dd79-b895-48c0-91fa-11306e18f9cd/site-packages']
Parent sys.argv ['parent.py']
Parent PYTHONPATH None
CompletedProcess(args=['/Users/kedo/.pyenv/versions/3.6.5/bin/python', 'child.py'], returncode=1, stdout=b'', stderr=b'Traceback (most recent call last):\n  File "child.py", line 1, in <module>\n    import requests\nModuleNotFoundError: No module named \'requests\'\n')

Is there a non-hacky way for parent to know that it's running via a shiv, and that it needs to pass down info about the shiv to the child? Could we set some sort of environment variable in https://github.com/linkedin/shiv/blob/master/src/shiv/bootstrap/__init__.py#L86 that parent.py could check and use?

I have a microservice zipapp with a lot of dependencies, which fails to bootstrap in a memory limited docker container:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/microservice/app.pyz/__main__.py", line 3, in <module>
  File "/microservice/app.pyz/_bootstrap/__init__.py", line 106, in bootstrap
  File "/microservice/app.pyz/_bootstrap/__init__.py", line 80, in extract_site_packages
  File "/usr/local/lib/python3.6/compileall.py", line 86, in compile_dir
    success = min(results, default=True)
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I am wondering/asking if it would be possible to limit the number of workers used to compile the site packages in https://github.com/linkedin/shiv/blob/master/src/shiv/bootstrap/__init__.py#L80 by specifying it at zipapp creation time or even completely disable pre-compilation (I guess, at least in my case a lot of modules from some dependencies are not used anyway)?

compiling with max workers(*):

no pre-compiling:

In my case (zipapp in docker to save image size) compilation does not work well, it takes a lot of memory and extends the bootstrap phase unnecessary.

* - profiling done with https://pypi.org/project/memory_profiler/

Hello,
For the generated zipapp to be fully standalone, is it technically possible to ship the full Python interpreter along? This way you could deploy your app on environments with different or no Python interpreter.

Looks like you're getting bit by pypi/warehouse#3664

This should fix it prior to the next upload:

pip install -U setuptools wheel

running :

shiv -c hello.py -o hello.pyz

Fails with

"No entry point 'hello.py' found in the console_scripts!"

whereas :
shiv -e hello:main -o hello.pyz

succeeds. I guess the console script needs to reside in some package ? either from a dependency or in the app source tree. But for a naive new user - you'd kind of expect it to work.

It appears a module I am shiv'ing includes a file that loses its executable flag when unzipped by the shiv bootstrapper. Is this something that is expected behavior from your side?

From the discussion in the packaged module where this happens:

shiv simply calls extract method of ZipFile, thus it loses any previous exectutable permissions. It should instead do something similar to what the pip installer does when unpacking a wheel (which is also a zip file)

For more details.

Hello,

I played a bit with shiv and noticed that it always writes the full path to a python interpreter (sys.executable or user provided one) in the zipapp's shebang. Which means, if I try to run the same pyz on a different machine where python3.6 is installed in a different location, it won't run.

With pex, one can specify the interpreter to use either as absolute path or just the name of a Python interpreter within the environment.

It would be nice if shiv also allowed to do that.

Thank you

I have a wrapper for building executables that I would like to move from pex to shiv. However, the API to build executables is not exposed. It seems like starting around the with Temporary.... would be a good place to split up and call an API function.

Would that make sense as a PR?

I tried to pack a script that uses multiprocessing. Unfortunately I was not successful executing the package. The script is of the following sort:

import multiprocessing

def process_image(chunk):
    for r in chunk:
        print("%s" %r)

chunks=[[1,2,3,4],['one','two','three','four']]

multiprocessing.freeze_support()

pool = multiprocessing.Pool(processes=2)

pool.map(process_image, chunks)

If run in the Linux command line there is no issue - but as soon as I pack it with shiv it hangs at pool = multiprocessing.Pool(processes=2) . I am using a Python 3.6.3 interpreter. Any ideas?

It would be nice if the CLI had a --version option :)

For the given source code: https://github.com/buxx/rolling
And following instructions:

• pip install . --target dist/
• shiv --site-packages dist --compressed -p venv3.7/bin/python -o rolling-gui.pyz -e rolling.gui.run:main

When executing ./rolling-gui.pyz in debian docker container, result is:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff2cfd4a440 (most recent call first):
Aborted (core dumped)

Does something is missing ?

Are there any plans to support executing modules directly (like python -m http.server) or running different entry points from the same file (like PEX_SCRIPT=gunicorn my.pex)? (Or would there be any opposition for supporting this functionality?)

We have a monolithic codebase that has a single set of Python requirements, but different ways of running things. For our web apps, we run gunicorn, but for some other services, we directly call python on a file. It would be great to be able to build once, and then be able to support both runtime situations, like one can with PEX.

For now, I have a workaround that just creates a dead simple Python file that instantiates and runs a gunicorn WSGIApplication object directly, but that is not ideal.

Running shiv -o myproject .

Here is the traceback:

Traceback (most recent call last):
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/bin/shiv", line 11, in <module>
    sys.exit(main())
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/shiv/cli.py", line 180, in main
    shared_object_map=map_shared_objects(site_packages),
  File "/Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/shiv/cli.py", line 85, in map_shared_objects
    module_name, import_tag, extension = fullpath.parts[-1].split(".")
ValueError: too many values to unpack (expected 3)

The file causing the error is:

>>> import pdb; pdb.pm()
> /Users/pete/.virtualenvs/saltdash-apCx4VH9/lib/python3.6/site-packages/shiv/cli.py(85)map_shared_objects()
-> module_name, import_tag, extension = fullpath.parts[-1].split(".")
(Pdb) fullpath
PosixPath('/var/folders/nw/7ftbjmn50y9fy047fgyv4c3r0000gn/T/tmpdjxe_i80/site-packages/psycopg2/.dylibs/libcrypto.1.0.0.dylib')

I have a dependency on psycopg2-binary==2.7.4

Hello,

I have use cases where I want to deploy a "compact" python distribution in terms of files and Mb.
I am using the python embedded distribution + shiv to come down to ~35 files and 48 Mb on windows with pandas+matplotlib which is what I want.
However, the core entry point of the distribution should still be the python.exe and not the shiv file (I am using the python/powerBI itnerface that requires a python.exe).

So I am ending doing some "dark magic" in the sitecustomize.py to boostrap the "extra.pyz" (containing all dependencies not packaged with the python embedded distribution: pandas, matplotlib, etc) by reusing the logic of the _bootstrap.bootstrap() method but without the last part (the run or execute_interpreter call) which works.

Would there be some interest to add this use case (supporting on the fly cache compressed packages including .dll/.so files) to shiv capabilities ?

This way we have the data to choose the solution most suited to our needs

Since python 3.7, zipapp are able to filter particular pattern files.
It is useful for a larger project, which may content different kinds of ignore files.
I propose shiv can support both of the following way to filter out the ignore files.

1. Use filter function like 3.7 does

create_archive(APP_APTH, APP_NAME, SHEBANG, MAIN, FILTER_FUNCTION, COMPRESS?)

2. Use .pyzignore, like .gitignore, to define the filter pattern.
   They can be applied at the same time.

@sixninetynine! Shiv looks rad. I've been lamenting about pkg_resources slowing down our Python CLI tools @NerdWallet.

One pex feature we rely on is building a single .pex file for multiple platforms (i.e. pex --platform linux-x86_64 --platform macosx-10.11-x86_64 ...). We pre-compile all of our own wheel files in a private repo so this means we end up with a single .pex full of our custom built wheels (fast builds, happy security folks, etc...)

#7 talks about invoking shiv for a different target, but it doesn't talk about multiple targets.

Is this even possible with Shiv? I hope so, I'd love to contribute back to shiv and explore moving to it!

hello,
I just discovered this very cool project. At first sight, this looks very similar to https://github.com/pantsbuild/pex. For casual users passing by, it would be nice if the README contained a brief paragraph with the similarities and differences between the two projects.
Thank you.

As written, the example leaves the shiv pyz file with a shebang pointing at the (presumably temporary) virtual environment used to build it.

The example should probably include a -p "/usr/bin/env python3" argument to specify an explicit interpreter.

Actually, given that shiv pyz files are self-contained, there's no real benefit to them ever having shebangs that refer to a virtualenv - maybe it would be worth making the default shebang /usr/bin/env python3, or at least locate the "base" Python interpreter for a virtualenv - which can be done as

Path(getattr(sys, 'real_prefix', sys.base_prefix)) / Path(sys.executable).relative_to(sys.prefix)

This may not be easy to solve - and indeed, it may be that all that needs to be done is document this as an unsupported use case - but it prevents the virtual environment manager pew being bundled as a zipapp with shiv.

The only possible way of addressing this would be by putting shiv's site-packages into the PYTHONPATH environment variable for subprocesses, But that might have its own issues - I haven't thought the implications through.

Simple test case to reproduce the issue:

>shiv -o ex.pyz -e main:main --site-packages x virtualenv
 shiv! 🔪
Collecting virtualenv
  Using cached https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-16.0.0
 done
>py .\ex.pyz
C:\[...]\Python37\python.exe: No module named virtualenv
Traceback (most recent call last):
  File "C:\[...]\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\[...]\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File ".\ex.pyz\__main__.py", line 3, in <module>
  File ".\ex.pyz\_bootstrap\__init__.py", line 125, in bootstrap
  File "C:\[...]\.shiv\ex_8792863b-a069-46b2-ac25-7d2750b38a7b\site-packages\main.py", line 5, in main
    subprocess.check_call([sys.executable, '-m', 'virtualenv', '--help'])
  File "C:\[...]\Python37\lib\subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\[...]\\Python37\\python.exe', '-m', 'virtualenv', '--help']' returned non-zero exit status 1.

In build systems that allow products to build multiple Python packages (apps) together in parallel, such as PyGradle, the solution from #16 breaks the build of a package that does not use shiv. It was implemented in #27 but we did not notice until now the negative effect it has.

Since that solution is implemented only for homebrew Python, I believe it's unacceptable that it breaks standard Python builds and packages. We need to make this usage conditional, with some setting, so that users who need it can apply it, but the standard usage should not be doing this by default.

I'm inclined to just plain revert #27 to get the fix out ASAP.
Users who need this change can use the releases of shiv that have it, and later can use the future release of shiv that implements this conditionally.

Can we execute this plan promptly?
Thanks!

$ shiv -o glances.pyz -e glances:main glances
$ ./glances.pyz
Traceback (most recent call last):
  File "./glances.pyz/_bootstrap/__init__.py", line 125, in bootstrap
TypeError: 'module' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "./glances.pyz/__main__.py", line 3, in <module>
  File "./glances.pyz/_bootstrap/__init__.py", line 130, in bootstrap
AttributeError: module 'glances.main' has no attribute 'main'

Glances (strangely enough) has a module named main as well as a function named main. These work:

import glances.main; glances.main()
import glances; getattr(glances, "main")()

This (shiv's technique) does not:

from importlib import import_module
import_module("glances.main")()

Shiv includes system site packages in the Python path. Roughly the equivalent of virtualenv's --system-site-packages. In general (or at least for my uses) this is not desirable. Would you consider changing it to not include other site-packages or adding a flag to toggle the behavior at build time?

I ran into an issue where my pyz packages pyyaml 5.1 but when run and extracted, the pyyaml it actually imported was an outdated version from the core python installation. This behavior seems consistent whether it is a package or module, part of the standard library or something installed into the root python etc.

Is this intended behavior and if so, is there a best practice?

Would you consider a PR which made the setup process (currently supporting pip/setuptools) pluggable so projects using alternate build systems could work with shiv? I'm interested in poetry, but others like flit or pipenv could be supported as well.

Hello! I'm trying to have shiv create a file that has the asynq package. asynq uses Cython, and depends on qcore (which also uses Cython).

When I try a naive shiv asynq -o foo.shiv, it fails to install because compiling asynq requires qcore's Cython-specific files. It seems that when shiv runs pip install --target blah, the information about the target is not passed down to Cython, so Cython is trying to look in the normal Python path for the qcore files instead of the temporary site-packages directory that we are actually installing things into.

Is there a supported solution for how to install packages that depend on other Cython packages like this?

If not, I made a janky proof of concept (master...kennydo:kedo-set-pythonpath) for getting this to work.
This branch:

1. sets PYTHONPATH when calling pip to the target site-packages directory (so that Cython is able to look in the right place for the qcore files) and
2. makes individual pip install calls (so that when we know that qcore has been installed by the time we try to compile asynq).
   I was able to create a shiv successfully by running: shiv qcore asynq -o foo.shiv.

platform : Mac OS X, 10.14.3
Python version : 3.7.3
Pip version ( if relevant ) : 9.0.3

I was getting test failures on a branch and so checkout master as a baseline. Turns out that's failing for me as well.

git clone [email protected]:linkedin/shiv.git
cd shiv
python3 setup.py venv
source activate
python3 setup.py develop

then

pip install tox
tox -e py37

gives

        # check that the command successfully completed
        assert result.exit_code == 0

        # ensure the created file actually exists
        assert output_file.exists()

        # now run the produced zipapp
        proc = subprocess.run(
            [str(output_file)],
            input=b"import hello;print(hello)",
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            shell=True,
            env=os.environ,
        )

>       assert proc.returncode == 0
E       assert 2 == 0
E        +  where 2 = CompletedProcess(args=['/private/var/folders/x_/h517ndbj2qx9m6ydt_vtsrv00000gn/T/pytest-of-ianmaclean/pytest-27/test_n...x83\xa0\x88\x08\x08\xe8&\x85\x11!B\x13\xc4\x92\xaf\x9f\xe9\xa9\xb6\xba7\xd6[\xde\xba\xe7.\x9e$\r$ip&\xe5 K\x8a\x82'\n").returncode

test/test_cli.py:189: AssertionError

doing some more digging to see exactly whats failing. pyz's built from the built shiv package seem to work fine.

Has there been any investigation done into whether/how to support niceties from the normal Python REPL like tab completion and command history?

I've tried using import site; site.enablerlcompleter() and import rlcompleter; import readline; readline.parse_and_bind("tab: complete") before calling code.interact(), but neither seem to be enough.

With shiv 0.0.14 from PyPI:

$ shiv aws -c aws -p $(which python3.6) -o blergh
 shiv! 🔪
Collecting aws
Collecting fabric>=1.6 (from aws)
Collecting boto (from aws)
  Using cached https://files.pythonhosted.org/packages/bd/b7/a88a67002b1185ed9a8e8a6ef15266728c2361fcb4f1d02ea331e4c7741d/boto-2.48.0-py2.py3-none-any.whl
Collecting prettytable>=0.7 (from aws)
Collecting paramiko<3.0,>=1.10 (from fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/3e/db/cb7b6656e0e7387637ce850689084dc0b94b44df31cc52e5fc5c2c4fd2c1/paramiko-2.4.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/74/8e/a6c0d340972d9e2f1a405aaa3f2460950b4c0337f92db0291a4355974529/PyNaCl-1.2.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting bcrypt>=3.1.3 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/7e/59/d48fd712941da1a5d6490964a37bb3de2e526965b6766273f6a7049ee590/bcrypt-3.1.4-cp36-cp36m-macosx_10_6_intel.whl
Collecting cryptography>=1.5 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/40/87/acdcf84ce6d25a7db1c113f4b9b614fd8d707b7ab56fbf17cf18cd26a627/cryptography-2.2.2-cp34-abi3-macosx_10_6_intel.whl
Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/8e/be/40b1bc2c3221acdefeb9dab6773d43cda7543ed0d8c8df8768f05af2d01e/cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl
Collecting six (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
Installing collected packages: pyasn1, pycparser, cffi, six, pynacl, bcrypt, idna, asn1crypto, cryptography, paramiko, fabric, boto, prettytable, aws
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 335, in run
    use_user_site=options.use_user_site,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 49, in install_given_reqs
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 748, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 961, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/wheel.py", line 216, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/locations.py", line 165, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

Pip install failed!

Here's the packages installed systemwide alongside shiv:

click==6.7
importlib-resources==0.5
pip==10.0.1
setuptools==39.1.0
shiv==0.0.14
wheel==0.31.0

OS X, Python 3.6.5 from Homebrew.

(Original discussion in pypa/pip#5355).

Unlike "conventional" zipapps, it seems that shiv adds bootstrap code to unpack the archive and run it from an unpacked directory. This allows it to support compiled extension modules, but it also means that the unpacked archives get left around (at least on Windows, as you can't delete a loaded DLL in Windows, and I believe the code doesn't even try to tidy the unpacked archive up anyway).

I think this behaviour should be documented, at least in the "gotchas" section of the readme, as it's likely to be unexpected for at least some users (and may even be a privacy concern for some).

It would be nice if the tool only unpacked when needed, and warned when building an archive that needed unpacking. But I doubt that's possible in practice (it's essentially the "zip safe" flag that setuptools tried to implement for eggs, that is generally considered to have failed).

If I find this project via PyPI, it doesn't show any links to the project homepage or to the documentation. Please can these be added to the package metadata?

I'd like to be able to have a shiv option that would instruct the bootstrap code to clean out old directories in .shiv for the current app. Maybe:

shiv --delete-other-builds

With that option, when the first bootstrap of a build finished, the boostrap code would essentially run:

rm -rf ~/.shiv/the_app_name_*

Excluding the folder that contains the files for the currently running app.

https://en.wikipedia.org/wiki/Shiv

Not to mention that there are folks who go by the name shiv

build id can be sourced from environment like SHIV_BUILD_ID, therefore, it is possible pass in git HEAD hash code for the build id.

This tripped me up using pytest:main as an entrypoint today. That function returns an integer, that is expected to be used as the exit code. Apparently this is the standard for entry point scripts: https://packaging.python.org/specifications/entry-points/#use-for-scripts

When attempting to install shiv using an older version of Python, the installation attempt fails without the intended output message of "shiv requires at least Python 3.6!". This can cause confusion for users who are attempting to use shiv and didn't pay attention to the Python 3.6 requirement.

Example Output:

whitbox:shiv whitlock$ pip --version
pip 10.0.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)
whitbox:shiv whitlock$ pip install shiv
Collecting shiv
  Using cached https://files.pythonhosted.org/packages/67/92/4cc2db2f2da137ea776608687ab9511c8fdcad24f0124c727c9588a7ab80/shiv-0.0.13.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/3d/rlv56c1x3k7g07h70h8_rrz8000f2y/T/pip-install-3hzAeW/shiv/setup.py", line 93
        ...
        ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/3d/rlv56c1x3k7g07h70h8_rrz8000f2y/T/pip-install-3hzAeW/shiv/
whitbox:shiv whitlock$

Just a couple of things that could make the usage experience a little better ;

• Confirmation that an output file was written. ie generated pyz to <output_file> or similar
• Flagging of success vs some error condition. I was getting an error for a while - but wasn't entirely sure if it was just a warning - since the last output line is just "done".

If there is agreement that this would be useful I'm happy to put a quick PR together.

On Windows, the console supports UTF8, and since Python 3.6 console output does too. But output to a pipe or file (and presumably any other case where the output doesn't support full Unicode, such as a Unix system with an encoding other than UTF-8 specified) crashes:

>shiv shiv >a.txt
Traceback (most recent call last):
  File "C:\[...]\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\[...]\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Utils\Misc\shiv.pyz\__main__.py", line 3, in <module>
  File "C:\Utils\Misc\shiv.pyz\_bootstrap\__init__.py", line 137, in bootstrap
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\shiv\cli.py", line 150, in main
    click.secho(" shiv! " + SHIV, bold=True)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\termui.py", line 420, in secho
    return echo(style(text, **styles), file=file, nl=nl, err=err, color=color)
  File "C:\[...]\.shiv\shiv_fbaa7543-371f-4f00-99fc-7a416dfdc32a\site-packages\click\utils.py", line 259, in echo
    file.write(message)
  File "C:\[...]\Python37\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f52a' in position 7: character maps to <undefined>

Something like the following fix to cli.py should address the issue:

# This is the 'knife' emoji
SHIV = u"\U0001F52A"
try:
    SHIV.encode(sys.stdout.encoding)
except UnicodeEncodeError:
    SHIV = "!"

Hi, thank you for making an excellent project like this open-source. I have a Unicode character in my folder path and therefore get the following error:

Traceback (most recent call last):
  File "c:\programdata\miniconda3\envs\energi_xl\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\miniconda3\envs\energi_xl\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Miniconda3\envs\energi_xl\Scripts\shiv.exe\__main__.py", line 9, in <module>
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\shiv\cli.py", line 169, in main
    pip.install(["--target", str(tmp_site_packages)] + list(pip_args))
  File "c:\programdata\miniconda3\envs\energi_xl\lib\site-packages\shiv\pip.py", line 65, in install
    click.echo(output.decode().rstrip())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 15: invalid start byte

I fixed it by modifying the install function in shiv/src/shiv/pip.py.
I added the argument text=True to subprocess.Popen and removed decode() from click.echo(output.decode().rstrip()) (line 65).

i.e. from

...
    process = subprocess.Popen(
        [sys.executable, "-m", "pip", "--disable-pip-version-check", "install", *args],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        env=subprocess_env,
    )

for output in process.stdout:
    if output:
        click.echo(output.decode().rstrip())
...

to

...
    process = subprocess.Popen(
        [sys.executable, "-m", "pip", "--disable-pip-version-check", "install", *args],
	stdout=subprocess.PIPE,
	stderr=subprocess.STDOUT,
	env=subprocess_env,
	text=True,
	)

for output in process.stdout:
    if output:
	click.echo(output.rstrip())
...

I do not know if this is the best, or even, a good solution but in my case it gets the job done.

doing
shiv -e cpe_m.cpe_mi:cli -o my_command.pyz . in setup.py dir
and then try to run command I get this strange error msg?

[I] ➜ ./my_command.pyz
zsh: permission denied: ./my_command.pyz

Moving to azure pipelines broke our coveralls configuration

Attempting to follow the example given in the readme, using OSX 10.11.6:

$ mktmpenv --python=`which python3.6`
$ which python3
/Users/<username>/Virtualenvs/tmp-cf98a678da0c70e/bin/python3

$ python -V
Python 3.6.2

$ python3 -m pip download boto
[...] Saved ./boto-2.49.0-py2.py3-none-any.whl

$ python3 -m pip install shiv
[...]

$ shiv -o boto.pyz --find-links . --no-index boto
[...] Successfully installed boto-2.49.0

$ ./boto.pyz
Traceback (most recent call last):
  File "/Users/<username>/.pyenv/versions/3.6.2/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/<username>/.pyenv/versions/3.6.2/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "boto.pyz/__main__.py", line 3, in <module>
  File "boto.pyz/_bootstrap/__init__.py", line 127, in bootstrap
  File "boto.pyz/_bootstrap/__init__.py", line 79, in extract_site_packages
  File "boto.pyz/_bootstrap/filelock.py", line 71, in __enter__
  File "boto.pyz/_bootstrap/filelock.py", line 39, in acquire_nix
FileNotFoundError: [Errno 2] No such file or directory: '/Users/<username>/.shiv/boto_cab7037d-ea57-4001-9b01-da4b73096d73.lock'

$ ls -al $HOME/.shiv
ls: /Users/<username>/.shiv: No such file or directory

$ mkdir $HOME/.shiv

$ ./boto.pyz
Python 3.6.2 [...]
>>> 

It seems like the lock acquisition assumes the directory exists, but I can't see where that's explained in the documentation.

Adding an issue for myself to update documentation since we've added some features

Would you consider a PR that added a flag like --from-site-packages which would copy the contents of an existing site-packages directory into the working path?

My use case is that part of my build process requires creating a working virtualenv (needed to generate static files for a Django app). It would be an optimization to just use the virtualenv I already have setup to create the zipapp instead of rebuilding one from scratch.

Is there a pressing reason not to use click 7.0 as indicated by click>=6.7,!=7.0 in the install requirements?

The change in #48 makes sure that shiv packed content (packages) has the precedence over the vended Python distribution packages or things installed into system Python site-packages. Fair enough. We want to run what we packed into the shiv.

However, it does have a potentially buggy behavior. Example:

• an old deprecated 3rd party package such as uuid or argparse is pulled in transitively
• these packages are unmaintained and did not get any changes (uuid since Python 2.5, for example)
• the standard library modules have been maintained and added new APIs (uuid did)
• since shiv's site-packages are stored on the sys.path before the standard library path, an obsolete 3rd party backward-compatibility package (but not forward compatible) will override the corresponding standard library module
• if any other package uses the new APIs from the affected package (e.g., new uuid APIs), the application will break because the old 3rd party package does not have these functions/methods.

I believe this requires changes to what was done in #48 to insert shiv site-packages before any other site-packages paths, but after the standard library path.

I've found launching multiple shiv apps concurrently in a clean environment can lead to processes tripping each other up, as they all try to unpack into ~/.shiv at the same time.

I know we can set SHIV_ROOT, but it would be nice to have a less cumbersome alternative for external callers of shiv apps. Callers who set a custom SHIV_ROOT to work around concurrency issues are then in a position of managing that directory (or hey, not), and it adds up to a lot of boilerplate and leaky implementation. My suggestion is to support ephemeral environments from right inside of shiv.

Example implementation at https://github.com/linkedin/shiv/compare/master...aayars:ephemeral?expand=1 -- if that seems okay, I'll move ahead with a PR.

An alternative implementation might have been to grab a lock file, but that seemed too heavy-handed. This approach preserves existing behavior unless the caller requests an ephemeral environment.

Certain systems* don't allow the use of entry points after pip installing a package. Would anyone be against me adding an __init__.py module so that it can also be run via python -m shiv ...?

*Jenkins pipelines + ephemeral docker agents

I am trying to make python a first class language in my enterprise. Problem is we do development on windows and deploys to linux.

The thing that has prevented us from beginning to write stuff in python before was Pex was never windows compatible (because it relied on symlinks). Is shiv?