http://github.com/nir0s/distro is a new, more robust (and is now used by pip) implementation of Python's platform.linux_distribution() which will be removed in Python 3.7. We should first specify distro as an optional dependency for wagon (maybe allow to install it by running pip install wagon[dist]?) so that it can be used. We should keep using Python's implementation for now but deprecate it and then fully move to using distro (if still relevant) in Wagon v1.0.0

Wheel's API provides the following:

• wheel.archive
• wheel.install
• wheel.pkginfo
• wheel.util

Maybe we should change the names of the subcommands to:

• wagon archive (instead of create)
• wagon pkginfo (instead of show)

WDYT?

To allow users to know which version of Wagon creates a specific wagon, we should provide it with each wagon's metadata. This will allow users to handle backward compatibility issues.

We can use:

import pkg_resources
wagon_version = pkg_resources.get_distribution('wagon').version

to get the version and then append it in the metadata: created_with: 'wagon-{0}'.format(version)

When building a wagon containing (e.g.) cffi, pip will happily pull down a manylinux1_x86_64 wheel. wagon will (sometimes; see #60) create a wagon with a supported platform of manylinux1_x86_64, but will then refuse to install this on a linux_x86_64 system, because the two strings don't match.

This is the output (note that I've opened #59 for the traceback):

+ wagon install -s foo-0.1.0-py27-none-manylinux1_x86_64-Ubuntu-precise.wgn
INFO - Installing foo-0.1.0-py27-none-manylinux1_x86_64-Ubuntu-precise.wgn
ERROR - Platform unsupported for package (linux_x86_64).
Traceback (most recent call last):
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/bin/wagon", line 11, in <module>
    sys.exit(main())
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/wagon/wagon.py", line 519, in install
    virtualenv, requirements_file, upgrade, ignore_platform, install_args)
  File "/srv/jenkins/jobs/TEMP-Wagon-Test/workspace/wagon_venv/local/lib/python2.7/site-packages/wagon/wagon.py", line 180, in install
    sys.exit(codes.errors['unsupported_platform_for_package'])
KeyError: 'unsupported_platform_for_package'

To accommodate for the inconsistencies between wagon and pip, and to allow for additional functionality, we will have to perform breaking changes until we can release v1.0.0. This should be stated in the docs so that people know they have to use specific versions of wagon to both create and install wagons.

With the advent of manylinux, a wagon can (validly) contain wheels for two platforms. In my case, the two platforms are linux_x86_64 and manylinux1_x86_64.

https://github.com/cloudify-cosmo/wagon/blob/master/wagon/utils.py#L241 assumes that the first non-any platform it finds applies to all wheels in the wagon, which means that a wagon containing exactly the same set of packages (some with linux_x86_64 and some with manylinux1_x86_64) can have a different supported_platform depending on the sort order of the directory containing the wheels.

DEBUG - installing to build/bdist.linux-x86_64/wheel
DEBUG - running install
DEBUG - Traceback (most recent call last):
DEBUG - File "", line 1, in
DEBUG - File "/tmp/pip-5nImSl-build/setup.py", line 82, in
DEBUG - 'cloudify-plugins-common>=3.4'
DEBUG - File "/opt/cfy/embedded/lib/python2.7/distutils/core.py", line 151, in setup
DEBUG - dist.run_commands()
DEBUG - File "/opt/cfy/embedded/lib/python2.7/distutils/dist.py", line 953, in run_commands
DEBUG - self.run_command(cmd)
DEBUG - File "/opt/cfy/embedded/lib/python2.7/distutils/dist.py", line 972, in run_command
DEBUG - cmd_obj.run()
DEBUG - File "/opt/cfy/embedded/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 215, in run
DEBUG - self.run_command('install')
DEBUG - File "/opt/cfy/embedded/lib/python2.7/distutils/cmd.py", line 326, in run_command
DEBUG - self.distribution.run_command(command)
DEBUG - File "/opt/cfy/embedded/lib/python2.7/distutils/dist.py", line 972, in run_command
DEBUG - cmd_obj.run()
DEBUG - File "/tmp/pip-5nImSl-build/setup.py", line 68, in run
DEBUG - install_requirements()
DEBUG - File "/tmp/pip-5nImSl-build/setup.py", line 52, in install_requirements
DEBUG - pip_win_path = path.join(environ.get('VIRTUAL_ENV'), 'Scripts/pip.exe')
DEBUG - File "/opt/cfy/embedded/lib/python2.7/posixpath.py", line 77, in join
DEBUG - elif path == '' or path.endswith('/'):
DEBUG - AttributeError: 'NoneType' object has no attribute 'endswith'

Currently, Wagon's default format when creating an archive is tar.gz on all platforms. This will make it harder to work with when using Windows. We should generate a zip by default to correspond with pip's implementation.

Virtualenv is currently a requirement because it is used in wagon validation but is not mandatory for creating and installing wagons.

Should we provide it as an extra requirement so that users can install it via pip install wagon['validate'] instead?

By passing the --with-requirements flag, we try to retrieve requirement files (namely, dev-requirements.txt and requirements.txt) automatically from a pip installable archive or a local directory. While this might be nice for the user, it's not explicit and might confuse users. It is also not consistent what pip and wheel which require explicitly passing the relevant requirement files via multiple -r flags.

I suggest we remove that and allow users to pass multiple -r flags with explicit paths (local or URLs) to requirement files.

I wonder why you did not elect to create aggregated wheels instead which is a scenario that has been always support by wheel? (eg rewheeling several wheels in one single wheel?)

When running wagon against a file system source project, it keeps trying to create the package in the current directory. For example, if the source project is in the current directory, it complains that the directory already exists. If I move to another directory (like /tmp, which is set as the output directory anyway) it works fine.

From the README:

If you're using pip>=10.x, please make sure you use wagon>=0.8.0

As far as I can tell, wagon 0.8.0 hasn't yet been published. Would it be possible to get a release so we can take the fix for #120 and start using wagon with pip >= 10.x?

While we can declare that wagon only supports manylinux1 if the wheels in it are manylinux1, I think we should add the ability to convert linux_x64_86 to manylinux1 to make the job a bit more comfortable for the user.

Relevant links:

• This PEP describes the tag.
• Auditwheel

Since auditwheel requires Centos 5 (not older) to work (due to forward backward compat of glibc), we can verify that version of the distribution and only if it's Centos 5-, allow to run wagon repair which will recursively repair all wheels in the wagon and recreate the wagon.

To test creating a manylinux1 compat wheel, try

$ docker run -i -t -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /bin/bash
$ /opt/python/cp35-cp35m/bin/pip wheel flask
$ auditwheel repair MarkupSafe-0.23-cp35-cp35m-linux_x86_64.whl
...

Repairing MarkupSafe-0.23-cp35-cp35m-linux_x86_64.whl
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp35-cp35m-linux_x86_64
New WHEEL info tags: cp35-cp35m-manylinux1_x86_64

Fixed-up wheel written to /wheelhouse/MarkupSafe-0.23-cp35-cp35m-manylinux1_x86_64.whl
...

WDYT?

When performing an install with wagon 0.6.0, I see the following lines in the console output (with the app name replaced):

Extracting zip .../foo.wgn to /tmp/tmpWN6MLw...
Source is: /tmp/tmpWN6MLw/foo

Once installation is complete, /tmp/tmpWN6MLw still exists. This has caused a disk space issue for us in a Jenkins environment where we are installing wagons a lot.

Wagon should support Python 3.x

The test environment isn't very organized so we should also arrange it

Currently, Wagon appends two additional fields to each archive's file name if running on Linux - distribution name and distribution codename. These tags are not a part of the wheel standard and are only there to help with installing on Linux (if not all wheels are manylinux1).

To accommodate the need to still identify the Linux distribution and release the wagon was created on, these tags should be removed from the file name and remain in the metadata only (where they also currently reside). Once manylinux1 is (hopefully) easier to generate, we would be able to deprecate this information from the metadata as well.

Practically, this means that if you want to auto-identify on which distribution and release you most probably can install the wagon, you would have to either use the CLI or the API to retrieve its metadata and read the relevant fields.

Currently, the supported python versions are either provided via user input or automatically deduced (as a single version) from the python version currently used to create the wagon.

1. We should rethink allowing users to provide the versions. It might be irrelevant
2. We should redo the mechanism which understands the python versions supported as the current one is technically not true. We might want to iterate through all wheels and get the set of common versions and use them instead.

Currently, when simply running wagon, it will error out on "too few arguments".
We should display the help message instead.

I have a requirements file for a bunch of packages I want to install on some servers. I want to use Wagon to easily package and install them. However, Wagon forces me to select a "source" package. I can arbitrarily select one of the packages I want to install, specify it as "soruce", and remove it from the requirement files (to avoid a duplication error), but I think it would be best if I could just use Wagon to "archive" a list of unrelated packages.

When a wagon is created and the platform tag is "linux_x86_64, the distro and version tags might be "none" since they were not retrieved by platform.linux_distribution() (like on arch). It should be stated in the docs.

Currently, wagon just assumes that the ABI if none. The ABI is wheel specific and so I'm not even sure there's a smart way to calculate it for multiple dependencies. WDYT? For now, it remains None, until we can figure out if we should at all address this issue.

We currently try to import virtualenv but also have many virtualenv variables in the code. We should probably change them to venv or something of that sort.

I've verified that the only place where we use the virtualenv module itself is where we check if it's found in globals. This should be changed to something like:

try:
    import virtualenv
    VIRTUALENV_EXISTS=True
except ImportError:
    VIRTUALENV_EXISTS=False

I went through the process of building "waggoned" plugins.

We are still using the 3.2 version of the Cloudify management server. It tries to pip install our Wagonned plugins, which fails, because there is no longer a setup.py file.

Is there any way to use wagons with cloudify 3.2?

As per file naming convention a build-tag can be specified as an argument to wheel via --build-number option. But currently specifying a tag has no impact on archive_name.

Wagon's CLI's structure is not consistent with pip/wheel. For instance, providing a requirements file for wagon requires using the -a '-r requirements.txt' flag. Currently, this is not so important as the main idea by the -a flag is to allow users to pass any arguments to pip or wheel. At some point, we should make this API consistent.

pip 10 was released yesterday and is currently breaking Wagon. The --use-wheel flag was removed from pip and replaced by --only-binary.

This is actually probably 2 issues.

Issue 1

Steps to reproduce:

• wagon create pip
• wagon validate <RESULTING_WAGON>
  The validation will fail.

Issue 2

The steps to reproduce are the same, but here the cause is clear - after pip install _s change to -s, and so this fails.
For example, here's a package on which create/validate will fail.

1. The wagon logger logs is set to INFO level by default, perhaps this should be changed to DEBUG
2. The functions exposed by the module have a verbose flag (which is just fine). Only problem is that this flag sets the logger logging level on each invocation. This may lead to unexpected behavior by users that may configure loggers in their app, etc...

Much clearer and easier to maintain

(First, cool tool! Python needs something like a fat-jar equivalent from the java world)

If you look at _get_pip_path you can see it does not handle pip installs on machines that are:

1. Not windows
2. Not in a virtualenv
3. Pip is not installed at /usr/bin/pip

https://github.com/cloudify-cosmo/wagon/blob/master/wagon.py#L448-L449

This happened for me because pip is installed here (default location by install from get-pip.py):

gnicholas$ which pip
/usr/local/bin/pip

Which results in the following when I try to create a package:

gnicholas$ wagon create . -r requirements.txt 
Creating archive for ....
Retrieving source...
Source is: .
setup.py file found. Retrieving name and version...
ae_deploy_utils
0.0.1
Downloading Wheels for ....
/bin/sh: /usr/bin/pip: No such file or directory
Failed to download wheels for: ['requirements.txt']

Possible fix?

You can invoke pip without knowing it's exact path using python's '-m' flag:
python -m "pip"

We currently allow to install Wagon into a virtualenv directly by passing the -e VENV_PATH flag. This doesn't exist in pip, and so, for consistency's sake, we should remove it (specifically before Wagon v1.0.0).

Users can always activate a venv or directly pass its pip executable when installing a wagon.

If a creation failed, running the same creation again will state that the "folder already exists".
On failure, Wagon should perform a cleanup.

The idea behind excluding packages was to allow users to install wagons in environments where some requirements were already installed (whether of the required dependency version or otherwise). Using constraint files, this feature isn't needed anymore and you can simply put a constraint on the already installed package in your environment.

Thus, the feature of excluding packages should be removed.

This is only helpful for environments where pip (and therefore wheel) (and currently virtualenv - see #64) is installed as they can simply install Wagon's wheel via pip and don't require any additional dependencies. Funnily enough, if they don't have wheel they will need a wagon of wagon and need wagon to install it :)

Wagon 0.3.1 uses sys.executable as a reference to where pip is found. In Windows, that might not be in the sys.executable's path but rather under scripts. The tests are not broken since both Travis and Circle provide a build in version of Python where pip is available in the correct path.

We should provide special treatment for Windows.

https://github.com/pypa/auditwheel allows to create precompiled wheels supporting different linux distros. Maybe we can utilize it to modify the wheels while the wagon is created

This can be reproduced like this:

lxc launch ubuntu:x reproducer
lxc exec reproducer /bin/bash
apt update -y
apt install -y virtualenv
virtualenv -p $(which python3) venv
. venv/bin/activate
pip install git+https://github.com/cloudify-cosmo/wagon
wagon create Django
/etc/init.d/networking stop
wagon install Django-2.1.5-py27-none-any.wgn

This will eventually succeed, but takes a lot longer than normal because every individual package installation has lines like this:

Collecting Django
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f68eaca8b70>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/django/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f68eaca8898>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/django/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f68ead04f28>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/django/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f68ead04fd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/django/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f68eaca83c8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/django/

I've tracked this down: when wagon install calls pip, it does this:

/root/venv/bin/pip install Django --only-binary --no-index --find-links /tmp/tmpzLn8ik/Django/wheels --pre

However, --only-binary actually takes an argument, so it swallows the --no-index from the command-line, meaning that pip tries to contact PyPI.

(This may be related to #56, but I'm not seeing the same error output so I'm not sure.)

When using offline with a requirements.txt file and all packages found locally, with cmd wagon create -r -s my/folder/, Wagon still tries to reach for pypi.python.org to download wheels.
After about 2 minutes that it hangs it continues and create the wagon successfully.

how to reproduce:
install a package with embedded python like:
https://gigaspaces-repository-eu.s3.amazonaws.com/org/cloudify3/3.4.0/m5/mgmt-worker-3.4.0~m5-394.el7.x86_64.rpm
create a constraints file: pip freeze > const.txt
download a wagon which has a dependency that exists in the constraints and doesn't exist in the system python like:
http://repository.cloudifysource.org/org/cloudify3/wagons/cloudify-diamond-plugin/1.3.2/cloudify_diamond_plugin-1.3.2-py27-none-linux_x86_64-centos-Core.wgn
install with prefix and constraint:
wagon install -s cloudify_diamond_plugin-1.3.2-py27-none-linux_x86_64-centos-Core.wgn -a '--prefix /opt/mgmtworker/env/plugins/cloudify-diamond-plugin-1.3.2 --constraint /opt/mgmtworker/env/const.txt --verbose'

one of the errors show usage of wrong pip:

Exception information:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
wb.build(autobuilding=True)
File "/usr/lib/python2.7/site-packages/pip/wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/lib/python2.7/site-packages/pip/index.py", line 489, in find_requirement
'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for cloudify-plugins-common==3.4a5 (from -c /opt/mgmtworker/env/const.txt (line 8))

pip wheel creates a directory where all wheels are stored. Wagon should do the same.

Right now, we allow arbitrary flags using a -a flag.

I have a virtualenv that's activated but that virtualenv does not have virtualenv installed (nor is it installed globally). Wagon fails due to a validation error that virtualenv is not available. I really can't imagine that this has an actual requirement (unless Wagon is creating its own virtualenv?!?) so I'm wondering why this is a fatal-error-level issue.

(venv) [gibson ~]$ wagon create -o ./wbuild --validate -v airflow
virtualenv is not installed and is required for the validation process. Please make sure virtualenv is installed and is in the path. (You can run 'pip install wagon[venv]')

(venv) [gibson ~]$ env | grep VIRTUAL_ENV
VIRTUAL_ENV=/home/ec2-user/venv

Allow an uninstall command to pip uninstall with dependencies. Since the metadata contains the entire set of the dependencies for the package, it can easily do that.

The caveat is that it might uninstall packages currently used in the same environment. Yuch.

codename (as seen in http://github.com/nir0s/distro) is not consistent. We should be using the version.

Currently, wagon doesn't support packaging dependencies found under extras_require much more than wheel does: http://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies

Of course, this can be easily worked around by adding an optional-requirements.txt file and passing it to wagon. Maybe we should add an --include-extras flag for that?

[jp@cfy-ctrl wagons]$ wagon create git+https://github.com/Cloudify-PS/cloudify-versa-plugin.git
2017-01-31 17:31:58,896 - wagon - INFO - Creating archive for git+https://github.com/Cloudify-PS/cloudify-versa-plugin.git...
Source URL type git+https is not supported
GitHub
Cloudify-PS/cloudify-versa-plugin
Contribute to cloudify-versa-plugin development by creating an account on GitHub.