One of us will add an issue I guess, but here's the general gist of what we're trying to solve.
First of all, here's an example tox.ini file:
https://github.com/openstack/python-glanceclient/blob/master/tox.ini
The OpenStack CI infrastructure is an automated gated trunk. That means that we push changes up to a gerrit instance code review, and then if they are approved, Jenkins grabs the changes, tests them, and if they pass tests it submits a message back to Gerrit which causes the change to be merged. We run all of these tests in clean environments, and for unittests and pep8 checks, we use tox to manage the creation of virtualenvs and the running of tests in those environments.
https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-python26/configure
https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-python27/configure
https://jenkins.openstack.org/view/Gate/job/gate-python-glanceclient-pep8/configure
https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/run-tox.sh
Perfect, right?
Problem is, we test somewhere around 200 changes a day all hands off, so things like transient network problems between our build hosts and PyPI (which happen often) become a major impediment - even if all we're talking about is the index look up for the dep name.
So as an optimization, we have an out-of-band job which runs pip bundle on our deps lists:
https://jenkins.openstack.org/view/Gate/job/python-glanceclient-venv/configure
https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/build-bundle.sh
Installing deps into a virtualenv from a bundle file avoids all network accesses, as all of the index lookups are pre-done. Problem is, there isn't a great way to do conditional deps lists - I can't override deps in tox:jenkins. So we make a bunch of 'fake' environments in our tox.ini file that jenkins can use jenkins26, jenkins27, etc.
This is a bit ugly though, as it encodes specifics of our CI implementation and specifically the caching concerns we have on a system that runs gobs of tests per day that don't apply to any of the individual developers.
You may remember I took a stab at making tox virtualenvs relocatable a while back as an attempt to solve this systemically - turns out that virtualenvs just are not possible to relocate in a sensible manner.
What I REALLY want is a way for tox to understand bundle-based caching, so that I could run a tox command in build-bundle that would produce something I could later drop into a location on a different machine, so that when tox decides it needs to create a virtualenv it could use the discovered bundle file instead of the directly listed deps, at which point I could get rid of the jenkins* envs inside of tox.ini and treat the problem of caching as a step that the CI system needs to do inside of a pipeline of activity.
As a half-way step to that (which I figured would be a much more invasive patch) we wrote the patch to do dep override on the command line, because at least that way we can have jenkins job do things that are specific to jenkins and have nothing to do with a normal use of the repo, like::
tox -epy27 --deps=.cache.bundle
py27 can then simply describe how to run the job, and just because we've already pre-created the dep bundle, we can tell it to use that. That gets us 95% of the way there- we can clean up the tox.ini files in the close to 30 branches we're managing this way and then can treat the further solution to the problem on the CI side.
Does that make sense? Sorry for the novel here.