Comments (22)
Hi all, I suggest to all the people interesting in this issue, to have a look to pytest-depends
(https://pypi.org/project/pytest-depends/; https://gitlab.com/MaienM/pytest-depends).
from pytest-dependency.
FWIW: I added an option to pytest-order (a fork of pytest-ordering) to also order dependencies if needed (use --order-dependencies
if the plugin is installed). For the time being, this can be used to work around this issue.
from pytest-dependency.
@DevilXD - as I wrote, I think the plugins complement each other. If you only need ordering, you are probably better of with pytest-order
or similar, as pytest-dependency
is not about ordering. That doesn't mean that pytest-order
is superior in any way - it just has different goals. Also note that pytest-order
is pretty new (at least the part it hasn't inherited from pytest-ordering
, which is the relevent part here) and likely still has bugs...
from pytest-dependency.
Using both plugins forces to define order twice (implicitly with pytest-dependency and explicitly with pytest-order), while pytest-dependency already determines the execution order. Plus, it's easy to create conflicting orders:
@pytest.mark.depency(name="test1")
@pytest.mark.order(2)
def test_test1():
pass
@pytest.mark.depency(depends="test1")
@pytest.mark.order(1)
def test_test2():
pass
Pytest-dependency already defines execution order, but does not enforce it in any way. It should be essential part of it.
Looking at a car, all a driver does is turning a steering wheel and pushing pedals in order to travel. But there is a lot more required in addition to supporting the steering wheel and the pedals in order to make the driver travel.
from pytest-dependency.
I took a stab at this problem separately from this package. https://gist.github.com/jwineinger/e41473cd5500554a189d8c97ac28f883 does work to reorder dependencies and does so across files and classes as well. I couldn't get it to play nice with the result tracking/skipping in this package but I'm intending to run pytest with -x anyway, so I don't care so much about that part.
from pytest-dependency.
@jwineinger - seems like what I'm looking for. were you able to link tests together?
from pytest-dependency.
@Formartha if by link you mean specify dependencies and reorder so that dependencies are run first, then yes. To be clear though, my linked gist is separate from pytest-dependency, so you don't get any of the features it provides. Just drop the conftest.py into your top-level test directory, install deps, and decorate your tests.
from pytest-dependency.
I have a similar need, make test run in dependency order.
There is another module trying this as well: https://github.com/ftobia/pytest-ordering
He can do ordering but not "relative dependency" as pytest dependency do.
from pytest-dependency.
I just ran into the same issue. I have a bunch of unit tests and integration tests.
Obviously, if a unit-test fails, the corresponding integration test will fail as well but I am only directly interested in the unit-test as it will usually have a simpler more precise setup. This makes it easier to identify the problem.
The integration tests have a much more complex setup and usually have much more output as well in the failure reports. So I would love to be able to skip them when the linked unit-test fails.
But as the tests are run in a different order (I'm not going to rename my tests to control ordering) this will currently always skip the integration test. Even if the unit-test passes.
from pytest-dependency.
Hi all, I think reordering of tests with dependencies is very useful. I used @jwineinger gist code to achieve this, I think it should be very nice to embed that idea in pytest-dependency.
from pytest-dependency.
I've put together a conftest.py
hook that should reorder the dependent tests based on their dependencies. It even leaves you a RuntimeWarning in case you'd mistype a dependency name (I know that using node IDs can be tricky sometimes).
https://gist.github.com/DevilXD/afc7f923049a5abedea55ba186e7219c
Note that there's no cyclic dependencies validation here, and using this with those will most likely cause pytest to hang at the collecting phase. I'm leaving this here in case anyone would like to use it, and push this issue forward, at least a tiny bit.
EDIT: Updated to ignore parametrization, and properly detect dependency cycles.
from pytest-dependency.
Will there be a new release of pytest-dependency after 0.5.1 that fixed this? Seems this issue affects many people, see for example https://stackoverflow.com/questions/17571438/test-case-execution-order-in-pytest#comment116177289_46393615
from pytest-dependency.
Actually I think it is not a bad thing that pytest-dependency
does not order the tests. I've meanwhile read a couple of comments by @RKrahl that state that he would prefer for the plugin to stay small and focused on one task, and I agree with that philosophy. Ordering tests is another task which can be accomplished using a separate plugin. I adapted pytest-order
specifically to play nice with pytest-dependency
, so that the combination should work ok (provided there are no bugs in pytest-order
, of course).
from pytest-dependency.
This makes pytest-order
superior to this library then, pretty sure - unless someone would really like to skip tests when others fail, I guess. Given that it's maintainer has neglected at the very least responding with their thoughts, to this issue and to the ordering PR I did...
I'm considering switching to pytest-order
at this point, because @mrbean-bremen clearly appears to care more about functionality like this. I shall give it a shot.
from pytest-dependency.
Pytest-dependecy is grossly incomplete without test ordering. It actually introduces to pytest a new logical execution order, but without any engine to enforce it. "Just" skipping is implemented.
Adding another plugin to the mix would add another level of ordering (pytest -> pytest-dependency->pytest-ordering) without any guarantee, that pytest-dependency order would not conflict with pytest-ordering one.
from pytest-dependency.
Pytest-dependecy is grossly incomplete without test ordering. It actually introduces to pytest a new logical execution order, but without any engine to enforce it. "Just" skipping is implemented. Adding another plugin to the mix would add another level of ordering (pytest -> pytest-dependency->pytest-ordering) without any guarantee, that pytest-dependency order would not conflict with pytest-ordering one.
pytest-dependency
was never meant to be about ordering. It's all about (conditionally) skipping of tests. This may have been considered a deficiency while pytest-order
was not available. But now that we have pytest-order
, this issue is solved. If you believe that this solution is not appropriate and want to advance the discussion, you should come up with a concrete example where the solution does not work properly.
from pytest-dependency.
Pytest runs unit tests in an arbitray order. It's usually alphebetical but this cannot be relied on.
How can this plugin alone even work? I'm with @lukjak and @DevilXD on this.
from pytest-dependency.
Is there any way to run @mark.depends without changing the tests order?
I want that my test will still run in the order they appear in the class but that some of the tests will be dependent on others,
For example I have this tests:
- test_login_with_wrong_user_name
- test_correct_login
@pytest.mark.depends(on=["test_correct_login"]
3. test_verify_user_name_after_login
In this example I want the tests will run in this order but if test_correct_login fails I don't want that test no' 3 will run.
What actually happens, when I'm running tests from some classes he mixes the run order between all the classes and making test to fail because now they running in some random order
from pytest-dependency.
@LiorPelet pytest-dependency
currently does no test reordering, and this issue is all about adding said reordering to the plugin. If your tests are getting reordered, then it's not because of this plugin. Even with the reordering implemented, dependent tests would always run after the tests they depend on, so you wouldn't get this issue in either case.
I'd recommend taking a look at your pytest env to see if there isn't any other plugin that'd reorder your tests instead.
from pytest-dependency.
@LiorPelet - you seem to use pytest-depends
, not pytest-dependency
. pytest-depends
always reorders tests in an unpredictable manner, even without any dependencies defined. There is already a bug for this filed, but it looks like it will not be fixed.
That being said, pytest-depends
does reorder the tests in a manner that the dependencies are executed in the correct order, it just does so in an unpredictable manner.
If you want your tests not been reordered (e.g. they are already in the correct order), you can use pytest-dependency
instead. Note that you have to uninstall pytest-depends
in this case, as that always reorders the tests, regardless of the markers.
from pytest-dependency.
FWIW: I added an option to pytest-order (a fork of pytest-ordering) to also order dependencies if needed (use
--order-dependencies
if the plugin is installed). For the time being, this can be used to work around this issue.
@mrbean-bremen I'm confused about how --order-dependencies
work. Does this mean I just have to install both plugins, specify my dependencies and set --order-dependencies
and it will work? Or will I have to annotate all the tests with both @pytest.mark.order
and @pytest.mark.depency
? Also is --order-dependencies
configured in project.toml or setup.cfg?
from pytest-dependency.
Does this mean I just have to install both plugins, specify my dependencies and set --order-dependencies and it will work?
Yes, exactly.
You can configure it in pytest.ini.
from pytest-dependency.
Related Issues (20)
- Test suite incompatibility with pytest 6.2 HOT 2
- Tests skipped on linux(3.6.13)(with no test failed) but executing on windows(3.6.10) HOT 3
- Dependency between two tests HOT 1
- session scope not working HOT 1
- Depends on all test methods in class HOT 1
- Conditional dependency on 2 tests HOT 2
- 0.5.1: pytest is failing HOT 4
- Is ordering on fixtures supported?
- only test_b is passed when install pytest-xdist with n>2.
- Not skipping a test when parent test skips or xfails
- The test was skipped,although the dependency test is success. HOT 5
- Dependency call is skipped by executing test with mark HOT 3
- pytest dependency not skipping the dependent tests when failed HOT 3
- when the dependency case use @pytest.mark.parametrize, depened cases was skipped,although the dependency test is success HOT 1
- is @pytest.mark.dependency() can be used for class level? like API test class depends on BVT test class...
- pyproject.toml missing
- Inheriting from test class with dependencies doesn't work correctly HOT 1
- Is pytest-dependency suppose to tell me why a test was skipped? I see nothing in the output explaining why. HOT 1
- Rerunning skipped cases when the parent test fails
- 0.6.0: pep517 based build fails with `Invalid version: 'UNKNOWN'` error message HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-dependency.