Add all previous tests as dependency? Or wildcard? about pytest-dependency HOT 3 OPEN

See
JoeSc@ca6212e

from pytest-dependency.

I'm not sure if I like the idea. I see at least two issues:

1. The use of magic names. Note that "all" is a perfectly valid name for a test.

2. It changes the nature of the dependency relation in an inconsistent manner. Until now this relation is static. The dependencies are given as a fixed list of tests. It is a priorily determined whether any other test belongs to the dependencies of a given test. This is independent of the current invocation of pytest. This is by design and I have use cases that depend on this. What your are proposing here is something dynamic. The dependencies of a tests in your proposal are all the tests that have been run until now in the current invocation of pytest. This is not determined a priorily.

Some examples to illustrate the issues this:

Example for the first issue

The following is perfectly valid with the current pytest-dependency:

import pytest

@pytest.mark.dependency(name="whole")
@pytest.mark.xfail(reason="deliberate fail")
def test_a():
    assert False

@pytest.mark.dependency(name="all")
def test_b():
    pass

@pytest.mark.dependency(name="universe", depends=["all"])
def test_c():
    pass

test_c does not depend on test_a. It will be run and pass. Yes, I do recongize the difference between depends=["all"] and depends="all". With your proposal in place, we might have the following:

import pytest

@pytest.mark.dependency(name="whole")
@pytest.mark.xfail(reason="deliberate fail")
def test_a():
    assert False

@pytest.mark.dependency(name="all")
def test_b():
    pass

@pytest.mark.dependency(name="universe", depends="all")
def test_c():
    pass

Now, test_c would be skipped. I would find this confusing.

Example for the second issue

Assume your proposal to be in place. Consider:

import pytest

@pytest.mark.dependency(name="a")
@pytest.mark.xfail(reason="deliberate fail")
def test_a():
    assert False

@pytest.mark.dependency(name="b")
def test_b():
    pass

@pytest.mark.dependency(name="c", depends="all")
def test_c():
    pass

From reading the code, one would assume it to be equivalent with:

import pytest

@pytest.mark.dependency(name="a")
@pytest.mark.xfail(reason="deliberate fail")
def test_a():
    assert False

@pytest.mark.dependency(name="b")
def test_b():
    pass

@pytest.mark.dependency(name="c", depends=["a", "b"])
def test_c():
    pass

But it is not. Assume the example to be saved as test.py and running python -m pytest test.py::test_b test.py::test_c respectively, test_c would not be skipped in the first variant, but skipped in the second one. I'd call this inconsistent.

from pytest-dependency.

Issue 1 I could see being solved by adding a unique flag in the mark to avoid confusion @pytest.mark.dependency(all_previous=True)

Issue 2 I see no way around since all by definition is dynamic.

Do you see any way to bring something like this into pytest-dependency? Or is the static vs dynamic definition of dependencies too great of barrier.

I have other uses for pytest-dependency so it'd be nice to keep it all clean with dependency marks. The only alternative I can see is using the request fixture

def test_d(request):
   if request.session.testsfailed:
       pytest.skip()
   pass

from pytest-dependency.