Long time user of flakefinder here - thank you for providing this awesome workaround for the silly pytest limitation. I have run into this extension failing with a specific type of tests. Please allow me to walk you through the issue:
Let's write a test that uses @pytest.mark.parametrize
$ cat << EOT >> test_1.py
import pytest
@pytest.mark.parametrize(
"a, b",
[(1, 2), (3, 4)],
)
def test_flake(a, b): pass
EOT
$ pytest --disable-warnings --collect-only -q test_1.py
test_1.py::test_flake[3-4]
test_1.py::test_flake[1-2]
2 tests collected in 1.31s
Let's run one of them:
$ pytest test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1277854280
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item
test_1.py . [100%]
=========================================================== 1 passed in 1.31s ============================================================
Let's use flakefinder:
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3718677389
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 0 items
========================================================= no tests ran in 1.29s ==========================================================
ERROR: not found: /mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]
(no name '/mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]' in any of [<Module test_1.py>])
this breaks, because I think flakefinder internally uses parametrize:
|
metafunc.parametrize( |
|
argnames=fixture_name, |
|
argvalues=list(range(self.flake_runs)), |
|
) |
which clashes with the test's parametrize
Note that tests that use a unitest parameterized have no problem as there it uses a different format like test_flake_1_2
I tried to fix this and it appears that if I immediately return from the overridden pytest_generate_tests this starts working just fine, including duplicating
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1049050234
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item
test_1.py .....
================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
=========================================================== 5 passed in 1.28s ============================================================
But I'm not sure how inside pytest_generate_tests to tell if a test has a @pytest.mark.parametrize decorator, I think the fix should be something like:
@pytest.hookimpl(tryfirst=True)
def pytest_generate_tests(self, metafunc):
"""For all true pytest tests use metafunc to add all the duplicates."""
if metafunc.has_parametrize_decorator???: # no idea how to test for this
return
fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.fixturenames.append(fixture_name)
metafunc.parametrize(
argnames=fixture_name,
argvalues=list(range(self.flake_runs)),
)
fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.function._pytest_duplicated = True
except for the weird collection report with minus values:
collected 2 items / 5 deselected / -3 selected
Also if I don't use the specific subtest it works too with the original code:
$ pytest --flake-finder --flake-runs=5 test_1.py
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3507953969
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 2 items
test_1.py ..........
================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
=========================================================== 10 passed in 1.29s ===========================================================
But I need to repeat a specific sub-test.
And I can't use pytest --flake-finder --flake-runs=5 test_1.py -k 1-2 either, because I have other tests that have the same params so it'd run other tests.
The clash happens between the specific:
test_1.py::test_flake[1-2]
and 2nd parametrize:
test_1.py::test_flake[1-1-2]
and that's why they don't match.
Thank you.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent