Git Product home page Git Product logo

pytest-plus's Introduction

PyTest Plus Plugin :: extends pytest functionality

PyPI version Python versions CI Python Black Code Style

This plugin aims to be used to host multiple basic pytest extensions that meet the following criteria:

  • Downgrade gracefully, meaning that if the plugin is removed, you will still be able to run pytest

PYTEST_REQPASS

If you define environment variable PYTEST_REQPASS=123 and at the end of the testing the number of passed tests is not exactly 123, pytest will return exit code 1.

This feature is aimed for CI usage in order to prevent accidental skipping of some tests. We do expect users to define this variable within their own CI job definitions. The number of tests executed is likely to be dependent on the CI job.

We discourage defining this inside places like tox.ini because when a developer runs tests, they are likely to endup running a different number of tests. Also, this feature makes no sense if you try to mention a specific test.

Avoiding duplicate test function names

While pytest allows users to have the same test function names in different files, that makes it harder to identify and copy/paste the test name in order to reproduce the failure locally. That is why this plugin forces its users to avoid having the same function name anywhere in the tested project.

You can disable this check by defining PYTEST_CHECK_TEST_DUPLICATE=0.

Avoiding problematic test identifiers

This plugin will raise errors when it encounters test IDs that are either too long or that contain unsafe characters. While pytest is very flexible in allowing a wide range of test IDs, using these does make development harder as it prevents people from doing a copy/paste with failed test and pasting in in their terminal to reproduce the failed test locally.

You can disable regex check by defining PYTEST_CHECK_TEST_ID_REGEX=0.

You can disable the length check by defining PYTEST_MAX_TEST_ID_LENGTH=0.

Prepare pytest log files for collection on CI

As pytest log files are created on temp directory and some CI systems refuse to collect files from outside the current project, we do copy these files inside $VIRTUAL_ENV/log, same directory used by tox itself. To collect the logs on Github Actions, you only need a step like:

- name: Archive logs
  uses: actions/upload-artifact@v4
  with:
    name: logs-${{ matrix.name }}.zip
    path: .tox/**/log/

Release process

Releases are triggered from GitHub Releases page.

Links

pytest-plus's People

Contributors

dependabot[bot] avatar ssbarnea avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

pytest-plus's Issues

RFE Support configuration via pyproject.toml

Pytest can be configured via the pyproject.toml file.

This plugin should allow for the same.

It is a little cumbersome to pass envvars to pytest without additional plugins.

(for configs like PYTEST_CHECK_TEST_DUPLICATE=0)

passed sumarry error gest in the middle of the console output

As seen on https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/ansible_41/2641/501baedbe757ee6523aa86e7a84031bc610d2bf3/check/molecule-tox-devel-functional/bd492b5/job-output.txt the error message is not the last thing printed before exiting pytest. The output after seems to be from stdout/stderr of executed commands.

I suspect that some buffers are not flushed in right order but I am not sure which should be the fix.

Minimum pass count

Currently only an exact number of tests passing is allowed.

When running tests on lots of arch and platforms and py versions, usually the number varies a bit.

Either a new envvar for minimum, or another "variance" envvar to determine +/- how many is allowable.

INTERNALERROR appears when running alongside pytest-flakefinder

I tested adding pytest-plus into the test suite for keyrings.envvars and it's failing when running alongside pytest-flakefinder.

nox > pytest --flake-finder --flake-runs=2 --numprocesses=auto --cov=src/keyrings/envvars/ --cov-report=xml '--log-format='"'"'%(asctime)s %(levelname)s %(message)s'"'"'' '--log-date-format='"'"'%Y-%m-%d %H:%M:%S'"'"'' --log-cli-level=INFO
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.12.1, pytest-7.4.4, pluggy-1.4.0
Using --randomly-seed=3431830401
rootdir: /home/devuan/work/public/keyrings.envvars
configfile: pyproject.toml
plugins: plus-0.6.1, antilru-1.1.1, randomly-3.15.0, xdist-3.5.0, flakefinder-1.1.0, cov-4.1.0
10 workers [80 items]     
scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/xdist/dsession.py", line 148, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/devuan/work/public/keyrings.envvars/.nox/pytest/lib/python3.12/site-packages/xdist/dsession.py", line 187, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('src/keyrings/envvars/tests/keyring_test.py::TestKeyring::test_get_invalid_service[0]', <WorkerController gw9>)
INTERNALERROR> assert not 'src/keyrings/envvars/tests/keyring_test.py::TestKeyring::test_get_invalid_service[0]'

================================================================================================================================= no tests ran in 1.10s ==================================================================================================================================
nox > Command pytest --flake-finder --flake-runs=2 --numprocesses=auto --cov=src/keyrings/envvars/ --cov-report=xml '--log-format='"'"'%(asctime)s %(levelname)s %(message)s'"'"'' '--log-date-format='"'"'%Y-%m-%d %H:%M:%S'"'"'' --log-cli-level=INFO failed with exit code 3
nox > Session pytest failed.

Is this configuration not supported by pytest-plus?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.