pytest-dev / pytest-xprocess Goto Github PK

View Code? Open in Web Editor NEW

96.0 7.0 20.0 228 KB

pytest external process plugin

License: MIT License

Python 100.00%

pytest-xprocess's Introduction

pytest-xprocess

A pytest plugin for managing external processes across test runs.

Installing

Install using pip:

pip install pytest-xprocess

Useful Links

Documentation: https://pytest-xprocess.readthedocs.io/
Changelog: https://pytest-xprocess.readthedocs.io/en/latest/changes.html
PyPI Releases: https://pypi.org/project/pytest-xprocess/
Source Code: https://github.com/pytest-dev/pytest-xprocess
Issue Tracker: https://github.com/pytest-dev/pytest-xprocess/issues/
Pytest IRC channel: ircs://irc.libera.chat:6697/#pytest
Pytest Discord Channel: https://discord.gg/k7F2ZFvJV3

pytest-xprocess's People

Contributors

Stargazers

Watchers

Forkers

danwos nicoddemus sneha-nd kmuszyn tsteinholz davidparsson northernsage pizzapanther python-repository-hub isabella232 federicober jezdez parnassius frenzymadness roehling jclerman sysfce2

pytest-xprocess's Issues

Pattern only reads first 50 lines of log

It took me forever to figure out why my process was not being detected, but it was due to my logs running in verbose mode, which outputs approx. 500 lines before actually running.

This meant that my server would run, however, the process would say it didn't and fail.

https://github.com/pytest-dev/pytest-xprocess/blob/master/xprocess.py#L197

I also attempted to override this function, but it seemed to prefer the super version... This would be really good to have configurable.

docs lack examples on how to use --xshow and --xkill command line utilities

Our documentation currently only briefly mentions the existence of these command line utilities, we could use some examples illustrating their use as well.

test_interruption_does_not_cleanup and test_functional_work_flow fails

Describe the bug
When I run tests with Python 3.9.15 two tests fails.

To Reproduce
Steps to reproduce the behavior:
tox-3.9 --current-env --no-provision --recreate -e py39

Expected behavior
All tests pass

Screenshots

______________________ test_interruption_does_not_cleanup ______________________

testdir = <Testdir local('/tmp/pytest-of-marcel/pytest-33/test_interruption_does_not_cleanup0')>
tcp_port = 53786

    def test_interruption_does_not_cleanup(testdir, tcp_port):
        server_path = Path(__file__).parent.joinpath("server.py").absolute()
        testdir.makepyfile(
            """
            import sys
            import socket
            from xprocess import ProcessStarter
    
            def test_servers_start(request, xprocess):
                port = %r
                server_path = %r
    
                class Starter(ProcessStarter):
                    pattern = "started"
                    args = [sys.executable, server_path, port]
    
                xprocess.ensure("server_test_interrupt_no_terminate", Starter)
    
                raise KeyboardInterrupt
            """
            % (tcp_port, str(server_path))
        )
        result = testdir.runpytest_subprocess()
        result.stdout.fnmatch_lines("*KeyboardInterrupt*")
        result = testdir.runpytest("--xshow")
>       result.stdout.fnmatch_lines("*LIVE*")
E       Failed: remains unmatched: '*LIVE*'

$(BUILDPATH)/tests/test_interruption_clean_up.py:58: Failed
----------------------------- Captured stdout call -----------------------------
running: /usr/bin/python3.9 -mpytest --basetemp=/tmp/pytest-of-marcel/pytest-33/test_interruption_does_not_cleanup0/runpytest-0
     in: /tmp/pytest-of-marcel/pytest-33/test_interruption_does_not_cleanup0
__________________________ test_functional_work_flow ___________________________

testdir = <Testdir local('/tmp/pytest-of-marcel/pytest-33/test_functional_work_flow0')>
tcp_port = 42620

    def test_functional_work_flow(testdir, tcp_port):
        server_path = Path(__file__).parent.joinpath("server.py").absolute()
        testdir.makepyfile(
            """
            import sys
            import socket
            from xprocess import ProcessStarter
    
            def test_server(request, xprocess):
                port = %r
                data = "spam\\n"
                server_path = %r
    
                class Starter(ProcessStarter):
                    pattern = "started"
                    max_read_lines = 200
                    args = [sys.executable, server_path, port]
    
                # required so test won't hang on pytest_unconfigure
                xprocess.proc_wait_timeout = 1
    
                xprocess.ensure("server_workflow_test", Starter)
    
                with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
                    sock.connect(("localhost", port))
                    sock.sendall(bytes(data, "utf-8"))
                    received = str(sock.recv(1024), "utf-8")
                    assert received == data.upper()
        """
            % (tcp_port, str(server_path))
        )
        result = testdir.runpytest()
>       result.stdout.fnmatch_lines("*1 passed*")
E       Failed: remains unmatched: '*1 passed*'

$(BUILDPATH)/tests/test_functional_workflow.py:36: Failed
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/vendor-packages/_pytest/pytester.py", line 1170, in runpytest_inprocess
    reprec = self.inline_run(*args, **kwargs)
  File "/usr/lib/python3.9/vendor-packages/_pytest/pytester.py", line 1135, in inline_run
    ret = main([str(x) for x in args], plugins=plugins)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/lib/python3.9/vendor-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/lib/python3.9/vendor-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/lib/python3.9/vendor-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/usr/lib/python3.9/vendor-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/lib/python3.9/vendor-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3.9/vendor-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 1229, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/usr/lib/python3.9/vendor-packages/pluggy/_manager.py", line 288, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 489, in register
    ret: Optional[str] = super().register(plugin, name)
  File "/usr/lib/python3.9/vendor-packages/pluggy/_manager.py", line 103, in register
    hookimpl_opts = self.parse_hookimpl_opts(plugin, name)
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 460, in parse_hookimpl_opts
    return _get_legacy_hook_marks(
  File "/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py", line 374, in _get_legacy_hook_marks
    warn_explicit_for(cast(FunctionType, method), message)
  File "/usr/lib/python3.9/vendor-packages/_pytest/warning_types.py", line 171, in warn_explicit_for
    raise type(w)(f"{w}\n at {filename}:{lineno}") from None
pytest.PytestDeprecationWarning: The hookimpl pytest_runtest_makereport uses old-style configuration options (marks or attributes).
Please use the pytest.hookimpl(hookwrapper=True) decorator instead
 to configure the hooks.
 See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers
$(INSTALLPATH)/usr/lib/python3.9/vendor-packages/xprocess/pytest_xprocess.py:48

Environment (please complete the following information):

OS: OpenIndiana
Python Version 3.9.15
pytest-xprocess version 0.20.0

Additional context
NA

IsADirectoryError on open log file

Hi. After upgrading from 0.17.1 to 0.22.2, i got following error

  File "/Users/20010116/.pyenv/versions/3.9.17/envs/venv-ddiapi2-py39/lib/python3.9/site-packages/xprocess/pytest_xprocess.py", line 59, in xprocess
    open(log_file, errors="surrogateescape").close()
IsADirectoryError: [Errno 21] Is a directory: '/Users/20010116/Documents/Workspace/ddi/infrasvc-ddi-api/.pytest_cache/d/.xprocess/consumer_start_consumer/log'

Digging the error, i found that get_log_files list all objects in root directory, including directory (i have one that is named "log")
I suggest to add a condition to verify if object is a file

def get_log_files(root_dir):
    proc_dirs = [f.path for f in os.scandir(root_dir) if f.is_dir()]
    return [
        os.path.join(proc_dir, f)
        for proc_dir in proc_dirs
        for f in os.listdir(proc_dir)
        if f.endswith("log") and os.path.isfile(os.path.join(proc_dir, f))
    ]

Make _checkpattern(count=50) configureable

The wait_pattern seems to be checked only for the first 50 lines of process stdout.
This is really not much during test runs, where logs are mostly configured to run on DEBUG level.

The hard coded value count=50 from https://github.com/pytest-dev/pytest-xprocess/blob/master/xprocess.py#L149
should be configurable via a parameter inside the ensure function:

def ensure(self, name, preparefunc, restart=False, count=50):
    ...
    if not callable(wait):
                check = lambda: self._checkpattern(f, wait, count)
            else:
                check = wait
            if check():
                self.log.debug("%s process startup detected", name)
            else:
                raise RuntimeError("Could not start process %s" % name)

Beside this, everything works like a charm.

report can fail to read log file as text

The stdout file passed to Popen in xprocess.ensure is opened in binary mode (a+b). However, the file is also opened in text read mode with the system encoding for xresource.fhandle, which is then read by pytest_runtest_makereport. If the process writes junk binary data to the stream, then reading the data can result in UnicodeDecodeError.

pytest-xprocess/xprocess/xprocess.py

Line 235 in 2dd5b9f

stdout = open(str(info.logpath), "a+b", 0)

pytest-xprocess/xprocess/xprocess.py

Line 271 in 2dd5b9f

xresource.fhandle = info.logpath.open()

pytest-xprocess/xprocess/pytest_xprocess.py

Line 70 in 5b59835

content = logfiles[name].read()

Just upgraded to 0.21 from 0.20 for Werkzeug and started getting this issue. Here's an example of the error: https://github.com/pallets/werkzeug/actions/runs/3803459746/jobs/6469880268#step:7:595

The file should probably be opened with errors="replace", or some other check should be used to avoid bytes/text issues.

Spurious ResourceWarnings about unterminated processes

Description
A ResourceWarning is triggered during tests using fixtures that start an xprocess. This seems like it might be a indication that Popen isn't being treated correctly and results in spurious user-facing errors when telling pytest to escalate warnings to errors.

To Reproduce

tests/conftest.py

from pytest import fixture
from xprocess import ProcessStarter


@fixture # using (scope='session') results in similar behaviour but only for the first test to use the fixture
def example(xprocess):
    class Starter(ProcessStarter):
        pattern = "foo"
        args = ('sh', '-c', 'echo foo; sleep 10; echo bar')
    log = xprocess.ensure('example', Starter)
    yield
    xprocess.getinfo('example').terminate()

tests/pytest.ini

[pytest]
filterwarnings =
    error

tests/test_example.py

def test_hello(example):
    pass

def test_world(example):
    pass

python3 -m pytest tests/test_example.py

Expected behaviour
No ResourceWarning: subprocess X is still running should occur until after the expected process termination has failed, at the end of the fixture's scope.

Screenshots

>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Popen.__del__ at 0x7f871d81d430>
E
E               Traceback (most recent call last):
E                 File "/usr/local/lib/python3.8/subprocess.py", line 942, in __del__
E                   _warn("subprocess %s is still running" % self.pid,
E               ResourceWarning: subprocess 376 is still running

/usr/local/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E
E               Traceback (most recent call last):
E                 File "/usr/local/lib/python3.8/site-packages/xprocess.py", line 164, in ensure
E                   logfiles[name] = f
E               ResourceWarning: unclosed file <_io.TextIOWrapper name='/src/tests/.pytest_cache/d/.xprocess/example/xprocess.log' mode='r' encoding='UTF-8'>

Environment (please complete the following information):

OS: Ubuntu 20.04
Python Version: 3.8.6
pytest-xprocess: version 0.17.0

Additional context

platform linux -- Python 3.8.6, pytest-6.2.0, py-1.10.0, pluggy-0.13.1                                                                                                                                             
rootdir: /src/tests, configfile: pytest.ini                                                                                                                                                                        
plugins: xprocess-0.17.0

Support starting only once with xdist

Is your feature request related to a problem? Please describe.
It is not uncommon wanting to start and stop a global service during a pytest run. Often test runs are also parallelized using pytest-xdist.

Describe the solution you'd like
Possibility to start a process when the fixture is first needed, and destruct when the fixture is not in use anymore at all. Both with single worker and multiple (local) workers.

Describe alternatives you've considered
Using https://pytest-xdist.readthedocs.io/en/stable/how-to.html#xdist.is_xdist_master separately or together for this purpose does not seem to work.

An alternative is another entry point that then starts pytest.

Additional context
Add any other context or screenshots about the feature request here.

TypeError: Can't instantiate abstract class Starter with abstract methods pattern

Describe the bug
When not using the pattern parameter, this error is returned:

TypeError: Can't instantiate abstract class Starter with abstract methods pattern

To Reproduce
Steps to reproduce the behavior:

Duplicate the Quickstart code, but comment out the pattern parameter.
see: https://pytest-xprocess.readthedocs.io/en/latest/

Expected behavior
The documentation does not make it clear that the pattern parameter is required. Also the error message produced is not very informative.

Personally, I would perfer to use the startup_check function to check that my server is working rather than pattern. Perhaps one of the parameters timeout, pattern, or startup_check should be required instead of only pattern being required?

Screenshots
NA

Environment (please complete the following information):
Windows 11

Additional context
NA

Lack of dedicated documentation page

Our README.rst is starting to get quite large so I believe it would be nice to start working on dedicated doc pages for pytest-xprocess (probably using something like Sphinx). I'll try to get started on this in the next few days. I'm opening the issue for documenting the process and in case anyone has suggestions or want to help.

Flaky test behavior introduced for py38

As title says, we are now getting tests fails related to proper process termination/cleanup ofr python 3.8. Traceback bellow:

________________________ test_interruption_cleanup __________________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x000002B0B6A17B80>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\_pytest\runner.py:338: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\_pytest\runner.py:259: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\pluggy\_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\pluggy\_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\_pytest\unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Popen.__del__ at 0x000002B0B66CE280>
E               
E               Traceback (most recent call last):
E                 File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 946, in __del__
E                   _warn("subprocess %s is still running" % self.pid,
E               ResourceWarning: subprocess 5[68](https://github.com/pytest-dev/pytest-xprocess/runs/6564442440?check_suite_focus=true#step:5:69)8 is still running

D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\lib\site-packages\_pytest\unraisableexception.py:78: PytestUnraisableExceptionWarning
---------------------------- Captured stdout call -----------------------------
running: D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\Scripts\python.EXE -mpytest --basetemp=C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_interruption_cleanup0\runpytest-0
     in: C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_interruption_cleanup0
============================= test session starts =============================
platform win32 -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_interruption_cleanup0
plugins: xprocess-0.1.dev1+g3c65e4b
collected 1 item

test_interruption_cleanup.py 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_interruption_cleanup0\test_interruption_cleanup.py:16: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
============================ no tests ran in 0.49s ============================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
10[76](https://github.com/pytest-dev/pytest-xprocess/runs/6564442440?check_suite_focus=true#step:5:77) server_test_interrupt DEAD C:\Users\runneradmin\AppData\Local\Temp\pytest-of-unknown\pytest-0\test_interruption_cleanup0\.pytest_cache\d\.xprocess\server_test_interrupt\xprocess.log
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
=========================== short test summary info ===========================
FAILED test_interruption_clean_up.py::test_interruption_cleanup - pytest.Pyte...
================== 1 failed, 41 passed, 3 skipped in 32.46s ===================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
ERROR: InvocationError for command 'D:\a\pytest-xprocess\pytest-xprocess\.tox\py38\Scripts\coverage.EXE' run -m pytest -v (exited with code 1)
___________________________________ summary ___________________________________
ERROR:   py38: commands failed

RFE: is it possible to start making github releases?🤔

On create github release entry is created email notification to those whom have set in your repo the web UI Watch->Releases.
gh release can contain additional comments (li changelog) or additional assets like release tar balls (by default it contains only assets from git tag) however all those part are not obligatory.
In simplest variant gh release can be empty because subiekt of the sent email contains git tag name.

I'm asking because my automation process uses those email notifications by trying to make preliminary automated upgrades of building packages, which allows saving some time on maintaining packaging procedures.
Probably other people may be interested to be instantly informed about release new version as well.

Documentation and examples of generate gh releases:
https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
https://cli.github.com/manual/gh_release_upload/
jbms/sphinx-immaterial#282
https://github.com/marketplace/actions/github-release
https://pgjones.dev/blog/trusted-plublishing-2023/
jbms/sphinx-immaterial#281 (comment)
tox target to publish on pypi and make gh release https://github.com/jaraco/skeleton/blob/928e9a86d61d3a660948bcba7689f90216cc8243/tox.ini#L42-L58

py.test fails on packaging in chroot

Hi!
I'm currently moving python{,2}-pytest-xprocess as a package to Arch Linux's [community] repository.
Packages are built in a clean chroot to ensure integrity and dependency completeness.

However, with this package, I'm having trouble running the tests:

============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1, inifile:
collected 6 items

example/test_server.py EEEEEF                                            [100%]

==================================== ERRORS ====================================
________________________ ERROR at setup of test_server _________________________
file /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py, line 12
  def test_server(xprocess):
E       fixture 'xprocess' not found
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:12
________________________ ERROR at setup of test_server2 ________________________
file /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py, line 23
  def test_server2(xprocess):
E       fixture 'xprocess' not found
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:23
______________________ ERROR at setup of test_server_env _______________________
file /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py, line 34
  def test_server_env(xprocess):
E       fixture 'xprocess' not found
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:34
  def test_server_env(xprocess):                                                           
E       fixture 'xprocess' not found                                                     
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.                                           
                                                                                                                 
/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:34                
_______________________ ERROR at setup of test_shutdown ________________________       
file /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py, line 48
  def test_shutdown(xprocess):                                                     
E       fixture 'xprocess' not found                                                       
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.
                
/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:48
____________________ ERROR at setup of test_shutdown_legacy ____________________
file /build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py, line 54
  def test_shutdown_legacy(xprocess):
E       fixture 'xprocess' not found
>       available fixtures: LineMatcher, _pytest, anypython, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, linecomp, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, testdir, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.
                  
/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:54
=================================== FAILURES ===================================
__________________________ test_functional_work_flow ___________________________
                                                                                
testdir = <Testdir local('/tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0')>
                                                                                  
    def test_functional_work_flow(testdir):
        testdir.makepyfile("""
            import sys                                                         
            def test_server(request, xprocess):
                xprocess.ensure("server", lambda cwd:                           
                    ("started", [sys.executable, %r, 6700]))                    
                import socket                                                                       
                sock = socket.socket()
                sock.connect(("localhost", 6700))
                sock.sendall("world\\n".encode("utf8"))                                                                                                                                                                                                          
                c = sock.recv(1)                            
                assert c == "1".encode("utf8")
        """ % str(server_path))                                                          
        result = testdir.runpytest()                                            
>       result.stdout.fnmatch_lines("*1 passed*")                                                   
                             
/build/python-pytest-xprocess/src/python-pytest-xprocess-0.12.1/example/test_server.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                                                                                                                  
                                                                                         
self = <_pytest.pytester.LineMatcher object at 0x6c67b8a195c0>                         
lines2 = '*1 passed*'                                                                               
                                                                                           
    def fnmatch_lines(self, lines2):                                                     
        """Search captured text for matching lines using ``fnmatch.fnmatch``.                                                                                                                                                                                    
                                                                                                       
            The argument is a list of lines which have to match and can use glob                                 
            wildcards.  If they do not match a pytest.fail() is called.  The                             
            matches and non-matches are also printed on stdout.                        
                                                                                                    
            """                                                                    
>       self._match_lines(lines2, fnmatch, 'fnmatch')                                      
E       Failed: nomatch: '*1 passed*'                                                                                                                                                                                                                            
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.6.4, pytest-3.4.0, py-1.5.2, pluggy-0.6.0'
E           and: 'rootdir: /tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0, inifile:'
E           and: 'collected 1 item'                                             
E           and: ''                                                                                 
E           and: 'test_functional_work_flow.py E                                           [100%]'
E           and: ''                 
E           and: '==================================== ERRORS ===================================='                                                                                                                                                              
E           and: '________________________ ERROR at setup of test_server _________________________'
E           and: 'file /tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0/test_functional_work_flow.py, line 2'
E           and: '  def test_server(request, xprocess):'                                 
E           and: "E       fixture 'xprocess' not found"                         
E           and: '>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, tmpdir, tmpdir_factory'
E           and: ">       use 'pytest --fixtures [testpath]' for help on them." 
E           and: ''                                                                      
E           and: '/tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0/test_functional_work_flow.py:2'
E           and: '=========================== 1 error in 0.02 seconds ============================'
E           and: ''           
E       remains unmatched: '*1 passed*'                                        
                                               
/usr/lib/python3.6/site-packages/_pytest/pytester.py:1175: Failed               
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================                    
platform linux -- Python 3.6.4, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0, inifile:
collected 1 item                                                                                                                                                                                                                                                 
                                                            
test_functional_work_flow.py E                                           [100%]
                                                                                         
==================================== ERRORS ====================================
________________________ ERROR at setup of test_server _________________________                    
file /tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0/test_functional_work_flow.py, line 2
  def test_server(request, xprocess):                                                     
E       fixture 'xprocess' not found                                                                                                                                                                                                                             
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_xml_attribute, record_xml_property, recwarn, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.  
                                                                                         
/tmp/pytest-of-builduser/pytest-0/test_functional_work_flow0/test_functional_work_flow.py:2
=========================== 1 error in 0.02 seconds ============================                    
====================== 1 failed, 5 error in 0.12 seconds =======================

Before running py.test{,2} I'm exporting the absolute build path to PYTHONPATH, so in theory xprocess should be found... but it isn't.

Maybe you can offer some help? I'd like to use the tests as well!

xprocess fails with UnicodeDecodeError:

Using python3 and pytest-xprocess (pytest_xprocess-0.12.1-py2.py3-none-any.whl), when pytest-xprocess launches a process that produces logs containing characters such as 'Ê��pP��requestsX�P��address', it fails with the following error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/main.py", line 169, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/runner.py", line 68, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/runner.py", line 76, in runtestprotocol
INTERNALERROR>     rep = call_and_report(item, "setup", log)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/runner.py", line 164, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/hypothesis/extra/pytestplugin.py", line 108, in pytest_runtest_makereport
INTERNALERROR>     report = (yield).get_result()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/usr/local/lib/python3.4/dist-packages/pytest_xprocess.py", line 55, in pytest_runtest_makereport
INTERNALERROR>     content = logfiles[name].read()
INTERNALERROR>   File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
INTERNALERROR>     return codecs.ascii_decode(input, self.errors)[0]
INTERNALERROR> UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 34649: ordinal not in range(128)

How to hide output while using the pattern option

Is there a way to hide all the ouput while using the pattern option? I've tried this:

class Starter(ProcessStarter):
    env = {"PYTHONUNBUFFERED": "1", **os.environ}
    pattern = BOOT_MSG
    popen_kwargs = {
        "stdout": subprocess.DEVNULL,
        "stderr": subprocess.STDOUT,
    }

but it breaks the pattern check :)

Implicit dependency on `py` causes failure of pytest==7.2.0 runs when xprocess is installed

Describe the bug
Missing dependency on py causes failure of pytest==7.2.0 runs when xprocess is installed.

To Reproduce
From a clean environment (say using Python 3.9; this issue affects at least python 3.8 and 3.9):

% pip install pytest==7.2.0
% pip install pytest-xprocess==0.20.0
% python
>>> import xprocess

Expected behavior
No error should be raised.

Actual result

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/xprocess/__init__.py", line 1, in <module>
    from .xprocess import ProcessStarter
  File "/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/xprocess/xprocess.py", line 13, in <module>
    from py import std
ImportError: cannot import name 'std' from 'py' (/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/py.py)
>>> from xprocess import ProcessStarter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/xprocess/__init__.py", line 1, in <module>
    from .xprocess import ProcessStarter
  File "/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/xprocess/xprocess.py", line 13, in <module>
    from py import std
ImportError: cannot import name 'std' from 'py' (/Users/***/Library/Caches/pypoetry/virtualenvs/mypackage-l0EsK_Ku-py3.9/lib/python3.9/site-packages/py.py)

Environment:

OS: macOS Monterey
Python Version 3.9.13
pytest-xprocess version: 0.20.0 (also tested with 0.18.1)

Additional context

This problem only occurs with pytest 7.2.0, which was just released. The issue seems to be that pytest has dropped dependency on the py package, which is no longer included when pytest is installed. pytest-xprocess has an implicit dependency on py but does not name it as a dependency in setup.py, so the import statement in xprocess.py fails.

Related pytest PR: pytest-dev/pytest#10396

Test suite fails because of Zombie processes with Python 3.11

With Python 3.11, the test suite will fail to terminate some server.py instances because they left behind zombie children, which will be returned by psutil.wait_procs as "alive" and thereby cause XProcessInfo.terminate to return failure.

To Reproduce

Run the test suite
Observe failing tests as detailed below

Expected behavior
No test failures

Environment (please complete the following information):

OS: [Debian unstable]
Python Version [3.11.1]
pytest-xprocess version [0.21.0]

Additional context

============================= test session starts ==============================
platform linux -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0+repack -- /usr/bin/python3.11
cachedir: .pytest_cache
rootdir: /<<PKGBUILDDIR>>, configfile: tox.ini
plugins: xprocess-0.21.0
collecting ... collected 45 items / 2 deselected / 43 selected

tests/test_callback.py::test_callback_success[s1] PASSED                 [  2%]
tests/test_callback.py::test_callback_success[s2] PASSED                 [  4%]
tests/test_callback.py::test_callback_success[s3] PASSED                 [  6%]
tests/test_callback.py::test_callback_fail[s1] PASSED                    [  9%]
tests/test_callback.py::test_callback_fail[s2] PASSED                    [ 11%]
tests/test_callback.py::test_callback_fail[s3] PASSED                    [ 13%]
tests/test_functional_workflow.py::test_functional_work_flow FAILED      [ 16%]
tests/test_process_initialization.py::test_servers_start[s1] PASSED      [ 18%]
tests/test_process_initialization.py::test_servers_start[s2] PASSED      [ 20%]
tests/test_process_initialization.py::test_servers_start[s3] PASSED      [ 23%]
tests/test_process_initialization.py::test_ensure_not_restart[s1] PASSED [ 25%]
tests/test_process_initialization.py::test_ensure_not_restart[s2] PASSED [ 27%]
tests/test_process_initialization.py::test_ensure_not_restart[s3] PASSED [ 30%]
tests/test_process_initialization.py::test_startup_detection_max_read_lines[s1-started-20] PASSED [ 32%]
tests/test_process_initialization.py::test_startup_detection_max_read_lines[s2-spam, bacon, eggs-30] PASSED [ 34%]
tests/test_process_initialization.py::test_startup_detection_max_read_lines[s3-finally started-62] PASSED [ 37%]
tests/test_process_initialization.py::test_runtime_error_on_start_fail[s1] PASSED [ 39%]
tests/test_process_initialization.py::test_runtime_error_on_start_fail[s2] PASSED [ 41%]
tests/test_process_initialization.py::test_runtime_error_on_start_fail[s3] PASSED [ 44%]
tests/test_process_initialization.py::test_popen_kwargs[s1] PASSED       [ 46%]
tests/test_process_initialization.py::test_popen_kwargs[s2] PASSED       [ 48%]
tests/test_process_initialization.py::test_popen_kwargs[s3] PASSED       [ 51%]
tests/test_process_termination.py::test_clean_shutdown[s1] FAILED        [ 53%]
tests/test_process_termination.py::test_clean_shutdown[s2] FAILED        [ 55%]
tests/test_process_termination.py::test_clean_shutdown[s3] FAILED        [ 58%]
tests/test_process_termination.py::test_terminate_no_pid[s1] PASSED      [ 60%]
tests/test_process_termination.py::test_terminate_no_pid[s2] PASSED      [ 62%]
tests/test_process_termination.py::test_terminate_no_pid[s3] PASSED      [ 65%]
tests/test_process_termination.py::test_terminate_only_parent[s1] PASSED [ 67%]
tests/test_process_termination.py::test_terminate_only_parent[s2] PASSED [ 69%]
tests/test_process_termination.py::test_terminate_only_parent[s3] PASSED [ 72%]
tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s1] FAILED [ 74%]
tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s2] FAILED [ 76%]
tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s3] FAILED [ 79%]
tests/test_process_termination.py::test_return_value_on_failure[s1] PASSED [ 81%]
tests/test_process_termination.py::test_return_value_on_failure[s2] PASSED [ 83%]
tests/test_process_termination.py::test_return_value_on_failure[s3] PASSED [ 86%]
tests/test_resource_cleanup.py::test_0 PASSED                            [ 88%]
tests/test_resource_cleanup.py::test_1 PASSED                            [ 90%]
tests/test_resource_cleanup.py::test_2 PASSED                            [ 93%]
tests/test_startup_timeout.py::test_timeout_raise_exception[s1] PASSED   [ 95%]
tests/test_startup_timeout.py::test_timeout_raise_exception[s2] PASSED   [ 97%]
tests/test_startup_timeout.py::test_timeout_raise_exception[s3] PASSED   [100%]

=================================== FAILURES ===================================
__________________________ test_functional_work_flow ___________________________

testdir = <Testdir local('/tmp/pytest-of-roehling/pytest-0/test_functional_work_flow0')>
tcp_port = 47033

    def test_functional_work_flow(testdir, tcp_port):
        server_path = Path(__file__).parent.joinpath("server.py").absolute()
        testdir.makepyfile(
            """
            import sys
            import socket
            from xprocess import ProcessStarter
    
            def test_server(request, xprocess):
                port = %r
                data = "spam\\n"
                server_path = %r
    
                class Starter(ProcessStarter):
                    pattern = "started"
                    max_read_lines = 200
                    args = [sys.executable, server_path, port]
    
                # required so test won't hang on pytest_unconfigure
                xprocess.proc_wait_timeout = 1
    
                xprocess.ensure("server_workflow_test", Starter)
    
                with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
                    sock.connect(("localhost", port))
                    sock.sendall(bytes(data, "utf-8"))
                    received = str(sock.recv(1024), "utf-8")
                    assert received == data.upper()
        """
            % (tcp_port, str(server_path))
        )
        result = testdir.runpytest()
        result.stdout.fnmatch_lines("*1 passed*")
        result = testdir.runpytest("--xshow")
        result.stdout.fnmatch_lines("*LIVE*")
        result = testdir.runpytest("--xkill")
>       result.stdout.fnmatch_lines("*TERMINATED*")
E       Failed: nomatch: '*TERMINATED*'
E           and: "could not terminated process [psutil.Process(pid=175, name='python3.11', status='zombie', started='20:21:21'), psutil.Process(pid=176, name='python3.11', status='zombie', started='20:21:21'), psutil.Process(pid=177, name='python3.11', status='zombie', started='20:21:21')]"
E           and: '174 server_workflow_test FAILED TO TERMINATE'
E       remains unmatched: '*TERMINATED*'

/<<PKGBUILDDIR>>/tests/test_functional_workflow.py:40: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0+repack
rootdir: /tmp/pytest-of-roehling/pytest-0/test_functional_work_flow0
plugins: xprocess-0.21.0
collected 1 item

test_functional_work_flow.py .                                           [100%]

============================== 1 passed in 0.23s ===============================
174 server_workflow_test LIVE /tmp/pytest-of-roehling/pytest-0/test_functional_work_flow0/.pytest_cache/d/.xprocess/server_workflow_test/xprocess.log
could not terminated process [psutil.Process(pid=175, name='python3.11', status='zombie', started='20:21:21'), psutil.Process(pid=176, name='python3.11', status='zombie', started='20:21:21'), psutil.Process(pid=177, name='python3.11', status='zombie', started='20:21:21')]
174 server_workflow_test FAILED TO TERMINATE
___________________________ test_clean_shutdown[s1] ____________________________

tcp_port = 47401, proc_name = 's1'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_clean_shutdown(tcp_port, proc_name, xprocess):
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        assert info.isrunning()
        children = psutil.Process(info.pid).children()
>       assert info.terminate() == 1
E       assert -1 == 1
E        +  where -1 = <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f369090b310>>()
E        +    where <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f369090b310>> = <xprocess.xprocess.XProcessInfo object at 0x7f369090b310>.terminate

tests/test_process_termination.py:22: AssertionError
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s1$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 47401
process 's1' started pid=193
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s1 process startup detected
could not terminated process [psutil.Process(pid=194, name='python3.11', status='zombie', started='20:22:05'), psutil.Process(pid=195, name='python3.11', status='zombie', started='20:22:05'), psutil.Process(pid=196, name='python3.11', status='zombie', started='20:22:05')]
___________________________ test_clean_shutdown[s2] ____________________________

tcp_port = 44317, proc_name = 's2'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_clean_shutdown(tcp_port, proc_name, xprocess):
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        assert info.isrunning()
        children = psutil.Process(info.pid).children()
>       assert info.terminate() == 1
E       assert -1 == 1
E        +  where -1 = <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f3690908250>>()
E        +    where <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f3690908250>> = <xprocess.xprocess.XProcessInfo object at 0x7f3690908250>.terminate

tests/test_process_termination.py:22: AssertionError
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s2$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 44317
process 's2' started pid=197
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s2 process startup detected
could not terminated process [psutil.Process(pid=199, name='python3.11', status='zombie', started='20:22:46'), psutil.Process(pid=200, name='python3.11', status='zombie', started='20:22:46'), psutil.Process(pid=198, name='python3.11', status='zombie', started='20:22:46')]
___________________________ test_clean_shutdown[s3] ____________________________

tcp_port = 37985, proc_name = 's3'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_clean_shutdown(tcp_port, proc_name, xprocess):
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        assert info.isrunning()
        children = psutil.Process(info.pid).children()
>       assert info.terminate() == 1
E       assert -1 == 1
E        +  where -1 = <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f3690903bd0>>()
E        +    where <bound method XProcessInfo.terminate of <xprocess.xprocess.XProcessInfo object at 0x7f3690903bd0>> = <xprocess.xprocess.XProcessInfo object at 0x7f3690903bd0>.terminate

tests/test_process_termination.py:22: AssertionError
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s3$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 37985
process 's3' started pid=201
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s3 process startup detected
could not terminated process [psutil.Process(pid=204, name='python3.11', status='zombie', started='20:23:26'), psutil.Process(pid=202, name='python3.11', status='zombie', started='20:23:26'), psutil.Process(pid=203, name='python3.11', status='zombie', started='20:23:26')]
____________________ test_sigkill_after_failed_sigterm[s1] _____________________

self = <psutil._pslinux.Process object at 0x7f36908f34c0>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
>           ret = self._cache[fun]
E           AttributeError: 'Process' object has no attribute '_cache'

/usr/lib/python3/dist-packages/psutil/_common.py:443: AttributeError

During handling of the above exception, another exception occurred:

self = <psutil._pslinux.Process object at 0x7f36908f34c0>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f34c0>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
            ret = self._cache[fun]
        except AttributeError:
            # case 2: we never entered oneshot() ctx
>           return fun(self)

/usr/lib/python3/dist-packages/psutil/_common.py:446: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f34c0>

    @wrap_exceptions
    @memoize_when_activated
    def _parse_stat_file(self):
        """Parse /proc/{pid}/stat file and return a dict with various
        process info.
        Using "man proc" as a reference: where "man proc" refers to
        position N always subtract 3 (e.g ppid position 4 in
        'man proc' == position 1 in here).
        The return value is cached in case oneshot() ctx manager is
        in use.
        """
>       data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1687: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/229/stat', fallback = <object object at 0x7f36917e8d80>

    def bcat(fname, fallback=_DEFAULT):
        """Same as above but opens file in binary mode."""
>       return cat(fname, fallback=fallback, _open=open_binary)

/usr/lib/python3/dist-packages/psutil/_common.py:776: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/229/stat', fallback = <object object at 0x7f36917e8d80>
_open = <function open_binary at 0x7f3690af3a60>

    def cat(fname, fallback=_DEFAULT, _open=open_text):
        """Read entire file content and return it as a string. File is
        opened in text mode. If specified, `fallback` is the value
        returned in case of error, either if the file does not exist or
        it can't be read().
        """
        if fallback is _DEFAULT:
>           with _open(fname) as f:

/usr/lib/python3/dist-packages/psutil/_common.py:764: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/229/stat'

    def open_binary(fname):
>       return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
E       FileNotFoundError: [Errno 2] No such file or directory: '/proc/229/stat'

/usr/lib/python3/dist-packages/psutil/_common.py:728: FileNotFoundError

During handling of the above exception, another exception occurred:

self = psutil.Process(pid=229, status='terminated'), pid = 229
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
>           self.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=229, status='terminated')

    def create_time(self):
        """The process creation time as a floating point number
        expressed in seconds since the epoch.
        The return value is cached after first call.
        """
        if self._create_time is None:
>           self._create_time = self._proc.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:714: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f34c0>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f34c0>

    @wrap_exceptions
    def create_time(self):
>       ctime = float(self._parse_stat_file()['create_time'])

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1855: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f34c0>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
            return fun(self, *args, **kwargs)
        except PermissionError:
            raise AccessDenied(self.pid, self._name)
        except ProcessLookupError:
            raise NoSuchProcess(self.pid, self._name)
        except FileNotFoundError:
            if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):
>               raise NoSuchProcess(self.pid, self._name)
E               psutil.NoSuchProcess: process no longer exists (pid=229)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1652: NoSuchProcess

During handling of the above exception, another exception occurred:

tcp_port = 39609, proc_name = 's1'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.skipif(
        sys.platform == "win32",
        reason="on windows SIGTERM is treated as an alias for kill()",
    )
    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_sigkill_after_failed_sigterm(tcp_port, proc_name, xprocess):
        # explicitly tell xprocess_starter fixture to make
        # server instance ignore SIGTERM
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port, "--ignore-sigterm"]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        # since terminate with sigterm will fail, set a lower
        # timeout before sending sigkill so tests won't take too long
>       assert (
            info.terminate(timeout=2) == 1
            or psutil.Process(info.pid).status() == psutil.STATUS_ZOMBIE
        )

tests/test_process_termination.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/psutil/__init__.py:332: in __init__
    self._init(pid)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=229, status='terminated'), pid = 229
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
            self.create_time()
        except AccessDenied:
            # We should never get here as AFAIK we're able to get
            # process creation time on all platforms even as a
            # limited user.
            pass
        except ZombieProcess:
            # Zombies can still be queried by this class (although
            # not always) and pids() return them so just go on.
            pass
        except NoSuchProcess:
            if not _ignore_nsp:
>               raise NoSuchProcess(pid, msg='process PID not found')
E               psutil.NoSuchProcess: process PID not found (pid=229)

/usr/lib/python3/dist-packages/psutil/__init__.py:373: NoSuchProcess
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s1$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 39609 --ignore-sigterm
process 's1' started pid=229
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s1 process startup detected
could not terminated process [psutil.Process(pid=232, name='python3.11', status='zombie', started='20:26:08'), psutil.Process(pid=230, name='python3.11', status='zombie', started='20:26:08'), psutil.Process(pid=231, name='python3.11', status='zombie', started='20:26:08')]
____________________ test_sigkill_after_failed_sigterm[s2] _____________________

self = <psutil._pslinux.Process object at 0x7f36908f1990>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
>           ret = self._cache[fun]
E           AttributeError: 'Process' object has no attribute '_cache'

/usr/lib/python3/dist-packages/psutil/_common.py:443: AttributeError

During handling of the above exception, another exception occurred:

self = <psutil._pslinux.Process object at 0x7f36908f1990>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f1990>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
            ret = self._cache[fun]
        except AttributeError:
            # case 2: we never entered oneshot() ctx
>           return fun(self)

/usr/lib/python3/dist-packages/psutil/_common.py:446: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f1990>

    @wrap_exceptions
    @memoize_when_activated
    def _parse_stat_file(self):
        """Parse /proc/{pid}/stat file and return a dict with various
        process info.
        Using "man proc" as a reference: where "man proc" refers to
        position N always subtract 3 (e.g ppid position 4 in
        'man proc' == position 1 in here).
        The return value is cached in case oneshot() ctx manager is
        in use.
        """
>       data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1687: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/233/stat', fallback = <object object at 0x7f36917e8d80>

    def bcat(fname, fallback=_DEFAULT):
        """Same as above but opens file in binary mode."""
>       return cat(fname, fallback=fallback, _open=open_binary)

/usr/lib/python3/dist-packages/psutil/_common.py:776: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/233/stat', fallback = <object object at 0x7f36917e8d80>
_open = <function open_binary at 0x7f3690af3a60>

    def cat(fname, fallback=_DEFAULT, _open=open_text):
        """Read entire file content and return it as a string. File is
        opened in text mode. If specified, `fallback` is the value
        returned in case of error, either if the file does not exist or
        it can't be read().
        """
        if fallback is _DEFAULT:
>           with _open(fname) as f:

/usr/lib/python3/dist-packages/psutil/_common.py:764: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/233/stat'

    def open_binary(fname):
>       return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
E       FileNotFoundError: [Errno 2] No such file or directory: '/proc/233/stat'

/usr/lib/python3/dist-packages/psutil/_common.py:728: FileNotFoundError

During handling of the above exception, another exception occurred:

self = psutil.Process(pid=233, status='terminated'), pid = 233
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
>           self.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=233, status='terminated')

    def create_time(self):
        """The process creation time as a floating point number
        expressed in seconds since the epoch.
        The return value is cached after first call.
        """
        if self._create_time is None:
>           self._create_time = self._proc.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:714: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f1990>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f1990>

    @wrap_exceptions
    def create_time(self):
>       ctime = float(self._parse_stat_file()['create_time'])

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1855: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36908f1990>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
            return fun(self, *args, **kwargs)
        except PermissionError:
            raise AccessDenied(self.pid, self._name)
        except ProcessLookupError:
            raise NoSuchProcess(self.pid, self._name)
        except FileNotFoundError:
            if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):
>               raise NoSuchProcess(self.pid, self._name)
E               psutil.NoSuchProcess: process no longer exists (pid=233)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1652: NoSuchProcess

During handling of the above exception, another exception occurred:

tcp_port = 57689, proc_name = 's2'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.skipif(
        sys.platform == "win32",
        reason="on windows SIGTERM is treated as an alias for kill()",
    )
    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_sigkill_after_failed_sigterm(tcp_port, proc_name, xprocess):
        # explicitly tell xprocess_starter fixture to make
        # server instance ignore SIGTERM
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port, "--ignore-sigterm"]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        # since terminate with sigterm will fail, set a lower
        # timeout before sending sigkill so tests won't take too long
>       assert (
            info.terminate(timeout=2) == 1
            or psutil.Process(info.pid).status() == psutil.STATUS_ZOMBIE
        )

tests/test_process_termination.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/psutil/__init__.py:332: in __init__
    self._init(pid)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=233, status='terminated'), pid = 233
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
            self.create_time()
        except AccessDenied:
            # We should never get here as AFAIK we're able to get
            # process creation time on all platforms even as a
            # limited user.
            pass
        except ZombieProcess:
            # Zombies can still be queried by this class (although
            # not always) and pids() return them so just go on.
            pass
        except NoSuchProcess:
            if not _ignore_nsp:
>               raise NoSuchProcess(pid, msg='process PID not found')
E               psutil.NoSuchProcess: process PID not found (pid=233)

/usr/lib/python3/dist-packages/psutil/__init__.py:373: NoSuchProcess
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s2$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 57689 --ignore-sigterm
process 's2' started pid=233
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s2 process startup detected
could not terminated process [psutil.Process(pid=236, name='python3.11', status='zombie', started='20:26:12'), psutil.Process(pid=234, name='python3.11', status='zombie', started='20:26:12'), psutil.Process(pid=235, name='python3.11', status='zombie', started='20:26:12')]
____________________ test_sigkill_after_failed_sigterm[s3] _____________________

self = <psutil._pslinux.Process object at 0x7f36907d5210>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
>           ret = self._cache[fun]
E           AttributeError: 'Process' object has no attribute '_cache'

/usr/lib/python3/dist-packages/psutil/_common.py:443: AttributeError

During handling of the above exception, another exception occurred:

self = <psutil._pslinux.Process object at 0x7f36907d5210>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36907d5210>

    @functools.wraps(fun)
    def wrapper(self):
        try:
            # case 1: we previously entered oneshot() ctx
            ret = self._cache[fun]
        except AttributeError:
            # case 2: we never entered oneshot() ctx
>           return fun(self)

/usr/lib/python3/dist-packages/psutil/_common.py:446: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36907d5210>

    @wrap_exceptions
    @memoize_when_activated
    def _parse_stat_file(self):
        """Parse /proc/{pid}/stat file and return a dict with various
        process info.
        Using "man proc" as a reference: where "man proc" refers to
        position N always subtract 3 (e.g ppid position 4 in
        'man proc' == position 1 in here).
        The return value is cached in case oneshot() ctx manager is
        in use.
        """
>       data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1687: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/237/stat', fallback = <object object at 0x7f36917e8d80>

    def bcat(fname, fallback=_DEFAULT):
        """Same as above but opens file in binary mode."""
>       return cat(fname, fallback=fallback, _open=open_binary)

/usr/lib/python3/dist-packages/psutil/_common.py:776: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/237/stat', fallback = <object object at 0x7f36917e8d80>
_open = <function open_binary at 0x7f3690af3a60>

    def cat(fname, fallback=_DEFAULT, _open=open_text):
        """Read entire file content and return it as a string. File is
        opened in text mode. If specified, `fallback` is the value
        returned in case of error, either if the file does not exist or
        it can't be read().
        """
        if fallback is _DEFAULT:
>           with _open(fname) as f:

/usr/lib/python3/dist-packages/psutil/_common.py:764: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = '/proc/237/stat'

    def open_binary(fname):
>       return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
E       FileNotFoundError: [Errno 2] No such file or directory: '/proc/237/stat'

/usr/lib/python3/dist-packages/psutil/_common.py:728: FileNotFoundError

During handling of the above exception, another exception occurred:

self = psutil.Process(pid=237, status='terminated'), pid = 237
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
>           self.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=237, status='terminated')

    def create_time(self):
        """The process creation time as a floating point number
        expressed in seconds since the epoch.
        The return value is cached after first call.
        """
        if self._create_time is None:
>           self._create_time = self._proc.create_time()

/usr/lib/python3/dist-packages/psutil/__init__.py:714: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36907d5210>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
>           return fun(self, *args, **kwargs)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1645: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36907d5210>

    @wrap_exceptions
    def create_time(self):
>       ctime = float(self._parse_stat_file()['create_time'])

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1855: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <psutil._pslinux.Process object at 0x7f36907d5210>, args = ()
kwargs = {}

    @functools.wraps(fun)
    def wrapper(self, *args, **kwargs):
        try:
            return fun(self, *args, **kwargs)
        except PermissionError:
            raise AccessDenied(self.pid, self._name)
        except ProcessLookupError:
            raise NoSuchProcess(self.pid, self._name)
        except FileNotFoundError:
            if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):
>               raise NoSuchProcess(self.pid, self._name)
E               psutil.NoSuchProcess: process no longer exists (pid=237)

/usr/lib/python3/dist-packages/psutil/_pslinux.py:1652: NoSuchProcess

During handling of the above exception, another exception occurred:

tcp_port = 49877, proc_name = 's3'
xprocess = <xprocess.xprocess.XProcess object at 0x7f3690b60850>

    @pytest.mark.skipif(
        sys.platform == "win32",
        reason="on windows SIGTERM is treated as an alias for kill()",
    )
    @pytest.mark.parametrize("proc_name", ["s1", "s2", "s3"])
    def test_sigkill_after_failed_sigterm(tcp_port, proc_name, xprocess):
        # explicitly tell xprocess_starter fixture to make
        # server instance ignore SIGTERM
        class Starter(ProcessStarter):
            pattern = "started"
            args = [sys.executable, server_path, tcp_port, "--ignore-sigterm"]
    
        xprocess.ensure(proc_name, Starter)
        info = xprocess.getinfo(proc_name)
        # since terminate with sigterm will fail, set a lower
        # timeout before sending sigkill so tests won't take too long
>       assert (
            info.terminate(timeout=2) == 1
            or psutil.Process(info.pid).status() == psutil.STATUS_ZOMBIE
        )

tests/test_process_termination.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/psutil/__init__.py:332: in __init__
    self._init(pid)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = psutil.Process(pid=237, status='terminated'), pid = 237
_ignore_nsp = False

    def _init(self, pid, _ignore_nsp=False):
        if pid is None:
            pid = os.getpid()
        else:
            if not _PY3 and not isinstance(pid, (int, long)):
                raise TypeError('pid must be an integer (got %r)' % pid)
            if pid < 0:
                raise ValueError('pid must be a positive integer (got %s)'
                                 % pid)
        self._pid = pid
        self._name = None
        self._exe = None
        self._create_time = None
        self._gone = False
        self._pid_reused = False
        self._hash = None
        self._lock = threading.RLock()
        # used for caching on Windows only (on POSIX ppid may change)
        self._ppid = None
        # platform-specific modules define an _psplatform.Process
        # implementation class
        self._proc = _psplatform.Process(pid)
        self._last_sys_cpu_times = None
        self._last_proc_cpu_times = None
        self._exitcode = _SENTINEL
        # cache creation time for later use in is_running() method
        try:
            self.create_time()
        except AccessDenied:
            # We should never get here as AFAIK we're able to get
            # process creation time on all platforms even as a
            # limited user.
            pass
        except ZombieProcess:
            # Zombies can still be queried by this class (although
            # not always) and pids() return them so just go on.
            pass
        except NoSuchProcess:
            if not _ignore_nsp:
>               raise NoSuchProcess(pid, msg='process PID not found')
E               psutil.NoSuchProcess: process PID not found (pid=237)

/usr/lib/python3/dist-packages/psutil/__init__.py:373: NoSuchProcess
----------------------------- Captured stdout call -----------------------------
/<<PKGBUILDDIR>>/.pytest_cache/d/.xprocess/s3$ /usr/bin/python3.11 /<<PKGBUILDDIR>>/tests/server.py 49877 --ignore-sigterm
process 's3' started pid=237
0 , % /.%,@%@._%%# #/%/ %

1 , % /.%,@%@._%%# #/%/ %

2 , % /.%,@%@._%%# #/%/ %

3 , % /.%,@%@._%%# #/%/ %

4 , % /.%,@%@._%%# #/%/ %

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

Ê�æ�pP��çîöē�P��adåráøū

started

s3 process startup detected
could not terminated process [psutil.Process(pid=238, name='python3.11', status='zombie', started='20:26:17'), psutil.Process(pid=239, name='python3.11', status='zombie', started='20:26:17'), psutil.Process(pid=240, name='python3.11', status='zombie', started='20:26:17')]
=========================== short test summary info ============================
FAILED tests/test_functional_workflow.py::test_functional_work_flow - Failed:...
FAILED tests/test_process_termination.py::test_clean_shutdown[s1] - assert -1...
FAILED tests/test_process_termination.py::test_clean_shutdown[s2] - assert -1...
FAILED tests/test_process_termination.py::test_clean_shutdown[s3] - assert -1...
FAILED tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s1]
FAILED tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s2]
FAILED tests/test_process_termination.py::test_sigkill_after_failed_sigterm[s3]
============ 7 failed, 36 passed, 2 deselected in 443.72s (0:07:23) ============

Published docs out-of-date (v0.18.0 'UNRELEASED'; should be v0.18.1)

Describe the bug
The published documentation is out-of-date - looks like it's for v0.18.0, but repo and published package at PyPi are at 0.18.1. Perhaps documentation-build or deployment wasn't done for v0.18.1 when it was published to PyPi?

This isn't really an issue with the source code here, but this seems like the best place to file it.

To Reproduce
Steps to reproduce the behavior:

https://pytest-xprocess.readthedocs.io/en/latest/changes.html
Note that the latest version is listed as "0.18.0 (UNRELEASED)"

Expected behavior
The CHANGELOG.rst shown in the latest version of the repo, associated with v0.18.1, should be served at the readthedocs.io site.

evaluate getting a named dir from the pytest cache

instead of putting another dot-dir into the rootdir, we should take a look at asking one form pytest-cache

Pattern matching optional

Is your feature request related to a problem? Please describe.
Some servers run in "quiet" mode where they do not provide any log output. Instead, they will have a health check API. xprocess does not work with these servers because pattern matching is mandatory and it must match with some type of output before running startup_check().

Describe the solution you'd like
Pattern matching should be optional if startup_check() is user-defined. A backoff mechanism can be used to retry the startup_check() call while the server boots up.

Describe alternatives you've considered
N/A

Additional context
N/A

Add child process terminate features.

love your work which makes me easily test my socket server.
Here's one problem is that since my server preforks several child process in order to improving handling capacity. Xprocessinfo.terminate only kills the parent but leaves others become zombie process ,So that I have to handle it myself by doing some wired hacking.
Could you please add features that allow process terminate itself and its children at framework level? Thanks.

ResourceWarning in xprocess.ensure

Just upgraded to 0.21 from 0.20 for Werkzeug, and am now getting resource warnings from within pytest-xprocess.

pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>

Traceback (most recent call last):
  File "/home/david/.local/share/virtualenvs/werkzeug/lib/python3.10/site-packages/xprocess/xprocess.py", line 274, in ensure
    lines = info.logpath.open().readlines()
ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/david/Projects/pallets/werkzeug/.pytest_cache/d/.xprocess/dev_server-test_streaming_chunked_response/xprocess.log' mode='r' encoding='UTF-8'>

The line it's pointing to is this, where a file is opened but not closed.

pytest-xprocess/xprocess/xprocess.py

Lines 273 to 274 in 5363447

 # skip previous process logs 

 lines = info.logpath.open().readlines()

configurable timeout for xprocess.ensure pattern match

Problem description
Currently xprocess.ensure will hang indefinitely in the event of an unmatched start pattern given fewer lines than the number specified by ProcessStarter.max_read_lines have been printed.

Proposed solution
Make xprocess.ensure take an optional argument (timeout), allowing the user to specify for how long to wait process startup before failing with a TimeoutError

Additional context
I'm opening the issue for the sake of documentation and to express my intention to work on this. Also, if anyone has any thoughts/suggestions on how this should be done (or if it should/shouldn't be done at all), please let me know. I will get started on a PR for this soon.

_internal.py isn't shipped with the 0.18.0 version installed by PyPi

Describe the bug

File "/home/ipaleka/dev/venvs/algt/lib/python3.9/site-packages/pytest_xprocess.py", line 4, in <module>
  from _internal import getrootdir
ModuleNotFoundError: No module named '_internal'

To Reproduce
Steps to reproduce the behavior:

$ pip install pytest-xprocess

from xprocess import ProcessStarter

Expected behavior
Flawless execution like in 0.17.1.

Environment (please complete the following information):

OS: Debian Stable (5.10.46-2 (2021-07-20) x86_64 GNU/Linux)
Python Version: 3.9.2
pytest-xprocess version: 0.18.0

A workaround is, well, to just download that file and place it in the environment's site-packages directory next to pytest_xprocess.py.

0.18.1: old version in copy.py

Looks like in https://github.com/pytest-dev/pytest-xprocess/blob/master/docs/source/conf.py#L21 is hardcoded old version.

When pytest exits on error (for ex Ctrl-C), xprocess terminate() does not get called

With this code inside a fixture, using yield then terminate(), terminate() doesn't seem to be called when pytest exits on an error

    # ensure process is running and return its logfile
    logfile = xprocess.ensure("dummy_echo_server", Starter)

    yield (_address, _port)

    # clean up whole process tree afterwards
    xprocess.getinfo("dummy_echo_server").terminate()

How to replicate: run pytest with an xprocess fixture. Hit Ctrl-C. Check process: it's still running.

Describe the solution you'd like
Provide a way to always call terminate() on Python exit.

How to control the current working directory of the launched process?

What would you like to know?.
How to control the current working directory of the launched process?

Additional context
When using pytest-xprocess to launch a server process I notice that it is launched in a different directory:
<test_root>/.pytest_cache/d/.xprocess/<test_fixture_name>

I want the freedom to change that, because my poc server is unfortunately relying on local files.

`tests/server.py` should bind to a random port and report that back to test code

Is your feature request related to a problem? Please describe.

Currently test code specifies which port the TestServer should bind to. This could cause conflict in a busy system.

Describe the solution you'd like

Let TestServer bind to a random port (e.g. port=0), and print out that port so that the test code can connect to it.

Describe alternatives you've considered

None

Additional context

None

Crashes when pytest cache is disabled

Describe the bug
xprocess fixture fails to initialize when running pytest with -p no:cacheprovider:

    @pytest.fixture(scope="session")
    def xprocess(request):
        """yield session-scoped XProcess helper to manage long-running
        processes required for testing."""

>       rootdir = getrootdir(request.config)

../../.pyenv/versions/3.7.9/envs/[email protected]/lib/python3.7/site-packages/pytest_xprocess.py:40:

    def getrootdir(config):
>       return config.cache.makedir(".xprocess")
E       AttributeError: 'Config' object has no attribute 'cache'

To Reproduce
Steps to reproduce the behavior:

Start tests with pytest -p no:cacheprovider

Expected behavior
Successful test run

Environment (please complete the following information):

OS: macOS Big Sur
Python Version 3.7.9
pytest-xprocess version 0.18.1

Server failed to start

I'm trying a quite minimal example with pytest 5.3.4, xprocess 0.12.1 and Python 3.8:

conftest.py:

import sys, socket, pytest, py, pathlib
from xprocess import ProcessStarter

@pytest.fixture
def serversock(xprocess):
    port = 5777
    curdir = pathlib.Path(__file__).parent

    class Starter(ProcessStarter):
        pattern = "started"
        args = [sys.executable, curdir / 'server.py', port]

    xprocess.ensure("server-%s" % port, Starter)
    sock = socket.socket()
    sock.connect(("localhost", port))
    return sock

server.py:

import socketserver
import time
import sys

class MainHandler(socketserver.StreamRequestHandler):
    count = 0
    def handle(self):
        while True:
            line = self.rfile.readline()
            if not line:
                break
            sys.stderr.write("[%d] %r\n" %(self.count, line))
            self.request.sendall(b"1")
            MainHandler.count += 1


class MyServer(socketserver.TCPServer):
    allow_reuse_address = True


if __name__ == "__main__":
    mainserver = MyServer(("localhost", int(sys.argv[1])), MainHandler)
    time.sleep(3) # simulate long startup time
    sys.stderr.write("started\n")
    mainserver.serve_forever()

test_server.py:

def test_server(serversock):
    serversock.sendall(b'hello\n')
    c = serversock.recv(1)
    assert c == b'1'

When I start the server by hand, it runs like expected:

$ python3 server.py 5778                                                                                                                        
started

When I try to run xdist, it also starts it, but then complains that it couldn't start it:

conftest.py:13: in serversock
    xprocess.ensure("server-%s" % port, Starter)
.venv/lib/python3.8/site-packages/xprocess.py:140: in ensure
    raise RuntimeError("Could not start process %s" % name)
E   RuntimeError: Could not start process server-5777
------------------------------------------------------------------------------------ Captured stdout setup ------------------------------------------------------------------------------------
.../.xprocess/server-5777$ .../python3 .../server.py 5777
process 'server-5777' started pid=184070

Yet when I run --xshow it is started:

184070 server-5777 LIVE .../.xprocess/server-5777/xprocess.log

And when I rerun the tests, they work fine.

What's going wrong? Is my pattern wrong in the ProcessStarter somehow? The example in the repo only seems to use a preparefunc which is deprecated...

Unnecessary Print

Is your feature request related to a problem? Please describe.
Can this reminder be made suppressible, or hidden depending on a verbosity level?

pytest-xprocess/xprocess/pytest_xprocess.py

Lines 62 to 67 in 6dac644

 def pytest_unconfigure(config): 

 print( 

 "pytest-xprocess reminder::Be sure to terminate the started process by running " 

 "'pytest --xkill' if you have not explicitly done so in your fixture with " 

 "'xprocess.getinfo(<process_name>).terminate()'." 

 )

Running xprocess in a Docker container causes pytest to hang?

Describe the bug
Running a TCP server with pytest-xprocess works fine in my local Ubuntu environment, but when running in Docker, pytest seems to hang without completing any of the tests.

To Reproduce
Steps to reproduce the behavior:

Using the following fixture in my pytest script

@pytest.fixture(scope = "session", autouse = True)
def host_sim_server(xprocess):
    python_executable_full_path = sys.executable

    class Starter(ProcessStarter):
        pattern = "HostSim server running"
        args = [python_executable_full_path, '-m', 'my_package.server_simulator']

    logfile = xprocess.ensure("host_sim", Starter)  # noqa: F841
    yield
    xprocess.getinfo("host_sim").terminate()

and my_package.server_simulator contains is a simple TCP server, with additional functions needed for testing added in the Handler class (essentially just functions to parse request packets, and respond, nothing that I think would effect pytest):

 class HostSim:
     def __init__(self):
         HOST = "127.0.0.1"
         PORT = 1234
         socketserver.TCPServer.allow_reuse_address = True
         self.tcp_server = socketserver.TCPServer((HOST, PORT), Handler)

         logger.info("HostSim server running")
         self.tcp_server.serve_forever()

     def __del__(self):
         self.tcp_server.server_close()

if __name__ == "__main__":
    sim = HostSim()

Expected behavior
I'd expect the server to start, allowing all the other tests to complete successfully by connecting to localhost:1234 (ie: where the server is listening) and performing the tests.

Log of running in my VM, succeeding:

$ PYTHONPATH=. pytest --cov=my_package tests/
====================================================================== test session starts =======================================================================
platform linux -- Python 3.8.5, pytest-6.1.0, py-1.9.0, pluggy-0.13.1
rootdir: /home/cquick/cquick/my_package
plugins: cov-2.10.1, xprocess-0.14.0
collected 22 items

tests/a_test.py ..                                                                                                                                       [  9%]
tests/b_test.py ...                                                                                                                                     [ 22%]
tests/server_test.py .............                                                                                                                             [ 81%]
tests/c_test.py ....                                                                                                                                     [100%]

... [redacted, coverage report]

======================================================================= 22 passed in 3.17s =======================================================================

Logs

Log of running in Docker, and having it hang

$ docker run -it --rm --entrypoint "/bin/bash" -v "$PWD:/opt" my_pytest_image:xprocess
root@22fff798fc3b:/# cd opt/
root@22fff798fc3b:/opt# PYTHONPATH=. pytest --cov my_package tests/
====================================================================== test session starts =======================================================================
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /opt
plugins: xprocess-0.15.0, cov-2.10.0
collected 22 items

tests/a_test.py ..                                                                                                                                       [  9%]
tests/b_test.py ...                                                                                                                                     [ 22%]
tests/server_test.py

Here it just hangs

Environment:

OS: Ubuntu 20.04.1 LTS
Python Version: 3.8.5
pytest-xprocess version: 0.14.0 and 0.15.0
Docker Version: 19.03.13, build 4484c46d9d
Docker Container Base Image: python:3-buster

Also, our Dockerfile is simply

FROM python:3-buster

RUN apt-get update && \
    apt-get -y full-upgrade && \
    apt-get -y install python3-pip

RUN pip install pytest pytest-cov pytest-xprocess

Additional context
In the Docker container, I am able to manually background the process and everything works, so I am fairly certain that Docker supports the multiprocessing/multithreading stuff.

$ python -m my_project.host_sim &
$ proc_id=$!
$ PYTHONPATH=. pytest --cov my_project tests/
$ kill $proc_id

My main reason for using the xprocess package is to incorporate the server simulator in our CI testing, which uses Docker to run the CI jobs. As a work around, having the CI script be what I added in "Additional Context" does allow the tests to complete, but if course that isn't using xprocess. It seems much cleaner to be able to start the server from within pytest, which we'd like to get working.

I can provide additional information if requested.

Thanks you!

DOC: How to shutdown the process after the test run

Is this the correct way to kill the process on scope teardown?

from pathlib import Path
import pytest
from xprocess import ProcessStarter

PATH = Path(__file__) / ".." / "docs"
PORT = 10000

@pytest.fixture(scope='module')
def myserver(xprocess):
    class Starter(ProcessStarter):
        pattern = "^Serving HTTP on"
        args = ["python", "-m", "http.server", "--directory", PATH, PORT]
    logfile = xprocess.ensure("myserver", Starter)  # == logfile
    yield logfile
    xprocess.getinfo("myserver").kill()

def test_name(myserver):
    from urllib.request import urlopen
    resp = urlopen(f"http://localhost:{PORT}")
    assert resp.code == 200

http://doc.pytest.org/en/latest/fixture.html#fixture-finalization-executing-teardown-code

This seems to be hanging between subsequent pytest invocations; probably due to open connections?

sdist is missing `conftest.py` and `server.py`

Describe the bug

The sdist package at PyPI is missing tests/conftest.py and tests/server.py. Without these files it is not possible to run tests successfuly.

To Reproduce
Steps to reproduce the behavior:

Download sdist from PyPI.
Unpack it.
Run tests.

Expected behavior

Testing pass.

Environment (please complete the following information):

OS: OpenIndiana
Python Version 3.9.18
pytest-xprocess version 1.0.0

Should the documentation still describe this as "experimental"?

The README describes this as experimental - is it still? It looks pretty robust and mature to me.

pytest-xprocess/README.rst

Line 14 in 942c761

 Experimental `pytest <https://docs.pytest.org/en/latest>`_ plugin for managing processes 

Alternative approach to check process state during xprocess.ensure call

XProcess.ensure currently relies on string matching (XProcessStarter.pattern) to check if a process has finished starting. While this works in most situations it starts getting a bit inconvenient when working with processes that don't have too much output or no output at all. We could provide an additional way of checking if processes are ready to answer queries by allowing the user to pass in a callback function to xprocess which will (optionally) be used to assert process responsiveness instead of (or in conjunction with) the pattern matching.

I would be willing to work on a PR if we decide to go ahead with the idea.

0.18.1: pytest is failing

Describe the bug
Looks like new pytest shows one unit failing.

To Reproduce
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Expected behavior

Screenshots
Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-xprocess-0.18.1-3.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-xprocess-0.18.1-3.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-xprocess-0.18.1, configfile: tox.ini
plugins: xprocess-0.18.1
collected 45 items

tests/test_callback.py ......                                                                                                                                        [ 13%]
tests/test_functional_workflow.py .                                                                                                                                  [ 15%]
tests/test_interruption_clean_up.py F.                                                                                                                               [ 20%]
tests/test_process_initialization.py ...............                                                                                                                 [ 53%]
tests/test_process_termination.py ...............                                                                                                                    [ 86%]
tests/test_resource_cleanup.py ...                                                                                                                                   [ 93%]
tests/test_startup_timeout.py ...                                                                                                                                    [100%]

================================================================================= FAILURES =================================================================================
________________________________________________________________________ test_interruption_cleanup _________________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f1b33a624c0>, when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.

        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

/usr/lib/python3.8/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/_pytest/runner.py:259: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/lib/python3.8/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/lib/python3.8/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Popen.__del__ at 0x7f1b34246f70>
E
E               Traceback (most recent call last):
E                 File "/usr/lib64/python3.8/subprocess.py", line 946, in __del__
E                   _warn("subprocess %s is still running" % self.pid,
E               ResourceWarning: subprocess 1673106 is still running

/usr/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-237/test_interruption_cleanup0/runpytest-0
     in: /tmp/pytest-of-tkloczko/pytest-237/test_interruption_cleanup0
============================= test session starts ==============================
platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-237/test_interruption_cleanup0
plugins: xprocess-0.18.1
collected 1 item

test_interruption_cleanup.py

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/tmp/pytest-of-tkloczko/pytest-237/test_interruption_cleanup0/test_interruption_cleanup.py:16: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
============================ no tests ran in 0.42s =============================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
1673112 server_test_interrupt DEAD /tmp/pytest-of-tkloczko/pytest-237/test_interruption_cleanup0/.pytest_cache/d/.xprocess/server_test_interrupt/xprocess.log
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.
========================================================================= short test summary info ==========================================================================
FAILED tests/test_interruption_clean_up.py::test_interruption_cleanup - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Popen.__del__ at 0x7f1b3...
====================================================================== 1 failed, 44 passed in 40.34s =======================================================================
ocess_name>).terminate()'.
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-7f45f463-80bf-49d4-8af6-65ed5d4cc065
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-7f45f463-80bf-49d4-8af6-65ed5d4cc065'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:79: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-638dfb38-0e22-42fa-994b-6fa837a10eb9
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-638dfb38-0e22-42fa-994b-6fa837a10eb9'
  warnings.warn(

Environment (please complete the following information):

OS: Linux x86/64
Python Version: 3.8.13
pytest-xprocess version: 0.18.1

Additional context
List of modules installed in build env

Coverage report from process

What would you like to know?.
Is there a way to get a coverage report from a process like serverless-offline started with xprocess?

isrunning() does not detect termination of short-lived processes

XProcessInfo.isrunning() is unable to detect when short-lived processes are terminated during test execution, due to giampaolo/psutil#1008.

I'm seeing this on MacOS.

This prevents me from easily using this to execute scripts in tests and wait until they are completed.

Flaky termination behaviour due to 0.18.0 changes

It seems changes from #64 have introduced flaky behaviour related to process termination. From CI logs:

FAILED test_functional_workflow.py::test_functional_work_flow
FAILED test_process_termination.py::test_clean_shutdown[6777-s1] - AssertionE..
...
6-29T22:46:03.6561320Z > result.stdout.fnmatch_lines("TERMINATED")
2021-06-29T22:46:03.6562270Z E Failed: nomatch: 'TERMINATED'
2021-06-29T22:46:03.6563150Z E and: '1231 server_workflow_test FAILED TO TERMINATE'
2021-06-29T22:46:03.6565050Z E and: "pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'."
2021-06-29T22:46:03.6566460Z E remains unmatched: 'TERMINATED'
...

The problem doesn't happen frequently, but I managed to reproduce it a few times by re-running CI jobs. I will look into it as soon as possible so we can go through with #73

Impolite service termination

xprocess uses 'terminate' and 'kill' like synonyms, but in practice, uses .kill() or SIGKILL for terminating processes. Instead, it should use a terminate/wait/kill loop similar to what pytest.services does.

Fatal error when subprocess has special characters in stdout

This line throws a TypeError: not enough arguments for format string when a started subprocess tries to print a line that python thinks needs argument substitutions. An example is ' , % /. ,@@@. %%# #/%%/ \n'

ResourceWarnings on windows wsl2

Possibly related to #59

ResourceWarnings are being raised on ubuntu 18.04 LTS on wsl2, which indicates a possible resource leak or improper handling of popen instances.

#86 touches how process resources are handled so I'll probably hold further work on this issue until after the next release is out 💭

Install pytest_xprocess module requires non-installed _internal module

Describe the bug
The package installs pytest_xprocess module by default:

py_modules = 
	pytest_xprocess
	xprocess

However, this module requires _internal that isn't (and with that name, mustn't be) installed:

from _internal import getrootdir

Therefore, after installing it pytest keeps erroring out due to missing module, e.g.:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/pytest", line 33, in <module>
    sys.exit(load_entry_point('pytest==6.2.4', 'console_scripts', 'pytest')())
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/lib/python3.10/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1172, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 162, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/usr/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/usr/lib/python3.10/site-packages/pytest_xprocess.py", line 4, in <module>
    from _internal import getrootdir
ModuleNotFoundError: No module named '_internal'

To Reproduce
Steps to reproduce the behavior:

Install the package.
Attempt to run pytest anywhere.

Expected behavior
Working pytest.

Environment (please complete the following information):

OS: Gentoo Linux
Python Version 3.10.0b4
pytest-xprocess version 0.18.0

Logs are overwritten for each new process

Is your feature request related to a problem? Please describe.
The log file is overwritten every time a new process is launched, making it difficult to see the process's log for a failed run that occurred in the middle of the test suite.

Describe the solution you'd like
I think a better approach would be to delete the log at the start of the pytest session, then append to it throughout the session instead of overwriting it. In addition to the process's log, adding the current test information would be useful too to match the log to the test. It could also be an option to the user to overwrite or append to the log.

Describe alternatives you've considered
To see a particular failing test, I've filtered pytest to only run the one test so I can see the process's log.

Additional context
N/A

Encapsulate xprocess internals info, popen and log handles into single object for resource cleanup

Following @RonnyPfannschmidt suggestion in #64, encapsulating the references and objects used for process resource clean-up in XProcess seems like a natural follow up to improve the PR and make the code readable and less error-prone when compared to the current approach of separate yet logically/semantically related lists.

	# skip previous process logs
	lines = info.logpath.open().readlines()

	def pytest_unconfigure(config):
	print(
	"pytest-xprocess reminder::Be sure to terminate the started process by running "
	"'pytest --xkill' if you have not explicitly done so in your fixture with "
	"'xprocess.getinfo(<process_name>).terminate()'."
	)

pytest-dev / pytest-xprocess Goto Github PK

pytest-xprocess's Introduction

pytest-xprocess

Installing

Useful Links

pytest-xprocess's People

Contributors

Stargazers

Watchers

Forkers

pytest-xprocess's Issues

Recommend Projects

Recommend Topics

Recommend Org