se2p / pynguin Goto Github PK

View Code? Open in Web Editor NEW

1.2K 26.0 74.0 5.57 MB

The PYthoN General UnIt Test geNerator is a test-generation tool for Python

Home Page: https://www.pynguin.eu

License: MIT License

Makefile 0.15% Dockerfile 0.14% Shell 0.11% Python 99.47% HTML 0.13%

python testing-tools test-generation test-generator search-based-software-testing

pynguin's Introduction

Pynguin

Pynguin (IPA: ˈpɪŋɡuiːn), the PYthoN General UnIt test geNerator, is a tool that allows developers to generate unit tests automatically.

Testing software is often considered to be a tedious task. Thus, automated generation techniques have been proposed and mature tools exist—for statically typed languages, such as Java. There is, however, no fully-automated tool available that produces unit tests for general-purpose programs in a dynamically typed language. Pynguin is, to the best of our knowledge, the first tool that fills this gap and allows the automated generation of unit tests for Python programs.

Internal Pipeline Status

Attention

Please Note:

Pynguin executes the module under test! As a consequence, depending on what code is in that module, running Pynguin can cause serious harm to your computer, for example, wipe your entire hard disk! We recommend running Pynguin in an isolated environment; use, for example, a Docker container to minimize the risk of damaging your system.

Pynguin is only a research prototype! It is not tailored towards production use whatsoever. However, we would love to see Pynguin in a production-ready stage at some point; please report your experiences in using Pynguin to us.

Prerequisites

Before you begin, ensure you have met the following requirements:

You have installed Python 3.10 (we have not yet tested with Python 3.11, there might be some problems due to changed internals regarding the byte-code instrumentation).

Attention: Pynguin now requires Python 3.10! Older versions are no longer supported!
You have a recent Linux/macOS/Windows machine.

Please consider reading the online documentation to start your Pynguin adventure.

Installing Pynguin

Pynguin can be easily installed using the pip tool by typing:

pip install pynguin

Make sure that your version of pip is that of a supported Python version, as any older version is not supported by Pynguin!

Using Pynguin

Before you continue, please read the quick start guide

Pynguin is a command-line application. Once you installed it to a virtual environment, you can invoke the tool by typing pynguin inside this virtual environment. Pynguin will then print a list of its command-line parameters.

A minimal full command line to invoke Pynguin could be the following, where we assume that a project foo is located in /tmp/foo, we want to store Pynguin's generated tests in /tmp/testgen, and we want to generate tests using a whole-suite approach for the module foo.bar (wrapped for better readability):

pynguin \
  --project-path /tmp/foo \
  --output-path /tmp/testgen \
  --module-name foo.bar

Please find a more detailed example in the quick start guide.

Contributing to Pynguin

For the development of Pynguin you will need the poetry dependency management and packaging tool. To start developing, follow these steps:

Clone the repository
Change to the pynguin folder: cd pynguin
Create a virtual environment and install dependencies using poetry: poetry install
Make your changes
Run make check to verify that your changes pass all checks

Please see the poetry documentation for more information on this tool.

Contributors

Pynguin is developed at the Chair of Software Engineering II of the University of Passau.

Maintainer: Stephan Lukasczyk

Contributors:

Development using PyCharm.

If you want to use the PyCharm IDE you have to set up a few things:

Import pynguin into PyCharm.
Let PyCharm configure configure a virtual environment using poetry.
Set the default test runner to pytest
Set the DocString format to Google

License

This project is licensed under the terms of the MIT License. Pynguin was using the GNU Lesser General Public License (LGPL) until version 0.29.0, its licence was changed with version 0.30.0.

Star History

pynguin's People

Contributors

Stargazers

Watchers

pynguin's Issues

pynguin does not finish. Does not point to the piece of code that caused a problem

windows, python 3.10.5, pynguin version 0.25.2 using a venv not shared with other projects.
(venv) PS C:\Users\xxxxx\PycharmProjects\Fighting> pynguin --project-path . --output_path ./test --module_name main_one_file ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ C:\Python\Python310\lib\runpy.py:196 in _run_module_as_main │ │ │ │ 193 │ main_globals = sys.modules["__main__"].__dict__ │ │ 194 │ if alter_argv: │ │ 195 │ │ sys.argv[0] = mod_spec.origin │ │ ❱ 196 │ return _run_code(code, main_globals, None, │ │ 197 │ │ │ │ │ "__main__", mod_spec) │ │ 198 │ │ 199 def run_module(mod_name, init_globals=None, │ │ │ │ C:\Python\Python310\lib\runpy.py:86 in _run_code │ │ │ │ 83 │ │ │ │ │ __loader__ = loader, │ │ 84 │ │ │ │ │ __package__ = pkg_name, │ │ 85 │ │ │ │ │ __spec__ = mod_spec) │ │ ❱ 86 │ exec(code, run_globals) │ │ 87 │ return run_globals │ │ 88 │ │ 89 def _run_module_code(code, init_globals=None, │ │ │ │ C:\Users\xxxxx\PycharmProjects\Fighting\venv\Scripts\pynguin.exe\__main__.py:7 in <module> │ │ │ │ [Errno 2] No such file or directory: │ │ 'C:\\Users\\xxxxx\\PycharmProjects\\Fighting\\venv\\Scripts\\pynguin.exe\\__main__.py' │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\cli.py:190 in main │ │ │ │ 187 │ set_configuration(parsed.config) │ │ 188 │ if console is not None: │ │ 189 │ │ with console.status("Running Pynguin..."): │ │ ❱ 190 │ │ │ return run_pynguin().value │ │ 191 │ else: │ │ 192 │ │ return run_pynguin().value │ │ 193 │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generator.py:107 in │ │ run_pynguin │ │ │ │ 104 │ """ │ │ 105 │ try: │ │ 106 │ │ _LOGGER.info("Start Pynguin Test Generation…") │ │ ❱ 107 │ │ return _run() │ │ 108 │ finally: │ │ 109 │ │ _LOGGER.info("Stop Pynguin Test Generation…") │ │ 110 │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generator.py:330 in _run │ │ │ │ 327 │ │ executor, test_cluster, constant_provider │ │ 328 │ ) │ │ 329 │ _LOGGER.info("Start generating test cases") │ │ ❱ 330 │ generation_result = algorithm.generate_tests() │ │ 331 │ if algorithm.resources_left(): │ │ 332 │ │ _LOGGER.info("Algorithm stopped before using all resources.") │ │ 333 │ else: │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generation\algorithms\dyn │ │ amosastrategy.py:70 in generate_tests │ │ │ │ 67 │ │ │ self.create_test_suite(self._archive.solutions) │ │ 68 │ │ ) │ │ 69 │ │ while self.resources_left() and len(self._archive.uncovered_goals) > 0: │ │ ❱ 70 │ │ │ self.evolve() │ │ 71 │ │ │ self.after_search_iteration(self.create_test_suite(self._archive.solutions)) │ │ 72 │ │ │ │ 73 │ │ self.after_search_finish() │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generation\algorithms\dyn │ │ amosastrategy.py:84 in evolve │ │ │ │ 81 │ │ """Runs one evolution step.""" │ │ 82 │ │ offspring_population: list[ │ │ 83 │ │ │ tcc.TestCaseChromosome │ │ ❱ 84 │ │ ] = self._breed_next_generation() │ │ 85 │ │ │ │ 86 │ │ # Create union of parents and offspring │ │ 87 │ │ union: list[tcc.TestCaseChromosome] = [] │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generation\algorithms\abs │ │ tractmosastrategy.py:60 in _breed_next_generation │ │ │ │ 57 │ │ │ │ offspring_population.append(offspring_1) │ │ 58 │ │ │ │ │ 59 │ │ │ # Apply mutation on offspring_2 │ │ ❱ 60 │ │ │ self._mutate(offspring_2) │ │ 61 │ │ │ if offspring_2.has_changed() and offspring_2.size() > 0: │ │ 62 │ │ │ │ offspring_population.append(offspring_2) │ │ 63 │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\generation\algorithms\abs │ │ tractmosastrategy.py:85 in _mutate │ │ │ │ 82 │ │ │ 83 │ @staticmethod │ │ 84 │ def _mutate(offspring: tcc.TestCaseChromosome) -> None: │ │ ❱ 85 │ │ offspring.mutate() │ │ 86 │ │ if not offspring.has_changed(): │ │ 87 │ │ │ # if offspring is not changed, we try to mutate it once again │ │ 88 │ │ │ offspring.mutate() │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\ga\testcasechromosome.py: │ │ 122 in mutate │ │ │ │ 119 │ │ │ randomness.next_float() │ │ 120 │ │ │ <= config.configuration.search_algorithm.test_delete_probability │ │ 121 │ │ ): │ │ ❱ 122 │ │ │ if self._mutation_delete(): │ │ 123 │ │ │ │ changed = True │ │ 124 │ │ │ │ 125 │ │ if ( │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\ga\testcasechromosome.py: │ │ 159 in _mutation_delete │ │ │ │ 156 │ │ │ if idx >= self.size(): │ │ 157 │ │ │ │ continue │ │ 158 │ │ │ if randomness.next_float() <= p_per_statement: │ │ ❱ 159 │ │ │ │ changed |= self._delete_statement(idx) │ │ 160 │ │ return changed │ │ 161 │ │ │ 162 │ def _delete_statement(self, idx: int) -> bool: │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\ga\testcasechromosome.py: │ │ 164 in _delete_statement │ │ │ │ 161 │ │ │ 162 │ def _delete_statement(self, idx: int) -> bool: │ │ 163 │ │ assert self._test_factory, "Mutation requires a test factory." │ │ ❱ 164 │ │ modified = self._test_factory.delete_statement_gracefully(self._test_case, idx) │ │ 165 │ │ return modified │ │ 166 │ │ │ 167 │ def _mutation_change(self) -> bool: │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\testcase\testfactory.py:6 │ │ 64 in delete_statement_gracefully │ │ │ │ 661 │ │ changed = False │ │ 662 │ │ if variable is not None: │ │ 663 │ │ │ for i in range(position + 1, test_case.size()): │ │ ❱ 664 │ │ │ │ alternatives = test_case.get_objects(variable.type, i) │ │ 665 │ │ │ │ try: │ │ 666 │ │ │ │ │ alternatives.remove(variable) │ │ 667 │ │ │ │ except ValueError: │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\testcase\testcase.py:254 │ │ in get_objects │ │ │ │ 251 │ │ │ var = statement.ret_val │ │ 252 │ │ │ if var is None: │ │ 253 │ │ │ │ continue │ │ ❱ 254 │ │ │ if not var.is_none_type() and is_consistent_with(var.type, parameter_type): │ │ 255 │ │ │ │ variables.append(var) │ │ 256 │ │ │ │ 257 │ │ return variables │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\utils\type_utils.py:162 │ │ in is_consistent_with │ │ │ │ 159 │ │ typ1 is typing.Any or typ2 is typing.Any │ │ 160 │ ): # pylint:disable=comparison-with-callable │ │ 161 │ │ return True │ │ ❱ 162 │ return is_subtype_of(typ1, typ2) │ │ 163 │ │ 164 │ │ 165 def extract_non_generic_class(obj) -> type | None: │ │ │ │ c:\users\xxxxx\pycharmprojects\fighting\venv\lib\site-packages\pynguin\utils\type_utils.py:136 │ │ in is_subtype_of │ │ │ │ 133 │ ) is not None: │ │ 134 │ │ # Hacky way to handle generics by trying to extract non-generic classes │ │ 135 │ │ # and checking for subclass relation │ │ ❱ 136 │ │ return issubclass(non_generic_tp1, non_generic_tp2) │ │ 137 │ if is_union_type(typ2): │ │ 138 │ │ if is_union_type(typ1): │ │ 139 │ │ │ # Union[int, str] is a subtype of Union[str, float, int] │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: issubclass() argument 2 cannot be a parameterized generic

Any help would be great. Thanks!

pynguin does not generate tests for all functions in the module

I've tried pynguin out of curiosity for my dpaster project, and it was only able to generate test cases for one function - get_syntax. The other two functions in the application module were not considered. Also, IMHO, the test cases generated for the get_syntax function looked very machine-generated, at best.

How to prevent pynguin from generating failed test cases

Related to #38

Though I understand test cases which raises exceptions are useful, I need also non-failing test cases.
Especially, we cannot get non-failing test cases if a failed test case achieves 100% coverage. And the case frequently occurs for simple code.
Is there any way to prevent pynguin from generating failed test cases?
I expect some options like --checked-exception=INVALID --unchecked-exception=INVALID of Randoop.

Pynguin encounters error when running against webpy

We are evaluating our project pyster(also a Python test generator) with Pynguin and found that Pynguin throws the following exception when running against webpy:

pynguin --algorithm WSPY --project_path public_repo/webpy --module_name web.db --output_path ./public_repo/webpy --budget 600
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 805, in successors
    return iter(self._succ[n])
KeyError: ProgramGraphNode(index=1, basic_block=[<FOR_ITER arg=[<LOAD_CONST arg=ExecutionTracer lineno=369>, <LOAD_METHOD arg='executed_bool_predicate' lineno=369>, <LOAD_CONST arg=False lineno=369>, <LOAD_CONST arg=45 lineno=369>, <CALL_METHOD arg=2 lineno=369>, <POP_TOP lineno=369>, <JUMP_ABSOLUTE arg=[<LOAD_GLOBAL arg='SQLQuery' lineno=375>, <LOAD_METHOD arg='join' lineno=375>, <LOAD_FAST arg='result' lineno=375>, <LOAD_CONST arg='' lineno=375>, <CALL_METHOD arg=2 lineno=375>, <RETURN_VALUE lineno=375>] lineno=369>] lineno=369>])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/pynguin", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/pynguin/cli.py", line 136, in main
    return generator.run().value
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 101, in run
    return self._run()
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 224, in _run
    if (setup_result := self._setup_and_check()) is None:
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 186, in _setup_and_check
    if not self._load_sut():
  File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 141, in _load_sut
    importlib.import_module(config.INSTANCE.module_name)
  File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 32, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 779, in exec_module
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 49, in get_code
    return instrumentation.instrument_module(to_instrument)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 326, in instrument_module
    return self._instrument_code_recursive(module_code)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 114, in _instrument_code_recursive
    return self._instrument_inner_code_objects(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 75, in _instrument_inner_code_objects
    self._instrument_code_recursive(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 113, in _instrument_code_recursive
    self._instrument_cfg(cfg, code_object_id)
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 130, in _instrument_cfg
    predicate_id = self._instrument_node(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 172, in _instrument_node
    predicate_id = self._instrument_for_loop(
  File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 423, in _instrument_for_loop
    for successor in dominator_tree.get_transitive_successors(node):
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 188, in get_transitive_successors
    return self._get_transitive_successors(node, set())
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 192, in _get_transitive_successors
    for successor_node in self.get_successors(node):
  File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 129, in get_successors
    for successor in self._graph.successors(node):
  File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 807, in successors
    raise NetworkXError(f"The node {n} is not in the digraph.") from e
networkx.exception.NetworkXError: The node ProgramGraphNode(1) is not in the digraph.

Clarification on pynguin's potential to cause serious harm

Currently the docs make very prominent mention of the fact that pynguin can cause serious harm to your system, and even requires the user to sign a waiver by setting an env variable acknowledging this. What exactly does this mean?

As far as I can tell (I may be wrong) it just means that pynguin executes the module it's generating tests for instead of analyzing the code statically. If that's correct, then pynguin seems no more or less dangerous than python unit testing conventions themselves: if I want to test a module foo then my test script test_foo.py will contain a line like from foo import bar which will execute top-level code is in foo.py (excepting the if __name__ == "__main__" block if applicable).

So, is pynguin any more dangerous than e.g. pytest? Further clarification of this point in the docs and in the warning message when pynguin is run with PYNGUIN_DANGER_AWARE unset would be welcome.

Thanks!

How to use pynguin for flask app (with get and post method)

I want to generate unit test cases for the flask app using pynguin which is containing get and post methods. How can we do this?

How can I only generate testcases for the functions in the input source file?

I want to ignore the functions in the dependent modules included in the souce file. Is there any configuration to help me only generate testcases for the functions in the input source file? Thanks.

Generating weird folders during test gen

Hi, I installed pynguin==0.7.2
and run it with command

pynguin --algorithm WHOLE_SUITE --project_path /Users/iuliia_volkova2/work/simple-ddl-parser/temp --output_path testgen --module_name simple_ddl_parser.output.common

I got a dozens of strange named folders in project path (take a look at the screen).

And got the error )

Behaviour re-produceable.
Python 3.8.7
OS: Darwin (MacOS)

[Push-Request] Add minimum coverage

There are many programs that do not achieve 100% coverage, so Pynguin runs for a very looong time.
Therefore, the user should be able to decide if they want to reach a certain minimum coverage (e.g. 75%) and if the coverage does not increase (e.g. same coverage over 10 iterations) after that minimum, the program should stop.

RuntimeError: The current thread shall not be executed any more, thus I kill it.

pynguin 0.27.0
Python 3.10.7
mac arm

I'm getting a RuntimeError logged to the console, I'm trying to figure out what is causing it, I tried setting maximum-search-time to 18000 (30 minutes) and maximum_iterations to 10000 but it still log the same error.
Anyway it generates 4 test cases with 3 of them with @pytest.mark.xfail(strict=True)

error below:

ERROR    Exception in Thread: <Thread(Thread-266595 (_execute_test_case), started daemon 6209482752)>                                                execution.py:2081
                    ╭─────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────╮                  
                    │ /Users/avraham/.pyenv/versions/3.10.7/lib/python3.10/threading.py:1016 in _bootstrap_inner                                              │                  
                    │                                                                                                                                         │                  
                    │   1013 │   │   │   │   _sys.setprofile(_profile_hook)                                                                                   │                  
                    │   1014 │   │   │                                                                                                                        │                  
                    │   1015 │   │   │   try:                                                                                                                 │                  
                    │ ❱ 1016 │   │   │   │   self.run()                                                                                                       │                  
                    │   1017 │   │   │   except:                                                                                                              │                  
                    │   1018 │   │   │   │   self._invoke_excepthook(self)                                                                                    │                  
                    │   1019 │   │   finally:                                                                                                                 │                  
                    │                                                                                                                                         │                  
                    │ /Users/avraham/.pyenv/versions/3.10.7/lib/python3.10/threading.py:953 in run                                                            │                  
                    │                                                                                                                                         │                  
                    │    950 │   │   """                                                                                                                      │                  
                    │    951 │   │   try:                                                                                                                     │                  
                    │    952 │   │   │   if self._target is not None:                                                                                         │                  
                    │ ❱  953 │   │   │   │   self._target(*self._args, **self._kwargs)                                                                        │                  
                    │    954 │   │   finally:                                                                                                                 │                  
                    │    955 │   │   │   # Avoid a refcycle if the thread is running a function with                                                          │                  
                    │    956 │   │   │   # an argument that has a member that points to the thread.                                                           │                  
                    │                                                                                                                                         │                  
                    │ /Users/avraham/Library/Caches/pypoetry/virtualenvs/M5hsRoCT-py3.10/lib/python3.10/site-packages/pynguin/testcase/execu │                  
                    │ tion.py:2184 in _execute_test_case                                                                                                      │                  
                    │                                                                                                                                         │                  
                    │   2181 │   │   for idx, statement in enumerate(test_case.statements):                                                                   │                  
                    │   2182 │   │   │   ast_node = self._before_statement_execution(statement, exec_ctx)                                                     │                  
                    │   2183 │   │   │   exception = self.execute_ast(ast_node, exec_ctx)                                                                     │                  
                    │ ❱ 2184 │   │   │   self._after_statement_execution(statement, exec_ctx, exception)                                                      │                  
                    │   2185 │   │   │   if exception is not None:                                                                                            │                  
                    │   2186 │   │   │   │   result.report_new_thrown_exception(idx, exception)                                                               │                  
                    │   2187 │   │   │   │   break                                                                                                            │                  
                    │                                                                                                                                         │                  
                    │ /Users/avraham/Library/Caches/pypoetry/virtualenvs/M5hsRoCT-py3.10/lib/python3.10/site-packages/pynguin/testcase/execu │                  
                    │ tion.py:2283 in _after_statement_execution                                                                                              │                  
                    │                                                                                                                                         │                  
                    │   2280 │   │   # See comments in _before_statement_execution                                                                            │                  
                    │   2281 │   │   if self.tracer.current_thread_identifier != threading.current_thread().ident:                                            │                  
                    │   2282 │   │   │   # Kill this thread                                                                                                   │                  
                    │ ❱ 2283 │   │   │   raise RuntimeError(                                                                                                  │                  
                    │   2284 │   │   │   │   "The current thread shall not be executed any more, thus I kill it."                                             │                  
                    │   2285 │   │   │   )                                                                                                                    │                  
                    │   2286                                                                                                                                  │                  
                    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                  
                    RuntimeError: The current thread shall not be executed any more, thus I kill it.

doc is dead: 500 or 522 error

https://pynguin.readthedocs.io/en/latest/user/quickstart.html
Hi, one issue about the doc: when clicking the Assertions link, it breaks the whole doc site and return 500 error in Chrome and 522 error in Edge, could you replicate that and fix it?

Equivalent of #pragma: no cover

Hello,
Thank you for open-sourcing this package. I am testing your tool with some of my code.
Do you have a way to exclude some codes from the test generation step ? Like #pragma: no cover in the coverage python package.

For example my traceback when encountered some numba functions.

numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
cannot determine Numba type of <class 'pynguin.testcase.execution.executiontracer.ExecutionTracer'>

File "fklab/events/basic_algorithms.py", line 671:
def _bsearchi(vector, key):  # pragma: no cover
    <source elided>

    nmemb = len(vector)
    ^

Thank you !

what should be the value of PYNGUIN_DANGER_AWARE

The docs says PYNGUIN_DANGER_AWARE should be set but what value ? It's not specified anywhere.

Exception: Statement is not part of it's test case

Hi~
I encountered a problem when I attempted to generate test cases for a toy program named test_script.py shown as below:

import numpy as np


def add1(a:np.ndarray,b:np.ndarray)->np.ndarray:
    if a.shape == b.shape:
        return a+b
    else:
        return np.array([])
if __name__ == "__main__":
    print(add1(np.array([1,2,3]),np.array([5,6,7,9])))

I placed test_pynguin.py at /data02/projects/test_pynguin and ran the command:

cd /data02/projects/test_pynguin
pynguin --project-path . --output-path /data02/projects/output --module-name test_script

Then I received an exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│                                                                                                  │
│ /data02/anaconda3/envs/py38/bin/pynguin:8 in <module>                                    │
│                                                                                                  │
│   5 from pynguin.cli import main                                                                 │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/cli.py:172 in main       │
│                                                                                                  │
│   169 │                                                                                          │
│   170 │   set_configuration(parsed.config)                                                       │
│   171 │   with console.status("Running Pynguin..."):                                             │
│ ❱ 172 │   │   return run_pynguin().value                                                         │
│   173                                                                                            │
│   174                                                                                            │
│   175 if __name__ == "__main__":                                                                 │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:98 in       │
│ run_pynguin                                                                                      │
│                                                                                                  │
│    95 │   """                                                                                    │
│    96 │   try:                                                                                   │
│    97 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱  98 │   │   return _run()                                                                      │
│    99 │   finally:                                                                               │
│   100 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   101                                                                                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:266 in _run │
│                                                                                                  │
│   263 │   │   executor, test_cluster                                                             │
│   264 │   )                                                                                      │
│   265 │   _LOGGER.info("Start generating test cases")                                            │
│ ❱ 266 │   generation_result = algorithm.generate_tests()                                         │
│   267 │   if algorithm.stopping_condition.is_fulfilled():                                        │
│   268 │   │   _LOGGER.info("Used up all resources (%s).", algorithm.stopping_condition)          │
│   269 │   else:                                                                                  │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
│ namosastrategy.py:70 in generate_tests                                                           │
│                                                                                                  │
│    67 │   │   │   self.create_test_suite(self._archive.solutions)                                │
│    68 │   │   )                                                                                  │
│    69 │   │   while self.resources_left() and len(self._archive.uncovered_goals) > 0:            │
│ ❱  70 │   │   │   self.evolve()                                                                  │
│    71 │   │   │   self.after_search_iteration(self.create_test_suite(self._archive.solutions))   │
│    72 │   │                                                                                      │
│    73 │   │   self.after_search_finish()                                                         │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
│ namosastrategy.py:84 in evolve                                                                   │
│                                                                                                  │
│    81 │   │   """Runs one evolution step."""                                                     │
│    82 │   │   offspring_population: List[                                                        │
│    83 │   │   │   tcc.TestCaseChromosome                                                         │
│ ❱  84 │   │   ] = self._breed_next_generation()                                                  │
│    85 │   │                                                                                      │
│    86 │   │   # Create union of parents and offspring                                            │
│    87 │   │   union: List[tcc.TestCaseChromosome] = []                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
│ stractmosastrategy.py:53 in _breed_next_generation                                               │
│                                                                                                  │
│    50 │   │   │   │   │   continue                                                               │
│    51 │   │   │                                                                                  │
│    52 │   │   │   # Apply mutation on offspring_1                                                │
│ ❱  53 │   │   │   self._mutate(offspring_1)                                                      │
│    54 │   │   │   if offspring_1.has_changed() and offspring_1.size() > 0:                       │
│    55 │   │   │   │   offspring_population.append(offspring_1)                                   │
│    56                                                                                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
│ stractmosastrategy.py:83 in _mutate                                                              │
│                                                                                                  │
│    80 │                                                                                          │
│    81 │   @staticmethod                                                                          │
│    82 │   def _mutate(offspring: tcc.TestCaseChromosome) -> None:                                │
│ ❱  83 │   │   offspring.mutate()                                                                 │
│    84 │   │   if not offspring.has_changed():                                                    │
│    85 │   │   │   # if offspring is not changed, we try to mutate it once again                  │
│    86 │   │   │   offspring.mutate()                                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
│ :129 in mutate                                                                                   │
│                                                                                                  │
│   126 │   │   │   randomness.next_float()                                                        │
│   127 │   │   │   <= config.configuration.search_algorithm.test_change_probability               │
│   128 │   │   ):                                                                                 │
│ ❱ 129 │   │   │   if self._mutation_change():                                                    │
│   130 │   │   │   │   changed = True                                                             │
│   131 │   │                                                                                      │
│   132 │   │   if (                                                                               │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
│ :182 in _mutation_change                                                                         │
│                                                                                                  │
│   179 │   │   │   │   │   continue                                                               │
│   180 │   │   │   │   old_distance = statement.ret_val.distance                                  │
│   181 │   │   │   │   ret_val = statement.ret_val                                                │
│ ❱ 182 │   │   │   │   if statement.mutate():                                                     │
│   183 │   │   │   │   │   changed = True                                                         │
│   184 │   │   │   │   else:                                                                      │
│   185 │   │   │   │   │   assert self._test_factory, "Mutation requires a test factory."         │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 16 in mutate                                                                                     │
│                                                                                                  │
│   1013 │   │   if mutable_param_count > 0:                                                       │
│   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
│   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
│ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
│   1017 │   │   return changed                                                                    │
│   1018 │                                                                                         │
│   1019 │   def _mutable_argument_count(self) -> int:                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 52 in _mutate_parameters                                                                         │
│                                                                                                  │
│   1049 │   │   changed = False                                                                   │
│   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
│   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
│ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
│   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
│   1054 │   │   │   │   )                                                                         │
│   1055                                                                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:11 │
│ 16 in _mutate_parameter                                                                          │
│                                                                                                  │
│   1113 │   │   │   │   │   },                                                                    │
│   1114 │   │   │   │   ),                                                                        │
│   1115 │   │   │   )                                                                             │
│ ❱ 1116 │   │   │   copy.mutate()                                                                 │
│   1117 │   │   │   possible_replacements.append(copy.ret_val)                                    │
│   1118 │   │                                                                                     │
│   1119 │   │   # TODO(fk) Use param_type instead of to_mutate.variable_type,                     │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 16 in mutate                                                                                     │
│                                                                                                  │
│   1013 │   │   if mutable_param_count > 0:                                                       │
│   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
│   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
│ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
│   1017 │   │   return changed                                                                    │
│   1018 │                                                                                         │
│   1019 │   def _mutable_argument_count(self) -> int:                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 52 in _mutate_parameters                                                                         │
│                                                                                                  │
│   1049 │   │   changed = False                                                                   │
│   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
│   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
│ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
│   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
│   1054 │   │   │   │   )                                                                         │
│   1055                                                                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 72 in _mutate_parameter                                                                          │
│                                                                                                  │
│   1069 │   │   current = self._args.get(param_name, None)                                        │
│   1070 │   │   param_type = inf_sig.parameters[param_name]                                       │
│   1071 │   │   possible_replacements = self.test_case.get_objects(                               │
│ ❱ 1072 │   │   │   param_type, self.get_position()                                               │
│   1073 │   │   )                                                                                 │
│   1074 │   │                                                                                     │
│   1075 │   │   # Param has to be optional, otherwise it would be set.                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:17 │
│ 6 in get_position                                                                                │
│                                                                                                  │
│    173 │   │   for idx, stmt in enumerate(self._test_case.statements):                           │
│    174 │   │   │   if stmt == self:                                                              │
│    175 │   │   │   │   return idx                                                                │
│ ❱  176 │   │   raise Exception("Statement is not part of it's test case")                        │
│    177 │                                                                                         │
│    178 │   def add_assertion(self, assertion: ass.Assertion) -> None:                            │
│    179 │   │   """Add the given assertion to this statement.                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Looking forward to your reply and thanks in advance!

PLEASE DELETE, BEEN RESOLVED

Use @pytest.mark.parametrize when generating pytest test cases

The currently generated tests have a lot of repeated code.
Is it possible to just generate a list of tuples of input data and expected result that are usable by pytest parametrize mark decorator?

Let me demonstrate using the example from the documentation:

def test_case_1():
    var0 = -2603
    var1 = module0.triangle(var0, var0, var0)
    assert var1 == "Equilateral triangle"
    var2 = 1272
    var3 = module0.triangle(var0, var2, var2)
    assert var3 == "Isosceles triangle"

This can be rewritten using pytest.mark.parametrize:

@pytest.mark.parametrize(
    "a, b, c, expected",
    [
        (-2603, -2603, -2603, "Equilateral triangle"),
        (-2603, 1272, 1272, "Isosceles triangle"),
    ],
)
def test_case_1(a, b, c, expected):
    assert module0.triangle(a, b, c) == expected

This can be even improved if testdata would be declared separately as a list of tuples and later used once in the tescase decorator.
See more examples here: https://docs.pytest.org/en/6.2.x/example/parametrize.html#paramexamples

It would be nice to also have test case names or ids to reflect what area of testing (e.g. boundary-value analysis or some edge case). Perhaps that is more of a n issue for the algorithm used to generate test cases. But i am cramming all this in one issue, which is not right.

For me personally the pytest parametrize usage would instantly upgrade this tool from academic interest to imediately usefull tool.

Generate test cases with unittest/syntax equivalent to PyCharm generated test cases syntax

Hi, thanks a lot for open-sourcing this project! I personally find the "MOSA" algorithm most helpful for my use. If possible, can you add the feature to generate the syntax with unittest syntax (PyCharm class -> Generate... -> Test...) so that it is more industry standard and easier to run?

Segmantation fault while running mutant

Hello.

I got an segmentation fault error while trying to generate test cases using pynguin.

Error Message : Running Pynguin...Segmentation fault (core dumped)

After further analysis, I found the cause.
Whenever I use dict.keys() or items() method and the mutant's input to be an empty dictionary({}), the tests fail.

I prepared a sample code below.

class Sample:
    def __init__(self):
        pass

    def get_user_names(self, user_dict: dict):
        headers = []

        for key in user_dict.keys():
            name = user_dict[key]['name']
            headers.append(name)

        print(headers)
        return headers

If I check length of the keys first, then It works fine.

or change line 8 to

for key in list(user_dict.keys()):

, it works too.

I think this is a general case.
I mean, many developers could encounter this kind of situation.
There are too many code lines like this sample in my projects... Hard to fix all of them.

Please fix this bug or let me know another way to avoid this error.
Thanks.

Environment variable 'PYNGUIN_DANGER_AWARE' not set.

Hi!
I have tried to run
pynguin

and i am getting this

Environment variable 'PYNGUIN_DANGER_AWARE' not set..
Aborting to avoid harming your system..
Please refer to the documentation.
(https://pynguin.readthedocs.io/en/latest/user/quickstart.html).
to see why this happens and what you must do to prevent it.

Am I doing something wrong on setup?

Generate test cases for specified types only

I'm looking for a tool to automatically generate tests for python functions, and came across pynguin. It looks really cool, and I was able to install it and run the tests from your docs.

However, when generating test, I notice that a lot of the time, pynguin seems to "ignore"(?) the type hints for function signatures. E.g., even for the triangle example where the hints specify x, y, z to be integers, many (the majority of) tests set some or all of these to be None types, or Booleans, or sometimes other datatypes.

I am wondering, is there an option to restrict test cases to using only types specified by the functions? I could not find anything like that in the documentation.

ModuleNotFoundError: No module named 'XYZ'

Hello,

I am using penguin in my project and getting the following error.

ModuleNotFoundError: No module named 'webapp'.

My project structure is as follows

Folder structure:

webapp (does not contain __init__.py)
- server (does not contain __init__.py)
- - module1 (contains __init__.py)
- - module2 (contains __init__.py)
    ....
- - authUtilities (contains __init__.py)

The code in module1/__init__.py has an import statement as follows

from webapp.server.authUtilities import verify_token

Here's the penguin command I am executing.

pynguin --project-path XXX/webapp/server/module1 --output-path XXX/webapp/testCases --module-name __init__ -v

The error I get is

ModuleNotFoundError: No module named 'webapp'

Here's the entire trace

...../.venv/lib/python3.9/site-packages/pynguin/generator.py:141 in _load_sut
│ 138 │ try:
│ 139 │ │ # We need to set the current thread ident so the import trace is recorded.
│ 140 │ │ tracer.current_thread_identifier = threading.current_thread().ident
| ❱ 141 │ │ importlib.import_module(config.configuration.module_name)
│ 142 │ except ImportError as ex:
│ 143 │ │ # A module could not be imported because some dependencies
144 │ │ # are missing or it is malformed

..../opt/anaconda3/lib/python3.9/importlib/__init__.py:127 in import_module
│
│ 124 │ │ │ if character != '.':
│ 125 │ │ │ │ break
│ 126 │ │ │ level += 1
│ ❱ 127 │ return _bootstrap._gcd_import(name[level:], package, level)
│ 128
| 129
│ 130 _RELOADING = {}
│ :1030 in _gcd_import
│ :1007 in _find_and_load
│ :986 in _find_and_load_unlocked
│ :680 in _load_unlocked

...... /.venv/lib/python3.9/site-packages/pynguin/instrumentation/machinery.py:44 in exec_module

                │    41 │  
                │    42 │   def exec_module(self, module):                 
                │    43 │   │   self._tracer.reset()     
                │ ❱  44 │   │   super().exec_module(module)                 
                │    45 │   │   self._tracer.store_import_trace()               
                │    46 │  
                │    47 │   def get_code(self, fullname) -> CodeType:   
                
                │ <frozen importlib._bootstrap_external>:850 in exec_module      
                
                │ <frozen importlib._bootstrap>:228 in _call_with_frames_removed

XXX/webapp/server/module1/__init__.py:4 in
│ 1 import json
|
│ ❱ 4 from webapp.server.authUtilities import verify_token

I am executing penguin in a virtual environment however not inside a docker.

Is there a specific switch that I should be using?

Kindly excuse if this is a novice question, would appreciate any direction.

Restructuring help and questions/discussion. (Issue was: honor max-iteration setting)

I am runing pynguin 0.17.0 with Python 3.9.10. I created some example code here:

https://github.com/inktrap/pynguin-example

Basicly it seems that pynguin does not value the max iteration option and perhaps other options too.

`in` operator seems not to be supported

Consider a simple function:

def detector(s: str):
    if 'py' in s:
        return True
    else:
        return False

Pynguin test generation fails on it with an unclear AttributeError message:

pynguin \
    --output-path ./pynguin --project-path . \
    --module-name pynguin-example \
    -v
[11:59:10] INFO     Start Pynguin Test Generation…                                                                  generator.py:110
           INFO     Collecting static constants from module under test                                              generator.py:209
           INFO     No constants found                                                                              generator.py:212
           INFO     Setting up runtime collection of constants                                                      generator.py:221
           INFO     Stop Pynguin Test Generation…                                                                   generator.py:113
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/user/miniforge3/envs/pynguin/bin/pynguin:8 in <module>                                  │
│                                                                                                  │
│   5 from pynguin.cli import main                                                                 │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/cli.py:190 in main    │
│                                                                                                  │
│   187 │   set_configuration(parsed.config)                                                       │
│   188 │   if console is not None:                                                                │
│   189 │   │   with console.status("Running Pynguin..."):                                         │
│ ❱ 190 │   │   │   return run_pynguin().value                                                     │
│   191 │   else:                                                                                  │
│   192 │   │   return run_pynguin().value                                                         │
│   193                                                                                            │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/generator.py:111 in   │
│ run_pynguin                                                                                      │
│                                                                                                  │
│   108 │   """                                                                                    │
│   109 │   try:                                                                                   │
│   110 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱ 111 │   │   return _run()                                                                      │
│   112 │   finally:                                                                               │
│   113 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   114                                                                                            │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/generator.py:496 in   │
│ _run                                                                                             │
│                                                                                                  │
│   493                                                                                            │
│   494                                                                                            │
│   495 def _run() -> ReturnCode:                                                                  │
│ ❱ 496 │   if (setup_result := _setup_and_check()) is None:                                       │
│   497 │   │   return ReturnCode.SETUP_FAILED                                                     │
│   498 │   executor, test_cluster, constant_provider = setup_result                               │
│   499 │   # traces slices for test cases after execution                                         │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/generator.py:253 in   │
│ _setup_and_check                                                                                 │
│                                                                                                  │
│   250 │   │   set(config.configuration.statistics_output.coverage_metrics),                      │
│   251 │   │   dynamic_constant_provider,                                                         │
│   252 │   )                                                                                      │
│ ❱ 253 │   if not _load_sut(tracer):                                                              │
│   254 │   │   return None                                                                        │
│   255 │   if not _setup_report_dir():                                                            │
│   256 │   │   return None                                                                        │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/generator.py:164 in   │
│ _load_sut                                                                                        │
│                                                                                                  │
│   161 │   try:                                                                                   │
│   162 │   │   # We need to set the current thread ident so the import trace is recorded.         │
│   163 │   │   tracer.current_thread_identifier = threading.current_thread().ident                │
│ ❱ 164 │   │   importlib.import_module(config.configuration.module_name)                          │
│   165 │   except ImportError as ex:                                                              │
│   166 │   │   # A module could not be imported because some dependencies                         │
│   167 │   │   # are missing or it is malformed                                                   │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/importlib/__init__.py:126 in import_module  │
│                                                                                                  │
│   123 │   │   │   if character != '.':                                                           │
│   124 │   │   │   │   break                                                                      │
│   125 │   │   │   level += 1                                                                     │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)                            │
│   127                                                                                            │
│   128                                                                                            │
│   129 _RELOADING = {}                                                                            │
│ <frozen importlib._bootstrap>:1050 in _gcd_import                                                │
│ <frozen importlib._bootstrap>:1027 in _find_and_load                                             │
│ <frozen importlib._bootstrap>:1006 in _find_and_load_unlocked                                    │
│ <frozen importlib._bootstrap>:688 in _load_unlocked                                              │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/machi │
│ nery.py:56 in exec_module                                                                        │
│                                                                                                  │
│    53 │                                                                                          │
│    54 │   def exec_module(self, module):                                                         │
│    55 │   │   self._tracer.reset()                                                               │
│ ❱  56 │   │   super().exec_module(module)                                                        │
│    57 │   │   self._tracer.store_import_trace()                                                  │
│    58 │                                                                                          │
│    59 │   def get_code(self, fullname) -> CodeType:                                              │
│ <frozen importlib._bootstrap_external>:879 in exec_module                                        │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/machi │
│ nery.py:71 in get_code                                                                           │
│                                                                                                  │
│    68 │   │   """                                                                                │
│    69 │   │   to_instrument = cast(CodeType, super().get_code(fullname))                         │
│    70 │   │   assert to_instrument, "Failed to get code object of module."                       │
│ ❱  71 │   │   return self._transformer.instrument_module(to_instrument)                          │
│    72                                                                                            │
│    73                                                                                            │
│    74 def build_transformer(                                                                     │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:205 in instrument_module                                                           │
│                                                                                                  │
│    202 │   │   │   │   # Abort instrumentation, since we have already                            │
│    203 │   │   │   │   # instrumented this code object.                                          │
│    204 │   │   │   │   assert False, "Tried to instrument already instrumented module."          │
│ ❱  205 │   │   return self._instrument_code_recursive(module_code)                               │
│    206 │                                                                                         │
│    207 │   def _instrument_code_recursive(                                                       │
│    208 │   │   self,                                                                             │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:244 in _instrument_code_recursive                                                  │
│                                                                                                  │
│    241 │   │   for adapter in self._instrumentation_adapters:                                    │
│    242 │   │   │   adapter.visit_entry_node(real_entry_node.basic_block, code_object_id)         │
│    243 │   │   self._instrument_cfg(cfg, code_object_id)                                         │
│ ❱  244 │   │   return self._instrument_inner_code_objects(                                       │
│    245 │   │   │   cfg.bytecode_cfg().to_code(), code_object_id                                  │
│    246 │   │   )                                                                                 │
│    247                                                                                           │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:265 in _instrument_inner_code_objects                                              │
│                                                                                                  │
│    262 │   │   │   if isinstance(const, CodeType):                                               │
│    263 │   │   │   │   # The const is an inner code object                                       │
│    264 │   │   │   │   new_consts.append(                                                        │
│ ❱  265 │   │   │   │   │   self._instrument_code_recursive(                                      │
│    266 │   │   │   │   │   │   const, parent_code_object_id=parent_code_object_id                │
│    267 │   │   │   │   │   )                                                                     │
│    268 │   │   │   │   )                                                                         │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:243 in _instrument_code_recursive                                                  │
│                                                                                                  │
│    240 │   │   assert real_entry_node.basic_block is not None, "Basic block cannot be None."     │
│    241 │   │   for adapter in self._instrumentation_adapters:                                    │
│    242 │   │   │   adapter.visit_entry_node(real_entry_node.basic_block, code_object_id)         │
│ ❱  243 │   │   self._instrument_cfg(cfg, code_object_id)                                         │
│    244 │   │   return self._instrument_inner_code_objects(                                       │
│    245 │   │   │   cfg.bytecode_cfg().to_code(), code_object_id                                  │
│    246 │   │   )                                                                                 │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:290 in _instrument_cfg                                                             │
│                                                                                                  │
│    287 │   │   │   │   node.basic_block is not None                                              │
│    288 │   │   │   ), "Non artificial node does not have a basic block."                         │
│    289 │   │   │   for adapter in self._instrumentation_adapters:                                │
│ ❱  290 │   │   │   │   adapter.visit_node(cfg, code_object_id, node, node.basic_block)           │
│    291                                                                                           │
│    292                                                                                           │
│    293 class BranchCoverageInstrumentation(InstrumentationAdapter):                              │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:345 in visit_node                                                                  │
│                                                                                                  │
│    342 │   │   │   │   │   code_object_id=code_object_id,                                        │
│    343 │   │   │   │   )                                                                         │
│    344 │   │   │   elif maybe_jump.is_cond_jump():                                               │
│ ❱  345 │   │   │   │   predicate_id = self._instrument_cond_jump(                                │
│    346 │   │   │   │   │   code_object_id=code_object_id,                                        │
│    347 │   │   │   │   │   maybe_compare_idx=maybe_compare_idx,                                  │
│    348 │   │   │   │   │   jump=maybe_jump,                                                      │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:387 in _instrument_cond_jump                                                       │
│                                                                                                  │
│    384 │   │   │   and maybe_compare.opcode in op.OP_COMPARE                                     │
│    385 │   │   ):                                                                                │
│    386 │   │   │   assert maybe_compare_idx is not None                                          │
│ ❱  387 │   │   │   return self._instrument_compare_based_conditional_jump(                       │
│    388 │   │   │   │   block=block,                                                              │
│    389 │   │   │   │   code_object_id=code_object_id,                                            │
│    390 │   │   │   │   compare_idx=maybe_compare_idx,                                            │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/site-packages/pynguin/instrumentation/instr │
│ umentation.py:481 in _instrument_compare_based_conditional_jump                                  │
│                                                                                                  │
│    478 │   │   │   │   # bytecode library.                                                       │
│    479 │   │   │   │   compare = Compare.IS_NOT if operation.arg else Compare.IS                 │
│    480 │   │   │   case "CONTAINS_OP":                                                           │
│ ❱  481 │   │   │   │   compare = Compare.NOT_IN if operation.arg else Compare.IN                 │
│    482 │   │   │   case _:                                                                       │
│    483 │   │   │   │   raise RuntimeError(f"Unknown comparison OP {operation}")                  │
│    484                                                                                           │
│                                                                                                  │
│ /Users/user/miniforge3/envs/pynguin/lib/python3.10/enum.py:437 in __getattr__                  │
│                                                                                                  │
│    434 │   │   try:                                                                              │
│    435 │   │   │   return cls._member_map_[name]                                                 │
│    436 │   │   except KeyError:                                                                  │
│ ❱  437 │   │   │   raise AttributeError(name) from None                                          │
│    438 │                                                                                         │
│    439 │   def __getitem__(cls, name):                                                           │
│    440 │   │   return cls._member_map_[name]                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: IN

From what I understand, this functionality is still in development, which is fine since it's a free project. :) But perhaps, at least give the user a clearer message? I was kind of confused by the message AttributeError: IN.

And it also seems to me that in is a very popular operator, so please consider it a feature request. :)

Other than that, thanks and keep up the good job!

How to make pynguin continue even when coverage reach to 100%

I want to generate various test inputs, but pynguin seems to stop when coverage is 100%.
(I plan to use pynguin to generate robust test cases from the code of the model answer in order to evaluate code submitted by various users.)

StoppingConfiguration provides maximum options, but there are no minimum (at least) options.
Is there a way to keep pynguin continue?

If there is any `.so` (e.g., `_C.so`) file in a package, Pynguin stops generating tests and reports `FileNotFoundError: [Errno 2] No such file or directory: '.../_C.py'`

I am trying to generate tests for the project torchvision.

After compiling torchvision, a file named _C.so is generated inside the package torchvision. Running Pynguin on the compiled project results in the following error:
FileNotFoundError: [Errno 2] No such file or directory: '.../_C.py'.

I found a workaround for now (I just need to add an empty file named _C.py next to the _C.so file).

I looked into Pynguin's source code. I think the bug is inside the following function:
https://github.com/se2p/pynguin/blob/main/pynguin/analyses/seeding/constantseeding.py#L139, which is:

Module: pynguin/analyses/seeding/constantseeding.py
Function: _find_modules
Line: 139

The problem is that the function iter_modules returns .so files as well as .py files.
If I am right, this function needs to check if the modules returned by iter_modules are definitely .py files and skipped otherwise.

This bug (if it is a bug) can be reproduced by simply putting an empty .so file in a package and trying to generate tests for the modules inside that package, using Pynguin.

PS: awesome tool!

AssertionError when using LINE Coverage

I copied "queue_example.py" from the quickstart guide on https://pynguin.readthedocs.io/en/stable/user/quickstart.html.

When i try to get the coverage report "LINE i get the error:

Commands used:
pynguin --project-path ../Badge5 --output-path ./tmp/pynguin-results --module-name main -v --create-coverage-report True --seed 1629381673714481067 --coverage_metrics LINE

pynguin --project-path ../Badge5 --output-path ./tmp/pynguin-results --module-name main -v --create-coverage-report True --seed 1629381673714481067 --coverage_metrics BRANCH LINE

I just followed the guide. What am I doing wrong?

Test case timeout causes race condition/multiple instances

System Info:
Pynguin version '0.32.0.dev'
Ubuntu 22.04, Python 3.10.6
Or when inside docker, Debian Bullseye Python 3.10.10.
Occurs in either.

After a test case times out (with the error message mentioned here : #29 ) , Pynguin spawns another instance of itself with the exact same command line being run and starting from the beginning. The previous instance is not terminated, and attempting to terminate either proccess kills the other(s).

Besides the obvious issues with possibly overwriting Pynguin's outputted result/generated files, this also sometimes occurs repeatedly, spawning enough processes to take up all available RAM and throttles the CPU. Unfortunately, I could not reproduce it when running under a debugger capable of dealing with threads (PyCharm), so I have no helpful backtrace to provide, but here's an example of the output with a single '--verbose'.

[21:55:39] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:55:50]       ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

               ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                             
                                                                                                                                                                                                                                 
[21:55:51] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058537564621002                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:55:58] INFO     Initial Population, Coverage: 0.104413                                                                                                                                                   searchobserver.py:69
[21:56:09] INFO     Iteration:       1, Coverage: 0.108763                                                                                                                                                   searchobserver.py:73
[21:56:14] INFO     Iteration:       2, Coverage: 0.110628                                                                                                                                                   searchobserver.py:73
[21:56:18] INFO     Iteration:       3, Coverage: 0.114357                                                                                                                                                   searchobserver.py:73
[21:56:23] INFO     Iteration:       4, Coverage: 0.114978                                                                                                                                                   searchobserver.py:73
[21:56:29] INFO     Iteration:       5, Coverage: 0.114978                                                                                                                                                   searchobserver.py:73
[21:56:33] INFO     Iteration:       6, Coverage: 0.115600                                                                                                                                                   searchobserver.py:73
[21:56:35] INFO     Iteration:       7, Coverage: 0.118086                                                                                                                                                   searchobserver.py:73
[21:56:39] INFO     Iteration:       8, Coverage: 0.118707                                                                                                                                                   searchobserver.py:73
[21:56:44] INFO     Iteration:       9, Coverage: 0.130516                                                                                                                                                   searchobserver.py:73
[21:56:48] INFO     Iteration:      10, Coverage: 0.134245                                                                                                                                                   searchobserver.py:73
[21:56:49] ERROR    Exception in Thread: <Thread(Thread-2061 (_execute_test_case), started daemon 139705112209152)>                                                                                             execution.py:2075
                    ╭─────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮                  
                    │ /usr/local/lib/python3.10/threading.py:1016 in _bootstrap_inner                                                                                                                         │                  
                    │                                                                                                                                                                                         │                  
                    │   1013 │   │   │   │   _sys.setprofile(_profile_hook)                                                                                                                                   │                  
                    │   1014 │   │   │                                                                                                                                                                        │                  
                    │   1015 │   │   │   try:                                                                                                                                                                 │                  
                    │ ❱ 1016 │   │   │   │   self.run()                                                                                                                                                       │                  
                    │   1017 │   │   │   except:                                                                                                                                                              │                  
                    │   1018 │   │   │   │   self._invoke_excepthook(self)                                                                                                                                    │                  
                    │   1019 │   │   finally:                                                                                                                                                                 │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/threading.py:953 in run                                                                                                                                       │                  
                    │                                                                                                                                                                                         │                  
                    │    950 │   │   """                                                                                                                                                                      │                  
                    │    951 │   │   try:                                                                                                                                                                     │                  
                    │    952 │   │   │   if self._target is not None:                                                                                                                                         │                  
                    │ ❱  953 │   │   │   │   self._target(*self._args, **self._kwargs)                                                                                                                        │                  
                    │    954 │   │   finally:                                                                                                                                                                 │                  
                    │    955 │   │   │   # Avoid a refcycle if the thread is running a function with                                                                                                          │                  
                    │    956 │   │   │   # an argument that has a member that points to the thread.                                                                                                           │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/site-packages/pynguin/testcase/execution.py:2178 in _execute_test_case                                                                                        │                  
                    │                                                                                                                                                                                         │                  
                    │   2175 │   │   for idx, statement in enumerate(test_case.statements):                                                                                                                   │                  
                    │   2176 │   │   │   ast_node = self._before_statement_execution(statement, exec_ctx)                                                                                                     │                  
                    │   2177 │   │   │   exception = self.execute_ast(ast_node, exec_ctx)                                                                                                                     │                  
                    │ ❱ 2178 │   │   │   self._after_statement_execution(statement, exec_ctx, exception)                                                                                                      │                  
                    │   2179 │   │   │   if exception is not None:                                                                                                                                            │                  
                    │   2180 │   │   │   │   result.report_new_thrown_exception(idx, exception)                                                                                                               │                  
                    │   2181 │   │   │   │   break                                                                                                                                                            │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/site-packages/pynguin/testcase/execution.py:2277 in _after_statement_execution                                                                                │                  
                    │                                                                                                                                                                                         │                  
                    │   2274 │   │   # See comments in _before_statement_execution                                                                                                                            │                  
                    │   2275 │   │   if self.tracer.current_thread_identifier != threading.current_thread().ident:                                                                                            │                  
                    │   2276 │   │   │   # Kill this thread                                                                                                                                                   │                  
                    │ ❱ 2277 │   │   │   raise RuntimeError(                                                                                                                                                  │                  
                    │   2278 │   │   │   │   "The current thread shall not be executed any more, thus I kill it."                                                                                             │                  
                    │   2279 │   │   │   )                                                                                                                                                                    │                  
                    │   2280                                                                                                                                                                                  │                  
                    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                  
                    RuntimeError: The current thread shall not be executed any more, thus I kill it.
⠙ Running Pynguin...[21:56:52] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:56:52 PM] INFO     Iteration:      11, Coverage: 0.139838                                                                                                                                                searchobserver.py:73
[09:56:57 PM] INFO     Iteration:      12, Coverage: 0.141081                                                                                                                                                searchobserver.py:73
⠼ Running Pynguin...[21:56:57] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:57:01 PM] INFO     Iteration:      13, Coverage: 0.146053                                                                                                                                                searchobserver.py:73
                                                                                                                                                                                                                                 
⠙ Running Pynguin...[21:57:05] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠋ Running Pynguin...[21:57:05] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:57:06] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058610586713330                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[09:57:07 PM] INFO     Iteration:      14, Coverage: 0.148539                                                                                                                                                searchobserver.py:73                                                                                                                                                                                                  
                                                                                                                                                                                                                                 
[21:57:12] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058615406288740                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
⠹ Running Pynguin...[21:57:13] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠇ Running Pynguin...[21:57:13] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠼ Running Pynguin...[21:57:14] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠸ Running Pynguin...[21:57:14] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:57:17 PM] INFO     Iteration:      15, Coverage: 0.157241                                                                                                                                                searchobserver.py:73
[21:57:25] INFO     Initial Population, Coverage: 0.099441                                                                                                                                                   searchobserver.py:69
[21:57:29]     ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

 ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                           
                                                                                                                                                                                                                                 
[21:57:29] INFO     Initial Population, Coverage: 0.110006                                                                                                                                                   searchobserver.py:69
[21:57:30] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058622915104278                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
[21:57:31] INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:31] ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

                                                                                                                                                                                                  
                                                                                                                                                                                                                                 
[21:57:32] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058622959935474                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523                                                                                                                                                                                                  
                                                                                                                                                                                                                                 
[09:57:39 PM] INFO     Iteration:      17, Coverage: 0.167806                                                                                                                                                searchobserver.py:73
[21:57:40] ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call
                                                                                                                                                                                                  
                                                                                                                                                                                                                                
[21:57:40]           ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call
                                                                                                                                                                                        
                                                                                                                                                                                                                                 
[21:57:41] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
[21:57:41] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630145117367                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     Using seed 1680058630236044284                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:42] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630223084926                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:45]      ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                                             
                                                                                                                                                                                                                                 
[21:57:46] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630454705334                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[09:57:48 PM] INFO     Iteration:      18, Coverage: 0.172778                                                                                                                                                searchobserver.py:73
[21:57:51] INFO     Initial Population, Coverage: 0.112492                                                                                                                                                   searchobserver.py:69
[09:57:52 PM] INFO     Initial Population, Coverage: 0.123058                                                                                                                                                searchobserver.py:69
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427756 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427822 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427899 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427938 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427955 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427961 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427972 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
[21:57:53] INFO     Iteration:       1, Coverage: 0.100062                                                                                                                                                   searchobserver.py:73
[21:57:54] INFO     Iteration:       1, Coverage: 0.114357                                                                                                                                                   searchobserver.py:73
⠼ Running Pynguin...[21:57:59] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠦ Running Pynguin...[21:57:59] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…                                                                                                                                                             generator.py:113
           INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…                                                                                                                                                             generator.py:113
           INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…

At which point I had begun interrupting the process.

Incidentally, as seen in the log above, any file that imports urlllib appears to run into two errors that do not halt the program. These are the only logging ERRORs that occur. It does not appear related, but I'll put it here in case:

DEBUG Analysing method module.py:1188
urllib.request.URLopener.open_ftp
ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

...
DEBUG Analysing method module.py:1188
urllib.request.ftpwrapper.real_close
ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

Nothing else in the debug level log appears relevant, but I can provide it if wanted.

Unfortunately a bug in the bytecode module prevented it from handling EXTENDED_ARG NOPs in certain situations, which bottle generates. Backporting the fix was declined by the bytecode maintainers, so this means you will either have to:

A) Update Pynguin to be compatible with the latest bytecode version for which a PR fix I sent was accepted. I don't think this fix is released on PyPI yet, but could be easily installed from source.

B) Install a locally modified version of 0.13.0 with the fix shown here https://github.com/EmmaJaneBonestell/bytecode/blob/54a1af74f33dfa323d920540fc8bab8e18b7e64c/bytecode/concrete.py#L371 , or just install from my fork/branch directly: pip install git+https://github.com/EmmaJaneBonestell/[email protected]

After that, no special commands are required to reproduce. e.g.:

git clone https://github.com/bottlepy/bottle ./bottle
pynguin --project-path ./bottle --output-path ./pynguin_output --module-name bottle

This normally happens around 30 iterations. It may take fewer, as it did in the above log, or nearly 200. It has never managed to reach the 600 second default timeout/full coverage without occurring.

Doesn't work when run on a not supported Python version

(env) D:\Projects\Python\PynguinTest>pynguin
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "D:\Projects\Python\PynguinTest\env\Scripts\pynguin.exe_main.py", line 4, in
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin_init.py", line 9, in
import pynguin.generator as gen
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\generator.py", line 31, in
import pynguin.analyses.seeding as seeding # pylint: disable=consider-using-from-import
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\analyses\seeding.py", line 25, in
import pynguin.ga.testcasechromosome as tcc
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\ga\testcasechromosome.py", line 13, in
import pynguin.ga.chromosome as chrom
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\ga\chromosome.py", line 13, in
import pynguin.ga.computations as ff
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\ga\computations.py", line 17, in
from pynguin.testcase.execution import ExecutionTrace
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\testcase\execution.py", line 425, in
class ExecutionTracer:
File "D:\Projects\Python\PynguinTest\env\Lib\site-packages\pynguin\testcase\execution.py", line 460, in ExecutionTracer
Compare.IN: lambda val1, val2: (
^^^^^^^^^^
File "C:\Users\Jstar\AppData\Local\Programs\Python\Python311\Lib\enum.py", line 787, in getattr
raise AttributeError(name) from None
AttributeError: IN

Support more (older) Python versions

Is your feature request related to a problem? Please describe.

Currently, the README states that only Python 3.10 is supported. Python 3.11 is not supported yet, and support for Python < 3.10 was apparently dropped.

Environments that cannot run Python 3.10 (typically banks and similar, large institutions) are hence unable to use Pynguin. Those organisations may need Pynguin, though, so that they can stop doing manual testing.

Describe the solution you'd like

I'd like to understand whether there is a hard technical reason for supporting only Python 3.10.

If that's not the case we should make the code as backward-compatible as possible, and have a test matrix that runs the test suite also against older Python versions.

Describe alternatives you've considered

I don't see any, for the moment.

Additional context

I work in a restricted corporate environment and try to beef up the test suite, so that we can demonstrate that automated tests actually make sense.

I run Pynguin 0.17.0 with Python 3.9 on Windows, and executing pynguin results in the following error:

pynguin (click to expand)

$ pynguin
Traceback (most recent call last): 
  File "D:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main 
    return _run_code(code, main_globals, None,
  File "D:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\python\venv\Scripts\pynguin.exe\__main__.py", line 4, in <module>
  File "D:\python\venv\lib\site-packages\pynguin\__init__.py", line 9, in <module>
    import pynguin.generator as gen
  File "D:\python\venv\lib\site-packages\pynguin\generator.py", line 31, in <module>
    import pynguin.analyses.seeding.initialpopulationseeding as initpopseeding
  File "D:\python\venv\lib\site-packages\pynguin\analyses\seeding\initialpopulationseeding.py", line 17, in <module>
    import pynguin.ga.testcasechromosome as tcc
  File "D:\python\venv\lib\site-packages\pynguin\ga\testcasechromosome.py", line 13, in <module>
    import pynguin.ga.chromosome as chrom
  File "D:\python\venv\lib\site-packages\pynguin\ga\chromosome.py", line 13, in <module>
    import pynguin.ga.computations as ff
  File "D:\python\venv\lib\site-packages\pynguin\ga\computations.py", line 17, in <module>
    from pynguin.testcase.execution import ExecutionTrace
  File "D:\python\venv\lib\site-packages\pynguin\testcase\execution.py", line 428, in <module>
    class ExecutionTracer:
  File "D:\python\venv\lib\site-packages\pynguin\testcase\execution.py", line 463, in ExecutionTracer
    Compare.IN: lambda val1, val2: (
  File "D:\Program Files\Python39\lib\enum.py", line 429, in __getattr__
    raise AttributeError(name) from None

I'd be willing to make the necessary contributions that make Pynguin run with older Pythons, once I understand the situation better.

Error when generating tests for a file using numpy as a dependency

Describe the bug
When I try to generate tests for a file that imports numpy, Pynguin crashes.

To Reproduce
Steps to reproduce the behaviour:

Use Pynguin version '0.34.0'
Use the following (minimal) code as a subject for test generation:

import numpy

def foo(x: int) -> int:
    return x * 2

Use the following command line arguments to Pynguin:

--module-name <module-name>
--project-path <project-name>
--output-path <output-name>

Give the error (stack trace, etc) you are encountering:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/lucas/.conda/envs/2324-master-thesis/bin/pynguin:8 in <module>                             │
│                                                                                                  │
│   5 from pynguin.cli import main                                                                 │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/cli.py:193 in main                              │
│                                                                                                  │
│   190 │   set_configuration(parsed.config)                                                       │
│   191 │   if console is not None:                                                                │
│   192 │   │   with console.status("Running Pynguin..."):                                         │
│ ❱ 193 │   │   │   return run_pynguin().value                                                     │
│   194 │   else:                                                                                  │
│   195 │   │   return run_pynguin().value                                                         │
│   196                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:108 in run_pynguin                 │
│                                                                                                  │
│   105 │   """                                                                                    │
│   106 │   try:                                                                                   │
│   107 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱ 108 │   │   return _run()                                                                      │
│   109 │   finally:                                                                               │
│   110 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   111                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:507 in _run                        │
│                                                                                                  │
│   504                                                                                            │
│   505                                                                                            │
│   506 def _run() -> ReturnCode:                                                                  │
│ ❱ 507 │   if (setup_result := _setup_and_check()) is None:                                       │
│   508 │   │   return ReturnCode.SETUP_FAILED                                                     │
│   509 │   executor, test_cluster, constant_provider = setup_result                               │
│   510 │   # traces slices for test cases after execution                                         │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:258 in _setup_and_check            │
│                                                                                                  │
│   255 │                                                                                          │
│   256 │   # Analyzing the SUT should not cause any coverage.                                     │
│   257 │   tracer.disable()                                                                       │
│ ❱ 258 │   if (test_cluster := _setup_test_cluster()) is None:                                    │
│   259 │   │   return None                                                                        │
│   260 │   tracer.enable()                                                                        │
│   261                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:114 in _setup_test_cluster         │
│                                                                                                  │
│   111                                                                                            │
│   112                                                                                            │
│   113 def _setup_test_cluster() -> ModuleTestCluster | None:                                     │
│ ❱ 114 │   test_cluster = generate_test_cluster(                                                  │
│   115 │   │   config.configuration.module_name,                                                  │
│   116 │   │   config.configuration.type_inference.type_inference_strategy,                       │
│   117 │   │   query_type4py=config.configuration.type_inference.type4py,                         │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1447 in                      │
│ generate_test_cluster                                                                            │
│                                                                                                  │
│   1444 │   Returns:                                                                              │
│   1445 │   │   A new test cluster for the given module                                           │
│   1446 │   """                                                                                   │
│ ❱ 1447 │   return analyse_module(                                                                │
│   1448 │   │   parse_module(module_name, query_type4py=query_type4py),                           │
│   1449 │   │   type_inference_strategy,                                                          │
│   1450 │   │   query_type4py=query_type4py,                                                      │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1422 in analyse_module       │
│                                                                                                  │
│   1419 │   │   A test cluster for the module                                                     │
│   1420 │   """                                                                                   │
│   1421 │   test_cluster = ModuleTestCluster(linenos=parsed_module.linenos)                       │
│ ❱ 1422 │   __resolve_dependencies(                                                               │
│   1423 │   │   root_module=parsed_module,                                                        │
│   1424 │   │   type_inference_strategy=type_inference_strategy,                                  │
│   1425 │   │   test_cluster=test_cluster,                                                        │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1304 in                      │
│ __resolve_dependencies                                                                           │
│                                                                                                  │
│   1301 │   │   )                                                                                 │
│   1302 │   │                                                                                     │
│   1303 │   │   # Analyze all functions found in the current module                               │
│ ❱ 1304 │   │   __analyse_included_functions(                                                     │
│   1305 │   │   │   module=current_module,                                                        │
│   1306 │   │   │   root_module_name=root_module.module_name,                                     │
│   1307 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1398 in                      │
│ __analyse_included_functions                                                                     │
│                                                                                                  │
│   1395 │   │   │   func_name=current.__qualname__,                                               │
│   1396 │   │   │   func=current,                                                                 │
│   1397 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│ ❱ 1398 │   │   │   module_tree=parse_results[current.__module__].syntax_tree,                    │
│   1399 │   │   │   type4py_data=parse_results[current.__module__].type4py_data,                  │
│   1400 │   │   │   test_cluster=test_cluster,                                                    │
│   1401 │   │   │   add_to_test=current.__module__ == root_module_name,                           │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1248 in __missing__          │
│                                                                                                  │
│   1245 │                                                                                         │
│   1246 │   def __missing__(self, key):                                                           │
│   1247 │   │   # Parse module on demand                                                          │
│ ❱ 1248 │   │   res = self[key] = parse_module(key, query_type4py=self._query_type4py)            │
│   1249 │   │   return res                                                                        │
│   1250                                                                                           │
│   1251                                                                                           │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:247 in parse_module          │
│                                                                                                  │
│    244 │   Returns:                                                                              │
│    245 │   │   A tuple of the imported module type and its optional AST                          │
│    246 │   """                                                                                   │
│ ❱  247 │   module = importlib.import_module(module_name)                                         │
│    248 │   type4py_data: Type4pyData | None = None                                               │
│    249 │   syntax_tree: astroid.Module | None = None                                             │
│    250 │   linenos: int = -1                                                                     │
│                                                                                                  │
│ /home/lucas/.conda/envs/2324-master-thesis/lib/python3.10/importlib/__init__.py:126 in           │
│ import_module                                                                                    │
│                                                                                                  │
│   123 │   │   │   if character != '.':                                                           │
│   124 │   │   │   │   break                                                                      │
│   125 │   │   │   level += 1                                                                     │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)                            │
│   127                                                                                            │
│   128                                                                                            │
│   129 _RELOADING = {}                                                                            │
│ in _gcd_import:1050                                                                              │
│ in _find_and_load:1027                                                                           │
│ in _find_and_load_unlocked:1004                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'numpy.char'

Expected behavior
Pynguin should have generated a test for the function foo.

Software Version

OS: fedora 39
Python version: 3.10.13
Pynguin Version: 0.34.0

No signature found error when generating tests on a Tensorflow file

Describe the bug
When I try to generate tests for the tf_import_time.py file in the Tensorflow project, Pynguin crashes.

To Reproduce
Steps to reproduce the behaviour:

Use Pynguin version '0.34.0'
Use the following (minimal) code as a subject for test generation:

The problem seems to come from a C extension module so I think that the easiest way to reproduce the bug is to clone the tensorflow git repository and to install it from there.

Use the following command line arguments to Pynguin:

--module-name tf_import_time
--project-path <path-to-tensorflow-git-directory>/tensorflow/python/tools/
--output-path <output-name>

Give the error (stack trace, etc) you are encountering:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/lucas/.conda/envs/2324-master-thesis/bin/pynguin:8 in <module>                             │
│                                                                                                  │
│   5 from pynguin.cli import main                                                                 │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/cli.py:193 in main                              │
│                                                                                                  │
│   190 │   set_configuration(parsed.config)                                                       │
│   191 │   if console is not None:                                                                │
│   192 │   │   with console.status("Running Pynguin..."):                                         │
│ ❱ 193 │   │   │   return run_pynguin().value                                                     │
│   194 │   else:                                                                                  │
│   195 │   │   return run_pynguin().value                                                         │
│   196                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:108 in run_pynguin                 │
│                                                                                                  │
│   105 │   """                                                                                    │
│   106 │   try:                                                                                   │
│   107 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱ 108 │   │   return _run()                                                                      │
│   109 │   finally:                                                                               │
│   110 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   111                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:507 in _run                        │
│                                                                                                  │
│   504                                                                                            │
│   505                                                                                            │
│   506 def _run() -> ReturnCode:                                                                  │
│ ❱ 507 │   if (setup_result := _setup_and_check()) is None:                                       │
│   508 │   │   return ReturnCode.SETUP_FAILED                                                     │
│   509 │   executor, test_cluster, constant_provider = setup_result                               │
│   510 │   # traces slices for test cases after execution                                         │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:258 in _setup_and_check            │
│                                                                                                  │
│   255 │                                                                                          │
│   256 │   # Analyzing the SUT should not cause any coverage.                                     │
│   257 │   tracer.disable()                                                                       │
│ ❱ 258 │   if (test_cluster := _setup_test_cluster()) is None:                                    │
│   259 │   │   return None                                                                        │
│   260 │   tracer.enable()                                                                        │
│   261                                                                                            │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/generator.py:114 in _setup_test_cluster         │
│                                                                                                  │
│   111                                                                                            │
│   112                                                                                            │
│   113 def _setup_test_cluster() -> ModuleTestCluster | None:                                     │
│ ❱ 114 │   test_cluster = generate_test_cluster(                                                  │
│   115 │   │   config.configuration.module_name,                                                  │
│   116 │   │   config.configuration.type_inference.type_inference_strategy,                       │
│   117 │   │   query_type4py=config.configuration.type_inference.type4py,                         │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1447 in                      │
│ generate_test_cluster                                                                            │
│                                                                                                  │
│   1444 │   Returns:                                                                              │
│   1445 │   │   A new test cluster for the given module                                           │
│   1446 │   """                                                                                   │
│ ❱ 1447 │   return analyse_module(                                                                │
│   1448 │   │   parse_module(module_name, query_type4py=query_type4py),                           │
│   1449 │   │   type_inference_strategy,                                                          │
│   1450 │   │   query_type4py=query_type4py,                                                      │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1422 in analyse_module       │
│                                                                                                  │
│   1419 │   │   A test cluster for the module                                                     │
│   1420 │   """                                                                                   │
│   1421 │   test_cluster = ModuleTestCluster(linenos=parsed_module.linenos)                       │
│ ❱ 1422 │   __resolve_dependencies(                                                               │
│   1423 │   │   root_module=parsed_module,                                                        │
│   1424 │   │   type_inference_strategy=type_inference_strategy,                                  │
│   1425 │   │   test_cluster=test_cluster,                                                        │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1294 in                      │
│ __resolve_dependencies                                                                           │
│                                                                                                  │
│   1291 │   │   │   continue                                                                      │
│   1292 │   │                                                                                     │
│   1293 │   │   # Analyze all classes found in the current module                                 │
│ ❱ 1294 │   │   __analyse_included_classes(                                                       │
│   1295 │   │   │   module=current_module,                                                        │
│   1296 │   │   │   root_module_name=root_module.module_name,                                     │
│   1297 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1354 in                      │
│ __analyse_included_classes                                                                       │
│                                                                                                  │
│   1351 │   │                                                                                     │
│   1352 │   │   type_info = test_cluster.type_system.to_type_info(current)                        │
│   1353 │   │                                                                                     │
│ ❱ 1354 │   │   __analyse_class(                                                                  │
│   1355 │   │   │   type_info=type_info,                                                          │
│   1356 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│   1357 │   │   │   module_tree=parse_results[current.__module__].syntax_tree,                    │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/module.py:1110 in __analyse_class      │
│                                                                                                  │
│   1107 │   else:                                                                                 │
│   1108 │   │   generic = GenericConstructor(                                                     │
│   1109 │   │   │   type_info,                                                                    │
│ ❱ 1110 │   │   │   test_cluster.type_system.infer_type_info(                                     │
│   1111 │   │   │   │   type_info.raw_type.__init__,                                              │
│   1112 │   │   │   │   type4py_data=find_predicted_signature(                                    │
│   1113 │   │   │   │   │   type4py_data, type_info.qualname + ".__init__", type_info.qualname    │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/typesystem.py:1529 in infer_type_info  │
│                                                                                                  │
│   1526 │   │   """                                                                               │
│   1527 │   │   match type_inference_strategy:                                                    │
│   1528 │   │   │   case config.TypeInferenceStrategy.TYPE_HINTS:                                 │
│ ❱ 1529 │   │   │   │   return self.infer_signature(                                              │
│   1530 │   │   │   │   │   method, type4py_data, self.type_hints_provider                        │
│   1531 │   │   │   │   )                                                                         │
│   1532 │   │   │   case config.TypeInferenceStrategy.NONE:                                       │
│                                                                                                  │
│ /home/lucas/Documents/GitHub/pynguin/src/pynguin/analyses/typesystem.py:1593 in infer_signature  │
│                                                                                                  │
│   1590 │   │   Returns:                                                                          │
│   1591 │   │   │   The inference result                                                          │
│   1592 │   │   """                                                                               │
│ ❱ 1593 │   │   method_signature = inspect.signature(method)                                      │
│   1594 │   │   hints = type_hint_provider(method)                                                │
│   1595 │   │   parameters: dict[str, ProperType] = {}                                            │
│   1596 │   │   type4py_parameters: dict[str, list[ProperType]] = {}                              │
│                                                                                                  │
│ /home/lucas/.conda/envs/2324-master-thesis/lib/python3.10/inspect.py:3254 in signature           │
│                                                                                                  │
│   3251                                                                                           │
│   3252 def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
│   3253 │   """Get a signature object for the passed callable."""                                 │
│ ❱ 3254 │   return Signature.from_callable(obj, follow_wrapped=follow_wrapped,                    │
│   3255 │   │   │   │   │   │   │   │      globals=globals, locals=locals, eval_str=eval_str)     │
│   3256                                                                                           │
│   3257                                                                                           │
│                                                                                                  │
│ /home/lucas/.conda/envs/2324-master-thesis/lib/python3.10/inspect.py:3002 in from_callable       │
│                                                                                                  │
│   2999 │   def from_callable(cls, obj, *,                                                        │
│   3000 │   │   │   │   │     follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
│   3001 │   │   """Constructs Signature for the given callable object."""                         │
│ ❱ 3002 │   │   return _signature_from_callable(obj, sigcls=cls,                                  │
│   3003 │   │   │   │   │   │   │   │   │   │   follow_wrapper_chains=follow_wrapped,             │
│   3004 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
│   3005                                                                                           │
│                                                                                                  │
│ /home/lucas/.conda/envs/2324-master-thesis/lib/python3.10/inspect.py:2468 in                     │
│ _signature_from_callable                                                                         │
│                                                                                                  │
│   2465 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
│   2466 │                                                                                         │
│   2467 │   if _signature_is_builtin(obj):                                                        │
│ ❱ 2468 │   │   return _signature_from_builtin(sigcls, obj,                                       │
│   2469 │   │   │   │   │   │   │   │   │      skip_bound_arg=skip_bound_arg)                     │
│   2470 │                                                                                         │
│   2471 │   if isinstance(obj, functools.partial):                                                │
│                                                                                                  │
│ /home/lucas/.conda/envs/2324-master-thesis/lib/python3.10/inspect.py:2275 in                     │
│ _signature_from_builtin                                                                          │
│                                                                                                  │
│   2272 │                                                                                         │
│   2273 │   s = getattr(func, "__text_signature__", None)                                         │
│   2274 │   if not s:                                                                             │
│ ❱ 2275 │   │   raise ValueError("no signature found for builtin {!r}".format(func))              │
│   2276 │                                                                                         │
│   2277 │   return _signature_fromstr(cls, func, s, skip_bound_arg)                               │
│   2278                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: no signature found for builtin <instancemethod __init__ at 0x7f1c03e45960>

Expected behavior
Pynguin should have tried to generate a test for file.

Software Version

OS: fedora 39
Python version: 3.10.13
Pynguin Version: 0.34.0

Additional context
I'm trying to reproduce the results of this paper, that is why it is a very specific case.

[Windows 10] pynguin cannot start, looking for folder name "python.exe"

Hi, when run under Windows 10 pynguin produces a weird path and fails to run. Error below:

user@... MINGW64 ~/git/dsdatabase (feature/orm-phase-2)
$ poetry run pynguin --algorithm RANDOM --project_path ./src/dsDataBase --output_path ./src/testgen --module_name utils -v --budget 10
[17:24:46] INFO     Start Pynguin Test Generation…                                                                                                                                                                             generator.py:89
           INFO     Instrumented compare_op                                                                                                                                                                             instrumentation.py:715
           INFO     Stop Pynguin Test Generation…                                                                                                                                                                              generator.py:92
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\runpy.py:197 in _run_module_as_main  │
│                                                                                                  │
│   194 │   main_globals = sys.modules["__main__"].__dict__                                        │
│   195 │   if alter_argv:                                                                         │
│   196 │   │   sys.argv[0] = mod_spec.origin                                                      │
│ ❱ 197 │   return _run_code(code, main_globals, None,                                             │
│   198 │   │   │   │   │    "__main__", mod_spec)                                                 │
│   199                                                                                            │
│   200 def run_module(mod_name, init_globals=None,                                                │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\runpy.py:87 in _run_code             │
│                                                                                                  │
│    84 │   │   │   │   │      __loader__ = loader,                                                │
│    85 │   │   │   │   │      __package__ = pkg_name,                                             │
│    86 │   │   │   │   │      __spec__ = mod_spec)                                                │
│ ❱  87 │   exec(code, run_globals)                                                                │
│    88 │   return run_globals                                                                     │
│    89                                                                                            │
│    90 def _run_module_code(code, init_globals=None,                                              │
│                                                                                                  │
│ C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\dsdatabase-LPBksMnQ-py3.9\Scripts\pyng │
│ uin.exe\__main__.py:7 in <module>                                                                │
│                                                                                                  │
│ [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\pypoetry\\Cache\\virtu │
│ alenvs\\dsdatabase-LPBksMnQ-py3.9\\Scripts\\pynguin.exe\\__main__.py'

The rest of the exception I think is not the problem, but here it goes:

│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\cli.py:150 in main                                                                 │
│                                                                                                  │
│   147 │                                                                                          │
│   148 │   set_configuration(parsed.config)                                                       │
│   149 │   with console.status("Running Pynguin..."):                                             │
│ ❱ 150 │   │   return run_pynguin().value                                                         │
│   151                                                                                            │
│   152                                                                                            │
│   153 if __name__ == "__main__":                                                                 │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\generator.py:90 in run_pynguin                                                     │
│                                                                                                  │
│    87 │   """                                                                                    │
│    88 │   try:                                                                                   │
│    89 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱  90 │   │   return _run()                                                                      │
│    91 │   finally:                                                                               │
│    92 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│    93                                                                                            │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\generator.py:224 in _run                                                           │
│                                                                                                  │
│   221                                                                                            │
│   222                                                                                            │
│   223 def _run() -> ReturnCode:                                                                  │
│ ❱ 224 │   if (setup_result := _setup_and_check()) is None:                                       │
│   225 │   │   return ReturnCode.SETUP_FAILED                                                     │
│   226 │   executor, test_cluster = setup_result                                                  │
│   227                                                                                            │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\generator.py:183 in _setup_and_check                                               │
│                                                                                                  │
│   180 │   tracer = _setup_import_hook()                                                          │
│   181 │   if not _load_sut(tracer):                                                              │
│   182 │   │   return None                                                                        │
│ ❱ 183 │   if (test_cluster := _setup_test_cluster()) is None:                                    │
│   184 │   │   return None                                                                        │
│   185 │   executor = TestCaseExecutor(tracer)                                                    │
│   186 │   _track_sut_data(tracer, test_cluster)                                                  │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\generator.py:97 in _setup_test_cluster                                             │
│                                                                                                  │
│    94                                                                                            │
│    95 def _setup_test_cluster() -> Optional[TestCluster]:                                        │
│    96 │   with Timer(name="Test-cluster generation time", logger=None):                          │
│ ❱  97 │   │   test_cluster = TestClusterGenerator(                                               │
│    98 │   │   │   config.configuration.module_name                                               │
│    99 │   │   ).generate_cluster()                                                               │
│   100 │   │   if test_cluster.num_accessible_objects_under_test() == 0:                          │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\setup\testclustergenerator.py:110 in generate_cluster                              │
│                                                                                                  │
│   107 │   │   │   self._test_cluster.add_generator(generic_function)                             │
│   108 │   │   │   self._test_cluster.add_accessible_object_under_test(generic_function)          │
│   109 │   │   │   self._add_callable_dependencies(generic_function, 1)                           │
│ ❱ 110 │   │   self._resolve_dependencies_recursive()                                             │
│   111 │   │   return self._test_cluster                                                          │
│   112 │                                                                                          │
│   113 │   def _add_callable_dependencies(                                                        │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\setup\testclustergenerator.py:234 in _resolve_dependencies_recursive               │
│                                                                                                  │
│   231 │   │   """Resolve the currently open dependencies."""                                     │
│   232 │   │   while self._dependencies_to_solve:                                                 │
│   233 │   │   │   to_solve = self._dependencies_to_solve.pop()                                   │
│ ❱ 234 │   │   │   self._add_dependency(                                                          │
│   235 │   │   │   │   to_solve.dependency_type, to_solve.recursion_level + 1, False              │
│   236 │   │   │   )                                                                              │
│   237                                                                                            │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\setup\testclustergenerator.py:162 in _add_dependency                               │
│                                                                                                  │
│   159 │   │   self._analyzed_classes.add(klass)                                                  │
│   160 │   │   self._logger.debug("Analyzing class %s", klass)                                    │
│   161 │   │   generic_constructor = GenericConstructor(                                          │
│ ❱ 162 │   │   │   klass, self._inference.infer_type_info(klass.__init__)[0]                      │
│   163 │   │   )                                                                                  │
│   164 │   │   if self._discard_accessible_with_missing_type_hints(generic_constructor):          │
│   165 │   │   │   return                                                                         │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\typeinference\typeinference.py:75 in infer_type_info                               │
│                                                                                                  │
│   72 │   │   """                                                                                 │
│   73 │   │   method_types: List[InferredSignature] = []                                          │
│   74 │   │   for strategy in self._strategies:                                                   │
│ ❱ 75 │   │   │   method_types.append(strategy.infer_type_info(method))                           │
│   76 │   │   return method_types                                                                 │
│   77                                                                                             │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\typeinference\typehintsstrategy.py:27 in infer_type_info                           │
│                                                                                                  │
│   24 │   def infer_type_info(self, method: Callable) -> InferredSignature:                       │
│   25 │   │   if inspect.isclass(method) and hasattr(method, "__init__"):                         │
│   26 │   │   │   return self._infer_type_info_for_callable(getattr(method, "__init__"))          │
│ ❱ 27 │   │   return self._infer_type_info_for_callable(method)                                   │
│   28 │                                                                                           │
│   29 │   @staticmethod                                                                           │
│   30 │   def _infer_type_info_for_callable(method: Callable) -> InferredSignature:               │
│                                                                                                  │
│ c:\users\user\appdata\local\pypoetry\cache\virtualenvs\dsdatabase-lpbksmnq-py3.9\lib\site-pac │
│ kages\pynguin\typeinference\typehintsstrategy.py:33 in _infer_type_info_for_callable             │
│                                                                                                  │
│   30 │   def _infer_type_info_for_callable(method: Callable) -> InferredSignature:               │
│   31 │   │   signature = inspect.signature(method)                                               │
│   32 │   │   parameters: Dict[str, Optional[type]] = {}                                          │
│ ❱ 33 │   │   hints = typing.get_type_hints(method)                                               │
│   34 │   │   for param_name in signature.parameters:                                             │
│   35 │   │   │   if param_name == "self":                                                        │
│   36 │   │   │   │   continue                                                                    │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:1449 in get_type_hints     │
│                                                                                                  │
│   1446 │   │   │   value = type(None)                                                            │
│   1447 │   │   if isinstance(value, str):                                                        │
│   1448 │   │   │   value = ForwardRef(value)                                                     │
│ ❱ 1449 │   │   value = _eval_type(value, globalns, localns)                                      │
│   1450 │   │   if name in defaults and defaults[name] is None:                                   │
│   1451 │   │   │   value = Optional[value]                                                       │
│   1452 │   │   hints[name] = value                                                               │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:283 in _eval_type          │
│                                                                                                  │
│    280 │   with recursive ForwardRef.                                                            │
│    281 │   """                                                                                   │
│    282 │   if isinstance(t, ForwardRef):                                                         │
│ ❱  283 │   │   return t._evaluate(globalns, localns, recursive_guard)                            │
│    284 │   if isinstance(t, (_GenericAlias, GenericAlias)):                                      │
│    285 │   │   ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__a  │
│    286 │   │   if ev_args == t.__args__:                                                         │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:543 in _evaluate           │
│                                                                                                  │
│    540 │   │   │   │   "Forward references must evaluate to types.",                             │
│    541 │   │   │   │   is_argument=self.__forward_is_argument__,                                 │
│    542 │   │   │   )                                                                             │
│ ❱  543 │   │   │   self.__forward_value__ = _eval_type(                                          │
│    544 │   │   │   │   type_, globalns, localns, recursive_guard | {self.__forward_arg__}        │
│    545 │   │   │   )                                                                             │
│    546 │   │   │   self.__forward_evaluated__ = True                                             │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:285 in _eval_type          │
│                                                                                                  │
│    282 │   if isinstance(t, ForwardRef):                                                         │
│    283 │   │   return t._evaluate(globalns, localns, recursive_guard)                            │
│    284 │   if isinstance(t, (_GenericAlias, GenericAlias)):                                      │
│ ❱  285 │   │   ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__a  │
│    286 │   │   if ev_args == t.__args__:                                                         │
│    287 │   │   │   return t                                                                      │
│    288 │   │   if isinstance(t, GenericAlias):                                                   │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:285 in <genexpr>           │
│                                                                                                  │
│    282 │   if isinstance(t, ForwardRef):                                                         │
│    283 │   │   return t._evaluate(globalns, localns, recursive_guard)                            │
│    284 │   if isinstance(t, (_GenericAlias, GenericAlias)):                                      │
│ ❱  285 │   │   ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__a  │
│    286 │   │   if ev_args == t.__args__:                                                         │
│    287 │   │   │   return t                                                                      │
│    288 │   │   if isinstance(t, GenericAlias):                                                   │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:283 in _eval_type          │
│                                                                                                  │
│    280 │   with recursive ForwardRef.                                                            │
│    281 │   """                                                                                   │
│    282 │   if isinstance(t, ForwardRef):                                                         │
│ ❱  283 │   │   return t._evaluate(globalns, localns, recursive_guard)                            │
│    284 │   if isinstance(t, (_GenericAlias, GenericAlias)):                                      │
│    285 │   │   ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__a  │
│    286 │   │   if ev_args == t.__args__:                                                         │
│                                                                                                  │
│ C:\Users\user\AppData\Local\Programs\Python\Python39\lib\typing.py:539 in _evaluate           │
│                                                                                                  │
│    536 │   │   │   elif localns is None:                                                         │
│    537 │   │   │   │   localns = globalns                                                        │
│    538 │   │   │   type_ =_type_check(                                                           │
│ ❱  539 │   │   │   │   eval(self.__forward_code__, globalns, localns),                           │
│    540 │   │   │   │   "Forward references must evaluate to types.",                             │
│    541 │   │   │   │   is_argument=self.__forward_is_argument__,                                 │
│    542 │   │   │   )                                                                             │
│ <string>:1 in <module>                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NameError: name 'ExtensionDtype' is not defined

Error for python 3.11 File "<frozen runpy>, line 198, in _ron_module_as_main

If I try to use pynguin with following command and Python 3.11.1 on Windows:

pynguin --project-path ./src --output-path ./test --module-name example

I get following error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main                                                                   
  File "<frozen runpy>", line 88, in _run_code                                                                              
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Scripts\pynguin.exe\__main__.py", line 4, in <module>               
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\__init__.py", line 9, in <module>         
    import pynguin.generator as gen                                                                                         
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\generator.py", line 31, in <module>       
    import pynguin.analyses.seeding as seeding  # pylint: disable=consider-using-from-import                                
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                              
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\analyses\seeding.py", line 25, in <module>
    import pynguin.ga.testcasechromosome as tcc                                                                             
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\ga\testcasechromosome.py", line 13, in <module>
    import pynguin.ga.chromosome as chrom
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\ga\chromosome.py", line 13, in <module>
    import pynguin.ga.computations as ff
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\ga\computations.py", line 17, in <module>
    from pynguin.testcase.execution import ExecutionTrace
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\testcase\execution.py", line 425, in <module>
    class ExecutionTracer:
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\site-packages\pynguin\testcase\execution.py", line 460, in ExecutionTracer
    Compare.IN: lambda val1, val2: (
    ^^^^^^^^^^
  File "C:\python_env\App\WinPython\python-3.11.1.amd64\Lib\enum.py", line 787, in __getattr__
    raise AttributeError(name) from None
AttributeError: IN

=> Could you please update pynguin to work with Python 3.11?

How to avoid digging deep into the dependencies

Hi,

In my project, I am using dependencies such as sqlalchemy, boto3, and pynguin reports an error as -
│ /work/venvs/py310/lib/python3.10/site-packages/sqlalchemy/orm/decl_api.py:309 in cascading │ │ │ │ 306 │ │ │ │ 307 │ │ │ │ 308 │ │ """ │ │ ❱ 309 │ │ return cls._stateful(cascading=True) │ │ 310 │ │ 311 │ │ 312 class _stateful_declared_attr(declared_attr): │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: _stateful_declared_attr._stateful() missing 1 required positional argument: 'self'

My project runs fine although.

Is there a way to make penguin avoid looking deeper into the code, which I have not written such as these.

What is the guidelines to ignore these dependencies, shall we mock them from our side before running penguin. Please guide.

Thanks.

The type information from type hints is not be used or from the type4py in the module under test.

Describe the bug
A clear and concise description of what the bug is.
When running the example.py in the quick start guide, although the example provides type annotation information for input and output data (input is int, output is str), the generated unit test code still includes test cases with non-int inputs. And when using type4py to obtain type information, it also does not take effect.
To Reproduce
Steps to reproduce the behaviour:

Use Pynguin version '0.35.0'
Use the following (minimal) code as a subject for test generation: './docs/source/_static/example.py'
Use the following command line arguments to Pynguin: 'pynguin --project-path ./docs/source/_static/ --output-path ../tmp/pynguin-results --module-name example -v'

Expected behavior
All generated unit test cases have input types as int, just as shown in the official documentation.

Screenshots
The generated unit test cases:

The expected generated unit test cases:

Software Version (please complete the following information):

OS: [ubuntu20.04]
Python version [3.10.13]
Pynguin Version 0.35.0

TypeError: unsupported operand type(s) for |: 'NoneType' and 'NoneType'

Describe the bug
Error when generating tests: TypeError: unsupported operand type(s) for |: 'NoneType' and 'NoneType'

To Reproduce
Steps to reproduce the behaviour:

Use Pynguin version '0.34.0'
Use the following (minimal) code as a subject for test generation:

import os
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt


class FilterCorrelatedFeatures:

    def __init__(self, corr_threshold=0.80):
        self.corr_threshold = corr_threshold
        self.correlated_features = []
        self.corr_matrix = pd.DataFrame()

Use the following command line arguments to Pynguin:
python -m pynguin --output-path . -v --project-path src/data_preprocessing/ --module-name hello
Give the error (stack trace, etc) you are encountering: '...'
pynguin-log.txt

Expected behavior
generate a simple test.

Screenshots
NA

Software Version (please complete the following information):

OS: [Ubuntu 20.04.6 LTS]
Python version [3.10.13]
Pynguin Version [0.34.0]

Support for asyncio

hello.

As the title suggests, function test code generation using the async / await syntax doesn't seem to work.
This is just an async of the documentation code:

async def triangle(x: int, y: int, z: int) -> str:
    if x == y == z:
        return "Equilateral triangle"
    elif x == y or y == z or x == z:
        return "Isosceles triangle"
    else:
        return "Scalene triangle"

The result of executing this code is as follows. (It's too long, so I'll omit it.)

export PYNGUIN_DANGER_AWARE=ANY && \
	poetry run pynguin \
		--project-path ./ \
		--output-path ./pynguin-results \
		-v \
		--module-name python_playground.pynguin_example_3 && \
	export -n PYNGUIN_DANGER_AWARE
[10:53:34] INFO     Start Pynguin Test Generation…                                                                                                                                           generator.py:97
           INFO     No seed given. Using 1640051613212106000                                                                                                                                generator.py:173
           INFO     Collecting constants from SUT.                                                                                                                                          generator.py:182
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                     generationalgorithmfactory.py:235
           INFO     Instantiated 11 fitness functions                                                                                                                      generationalgorithmfactory.py:311
[10:53:35] INFO     Using CoverageArchive                                                                                                                                  generationalgorithmfactory.py:279
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                               generationalgorithmfactory.py:254
           INFO     Using stopping condition: StoppingCondition.MAX_TIME                                                                                                    generationalgorithmfactory.py:90
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                 generationalgorithmfactory.py:267
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                     generationalgorithmfactory.py:287
           INFO     Start generating test cases                                                                                                                                             generator.py:265
           INFO     Iteration:     0, Coverage: 0.090909                                                                                                                                searchobserver.py:66
           INFO     Iteration:     1, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     2, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     3, Coverage: 0.090909                                                                                                                                searchobserver.py:72
[10:53:36] INFO     Iteration:     4, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     5, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     6, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     7, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:     8, Coverage: 0.090909                                                                                                                                searchobserver.py:72
[10:53:37] INFO     Iteration:     9, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    10, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    11, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    12, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    13, Coverage: 0.090909                                                                                                                                searchobserver.py:72
[10:53:38] INFO     Iteration:    14, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    15, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    16, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    17, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    18, Coverage: 0.090909                                                                                                                                searchobserver.py:72
[10:53:39] INFO     Iteration:    19, Coverage: 0.090909                                                                                                                                searchobserver.py:72
           INFO     Iteration:    20, Coverage: 0.090909                                                                                                                                searchobserver.py:72
...

Is there a way or implementation to avoid this?

Pynguin does not create test cases

Hello,

I tried to generate test cases for a private project. Unfortunately Pynguin was not able to create a working test case, but there were some failed test cases.
I tried all documented strategies but the situation did not really change. The functions in the module I used are a bit complex and the code is a mixture of functional and procedural code, there is no object oriented code.

Do you have an idea how I can create test cases with Pynguin? Do I have to change the code or are there known problems with code similar to that one I described? Can the problem be solved with changing the parameters for Pynguin?

Low coverage generating tests on a ROS Node

We are trying to generate tests for our ROS project. Running Pynguin on a simple vehicle class shows a coverage of 0.187500 and the test generated is not very useful:

# Test cases automatically generated by Pynguin (https://www.pynguin.eu).
# Please check them before you use them.
import pytest
import vehicle as module_0


@pytest.mark.xfail(strict=True)
def test_case_0():
    module_0.Vehicle()

We are guessing that Pynguin gets lost at one point, and are looking for some insight on what we can do.

To Reproduce
We made a minimal example here.

Expected behavior
We would expect the coverage to be a bit higher, with some relevant tests (test on the speed_profile or even the quickstart example).

Software Version (please complete the following information):

OS: Docker Ubuntu 22.04 + ROS Humble
Python version: 3.10.12
Pynguin Version: 0.34.0

install pynguin error - with astmonkey

Hello, I encountered a problem when installing
Collecting astmonkey>=0.3.6
Using cached astmonkey-0.3.6.tar.gz (10 kB)

[6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\wTat\AppData\Local\Temp\pip-install-s2zp1fpl\astmonkey_176ba39be03b487bbbf28bea3f7c3b7e\setup.py", line 7, in
long_description = f.read()
UnicodeDecodeError: 'cp950' codec can't decode byte 0xc5 in position 3079: illegal multibyte sequence
[end of output]

Please tell me how to deal with and solve it?

Docker build issues

Hi,

I want use pynguin as a docker container. I tried to build the dockerfile which is in the repository.("docker build --tag pynguin .") but is did not build. The reason; Errors related to "poetry"..

I copied pymyproject.toml, README.md and pynguin folder whic is in the repo to the dokcer folder. After; i tryed build ,and builded successfull. I tryed docker run with volumes parameters. The container gave the following as output

where is the i do mistakes, can your help me?.

The --maximum_iteration flag doesn't actually do anything

I was trying to limit the time waiting for the generation of the test cases, and for that I used the flag. Nonetheless, the flag didn't limit the number of iterations of the program.

pynguin takes long time even when specifying timeout options

Describe the bug
When pynguin generates test cases for a program that may occur an infinite loop, pynguin takes long time (even when specifying timeout options).

To Reproduce

Create loop.py with the following content:

def func(n: int):
    while n == 0:
        pass
    return n

Run pynguin with the following command: PYNGUIN_DANGER_AWARE=1 pynguin --project-path . --output-path output --module-name loop --maximum-search-time 1 --maximum-slicing-time 1 --maximum-test-execution-timeout 1 --maximum-test-execution-timeout 1 --test-execution-time-per-statement 1 --max-size 1 --max-length-test-case 1 --max-int 10 --assertion_generation SIMPLE --algorithm MOSA
pynguin takes about 30 seconds

Expected behavior
I can imagine there are various mandatory processes where it is difficult to limit execution times, but, I'd like pynguin stop until 5 seconds when specifying the above options.

Screenshots

Software Version:

OS: macOS
Python: 3.10.9
Poetry: 1.4.0
Pynguin: 0.33.0

ConstructionFailedException: Found no variables of type at position

Hi,

First of all thank you for the project, I recently started playing with it and I am very impressed by the results.

I'm using a third party project as a test case (dnspython), with the following command line:
pynguin --project-path . --output-path guin/dnsresults --module-name dns.ipv4

It successfully generates 6 test cases but it also outputs some errors, I'm not sure if they are expected or not, I'm looking for help to understand what is causing the error and what should I change so it can run successfully.

This is the code used: ipv4.py

ERROR Failed to change call for statement. testfactory.py:757
╭────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────╮
│ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:754 in change_random_call │
│ │
│ 751 │ │ │
│ 752 │ │ call = randomness.choice(calls) │
│ 753 │ │ try: │
│ ❱ 754 │ │ │ self.change_call(test_case, statement, call) │
│ 755 │ │ │ return True │
│ 756 │ │ except ConstructionFailedException: │
│ 757 │ │ │ self._logger.exception("Failed to change call for statement.") │
│ │
│ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:782 in change_call │
│ │
│ 779 │ │ if call.is_method(): │
│ 780 │ │ │ method = cast(gao.GenericMethod, call) │
│ 781 │ │ │ assert method.owner │
│ ❱ 782 │ │ │ callee = self._get_random_non_none_object(test_case, method.owner, position) │
│ 783 │ │ │ parameters = self._get_reuse_parameters( │
│ 784 │ │ │ │ test_case, method.inferred_signature, position │
│ 785 │ │ │ ) │
│ │
│ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:850 in get_random_non_none_object │
│ │
│ 847 │ │ │ ) │
│ 848 │ │ ] │
│ 849 │ │ if len(variables) == 0: │
│ ❱ 850 │ │ │ raise ConstructionFailedException( │
│ 851 │ │ │ │ f"Found no variables of type {type} at position {position}" │
│ 852 │ │ │ ) │
│ 853 │ │ return randomness.choice(variables) │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ConstructionFailedException: Found no variables of type <class 'dns.exception.DNSException'> at position 2

Using mock

does Pynguin use mock? If so, how can I configure its use?

Pynguin throws module not found error for internal packages

Hi There,

I am using Pynguin to generate unit tests for my project. My project has an internal package dependency. Let's call it datapackage. This is installed in a virtual environment. My Pycharm and REPL are able to detect datapackage and import them.

However, Pynguin throws module not found error.

Please find the stacktrace below.

$pynguin --algorithm WHOLE_SUITE --project_path src --output_path tests\unit --module_name client.py

[13:22:35] ERROR    Failed to load SUT: No module named 'datapackage'                                                                                         generator.py:137
                    ┌──────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────┐
                    │ c:\users\username\appdata\local\programs\python\python38\lib\site-packages\pynguin\generator.py:133 in _load_sut                  │
                    │                                                                                                                                    │
                    │   130 │   try:                                                                                                                     │
                    │   131 │   │   # We need to set the current thread ident so the import trace is recorded.                                           │
                    │   132 │   │   tracer.current_thread_ident = threading.currentThread().ident                                                        │
                    │ > 133 │   │   importlib.import_module(config.configuration.module_name)                                                            │
                    │   134 │   except ImportError as ex:                                                                                                │
                    │   135 │   │   # A module could not be imported because some dependencies                                                           │
                    │   136 │   │   # are missing or it is malformed                                                                                     │
                    │                                                                                                                                    │
                    │ c:\users\username\appdata\local\programs\python\python38\lib\importlib\__init__.py:127 in import_module                           │
                    │                                                                                                                                    │
                    │   124 │   │   │   if character != '.':                                                                                             │
                    │   125 │   │   │   │   break                                                                                                        │
                    │   126 │   │   │   level += 1                                                                                                       │
                    │ > 127 │   return _bootstrap._gcd_import(name[level:], package, level)                                                              │
                    │   128                                                                                                                              │
                    │   129                                                                                                                              │
                    │   130 _RELOADING = {}                                                                                                              │
                    │ <frozen importlib._bootstrap>:1014 in _gcd_import                                                                                  │
                    │ <frozen importlib._bootstrap>:991 in _find_and_load                                                                                │
                    │ <frozen importlib._bootstrap>:961 in _find_and_load_unlocked                                                                       │
                    │ <frozen importlib._bootstrap>:219 in _call_with_frames_removed                                                                     │
                    │ <frozen importlib._bootstrap>:1014 in _gcd_import                                                                                  │
                    │ <frozen importlib._bootstrap>:991 in _find_and_load                                                                                │
                    │ <frozen importlib._bootstrap>:975 in _find_and_load_unlocked                                                                       │
                    │ <frozen importlib._bootstrap>:671 in _load_unlocked                                                                                │
                    │ <frozen importlib._bootstrap_external>:783 in exec_module                                                                          │
                    │ <frozen importlib._bootstrap>:219 in _call_with_frames_removed                                                                     │
                    │                                                                                                                                    │
                    │ C:\<path to your module>:2 in <module>                       │
                    │                                                                                                                                    │
                    │     1 from datapackage.http.request_dispatcher import dispatch_retry_session                                                          │
                    │ >   2 from requests import HTTPError                                                                                               │
                    │     3                                                                                                                              │
                    │                                                          │
                    │                                                                                     │
                    └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
                    ModuleNotFoundError: No module named 'datapackage'

ERROR SUT contains nothing we can test.

Problem description

Hello,

I have tried this tool on multiple libraries (private and public) and I cannot get it to work. All I get is this error:

ERROR    SUT contains nothing we can test.

Am I doing something wrong 😕 ?

Code to reproduce

git clone [email protected]:ThibTrip/pangres.git
cd pangres
conda env create -f environment.yml # create the virtual environment
conda activate pangres-dev
pip install pynguin
pynguin --algorithm WHOLE_SUITE --project_path . --output_path ./test_pynguin --module_name pangres

I also tried other commands with the same result.

pynguin --algorithm WHOLE_SUITE --project_path ./pangres --output_path ./test_pynguin --module_name pangres
pynguin --algorithm WHOLE_SUITE --project_path ./pangres --output_path ./test_pynguin --module_name core
pynguin --algorithm WHOLE_SUITE --project_path ./pangres --output_path ./test_pynguin --module_name core.py

Trouble with testing a repository

Hi, I've read your paper and saw you have tried to test pynguin in repositories such as sanic.

However, when I configured a local environment for sanic, and try to run the command such as
PYNGUIN_DANGER_AWARE=1 pynguin --algorithm DYNAMOSA --project_path . --output_path ./test_pynguin --module_name sanic.http --assertion_generation SIMPLE

I got this error message.

ERROR    SUT contains nothing we can test.

Issue #10 looks similar, but I think I tried the exact suggestion given from one of the contributors.

What should be the correct command to use pynguin for such cases?

Cannot generate testcase with import statement e.g. `numpy`

I try to run pynguin to generate the test case for this example module (numpy_example) containing the following snippet.

import numpy as np

def gen_array(value):
    return np.array(value)

but i got an error as ValueError: no signature found for builtin <built-in function where>

This is the full stack trace.

[03:08:51] INFO     Start Pynguin Test Generation…                                                                                        generator.py:110
           INFO     Collecting static constants from module under test                                                                    generator.py:209
           INFO     No constants found                                                                                                    generator.py:212
           INFO     Setting up runtime collection of constants                                                                            generator.py:221
[03:08:54] INFO     Stop Pynguin Test Generation…                                                                                         generator.py:113
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/bin/pynguin:10 in <module>                                                            │
│                                                                                                  │
│    7 if __name__ == '__main__':                                                                  │
│    8 │   os.environ["PYNGUIN_DANGER_AWARE"] = "1"                                                │
│    9 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                        │
│ ❱ 10 │   sys.exit(main())                                                                        │
│   11                                                                                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/cli.py:190 in main                               │
│                                                                                                  │
│   187 │   set_configuration(parsed.config)                                                       │
│   188 │   if console is not None:                                                                │
│   189 │   │   with console.status("Running Pynguin..."):                                         │
│ ❱ 190 │   │   │   return run_pynguin().value                                                     │
│   191 │   else:                                                                                  │
│   192 │   │   return run_pynguin().value                                                         │
│   193                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:111 in run_pynguin                  │
│                                                                                                  │
│   108 │   """                                                                                    │
│   109 │   try:                                                                                   │
│   110 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱ 111 │   │   return _run()                                                                      │
│   112 │   finally:                                                                               │
│   113 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   114                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:496 in _run                         │
│                                                                                                  │
│   493                                                                                            │
│   494                                                                                            │
│   495 def _run() -> ReturnCode:                                                                  │
│ ❱ 496 │   if (setup_result := _setup_and_check()) is None:                                       │
│   497 │   │   return ReturnCode.SETUP_FAILED                                                     │
│   498 │   executor, test_cluster, constant_provider = setup_result                               │
│   499 │   # traces slices for test cases after execution                                         │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:260 in _setup_and_check             │
│                                                                                                  │
│   257 │                                                                                          │
│   258 │   # Analyzing the SUT should not cause any coverage.                                     │
│   259 │   tracer.disable()                                                                       │
│ ❱ 260 │   if (test_cluster := _setup_test_cluster()) is None:                                    │
│   261 │   │   return None                                                                        │
│   262 │   tracer.enable()                                                                        │
│   263                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:117 in _setup_test_cluster          │
│                                                                                                  │
│   114                                                                                            │
│   115                                                                                            │
│   116 def _setup_test_cluster() -> ModuleTestCluster | None:                                     │
│ ❱ 117 │   test_cluster = generate_test_cluster(                                                  │
│   118 │   │   config.configuration.module_name,                                                  │
│   119 │   │   config.configuration.type_inference.type_inference_strategy,                       │
│   120 │   )                                                                                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1303 in generate_test_cluster │
│                                                                                                  │
│   1300 │   Returns:                                                                              │
│   1301 │   │   A new test cluster for the given module                                           │
│   1302 │   """                                                                                   │
│ ❱ 1303 │   return analyse_module(parse_module(module_name), type_inference_strategy)             │
│   1304                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1282 in analyse_module        │
│                                                                                                  │
│   1279 │   │   A test cluster for the module                                                     │
│   1280 │   """                                                                                   │
│   1281 │   test_cluster = ModuleTestCluster(linenos=parsed_module.linenos)                       │
│ ❱ 1282 │   __resolve_dependencies(                                                               │
│   1283 │   │   root_module=parsed_module,                                                        │
│   1284 │   │   type_inference_strategy=type_inference_strategy,                                  │
│   1285 │   │   test_cluster=test_cluster,                                                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1169 in                       │
│ __resolve_dependencies                                                                           │
│                                                                                                  │
│   1166 │   │   )                                                                                 │
│   1167 │   │                                                                                     │
│   1168 │   │   # Analyze all functions found in the current module                               │
│ ❱ 1169 │   │   __analyse_included_functions(                                                     │
│   1170 │   │   │   module=current_module,                                                        │
│   1171 │   │   │   root_module_name=root_module.module_name,                                     │
│   1172 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1258 in                       │
│ __analyse_included_functions                                                                     │
│                                                                                                  │
│   1255 │   │   if current in seen_functions:                                                     │
│   1256 │   │   │   continue                                                                      │
│   1257 │   │   seen_functions.add(current)                                                       │
│ ❱ 1258 │   │   __analyse_function(                                                               │
│   1259 │   │   │   func_name=current.__qualname__,                                               │
│   1260 │   │   │   func=current,                                                                 │
│   1261 │   │   │   type_inference_strategy=type_inference_strategy,                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:941 in __analyse_function     │
│                                                                                                  │
│    938 │   │   return                                                                            │
│    939 │                                                                                         │
│    940 │   LOGGER.debug("Analysing function %s", func_name)                                      │
│ ❱  941 │   inferred_signature = test_cluster.type_system.infer_type_info(                        │
│    942 │   │   func, type_inference_strategy                                                     │
│    943 │   )                                                                                     │
│    944 │   func_ast = get_function_node_from_ast(module_tree, func_name)                         │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/typesystem.py:1157 in infer_type_info   │
│                                                                                                  │
│   1154 │   │   """                                                                               │
│   1155 │   │   match type_inference_strategy:                                                    │
│   1156 │   │   │   case config.TypeInferenceStrategy.TYPE_HINTS:                                 │
│ ❱ 1157 │   │   │   │   return self.infer_signature(method, self.type_hints_provider)             │
│   1158 │   │   │   case config.TypeInferenceStrategy.NONE:                                       │
│   1159 │   │   │   │   return self.infer_signature(method, self.no_type_hints_provider)          │
│   1160 │   │   │   case _:                                                                       │
│                                                                                                  │
│ /usr/local/lib/python3.10/site-packages/pynguin/analyses/typesystem.py:1214 in infer_signature   │
│                                                                                                  │
│   1211 │   │   if inspect.isclass(method) and hasattr(method, "__init__"):                       │
│   1212 │   │   │   return self.infer_signature(getattr(method, "__init__"), type_hint_provider)  │
│   1213 │   │                                                                                     │
│ ❱ 1214 │   │   method_signature = inspect.signature(method)                                      │
│   1215 │   │   hints = type_hint_provider(method)                                                │
│   1216 │   │   parameters: dict[str, ProperType] = {}                                            │
│   1217 │   │   for param_name in method_signature.parameters:                                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/inspect.py:3253 in signature                                           │
│                                                                                                  │
│   3250                                                                                           │
│   3251 def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
│   3252 │   """Get a signature object for the passed callable."""                                 │
│ ❱ 3253 │   return Signature.from_callable(obj, follow_wrapped=follow_wrapped,                    │
│   3254 │   │   │   │   │   │   │   │      globals=globals, locals=locals, eval_str=eval_str)     │
│   3255                                                                                           │
│   3256                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/inspect.py:3001 in from_callable                                       │
│                                                                                                  │
│   2998 │   def from_callable(cls, obj, *,                                                        │
│   2999 │   │   │   │   │     follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
│   3000 │   │   """Constructs Signature for the given callable object."""                         │
│ ❱ 3001 │   │   return _signature_from_callable(obj, sigcls=cls,                                  │
│   3002 │   │   │   │   │   │   │   │   │   │   follow_wrapper_chains=follow_wrapped,             │
│   3003 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
│   3004                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/inspect.py:2467 in _signature_from_callable                            │
│                                                                                                  │
│   2464 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
│   2465 │                                                                                         │
│   2466 │   if _signature_is_builtin(obj):                                                        │
│ ❱ 2467 │   │   return _signature_from_builtin(sigcls, obj,                                       │
│   2468 │   │   │   │   │   │   │   │   │      skip_bound_arg=skip_bound_arg)                     │
│   2469 │                                                                                         │
│   2470 │   if isinstance(obj, functools.partial):                                                │
│                                                                                                  │
│ /usr/local/lib/python3.10/inspect.py:2274 in _signature_from_builtin                             │
│                                                                                                  │
│   2271 │                                                                                         │
│   2272 │   s = getattr(func, "__text_signature__", None)                                         │
│   2273 │   if not s:                                                                             │
│ ❱ 2274 │   │   raise ValueError("no signature found for builtin {!r}".format(func))              │
│   2275 │                                                                                         │
│   2276 │   return _signature_fromstr(cls, func, s, skip_bound_arg)                               │
│   2277                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: no signature found for builtin <built-in function where>

Pynguin find classes that are not part of production code

Describe the bug
I'm using Pynguin to generate test cases for a Python script. The script does not contain classes, but only functions (14 functions).
But when i use Pynguin it finds more than 600 classes and the generation of test cases takes a lot of time.

To Reproduce
Steps to reproduce the behavior:

Use the latest Pynguin version
Use the following command line arguments to Pynguin:

pynguin --project-path ./coupling --output-path ./tests/developer_coupling/ --module-name developer_coupling -v

[13:12:03] INFO     Start Pynguin Test Generation…                                                                                                              generator.py:107
           INFO     Collecting static constants from module under test                                                                                          generator.py:208
           INFO     No constants found                                                                                                                          generator.py:211
           INFO     Setting up runtime collection of constants                                                                                                  generator.py:220
[13:12:09] INFO     Analyzed project to create test cluster                                                                                                       module.py:1318
           INFO     Modules:      49                                                                                                                              module.py:1319
           INFO     Functions:   109                                                                                                                              module.py:1320
           INFO     Classes:     166                                                                                                                              module.py:1321

Expected behavior
I expect that only the classes and functions on my script will be tested.

Software Version (please complete the following information):

OS: macOS
Python version Python 3.10
Pynguin Version latest

se2p / pynguin Goto Github PK

pynguin's Introduction

Pynguin

Attention

Prerequisites

Installing Pynguin

Using Pynguin

Contributing to Pynguin

Contributors

Development using PyCharm.

License

Star History

pynguin's People

Contributors

Stargazers

Watchers

Forkers

pynguin's Issues

Problem description

Code to reproduce

Recommend Projects

Recommend Topics

Recommend Org