Pynguin, The PYthoN General UnIt Test geNerator is a test-generation tool for Python

Related tags

Testing pynguin
Overview

Pynguin

Pynguin, the PYthoN General UnIt test geNerator, is a tool that allows developers to generate unit tests automatically.

Testing software is a tedious task. Thus, automated generation techniques have been proposed and mature tools exist—for statically typed languages, such as Java. There is, however, no fully-automated tool available that produces unit tests for general-purpose programs in a dynamically typed language. Pynguin is, to the best of our knowledge, the first tool that fills this gap and allows the automated generation of unit tests for Python programs.

Pynguin is developed at the Chair of Software Engineering II of the University of Passau.

License LGPL v3 Code style: black PyPI version Supported Python Versions Documentation Status DOI REUSE status Downloads

Pynguin Logo

Attention

Pynguin executes the module under test! As a consequence, depending on what code is in that module, running Pynguin can cause serious harm to your computer, for example, wipe your entire hard disk! We recommend running Pynguin in an isolated environment, for example, a Docker container, to minimise the risk of damaging your system.

Prerequisites

Before you begin, ensure you have met the following requirements:

  • You have installed Python 3.8 (we have not yet tested with Python 3.9, there might be some problems due to changed internals regarding the byte-code instrumentation).
  • You have a recent Linux/macOS/Windows machine.

Please consider reading the online documentation to start your Pynguin adventure.

Installing Pynguin

Pynguin can be easily installed using the pip tool by typing:

pip install pynguin

Make sure that your version of pip is the one of the Python 3.8 interpreted or a virtual environment that uses Python 3.8 as its interpreter as any older version is not supported by Pynguin!

Using Pynguin

Pynguin is a command-line application. Once you installed it to a virtual environment, you can invoke the tool by typing pynguin inside this virtual environment. Pynguin will then print a list of its command-line parameters.

A minimal full command line to invoke Pynguin could be the following, where we assume that a project foo is located in /tmp/foo, we want to store Pynguin's in /tmp/testgen, and we want to generate tests using a whole-suite approach for the module foo.bar (wrapped for better readability):

pynguin \
  --algorithm WHOLE_SUITE \
  --project_path /tmp/foo \
  --output_path /tmp/testgen \
  --module_name foo.bar

Contributing to Pynguin

For the development of Pynguin you will need the poetry dependency management and packaging tool. To start developing, follow these steps:

  1. Clone the repository

  2. Change to the pynguin folder: cd pynguin

  3. Create a virtual environment and install dependencies using poetry: poetry install

  4. Make your changes

  5. Run poetry shell to switch to the virtual environment in your current shell

  6. Run make check to verify that your changes pass all checks

    Please see the poetry documentation for more information on this tool.

Development using PyCharm.

If you want to use the PyCharm IDE you have to set up a few things:

  1. Import pynguin into PyCharm.
  2. Find the location of the virtual environment by running poetry env info in the project directory.
  3. Go to Settings / Project: pynguin / Project interpreter
  4. Add and use a new interpreter that points to the path of the virtual environment
  5. Set the default test runner to pytest

License

This project is licensed under the terms of the GNU Lesser General Public License.

Comments
  • ConstructionFailedException: Found no variables of type at position

    ConstructionFailedException: Found no variables of type at position

    Hi,

    First of all thank you for the project, I recently started playing with it and I am very impressed by the results.

    I'm using a third party project as a test case (dnspython), with the following command line: pynguin --project-path . --output-path guin/dnsresults --module-name dns.ipv4

    It successfully generates 6 test cases but it also outputs some errors, I'm not sure if they are expected or not, I'm looking for help to understand what is causing the error and what should I change so it can run successfully.

    This is the code used: ipv4.py

    ERROR Failed to change call for statement. testfactory.py:757 ╭────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────╮ │ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:754 in change_random_call │ │ │ │ 751 │ │ │ │ 752 │ │ call = randomness.choice(calls) │ │ 753 │ │ try: │ │ ❱ 754 │ │ │ self.change_call(test_case, statement, call) │ │ 755 │ │ │ return True │ │ 756 │ │ except ConstructionFailedException: │ │ 757 │ │ │ self._logger.exception("Failed to change call for statement.") │ │ │ │ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:782 in change_call │ │ │ │ 779 │ │ if call.is_method(): │ │ 780 │ │ │ method = cast(gao.GenericMethod, call) │ │ 781 │ │ │ assert method.owner │ │ ❱ 782 │ │ │ callee = self._get_random_non_none_object(test_case, method.owner, position) │ │ 783 │ │ │ parameters = self._get_reuse_parameters( │ │ 784 │ │ │ │ test_case, method.inferred_signature, position │ │ 785 │ │ │ ) │ │ │ │ G:\envs\testccode\lib\site-packages\pynguin\testcase\testfactory.py:850 in get_random_non_none_object │ │ │ │ 847 │ │ │ ) │ │ 848 │ │ ] │ │ 849 │ │ if len(variables) == 0: │ │ ❱ 850 │ │ │ raise ConstructionFailedException( │ │ 851 │ │ │ │ f"Found no variables of type {type} at position {position}" │ │ 852 │ │ │ ) │ │ 853 │ │ return randomness.choice(variables) │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ConstructionFailedException: Found no variables of type <class 'dns.exception.DNSException'> at position 2

    opened by tovmeod 9
  • Pynguin does not create test cases

    Pynguin does not create test cases

    Hello,

    I tried to generate test cases for a private project. Unfortunately Pynguin was not able to create a working test case, but there were some failed test cases. I tried all documented strategies but the situation did not really change. The functions in the module I used are a bit complex and the code is a mixture of functional and procedural code, there is no object oriented code.

    Do you have an idea how I can create test cases with Pynguin? Do I have to change the code or are there known problems with code similar to that one I described? Can the problem be solved with changing the parameters for Pynguin?

    opened by kipfstuhl 7
  • Pynguin encounters error when running against webpy

    Pynguin encounters error when running against webpy

    We are evaluating our project pyster(also a Python test generator) with Pynguin and found that Pynguin throws the following exception when running against webpy:

    pynguin --algorithm WSPY --project_path public_repo/webpy --module_name web.db --output_path ./public_repo/webpy --budget 600
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 805, in successors
        return iter(self._succ[n])
    KeyError: ProgramGraphNode(index=1, basic_block=[<FOR_ITER arg=[<LOAD_CONST arg=ExecutionTracer lineno=369>, <LOAD_METHOD arg='executed_bool_predicate' lineno=369>, <LOAD_CONST arg=False lineno=369>, <LOAD_CONST arg=45 lineno=369>, <CALL_METHOD arg=2 lineno=369>, <POP_TOP lineno=369>, <JUMP_ABSOLUTE arg=[<LOAD_GLOBAL arg='SQLQuery' lineno=375>, <LOAD_METHOD arg='join' lineno=375>, <LOAD_FAST arg='result' lineno=375>, <LOAD_CONST arg='' lineno=375>, <CALL_METHOD arg=2 lineno=375>, <RETURN_VALUE lineno=375>] lineno=369>] lineno=369>])
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/bin/pynguin", line 8, in <module>
        sys.exit(main())
      File "/usr/local/lib/python3.8/site-packages/pynguin/cli.py", line 136, in main
        return generator.run().value
      File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 101, in run
        return self._run()
      File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 224, in _run
        if (setup_result := self._setup_and_check()) is None:
      File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 186, in _setup_and_check
        if not self._load_sut():
      File "/usr/local/lib/python3.8/site-packages/pynguin/generator.py", line 141, in _load_sut
        importlib.import_module(config.INSTANCE.module_name)
      File "/usr/local/Cellar/[email protected]/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 32, in exec_module
        super().exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 779, in exec_module
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/machinery.py", line 49, in get_code
        return instrumentation.instrument_module(to_instrument)
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 326, in instrument_module
        return self._instrument_code_recursive(module_code)
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 114, in _instrument_code_recursive
        return self._instrument_inner_code_objects(
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 75, in _instrument_inner_code_objects
        self._instrument_code_recursive(
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 113, in _instrument_code_recursive
        self._instrument_cfg(cfg, code_object_id)
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 130, in _instrument_cfg
        predicate_id = self._instrument_node(
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 172, in _instrument_node
        predicate_id = self._instrument_for_loop(
      File "/usr/local/lib/python3.8/site-packages/pynguin/instrumentation/branch_distance.py", line 423, in _instrument_for_loop
        for successor in dominator_tree.get_transitive_successors(node):
      File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 188, in get_transitive_successors
        return self._get_transitive_successors(node, set())
      File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 192, in _get_transitive_successors
        for successor_node in self.get_successors(node):
      File "/usr/local/lib/python3.8/site-packages/pynguin/analyses/controlflow/programgraph.py", line 129, in get_successors
        for successor in self._graph.successors(node):
      File "/usr/local/lib/python3.8/site-packages/networkx/classes/digraph.py", line 807, in successors
        raise NetworkXError(f"The node {n} is not in the digraph.") from e
    networkx.exception.NetworkXError: The node ProgramGraphNode(1) is not in the digraph.
    
    opened by JamZYu 6
  • The --maximum_iteration flag doesn't actually do anything

    The --maximum_iteration flag doesn't actually do anything

    I was trying to limit the time waiting for the generation of the test cases, and for that I used the flag. Nonetheless, the flag didn't limit the number of iterations of the program.

    opened by angelomorgado 4
  • RuntimeError: The current thread shall not be executed any more, thus I kill it.

    RuntimeError: The current thread shall not be executed any more, thus I kill it.

    pynguin 0.27.0 Python 3.10.7 mac arm

    I'm getting a RuntimeError logged to the console, I'm trying to figure out what is causing it, I tried setting maximum-search-time to 18000 (30 minutes) and maximum_iterations to 10000 but it still log the same error. Anyway it generates 4 test cases with 3 of them with @pytest.mark.xfail(strict=True)

    error below:

    ERROR    Exception in Thread: <Thread(Thread-266595 (_execute_test_case), started daemon 6209482752)>                                                execution.py:2081
                        ╭─────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────╮                  
                        │ /Users/avraham/.pyenv/versions/3.10.7/lib/python3.10/threading.py:1016 in _bootstrap_inner                                              │                  
                        │                                                                                                                                         │                  
                        │   1013 │   │   │   │   _sys.setprofile(_profile_hook)                                                                                   │                  
                        │   1014 │   │   │                                                                                                                        │                  
                        │   1015 │   │   │   try:                                                                                                                 │                  
                        │ ❱ 1016 │   │   │   │   self.run()                                                                                                       │                  
                        │   1017 │   │   │   except:                                                                                                              │                  
                        │   1018 │   │   │   │   self._invoke_excepthook(self)                                                                                    │                  
                        │   1019 │   │   finally:                                                                                                                 │                  
                        │                                                                                                                                         │                  
                        │ /Users/avraham/.pyenv/versions/3.10.7/lib/python3.10/threading.py:953 in run                                                            │                  
                        │                                                                                                                                         │                  
                        │    950 │   │   """                                                                                                                      │                  
                        │    951 │   │   try:                                                                                                                     │                  
                        │    952 │   │   │   if self._target is not None:                                                                                         │                  
                        │ ❱  953 │   │   │   │   self._target(*self._args, **self._kwargs)                                                                        │                  
                        │    954 │   │   finally:                                                                                                                 │                  
                        │    955 │   │   │   # Avoid a refcycle if the thread is running a function with                                                          │                  
                        │    956 │   │   │   # an argument that has a member that points to the thread.                                                           │                  
                        │                                                                                                                                         │                  
                        │ /Users/avraham/Library/Caches/pypoetry/virtualenvs/M5hsRoCT-py3.10/lib/python3.10/site-packages/pynguin/testcase/execu │                  
                        │ tion.py:2184 in _execute_test_case                                                                                                      │                  
                        │                                                                                                                                         │                  
                        │   2181 │   │   for idx, statement in enumerate(test_case.statements):                                                                   │                  
                        │   2182 │   │   │   ast_node = self._before_statement_execution(statement, exec_ctx)                                                     │                  
                        │   2183 │   │   │   exception = self.execute_ast(ast_node, exec_ctx)                                                                     │                  
                        │ ❱ 2184 │   │   │   self._after_statement_execution(statement, exec_ctx, exception)                                                      │                  
                        │   2185 │   │   │   if exception is not None:                                                                                            │                  
                        │   2186 │   │   │   │   result.report_new_thrown_exception(idx, exception)                                                               │                  
                        │   2187 │   │   │   │   break                                                                                                            │                  
                        │                                                                                                                                         │                  
                        │ /Users/avraham/Library/Caches/pypoetry/virtualenvs/M5hsRoCT-py3.10/lib/python3.10/site-packages/pynguin/testcase/execu │                  
                        │ tion.py:2283 in _after_statement_execution                                                                                              │                  
                        │                                                                                                                                         │                  
                        │   2280 │   │   # See comments in _before_statement_execution                                                                            │                  
                        │   2281 │   │   if self.tracer.current_thread_identifier != threading.current_thread().ident:                                            │                  
                        │   2282 │   │   │   # Kill this thread                                                                                                   │                  
                        │ ❱ 2283 │   │   │   raise RuntimeError(                                                                                                  │                  
                        │   2284 │   │   │   │   "The current thread shall not be executed any more, thus I kill it."                                             │                  
                        │   2285 │   │   │   )                                                                                                                    │                  
                        │   2286                                                                                                                                  │                  
                        ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                  
                        RuntimeError: The current thread shall not be executed any more, thus I kill it. 
    
    opened by tovmeod 4
  • doc is dead: 500 or 522 error

    doc is dead: 500 or 522 error

    https://pynguin.readthedocs.io/en/latest/user/quickstart.html Hi, one issue about the doc: when clicking the Assertions link, it breaks the whole doc site and return 500 error in Chrome and 522 error in Edge, could you replicate that and fix it?

    opened by Symbolk 4
  • Exception: Statement is not part of it's test case

    Exception: Statement is not part of it's test case

    Hi~ I encountered a problem when I attempted to generate test cases for a toy program named test_script.py shown as below:

    import numpy as np
    
    
    def add1(a:np.ndarray,b:np.ndarray)->np.ndarray:
        if a.shape == b.shape:
            return a+b
        else:
            return np.array([])
    if __name__ == "__main__":
        print(add1(np.array([1,2,3]),np.array([5,6,7,9])))
    
    

    I placed test_pynguin.py at /data02/projects/test_pynguin and ran the command:

    cd /data02/projects/test_pynguin pynguin --project-path . --output-path /data02/projects/output --module-name test_script

    Then I received an exception:

    ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/bin/pynguin:8 in <module>                                    │
    │                                                                                                  │
    │   5 from pynguin.cli import main                                                                 │
    │   6 if __name__ == '__main__':                                                                   │
    │   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
    │ ❱ 8 │   sys.exit(main())                                                                         │
    │   9                                                                                              │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/cli.py:172 in main       │
    │                                                                                                  │
    │   169 │                                                                                          │
    │   170 │   set_configuration(parsed.config)                                                       │
    │   171 │   with console.status("Running Pynguin..."):                                             │
    │ ❱ 172 │   │   return run_pynguin().value                                                         │
    │   173                                                                                            │
    │   174                                                                                            │
    │   175 if __name__ == "__main__":                                                                 │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:98 in       │
    │ run_pynguin                                                                                      │
    │                                                                                                  │
    │    95 │   """                                                                                    │
    │    96 │   try:                                                                                   │
    │    97 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
    │ ❱  98 │   │   return _run()                                                                      │
    │    99 │   finally:                                                                               │
    │   100 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
    │   101                                                                                            │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:266 in _run │
    │                                                                                                  │
    │   263 │   │   executor, test_cluster                                                             │
    │   264 │   )                                                                                      │
    │   265 │   _LOGGER.info("Start generating test cases")                                            │
    │ ❱ 266 │   generation_result = algorithm.generate_tests()                                         │
    │   267 │   if algorithm.stopping_condition.is_fulfilled():                                        │
    │   268 │   │   _LOGGER.info("Used up all resources (%s).", algorithm.stopping_condition)          │
    │   269 │   else:                                                                                  │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
    │ namosastrategy.py:70 in generate_tests                                                           │
    │                                                                                                  │
    │    67 │   │   │   self.create_test_suite(self._archive.solutions)                                │
    │    68 │   │   )                                                                                  │
    │    69 │   │   while self.resources_left() and len(self._archive.uncovered_goals) > 0:            │
    │ ❱  70 │   │   │   self.evolve()                                                                  │
    │    71 │   │   │   self.after_search_iteration(self.create_test_suite(self._archive.solutions))   │
    │    72 │   │                                                                                      │
    │    73 │   │   self.after_search_finish()                                                         │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
    │ namosastrategy.py:84 in evolve                                                                   │
    │                                                                                                  │
    │    81 │   │   """Runs one evolution step."""                                                     │
    │    82 │   │   offspring_population: List[                                                        │
    │    83 │   │   │   tcc.TestCaseChromosome                                                         │
    │ ❱  84 │   │   ] = self._breed_next_generation()                                                  │
    │    85 │   │                                                                                      │
    │    86 │   │   # Create union of parents and offspring                                            │
    │    87 │   │   union: List[tcc.TestCaseChromosome] = []                                           │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
    │ stractmosastrategy.py:53 in _breed_next_generation                                               │
    │                                                                                                  │
    │    50 │   │   │   │   │   continue                                                               │
    │    51 │   │   │                                                                                  │
    │    52 │   │   │   # Apply mutation on offspring_1                                                │
    │ ❱  53 │   │   │   self._mutate(offspring_1)                                                      │
    │    54 │   │   │   if offspring_1.has_changed() and offspring_1.size() > 0:                       │
    │    55 │   │   │   │   offspring_population.append(offspring_1)                                   │
    │    56                                                                                            │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
    │ stractmosastrategy.py:83 in _mutate                                                              │
    │                                                                                                  │
    │    80 │                                                                                          │
    │    81 │   @staticmethod                                                                          │
    │    82 │   def _mutate(offspring: tcc.TestCaseChromosome) -> None:                                │
    │ ❱  83 │   │   offspring.mutate()                                                                 │
    │    84 │   │   if not offspring.has_changed():                                                    │
    │    85 │   │   │   # if offspring is not changed, we try to mutate it once again                  │
    │    86 │   │   │   offspring.mutate()                                                             │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
    │ :129 in mutate                                                                                   │
    │                                                                                                  │
    │   126 │   │   │   randomness.next_float()                                                        │
    │   127 │   │   │   <= config.configuration.search_algorithm.test_change_probability               │
    │   128 │   │   ):                                                                                 │
    │ ❱ 129 │   │   │   if self._mutation_change():                                                    │
    │   130 │   │   │   │   changed = True                                                             │
    │   131 │   │                                                                                      │
    │   132 │   │   if (                                                                               │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
    │ :182 in _mutation_change                                                                         │
    │                                                                                                  │
    │   179 │   │   │   │   │   continue                                                               │
    │   180 │   │   │   │   old_distance = statement.ret_val.distance                                  │
    │   181 │   │   │   │   ret_val = statement.ret_val                                                │
    │ ❱ 182 │   │   │   │   if statement.mutate():                                                     │
    │   183 │   │   │   │   │   changed = True                                                         │
    │   184 │   │   │   │   else:                                                                      │
    │   185 │   │   │   │   │   assert self._test_factory, "Mutation requires a test factory."         │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
    │ 16 in mutate                                                                                     │
    │                                                                                                  │
    │   1013 │   │   if mutable_param_count > 0:                                                       │
    │   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
    │   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
    │ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
    │   1017 │   │   return changed                                                                    │
    │   1018 │                                                                                         │
    │   1019 │   def _mutable_argument_count(self) -> int:                                             │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
    │ 52 in _mutate_parameters                                                                         │
    │                                                                                                  │
    │   1049 │   │   changed = False                                                                   │
    │   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
    │   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
    │ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
    │   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
    │   1054 │   │   │   │   )                                                                         │
    │   1055                                                                                           │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:11 │
    │ 16 in _mutate_parameter                                                                          │
    │                                                                                                  │
    │   1113 │   │   │   │   │   },                                                                    │
    │   1114 │   │   │   │   ),                                                                        │
    │   1115 │   │   │   )                                                                             │
    │ ❱ 1116 │   │   │   copy.mutate()                                                                 │
    │   1117 │   │   │   possible_replacements.append(copy.ret_val)                                    │
    │   1118 │   │                                                                                     │
    │   1119 │   │   # TODO(fk) Use param_type instead of to_mutate.variable_type,                     │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
    │ 16 in mutate                                                                                     │
    │                                                                                                  │
    │   1013 │   │   if mutable_param_count > 0:                                                       │
    │   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
    │   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
    │ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
    │   1017 │   │   return changed                                                                    │
    │   1018 │                                                                                         │
    │   1019 │   def _mutable_argument_count(self) -> int:                                             │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
    │ 52 in _mutate_parameters                                                                         │
    │                                                                                                  │
    │   1049 │   │   changed = False                                                                   │
    │   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
    │   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
    │ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
    │   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
    │   1054 │   │   │   │   )                                                                         │
    │   1055                                                                                           │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
    │ 72 in _mutate_parameter                                                                          │
    │                                                                                                  │
    │   1069 │   │   current = self._args.get(param_name, None)                                        │
    │   1070 │   │   param_type = inf_sig.parameters[param_name]                                       │
    │   1071 │   │   possible_replacements = self.test_case.get_objects(                               │
    │ ❱ 1072 │   │   │   param_type, self.get_position()                                               │
    │   1073 │   │   )                                                                                 │
    │   1074 │   │                                                                                     │
    │   1075 │   │   # Param has to be optional, otherwise it would be set.                            │
    │                                                                                                  │
    │ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:17 │
    │ 6 in get_position                                                                                │
    │                                                                                                  │
    │    173 │   │   for idx, stmt in enumerate(self._test_case.statements):                           │
    │    174 │   │   │   if stmt == self:                                                              │
    │    175 │   │   │   │   return idx                                                                │
    │ ❱  176 │   │   raise Exception("Statement is not part of it's test case")                        │
    │    177 │                                                                                         │
    │    178 │   def add_assertion(self, assertion: ass.Assertion) -> None:                            │
    │    179 │   │   """Add the given assertion to this statement.                                     │
    ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
    
    

    Looking forward to your reply and thanks in advance!

    opened by Jacob-yen 4
  • ModuleNotFoundError: No module named 'XYZ'

    ModuleNotFoundError: No module named 'XYZ'

    Hello,

    I am using penguin in my project and getting the following error.

    ModuleNotFoundError: No module named 'webapp'.

    My project structure is as follows

    Folder structure:

    • webapp (does not contain __init__.py)
      • server (does not contain __init__.py)
        • module1 (contains __init__.py)
        • module2 (contains __init__.py) ....
        • authUtilities (contains __init__.py)

    The code in module1/__init__.py has an import statement as follows

    from webapp.server.authUtilities import verify_token

    Here's the penguin command I am executing.

    pynguin --project-path XXX/webapp/server/module1 --output-path XXX/webapp/testCases --module-name __init__ -v

    The error I get is

    ModuleNotFoundError: No module named 'webapp'

    Here's the entire trace

    ...../.venv/lib/python3.9/site-packages/pynguin/generator.py:141 in _load_sut │ 138 │ try:
    │ 139 │ │ # We need to set the current thread ident so the import trace is recorded. │ 140 │ │ tracer.current_thread_identifier = threading.current_thread().ident | ❱ 141 │ │ importlib.import_module(config.configuration.module_name) │ 142 │ except ImportError as ex: │ 143 │ │ # A module could not be imported because some dependencies 144 │ │ # are missing or it is malformed

    ..../opt/anaconda3/lib/python3.9/importlib/__init__.py:127 in import_module │ │ 124 │ │ │ if character != '.': │ 125 │ │ │ │ break │ 126 │ │ │ level += 1 │ ❱ 127 │ return _bootstrap._gcd_import(name[level:], package, level) │ 128 | 129
    │ 130 _RELOADING = {} │ :1030 in _gcd_import │ :1007 in _find_and_load │ :986 in _find_and_load_unlocked │ :680 in _load_unlocked

    ...... /.venv/lib/python3.9/site-packages/pynguin/instrumentation/machinery.py:44 in exec_module

                    │    41 │  
                    │    42 │   def exec_module(self, module):                 
                    │    43 │   │   self._tracer.reset()     
                    │ ❱  44 │   │   super().exec_module(module)                 
                    │    45 │   │   self._tracer.store_import_trace()               
                    │    46 │  
                    │    47 │   def get_code(self, fullname) -> CodeType:   
                    
                    │ <frozen importlib._bootstrap_external>:850 in exec_module      
                    
                    │ <frozen importlib._bootstrap>:228 in _call_with_frames_removed  
    

    XXX/webapp/server/module1/__init__.py:4 in │ 1 import json | │ ❱ 4 from webapp.server.authUtilities import verify_token

    I am executing penguin in a virtual environment however not inside a docker.

    Is there a specific switch that I should be using?

    Kindly excuse if this is a novice question, would appreciate any direction.

    opened by samreenmallick 3
  • Segmantation fault while running mutant

    Segmantation fault while running mutant

    Hello.

    I got an segmentation fault error while trying to generate test cases using pynguin.

    Error Message : Running Pynguin...Segmentation fault (core dumped)

    After further analysis, I found the cause. Whenever I use dict.keys() or items() method and the mutant's input to be an empty dictionary({}), the tests fail.

    I prepared a sample code below.

    class Sample:
        def __init__(self):
            pass
    
        def get_user_names(self, user_dict: dict):
            headers = []
    
            for key in user_dict.keys():
                name = user_dict[key]['name']
                headers.append(name)
    
            print(headers)
            return headers
    

    If I check length of the keys first, then It works fine.

    or change line 8 to

    for key in list(user_dict.keys()):
    

    , it works too.

    I think this is a general case. I mean, many developers could encounter this kind of situation. There are too many code lines like this sample in my projects... Hard to fix all of them.

    Please fix this bug or let me know another way to avoid this error. Thanks.

    opened by pjy0121 3
  • If there is any `.so` (e.g., `_C.so`) file in a package, Pynguin stops generating tests and reports `FileNotFoundError: [Errno 2] No such file or directory: '.../_C.py'`

    If there is any `.so` (e.g., `_C.so`) file in a package, Pynguin stops generating tests and reports `FileNotFoundError: [Errno 2] No such file or directory: '.../_C.py'`

    I am trying to generate tests for the project torchvision.

    After compiling torchvision, a file named _C.so is generated inside the package torchvision. Running Pynguin on the compiled project results in the following error: FileNotFoundError: [Errno 2] No such file or directory: '.../_C.py'.

    I found a workaround for now (I just need to add an empty file named _C.py next to the _C.so file).

    I looked into Pynguin's source code. I think the bug is inside the following function: https://github.com/se2p/pynguin/blob/main/pynguin/analyses/seeding/constantseeding.py#L139, which is:

    • Module: pynguin/analyses/seeding/constantseeding.py
    • Function: _find_modules
    • Line: 139

    The problem is that the function iter_modules returns .so files as well as .py files. If I am right, this function needs to check if the modules returned by iter_modules are definitely .py files and skipped otherwise.

    This bug (if it is a bug) can be reproduced by simply putting an empty .so file in a package and trying to generate tests for the modules inside that package, using Pynguin.

    PS: awesome tool!

    opened by mohrez86 3
  • Generating weird folders during test gen

    Generating weird folders during test gen

    Hi, I installed pynguin==0.7.2 and run it with command

    pynguin --algorithm WHOLE_SUITE --project_path /Users/iuliia_volkova2/work/simple-ddl-parser/temp --output_path testgen --module_name simple_ddl_parser.output.common

    I got a dozens of strange named folders in project path (take a look at the screen).

    And got the error ) Screen Shot 2021-04-14 at 1 16 49 PM

    Behaviour re-produceable. Python 3.8.7
    OS: Darwin (MacOS)

    opened by xnuinside 3
  • Cannot generate testcase with import statement e.g. `numpy`

    Cannot generate testcase with import statement e.g. `numpy`

    I try to run pynguin to generate the test case for this example module (numpy_example) containing the following snippet.

    import numpy as np
    
    def gen_array(value):
        return np.array(value)
    
    

    but i got an error as ValueError: no signature found for builtin <built-in function where>

    This is the full stack trace.

    [03:08:51] INFO     Start Pynguin Test Generation…                                                                                        generator.py:110
               INFO     Collecting static constants from module under test                                                                    generator.py:209
               INFO     No constants found                                                                                                    generator.py:212
               INFO     Setting up runtime collection of constants                                                                            generator.py:221
    [03:08:54] INFO     Stop Pynguin Test Generation…                                                                                         generator.py:113
    ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
    │ /usr/local/bin/pynguin:10 in <module>                                                            │
    │                                                                                                  │
    │    7 if __name__ == '__main__':                                                                  │
    │    8 │   os.environ["PYNGUIN_DANGER_AWARE"] = "1"                                                │
    │    9 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                        │
    │ ❱ 10 │   sys.exit(main())                                                                        │
    │   11                                                                                             │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/cli.py:190 in main                               │
    │                                                                                                  │
    │   187 │   set_configuration(parsed.config)                                                       │
    │   188 │   if console is not None:                                                                │
    │   189 │   │   with console.status("Running Pynguin..."):                                         │
    │ ❱ 190 │   │   │   return run_pynguin().value                                                     │
    │   191 │   else:                                                                                  │
    │   192 │   │   return run_pynguin().value                                                         │
    │   193                                                                                            │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:111 in run_pynguin                  │
    │                                                                                                  │
    │   108 │   """                                                                                    │
    │   109 │   try:                                                                                   │
    │   110 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
    │ ❱ 111 │   │   return _run()                                                                      │
    │   112 │   finally:                                                                               │
    │   113 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
    │   114                                                                                            │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:496 in _run                         │
    │                                                                                                  │
    │   493                                                                                            │
    │   494                                                                                            │
    │   495 def _run() -> ReturnCode:                                                                  │
    │ ❱ 496 │   if (setup_result := _setup_and_check()) is None:                                       │
    │   497 │   │   return ReturnCode.SETUP_FAILED                                                     │
    │   498 │   executor, test_cluster, constant_provider = setup_result                               │
    │   499 │   # traces slices for test cases after execution                                         │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:260 in _setup_and_check             │
    │                                                                                                  │
    │   257 │                                                                                          │
    │   258 │   # Analyzing the SUT should not cause any coverage.                                     │
    │   259 │   tracer.disable()                                                                       │
    │ ❱ 260 │   if (test_cluster := _setup_test_cluster()) is None:                                    │
    │   261 │   │   return None                                                                        │
    │   262 │   tracer.enable()                                                                        │
    │   263                                                                                            │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/generator.py:117 in _setup_test_cluster          │
    │                                                                                                  │
    │   114                                                                                            │
    │   115                                                                                            │
    │   116 def _setup_test_cluster() -> ModuleTestCluster | None:                                     │
    │ ❱ 117 │   test_cluster = generate_test_cluster(                                                  │
    │   118 │   │   config.configuration.module_name,                                                  │
    │   119 │   │   config.configuration.type_inference.type_inference_strategy,                       │
    │   120 │   )                                                                                      │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1303 in generate_test_cluster │
    │                                                                                                  │
    │   1300 │   Returns:                                                                              │
    │   1301 │   │   A new test cluster for the given module                                           │
    │   1302 │   """                                                                                   │
    │ ❱ 1303 │   return analyse_module(parse_module(module_name), type_inference_strategy)             │
    │   1304                                                                                           │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1282 in analyse_module        │
    │                                                                                                  │
    │   1279 │   │   A test cluster for the module                                                     │
    │   1280 │   """                                                                                   │
    │   1281 │   test_cluster = ModuleTestCluster(linenos=parsed_module.linenos)                       │
    │ ❱ 1282 │   __resolve_dependencies(                                                               │
    │   1283 │   │   root_module=parsed_module,                                                        │
    │   1284 │   │   type_inference_strategy=type_inference_strategy,                                  │
    │   1285 │   │   test_cluster=test_cluster,                                                        │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1169 in                       │
    │ __resolve_dependencies                                                                           │
    │                                                                                                  │
    │   1166 │   │   )                                                                                 │
    │   1167 │   │                                                                                     │
    │   1168 │   │   # Analyze all functions found in the current module                               │
    │ ❱ 1169 │   │   __analyse_included_functions(                                                     │
    │   1170 │   │   │   module=current_module,                                                        │
    │   1171 │   │   │   root_module_name=root_module.module_name,                                     │
    │   1172 │   │   │   type_inference_strategy=type_inference_strategy,                              │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:1258 in                       │
    │ __analyse_included_functions                                                                     │
    │                                                                                                  │
    │   1255 │   │   if current in seen_functions:                                                     │
    │   1256 │   │   │   continue                                                                      │
    │   1257 │   │   seen_functions.add(current)                                                       │
    │ ❱ 1258 │   │   __analyse_function(                                                               │
    │   1259 │   │   │   func_name=current.__qualname__,                                               │
    │   1260 │   │   │   func=current,                                                                 │
    │   1261 │   │   │   type_inference_strategy=type_inference_strategy,                              │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/module.py:941 in __analyse_function     │
    │                                                                                                  │
    │    938 │   │   return                                                                            │
    │    939 │                                                                                         │
    │    940 │   LOGGER.debug("Analysing function %s", func_name)                                      │
    │ ❱  941 │   inferred_signature = test_cluster.type_system.infer_type_info(                        │
    │    942 │   │   func, type_inference_strategy                                                     │
    │    943 │   )                                                                                     │
    │    944 │   func_ast = get_function_node_from_ast(module_tree, func_name)                         │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/typesystem.py:1157 in infer_type_info   │
    │                                                                                                  │
    │   1154 │   │   """                                                                               │
    │   1155 │   │   match type_inference_strategy:                                                    │
    │   1156 │   │   │   case config.TypeInferenceStrategy.TYPE_HINTS:                                 │
    │ ❱ 1157 │   │   │   │   return self.infer_signature(method, self.type_hints_provider)             │
    │   1158 │   │   │   case config.TypeInferenceStrategy.NONE:                                       │
    │   1159 │   │   │   │   return self.infer_signature(method, self.no_type_hints_provider)          │
    │   1160 │   │   │   case _:                                                                       │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/site-packages/pynguin/analyses/typesystem.py:1214 in infer_signature   │
    │                                                                                                  │
    │   1211 │   │   if inspect.isclass(method) and hasattr(method, "__init__"):                       │
    │   1212 │   │   │   return self.infer_signature(getattr(method, "__init__"), type_hint_provider)  │
    │   1213 │   │                                                                                     │
    │ ❱ 1214 │   │   method_signature = inspect.signature(method)                                      │
    │   1215 │   │   hints = type_hint_provider(method)                                                │
    │   1216 │   │   parameters: dict[str, ProperType] = {}                                            │
    │   1217 │   │   for param_name in method_signature.parameters:                                    │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/inspect.py:3253 in signature                                           │
    │                                                                                                  │
    │   3250                                                                                           │
    │   3251 def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
    │   3252 │   """Get a signature object for the passed callable."""                                 │
    │ ❱ 3253 │   return Signature.from_callable(obj, follow_wrapped=follow_wrapped,                    │
    │   3254 │   │   │   │   │   │   │   │      globals=globals, locals=locals, eval_str=eval_str)     │
    │   3255                                                                                           │
    │   3256                                                                                           │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/inspect.py:3001 in from_callable                                       │
    │                                                                                                  │
    │   2998 │   def from_callable(cls, obj, *,                                                        │
    │   2999 │   │   │   │   │     follow_wrapped=True, globals=None, locals=None, eval_str=False):    │
    │   3000 │   │   """Constructs Signature for the given callable object."""                         │
    │ ❱ 3001 │   │   return _signature_from_callable(obj, sigcls=cls,                                  │
    │   3002 │   │   │   │   │   │   │   │   │   │   follow_wrapper_chains=follow_wrapped,             │
    │   3003 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
    │   3004                                                                                           │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/inspect.py:2467 in _signature_from_callable                            │
    │                                                                                                  │
    │   2464 │   │   │   │   │   │   │   │   │   │   globals=globals, locals=locals, eval_str=eval_st  │
    │   2465 │                                                                                         │
    │   2466 │   if _signature_is_builtin(obj):                                                        │
    │ ❱ 2467 │   │   return _signature_from_builtin(sigcls, obj,                                       │
    │   2468 │   │   │   │   │   │   │   │   │      skip_bound_arg=skip_bound_arg)                     │
    │   2469 │                                                                                         │
    │   2470 │   if isinstance(obj, functools.partial):                                                │
    │                                                                                                  │
    │ /usr/local/lib/python3.10/inspect.py:2274 in _signature_from_builtin                             │
    │                                                                                                  │
    │   2271 │                                                                                         │
    │   2272 │   s = getattr(func, "__text_signature__", None)                                         │
    │   2273 │   if not s:                                                                             │
    │ ❱ 2274 │   │   raise ValueError("no signature found for builtin {!r}".format(func))              │
    │   2275 │                                                                                         │
    │   2276 │   return _signature_fromstr(cls, func, s, skip_bound_arg)                               │
    │   2277                                                                                           │
    ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
    ValueError: no signature found for builtin <built-in function where>
    
    opened by jdanceze 1
  • How to avoid digging deep into the dependencies

    How to avoid digging deep into the dependencies

    Hi,

    In my project, I am using dependencies such as sqlalchemy, boto3, and pynguin reports an error as - │ /work/venvs/py310/lib/python3.10/site-packages/sqlalchemy/orm/decl_api.py:309 in cascading │ │ │ │ 306 │ │ │ │ 307 │ │ │ │ 308 │ │ """ │ │ ❱ 309 │ │ return cls._stateful(cascading=True) │ │ 310 │ │ 311 │ │ 312 class _stateful_declared_attr(declared_attr): │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: _stateful_declared_attr._stateful() missing 1 required positional argument: 'self'

    My project runs fine although.

    Is there a way to make penguin avoid looking deeper into the code, which I have not written such as these.

    What is the guidelines to ignore these dependencies, shall we mock them from our side before running penguin. Please guide.

    Thanks.

    opened by ravindrabajpai 2
  • Restructuring help and questions/discussion. (Issue was: honor max-iteration setting)

    Restructuring help and questions/discussion. (Issue was: honor max-iteration setting)

    I am runing pynguin 0.17.0 with Python 3.9.10. I created some example code here:

    https://github.com/inktrap/pynguin-example

    Basicly it seems that pynguin does not value the max iteration option and perhaps other options too.

    opened by inktrap 9
  • Use @pytest.mark.parametrize when generating pytest test cases

    Use @pytest.mark.parametrize when generating pytest test cases

    The currently generated tests have a lot of repeated code. Is it possible to just generate a list of tuples of input data and expected result that are usable by pytest parametrize mark decorator?

    Let me demonstrate using the example from the documentation:

    def test_case_1():
        var0 = -2603
        var1 = module0.triangle(var0, var0, var0)
        assert var1 == "Equilateral triangle"
        var2 = 1272
        var3 = module0.triangle(var0, var2, var2)
        assert var3 == "Isosceles triangle"
    

    This can be rewritten using pytest.mark.parametrize:

    @pytest.mark.parametrize(
        "a, b, c, expected",
        [
            (-2603, -2603, -2603, "Equilateral triangle"),
            (-2603, 1272, 1272, "Isosceles triangle"),
        ],
    )
    def test_case_1(a, b, c, expected):
        assert module0.triangle(a, b, c) == expected
    

    This can be even improved if testdata would be declared separately as a list of tuples and later used once in the tescase decorator. See more examples here: https://docs.pytest.org/en/6.2.x/example/parametrize.html#paramexamples

    It would be nice to also have test case names or ids to reflect what area of testing (e.g. boundary-value analysis or some edge case). Perhaps that is more of a n issue for the algorithm used to generate test cases. But i am cramming all this in one issue, which is not right.

    For me personally the pytest parametrize usage would instantly upgrade this tool from academic interest to imediately usefull tool.

    opened by kbaikov 1
  • Equivalent of #pragma: no cover

    Equivalent of #pragma: no cover

    Hello, Thank you for open-sourcing this package. I am testing your tool with some of my code. Do you have a way to exclude some codes from the test generation step ? Like #pragma: no cover in the coverage python package.

    For example my traceback when encountered some numba functions.

    numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
    cannot determine Numba type of <class 'pynguin.testcase.execution.executiontracer.ExecutionTracer'>
    
    File "fklab/events/basic_algorithms.py", line 671:
    def _bsearchi(vector, key):  # pragma: no cover
        <source elided>
    
        nmemb = len(vector)
        ^
    

    Thank you !

    opened by MarineChap 1
Releases(0.27.0)
  • 0.27.0(Sep 23, 2022)

    • Write Pynguin version number to Cobertura XML report

    • Fix the computation of coverage values

      Pynguin 0.26.0 added the possibility to optimise for, e.g., branch coverage while yielding the line coverage of the resulting test suite in the end. This had some unintentional implications, such as the value of the Coverage output variable having weird values. We fix this by providing additional output variables FinalBranchCoverage and FinalLineCoverage that contain the final coverage values after all generation and postprocessing. See the runtimevariable.py module for details.

    • Rewrite the internal type handling in Pynguin

      This is a huge internal change that improves the internal type system and adds the possibility to trace types during test execution. Some highlights:

      • we got rid of our internal abstraction of type | None to mark whether type information exists
      • we allow to retrieve additional or new type information from the test execution
      • enhance our internal type representation to make it more flexible
      • add the missing primitive type complex
      • a more suitable subtyping check for Unions
    Source code(tar.gz)
    Source code(zip)
  • 0.26.0(Sep 19, 2022)

    • Allow the calculation of coverage values regardless of optimisation.

      Allows to yield, for example, the resulting line coverage of the test suite, while the optimisation during test generation was done for branch coverage. Add the required coverage values to your --output-variables list to activate this feature.

      Note: when doing this, the Coverage output variable will contain the average value of the incorporated coverage values!

    • Provide a Cobertura-like coverage report

      Pynguin already provides an HTML report that can be activated by setting --create-coverage-report True. This report is nice for human users but not very usable if one wants to automatically reason about the achieved coverage using tools. We thus also emit an XML report in the style of the Cobertura tool that can be used for further automated tools.

    • Fix typo on test_parameterizedstatements.py (see #27, thanks to @stavares843)

    • Fix typo on testcase.py (see #26, thanks to @stavares843)

    • Improve mutation-based assertion generation

      The assertion generation now does not more compare the recorded assertion traces but actually executes the assertions to make the whole process more reliable. Besides, we do not check for is not None as a fallback for object checks any more, but use an isinstance check on the object's type to have a more precise assertion.

    • Make statistics tests debuggable

      Some accidental circular import made it impossible to run a debugger on tests in pynguin.utils. We resolved this by moving tests to another package.

    • Partial rework of internal type system

      We added an abstraction layer over the existing type hints from a module to make handling and reasoning with types easier. This abstraction is based on the one used by mypy, however, we only cover a small part of what PEP-484 actually defines.

    • Make assertion generation more strict

    Source code(tar.gz)
    Source code(zip)
  • 0.25.2(Jul 14, 2022)

  • 0.25.1(Jul 6, 2022)

  • 0.25.0(Jul 5, 2022)

    • Fix further issues with test-case isolation during execution.
    • Fix some bugs reegarding type information.
    • Use astroid instead of Python's ast module efor our module analysis due to its enhanced features.
    Source code(tar.gz)
    Source code(zip)
  • 0.24.1(Jun 29, 2022)

  • 0.24.0(Jun 29, 2022)

  • 0.23.0(Jun 24, 2022)

    • Provide a naive inheritance graph to improve input generation.

    • Improve killing of long-running test-case executions

    • Add computation of mutation scores for MUTATION_ANALYSIS assertion generation.

      The output variables NumberOfCreatedMutants, NumberOfKilledMutants, NumberOfTimedOutMutants, and MutationScore allow to export those values.

    • Do not enable typing.TYPE_CHECKING for SUT analysis as this may cause circular imports.

    • Improve the black list of modules that shall not be incorporated into the test cluster.

    • Annotate failing tests with @pytest.mark.xfail(strict=True).

    • Improve log output of mutation-based assertion generation.

    • Add instrumentation to mutated modules to easier kill them.

      This change is relevant only to the MUTATION_ANALYSIS assertion-generation strategy.

    • Write errors in execution threads to the log instead of STDERR to avoid cluttering log output.

    • Add limits for amount and size of constants in the constant pool.

      The configuration options max_dynamic_length and max_dynamic_pool_size allow to set sizes for the maximum length of strings/bytes that shall be stored in the dynamic constant pool and the maximum numbers of constants of a type, respectively. This prevents the constant pool from growing unreasonably large.

    • Improve handling of type annotations.

    • Fix computation of cyclomatic complexity.

      Computing cyclomatic complexity does not work for functions that are not present in the AST, e.g., default constructors. We now omit those from the computation of the cyclomatic-complexity output variables.

    Source code(tar.gz)
    Source code(zip)
  • 0.22.0(Jun 8, 2022)

    • Fix selection of type-inference strategy.

    • Fix a bug in the type inference regarding cases where not type information is present.

    • Add a PyLint checker for calls to print().

    • Extend the blacklist of modules that shall not be analysed.

    • Raise RuntimeError from tracer when called from another thread.

    • Provide better exception messages for critical failures.

    • Apply a further limit to the execution time of a single generated test case to at most 10 seconds.

    • Exclude empty enum classes from test cluster to fix test generation.

      Parsing included modules raised an issue when the enum module is used: the test cluster then had a reference to the enum.Enum class, which obviously does not contain any fields. In the following, generating tests failed, as soon as this class was selected to fulfil parameter values because there was no field to select from, e.g., MyEnum.MY_FIELD. We now exclude empty enums from the test cluster.

    Source code(tar.gz)
    Source code(zip)
  • 0.21.0(May 25, 2022)

    • Fix a bug in the module analysis regarding nested functions

      Nested functions/closures caused Pynguin's module analysis to crash with an failing assertion.

    • Improve the branch-distance computation for bool values

    • Allow for more statistics variables regarding number of lines and cyclomatic complexity

    Source code(tar.gz)
    Source code(zip)
  • 0.20.1(May 24, 2022)

  • 0.20.0(May 24, 2022)

    Breaking Changes

    • Remove splitting into passing and failing test suite.

      Previously, we consider a test case passing if it did not raise any exception during its execution; it was considered failing otherwise. Pynguin did a split of the test cases into two test suites before exporting them. This was mainly an artefact from implementing the random algorithm in the very beginning of the project. Due to the improved assertion export for exception assertions we can now get rid of the split and export only one test module containing all generated test cases.

    • Remove the option to use a log file (--log_file or --log-file).

      Pynguin writes its output to STDOUT/STDERR now, if requested by the -v/-vv switch. This output is formatted by @willmcgugan's amazing rich library. A user can disable the output formatting by setting the --no-rich flag. Of course, because we believe that rich is such an awesome library, we also provide an alias for this flag, called --poor😉

    Further Changes

    • Distinguish between expected and unexpected exceptions.

      We consider an exception to be expected if it is explicitly raised in the code under test or is documented in the code's docstring. For those exceptions we build an with pytest.raises block around the exception-raising statement. All other exceptions are considered to be unexpected. We decorate the test method with the @pytest.mark.xfail decorator and let the exception happen. It is up to the user to decide whether such an exception is expected. An exception here is the AssertionError: it is considered to be expected as soon as there is an assert statement in the code under test.

    • Improve installation description to explicitly point to using a virtual environment (see GitHub issue #23, thanks to @tuckcodes).

    • Improve variable names and exception assertions

      The assertion generation got an improved handling for asserting on exceptions, which creates more meaningful and (hopefully) better understandable assertions for exceptions.

    • Enhance the module analysis

      This is basically a rewrite of our previously existing test cluster, which keeps track of all the callables from the subject under test as well as the subject's dependencies. It also incorporates an analysis of the subject's AST (if present) and allows for more and more precise information about the subject which can then improve the quality of the generated tests.

    • To distinguish bytecode instructions during instrumentation we add an ArtificialInstr for our own added instructions.

    • Fix a bug in the tracing of runtime types.

      During assertion generation Pynguin tracks the variable types to decide for which values it actually is able to generate assertions. Creating an assertion on a generator function does not work, as the type is not exposed by Python but only present during runtime—thus generating an object of this type always fail. We mitigate this by ignoring objects of type builtins.generator from the assertion generation.

    • Improve documentation regarding coverage measurement and the coverage report

    Source code(tar.gz)
    Source code(zip)
  • 0.19.0(Mar 16, 2022)

    Breaking Change

    • One can now use multiple stopping conditions at a time.

      This breaks the CLI in two ways:

      • The parameter --stopping-condition has been removed.
      • The parameter --budget was renamed to --maximum-search-time.

      Users have to change their run configurations accordingly!

      To specify stopping conditions, add one or many from --maximum-search-time, --maximum-test-executions, --maximum-statement-executions, and --maximum-iterations to your command line.

    Further Changes

    • Clarify log output for search phases
    • Pynguin now uses the ast.unparse function from Python's AST library instead of the third-party astor library to generate the source code of the test cases.
    Source code(tar.gz)
    Source code(zip)
  • 0.18.0(Mar 15, 2022)

    Breaking Change

    • We drop the support for Python 3.8 and Python 3.9 with this version!

      You need Python 3.10 to run Pynguin! We recommend using our Docker container, which is already based on Python 3.10, to run Pynguin.

    Further Changes

    • Add line coverage visualisation to the coverage report.
    • Add a citation reference to our freshly accepted ICSE'22 tool demo paper “Pynguin: Automated Unit Test Generation for Python.
    • Unify the modules for the analysis of the module under test.
    Source code(tar.gz)
    Source code(zip)
  • v0.17.0(Feb 4, 2022)

    • Add line coverage as another coverage type (thanks to @labrenz). The user can now choose between using either line or branch coverage or both by setting the parameter --coverage-metrics to LINE, BRANCH, or LINE,BRANCH. Default value is BRANCH.
    Source code(tar.gz)
    Source code(zip)
  • v0.16.1(Jan 17, 2022)

  • v0.16.0(Jan 17, 2022)

    • Refactor the assertion generation. This unifies the SIMPLE and the MUTATION_ANALYSIS approaches. Furthermore, Pynguin now uses the MUTATION_ANALYSIS approach again as the default.
    • Update the type annotations in Pynguin's code to the simplified, future versions (e.g. instead of Dict[str, Set[int]] we can now write dict[str, set[int]]) and do not need any imports from Python's typing module.
    • Fix a crash of the seeding when native modules are present. Fixes #20.
    • Provide a hint in the documentation that PyCharm 2021.3 now integrates poetry support, thus no plugin is required for this (and newer) versions (thanks to @labrenz).
    Source code(tar.gz)
    Source code(zip)
  • v0.15.0(Dec 13, 2021)

    • Fix a bug for mutating a statement that is not in the current test case (see #17).
    • Set default assertion generation to SIMPLE due to issues with the experimental new generation strategy.
    • Add GitHub Actions that also runs our CI chain on GitHub.
    Source code(tar.gz)
    Source code(zip)
  • v0.14.0(Dec 6, 2021)

    • Breaking: Simplify the logging such that Pynguin uses different log levels also for the log file. This removes the -q option to make all outputs quiet.
    • Pynguin now also supports field accesses during test generation. This is a preliminary feature.
    • Fix a deadlock in the executor
    • Pynguin now uses Python 3.10 as its default version for the provided Docker container as well as our CI. Still, Pynguin supports Python 3.8 and 3.9. Up to now, Python 3.11 is not yet supported.
    • Refactor the state-trace implementation
    • Remove the module-loader singleton
    • Fix loading of mutated module for assertion generation. Caused that no assertion was generated because the mutants were not loaded properly. This is a regression from merging the assertion-generation strategy in Pynguin 0.13.0.
    Source code(tar.gz)
    Source code(zip)
  • v0.13.2(Nov 10, 2021)

    • Add the reference to the preprint of our EMSE journal submission to documentation.
    • Clarify on the value of the PYNGUIN_DANGER_AWARE environment variable. One can set an arbitrary value for the variable, Pynguin only checks whether its defined (#16 ).
    Source code(tar.gz)
    Source code(zip)
  • v0.13.1(Nov 5, 2021)

  • v0.13.0(Nov 5, 2021)

    • Add an assertion-generation strategy based on mutation (thanks to @f-str).

      The new strategy is enabled by default (can be configured using the --assertion-generation parameter). It uses a custom fork of the mutation-testing framework MutPy, which mutates the subject under test and for each mutant traces the values of fields and returned by method calls. If these values differ between mutant and original code, an assertion is generated based on the result on the original subject under test. One can also control whether the strategy shall also generate assertions for unchanged values by the --generate-all-assertions flag.

      The release also updates the documentation accordingly.

      Note: This feature is an early prototype of such an assertion generation, which might cause unexpected behaviour. You can switch back to the previous strategy for assertion generation by setting --assertion-generation SIMPLE.

    Source code(tar.gz)
    Source code(zip)
  • v0.12.0(Nov 5, 2021)

    • Generate more reasonable variable names in tests. Before this release, Pynguin only generated variables named var0, var1, etc. A simple heuristics now attempts to generate more reasonable names depending on the type of the variable, such as int_0, bool_1, or str_2. We also adjusted the documentation to match this change.
    • We updated all provided PyCharm run configurations the use the more sophisticated queue example instead of the simple example module to see an improved output.
    • Prevent a potential regression when updating the dependencies to version 0.0.17 of the simple-parsing library for CLI argument parsing, which changed its API.
    Source code(tar.gz)
    Source code(zip)
  • v0.11.0(Nov 3, 2021)

    • Fix a control-dependency bug in DynaMOSA. Loops in the control-dependence graph caused DynaMOSA to not consider certain targets because they were control dependent on goals that had not yet been covered due to the loop.
    • Improve documentation
    • Split and extend FitnessValues to avoid expensive re-computations. This also extends the API of the FitnessValues and refactors large parts of the fitness handling.
    • Fix for bumpiness of flaky tests. Whenever Pynguin generates a test that behaves flaky result could be that coverage over time looks like ventricular fibrillation especially for the MIO algorithm. The fix prevents this by carefully revisiting the equality of chromosomes.
    • Improve handling of entry/exit nodes in the CFG; this fixes issues with Python 3.10
    Source code(tar.gz)
    Source code(zip)
  • v0.10.0(Oct 6, 2021)

    • Provide support for Python 3.10
    • Pynguin now set typing.TYPE_CHECKING = True explicitly before parsing the subject under test in order to be able to collect also information about types that are only imported due to type checking/providing type annotations.
    • Improved generation of collection statements
    • Cleanup the implementation of the algorithms
    • Add supports for enums in the test-generation process
    • Cleanup the implementation of the dynamic value seeding
    • Make Pynguin executions as deterministic as we possibly can
    • Make DynaMOSA the default algorithm
    • Allow the generation of an HTML coverage report similar to the one generated by Coverage.py. This allows to show the subject under test and the coverage achieved by the test cases generated by Pynguin in the web browser.
    • Add a CITATION.cff file
    • Improve the internal control-flow graph
    • Improve the documentation
    • Cleanup and remove unused code
    • Fix a bug in post-processing
    • Fix a bug in branch coverage instrumentation on for loops
    • Add a variant of the whole-suite algorithm that uses an archive
    • Guard imports that are only necessary for type checking in Pynguin's modules by if typing.TYPE_CHECKING conditions
    Source code(tar.gz)
    Source code(zip)
  • v0.9.2(Jun 21, 2021)

    • Add explicit code-execution prevention (thanks to @Wooza).

      Pynguin now requires you to set the PYNGUIN_DANGER_AWARE environment variable before it actually does test generation. This was added due to the fact that Pynguin executes the module under test, including its dependencies, which could potentially cause harm on the user's system. By requiring the variable to be set explicitly, a user confirms that they are aware of this issue. Inside the Docker container, the variable is set automatically; we highly recommend this way of executing Pynguin!

    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(Jun 17, 2021)

  • v0.9.0(Jun 7, 2021)

  • v0.8.1(Jun 1, 2021)

    • Regroup configuration options
    • Improve branch-distance calculations for data containers
    • Save import coverage to a separate output variable
    • Delete some unused code
    • Add warning notice to read-me file
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Apr 26, 2021)

    • Breaking: Renamed RANDOM_SEARCH to RANDOM_TEST_SUITE_SEARCH to select the random-sampling algorithm based on test suites introduced in Pynguin 0.7.0.

    • Improve input generation for collection types.

    • Add an implementation of tournament selection for the use with DynaMOSA, MOSA, and Whole Suite.

      For Whole Suite, on can choose the selection algorithm (either rank or tournament selection) by setting the value of the --selection parameter.

    • Add DynaMOSA test-generation algorithm.

      It can be selected via --algorithm DYNAMOSA.

    • Add MIO test-generation algorithm.

      It can be selected via --algorithm MIO.

    • Add a random sampling algorithm based on test cases.

      The algorithm is available by setting --algorithm RANDOM_TEST_CASE_SEARCH. It randomly picks one test case, adds all available fitness functions to it and adds it to the MOSA archive. If the test case is covering one fitness target it is retrieved by the archive. If it covers an already covered target but is shorter than the currently covering test case for that target, it replaces the latter.

    • Fix OSError from executors queue.

      The queue was kept open until the garbage collector delete the object. This caused an OSError because it reached the OS's limit of open resource handles.
      We now close the queue in the test-case executor manually to mitigate this.

    • Fix __eq__ and __hash__ of parameterised statements.

      Before this, functions such as foo(a) and bar(a) had been considered equivalent from their equals and hash-code implementation, which only compared the parameters and returns but not the actual function's name.

    • Fix logging to work properly again.

    Source code(tar.gz)
    Source code(zip)
Owner
Chair of Software Engineering II, Uni Passau
Chair of Software Engineering II, Uni Passau
Simple assertion library for unit testing in python with a fluent API

assertpy Simple assertions library for unit testing in Python with a nice fluent API. Supports both Python 2 and 3. Usage Just import the assert_that

null 19 Sep 10, 2022
A pytest plugin to run an ansible collection's unit tests with pytest.

pytest-ansible-units An experimental pytest plugin to run an ansible collection's unit tests with pytest. Description pytest-ansible-units is a pytest

Community managed Ansible repositories 9 Dec 9, 2022
AllPairs is an open source test combinations generator written in Python

AllPairs is an open source test combinations generator written in Python

Robson Agapito Correa 5 Mar 5, 2022
A small automated test structure using python to test *.cpp codes

Get Started Insert C++ Codes Add Test Code Run Test Samples Check Coverages Insert C++ Codes you can easily add c++ files in /inputs directory there i

Alireza Zahiri 2 Aug 3, 2022
a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)

pytest-sugar pytest-sugar is a plugin for pytest that shows failures and errors instantly and shows a progress bar. Requirements You will need the fol

Teemu 963 Dec 28, 2022
Ab testing - The using AB test to test of difference of conversion rate

Facebook recently introduced a new type of offer that is an alternative to the current type of bidding called maximum bidding he introduced average bidding.

null 5 Nov 21, 2022
Python wrapper of Android uiautomator test tool.

uiautomator This module is a Python wrapper of Android uiautomator testing framework. It works on Android 4.1+ (API Level 16~30) simply with Android d

xiaocong 1.9k Dec 30, 2022
A complete test automation tool

Golem - Test Automation Golem is a test framework and a complete tool for browser automation. Tests can be written with code in Python, codeless using

null 486 Dec 30, 2022
RAT-el is an open source penetration test tool that allows you to take control of a windows machine.

To prevent RATel from being detected by antivirus, please do not upload the payload to TOTAL VIRUS. Each month I will test myself if the payload gets detected by antivirus. So you’ll have a photo every month to prove RAtel’s discretion.

null 218 Dec 16, 2022
A simple tool to test internet stability.

pingtest Description A personal project for testing internet stability, intended for use in Linux and Windows.

chris 0 Oct 17, 2021
Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes in a variety of languages.

Mimesis - Fake Data Generator Description Mimesis is a high-performance fake data generator for Python, which provides data for a variety of purposes

Isaak Uchakaev 3.8k Dec 29, 2022
FakeDataGen is a Full Valid Fake Data Generator.

FakeDataGen is a Full Valid Fake Data Generator. This tool helps you to create fake accounts (in Spanish format) with fully valid data. Within this in

Joel GM 64 Dec 12, 2022
HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom

hey is a tiny program that sends some load to a web application. hey was originally called boom and was influenced from Tarek Ziade's tool at tarekzia

Jaana Dogan 14.9k Jan 7, 2023
WomboAI Art Generator

WomboAI Art Generator Automate AI art generation using wombot.art. Also integrated into SnailBot for you to try out. Setup Install Python Go to the py

nbee 7 Dec 3, 2022
To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations

To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations, a lot of data has to be collected to ensure the variance of the tests. This respository was established to collect a lot of different test data and related test cases of different member states in a standardized manner. Each member state can generate a folder in this section.

null 160 Jul 25, 2022
Green is a clean, colorful, fast python test runner.

Green -- A clean, colorful, fast python test runner. Features Clean - Low redundancy in output. Result statistics for each test is vertically aligned.

Nathan Stocks 756 Dec 22, 2022
splinter - python test framework for web applications

splinter - python tool for testing web applications splinter is an open source tool for testing web applications using Python. It lets you automate br

Cobra Team 2.6k Dec 27, 2022
A test fixtures replacement for Python

factory_boy factory_boy is a fixtures replacement based on thoughtbot's factory_bot. As a fixtures replacement tool, it aims to replace static, hard t

FactoryBoy project 3k Jan 5, 2023
A test fixtures replacement for Python

factory_boy factory_boy is a fixtures replacement based on thoughtbot's factory_bot. As a fixtures replacement tool, it aims to replace static, hard t

FactoryBoy project 2.4k Feb 5, 2021