domohuhn / mutation-test Goto Github PK

Having a command line flag that takes a string as argument for the verification command would users a quick start.

• Add -c/--command="xxxxx" and -t/--timeout=60 as command line options

For most CI tools, YAML files are used as inputs. The binary should switch from xml files to YAML files as inputs, as its syntax is easier for humans to read and write.

Specifying (flutter test test/some_test_file.dart) as command in the input files (ex: config.xml) is throwing the following Error / Exception :
"Error while processing:
ProcessException: Das System kann die angegebene Datei nicht finden.

Command: flutter test test/example_test.dart"

Are "flutter test" commands not supported ?

• the statistics block should contain all information in a single table
• program version should be in the report
• total runtime should be included
• Reporting should be moved out of the test runner

The reports currently only show the undetected mutations in the files. Showing all mutations gives a better picture in the reports.

The name flag for commands serves no purpose- remove it.

Users should be able to choose if they want the builtin rules or not.

Use list of files from the command line arguments as input files if the file extension is not .xml. This allows users to run tests without having to create a xml file with the inputs.

Having the ability to combine multiple rule documents gives users more flexibility how they use the application.

Creating output will fail with an exception if the path to the input file does not contain a /.

The output on the command line should be improved by including a progress bar and an remaining estimate.

The configuration should contain thresholds that describe the percent of undetected mutations for a run to be considered as failed.

For the given String 'a+b' the Regex will be a match and it will be executed, it's a false positive since it's not an arithmetic operation.

For String templates, only mutates if it's using variables such as '$a+$b', '${a + b}', etc.

The tests currently do not run code in several files (and codecov doesnot see them either):

• html_reporter.dart
• mutation_progress_bar.dart
• mutation_test.dart
• report_format.dart
• test_runnner.dart

The deliveries for linux and mac are missing the executable permissions and should not have the .exe extension. It is probably better to use .tgz for linux as archive format.

Ideally, this should be automated using a github action.

What could be causing such an Error ?

PS C:\Users\M\Desktop\my_app> dart run mutation_test mutation_testing/config.xml --rules mutation_testing/small_test.xml -d
Error while processing:
RangeError (start): Invalid value: Not in inclusive range 0..176: 179

My config.xml : 

<?xml version="1.0" encoding="UTF-8"?>
<mutations version="1.0">
  <!-- License: BSD-3-Clause  -->
  <directories>
    <directory recursive="true">lib
      <matching pattern="\.dart$"/>
    </directory>
  </directories>
  <exclude>
    <!-- lines can also be globally excluded  -->
    <lines begin="1" end="2"/>
  </exclude>
  
  <commands>
   <!-- <command group="test" expected-return="0" working-directory="./">dart //version</command> -->
    
    <command group="test" expected-return="0" working-directory="./">C:\scr\flutter\bin\flutetr.bat test</command>

  </commands>
</mutations>


my small_test.xml : 

<?xml version="1.0" encoding="UTF-8"?>
<mutations version="1.0">
   <!-- License: BSD-3-Clause  -->
   <!-- The rules element describes all mutations done during a mutation test -->
   <!-- The following children are parsed: literal and regex -->
   <!-- A literal element matches the literal text -->
   <!-- A regex element mutates source code if the regular expression matches -->
   <!-- Each of them must have at least one mutation child -->
   <rules>


    <!--this replaces (x op y,x) and (x op y,y)-->
    <regex pattern="([A-Za-z0-9._]+(?:\(\)|\s*))\s+([\*\/\+\-\~])\s+([A-Za-z0-9._]+(?:\(\)|\s*))" dotAll="false">
    <mutation text="$1"/>
    <mutation text="$3"/>
    </regex>
  
  </rules>
  <!-- This element creates a blacklist, allowing you to exclude parts from the mutations -->
  <exclude>

    <!-- excludes anything between two tokens  -->
    <token begin="//" end="\n"/>
    <token begin="#" end="\n"/>
    <token begin="import" end=";"/>

    <!-- excludes anything that matches a pattern  -->
    <regex pattern="/[*].*?[*]/" dotAll="true"/>
    <!-- exclude loops to prevent infinte tests -->
    <regex pattern="[\s]for[\s]*\(.*?\)[\s]*{" dotAll="true"/>
    <regex pattern="[\s]while[\s]*\(.*?\)[\s]*{.*?}" dotAll="true"/>
    <!-- lines can also be globally excluded  -->
    <!-- line index starts at 1  -->
    <lines begin="1" end="2"/>

  </exclude>

</mutations>

We are trying to run a mutation test for our project daily nightly. The estimated time to complete the test shown by the tool is ~21h 12m 17s, which makes it hard to run the mutation test for the whole project.

As far as I understood, this tool generates the mutants for the code regardless of whether the test suites cover it. If so, it will be better to generate mutants only for the code covered by the test suites.

I'd also like to know how mutants are subjected to test suites. Is every mutant subjected to the whole test suite or only the test that covers the mutant itself?

Running without any commandline options should assume the standard dart layout and glob all files in ./lib/ as inputs and use dart test as test command.

Maybe all OS licenses should be included in releases.

Similar to https://github.com/muter-mutation-testing/muter#muter-can-be-run-in-your-ci

Mutation tests are usually slow to run, I'd like to test only the modified files in my PR via CI.

The mutated lines should be enclosed in div elements and their style should be set via css instead of inline styles.

https://github.com/domohuhn/mutation-test/blob/9eff7dee88402e93a0e26fc6941d29fdbc088b0e/lib/src/configuration/builtin_rules.dart#L156C10-L160

Positional and Optional positional parameters may have bugs if misplaced, but I don't see how changing the order of named parameters would help.

I am currently working on a Flutter testing project to implement effective mutation testing techniques. However, these techniques require a kill matrix that can illustrate which test kills which mutant. I was wondering if there is any way to detect this using built-in mechanisms of the tool