Description

By seeing if the bytecode for a mutant is equal to the bytecode for the original program we can easily determine equivalence for some mutations.

Description

Currently vertigo expects that there are live test networks available.
Regularly one would estup several instances of ganache to allow for parallel mutation testing.
However, due to the nature of ganache this results in millions of files and hundreds of GB's being created over the course of one mutation run.

A more effective and user friendly approach is to dynamically create a new ganache instance for each test run, using a ramdisk rather than local storage.
This also removes the need to manually setup different ganache processes, which is another win in usability

@JoranHonig Hello. Great work on this project. I've just started playing with it a bit on a fairly large codebase. I ran a full test (--sample-ratio = 1.0) last night, and the results were 0/247 mutants killed. I was suprised by this, so I picked 1 mutation at random and manually tested it by switching the operator and running some tests targeting this bit of code. I quickly had 5 failing tests, which makes me question the results from this.

I'm wondering if there's any way to tweak settings to produce more accurate results?
I've tried running a smaller sample-ratio (0.1) which just produced

Mutation testing report:
Number of mutations:    24
Killed:                 0 / 24

Not sure if I'm doing something wrong here? If so, please feel free to close this issue.
For reference, here's the command & options I passed:

vertigo run --network vertigo_test_network_1 --network vertigo_test_network_2 --truffle-location /Users/overlord/DAPPS/unlock/smart-contracts/node_modules/.bin/truffle --output ./vertigo_output.txt --sample-ratio 0.1

Description

For a while now truffle has preferred the use of truffle-config.js over that of truffle.js
Vertigo currently only supports the latter, but should be changed to allow both to make the usage of vertigo that much more simple

When e.g. vertigo run --hardhat-parallel 8 is run in a newly-created Typescript Hardhat project, it errors with:

[*] Could not find supported project directory in <project dir>

I believe this is because the _directory_type function on line 203 of eth_vertigo/cli/main.py fails to check for hardhat.config.ts. However, when that file is renamed to hardhat.config.js the test still fails with a the following error:

[*] Starting mutation testing
[*] Starting analysis on project
[*] Initializing campaign run 
[*] Checking validity of project
[-] Encountered an error while running the framework's test command:
Encountered error during test output analysis

Note that I also tried updating eth_vertigo/cli/main.py and eth_vertigo/cli/hardhat/tester.py locally to add references to hardhat.config.ts and reinstalled vertigo but it still produces the error above.

I can also add that printing the test_result variable on line 133 of eth_vertigo/interfaces/common/tester.py prints nothing to the console. It looks like the preamble flag on line 123 never gets set to False when run in a TypeScript project, but does when run in a JavaScript project.

Description

The https://github.com/agroce/universalmutator is very cool and has already implemented some mutation operators for solidity.

Vertigo can be extended to leverage universalmutor in two aspects:

• the universalmutator solidity operators
• easy project specific mutation operators

Universal mutator paper: https://agroce.github.io/icse18t.pdf

The code currently doesn't look in the truffle config file for a contracts directory, it just looks in the repo root:

I should be able to use the configured truffle directory if I wanted to.

module.exports = {
  contracts_directory: "./src/contracts",
  networks: {

Should be an easy fix, if you really need I can make the PR :)

Vertigo showing this error when run with Hardhat. Can someone help me to solve this issue?

Current design of Vertigo will attempt to generate all possible mutants in the defined mutation space, only then report will be generated. However, it is likely that there is no enough budget to perform the evaluation of all possible mutants. For example, it can take ~20 mins to run the test suite of OpenZeppelin. So, it's impractical to wait for the testing task to complete for the report. As a result, a functionality to limit the execution boundary might be a good idea, and maybe also make Vertigo generate report at exit (potentially unexpected) as long as one mutant was generated and evaluated so that external tool like timeout can be used to bound the execution timeout.

Thanks

Hi,
I wonder if I can can set hardhat location with vertigo.
The problem I am facing is that my project structure is a bit unusual.

node_modules
project1
  contracts
    SomeContract.sol
  test
    SomeContact.test.js
project2
  contracts
    SomeContract2.sol
  test
    SomeContact2.test.js

As you can see the node_modules is not in the same level as contracts directory.
When I am in project1 directory, I run vertigo run --hardhat-parallel 8, I get the following error:

[-] Encountered an error while running the framework's test command:
Error HH12: Trying to use a non-local installation of Hardhat, which is not supported.

Then I created a new project structure. It's almost the same as above but node_modules is in the project1.

project1
  node_modules
  contracts
    SomeContract.sol
  test
    SomeContact.test.js

Then it works.

[+] Survivors
[*] Done!

By this experiment, I assume that vertigo can't find local hardhat.
Would it be possible to set local hardhat?
Or please let me know If I am pointing in wrong direction of the problem.

Thanks in advance.

When installing, I noticed that the repo still points to eth-vertigo and not https://github.com/joranhonig/vertigo in the build script's setup.py

It is very interesting to consider the use case of incremental mutation analysis.

The approach that I propose can be implemented in Vertigo is the following:

• Each time we evaluate a mutant we generate a tuple of the mutation & the tests that kill the mutant (if the mutant is killed).
• In each consecutive analysis run, for each mutant that we have seen before we first execute those tests that killed it the previous mutation analysis run.

This approach is based on the following hypothesis:

Within two versions of a program a majority of the mutants that are shared between the two increments of a program will be killed by the same tests.

Some related work:

• https://pitest.org/quickstart/incremental_analysis/
• Towards Incremental Mutation Testing - M.A. Cachia, M. Micallef, C. Colombo

Hi, I try to use this repo to test some Solidity Contracts.
I use last version of Truffle, Ganache and Python.
I follow your tutorial on medium with the MetaCoin and it works fine, but when I change the code in test folder to add asserts to kill some mutants, my new asserts seems ignored.
I understand how the mutation works but I don't really understand how the code works to test mutants.

How do you test contracts with your code ?
Do you run a truffle test ?
Do the code test .js or .sol files ?
Do i missed something ?

Thanks.

Hi @JoranHonig, thanks for the great tool.

Extreme mutation operators consist of removing the whole body of functions, see A Comprehensive Study of Pseudo-tested Methods.

They have the advantage of revealing important testing problems with very few mutants.

What about adding support for extreme mutation operators in vertigo?

cc/ @bbaudry @Deee92

For many mutation testing optimisations it is required / helpful if we have code coverage information.
Specifically we need to be able to determine which tests cover specific lines of code.

@JoranHonig I'm wondering whether the comparison of bytecodes is after stripping the metadata in the bytecode that are not semantic related? As I don't see any code doing so.
Also, optimization is currently not enforced which should already eliminate lots of the functionally equivalent ones.

See exerpt below:

[+] Survivors
Mutation:
    File: /home/.../Contract.sol
    Line nr: 495
    Result: Lived
    Original line:
             @notice Update the times for start, stop and uncap.

    Mutated line:
             @notice Update the times fo < art, stop and uncap.

Relevant code in my contract:

/**
    @notice Update the times for start, stop and uncap.
    @dev The start time cannot be changed if people already contributed. No time can be changed if the deal is over.
    @param startTimestamp Timestamp when the deal opens to investors
    @param endTimestamp Timestamp when the deal closes to investors
    @param uncapTimestamp Timestamp when the maxAlloc personal cap stops being enforced
    @param _publicTimestamp Timestamp when the whitelist stops being enforced. Set to `endTimestamp` to always enable,
        set to zero to always disable.
    */
    function updateTimestamps(
        uint256 startTimestamp,
        uint256 endTimestamp,
        uint256 uncapTimestamp,
        uint256 _publicTimestamp
    ) external onlyRole(ROLE_MANAGER) { /* ... */ }

Vertigo currently looks for hardhat.config.js when it's doing mutation testing on hardhat projects.

Hardhat actually allows you to use custom configuration names, so, Vertigo will need that too.

Let's add --hardhat-config <config_file> to allow users to provide custom configurations

hi
I need to define new operators. But I do not know where to define them?
According to the solidity document, there are a number of security considerations. I want to define an operator for these considerations.
https://docs.soliditylang.org/en/v0.6.6/security-considerations.html?highlight=Call%20stack%20depth#callstack-depth

Please tell me how to define these operators for solidity?

tx.origin to msg.sender replacement
msg.sender to tx.origin replacement
address to address payable replacement
Constant to immutable replacement
Remove selfdestruct from a function
Call stack depth >= 1024

Running the test suite on windows results in the following error:

==================================== FAILURES ==================================== 
_______________________________ test_relative_path _______________________________ 

tmp_path = WindowsPath('C:/Users/donger/AppData/Local/Temp/pytest-of-donger/pytest-5/test_relative_path0')

    def test_relative_path(tmp_path):
        # Arrange
        (tmp_path / "testdir").mkdir()
        sf_file: Path = tmp_path / "testdir" / "sf.txt"
        sf_file.write_text("some text", encoding="utf-8")
        sf = SourceFile(sf_file)
    
        mutation = Mutation(None, sf, None, tmp_path)
    
        # Act
        relative_path = mutation.relative_path
    
        # Assert
>       assert "testdir/sf.txt" == relative_path
E         - testdir/sf.txt
E         ?        ^
E         + testdir\sf.txt
E         ?        ^

test\mutation\test_mutation.py:42: AssertionError

I believe the way to make this work on both Posix and Windows would be to convert these both to Path objects before comparing them.

Hi Joran, thanks for a great tool!

Currently Vertigo checks for existence of either a truffle config or a hardhat config. If both are present though, it finds neither and exits.

It would be great if the logic was such that when both are present, Vertigo runs whichever framework the user specified in their cmd line input.

Currently vertigo leverages a range of command line parameters to configure mutation testing runs.
With a decently sized projects one might end up using +10 command line parameters, which is-- not ideal.

To solve this vertigo should be extended with the ability to use configuration files.

Example of what this could look like:

vertigo run -c vertigo.yml

Example configuration:

networks:
    - development 1