stryker-mutator / stryker-jest-runner Goto Github PK

View Code? Open in Web Editor NEW

27.0 6.0 5.0 274 KB

A plugin to use the Jest test runner and framework in Stryker, the JavaScript mutation testing framework

stryker-plugin stryker-testrunner

stryker-jest-runner's People

Contributors

Stargazers

Watchers

Forkers

gdaolewe jankuca mthmulders pawgre liamondrop

stryker-jest-runner's Issues

"SyntaxError: Unexpected token import" with create-react-app-ts

Currently trying out Stryker on my Jest + TypeScript React application. Work is on the feature/mutation-testing branch. I've added a work-around for #38, but now I'm facing a new issue.

The configuration seems fine and Stryker attempts its initial test run. Every test suite fails with this message:

    /Users/maarten/Projects/hyperion-web/.stryker-tmp/sandbox8787577/build/dist/__tests__/apiSpec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import * as fetchMock from 'fetch-mock';
                                                                                            ^^^^^^
    
    SyntaxError: Unexpected token import
    
    at new Script (vm.js:51:7)
        at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:118:7)

I'm using create-react-app-ts, which uses ts-jest under the hood. This hooks into Jest using a preprocessor which invokes tsc to transpile the Typescript code to JavaScript first.

I've tried the following in stryker.conf.js:

project: 'default',
project: 'react',
no project at all

All mutants survive because no coverage

@jankuca reported this issue in #2

Mutants are created but none fail. My npm test script is simply jest which runs tests and fails when I manually apply a mutant myself.

69 total mutants.
69 mutants survived.
  of which 69 were not covered by the tests.
0 mutants timed out.
0 mutants killed.
Ran 0.00 tests per mutant on average.
Mutation score based on covered code: n/a
Mutation score based on all code: 0.00%

My stryker.conf.js file:

module.exports = function (config) {
  config.set({
    files: [
      { pattern: 'src/**/!(*.spec).js', included: false, mutated: true },
      { pattern: 'src/**/*.spec.js', included: true, mutated: false },
    ],
    testRunner: 'jest-test-runner',
    coverageAnalysis: 'off',
    reporter: [ 'clear-text', 'progress' ],
  });
}

Default jest config file jest.config.js is ignored.

jest and NODE_ENV environment variable

When I run jest my process.env.NODE_ENV is equal to test.
But when I run mutation test it is not equal to test (it's undefined)

Error: Cannot find module './myComponent.component.js' - Trouble with require()

Hi all,

Currently I am facing some problems with this plugin.

I guess I configured everything correctly :

module.exports = function(config) {
  config.set({
    files: [
      "./directory/desktop/app/components/myComponent/myComponent.test.js",
    ],
    testRunner: "jest",
    mutator: "javascript",
    transpilers: [],
    reporter: ["html", "clear-text", "progress"],
    coverageAnalysis: "off"
  });
};

When I execute my unit test via gulp it works pretty fine but when I executed

node_modules/.bin/stryker run

here is what I get :

[2018-01-30 14:46:37.091] [ERROR] InitialTestExecutor - One or more tests resulted in an error:
        Error: Cannot find module './myComponent.component.js' from 'myComponent.test.js'
[2018-01-30 14:46:37.092] [ERROR] StrykerCli - an error occurred Error: Something went wrong in the initial test run

Is there anything wrong by using require that I am missing ? Also if I remove this require then the error is that angular is not defined so I was wondering if jest.init.js were exectuted at some point ? ( Which is for me the same problem because in jest.init.js I am mainly importing module via require().

Thanks in advance ;-)

CRA You need Jest version >= 22.0.0 to use Stryker error

First of all, I am elated to find a mutation test framework for JS. I come from a quick check and PIT testing background and place a high value on the quality of tests. I am trying to configure stryker with my CRA project and getting the following error even-though the peer dependencies are met. An entry in the CRA docs on how to configure stryker would be great for devs and growing the community.

stryker.conf.js

 module.exports = function (config) {
  config.set({
    jest: {
      project: 'react'
    },
    testRunner: 'jest',
    mutator: 'javascript',
    coverageAnalysis: 'off',
    mutate: ['src/**/*.js']
  });
};

I have the following dependencies and jest config in package.json

 "devDependencies": {
    "axe-testcafe": "^1.1.0",
    "axios-mock-adapter": "^1.12.0",
    "chalk": "^2.4.0",
    "enzyme": "^3.3.0",
    "enzyme-adapter-react-16": "^1.1.1",
    "eslint": "~4.12.1",
    "eslint-config-react-app": "^2.1.0",
    "eslint-plugin-testcafe": "^0.2.1",
    "flow-bin": "~0.70.0",
    "flow-typed": "2.4.0",
    "gulp": "^3.9.1",
    "gulp-maven-deploy": "~0.2.0",
    "jest-emotion": "^9.0.0-0",
    "react-scripts": "1.1.4",
    "redux-devtools-extension": "^2.13.2",
    "stryker": "0.21.0",
    "stryker-api": "0.15.0",
    "stryker-jest-runner": "^0.7.0",
    "testcafe": "^0.19.2"
  },
  "jest": {
    "collectCoverageFrom": [
      "src/**/*.js",
      "!**/mock-backend/**",
      "!**/types/**"
    ]
  }

package.json jest config not respected

Hi,

I've been trying to add Stryker to my project, but have been hitting several bumps along the way. Right now I think the issue comes from jest:
I'm using Enzyme for my tests. This requires me to have setupFiles configured for jest in package.json (https://facebook.github.io/jest/docs/en/configuration.html#setupfiles-array). It does not appear that those files are called, since enzyme throws an error.

An easy way to reproduce is something like this:

// __tests__/my-test.js
it('dummy', () => expect(setup).toBe(true))

// package.json
{
...
"jest": {"setupFiles": ["<rootDir>/setup.js"]}
...
}

// setup.js
global.setup = true;

//src.js
if(1 == 1) {
  console.log("ABC")
}

Expected: Test reports mutants survived (jest test passes)
Actual: ReferenceError: setup is not defined

Use Jest types once they are up to date

The types for Jest on definitelytyped are currently not working, we should however include them once they are.

Move to monorepo

@mthmulders is it time to move? Moving to the mono repo will help with keeping the stryker-jest-runner up-to-date with the latest stryker releases. It will also help with issue tracking amongst others.

Add stryker-api 0.15 support

Add support for new file API

JestConfigEditor test failing

When working on #38, I noticed that one test case is currently failing. It fails not only on my machine, but also on Travis:

    1) Integration JestConfigEditor
        should create a jest configuration for a react project:

        AssertionError: expected { Object (collectCoverageFrom, setupFiles, ...) } to deeply equal { Object (collectCoverageFrom, setupFiles, ...) }
        + expected - actual

            "src/**/*.{js,jsx,mjs}"
            ]
            "moduleFileExtensions": [
            "web.js"
        +    "mjs"
            "js"
            "json"
            "web.jsx"
            "jsx"
            "node"
        -    "mjs"
            ]
            "moduleNameMapper": {
            "^react-native$": "react-native-web"
            }

stryker-jest-runner unable to run

Hi, I was trying out Stryker on a project using jest and stumpled upon an error when trying to run Stryker with the newest dependency versions. It looks similar to the one reported in issue #354 in stryker and fixed in version 0.1.0 of the stryker-jest-runner plugin.
I have reproduced the error in a mutant-testing repo forked from the example project in issue #15. The dependency versions in package.json are updated to:

"devDependencies": {
  "babel-jest": "^21.2.0",
  "babel-preset-env": "^1.6.0",
  "jest": "^21.2.1",
  "jest-cli": "^21.2.1",
  "regenerator-runtime": "^0.11.0",
  "stryker": "^0.13.0",
  "stryker-api": "^0.11.0",
  "stryker-jest-runner": "^0.2.2"
}

Running Stryker then fails with the following output:

$ npm run stryker

> [email protected] stryker /home/ckp/mutant-testing
> stryker run stryker.conf.js

[2017-10-31 16:01:53.920] [WARN] PluginLoader - Error during loading "stryker-jest-runner" plugin:
  Cannot find module 'jest-cli/build/runTest'
[2017-10-31 16:01:53.924] [INFO] ReporterOrchestrator - Detected that current console does not support the "progress" reporter, downgrading to "progress-append-only" reporter
[2017-10-31 16:01:53.938] [INFO] InputFileResolver - Found 1 of 2 file(s) to be mutated.
[2017-10-31 16:01:53.939] [INFO] InitialTestExecutor - Starting initial test run. This may take a while.
[2017-10-31 16:01:54.090] [ERROR] TestRunnerChildProcessAdapter - /home/ckp/mutant-testing/node_modules/stryker-api/src/core/Factory.js:43
            throw new Error("Could not find a " + this.factoryName + " with name " + name + ", did you install it correctly (for example: npm install --save-dev " + this.importSuggestion(name) + ")?");
            ^

Error: Could not find a testrunner with name jest, did you install it correctly (for example: npm install --save-dev stryker-jest-runner)?
    at TestRunnerFactory.Factory.create (/home/ckp/mutant-testing/node_modules/stryker-api/src/core/Factory.js:43:19)
    at IsolatedTestRunnerAdapterWorker.start (/home/ckp/mutant-testing/node_modules/stryker/src/isolated-runner/IsolatedTestRunnerAdapterWorker.js:60:80)
    at process.<anonymous> (/home/ckp/mutant-testing/node_modules/stryker/src/isolated-runner/IsolatedTestRunnerAdapterWorker.js:20:27)
    at emitTwo (events.js:106:13)
    at process.emit (events.js:191:7)
    at process.nextTick (internal/child_process.js:787:12)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)

[2017-10-31 16:01:54.094] [ERROR] TestRunnerChildProcessAdapter - Child process exited with non-zero exit code 1. Last 10 message from the child process were: 
	[2017-10-31 16:01:54.081] [DEBUG] PluginLoader - Loading stryker-* from /home/ckp/mutant-testing/node_modules
	[2017-10-31 16:01:54.086] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner (matched with expression stryker-*)
	[2017-10-31 16:01:54.086] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner
	[2017-10-31 16:01:54.088] [WARN] PluginLoader - Error during loading "stryker-jest-runner" plugin:
  Cannot find module 'jest-cli/build/runTest'
	[2017-10-31 16:01:54.088] [DEBUG] IsolatedTestRunnerAdapterWorker - Changing current working directory for this process to /home/ckp/mutant-testing/.stryker-tmp/sandbox9923194
[2017-10-31 16:01:54.095] [ERROR] StrykerCli - an error occurred Test runner child process exited with non-zero exit code 1

My reason for updating all the dependencies was to try out mutation testing of Typescript code in the latest 0.10.0 version of Stryker.
Any ideas what could be wrong?

Jest v20 is not supported

When running stryker with jest@20 i get this error:

$ stryker run
[2017-08-10 18:49:25.840] [INFO] ConfigReader - Using stryker.conf.js in the current working directory.
[2017-08-10 18:49:26.015] [INFO] InputFileResolver - Found 1 of 2 file(s) to be mutated.
[2017-08-10 18:49:26.016] [INFO] SandboxCoordinator - Starting initial test run. This may take a while.
[2017-08-10 18:49:26.685] [ERROR] IsolatedTestRunnerAdapter - (node:17004) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Runtime.createHasteContext is not a function

[2017-08-10 18:49:26.685] [ERROR] IsolatedTestRunnerAdapter - (node:17004) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Reproduce:

git clone [email protected]:nicojs/mutant-testing.git
cd mutant-testing
npm i
npm run stryker

Support snapshot tests

Currently, when running Jest tests with snapshots, the stryker-jest-runner will fail.

Code to reproduce:

test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toMatchSnapshot();
});

Error message:

One or more tests failed in the initial test run:
	adds 1 + 2 to equal 3
		Error: expect(value).toMatchSnapshot()

New snapshot was not written. The update flag must be explicitly passed to write a new snapshot.

This is likely because this test is run in a continuous integration (CI) environment in which snapshots are not written by default.

Override jest config settings

Right now, the Jest config of the user is used without making any changes. I believe we should make some changes to speed up the stryker-jest-runner

Config setting	Default value	Suggest value	Reason
bail	`false`	`true`	As soon as a single test fails, we can stop testing. In the best case, we run 1 test. In the worst case we still run 'n' tests. We also do this with Mocha.
collectCoverage	`false`	`false`	We don't need to collect coverage since we don't have a test framework plugin for Jest.
verbose	`false`	`false`	Verbose logging slows down the test run.

All config options: https://facebook.github.io/jest/docs/en/configuration.html

Jest runner not returning empty error messages as strings

I am getting the following error in IsolatedTestRunnerAdapterWorker.reportResult

[2017-02-12 11:20:21.035] [ERROR] Stryker - One or more tests errored in the initial test run:
TypeError: Cannot read property 'toString' of undefined

The Jest runner is returning an array of undefined when the are no test failures.

jest runner not registering as a plugin correctly

The plugin is loaded as shown here

[2017-02-12_ 11:00:47.051] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner (matched with expression stryker-*)
[2017-02-12 11:00:47.054] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner

However this error follows

Error: Could not find a testrunner with name jest, did you install it correctly (for example: npm install --save-dev stryker-jest-runner)?
    at TestRunnerFactory.Factory.create

I believe this is due to this in index.ts

TestRunnerFactory.instance().register('jest-test-runner', JestTestRunner);

That should be

TestRunnerFactory.instance().register('jest', JestTestRunner);

Running without custom Jest configuration isn't possible

If you don't have a jest.config.js file, nor a jest key in the package.json, Stryker will fail:

[2018-04-02 20:52:17.345] [INFO] ConfigReader - Using stryker.conf.js in the current working directory.
/Users/maarten/Projects/hyperion-web/node_modules/stryker-jest-runner/src/configLoaders/DefaultJestConfigLoader.js:16
            throw new Error('Could not read Jest configuration, please provide a jest.config.js file or a jest config in your package.json');

As a work-around, create a jest.config.js with just

{
}

I don't think this is a nice approach. Maybe we should just let Stryker could handle this situation?

Downgrade node types to version 6

Since node 6 is still in LTS we should support it, it makes sense to downgrade the types for TypeScript to Node 6 to make sure we do not use any of the node functions that are not supported in Node 6. The build will eventually fail but it would be better to get instant feedback.

Failed to update from 0.3.0

I've tried to update stryker-jest-runner from 0.3.0 to latest 0.5.1, and got error:

[2018-03-09 16:27:12.818] [ERROR] InitialTestExecutor - One or more tests resulted in an error:
        : ? Validation Error:

  Module <rootDir>/node_modules/ts-jest/preprocessor.js in the transform option was not found.

  Configuration Documentation:
  https://facebook.github.io/jest/docs/configuration.html

? Validation Error:

  Module <rootDir>/node_modules/ts-jest/preprocessor.js in the transform option was not found.

  Configuration Documentation:
  https://facebook.github.io/jest/docs/configuration.html

[2018-03-09 16:27:12.819] [ERROR] StrykerCli - an error occurred Error: Something went wrong in the initial test run
    at InitialTestExecutor.validateResult (D:\My\Dev\nest-typescript-starter\node_modules\stryker\src\process\InitialTestExecutor.js:98:15)
    at InitialTestExecutor.<anonymous> (D:\My\Dev\nest-typescript-starter\node_modules\stryker\src\process\InitialTestExecutor.js:33:30)
    at step (D:\My\Dev\nest-typescript-starter\node_modules\tslib\tslib.js:133:27)
    at Object.next (D:\My\Dev\nest-typescript-starter\node_modules\tslib\tslib.js:114:57)
    at fulfilled (D:\My\Dev\nest-typescript-starter\node_modules\tslib\tslib.js:104:62)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Tried to update to 0.4.0 and got same error.
Version 0.3.0 worked without errors.

stryker.config.js

    config.set({
        files: [
            { pattern: 'app/**/*.graphql', included: false, mutated: false },
        ],
        testRunner: 'jest',
        mutator: 'typescript',
        transpilers: ['typescript'],
        reporter: ['html', 'clear-text', 'progress'],
        coverageAnalysis: 'off',
        tsconfigFile: 'tsconfig.json',
        jest: {
            project: 'default',
            config: require('./jest.config'),
        },
        mutate: [
            'app/**/cat.controller.ts',
        ],
        timeoutMs: 10 * 1000,
        // timeoutFactor: 2,
        htmlReporter: {
            baseDir: '.testresults/mutation'
        },
        logLevel: 'debug',
    });

jest.config.js

module.exports = {
    "testEnvironment": "node",
    "transform": {
        "^.+\\.tsx?$": "<rootDir>/node_modules/ts-jest/preprocessor.js"
    },
    "collectCoverage": false,
    "coverageDirectory": ".testresults",
    "coverageReporters": [
        "lcov",
        "text",
    ],
    "collectCoverageFrom": [
        "app/**/*.ts",
        "!app/**/*.spec.ts",
        "!app/**/*.ispec.ts"
    ],
    "testMatch": [
        "<rootDir>/app/**/*.spec.ts"
    ],
    "moduleFileExtensions": [
        "ts",
        "js"
    ]
};

Stryker stops if one of the mutations is not passing - using Jest

I started a project only to test Stryker, so i created a function that sum two number and created the test for it, and it worked great! But when i created more functions and tests and made stryker run, it creates the sandbox directores and gives me this error message:

[2018-02-28 08:38:28.924] [ERROR] TestRunnerChildProcessAdapter - Child process exited with non-zero exit code 1. Last 10 message from the child process were:
[2018-02-28 08:38:22.199] [DEBUG] PluginLoader - Loading stryker-* from C:\Users\davi.carvalho\Documents\stryker_mutant_killing\node_modules

    [2018-02-28 08:38:22.203] [DEBUG] PluginLoader - Loading plugins stryker-html-reporter (matched with expression stryker-*)

    [2018-02-28 08:38:22.204] [DEBUG] PluginLoader - Loading plugins stryker-javascript-mutator (matched with expression stryker-*)

    [2018-02-28 08:38:22.204] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner (matched with expression stryker-*)

    [2018-02-28 08:38:22.205] [DEBUG] PluginLoader - Loading plugins stryker-html-reporter

    [2018-02-28 08:38:22.258] [DEBUG] PluginLoader - Loading plugins stryker-javascript-mutator

    [2018-02-28 08:38:23.991] [DEBUG] PluginLoader - Loading plugins stryker-jest-runner

    [2018-02-28 08:38:24.052] [DEBUG] IsolatedTestRunnerAdapterWorker - Changing current working directory for this process to C:\Users\davi.carvalho\Documents\stryker_mutant_killing\.stryker-tmp\sandbox3640639

[2018-02-28 08:38:28.989] [ERROR] StrykerCli - an error occurred Test runner child process exited with non-zero exit code 1

After this message, i ran jest and saw that when jest tries to run the mutation tests, and if those tests fail, Stryker just stops running. I can't even get the report from the html-reporter, so I keep in the black without knowing what should I coverage to make them pass.
Url to repo i'm using: https://github.com/DavideCarvalho/mutant_test