jimivdw / grunt-mutation-testing Goto Github PK

When unit tests fail when generating the dummy coverage report, configuration of either the code or the specs is wrong. However, the program does not notice that, will give some warnings, and will not run properly (but still run somehow).

Issue is related to #31 and other reported problems (not reported on this GitHub).

Edit: the problem appeared to be even bigger: the check if tests were failing without mutations at all is broken.

As came forward in #40, it is not easy to exclude files from the configuration options.

Whereas in most Grunt plugins, one can simply do ['js/*.js', '!js/*.test.js'], this is not possible in our setup, or at least can only be achieved via ['js/!(*.test).js'].

It was suggested to me that Browserify might allow for in-memory mutations, as well as code parsing and formatting.

This, at the very least, calls for an investigation of the possibilities of Browserify.

In case the Karma code-specs pair does not contain a single spec for a certain file, this file still gets marked with a 100% mutation score. It should rather be marked as 'untested'.

Has anyone tried integrating this plugin with Jenkins?
There is https://github.com/jenkinsci/pitmutation-plugin for Java PIT. Maybe that one could be reused?

I've been trying to get the plugin to work, but wasn't successful. Here's my code: https://github.com/mattzeunert/grunt-mutation-testing-example

There are two problems I'm running into:

1. Getting "Tests fail without mutations" message
2. Says survived after removing function, but if do it manually in Karma it fails as expected

I'm getting the first issue with the setup as it is in my repo.

Here's the output I'm getting, after adding a console.log for the config that is passed to Karma: https://gist.github.com/mattzeunert/bc338a18a97d86fd0da6

It seems to pass in these files:

1. [ 'src/add.js', 'test/*.js' ]
2. [ 'src/add.js', 'test/add.spec.js' ]
3. [ 'src/add.js' ]

Is that expected? Why am I getting the "tests fail without mutation" even though they work in Karma?

I've worked around the first issue by forcing self._config.files = [ 'src/add.js', 'tests/add.spec.js' ]; in KarmaServerManager.

This is the output I get after that: https://gist.github.com/mattzeunert/3f8dcf666eaa2238a9a5

(13:40:31.573) INFO [mutation-testing]: Mutating file:     /var/folders/tg/r1vq4wwd641fbx971h2gyqp00000gn/T/mutation-testing11579-30414-h3y7d7/src/add.js
(13:40:31.575) INFO [mutation-testing]: Mutating line 1, 1/3 (33%)
(13:40:31.575) TRACE [KarmaServer]: Server status changed to: RUNNING
(13:40:31.710) TRACE [KarmaServer]: Server status changed to: READY

/src/add.js:1:1 Removed function add(a, b){ return a + b; } -> SURVIVED

When I remove the function myself the test doesn't survive in Karma. Is it possible to see exactly what the code looked like after the mutation?

Hi! Thanks for an amazing tool. It's been fun to put my tests on a test.

I found that some of the mutations almost always produce false positives:

• Replacing === with == and !== with != and vice versa.
• Replacing "use strict" with "MUTATION!" or removing it altogether.

Very rarely will these mutations produce a test failure. Like in my code base all files begin with "use strict" - so as a result I almost never get 100% mutation success rate. And whether == or != produces a failure is a moot point for me, as the coding standards require to always use triple-equals.

I think both of these mutations are better detected with code style checking tools.

For the triple-equals case I found a quick hack to turn it off. But so far I haven't quite managed to get rid of the "use strict" case.

I'm having trouble with this plugin when also using the karma-coverage plugin.

If I don't add karma-coverage to my karma conf file, I can run my tests with karma and also the mutation tests without issue. If I add the karma-coverage plugin, I can run my tests with karma and get the coverage report, but I can no longer run the mutation tests and I get errors like:

29 02 2016 17:20:21.646:WARN [mutation-testing-karma]: Could not find specs for file: /var/folders/rr/cbmh00gx3rn0_869mnd1n7g40000gn/T/mutation-testing116129-57978-fpmsst/public/javascripts/controllers/waterfallController.js
29 02 2016 17:20:21.646:WARN [mutation-testing]: Could not properly set up Karma server for file: /var/folders/rr/cbmh00gx3rn0_869mnd1n7g40000gn/T/mutation-testing116129-57978-fpmsst/public/javascripts/controllers/waterfallController.js, skipping...
/public/javascripts/controllers/waterfallController.js could not properly set up Karma server

I encountered the same thing when trying to run the mutation tests in https://github.com/lvandiest/mutation-test-demo, which also makes use of the karma-coverage plugin.

KarmaCodeSpecsMatcher expects to see a nesting for the coverage output, which is generally true. However, when there's no coverage, will be blank. As a result, the ".forEach" call invokes an error and the program outputs a warning about invalid format. It still falls back on a full mapping, which works, but I'm trying to improve run speed.

I was able to adjust this so it would return a blank result if the "lines-covered" was zero, and that seems to do what I expect it to. I can make a pull request if desired, https://github.com/divido/grunt-mutation-testing/tree/fix-karma-code-specs

However, I surprised this hasn't come up. Wouldn't everybody always have zero base coverage? It doesn't include any specs -- there's therefore nothing to run. I'm concerned that I'm missing the point, or have something misconfigured. Heck, maybe I broke it myself with #60.

In case it's related, I'm using the karma-dojo plugin, and all of my source uses AMD-style modules.

Many issues until now have been caused by a wrong configration (#47, #31, #3, and many others not filed on this GitHub). As suggested in #47, the documentation should be extended.

We need to look at fixing this error:

(node) warning: possible EventEmitter memory leak detected. 11 uncaughtException listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at process.addListener (events.js:179:15)
...

Could not the useful functionality of this codebase be extracted into something not depending upon using Grunt, and then a Grunt-runner for that extracted toolkit be written?

Not every project uses Grunt. And most that don't, don't wish to add an additional task-runner framework to their existing meta-project dead-weight. :P

Is there a particular rationale that I'm not seeing for this to be tightly tied to Grunt?

My projects structure is set up like this.

js
├── calc.js
└── calc.test.js

How can I exclude calc.test.js from the files for mutation? Here is what I tried.

options: {
  code: ['js/*.js'],
  specs: ['js/*.test.js'],
  mutate: ['js/*.js', '!js/calc.test.js']
}

I believe that ['js/*.js', '!js/calc.test.js'] is supported my minimatch and grunt, but it doesn't appear to work here. Is there something to mark files not to be mutated?

Hi @shybyte

I'm trying grunt-mutation-testing in a project I'm working on and I keep getting Potentially unhandled rejection [...] Error. What's the meaning of this? do you have any documentation or reference I can got to with debug messages?

Thanks!

When I run tests normally, they all pass.
When I comment out some tests then mutation works fine, but when I uncomment some tests then I get

(16:32:03.783) INFO [KarmaServerManager]: Karma server started after 3100ms and is listening on port 12111
[2016-05-27 16:32:03.894] [DEBUG] config - Loading config /home/project/test/karma.conf.js
(16:32:05.896) WARN [mutation-testing-karma]: Warning! Infinite loop detected. This may put a strain on your CPU.
(16:32:05.896) ERROR [mutation-testing-karma]: Fatal: Unfortunately the mutation test cannot recover from this error and will shut down
(16:32:05.898) WARN [mutation-testing]: Tests fail without mutations for file: /tmp/mutation-testing116427-5613-p3fcyj/activeArtifact.controller.js
(16:32:05.899) WARN [mutation-testing]: This failure may be due to a misconfiguration of either `code` or `specs`. Did you include your external libraries?

Arithmetic mutation (and probably also comparison expression) breaks the syntax of the mutated code.
Consider :

var answer = 6 + (7 + 31);

The relevant syntax tree fragment looks like this:
{ "range": [13,25], "type":"BinaryExpression", "operator":"+", "left":{ "range":[13,14], "type":"Literal", "value":6, "raw":"6" }, "right":{ "range":[18,2], "type":"BinaryExpression", "operator":"+", "left":{ "range":[18,19], "type":"Literal", "value":7, "raw":"7" }, "right":{ "range":[22,24], "type":"Literal", "value":31, "raw":"31" } } }
Evidently 'left' ends at range index 14 and 'right' starts at range index 18. That leaves 14(which is the space after 14),15,16 and 17, or ' + ('
The resulting mutation (after replacing + by -) is:

var answer = 6-7 + 31);

There are some workarounds. But the only real solution to me seems to me to do the mutation in the AST and (using codegen) serialise that.

I'm looking into using mutation testing on a dojo-heavy project, so I started using grunt-mutation-testing. I'm using karma and jasmine, and have used karma's dojo framework plugin for support.

When I run the mutation task in grunt, it complains about tests failing prior to mutation.

I believe I've traced this down to an interaction between the KarmaServerManager and Karma's Server module, during the handoff of karma's config structure. One of the fields that grunt-mutation-testing overwrites is the "files", which it supplies as an array of strings. This is valid for Karma, but karma also supports arrays of objects of the form { pattern: "", included: false }. For interactions with dojo, it is important to have all dojo modules specified, but not included, and have a single "bootstrap" or similar function that is included.

Just prior to sending the config to Karma, I replaced it with a hard-coded value unique to my test project, and it worked correctly. I'm looking into a more general solution now, but I wanted to bring it to your attention in case I missed something or you had some suggestions on how to implement the fix.

 appfiles git:(mutate-experiment) ✗ grunt mutationTest
Running "mutationTest:target" (mutationTest) task
Fatal error: Could not connect to a Karma server on port 9091 within 10 seconds


Execution Time (2015-11-03 09:17:14 UTC)
loading tasks        117ms  ▇▇ 1%
mutationTest:target    10s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99%
Total 10.1s

 appfiles git:(mutate-experiment) ✗ ps
  PID TTY           TIME CMD
 4171 ttys000    0:01.00 /usr/local/bin/node /Users/me/Documents/workspace/project/appfiles/node_modules/grunt-mutation-testing/lib/karma/KarmaWorker.js
98054 ttys000    0:01.81 -/bin/zsh
88114 ttys001    0:00.88 -/bin/zsh
 1478 ttys003    3:39.36 grunt
94526 ttys003    0:00.66 -/bin/zsh
 appfiles git:(mutate-experiment) ✗

Subsequent runs causes more zombie processes.

i'm trying to test out this project on an es6 project, but am getting the following error for each file:

(00:15:52.332) WARN [mutation-testing]: Tests fail without mutations for file: /var/folders/9l/q9klhdys15xcxrqxpmh76ccr0000gn/T/mutation-testing11659-56658-9msma5/src/index.js
(00:15:52.333) WARN [mutation-testing]: This failure may be due to a misconfiguration of either `code` or `specs`. Did you include your external libraries?

my test/mocha.opts file is configured to apply babel when running the tests, which works properly in all of the other ways that i run my tests. when defining the testFramework is set to mocha does it preserve the default mocha behavior of using the mocha.opts file under test/? even if so does the code need to be transpiled before the mutations can be applied?

also, is there some way to see the output of the failure that is happening? i'm sure that would be very informative related to understanding what needs to change. even with the logLevel set to ALL, i see none of the output from the underlying failure.

if it would be helpful, the project i'm currently trying to test this out on is https://github.com/travi/admin.travi.org-components

Hi All, for some reason this plugin does not seem to be working for karma. I debugged and what I could see is the server does not start. It always gives a error that "No server is running on ". I tried the plugin for mocha it seems to be working fine. But for karma it does not work. following is the code added to the grunt file -

    mutationTest: {
        karma: {
            options: {
                karma: {
                    configFile: 'node_modules/grunt-mutation-testing/test/fixtures/karma-mocha/karma.conf.js',
                    waitForServerTime: 5 // optional, default = 5s
                }
            },
            files: {
                'tmp/karma.txt': ['node_modules/grunt-mutation-testing/test/fixtures/karma-mocha/script*.js']
            }
        }
    },

Following is the content of the conf file -

// Karma configuration
// Generated on Wed Apr 30 2014 16:43:43 GMT+0200 (CEST)

    module.exports = function (config) {
        config.set({

        // base path that will be used to resolve all patterns (eg. files, exclude)
        basePath: '../',

        /**
         * This is the list of file patterns to load into the browser during testing.
         */
        files: [
            'node_modules/grunt-mutation-testing/test/fixtures/karma-mocha/script*.js',
            'node_modules/grunt-mutation-testing/test/fixtures/karma-mocha/karma-test.js'
        ],

        frameworks: [ 'jasmine' ],
        plugins: [ 'karma-jasmine', 'karma-firefox-launcher', 'karma-chrome-launcher', 'karma-coffee-preprocessor', 'karma-coverage' ],


        // list of files to exclude
        exclude: [
            //'app/scripts/sidebar.js'
        ],


        // preprocess matching files before serving them to the browser
        // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
        preprocessors: {
        },


        // test results reporter to use
        // possible values: 'dots', 'progress'
        // available reporters: https://npmjs.org/browse/keyword/karma-reporter
        reporters: ['progress', 'coverage'],

        port: 9876,
        runnerPort: 9100,
        urlRoot: '/',

        /**
         * Disable file watching by default.
         */
        autoWatch: false,

        // enable / disable colors in the output (reporters and logs)
        colors: true,

        // level of logging
        // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
        logLevel: config.LOG_INFO,


        // start these browsers
        // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
        browsers: ['Firefox'],

        // Continuous Integration mode
        // if true, Karma captures browsers, runs the tests and exits
        singleRun: false


    }
)
;

}
;

Following is the output of the karma.txt file -

   Tests fail without mutations.

Let me know if anyone can help.

Neat plugin but unfortunately all I can get this to do is:

`Running "mutationTest:target" (mutationTest) task
Warning: Not all required options have been set properly Used --force, continuin
g.
Warning: Cannot read property 'testFramework' of null Used --force, continuing.

Done, but with warnings.`

I have mocha installed. Not sure why this is being thrown.

Gruntfile is configured like this:
mutationTest: { options: { code: 'test/setup.js', specs: 'test/component/test.js', mutate: 'src/scripts/testing/QuestionStoreTest.js', }, target: { code: ['src/scripts/stores/testing/*.js', 'src/mutations/*.js'], specs: 'test/setup.js', mutate: 'src/scripts/stores/testing/*.js' } }

After changing the mutationTest command to:
mutationTest: { mocha: { options: { code: 'test/setup.js', specs: 'test/component/test.js', mutate: 'src/scripts/testing/QuestionStoreTest.js', ignore: /^\s*log\(/, ignoreReplacement: [/^console$/], reporters: { text: { file: 'mocha.txt' } } } } },

The same issue persists.

Some more documentation, or an example project would be greatly appreciated!

Hi @jimivdw

I have recently set up mutation testing in my JS framework. The thing is it dosent error out but it does not move from the below point.

Running "mutationTest:all" (mutationTest) task
(08:46:43.934) INFO [KarmaServerManager]: Karma server started after 2647ms and is listening on port 12111

It just stays at this point.

I took al the config from the git example that you had provided.

    mutationTest: {
        options: {
            karma: {
                configFile: 'karma.conf.js'
            },
            reporters: {
                logLevel: 'INFO',
                html: {
                    dir: 'reports/mutation-test-html'
                },
                text: {
                    dir: 'reports/mutation-test-text'
                }
            }
        },
        all: {
            options: {
                code: ['src/**/*.js'],
                specs: 'test/**/*.js',
                mutate: ['src/**/*.js']
            }
        }
    }

If you could provide some advice it would be great.

Regards,
Sanu

When not a single code-specs mapping is found, the application crashes with a

Fatal error: Cannot call method 'stop' of undefined

Hi all,

I used to make 0.5.1 work last year and I wanted to use the mutation test again with the new version.
I succeeded in having the grunt task set up but it seems nothing is tested. Here is the task definition I used :

mutationTest: {
      options: {
        karma: {
          configFile: 'karma.conf.js',
          fileSpecs: 'test/spec/mapping.spec.json',
          waitForRunnerTime: 10
        },
        reporters: {
          html:{
            dir: 'reports/grunt-mutation-testing',
            successThreshold: 90
          },
          console: true
        }
      },
      all: {
        options: {
          code: [
            'main/common/**/*.js',
            'main/features/**/*.js',
            'test/spec/**/*.js'
          ],
          specs: '**/*.spec.js',
          mutate: 'main/common/services/feature-manager.js',
          logLevel: 'ALL'
        }
      }
    }

where the test/spec/mapping.spec.json looks like this :

{
  "main/common/services/i18n.js" : ["test/spec/common/services/i18n.spec.js"],
  "main/common/services/feature-manager.js" : ["test/spec/common/services/feature-manager.spec.js"]
}

The karma.conf.js is correct because when I run karma the 500 tests pass (it uses jasmine test framework by the way).

Can you give me hint about this?

I will update the Readme consequently.

Thx

After upgrading to Jasmine2, the framework is no longer able to automatically determine which unit test file is covered by which spec file.

This is due to a change in the coverage reports: they used to use absolute paths, but now use relative paths instead.

Hi

I can't get it working with karma, I receive following output:

$ grunt
Running "mutationTest:goals" (mutationTest) task
(13:38:25.033) DEBUG [KarmaServerManager]: 29 01 2016 13:38:25.033:INFO [karma]: Karma v0.13.15 server started at http://localhost:12111/

Fatal error: Could not connect to a Karma server on port 12111 within 30 seconds

(I tried with longer timeout, didn't help)
Here's my Gruntfile.js

"use strict";
/* global module */
module.exports = function (grunt) {
    grunt.initConfig({
        mutationTest: {
            options: {
                logLevel: "DEBUG",
                mutateProductionCode: true,
                configFile: "karma.conf.js",
                karma: {
                    waitForServerTime: 90
                }
            },
            goals: {
                code: "build/assets/js/**/*.js",
                specs: "build/assets/js/spec/subapps/goals/**/*.js",
                mutate: "build/assets/js/app/subapps/goals/goals.subapp.js"
            }
        }
    });

    grunt.loadNpmTasks("grunt-mutation-testing");
    grunt.registerTask("default", ["mutationTest"]);
};

Am I doing something wrong?
My karma.conf is ok, it works.

Hi,

Do you plan to add a support for the nodeunit?

Cheers,
Adam

As was suggested in #47, an example project would greatly help in getting the configuration to work properly.

It is not possible to configure a port or port range on which Karma should run. This creates issues on (shared) environments where some port ranges are unavailable.

Hi !
I've tried installing and running grunt-mutation-testing v1.2.0 but unfortunately it seems to always fail with the following error :

D:\Documents\Git\Jamstash> grunt mutationTest
Loading "mutation-testing-mocha.js" tasks...ERROR
>> Error: Cannot find module 'mocha'
Loading "mutation-testing.js" tasks...ERROR
>> Error: Cannot find module 'mocha'
Warning: Task "mutationTest" failed. Use --force to continue.

Aborted due to warnings.

Here is the dedicated part of my Gruntfile :

mutationTest: {
      options: {
        code: [
          'bower_components/jquery/dist/jquery.js'
          // More bower files, removed for brevity...
        ],
        specs: 'app/**/*_test.js',
        mutate: 'app/**/*.js',
        mutateProductionCode: true,
        reporters: {
          html: {
            dir: 'reports/grunt-mutation-testing'
          }
        },
        karma: {
          browsers: ['Chrome'],
          configFile: 'karma.conf.js',
          fileSpecs: 'code-test-mapping.json',
          waitForServerTime: 30
        },
        testFramework: 'karma'
      },
      test: {}
    },

I am running on grunt v0.4.5, on Windows 8.1. This config does work with grunt-mutation-testing (although it seems I have other problems, I'll probably need to open other issues).
Let me know if you need more information and many thanks for this plugin, I look forward to improving the quality of our tests with its help !