myusernamejeep / grunt-saucelabs Goto Github PK

grunt-saucelabs

A Grunt task for running QUnit, Jasmine, Mocha and YUI tests using Sauce Labs' Cloudified Browsers.

Grunt is a task-based command line build tool for JavaScript projects, based on nodejs. QUnit is a powerful, easy-to-use JavaScript unit test suite used by the jQuery, jQuery UI and jQuery Mobile projects and is capable of testing any generic JavaScript code, including itself! Mocha is a JavaScript test framework for running serial asynchronous tests. YUI Test is a browser-based testing framework from Yahoo!. Sauce Labs offers browser environments on the cloud for testing code.

About the tool

The grunt-contrib-qunit task runs qunit based test suites on PhantomJS. The saucelabs-qunit task is very similar but runs the test suites on the cloudified browser environment provided by Sauce Labs. This ensures that subject of the test runs across different browser environment. The task also uses Sauce Connect to establish a tunnel between Sauce Labs browsers and the machine running Grunt to load local pages. This is typically useful for testing pages on localhost that are not publically accessible on the internet. The saucelabs-jasmine runs Jasmine tests in the Sauce Labs browser. The saucelabs-jasmine task requires jasmine-1.3.0. There are also saucelabs-mocha and saucelabs-yui tasks that let you run your Mocha and YUI tests on Sauce Labs cloudified browser environment.

Usage

This task is available as a node package and can be installed as npm install grunt-saucelabs. It can also be included as a devDependency in package.json in your node project.

To use the task in grunt.js, load the npmTask.

grunt.loadNpmTasks('grunt-saucelabs');

In the grunt.initConfig, add the configuration that looks like the following

'saucelabs-qunit': {
  all: {
      options: {
      username: 'saucelabs-user-name', // if not provided it'll default to ENV SAUCE_USERNAME (if applicable)
      key: 'saucelabs-key', // if not provided it'll default to ENV SAUCE_ACCESS_KEY (if applicable)
      urls: ['array or URLs to to load for QUnit'],
      concurrency: 'Number of concurrent browsers to test against. Will default to the number of overall browsers specified. Check your plan (free: 2, OSS: 3) and make sure you have got sufficient Sauce Labs concurrency.',
      tunneled: 'true (default) / false; false if you choose to skip creating a Sauce connect tunnel.',
      tunnelTimeout: 'A numeric value indicating the time to wait before closing all tunnels',
      testTimeout: 'Milliseconds to wait before timeout for qunit test per page',
      testInterval: 'Milliseconds between retries to check if the tests are completed',
      testReadyTimeout: 'Milliseconds to wait until the test-page is ready to be read',
      detailedError: 'false (default) / true; if true log detailed test results when a test error occurs',
      testname: 'Name of the test',
      tags: ['Array of tags'],
      browsers: [{
        browserName: 'opera'
      }],
      onTestComplete: function(){
        // Called after a qunit unit is done, per page, per browser
        // Return true or false, passes or fails the test
        // Returning undefined does not alter the test result

        // For async return, call
        var done = this.async();
        setTimeout(function(){
          // Return to this test after 1000 milliseconds
          done(/*true or false changes the test result, undefined does not alter the result*/);
        }, 1000);
      }
    }
  }
}

The configuration of saucelabs-jasmine, saucelabs-mocha, saucelabs-yui are exactly the same. Note the options object inside a grunt target. This was introduced in grunt-saucelabs-* version 4.0.0 to be compatiable with [email protected]

The parameters are

• username : The Sauce Labs username that will be used to connect to the servers. Required
• key : The Sauce Labs secret key. Since this is a secret, this should not be checked into the source code and may be available as an environment variable. Grunt can access this using process.env.saucekey. Required
• urls: An array or URLs that will be loaded in the browsers, one after another. Since SauceConnect is used, these URLs can also be localhost URLs that are available using the server task from grunt. Required
• tunneled: Defaults to true; Won't launch a Sauce Connect tunnel if set to false. Optional
• testname: The name of this test, displayed on the Sauce Labs dashboard. Optional
• tags: An array of tags displayed for this test on the Sauce Labs dashboard. This can be the build number, commit number, etc, that can be obtained from grunt. Optional
• browsers: An array of objects representing the various browsers on which this test should run. Optional
• tunnelTimeout: A numeric value indicating the time to wait before closing all tunnels (default: 120). Optional
• testTimeout : Number of milliseconds to wait for qunit tests on each page before timeout and failing the test (default: 300000). Optional
• testInterval : Number of milliseconds between each retry to see if a test is completed or not (default: 5000). Optional
• testReadyTimeout: Number of milliseconds to wait until the test-page is ready to be read (default: 5000). Optional
• onTestComplete : A callback that is called everytime a qunit test for a page is complete. Runs per page, per browser configuration. A true or false return value passes or fails the test, undefined return value does not alter the result of the test. For async results, call this.async() in the function. The return of this.async() is a function that should be called once the async action is completed. Optional

A typical test task running from Grunt could look like grunt.registerTask('test', ['server', 'qunit', 'saucelabs-qunit']); This starts a server and then runs the Qunit tests first on PhantomJS and then using the Sauce Labs browsers.

Test results details with Jasmine

You can make Job Details pages more infromative on Sauce by providing more data with each test. You will get info about each test run inside your suite directly on Sauce pages.

You can do that by using Jasmine JS Reporter that will let saucelabs-jasmine task provide in-depth data about each test as a JSON object.

All you need to do is to include the new jasmine-jsreporter reporter to the page running Jasmine tests by adding new script in header:

<script src="path/to/jasmine-jsreporter.js" type="text/javascript"></script>

and telling Jasmine to use it:

jasmineEnv.addReporter(new jasmine.JSReporter());

Examples

Some projects that use this task are as follows. You can take a look at their GruntFile.js for sample code

• This project
• Jquery-IndexedDB
• IndexedDBShim

If you have a project that uses this plugin, please add it to this list and send a pull request.

Integration with a CI system

Grunt tasks are usually run alongside a continuous integration system. For example, when using Travis, adding the following lines in the package.json ensures that the task is installed with npm install is run. Registering Sauce Labs in test task using grunt.registerTask('test', ['server', 'saucelabs-qunit']); ensures that the CI environment runs the tests using npm test. To secure the Sauce Key, the CI environment can be configured to provide the key as an environment variable instead of specifying it file. CI Environments like Travis provide ways to add secure variables in the initial configuration. The IndexedDBShim is a project that uses this plugin in a CI environment. Look at the .travis.yml and the grunt.js for usage example.