Hermione is the utility for integration testing of web pages using WebdriverIO and Mocha.

• Why should you choose hermione?
  • Easy to use
  • Parallel test running
  • Extensibility
  • Retries for failed tests
  • Executing a separate test
  • Skip tests in specific browsers
  • Flexible tests configuration
  • Auto initialization and closing grid sessions
• Prerequisites
• Skip
• Only
• WebdriverIO extensions
  • Sharable meta info
  • Execution context
• Quick start
• .hermione.conf.js
  • specs
  • browsers
  • gridUrl
  • baseUrl
  • httpTimeout
  • waitTimeout
  • sessionsPerBrowser
  • retry
  • system
    • debug
    • mochaOpts
  • plugins
  • prepareBrowser
  • prepareEnvironment
• CLI
• Overriding settings
• Programmatic API
• Environment variables
  • HERMIONE_SKIP_BROWSERS

Hermione provides several features which WebdriverIO doesn't and makes testing process easier.

If you are familiar with WebdriverIO and Mocha, you can start writing and running tests in 5 minutes! You need to install hermione via npm and add a tiny config to your project. See details in Quick start section.

When tests are run one by one, it takes a lot of time. Hermione can run tests in parallel sessions in different browsers out of the box.

WebdriverIO provides built-in commands for browser and page manipulation. Often projects need to store some common code and reuse it through all tests. So a developer should create some helpers and include them in the tests.

With hermione it's very simple and straightforward. You can add any number of custom commands in the hermione config and use them as this.browser.myCustomCommand in tests.

Moreover, hermione provides plugins which work as some kind of a hooks. They allow a developer to prepare environment for tests and react properly to test execution events.

Integration tests use a dynamic environment with a lot of dependencies where any of them can work unstable from time to time. As a result, integration tests become red randomly and make them undetermined. It spoils all testing process.

To prevent incidental fails hermione retries a failed test before marking it as a failed. It makes it possible to get rid of a majority of incidental fails. Number of retries can be specified for all browsers or for a specific browser.

⚠️ Hermione reruns tests in a new browser session to exclude situations when the browser environment is a cause of the fail.

Sometimes it is needed to run only specific tests but not all tests in a set. Hermione makes it possible. You can specify path to the test file

hermione tests/func/mytest.js

or filter describes by using --grep option

hermione --grep login

or simply use mocha only() API in the test

describe.only('user login', function() {...});

Sometimes you need to skip test not in all browsers but in a specific one. For example, you don't need to run some test in ugly IE browsers. In hermione you can do it with hermione helper. For example, you can skip some tests in the specific browser

describe('feature', function() {
    hermione.skip.in('ie8', 'it can not work in this browser');
    it('nowaday functionality', function() {...});
});

or run tests in one of the browsers

describe('feature', function() {
    // will be skipped in all browsers except chrome
    hermione.skip.notIn('chrome', 'it should work only in Chrome');
    it('specific functionality', function() {...});
});

In these cases you will see messages with a skip reason in reports.

To skip suite or test silently (without any messages in reports), you can pass the third argument with silent flag:

hermione.skip.in('ie8', 'skipReason', {silent: true});
// or
hermione.skip.notIn('chrome', 'skipReason', {silent: true});

Or you can use another hermione helper - only, which is silent by default:

hermione.only.in('chrome');

It will run tests only in one browser and skip rest silently.

Hermione has possibility to configure running some set of tests in specific browsers. For example,

specs: [
    'tests/common', // run common tests in all browsers specified in the config
    {
        files: 'tests/desktop',
        browsers: ['ie8', 'opera']
    },
    {
        files: 'tests/touch',
        browsers: ['iphone', 'android']
    }
]

See specs for more details.

All work with a grid client is incapsulated in hermione. Forget about client.init and client.end in your tests ;)

Because of hermione is based on WebdriverIO you need to set up Selenium before proceed further.

The simplest way to get started is to use one of the NPM selenium standalone packages like: vvo/selenium-standalone. After installing it (globally) you can run your server by executing:

selenium-standalone start

This feature allows you to ignore the specified suite or test in any browser with additional comment. You can do it by using global hermione.skip helper. It supports the following methods:

• .in — adds matchers for browsers with additional comment;
• .notIn — .in method with reverted value;

Each of these methods takes following arguments:

• browser {String|RegExp|Array<String|RegExp>} — matcher for browser(s) to skip;
• [comment] {String} — comment for skipped test;
• [options] {Object} - additional options;

Note that matchers will be compared with browserId specified in a config file, e.g. chrome-desktop.

For example,

describe('feature', function() {
    hermione.skip.in('chrome', "It shouldn't work this way in Chrome");
    it('should work this way', function() {
        return runTestThisWay();
    });

    it('should work that way', function() {
        return runTestThatWay();
    });

    hermione.skip.in(['chrome', 'firefox', /ie\d*/], 'Unstable test, see ticket TEST-487');
    it('should done some tricky things', function() {
        return runTrickyTest();
    });
});

in this case behaviour it should work this way will be skipped only in chrome browser, but will be run in other browsers. It should work that way will not be ignored. So skip will be applied only to the nearest test. If you need to skip all tests within a suite you can apply skip helper to a describe - all tests within this suite will be skipped with the same comment.

hermione.skip.in('chrome', 'skip comment');
describe('some feature', function() {
    it(...);
    it(...);
});

Also you can use .notIn method to invert matching. For example,

// ...
hermione.skip.notIn('chrome', 'some comment');
it('should work this way', function() {
    return doSomething();
});
// ...

in this case test will be skipped in all browsers except chrome.

All of these methods are chainable. So you can skip test in several browsers with different comments. For example,

// ...
hermione.skip
    .in('chrome', 'some comment')
    .notIn('ie9', 'another comment');
it('test1', function() {
    return doSomething();
});
// ...

If you need to skip test in all browsers without a comment you can use mocha .skip method instead of hermione.skip.in(/.*/);. The result will be the same.

This feature allows you to ignore the specified suite or test in any browser silently (without any messages in reports). You can do it by using global hermione.only helper. It supports only one method:

• .in — hermione.skip.notIn method with silent flag

This method takes following arguments:

• browser {String|RegExp|Array<String|RegExp>} — matcher for browser(s) to skip;

For example:

// ...
hermione.only.in('chrome');
it('should work this way', function() {
    return doSomething();
});
// ...

in this case test will be skipped in all browsers silently except chrome.

Hermione adds some usefull methods and properties to the webdriverio session after its initialization.

Implemented via two commands:

• setMeta(key, value)
• getMeta(key)

These methods allow to store some information between webdriver calls so it can be used in custom commands for example. This meta information will be shown in allure report.

Note: hermione saves in meta info last url opened in browser.

Example:

it('test1', function() {
    return this.browser
        .setMeta('foo', 'bar')
        .url('/foo/bar?baz=qux')
        .getMeta('foo')
        .then((val) => console.log(val)) // prints 'bar'
        .getMeta('url')
        .then((url) => console.log(url)); // prints '/foo/bar?baz=qux'
});

Execution context can be accessed by browser.executionContext property which contains current test/hook mocha object extended with browser id.

Example:

it('some test', function() {
    return this.browser
        .url('/foo/bar')
        .then(function() {
            console.log('test', this.executionContext);
        });
});

will print something like this

test: {
  "title": "some test",
  "async": 0,
  "sync": true,
  "timedOut": false,
  "pending": false,
  "type": "test",
  "body": "...",
  "file": "/foo/bar/baz/qux.js",
  "parent": "#<Suite>",
  "ctx": "#<Context>",
  "browserId": "chrome",
  "meta": {},
  "timer": {}
}

First of all, make sure that all prerequisites are satisfied.

Install the package.

npm install -g hermione

Then put .hermione.conf.js in the project root.

module.exports = {
    specs: ['tests/func'],

    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: 'chrome'
            }
        }
    }
};

Write your first test.

var assert = require('chai').assert;

describe('yandex', function() {
    it('should find itself', function() {
        return this.browser
            .url('https://yandex.com')
            .setValue('.search2__input input', 'yandex')
            .click('.search2__button button')
            .getText('.z-entity-card__title')
            .then(function(title) {
                assert.equal(title, 'Yandex')
            });
    });
});

Finally, run tests.

hermione

hermione is tuned using a configuration file. By default .hermione.conf.js is used but a path to the configuration file can be specified using --conf option.

There are only two required fields: specs and browsers.

module.exports = {
    specs: [
        'tests/desktop',
        'tests/touch'
    ],
    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: 'chrome'
            }
        }
    }
};

Required. The list of paths where hermione will look for tests.

For example,

specs: [
    {                          // run tests associated with this path in all browsers
        files: 'tests/desktop' // which are configured in option `browsers`
    },
    'tests/deskpad',           // the alias for the previous case
    {
        files: 'tests/desktop/*.hermione.js' // run tests matched with a mask
    },
    {
        files: 'tests/touch',  // run tests associated with this path in a browser with id `browser`
        browsers: ['browser']  // which is configured in option `browsers`
    }
]

Required. The list of browsers which should be used for running tests.

Browser section has the following format

browsers: {
    <browser_id>: {
        <option>:<value>
        <option>:<value>
    }
}

<browser-id> values is used for browser identification.

Available browser options:

Selenium grid URL. Default value is http://localhost:4444/wd/hub.

Base service-under-test url. Default value is http://localhost.

Timeout for any requests to Selenium server. Default value is 90000 ms.

Timeout for web page events. Default value is 1000 ms.

Number of sessions which are run simultaneously. Global value for all browsers. Default value is 1.

How many times test should be retried in case of a fail. Global value for all browsers. Default value is 0.

Turn webdriver debug mode on. Default value is false.

Extra options for mocha which are passed to mocha.setup. See Mocha documentation for the list of options. Default values are:

mochaOpts: {
    slow: 10000, // If test execution time is greater than this value, then test is slow.
    timeout: 60000 // timeout for test execution.
}

Hermione plugins are commonly used to extend built-in possibilities. For example, hermione-allure-reporter and hermione-tunnel.

Plugin is a module which exports a single function. The function has two arguments:

• hermione instance;
• plugin options from configuration file.

Plugins will be loaded before hermione runs tests.

It's strongly recommended to name hermione plugins with hermione- prefix. It makes search for user plugins very simple.

If a plugin name starts with hermione-, then the prefix can be ommited in the configuration file. If two modules with names hermione-some-module and some-module are specified, then module with prefix will have higher priority.

For example.

// .hermione.conf.js
// ...
plugins: {
    'my-cool-plugin': {
        param: 'value'
    }
}
// ...

// hermione-my-cool-plugin/index.js
module.exports = function(hermione, opts) {
    hermione.on(hermione.events.RUNNER_START, function(runner) {
        return setUp(hermione.config, opts.param); // config can be mutated
    });

    hermione.on(hermione.events.RUNNER_END, function() {
        return tearDown();
    });
}

Properties of hermione object

Available events

Prepare browser session before tests are run. For example, adding custom user commands.

prepareBrowser: function(browser) {
    // do setup here
}

browser argument is a WebdriverIO session.

Configuration data can be changed depending on extra conditions in prepareEnvironment function.

  Usage: hermione [options]

  Options:

    -h, --help                 Output usage information
    -c, --config <path>        Path to configuration file
    -r, --reporter <reporter>  Test reporter
    -b, --browser <browser>    Run tests only in specified browser
    --grep <grep>              Run only tests matching string or regexp

For example,

hermione --config ./config.js --reporter flat --browser firefox --grep name

Note. All CLI options override config values.

All options can also be overridden via command-line flags or environment variables. Priorities are the following:

• command-line option has the highest priority. It overrides environment variable and config file value.

• environment variable has second priority. It overrides config file value.

• config file value has the lowest priority.

• if no command-line option, environment variable or config file option specified, default is used.

To override config setting with CLI option, convert full option path to --kebab-case. For example, if you want to run tests against different base URL, call:

hermione path/to/mytest.js --base-url http://example.com

To change number of sessions for Firefox (considering you have browser with firefox id in the config):

hermione path/to/mytest.js --browsers-firefox-sessions-per-browser 7

To override setting with environment variable, convert its full path to snake_case and add hermione_ prefix. Above examples can be rewritten to use environment variables instead of CLI options:

hermione_base_url=http://example.com hermione path/to/mytest.js
hermione_browsers_firefox_sessions_per_browser=7 hermione path/to/mytest.js

With the help of API you can use Hermione programmatically in your scripts or build tools.

const Hermione = require('hermione');

const hermione = new Hermione(config, allowOverrides);

hermione.run(testPaths, options)
    .then((success) => process.exit(success ? 0 : 1))
    .catch((e) => {
        console.log(e.stack);
        process.exit(1);
    })
    .done();

• config (required) String|Object - path to configuration file which will be read relatively to process.cwd or configuration object.
• allowOverrides (optional) Object - switch on/off configuration override via environment variables or cli options:
  • env (optional) Boolean – switch on/off configuration override via environment variables. Default is false
  • cli (optional) Boolean - switch on/off configuration override via cli options. Default is false
• testPaths (optional) String[] - paths to tests relatively to process.cwd
• options (optional) Object
  • reporters (optional) String[] - test result reporters
  • browsers (optional) String[] - browsers in which to run tests
  • grep (optional) RegExp - pattern which indicates which tests to run

Skip browsers specified in a config by passing browser ids. Several browser ids should be split by commas (spaces after commas are allowed).

For example,

HERMIONE_SKIP_BROWSERS=ie10,ie11 hermione