Library for sending test run reports to your testomat.io project.

Get the {API_KEY} from testomat.

You can refer sample tests from example folder of this repo. This is a basic example. If you need something full fledged you can refer this example repo.

Add @testomatio/reporter package to your project:

npm i @testomatio/reporter --save

For testcafe use testcafe reporter:

npm i testcafe-reporter-testomatio

For newman use:

npm i newman-reporter-testomatio --save-dev

newman and newman-reporter-testomatio should be installed in the same directory.
If you run your tests using globally installed newman (newman run ...), intall newman-reporter-testomatio globally too (npm i newman-reporter-testomatio -g).
If you use locally installed newman (within the project) (npx newman run ...), install newman-reporter-testomatio locally (npm i newman-reporter-testomatio). You can verify installed packages via npm list or npm list -g.

Make sure you load all your tests using check-tests.

Add plugin to codecept conf:

plugins: {
  testomatio: {
    enabled: true,
    require: '@testomatio/reporter/lib/adapter/codecept',
    apiKey: process.env.TESTOMATIO || 'API_KEY', // pass in api key via config or env variable
  }
}

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx codeceptjs run

If tests run parallel, like workers in CodeceptJS use start-test-run command to get proper reports:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx codeceptjs run-workers 2'

Specify a command to run with -c option in start-test-run

Use --env-file <envfile> option to load environment variables from .env file. Inside env file TESTOMATIO credentials like TESTOMATIO api key or S3 config can be stored.

Add a reporter to Playwright config:

reporter: [
  ['list'],
  [
    '@testomatio/reporter/lib/adapter/playwright.js',
    {
      apiKey: process.env.TESTOMATIO,
    },
  ],
];

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx playwright test

Load the test using using check-tests if not done already. Get the test id from testomat account and add it to your mocha test like in this example.

Run the following command from you project folder:

mocha --reporter ./node_modules/testomat-reporter/lib/adapter/mocha.js --reporter-options apiKey={API_KEY}

Load the test using using check-tests. Add the test id to your tests like in this example.

Add the following line to jest.config.js:

reporters: ['default', ['@testomatio/reporter/lib/adapter/jest.js', { apiKey: process.env.TESTOMATIO }]],

Run your tests.

Load you test using check-cucumber.

Run the following command from you project folder:

TESTOMATIO={API_KEY} ./node_modules/.bin/cucumber-js --format ./node_modules/@testomatio/reporter/lib/adapter/cucumber.js

Load the test using using check-tests.

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx testcafe chrome -r testomatio

Run collection and specify testomatio as reporter:

TESTOMATIO={API_KEY} npx newman run {collection_name.json} -r testomatio

check-tests not supported for newman for now, tests will be created on testomatio by default

Load the test using using check-tests.

Register our cypress-plugin in cypress/plugins/index.js:

const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress-plugin');

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  testomatioReporter(on, config);

  return config;
};

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx cypress run

Load the test using using check-tests.

Add the following lines to conf.js:

const JasmineReporter = require('@testomatio/reporter/lib/adapter/jasmine');

exports.config = {
  onPrepare: () => {
    jasmine.getEnv().addReporter(new JasmineReporter({ apiKey: process.env.TESTOMATIO }));
  },
};

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx protractor conf.js'

Load the test using using check-tests.

Add the following lines to wdio.conf.js:

const testomatio = require('@testomatio/reporter/lib/adapter/webdriver');

exports.config = {
  // ...
  reporters: [
    [testomatio, {
      apiKey: $ {
        process.env.TESTOMATIO
      }
    }]
  ]
}

For making screenshots on failed tests add the following hook to wdio.conf.js:

    afterTest: function (test, context, { error, result, duration, passed, retries }) {
        if (error) {
            browser.takeScreenshot()
        }
    },

Run the following command from you project folder:

TESTOMATIO={API_KEY} npx start-test-run -c 'npx wdio wdio.conf.js'

Pipes allow you to get report inside different systems (e.g. Pull request comment, database etc) For now next pipes available:

This pipe adds comment with run report to GitHub Pull Request.

To use it:

1. run your tests using github actions in Pull Request
2. pass GH_PAT (GitHub Personal Access Token) as environment variable.

Last report (comment) will be replaced with the new one. To leave previous report pass GITHUB_KEEP_OUTDATED_REPORTS=1 env variable.

This pipe adds comment with run report to GitLab Merge Request.

To use it:

1. run your tests in Merge Request (pipeline trigger should be merge_request)
2. pass GITLAB_PAT (GitLab Personal Access Token) as environment variable.

Last report (comment) will be replaced with the new one. To leave previous report pass GITLAB_KEEP_OUTDATED_REPORTS=1 env variable.

Notice JUnit reports are supported since 0.6.0

Other frameworks and languages are supported via JUnit reports.

JUnit XML format is standard among test runners on various platforms. Testomat.io can load XML reports from test runners and create tests in a project from it. If your framework is not supported yet, generate JUnit report and upload it into Testomat.io

Testomat.io will not only create a run report but also collect all information about tests, including source code, and create tests in a system as regular importer do. The only difference that normal process of Testomat.io import does not require executing tests on import, while importing via repoter requires to have tests executed and XML report provided.

Tested Frameworks:

• JUnit (JUnit)
• Python (Pytest)
• Minitest (Ruby)
• PHPUnit (PHP)
• NUnit (C#)

To import JUnit reports into Testomat.io NodeJS >=14 is required.

Package @testomatio/reporter should be installed:

npm i @testomatio/reporter@latest --save-dev

Run your test framework and generate a JUnit report.

Then import XML report into Testomat.io

TESTOMATIO={API_KEY} npx report-xml "{pattern}" --lang={lang}

• pattern - is a glob pattern to match all XML files from report. For instance, "test/report/**.xml" or just report.xml
• --lang option can be specified to identify source code of the project. Example: --lang=Ruby or --lang=Java or --lang=Python. Possible values:
  • c#
  • java
  • ruby
  • python
  • php
• --java-tests option is avaiable for Java projects, and can be set if path to tests is different then src/test/java. When this option is enable, lang option is automatically set to java
• --env-file <envfile> option to load environment variables from .env file. Inside env file TESTOMATIO credentials like TESTOMATIO api key or S3 config can be stored.
• --timelimit <time> set a timer to silently kill a long-running reporter process due to network or other issues. For instance, use --set-timeout=3 to stop process after 3 secs.

Notice: All options from Advanced Usage are also available for JUnit reporter

Check the list of examples:

Run pytest tests and generate a report to report.xml:

pytest --junit-xml report.xml

Import report with this command

TESTOMATIO={API_KEY} npx report-xml report.xml --lang=python

Run tests via Maven, make sure JUnit report was configured in pom.xml.

mvn clean test

Import report with this command:

TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/*.xml" --java-tests

You can specify --java-test option to set a path to tests if they are located in path other than src/test/java

Generate NUnit XML report and run the following code:

TESTOMATIO={API_KEY} npx report-xml "report.xml" --lang="c#"

# test_helper.rb:

reporters = [Minitest::Reporters::DefaultReporter.new(color: true)]
# enable JUnit reporter
reporters << Minitest::Reporters::JUnitReporter.new

Minitest::Reporters.use! reporters

Launch tests:

rails test

Import reports from test/reports directory:

TESTOMATIO={API_KEY} npx report-xml "test/reports/*.xml" --lang ruby

Screenshots or videos from tests are uploaded if a test contains output with a path to file of following format:

file://path/to/screenshot.png

For instance, inside Java test you can use System.out.println to print a path to file that should be uploaded as a screenshot.

System.out.println("file://" + pathToScreenshot);

This will produce XML report which contains path to a file:

<testcase>
  <system-out><![CDATA[
    file://path/to/scrrenshot.png
  ]]></system-out>
</testcase>

When uploaded XML report, all files from file:// will be uploaded to corresponding tests.

Testomat.io will not create tests from the report if they have not been previously imported. To create tests during the report TESTOMATIO_CREATE option can be used:

TESTOMATIO={API_KEY} TESTOMATIO_CREATE=1 <actual run command>

This feature is widely used when a run is executed on CI. A run is created before the test is started and it is marked as scheduled. Then a report is assigned to that run using TESTOMATIO_RUN environment variable and {RUN_ID} of a run:

TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} <actual run command>

If multiple reports are added to the same run, each of them should not finalize the run. In this case use TESTOMATIO_PROCEED=1 environment variable, so the Run will be shown as Running

TESTOMATIO={API_KEY} TESTOMATIO_PROCEED=1 TESTOMATIO_RUN={RUN_ID} <actual run command>

After all reports were attached and run can be execute the following command:

TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx start-test-run --finish

Give a title to your reports by passing it as environment variable to TESTOMATIO_TITLE.

TESTOMATIO={API_KEY} TESTOMATIO_TITLE="title for the report" <actual run command>

Provide a shared unique title to all runs that will be running in parallel, and add TESTOMATIO_SHARED_RUN environment var. So all reports will be sent to this run.

TESTOMATIO={API_KEY} TESTOMATIO_TITLE="report for commit ${GIT_COMMIT}" TESTOMATIO_SHARED_RUN=1 <actual run command>

We recommend using a commit hash as a title to create a new Run. In this case we ensure that run title is unique and will be the same for all parallel jobs running exactly for this commit.

Create/Add run to group by providing TESTOMATIO_RUNGROUP_TITLE:

TESTOMATIO={API_KEY} TESTOMATIO_RUNGROUP_TITLE="Build ${BUILD_ID}" <actual run command>

Add environments to run by providing TESTOMATIO_ENV as comma seperated values:

TESTOMATIO={API_KEY} TESTOMATIO_ENV="Windows, Chrome" <actual run command>

Add an env to run by specifying the TESTOMATIO_CSV_FILENAME variable.

1. using default report name:

TESTOMATIO={API_KEY} TESTOMATIO_CSV_FILENAME="report.csv" <actual run command>

2. using unique report name:

TESTOMATIO={API_KEY} TESTOMATIO_CSV_FILENAME="test.csv" <actual run command>

It's create a new /export folder with csv files

To save a test artifacts (screenshots and videos) of a failed test use S3 storage. Please note, that the storage is not connected to Testomatio. This allows you to store your artifacts on your own account and not expose S3 credentials.

To save screenshots provide a configuration for S3 bucket via environment variables:

• S3_REGION - Region your bucket lies.
• S3_BUCKET - Bucket name.
• S3_ACCESS_KEY_ID - Access key.
• S3_SECRET_ACCESS_KEY - Secret.
• S3_ENDPOINT - for providers other than AWS

By default tests artifacts are uploaded to bucket with public-read permission. In this case uploaded files will be publicly accessible in Internet. These public links will be used by testomat.io to display images and videos.

To upload files with private access bucket add TESTOMATIO_PRIVATE_ARTIFACTS=1 environment value. Then update provide the same S3 credentials in "Settings > Artifacts" section of a testomat.io project, so testomat.io could connect to the same bucket and fetch uploaded artifacts. Links to files will be pre-signed and expires automatically in 10 minutes.

Example upload configuration in environment variables:

TESTOMATIO_PRIVATE_ARTIFACTS=1
S3_ACCESS_KEY_ID=11111111111111111111
S3_SECRET_ACCESS_KEY=2222222222222222222222222222222222222222222
S3_BUCKET=artifacts
S3_REGION=us-west-1

TESTOMATIO_PRIVATE_ARTIFACTS=1
S3_ENDPOINT=https://ams3.digitaloceanspaces.com
S3_ACCESS_KEY_ID=11111111111111111111
S3_SECRET_ACCESS_KEY=2222222222222222222222222222222222222222222
S3_BUCKET=artifacts
S3_REGION=ams3

S3_ENDPOINT=http://company.storage.com
S3_ACCESS_KEY_ID=minio
S3_SECRET_ACCESS_KEY=minio123
S3_BUCKET=testomatio
S3_FORCE_PATH_STYLE=true

It is important to add S3_FORCE_PATH_STYLE var for minio setup

For local testing, it is recommended to store configuration in .env file. If you load configuration from a test runner, use dotenv library to load it.

If you run a reporter via start-test-run or report-xml command use --env-file option to load variables from .env file:

npx start-test-run --env-file .env
npx report-xml --env-file .env

On CI set environment variables in CI config.

Test artifacts are automatically uploaded for these test runners:

• CodeceptJS
• Playwright
• Cypress
• WebdriverIO

To manually attach an artifact and upload it for a test use global.testomatioArtifacts array:

// attach a picture inside a test
global.testomatioArtifacts.push('img/file.png');
// attach a picture and add a name to it
global.testomatioArtifacts.push({ name: 'Screenshot', path: 'img/file.png' });

Artifacts will be uploaded for the current test when it is finished.

To disable uploading artifacts add TESTOMATIO_DISABLE_ARTIFACTS environment variable:

TESTOMATIO_DISABLE_ARTIFACTS=1

If you want to create a run and obtain its {RUN_ID} from testomat.io you can use --launch option:

TESTOMATIO={API_KEY} npx start-test-run --launch

This command will return {RUN_ID} which you can pass to other testrunner processes.

When executed with --launch a command provided by -c flag is ignored

If you want to finish a run started by --launch use --finish option. TESTOMATIO_RUN environment variable is required:

TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx start-test-run --finish

Pass DEBUG variable with module name e.g. DEBUG=@testomatio/reporter:pipe:github. (Module name could be taken directly from the required module code). To log all debug info pass DEBUG=*.