- What is Cucable
- Maven dependency
- Data flow
- Building
- Future improvements
- License
Cucable is a Maven plugin for Cucumber scenarios that simplifies running scenarios in parallel.
This plugin has two purposes:
- Generating single Cucumber features from .feature files
- Generating Cucumber runners for every generated feature file
Those generated runners and features can then be used with Maven Failsafe in order to parallelize test runs.
This plugin was inspired by the Cucumber Slices Maven Plugin.
<dependency>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>0.0.4</version>
</dependency>
The typical flow is
- Generation of runners and features
- Running them with Maven failsafe
- Aggregation of a single test report after all test runs
- Optional passing or failing of the build according to the test results
The following sections break down the above steps.
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable-plugin.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
<configuration>
<sourceRunnerTemplateFile>src/test/resources/parallel/cucable.template</sourceRunnerTemplateFile>
<sourceFeatureDirectory>src/test/resources/features</sourceFeatureDirectory>
<generatedFeatureDirectory>src/test/resources/parallel/features</generatedFeatureDirectory>
<generatedRunnerDirectory>src/test/java/parallel/runners</generatedRunnerDirectory>
</configuration>
</execution>
</executions>
</plugin>
The path to a text file (e.g. src/test/resources/parallel/cucable.template) with [FEATURE_FILE_NAME] placeholders for the generated feature file name. This file will be used to generate runners for every generated feature file.
Example:
package com.example;
import com.example.YourTestRunner;
import cucumber.api.CucumberOptions;
import org.junit.runner.RunWith;
@RunWith(YourTestRunner.class)
@CucumberOptions(
monochrome = false,
features = {"classpath:parallel/features/[FEATURE_FILE_NAME].feature"},
format = {"json:target/cucumber-report/[FEATURE_FILE_NAME].json"},
strict = false,
dryRun = false,
glue = {"com.example.glue"},
tags = {"~@ignore"}
)
public class [FEATURE_FILE_NAME] {
}
The path where your existing Cucumber .feature files are located (e.g. src/test/resources/features).
The path where the generated Cucumber .feature files should be located (e.g. src/test/resources/parallel).
Hint: This directory should be located under a valid resource folder to be included as a test source by Maven.
Caution: This directory will be wiped prior to the feature file generation!
The path where the generated runner classes should be located (e.g. src/test/java/parallel/runners).
Hint: This directory should be located under a valid source folder to be included as a test source by Maven.
Caution: This directory will be wiped prior to the runner file generation!
Below, you can see a full example of what Cucable does.
This is our source feature file. It contains a scenario and a scenario outline with two examples.
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
Scenario Outline: Second scenario
Given I am on the start page
And I add <amount> items
And I navigate to the shopping basket
Then I see <amount> items
Examples:
| amount |
| 12 |
| 85 |
This is the runner template file that is used to generate single scenario runners. The [FEATURE_FILE_NAME] placeholder will be automatically replaced with the name of each generated scenario.
It is possible to specify a custom runner using @RunWith(MyCustomCucumberRunner.class)
!
By specifying tags, we can also split the tests even further if needed. In this case we just ignore all tests that are annotated with ignore
.
The format option tells Cucumber where to put the json report files for the aggregated test report.
package parallel.runners; import com.trivago.trupi.runner.MyCustomCucumberRunner; import cucumber.api.CucumberOptions; import org.junit.runner.RunWith; @RunWith(MyCustomCucumberRunner.class) @CucumberOptions( monochrome = false, features = {"classpath:parallel/features/[FEATURE_FILE_NAME].feature"}, format = {"json:target/cucumber-report/[FEATURE_FILE_NAME].json"}, strict = false, dryRun = false, glue = {"com.trivago.glue"}, tags = {"~@ignore"} ) public class [FEATURE_FILE_NAME] { }
For each scenario, a single feature file is created:
Example_00001_IT.feature
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
Note that for the scenario outlines, each example is converted to its own scenario and feature file:
Example_00002_IT.feature
Feature: This is the feature name Scenario: Second scenario Given I am on the start page And I add 12 items And I navigate to the shopping basket Then I see 12 items
Example_00003_IT.feature
Feature: This is the feature name Scenario: Second scenario Given I am on the start page And I add 85 items And I navigate to the shopping basket Then I see 85 items
The generated runners point to each one of the generated feature files.
This is an example for one of the generated runners - note how the placeholders are now replaced with the name of the feature to run:
Example_00003_IT.java
package parallel.runners; import com.trivago.trupi.runner.MyCustomCucumberRunner; import cucumber.api.CucumberOptions; import org.junit.runner.RunWith; @RunWith(TrupiCucumberRunner.class) @CucumberOptions( monochrome = false, features = {"classpath:parallel/features/Example_00001_IT.feature"}, format = {"json:target/cucumber-report/Example_00001_IT.feature.json"}, strict = false, dryRun = false, glue = {"com.trivago.glue"}, tags = {"~@ignore"} ) public class Example_00001_IT { }
This will skip the unit tests (if any) and run the generated runner classes with Failsafe. Since all generated runner classes from the step before end with _IT, they are automatically considered integration tests and run with failsafe.
If all tests should be run regardless of their result, it is important to set <testFailureIgnore>true</testFailureIgnore>
for Failsafe - otherwise the plugin execution will stop on failing tests.
However, if this is specified, the build will not fail in case of failing tests! To circumvent that, it is possible to specify a custom Maven fail rule.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>Run parallel tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<forkCount>${maven.fork.count}</forkCount>
<reuseForks>false</reuseForks>
<argLine>-Dfile.encoding=UTF-8</argLine>
<disableXmlReport>true</disableXmlReport>
</configuration>
</plugin>
We use the Maven Cucumber Reporting library to aggregate all generated .json report files into one overall test report.
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<executions>
<execution>
<id>report</id>
<phase>post-integration-test</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber-report</cucumberOutput>
<projectName>My Report</projectName>
<buildNumber>1</buildNumber>
</configuration>
</execution>
</executions>
</plugin>
You can use a custom Maven fail rule that passes or fails the complete build based on test failures. It could check the Failsafe summary report that is generated for each test run. Without this rule we would have a successful build every time in case we specify.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<dependencies>
<dependency>
<groupId>com.trivago.rta</groupId>
<artifactId>your-custom-maven-fail-rule</artifactId>
<version>${fail.rule.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<phase>post-integration-test</phase>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<yourRule implementation="com.example.YourFailRule"/>
</rules>
<fail>true</fail>
</configuration>
</execution>
</executions>
</plugin>
This is the complete Maven profile that is used when invoking
mvn clean verify -Pparallel
So all specified plugins will execute one after the other.
<profiles>
<profile>
<id>parallel</id>
<build>
<plugins>
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable-plugin.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
<configuration>
<sourceRunnerTemplateFile>src/test/resources/parallel/cucable.template</sourceRunnerTemplateFile>
<sourceFeatureDirectory>src/test/resources/features</sourceFeatureDirectory>
<generatedFeatureDirectory>src/test/resources/parallel/features</generatedFeatureDirectory>
<generatedRunnerDirectory>src/test/java/parallel/runners</generatedRunnerDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>Run parallel tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<forkCount>${maven.fork.count}</forkCount>
<reuseForks>false</reuseForks>
<argLine>-Dfile.encoding=UTF-8</argLine>
<disableXmlReport>true</disableXmlReport>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<executions>
<execution>
<id>report</id>
<phase>post-integration-test</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber-report</cucumberOutput>
<projectName>My Report</projectName>
<buildNumber>1</buildNumber>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<dependencies>
<dependency>
<groupId>com.trivago.rta</groupId>
<artifactId>your-custom-maven-fail-rule</artifactId>
<version>${fail.rule.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<phase>post-integration-test</phase>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<yourRule implementation="com.example.YourFailRule"/>
</rules>
<fail>true</fail>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<pluginRepositories>
<pluginRepository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>parallel</id>
<name>libs-release</name>
<url>http://artifactory.rta.trivago.trv:8081/artifactory/libs-release</url>
</pluginRepository>
</pluginRepositories>
</profile>
Cucable requires Java 8 and it uses Maven for its dependencies.
- Offer the possibility to generate runners and features directly in the target folder
- Support running specific scenarios in parallel
- Support running the whole test suite multiple times
- Support running single tests multiple times
Copyright 2017 trivago GmbH
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.