Git Product home page Git Product logo

fxtest-jenkins-pipeline's Introduction

Shared libraries for Jenkins Pipeline

This repository holds shared libraries for Jenkins pipelines used by Firefox Test Engineering.

Build Status

Pipeline Steps

ircNotification

Sends a notification to IRC with the specified channel, nick, and server. By default it will connect to irc.mozilla.org:6697 as fxtest and join the #fx-test-alerts channel.

Examples

// use defaults
ircNotification()

// specify a channel
ircNotification('#fx-test-alerts')

// specify all values
ircNotification(
  channel: '#fx-test-alerts',
  nick: 'fxtest',
  server: 'irc.mozilla.org:6697'
)

publishToPulse

Publishes a message, to Pulse with the specified exchange and routingKey. If a schema is provided then it will be used to check that the message is valid. If the message fails to pass validation, details will be output to the console log and ProcessingException will be thrown.

Requirements

Examples

// configure environment variables from credentials
environment {
  PULSE = credentials('PULSE')
}

// send message without schema validation
publishToPulse(
  exchange: "exchange/${PULSE_USR}",
  routingKey: "${PULSE_USR}.foo",
  message: 'foo'
)

// send message with schema validation from resources
schema = libraryResource 'org/mozilla/fxtest/pulse/schemas/treeherder.json'
publishToPulse(
  exchange: "exchange/${PULSE_USR}",
  routingKey: "${PULSE_USR}.foo",
  message: 'foo',
  schema: schema
)

publishToS3

Publishes the files at path to the specified Amazon S3 bucket and region. Defaults to region us-east-1.

Requirements

  • Amazon S3 bucket with appropriate permissions.
  • S3 Plugin with profile configured in Jenkins.

Examples

// single file with default bucket and region
publishToS3('results.html')

// multiple files with specified bucket and region
publishToS3(
  path: 'results/*',
  bucket: 'foo',
  region: 'bar'
)

submitToActiveData

Publishes the structured log(s) at logPath to ActiveData.

Requirements

See publishToS3.

Examples

submitToActiveData('results/raw.txt')

submitToTreeherder

Submits the build result for project to Treeherder using the specified jobSymbol and jobName. If provided, files located by artifactPath and logPath will be published to Amazon S3 and linked from the build results. By default the job will have a groupSymbol of 'j', and a groupName of 'Executed by Jenkins'. To leave the job ungrouped pass a groupSymbol of '?'.

Requirements

See publishToS3 and publishToPulse.

Examples

// submit default grouped build results without artifacts or logs
submitToTreeherder(
  project: 'foo',
  jobSymbol: 'T',
  jobName: 'Tests'
)

// submit ungrouped build results without artifacts or logs
submitToTreeherder(
  project: 'foo',
  jobSymbol: 'T',
  jobName: 'Tests',
  groupSymbol: '?'
)

// submit custom grouped build results with artifacts and log
submitToTreeherder(
  project: 'foo',
  jobSymbol: 'I',
  jobName: 'Integration tests',
  artifactPath: 'results/*',
  logPath: 'results/tbpl.txt',
  groupSymbol: 'T',
  groupName: 'Tests'
)

writeCapabilities

Writes a JSON file containing the items from the capabilities map to the specified path (for use by pytest-selenium). If omitted, the path defaults to capabilities.json in the working directory.

Examples

capabilities = [
  browserName: 'Firefox',
  version: '51.0',
  platform: 'Windows 10'
]

// write capabilities to default path
writeCapabilities(capabilities)

// write capabilities to specified path
writeCapabilities(
  desiredCapabilities: capabilities,
  path: 'fx51win10.json'
)

ServiceBook

testProject

Queries the Service Book project API for the given project name, iterates over its associated test repositories, checks them out from SCM, and executes their run file(s). Finally, it returns exit 0 on successful/passing tests, and exit 1 in the event of failed builds.

Examples

@Library('fxtest') _

def sb = new org.mozilla.fxtest.ServiceBook()  
sb.testProject('kinto')

Requirements

How to run tests

Make sure you have the latest Gradle install and run:

$ gradle check
BUILD SUCCESSFUL in 1s
2 actionable tasks: 2 up-to-date

Version History

1.10 (2018-01-15)

  • Remove superfluous test stage when using Service Book.
  • Use GIT_COMMIT when reporting revision to Treeherder.

1.9 (2017-09-14)

  • Fixed the production Service Book API URL

1.8 (2017-09-14)

  • Greatly simplified Service Book API calls using the HTTP Request Plugin.

1.7 (2017-09-04)

  • Introduced ServiceBook class, with testProject method to execute tests for all pipeline associated with the specified project name.

1.6 (2017-04-13)

  • Changed TBPL log name to buildbot_text in Treeherder message for log parsing. (#12)
  • Switched to YAML schema for Treeherder message validation. (#2)
  • Added link to Treeherder results to console log. (#11)
  • Provided a default group for Treeherder jobs. (#13)

1.5 (2017-03-31)

  • Changed S3 profile to fx-test-jenkins-s3-publisher.

1.4 (2017-02-31)

  • Introduced publishToS3, publishToPulse, and submitToTreeherder steps.

1.3 (2017-02-23)

  • Don't mark jobs as unstable if submitToActiveData fails.

1.2 (2017-02-22)

  • Added submitToActiveData step for publishing structured logs to S3 for processing by ActiveData.

1.1 (2017-02-13)

  • Changed order of arguments for ircNotification for ease of specifying alternate channel.

1.0 (2017-02-13)

  • Initial release with ircNotification and writeCapabilities steps.

fxtest-jenkins-pipeline's People

Contributors

davehunt avatar mozilla-github-standards avatar oremj avatar rbillings avatar stephendonner avatar tarekziade avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fxtest-jenkins-pipeline's Issues

Accept optional 'mention' parameter for ircNotification

Add an optional argument for specifying nicks to mention in the IRC notification. Perhaps something like:

ircNotification(
  channel: '#fx-test-alerts',
  mention: ['davehunt', 'chartjes']
)

These nicks should be added to the message sent to the IRC channel:

Project foo build #1: FAILURE: https://jenkins/job/foo/1/ @davehunt @chartjes 

Limit certain features to specific Jenkins instances

Currently our pipelines using this shared library will execute on any Jenkins instance, which may be undesirable. For example, whilst testing new features for the shared library on a local Jenkins instance, we wouldn't want IRC notifications to be sent, or results to be submitted to ActiveData/Treeherder.

One option might be a list of allowed hosts, which we could default to include fx-test-jenkins, whilst allowing it to be overridden using an environment variable. Something like FXTEST_HOSTS. If our staging instance is not to be included in the default (I'm thinking it shouldn't) then it would be good to have a way to opt-in for individual adhoc builds, too.

Use latest versions of dependencies for JSON schema validation

Due to JENKINS-43195 we're currently unable to depend on our own versions of dependencies for JSON schema validation. Once Pipeline Model Definition v1.1.2 is released, we should specify our own dependencies for these packages. This means we'd be unaffected by any change in the Jenkins dependencies, and would be able to take advantage of features from later versions.

Write tests for and document Service Book class

We need to write/improve docs and tests for the code in #23 which will hopefully make it into our shared library in some form or fashion.

Particularly perhaps, for starters, testProject('project name')?

More specifics as we iterate on this, together!

Use shaded imports for JSON validation

Due to JENKINS-43195 we're currently dependent on the version of json-schema-validator that Jenkins uses. Once Pipeline Definition Model plugin v1.1.2 is released we will need to update these imports from:

import com.github.fge.jsonschema.exceptions.ProcessingException
import com.github.fge.jsonschema.main.JsonSchemaFactory
import com.github.fge.jsonschema.util.JsonLoader

to:

import org.jenkinsci.plugins.pipeline.modeldefinition.shaded.com.github.fge.jsonschema.exceptions.ProcessingException
import org.jenkinsci.plugins.pipeline.modeldefinition.shaded.com.github.fge.jsonschema.main.JsonSchemaFactory
import org.jenkinsci.plugins.pipeline.modeldefinition.shaded.com.github.fge.jsonschema.util.JsonLoader

Alternatively, we could depend on our own version of this (and the FasterXML) dependency. This would avoid any unexpected issues if Jenkins updates its dependencies, and would allow us to take advantage of features from later releases.

Provide a default group for Treeherder jobs

As discussed in bug 1353715, top level jobs will not automatically group and large numbers of jobs for the same revision would therefore make Treeherder's UI difficult to use. We may ultimately need to revisit whether we should be submitting monitoring style jobs to Treeherder, but regardless it makes sense to provide a default group name and symbol. I recommend following the TaskCluster approach:

  • Group symbol: j
  • Group name: Executed by Jenkins

Thread leak in JsonSchemaValidator.groovy

Caused by JsonSchemaFactory.byDefault().

Full stack trace of thread creation:

	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
	at com.github.fge.msgsimple.provider.LoadingMessageSourceProvider.getMessageSource(LoadingMessageSourceProvider.java:190)
	at com.github.fge.msgsimple.bundle.MessageBundle.getMessage(MessageBundle.java:122)
	at com.github.fge.msgsimple.bundle.MessageBundle.getMessage(MessageBundle.java:146)
	at com.github.fge.jackson.JsonNodeReader.readNode(JsonNodeReader.java:140)
	at com.github.fge.jackson.JsonNodeReader.fromInputStream(JsonNodeReader.java:103)
	at com.github.fge.jackson.JsonLoader.fromResource(JsonLoader.java:99)
	at com.github.fge.jsonschema.SchemaVersion.<init>(SchemaVersion.java:63)
	at com.github.fge.jsonschema.SchemaVersion.<clinit>(SchemaVersion.java:44)
	at com.github.fge.jsonschema.core.load.configuration.LoadingConfigurationBuilder.<init>(LoadingConfigurationBuilder.java:117)
	at com.github.fge.jsonschema.core.load.configuration.LoadingConfiguration.byDefault(LoadingConfiguration.java:151)
	at com.github.fge.jsonschema.main.JsonSchemaFactoryBuilder.<init>(JsonSchemaFactoryBuilder.java:67)
	at com.github.fge.jsonschema.main.JsonSchemaFactory.newBuilder(JsonSchemaFactory.java:123)
	at com.github.fge.jsonschema.main.JsonSchemaFactory.byDefault(JsonSchemaFactory.java:113)
	at com.github.fge.jsonschema.main.JsonSchemaFactory$byDefault.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at org.mozilla.fxtest.JsonSchemaValidator.validate(JsonSchemaValidator.groovy:20)
	at org.mozilla.fxtest.JsonSchemaValidator$validate.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
	at threadLeak.call(/var/lib/jenkins/jobs/oremj-test-2/builds/125/libs/oremjtest/vars/threadLeak.groovy:3)
	at WorkflowScript.run(WorkflowScript:5)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:74)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

Named attributes help

In the readme you suggest we can call a custom library using the named attributes:

// specify all values
ircNotification(
  channel: '#fx-test-alerts',
  nick: 'fxtest',
  server: 'irc.mozilla.org:6697'
)
slackNotify(
  message: 'custom message testing',
  channel: '#jenkinst-test',
  color: 'danger'
)

When I try the example I get an error but it works when I remove the names:

hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: slackNotify.call() is applicable for argument types: (java.util.LinkedHashMap) values: [[message:custom message testing, channel:#jenkins-test, color:danger]]
slackNotify(
  'custom message testing',
  '#jenkinst-test',
  'danger'
)

Is there a plugin or trick to get the names to work?

Wiki changes

FYI: The following changes were made to this repository's wiki:

These were made as the result of a recent automated defacement of publically writeable wikis.

Improve platform in Treeherder message

The platform in the Treeherder message is currently the operating system, version, and architecture separated by dashes. An example would be: linux-3-10-0-327-36-2-el7-x86_64-amd64. This isn't easy to read in Treeherder, so we should improve it. Treeherder uses a platform map for displaying platforms, however the keys don't appear to all follow the same format. We could probably do a combination of modifying the value in the message, and adding keys to the existing Treeherder map.

Add build cause to Treeherder message

Currently we hard-code the build reason as 'scheduled'. We should investigate if it's possible to retrieve the actual build cause from Jenkins. According to the documentation, currentBuild.rawBuild will return a hudson.model.Run object, which provides a getCauses method.

Note this additional information from the documentation:

only for trusted libraries or administrator-approved scripts outside the sandbox; the value will not be Serializable so you may only access it inside a method marked @NonCPS

This may mean that using rawBuild would require script approval for our shared library, which may be undesirable. Another option would be to reach out to the Jenkins development team to determine if there's any other way to access the build causes.

v1.8 fails to get the project repo from Service Book's API, and is returned a 404, instead

10aa90c7fab8fafd82abfeefdde4ceec--bat-signal-batman-logo

@tarekziade @davehunt HALP!

Version 1.8 fails to retrieve the project repo from the Service Book API endpoint, and instead receives a 404 after issuing its GET:

Testing project kinto
[Pipeline] httpRequest
HttpMethod: GET
URL: https://servicebook.stage.mozaws.net/api/project
Sending request to url: https://servicebook.stage.mozaws.net/api/project
Response Code: HTTP/1.1 404 NOT FOUND
[Pipeline] End of Pipeline

I suspect that it's because we might not be issuing the POSTs nor PUTs which we were in 1.7 (nor, it appears, sending content-type: application/json).

Working: https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/kinto.integration.stage/35/console
Failing: https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/kinto.integration.stage/28/console

More as I have it!

Full output here: https://gist.github.com/stephendonner/83e577e1f7ba420248579545f40ee25e

Write a suite of tests

We should introduce tests, linting, and coverage reporting for this shared library. This will give us confidence when introducing new functionality and reduce the chance of introducing regressions. These should be run automatically whenever there's a pull request or a push, using Travis CI or similar.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

  1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
  2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.