Git Product home page Git Product logo

james-jenkins's Introduction

Deprecation notice: This project relies on an outdated Jenkins version, subject to many vulnerabilities.

It is furthermore complex to maintain.

The Apache James project had been contributing a Jenkins2 pipeline that should be used instead.

Immutable CI

Summary

This project aims at describing our CI using jenkins DSL Plugin.

It starts Jenkins with Git and DSL Plugins, fetch a git repository to retrieve the DSL configuration and finally make sure this DSL is run at start and on every changes.

That way, you just have to write your jobs in build-script file and your jenkins or any future jenkins will make them run.

How to use it

Key container

We want our TLS keys to be in an other container. This allows us to publish our Jenkins stateless container, and to be able to easily change the TLS keys. Jenkins container will use the --volumes-from when run to access certificates.

  • First put your TLS keys in the $PWD/keys folder

You are expected to provide :

  • ca.pem : The CA of your server certificate

  • cert.pem : Your client certificate

  • key.pem : The key used to generate client certificate

To construct the key holding container :

$docker build -f DockerFileKeyContainer -t keys .
$docker run -v /keys --name keys keys

Launch Jenkins

To start a jenkins instance, you need to have a key container on the node you are launching jenkins. Then:

$docker build --tag myjenkins .
$docker run -p 8080:8080 --volumes-from keys --env-file=./env.file --name jks myjenkins

If you forecast to use slaves with your master, you should also export the 50000 port, so the previous line should be:

$docker run -p 8080:8080 -p 50000:50000 --volumes-from keys --env-file=./env.file --name jks myjenkins

You have to modify the env.file file with the following parameters:

  • JENKINS_URL: is the external URL of your jenkins (ex. "http://ci-james.org:8080/")

  • GITHUB_TOKEN: is the token used to access the GitHub API (https://github.com/settings/tokens)

  • DOCKER_USER: is the Docker Hub user used to publish the build, see publishing section (ex. myuser)
    If not given, publishing to Docker Hub will not be enabled.

  • GITLAB_TOKEN: is the GitLab token used to trigger a job generating packages on packages.linagora.com.Docker.
    If not given, packaging will not be enabled.

  • DOCKER_PASSWORD: is the password of the Docker Hub user

Then, you can open it in your browser :

$firefox http://`docker inspect --format {{.NetworkSettings.IPAddress}} jks`:8080
or
$firefox http://JENKINS_URL

Starting a slave

You may need to start one (or several) Jenkins slave.

$docker build -f DockerfileSlave -t jenkins-slave .
$docker run -d -v /var/run/docker.sock:/var/run/docker.sock --volumes-from=keys --rm --name jenkins-slave jenkins-slave -url <master-url> <secret> <host>

Where:

  • master-url: is the URL of your jenkins

  • secret: is the agent secret

  • host: is the agent name

Publishing to Docker

This feature is only available when building the master branch.

You may want to publish the image produced by the build on a Docker Hub account. To achieve that, you have to provide a user name and its password in the env.file file.

Jobs requirements

You need some extra container for the jobs to run well.

James Keystore

If you want to use james deploy script, you need to have a docker container with the James TLS certificates. You may not want to publish a container with such sensitive content, or even commit it in a source code management system.

First copy James TLS certificate to the directory containing this README. Or you can generate them using :

$keytool -genkey -alias james -keyalg RSA -keystore keystore
Note
To generate James keystore this way, you need to have keytool installed. This tools comes from the JDK.

Be sure that the password you provide matches with James configuration. You will find it in james-parent project, in the destination/conf directory. These files are 'imapserver.xml', 'pop3server.xml' and 'smtpserver.xml'. Edit them locally before launching james image build. Default value is set to 'james72laBalle').

Then, run these command to set up this container on nodes that will hosts James servers :

$docker build -f DockerFileKeystoreContainer -t keystore .
$docker run -v /keys --name keystore keystore

Launch Jenkins on the specified branch

If you change the workflow-job and would like to launch Jenkins build on new branch

Create new "Workflow" which point to you james-jenkins branch with

Repositories point to your github repository Repositories point to your branch Script Path point to groovy script that you want to run

Clone new job from base job: create "New item" -→ Choose "Copy existing Item" with the base job’s name

Edit the new cloned job

GitHub project point to your github repository
Projects to build point to above workflow
Define the parameter

repoURL=<yours_repository> branch=<yours_branch> sha1=<your_last_commit_id_at_branch> version=<the_James_version_of_this_branch>

james-jenkins's People

Contributors

aduprat avatar arsnael avatar blackheaven avatar chibenwa avatar hoangdat avatar matthieu-at-linagora avatar mbaechler avatar mburns avatar remk avatar rouazana avatar trantienduchn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

james-jenkins's Issues

Faster build with TMPFS on /root/.m2 directory ?

Currently, /root/.m2 within compilation containers is mapped on file system...

This means every time maven copies a local resource, it needs to read over the filesystem

I believe significant resources could be saved by having ".m2" folder being mapped to TMPFS...

Conflicts between unmerged PR during deployement tests

In linagora/james-project#485 I got an error that seems linked to an unmerged pull request :

09:19:03.732 [ERROR] o.a.j.m.i.c.CamelCompositeProcessor - Unable to init mailet Metrics: javax.mail.MessagingException: Can not load mailet Metrics;
  nested exception is:
	java.lang.ClassNotFoundException: org.apache.james.transport.mailets.Metrics

(https://james.open-paas.org/job/build%20pull-requests/670/)

  • GitHub pull request #485 of commit 9f48676

Looks like we end up with configuration from an other PR.

My thoughts about that :

  • We use the same workspace
  • Our build is dockerized, hence conflict free
  • But we use the same local folders for :
    • Copying binaries
    • And config files

We need to prefix such directories with buildId.

Use gradle to get exact version of Jenkins and plugins

I've been exploring immutable infrastructure for Jenkins as well. One of the primary challenges is that when installing Jenkins; it uses the latest version of all plugins. This is not good for immutable infrastructure because if you bootstrap your Jenkins a year from now it is not guaranteed to work because it will be on different versions.

One solution I've found to this is to use Gradle to download the WAR and exact version of all plugins. I took it a step further and the build.gradle packages an RPM and a DEB package which can be used to install the immutable Jenkins.

I thought you may find it interesting so I'm opening this issue to bring your attention to it.

dockerIp is null

This variable is not defined hence workflow command:

def integrationArguments = "${dockerIp} \$JAMES_PORT ${sha1ToBuild}"

[...]

sh "${findImapPortJPA} && docker run ${verbose} --name=${containers.integrationJPA} --entrypoint=\"/root/integration_tests.sh\" ${useMavenCache} --volumes-from=${containers.gitPublish} ${images.jamesCompile} ${integrationArguments}"

Result in:

[133] Running shell script
+ docker inspect -f {{(index (index .NetworkSettings.Ports "143/tcp") 0).HostPort}} james-server-cassandra-mpt-41-deploy-2af9b1f7-e013-4005-8c10-7970102fd666
+ export JAMES_PORT=41346
+ docker run -a stdout -a stderr --name=integration-cassandra-mpt-41-deploy-2af9b1f7-e013-4005-8c10-7970102fd666 --entrypoint=/root/integration_tests.sh -v maven-cache-volume:/root/.m2 --volumes-from=git-publish-mpt-41-deploy-2af9b1f7-e013-4005-8c10-7970102fd666 james-parent-compile-mpt-41-deploy-2af9b1f7-e013-4005-8c10-7970102fd666 null 41346 

And fails:

[55d605a8756ec86e29f9ec7ee4a4041b09d94a2b] Tests in error: 
[55d605a8756ec86e29f9ec7ee4a4041b09d94a2b]   DeploymentValidation.validateDeployment:50 » UnresolvedAddress

We need to set this to localhost

Remove Slack notification for master build

We changed our chat from Slack to Mattermost.

While waiting for Mattermost integration, we have to disable slack notifications so that a push to the no more active account do not make the build fail.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.