openscap / jenkins Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 10.0 225 KB

Issue tracker for our Jenkins continuous integration infrastructure.

Shell 100.00%

jenkins's People

Contributors

Stargazers

Watchers

Forkers

yuumasato dahaic redhatrises matejak cipherboy jan-cerny ggbecker mildas matusmarhefka swipswaps

jenkins's Issues

OAA PR filter is very hard to read

The OAA PR job configuration uses a matrix build together with a filter.
The filter is really hard to read:

   !(t_branch=="rhel7-branch" && distributions=="fedora")
&& !(t_branch=="master" && distributions=="rhel7")
&& !(t_branch=="rhel8-branch" && distributions=="rhel7")

What about replacing it with something like

   (t_branch=="rhel7-branch" && distributions=="rhel7")
|| (t_branch=="rhel8-branch" && distributions=="rhel8")
|| (t_branch=="master" && distributions=="fedora")

and in case when master is compatible with rhel8 just add that combination to the condition.

Moreover, I think that the first form works only because the rhel8 label is left out from the matrix.
As the second solution looks obviously better, I have a feeling that I may be missing something.

Jenkins raises exceptions while testing pull requests

It happened to me twice today. Jenkins failed to build a pull request from github. It gave a big exception.
See the log:

GitHub pull request #270 of commit 060f336dd7c8b2e72e900f974c4d202a22737560, no merge conflicts.
Setting status of 060f336dd7c8b2e72e900f974c4d202a22737560 to PENDING with url https://jenkins.open-scap.org/job/openscap-pull-requests/283/ and message: 'Build started sha1 is merged.'
[EnvInject] - Loading node environment variables.
Building remotely on el6 (node-el6 rhel6) in workspace /home/jenkins/workspace/openscap-pull-requests
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://github.com/OpenSCAP/openscap.git # timeout=10
Fetching upstream changes from git://github.com/OpenSCAP/openscap.git
 > git --version # timeout=10
 > git fetch --tags --progress git://github.com/OpenSCAP/openscap.git +refs/pull/*:refs/remotes/origin/pr/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://github.com/OpenSCAP/openscap.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:763)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1012)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1043)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
    at hudson.model.Run.execute(Run.java:1738)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://github.com/OpenSCAP/openscap.git +refs/pull/*:refs/remotes/origin/pr/*" returned status code 128:
stdout: 
stderr: fatal: Unable to look up github.com (port 9418) (Temporary failure in name resolution)

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1388)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to el6(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
    at hudson.remoting.Channel.call(Channel.java:781)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
    at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
    at com.sun.proxy.$Proxy51.execute(Unknown Source)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:761)
    ... 11 more
ERROR: null
An attempt to send an e-mail to empty list of recipients, ignored.
Setting commit status on GitHub for https://github.com/OpenSCAP/openscap/commit/588187a15286d75f2f4ab6154dfdcff3c52ed7d8
Setting status of 060f336dd7c8b2e72e900f974c4d202a22737560 to FAILURE with url https://jenkins.open-scap.org/job/openscap-pull-requests/283/ and message: ' No test results found.'
Finished: FAILURE

Jenkins service needs periodic restarts

If not restarted (after update?), user cannot log in. Possibly just update project needs to be fixed.

ShellCheck is missing at least on some slaves

EL6 slave SSG build:

Skipping ShellCheck analysis, ensure shellcheck executable is present in the PATH!

Build failures in OpenSCAP maint

https://jenkins.open-scap.org/job/openscap-maint/10/

Build and test itself looks OK, it successfullly creates the "dist tarball", but then it fails:

Cannot find config with Id [org.jenkinsci.plugins.managedscripts.ScriptConfig1445854237346]. Are you sure it exists? Please check the configuration.
Build step 'Execute managed script' marked build as failure
Sending e-mails to: [email protected]
Finished: FAILURE

Test `./configure --disable-probes`

We need ./configure --disable-probes to work for SCAP Workbench binary builds for Windows and OSX. This is only tested when I build SCAP Workbench and we often break this scenario.

Please add this configuration in Jenkins so that we continuously test it.

release-testing-run-all clogs the system

2 of these jobs have been running for several days. They are endangering the stability of the system because they clog some of the slaves.

If the job really takes this long we probably should decide to run it in parts.

Could we investigate this? Maybe we can disable this job for the time being?

Change Jenkins email "From" address

Right now it's [email protected] and redhat.com has SPF records which means the mail will get rejected when arriving to most SMTP servers.

We need to change it to something else.

Remove Fedora 23 node

Fedora 23 EOL was on December 20th, 2016
We waste resources by keeping it on.

[RFE] Update our Jenkins infrastructure WRT to JSA 2016-05-11

See:

for further details. Not sure some of the issues would be exploitable on our instance too, but we should definitely upgrade to stay on safe side.

Thank you, Jan.

P.S.: Giving to Zbynek, since he best knows how to perform the update. Are the necessary tasks need to be performed to update our Jenkins CI documented somewhere? (if so, I can have a look) If not, we should probably create such document.

RFE: Enhance scap-security-guide-nist-testsuite Jenkins job to perform ScapVal validation tests also for RHEL/7 SSG product

The scap-security-guide-nist-testsuite Jenkins CI job is currently performing ScapVal tests for SSG RHEL/6 content. Due to:

[1] https://github.com/OpenSCAP/scap-security-guide/labels/RHEL7-meet-ScapVal-1.2.14.1-requirements

we need to enable this Jenkins job also for RHEL/7 content and address the issues reported in [1].

OpenSCAP system_info probe is not being built on the el6 slave

Not sure about the other slaves. It says header files are missing.

 * Checking presence of required headers for the system_info probe
checking for arpa/inet.h... yes
checking for ctype.h... yes
checking for errno.h... yes
checking for ifaddrs.h... yes
checking for libdlpi.h... no
checking for netdb.h... yes
checking for net/if.h... yes
checking for net/if_types.h... no
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for sys/ioctl.h... yes
checking for sys/socket.h... yes
checking for sys/sockio.h... no
checking for sys/utsname.h... yes
checking for unistd.h... (cached) yes

 === probes ===
  system_info:                 NO (missing: header files)
  family:                      yes
  textfilecontent:             yes
  textfilecontent54:           yes
  variable:                    yes
  xmlfilecontent:              yes

Did we accidentally introduce a new dependency in OpenSCAP or was this always broken?

Please apply "JENKINS_JAVA_OPTIONS="-Dhudson.model.User.SECURITY_243_FULL_DEFENSE=false" in /etc/sysconfig/jenkins on OpenSCAP Jenkins slaves as a temporary workaround for JENKINS-34775 issue

JSA 2016-05-11 fixed couple of security issues (within the list also Malicious users with multiple user accounts can prevent other users from logging in SECURITY-243 / CVE-2016-3722).

But as has been proven in OpenSCAP Jenkins CI infrastructure for the jobs being built during the last week (they are stuck due to problem in Jenkins github-oauth-plugin) the fix for SECURITY-243 / CVE-2016-3722 introduces the following Jenkins Hudson traceback:

FATAL: org.acegisecurity.providers.UsernamePasswordAuthenticationToken cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken
java.lang.ClassCastException: org.acegisecurity.providers.UsernamePasswordAuthenticationToken cannot be cast to org.jenkinsci.plugins.GithubAuthenticationToken
    at org.jenkinsci.plugins.GithubSecurityRealm.loadUserByUsername(GithubSecurityRealm.java:639)
    at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1050)
    at hudson.model.User.get(User.java:395)
    at hudson.model.User.get(User.java:364)
    at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:374)
    at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:435)
    at hudson.model.AbstractBuild.getCulprits(AbstractBuild.java:350)
    at hudson.model.AbstractBuild.getCulprits(AbstractBuild.java:346)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:669)
    at hudson.model.Run.execute(Run.java:1766)
    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE

List of selected Jenkins CI jobs affected by this problem (to mention some of them):

This issue has been reported (2016-05-12) to Jenkins upstream:
[1] https://issues.jenkins-ci.org/browse/JENKINS-34775

but so far there isn't a patch for the problem yet (contributions welcome).

Though there's a suggested workaround for the SECURITY-243 issue problem to add:

JENKINS_JAVA_OPTIONS="-Dhudson.model.User.SECURITY_243_FULL_DEFENSE=false"

setting into /etc/sysconfig/jenkins.

We should consider applying that workaround to OpenSCAP Jenkins CI slaves / nodes till the issue [1] is fixed in Jenkins upstream.

Thank you, Jan

Include ruby-openscap project in CI

Hello,

Please include ruby-openscap project in our jenkins testing as well.

The repository includes very simple ./runtest.sh that should be fine for CI. https://github.com/OpenSCAP/ruby-openscap/blob/master/runtest.sh

It however needs to have gem and rubocop packages installed. So this is up perhaps up for a discussion.

Jenkins login is broken

This is what I get when trying to log in with github acc:

javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.jenkinsci.plugins.GithubSecurityRealm.doFinishLogin(GithubSecurityRealm.java:432)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:196)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:90)
    at sun.security.validator.Validator.getInstance(Validator.java:179)
    at sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:312)
    at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:171)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:184)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    ... 82 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
    at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
    at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
    at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:88)
    ... 94 more

Fedora 25 node missing wget

Last two runs of scap-security-guide-nist-testsuite in jenkins have failed because Fedora 25 node doesn't have wget installed, see following output:

[scap-security-guide-nist-testsuite] $ /bin/bash /tmp/hudson6603223906284917495.sh
/tmp/nist_testsuite_zip_SCAP%20Content%20Validation%20Tool%201.2.zip ~/workspace/scap-security-guide-nist-testsuite
/tmp/hudson6603223906284917495.sh: line 17: wget: command not found
Build step 'Execute shell' marked build as failure

Out of disk space jenkins

Free disk space was depleted on one of jenkins slave today.
Classic jobs should not grow in space too much, because we delete old files.

We have 20GB/slave, I think it is enough.

What we need:

Find what use too much diskspace
if lots of files rest in /tmp, we need to clean it in someway
- weekly jenkins job
- restart machine? (find you if it clean /tmp)
maybe monitor disk space - we can add simple check to script what run every week

any tips?

OpenSCAP CI started to comment on pull request

OpenSCAP CI started to write comments into pull requests, which pollutes the discussion and makes pull requests harder to understand.

Do not make openscap distcheck with -k

It is major PITA. I am swearing pretty bad. I do not like PITA.

In the dark past we have enabled make distcheck -k for nightly automation. That was during the times of irregular nightly builds. It made a kinda sense, because we wanted to see all the failures.

However, now things have changed. We have automation before&after each pull-request. We want to encourage people to look at the failed tests and see immediately what went wrong. Now, when uneducated user see the failed log from openscap jenkins, they basically have low chance to spot the failure. It takes me many many seconds before I get to the first failure. It makes me feel sick. It makes me not click on the jenkins result. I don't remember, if there were ever two failures in the log since the migration to public jenkins.

In conclusion the pain put on the shoulders of log readers is unjustifiable. I am very concerned. :-)

Provide test results

We can provide test result file after every test, so we will able to have nice jenkins-managed stats of tests.

Expected workflow:

Run tests over with | tee "$result"
Transform "$result" to some standardized test result format (.xml/..)
Enable job to collect result file
Enjoy stats
(Maybe more readable results on github)

https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin

SSG pull requests are tested serially and only on RHEL6

The task has a label rhel6, this means it can only be tested on one of the slaves.

Is this what we want? That means that we are never testing oval 5.11 in jenkins.

Can't connect to SSH

I need to examine build https://jenkins.open-scap.org/job/openscap-pull-requests/298/ of OpenSCAP/openscap#276, but I am not able to connect to ssh on it. Firstly I thought that simply my home network ISP doesn't allows such connection, but now it does not work also from office. SSH gives an error message. Here is the verbose output.

[jcerny@thinkpad ~]$ ssh -v 'jan-cerny@298#openscap-pull-requests.jenkins.open-scap.org'
OpenSSH_7.1p1, OpenSSL 1.0.2e-fips 3 Dec 2015
debug1: Reading configuration data /home/jcerny/.ssh/config
debug1: /home/jcerny/.ssh/config line 5: Applying options for *.jenkins.open-scap.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Executing proxy command: exec ssh -q -p 56917 jenkins.open-scap.org diagnose-tunnel -suffix .jenkins.open-scap.org 298#openscap-pull-requests.jenkins.open-scap.org
debug1: permanently_drop_suid: 1000
debug1: identity file /home/jcerny/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jcerny/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
ssh_exchange_identification: Connection closed by remote host

Add Fedora 24 node

Fedora 24 was released on June 21st. We need to set up Fedora 24 virtual machine, add it to Jenkins and use it for builds and tests on Jenkins.

The "scap-security-guide-nist-testsuite" Jenkins CI job is still reporting failure (even when it was passing in the past)

The "scap-security-guide-nist-testsuite" Jenkins CI job is still failing (regardless if the internal testing job passed or not). Have a look at this example:

https://jenkins.open-scap.org/job/scap-security-guide-nist-testsuite/140/console

As can be seen there were "0 Errors" detected, but the job was still marked as failing. The corresponding error / warning message was:

Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed:
No test report files were found. Configuration error?
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: FAILURE

This makes me to think there's some issue with the configuration of the testing job itself, rather than failure in the tests.

P.S.: Right now it's truly failing (example: https://jenkins.open-scap.org/job/scap-security-guide-nist-testsuite/145/console -- that's a different issue, we will fix in SSG directly). But this report is about the fact, despite of there being 0 Errors, the ssg-nist-testsuite Jenkins CI job is still marked / reported as failing.

Job scap-security-guide-nist-testsuite fails due to missing NIST SCAP Content Validator

Current link (http://scap.nist.gov/revision/SCAP%20Content%20Validation%20Tool%201.2.zip) to SCAP Content Validator returns 404.

From this page, it looks like new URL should be https://csrc.nist.gov/CSRC/media/Projects/Shared/tools/scap/1.2/SCAP%20Content%20Validation%20Tool%201_2.zip. But even this one returns 404.

An e-mail has been sent inquiring about this.
We should update job when the SCAP Content Validator is available on-line again.

Put 'execute shell' contents to this git

We have growing number of jobs.
Each job has an 'execute shell' section which lists commands, that should be run on the jenkins

I convinced we should put these shell scripts to this git, here are my thoughts:

Scripts are currently not visible for the community. Hard to understand what is going on. We should be more transparent.
Multiple jobs for each project should be always in sync. However, we currently rely on people to keep them in sync. Did you know that people make mistakes? Did you know how many jobs are there for openscap base?
They are not easily accessible. It always takes me time to find them, when I need to know what is tested and what is not. That makes this information source less appealing.
We don't have history tracked for these scripts. Why are they such and such?
We cannot have a discussion on changes (pr review) as we are used to for all the other stuff.
When I have to edit multiple same scripts, I try to make excuses. That means, we need to see major pain, before we start editing these scripts. That causes us to avoid small improvements. That makes us dull. :)

I suggest we have all the execute shell the same:

project= # openscap, scap-security-guide, ruby-openscap ...
git clone https://github.com/OpenSCAP/jenkins #or pull
exec bash -x scripts/$project.sh

Thoughts?

"scap-security-guide" SSG CI Jenkins job now returns "java.io.IOException: remote file operation failed" result on Fedora22 and Fedora23 slaves (rather than performing testing of the actual SSG pull request that got merged)

AFAICT this behaviour started after most recent Jenkins upgrade last Thursday (2016-01-21) IIRC:

Current result:

IOW rather the scap-security-guide to perform the testing of the particular / underlying PR that got merged, it returns some Jenkins CI error:

java.io.IOException: remote file operation failed: /home/jenkins/workspace/scap-security-guide at hudson.remoting.Channel@3af655b6:fedora23: java.io.IOException: Remote call on fedora23 failed
..
Caused by: java.io.IOException: Remote call on fedora23 failed
..
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8

Expected result:

Underlying merged PR is tested.

Additional information:

So far this behaviour has been noticed / seems to be to "scap-security-guide" Jenkins SSG CI job specific (other openscap Jenkins CI jobs seem to be actually testing what's desired AFAICT). Also so far this behaviour has been noticed on the following two Jenkins slaves:

Fedora22
Fedora23

Thanks, Jan.

Update Jenkins Jobs list in master.yml according xml files.

Update list:

jenkins/ansible/master.yml

Line 29 in d093856

jenkins_jobs:

according list of jobs present in ansible/jobs/.

Jenkins Security Advisory 2015-11-11 fixing multiple security flaws (CVE-2015-5317 up to CVE-2015-5326, CVE-2015-8103) => Jenkins update required

Multiple security flaws have been recently reported:
[1] http://article.gmane.org/gmane.comp.security.oss.general/18100

against Jenkins CI, and corrected upstream:
[2] https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2015-11-11

There's even an exploit available for the deserialization issue already (CVE-2015-8103):
[3] http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/#jenkins

We should update our infrastructure (used Jenkins version) to ensure we stay on the safe side.

Thank you, Jan.

TLS certificate from letsencrypt

Currently we use a self-signed AND relatively weak TLS cert, we should move to TLS 1.2 letsencrypt certificate that is trusted by browsers.

As part of the letsencrypt beta program I have whitelisted jenkins.open-scap.org. It needs coordination with somebody who knows the infrastructure to deploy it though.

A PR is pointing to wrong check in jenkins

Jenkins test link in OpenSCAP/scap-workbench#113 is pointing to wrong run of checks.

While it points to https://jenkins.open-scap.org/job/scap-workbench-pull-requests/1/ it should point to https://jenkins.open-scap.org/job/scap-workbench-pull-requests/38/

[RFE] Allow Jenkins CI job to show result of testing from more than just one slave

The scap-security-guide-pull-requests Jenkins CI job has been configured to run all pull requests checks in parallel on all three of the following systems:

el6 to verify OVAL-5.10 content is produced properly (mainly wrt to make validate targets),
el7 to verify OVAL-5.11 content is produced properly (mainly wrt to make validate targets), and
fedora23 to verify:
- OVAL-5.11 content is produced properly (mainly wrt to make validate targets), but
- also verify the ShellCheck executable won't report some regression on the syntax of provided remediation scripts

The example PR on which all three tests have been performed is the following one:
[1] ComplianceAsCode/content#1048

The issue is having look at All checks have passed section there's only 1 successful check report, not 3.

Since we need the scap-security-guide-pull-requests jenkins CI job to:

pass only if the specific PR has passed on all three nodes,
fail otherwise,

the All checks have passed section should be enhanced to list all three test results.

@ybznek You think this would be possible to implement?

Thanks, Jan.

[scap-security-guide-nist-testsuite] Jenkins CI testing job fails despite RHEL-6 content containing no errors according to scapval

The scap-security-guide-nist-testsuite is now currently failing despite ScapVal-1.2.14.1 not reporting any errors in the currently produced RHEL-6 SSG master content:

# java -jar lib/scapval-1.2.1.14.jar -scapversion 1.2 -file /root/scap-security-guide/RHEL/6/output/ssg-rhel6-ds.xml
...
INFO : APPLICATION - Gathering statistics from datastream
INFO : STATISTIC - Total Number of xccdf:Rule(s) with OVAL checks: 0
INFO : STATISTIC - Total Number of xccdf:Rule(s) with OCIL checks: 0
INFO : STATISTIC - Total Number of xccdf:Rule(s) with only OCIL checks: 0
INFO : STATISTIC - Total Number of xccdf:Rule(s) with at least 1 CCE: 0
INFO : STATISTIC - Total Number of runlevel_test: 798
INFO : STATISTIC - Total Number of rpminfo_test: 187
INFO : STATISTIC - Total Number of textfilecontent54_test: 679
INFO : STATISTIC - Total Number of sysctl_test: 42
INFO : STATISTIC - Total Number of family_test: 6
INFO : STATISTIC - Total Number of partition_test: 38
INFO : STATISTIC - Total Number of file_test: 98
INFO : STATISTIC - Total Number of variable_test: 26
INFO : STATISTIC - Total Number of xmlfilecontent_test: 44
INFO : STATISTIC - Total Number of uname_test: 4
INFO : STATISTIC - Total Number of rpmverifyfile_test: 8
INFO : STATISTIC - Total Number of interface_test: 2
INFO : STATISTIC - Total Number of selinuxsecuritycontext_test: 4
INFO : STATISTIC - Total Number of password_test: 2
INFO : STATISTIC - Total Number of environmentvariable58_test: 12
INFO : APPLICATION - Finished SCAP content validation in 00:00:52.453.
INFO : Generating the results report...
INFO : 616 Warnings and 0 Errors in results.
INFO : See results in scap-validation-result.xml.
# grep ERROR scap-validation-result.{xml,html}

The issue is here since 2016-Apr-22, when I have re-configured that Jenkins job (removed those previously whitelisted errors, that should be fixed in ScapVal-1.2.14.1 version already, thus don't need to be whitelisted anymore).

The issue seems to be the following statement in the configuration:

function transformateResult {
...
  errors=$(grep "^ERROR" "$resultFile")
}

But since errors is now empty, the statement fails with non-zero exit code, and whole testing job fails (the zero count of errors is even visible in previous runs of the job, e.g.:
https://jenkins.open-scap.org/job/scap-security-guide-nist-testsuite/103/console

Hopefully I will get today to fix this.