gocd / kubernetes-elastic-agents Goto Github PK

View Code? Open in Web Editor NEW

34.0 15.0 31.0 4 MB

Kubernetes Elastic agent plugin for GoCD

Home Page: https://www.gocd.org

License: Apache License 2.0

Java 95.36% HTML 4.64%

kubernetes-elastic-agents gocd-elastic-agents gocd kubernetes gke minikube kops

kubernetes-elastic-agents's Introduction

Kubernetes Elastic agent plugin for GoCD

Kubernetes Elastic Agent Plugin provides production grade support to run GoCD Elastic Agents on Kubernetes Cluster.

Building the code base

To build the jar, run ./gradlew clean test assemble

Documentation

Read about setting up a Kubernetes cluster and using GoCD on Kubernetes in GoCD's documentation. Installation and configuration documentation for this plugin is available here.

License

Copyright 2022 Thoughtworks, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

kubernetes-elastic-agents's People

Contributors

Stargazers

Watchers

kubernetes-elastic-agents's Issues

GoCD elastic agent/cluster profile misconfigured

I deployed GoCD kubernetes helm chart into Openshift OKD. GoCD automatically created cluster profile on startup but It didnt work, since status report or scheduling didnt work.

The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping'

Failed to handle request cd.go.elastic-agent.cluster-status-report

Then I modified cluster profile I changed cluster url to point to url I got from running "kubectl cluster-info", before it was some internal cluster ip

I changed security token to gocd namespace default secret and I tried with gocd-app secret token created by helm, didnt work

A cluster cannot be used by more than one Go Server

When the same cluster is used by 2 (or more) Go Servers, each Go Server plugin instance, based on the timeout for registering kills the other Go Server pods.

Example:
Go Server A creates an elastic agent A
Elastic agent A is running a job that takes more than the register timeout defined in Go Server B
Go Server B terminates Elastic agent A

From looking at the code, the criteria used to obtain the list of pods to check for register timeouts doesn't take into account the Go Server. It considers all the pods irregardless of the Go Server.

Documentation link in README.md doesn't work

Documentation link in README.md doesn't work:

https://docs.gocd.org/current/gocd_on_kubernetes/ redirects to 404 NOT FOUND.

Plugin throws NullPointerException while create agent request when pods for cluster profile are not refreshed

Plugin fails to create an agent with the following exception:

2019-09-05 13:42:04,895 ERROR [5611@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
java.lang.NullPointerException: null
	at cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:44)
	at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
	at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:60)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:43)
	at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
	at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
	at java.lang.Thread.run(Thread.java:748)

This happens due to this section of the code returning null. The function performs a null check to refreshAgentInstances if it doesn't exist. But, it does not change the agentInstances local variable (which is currently null)

Agent Status Fails If Pod Contains Multiple Containers

Hi,

We're using Elastic agents and the pod spec we use has multiple containers within them. This causes the 'Agent Status' page to break as the plugin is unable to retrieve logs etc. from the pod as it does not specify a given container:

Failure executing: GET at: https://<>/api/v1/namespaces/gocd/pods/gocd-admin-agent-8e18234c-d4f2-49f6-b389-646bb1572bd3/log?pretty=true. Message: a container name must be specified for pod gocd-admin-agent-8e18234c-d4f2-49f6-b389-646bb1572bd3, choose one of: [gocd-agent docker]. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=a container name must be specified for pod gocd-admin-agent-8e18234c-d4f2-49f6-b389-646bb1572bd3, choose one of: [gocd-agent docker], metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={}).

This just affects the status report for individual jobs (i.e. https://<>/go/admin/status_reports/cd.go.contrib.elasticagent.kubernetes/gocd-admin-agent-8e18234c-d4f2-49f6-b389-646bb1572bd3)

Would it be within scope to provide an option to specify the master container or something to that affect?

Limit Maximum Pending Pods on the Cluster

Issue Type

Bug Report

Description

Currently, the plugin creates a pod for each createAgentRequest. On a performance run, with hundreds of jobs, we have seen the issue where plugin keeps creating a pod for the createAgentRequest, due to limited resources of the cluster, the pods are in pending state for a longer time and later were cleared after the agentAutoRegusterTimeout.
The plugin was busy doing this loop due to resource constraints.

Do not allow the plugin to create more pods when the maximum pending pod limits have reached.

Possible Fix

Allow Administrators to specify the maximum pending kubernetes pods count.
maximumPendingKubernetesPodsCount value will be checked before creating any pod.

Display Job Information associated with pod on status report page

Status Report should include JobIdentifier information:
- Container building job.
  - Each container row should also include the information about the building job.
- Link to build_details_page
  - The job identifier field from the container row should link to the build_details_page of the current job.
- Highlight Pod-Job Relation
  - When a user navigates from build_details_page to status_report_page, the report should highlight the pod which is building the current job. (more information:gocd/gocd#4091 (comment) )

Agent auto-register timeout at Elastic Agent Profiles level

Hi Guys

Thank you for the awesome plugin.
Can I request a feature - for having the Agent auto-register timeout
at Elastic Agent Profiles.
We are planning to have different agents - java, node ,Golang etc
and each of the agents could have different up time.

I can contribute if needed

Thanks
Saif

Show error when using invalid pod spec

When trying to create an agent with an invalid pod.yaml, GoCD only shows 'waiting for agent'. Checking the logs on the server, it shows

2019-01-11 14:27:01,002 ERROR [155@MessageListener for CreateAgentListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:128 [plugin-cd.go.contrib.elasticagent.kubernetes] - Failed to handle request cd.go.elastic-agent.create-agent
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create]  for kind: [Pod]  with name: [null]  in namespace: [gocd]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:363)
	at cd.go.contrib.elasticagent.KubernetesInstanceFactory.createKubernetesPod(KubernetesInstanceFactory.java:133)
Caused by: java.net.SocketTimeoutException: timeout
2019-01-11 14:27:01,002 ERROR [155@MessageListener for CreateAgentListener] JMSMessageListenerAdapter:77 - Exception thrown in message handling by listener com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener@2ad4f87c
java.lang.RuntimeException: Interaction with plugin with id 'cd.go.contrib.elasticagent.kubernetes' implementing 'elastic-agent' extension failed while requesting for 'cd.go.elastic-agent.create-agent'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.create-agent']
	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:57)
	at com.thoughtworks.go.plugin.access.elastic.v3.ElasticAgentExtensionV3.createAgent(ElasticAgentExtensionV3.java:101)

Without any indication as to what the problem is.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: xxxx
spec:
  containers:
    - name: agent
      image: xxxx
      resources:
        limits:
          cpu: 1
          memory: 4096Mi
        requests:
          cpu: 1
          memory: 8096Mi

Should show the following error message which it would get from kubernetes, but this is not visible anywhere in GoCD.

The Pod "xxxx is invalid: spec.containers[0].resources.requests: Invalid value: "8096Mi": must be less than or equal to memory limit

A possible reason for this is that it seems to take kubernetes 30 secs to return the error (when using kubectl create -f pod.yaml) which is longer than the 10 second timeout in GoCD.

Kubernetes: v1.10.9-gke.0
GoCD: 18.6.0
Kubernetes Elastic Agent Plugin: 1.0.1

[Question] Does this plugin support AWS EKS?

Due to the special configuration required to use EKS I'm not able to figure out how to configure this plugin properly

It might not be compatible at all...?

Agents are killed after 10 minutes

With GoCd Server 18.11 and Kubernetes plugin 2.0.0, agents are killed after 10 minutes, even when the job has not finished execution yet.

I tried to set up a timeout of 60 minutes on the Kubernetes Plugin Configuration but it does not seems to have any effect .

Elastic Agent Reuse

Hi Guys,

I have noticed that the elastic agents do not get reused by the builds .
The plugin spins up another pod having the agent even if there is an idle one if I start the build

I am wondering if this is intentional.

I can contribute a change for this if its a worthwhile feature change.

Thanks,
Saif

Wrong size parsing in utils/Size.java

I see this code in module https://github.com/gocd/kubernetes-elastic-agents/blob/master/src/main/java/cd/go/contrib/elasticagent/utils/Size.java

public static Size parse(String size) {
        final Matcher matcher = SIZE_PATTERN.matcher(size);
        checkArgument(matcher.matches(), "Invalid size: " + size);

        final double count = Double.parseDouble(matcher.group(1));
        final SizeUnit unit = SUFFIXES.get(matcher.group(2));
        if (unit == null) {
            throw new IllegalArgumentException("Invalid size: " + size + ". Wrong size unit");
        }

        return new Size(count, unit);
    }

but kubectl describe node <node_name> could returns values without measure unit:

Allocatable:
  cpu:                3700m
  ephemeral-storage:  91396653315
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             7503080192
  pods:               110

And when I try to register a new cluster profile I get this error:

Invalid size: 7503080192. Wrong size unit

Performance impact of K8s plugin on the Go Server

During the performance run benchmarking on latest physical hardware, it was noticed that having K8s Elastic agent plugin used in the setup for performance test the time between a job moving from scheduled to assigned is increased.

Configuration of performance environment WITHOUT K8s:

Go Version : 18.2.0 and 18.3.0
Total pipelines: 1000
Pipelines run on static agents: 750
Pipelines run on ECS elastic agents: 250
Number of Static agents: 70
Server Max Mem: 10g
Server Min mem: 8g
Max EC2 instances: 4 c3.2xlarge

With this setup when the performance run was done for 12 hour period the average time for job instance to move fromscheduled to assgined was about a minute. So total run of each pipelines were approximately 200

Configuration of performance environment WITH K8s:

Go Version : 18.2.0 and 18.3.0
K8s Elastic agent version: 1.0.0-94
Total pipelines: 1100
Pipelines run on static agents: 750
Pipelines run on ECS elastic agents: 250
Pipelines run on K8s elastic agents: 100
Number of Static agents: 70
Server Max Mem: 10g
Server Min mem: 8g
Max EC2 instances: 4 c3.2xlarge
K8s Cluster nodes: 3 nodes with 4vCPU and 15GB memory

With this setup when the performance run was done for 12 hour period the average time for job instance to move fromscheduled to assgined was about 10 minutes. So total run of each pipelines were approximately only 80

Why there is a increase in delay for job instances to move from scheduled to assigned need to be investigated.

More data are captured on the server box(db backup, logs). Whoever is picking this issue, please message me, can pair to look into the captured data

Unusual error breaking kubernetes connection

GoCD new install with helm
GoCD 19.9.0, Plugin 3.2.0-187 / 3.3.0-191
While trying to generate a status report on the elastic agent:

jvm 1 | 2019-10-14 16:22:37,097 INFO [qtp309906614-36] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:72 [plugin-cd.go.contrib.elasticagent.kubernetes] - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
jvm 1 | 2019-10-14 16:22:37,100 INFO [qtp309906614-36] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:72 [plugin-cd.go.contrib.elasticagent.kubernetes] - [status-report] Generating status report.
jvm 1 | 2019-10-14 16:22:37,205 ERROR [qtp309906614-36] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:127 [plugin-cd.go.contrib.elasticagent.kubernetes] - Error while generating status report: For input string: "6694Mi"
jvm 1 | java.lang.NumberFormatException: For input string: "6694Mi"
jvm 1 | at java.base/java.lang.NumberFormatException.forInputString(Unknown Source)
jvm 1 | at java.base/java.lang.Long.parseLong(Unknown Source)
jvm 1 | at java.base/java.lang.Long.valueOf(Unknown Source)
jvm 1 | at cd.go.contrib.elasticagent.model.KubernetesNode.(KubernetesNode.java:56)
jvm 1 | at cd.go.contrib.elasticagent.model.KubernetesCluster.lambda$new$0(KubernetesCluster.java:37)
jvm 1 | at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
jvm 1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
jvm 1 | at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
jvm 1 | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
jvm 1 | at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
jvm 1 | at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)

Thanks for any idea how to fix this, been driving me up the wall for a few days now :(

Show Elastic Agent Starting state on agent status page

Issue Type

Bug Report

Summary

When kubernetes pod for the elastic agent is starting up (let's say pulling the image), accessing the agent status report page throws an error with message "ContainerCreating".

Possible Fix

Do not throw an error in case of the pending pod.
Show appropriate message depending on the state of the pod. Example: Starting Elastic Agent..

Does the elastic agent support kubernetes PVC volume?

Here is my pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-name-prefix-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: gocd/gocd-agent-docker-dind:v18.12.0
      securityContext:
        privileged: true
      volumeMounts:
      - name: ssh-key-secret
        mountPath: /home/go/.ssh
        readOnly: true
  volumes:
  - name: ssh-key-secret
    secret:
      secretName: ssh-key-secret

when I change volumes.secret to persistentVolumeClaim and use correct claimName.Then I assign the elastic agent to a job and I trigger the pipeline manually. But after a long time, the pod doesn't start and the job is still waiting...

Job not being scheduled to elastic agent

Hello guys, first thanks for the plugin!!!

So I am using the GoCD Chart to deploy my cluster and also elastic actors, I've followed the steps to configure the plugin and it was working with the demo that comes with the chart that uses elastic agents

So I created the following profile, name jdk-8-mvn-helm which is the bundle I need.

`apiVersion: v1
kind: Pod
metadata:
name: jdk8-{{ POD_POSTFIX }}
labels:
app: web
spec:
containers:

name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
image: travix/gocd-agent-gcloud-jdk-8:18.6.0
securityContext:
privileged: true
volumeMounts:
- name: ssh-secrets
  mountPath: /var/go/.ssh
  readOnly: false
- name: m2
  mountPath: /var/go/.m2/repository
  subPath: repository
  readOnly: false
- name: m2-config-volume
  mountPath: /var/go/.m2
  readOnly: false
  volumes:
name: m2-config-volume
configMap:
name: m2-config
items:
- key: settings.xml
path: settings.xml
mode: 511
name: m2
emptyDir: {}
name: ssh-secrets
secret:
secretName: gocd-ssh
defaultMode: 384`

And added it to my Job, so it now shows Elastic Profile Id as jdk-8-mvn-helm

When it is started, I see the creation of the container in my minikube, but my job never runs and just hang forever in an unassigned state.

In the server logs I get this exception
2018-07-19 14:00:04,674 WARN [qtp32863545-29] HttpChannel:568 - /go java.lang.IllegalStateException: Committed at org.eclipse.jetty.server.HttpChannel.resetBuffer(HttpChannel.java:841) at org.eclipse.jetty.server.HttpOutput$Interceptor.resetBuffer(HttpOutput.java:116) at org.eclipse.jetty.server.HttpOutput.resetBuffer(HttpOutput.java:928) at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1312) at org.eclipse.jetty.server.Response.sendRedirect(Response.java:720) at org.eclipse.jetty.server.Response.sendRedirect(Response.java:729) at org.eclipse.jetty.server.handler.ContextHandler.checkContext(ContextHandler.java:1048) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1100) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:530) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 2018-07-19 14:00:05,047 WARN [qtp32863545-25] HttpChannel:568 - /go java.lang.IllegalStateException: Committed at org.eclipse.jetty.server.HttpChannel.resetBuffer(HttpChannel.java:841) at org.eclipse.jetty.server.HttpOutput$Interceptor.resetBuffer(HttpOutput.java:116) at org.eclipse.jetty.server.HttpOutput.resetBuffer(HttpOutput.java:928) at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1312) at org.eclipse.jetty.server.Response.sendRedirect(Response.java:720) at org.eclipse.jetty.server.Response.sendRedirect(Response.java:729) at org.eclipse.jetty.server.handler.ContextHandler.checkContext(ContextHandler.java:1048) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1100) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:530) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)

here is the log before this exception on a manual start

`2018-07-19 13:59:57,124 INFO [qtp32863545-23] PipelineTriggerService:44 - [Pipeline Schedule] [Requested] Manual trigger of pipeline 'StocksCash' requested by anonymous
2018-07-19 13:59:57,127 INFO [qtp32863545-23] PipelineTriggerService:47 - [Pipeline Schedule] [Accepted] Manual trigger of pipeline 'StocksCash' accepted for user anonymous
2018-07-19 13:59:57,131 INFO [qtp32863545-23] PipelineTriggerService:49 - [Pipeline Schedule] [Processed] Manual trigger of pipeline 'StocksCash' processed with result 'com.thoughtworks.go.serverhealth.ServerHealthState@1ede2708[healthStateLevel=OK,type=<HealthStateType ARTIFACTS_DISK_FULL LogScope[GLOBAL, scope=GLOBAL]>,message=,description=,expiryTime=,timestamp=Thu Jul 19 13:59:57 GMT 2018]'
2018-07-19 13:59:59,884 INFO [ThreadPoolTaskScheduler-10] ScheduleService:155 - [Pipeline Schedule] Scheduling pipeline StocksCash with build cause [ManualForcedBuildCause: Forced by anonymous]
2018-07-19 13:59:59,889 INFO [ThreadPoolTaskScheduler-10] PipelineRepository:86 - Start updating pipeline timeline
2018-07-19 13:59:59,893 INFO [ThreadPoolTaskScheduler-10] PipelineRepository:92 - Pipeline timeline updated
2018-07-19 14:00:00,569 INFO [152@MessageListener for ServerPingListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:73 [plugin-cd.go.contrib.elasticagent.kubernetes] - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 4.
2018-07-19 14:00:03,749 INFO [149@MessageListener for CreateAgentListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:73 [plugin-cd.go.contrib.elasticagent.kubernetes] - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 4.
2018-07-19 14:00:03,759 INFO [149@MessageListener for CreateAgentListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:73 [plugin-cd.go.contrib.elasticagent.kubernetes] - [Create Agent] Creating K8s pod with spec: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={Privileged=true, MaxCPU=, Environment=GO_SERVER_URL=https://10.98.122.134:8154/go
GO_EA_SERVER_URL=https://10.98.122.134:8154/go, Elastic-Agent-Job-Identifier={"pipeline_name":"StocksCash","pipeline_counter":34,"pipeline_label":"34","stage_name":"Build","stage_counter":"1","job_name":"Test","job_id":74}, Image=travix/gocd-agent-gcloud-jdk-8:18.6.0, PodConfiguration=apiVersion: v1
kind: Pod
metadata:
name: jdk8-{{ POD_POSTFIX }}
labels:
app: web
spec:
containers:

name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
image: travix/gocd-agent-gcloud-jdk-8:18.6.0
securityContext:
privileged: true
volumeMounts:
- name: ssh-secrets
  mountPath: /var/go/.ssh
  readOnly: false
- name: m2
  mountPath: /var/go/.m2/repository
  subPath: repository
  readOnly: false
- name: m2-config-volume
  mountPath: /var/go/.m2
  readOnly: false
  volumes:
name: m2-config-volume
configMap:
name: m2-config
items:
- key: settings.xml
path: settings.xml
mode: 511
name: m2
emptyDir: {}
name: ssh-secrets
secret:
secretName: gocd-ssh
defaultMode: 384, SpecifiedUsingPodConfiguration=true, MaxMemory=512M}, clusterName=null, creationTimestamp=2018-07-19T14:00:03Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={app=web, Elastic-Agent-Created-By=cd.go.contrib.elasticagent.kubernetes, Elastic-Agent-Job-Id=74, kind=kubernetes-elastic-agent, Elastic-Agent-Environment-Name=Local}, name=jdk8-7026b4f2-bd3e-429d-ad5e-347e2faefc24, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[], env=[EnvVar(name=GO_EA_SERVER_URL, value=https://gocd-server:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_SERVER_URL, value=https://10.98.122.134:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_SERVER_URL, value=https://10.98.122.134:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_KEY, value=2f490bd3-7efe-4b82-aa5a-9ac02d428e3a, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ENVIRONMENT, value=Local, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_AGENT_ID, value=jdk8-7026b4f2-bd3e-429d-ad5e-347e2faefc24, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_PLUGIN_ID, value=cd.go.contrib.elasticagent.kubernetes, valueFrom=null, additionalProperties={})], envFrom=[], image=travix/gocd-agent-gcloud-jdk-8:18.6.0, imagePullPolicy=null, lifecycle=null, livenessProbe=null, name=gocd-agent-container-ef11c349-256e-4dd4-943e-5e5374bec29c, ports=[], readinessProbe=null, resources=null, securityContext=SecurityContext(capabilities=null, privileged=true, readOnlyRootFilesystem=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=null, volumeMounts=[VolumeMount(mountPath=/var/go/.ssh, name=ssh-secrets, readOnly=false, subPath=null, additionalProperties={}), VolumeMount(mountPath=/var/go/.m2/repository, name=m2, readOnly=false, subPath=repository, additionalProperties={}), VolumeMount(mountPath=/var/go/.m2, name=m2-config-volume, readOnly=false, subPath=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsPolicy=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector=null, restartPolicy=null, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=ConfigMapVolumeSource(defaultMode=null, items=[KeyToPath(key=settings.xml, mode=511, path=settings.xml, additionalProperties={})], name=m2-config, optional=null, additionalProperties={}), downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=m2-config-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=null, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=m2, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=ssh-secrets, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=384, items=[], optional=null, secretName=gocd-ssh, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={}).
2018-07-19 14:00:03,777 INFO [148@MessageListener for CreateAgentListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:73 [plugin-cd.go.contrib.elasticagent.kubernetes] - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 5.
2018-07-19 14:00:03,788 INFO [148@MessageListener for CreateAgentListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:73 [plugin-cd.go.contrib.elasticagent.kubernetes] - [Create Agent] Creating K8s pod with spec: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={Privileged=true, MaxCPU=, Environment=GO_SERVER_URL=https://10.98.122.134:8154/go
GO_EA_SERVER_URL=https://10.98.122.134:8154/go, Elastic-Agent-Job-Identifier={"pipeline_name":"StocksCash","pipeline_counter":34,"pipeline_label":"34","stage_name":"Build","stage_counter":"1","job_name":"Build","job_id":73}, Image=travix/gocd-agent-gcloud-jdk-8:18.6.0, PodConfiguration=apiVersion: v1
kind: Pod
metadata:
name: jdk8-{{ POD_POSTFIX }}
labels:
app: web
spec:
containers:
name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
image: travix/gocd-agent-gcloud-jdk-8:18.6.0
securityContext:
privileged: true
volumeMounts:
- name: ssh-secrets
  mountPath: /var/go/.ssh
  readOnly: false
- name: m2
  mountPath: /var/go/.m2/repository
  subPath: repository
  readOnly: false
- name: m2-config-volume
  mountPath: /var/go/.m2
  readOnly: false
  volumes:
name: m2-config-volume
configMap:
name: m2-config
items:
- key: settings.xml
path: settings.xml
mode: 511
name: m2
emptyDir: {}
name: ssh-secrets
secret:
secretName: gocd-ssh
defaultMode: 384, SpecifiedUsingPodConfiguration=true, MaxMemory=512M}, clusterName=null, creationTimestamp=2018-07-19T14:00:03Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={app=web, Elastic-Agent-Created-By=cd.go.contrib.elasticagent.kubernetes, Elastic-Agent-Job-Id=73, kind=kubernetes-elastic-agent, Elastic-Agent-Environment-Name=Local}, name=jdk8-8fdb7961-8346-4113-a502-78f3a9ed64a7, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[], env=[EnvVar(name=GO_EA_SERVER_URL, value=https://gocd-server:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_SERVER_URL, value=https://10.98.122.134:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_SERVER_URL, value=https://10.98.122.134:8154/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_KEY, value=2f490bd3-7efe-4b82-aa5a-9ac02d428e3a, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ENVIRONMENT, value=Local, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_AGENT_ID, value=jdk8-8fdb7961-8346-4113-a502-78f3a9ed64a7, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_PLUGIN_ID, value=cd.go.contrib.elasticagent.kubernetes, valueFrom=null, additionalProperties={})], envFrom=[], image=travix/gocd-agent-gcloud-jdk-8:18.6.0, imagePullPolicy=null, lifecycle=null, livenessProbe=null, name=gocd-agent-container-f3dfb3b4-a78d-427d-b185-6a3af58a476d, ports=[], readinessProbe=null, resources=null, securityContext=SecurityContext(capabilities=null, privileged=true, readOnlyRootFilesystem=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=null, volumeMounts=[VolumeMount(mountPath=/var/go/.ssh, name=ssh-secrets, readOnly=false, subPath=null, additionalProperties={}), VolumeMount(mountPath=/var/go/.m2/repository, name=m2, readOnly=false, subPath=repository, additionalProperties={}), VolumeMount(mountPath=/var/go/.m2, name=m2-config-volume, readOnly=false, subPath=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsPolicy=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector=null, restartPolicy=null, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=ConfigMapVolumeSource(defaultMode=null, items=[KeyToPath(key=settings.xml, mode=511, path=settings.xml, additionalProperties={})], name=m2-config, optional=null, additionalProperties={}), downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=m2-config-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=null, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=m2, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=ssh-secrets, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=384, items=[], optional=null, secretName=gocd-ssh, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={}).
`

Failed to validate namespace existence: default

When trying to configure the plugin it fails with the following message:

Failed to validate namespace existence: default Please check plugin log for more detail.

The only thing I can find in the go.cd server logs is the following:

ERROR [128@MessageListener for ServerPingListener] JMSMessageListenerAdapter:77 - Exception thrown in message handling by listener com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener@329b4cb2
 java.lang.RuntimeException: Interaction with plugin with id 'cd.go.contrib.elasticagent.kubernetes' implementing 'elastic-agent' extension failed while requesting for 'cd.go.elastic-agent.server-ping'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping']
 	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:57)
 	at com.thoughtworks.go.plugin.access.elastic.v3.ElasticAgentExtensionV3.serverPing(ElasticAgentExtensionV3.java:111)
 	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:82)
 	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:50)
 	at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:32)
 	at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:22)
 	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:73)
 	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:53)
 	at java.lang.Thread.run(Thread.java:748)
 Caused by: java.lang.RuntimeException: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping'
 	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:55)
 	... 8 common frames omitted

I've set up RBAC role, binding and service account with the following template and use https://kubernetes.default.svc.cluster.local as cluster url. I've copied the service account token from /var/run/secrets/kubernetes.io/serviceaccount/token and ca certificate data from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: server-role
 namespace: ${NAMESPACE}
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - get
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - pods/logs
  verbs:
  - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: server-role-binding
  namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: server-role
subjects:
- namespace: ${NAMESPACE}
  kind: ServiceAccount
  name: server
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: server
 namespace: ${NAMESPACE}

Is there any other place to find the actual logs from the plugin?

Use smarter defaults when server runs in Kubernetes as well

If go.cd server itself runs in Kubernetes a number of currently configurable fields can be set in a smarter way.

Cluster URL
Either use https://kubernetes.default.svc.cluster.local or https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} by default

Token
Default to loading from /var/run/secrets/kubernetes.io/serviceaccount/token

Ca.crt
Default to loading from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Namespace
Default to loading from /var/run/secrets/kubernetes.io/serviceaccount/namespace to run them in the same namespaces as the server

Validate Kubernetes cluster credentials

as part of the validate-plugin-settings call, the plugin should:

Validate cluster URL, username, password, and cluster ca certificate.
Validate the kubernetes cluster is accessible using provided user's credentials.
Validate the rbac authorization rules for the provided user to make sure that the user has the minimum set of permissions on the cluster to manipulate the k8s resources.

KeyUsage does not allow digital signatures

Hi guys:

Currently, we met a problem cause configure the Elastic Agent Plugin failed.

Environment

SLES 12 SP4
Kubernetes v1.11.2
GoCD Server URL: https://172.21.3.147:8514
K8S API Server URL: https://172.21.3.147:6443

Deployment

GoCD v18.10.0
GoCD Server has been deployed by using RPMs
GoCD Agent has been deployed by using GoCD helm chars（agent only mode）

GoCD Agent Deployments

ServiceAccount

# kubectl get sa -n gocd
NAME       SECRETS   AGE
default    1         1h
gocd-app   1         1h

ClusterRole

# kubectl get clusterrole gocd-app -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: 2018-11-20T06:13:45Z
  labels:
    app: gocd
    chart: gocd-1.5.5
    heritage: Tiller
    release: gocd-app
  name: gocd-app
  resourceVersion: "16937324"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/gocd-app
  uid: 6f97d2af-ec8b-11e8-b41c-0017fa00e2a6
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
  - get

ClusterRoleBinding

# kubectl get clusterrolebinding gocd-app -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: 2018-11-20T06:13:45Z
  labels:
    app: gocd
    chart: gocd-1.5.5
    heritage: Tiller
    release: gocd-app
  name: gocd-app
  resourceVersion: "16937326"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/gocd-app
  uid: 6f9af79a-ec8b-11e8-b41c-0017fa00e2a6
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gocd-app
subjects:
- kind: ServiceAccount
  name: gocd-app
  namespace: gocd

Deployments

# kubectl get all -n gocd
NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gocd-app-agent   0         0         0            0           1h

NAME                                        DESIRED   CURRENT   READY     AGE
replicaset.apps/gocd-app-agent-6f7cf7744b   0         0         0         1h

Logs

2018-11-20 07:35:35,951 ERROR [qtp1166807841-31] KubernetesPlugin:128 - Failed validation of plugin settings. The reasons could be - Cluster Url is configured incorrectly or the service account token might not have enough permissions to list namespaces or incorrect CA certificate.
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Namespace]  with name: [null]  in namespace: [null]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
.
.
.

Caused by: sun.security.validator.ValidatorException: KeyUsage does not allow digital signatures
	at sun.security.validator.EndEntityChecker.checkTLSServer(EndEntityChecker.java:271)
	at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:143)
	at sun.security.validator.Validator.validate(Validator.java:274)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
	... 566 common frames omitted

Any idea about this issue?

Thanks in advance.

Elastic agents don't get assigned

Issue Type

Bug Report

Summary

The elastic agents don't get created and previous agents don't get delete after deployment.

Environment

Basic environment details

Go Version: GoCD Version: 19.7.0 (9567-727ea9db824eb6971170ac2a886ff1072ff5a235).
JAVA Version: 12.0.1
OS: Linux 4.14.123-111.109.amzn2.x86_64
Browser vendor and version (if relevant): Chrome

Additional Environment Details

Steps to Reproduce

I'm not sure how to reproduce this issue, however trying a sleep for longer time may reproduce

Create a pipeline, with sleep for longer period
execute the pipeline in an elastic agent and let it run
see if the above said pipeline gets succeeded and agent deleted.
if the agent isn't delete then execute the pipeline again and see if new agents are created and auto assigned.

Expected Results

The Elastic agents are created assigned to pipelines to do the job.

Actual Results

Few previously run pipelines have left elastic agents behind, which I think is causing GoCD Server to not to create new agents and assign, I may be wrong.
After Restarting the server it works fine but it happens quite often.

One more thing I would like highlight is that GoCD Server Pod continues to run in behind however the ELB gives a http 503 error, The ELB health check gets passed and kubectl logs have below errors, but after sometimes the service becomes available

jvm 1    | 2020-03-23 09:26:51,430 ERROR [Thread-81] DefaultPluginManager:154 - This is an invalid request type :go.plugin-settings.get-configuration
jvm 1    | 2020-03-23 09:26:52,154 ERROR [Thread-81] p.c.g.a.l.c.g.a.l.LdapPlugin:127 [plugin-cd.go.authentication.ldap] - Error while executing request go.plugin-settings.get-configuration

Possible Fix

Restart GoCD Server pod

Log snippets

Code snippets/Screenshots

Any other info

Only display GoCD agent pods on status report page

Currently, the status page displays all pods running on the K8S cluster. Please filter this list to include only GoCD agent pods.

Broken help link

Here:

kubernetes-elastic-agents/src/main/resources/profile.template.html

Line 210 in 8d29fa5

 <a href="https://kubernetes-v1-4.github.io/docs/user-guide/pods/multi-container/#pod-configuration-file" 

Do not create extra GoCD Agents

Issue Type

Bug Report

Summary

Kubernetes Elastic Agent Plugin brings up more pods than necessary.

Description

On every create agent request, kubernetes-elastic-agent plugin brings up a new pod with the gocd-agent container on it.

If the plugin takes more than time (than job starvation timeout) to bring up a GoCD agent, the server will send another create agent request for the same job. In such cases, the plugin will bring up extra pods which are not necessary.

For each create agent request, plugin receives only elastic profile configurations, that might not suffice to decide whether the request is a for a newly scheduled job with the same elastic-agent-id or another request for a starving job.

Note: Excessive pods will be lying around for time specified as agent-auto-register timeout value. After this timeout, these pods will be cleaned by the plugin.

Retain pods for failed jobs a longer period

2.0.0 introduced terminating a pod immediately upon job completion which has proved useful when running many different pipelines with elastic agents, as nodes' resources are freed up quicker which reduces the chances of nodes have to autoscale.

This has introduced some extra difficulties when troubleshooting failed jobs though since the pods are cleaned up immediately it leaves nothing left to debug. It could be useful to set an alternate grace period for pods whose assigned jobs have failed, to give the option to look around the pod.

Failed to handle request cd.go.elastic-agent.create-agent' - 500 Response

Issue Type

Bug Report

Summary

When I kick off a pipeline build, an agent pod is not created by the elastic agent plugin. I checked the logs of the gocd-pod and see a Java runtime error in there:

java.lang.RuntimeException: Interaction with plugin with id 'cd.go.contrib.elasticagent.kubernetes' implementing 'elastic-agent' extension failed while requesting for 'cd.go.elastic-agent.create-agent'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.create-agent']
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:57)
        at com.thoughtworks.go.plugin.access.elastic.v4.ElasticAgentExtensionV4.createAgent(ElasticAgentExtensionV4.java:101)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:72)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:41)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:73)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:53)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failedto handle request cd.go.elastic-agent.create-agent'
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:55)

Environment

Basic environment details

Set up elastic agent plugin according to the documentation on install:

Put in the GoCD server address
Put in the correct cluster address
Provided the security token
Provided the CA certificate without the BEGIN and END Certificate tags
Put in the correct namespace (gocd)

Created a basic agent pod using a YAML file, agent is working fine. Tested locally.

Expected Results

Elastic agents pods should be created in gocd namespace

Actual Results

The create agent request fails.

Any other info

Odd thing is that the agent pods can launch when I leave the default credentials in there, but then they don't actually execute any of the tasks or jobs in the pipeline. So I saw that they were working, but when I changed the elastic agent plugin configuration, I get the Java runtime error above.

Agent status show IndexOutOfBoundsException when container inside pod is dies off.

Agent status report shows IndexOutOfBoundsException when container running inside the pod is terminated or killed because of resource constraints. Here is a stacktrace -

2018-03-15 17:43:10,442 ERROR [qtp212921632-192] KubernetesPlugin:127 - Error while generating status report: Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:657)
        at java.util.ArrayList.get(ArrayList.java:433)
        at cd.go.contrib.elasticagent.model.reports.agent.KubernetesElasticAgent.fromPod(KubernetesElasticAgent.java:44)
        at cd.go.contrib.elasticagent.executors.AgentStatusReportExecutor.execute(AgentStatusReportExecutor.java:71)
        at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:87)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:167)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:162)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:316)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:246)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:162)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:48)
        at com.thoughtworks.go.plugin.access.elastic.v3.ElasticAgentExtensionV3.getAgentStatusReport(ElasticAgentExtensionV3.java:141)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.getAgentStatusReport(ElasticAgentExtension.java:110)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.getAgentStatusReport(ElasticAgentPluginRegistry.java:70)
        at com.thoughtworks.go.server.service.ElasticAgentPluginService.getAgentStatusReport(ElasticAgentPluginService.java:214)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:481)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:336)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:68)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:236)
        at org.jruby.ast.CallThreeArgNode.interpret(CallThreeArgNode.java:61)
        at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
        at org.jruby.RubyClass.finvoke(RubyClass.java:524)
        at org.jruby.RubyBasicObject.send19(RubyBasicObject.java:1562)
        at org.jruby.RubyKernel.send19(RubyKernel.java:2207)
        at org.jruby.RubyKernel$INVOKER$s$send19.call(RubyKernel$INVOKER$s$send19.gen)
        at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:86)

volumeMounts in PodConfiguration are being overridden

Hi everyone,

in order to access private git repositories through SSH I was trying to mount a Secret or ConfigMap under /home/go/.ssh which contains everything that is required to SSH into our GitLab repositories (private keypair, prefilled known_hosts).

Basically what I did is the same what the images/profile-with-pod-yaml.png image is showing on the install.md page in this repo, but with a different directory. Is this intentional behaviour on the .ssh directory or am I facing a bug? (yes the secret exists and contains all data, tested it with a busybox mounting the directory in the same way)

Here is my pod configuration from the elastic agent profile:

apiVersion: v1
kind: Pod
metadata:
  name: pod-name-prefix-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: {{ GOCD_AGENT_IMAGE }}:{{ LATEST_VERSION }}
      securityContext:
        privileged: true
      volumeMounts:
        - name: ssh
          mountPath: /home/go/.ssh
          readOnly: true
  volumes:
    - name: ssh
      secret:
        secretName: gocd-ssh-key

Output of kubectl describe -n infra pod/k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a, which is the elastic agent pod. As you can see no extra volume was mounted:

Name:         k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
Namespace:    infra
Node:         MY-NODE/MY-IP
Start Time:   Thu, 28 Jun 2018 15:08:10 +0200
Labels:       Elastic-Agent-Created-By=cd.go.contrib.elasticagent.kubernetes
              Elastic-Agent-Job-Id=7
              kind=kubernetes-elastic-agent
Annotations:  Elastic-Agent-Job-Identifier={"pipeline_name":"hello_world_ssh","pipeline_counter":1,"pipeline_label":"1","stage_name":"default_stage","stage_counter":"1","job_name":"default_job","job_id":7}
              Environment=
              Image=gocd/gocd-agent-docker-dind:v18.6.0
              MaxCPU=
              MaxMemory=
              PodConfiguration=apiVersion: v1
kind: Pod
metadata:
  name: pod-name-prefix-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      ...
         Privileged=true
         SpecifiedUsingPodConfiguration=false
Status:  Running
IP:      100.96.0.22
Containers:
  k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a:
    Container ID:   docker://179c413dec10dd689e19af2794acba14c47a998fbb322628d5809501ffd4fe14
    Image:          gocd/gocd-agent-docker-dind:v18.6.0
    Image ID:       docker-pullable://gocd/gocd-agent-docker-dind@sha256:90521ed917de7c6535c072eae8432870e7d9004e0f08100a0dc7aa01b01107ac
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 28 Jun 2018 15:08:23 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      GO_EA_SERVER_URL:                       https://gocd-server:8154/go
      GO_EA_AUTO_REGISTER_KEY:                1df623f2-601f-45ec-8578-711a2ca9ba2a
      GO_EA_AUTO_REGISTER_ELASTIC_AGENT_ID:   k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
      GO_EA_AUTO_REGISTER_ELASTIC_PLUGIN_ID:  cd.go.contrib.elasticagent.kubernetes
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ms8jc (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  default-token-ms8jc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ms8jc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                                     Message
  ----    ------                 ----  ----                                                     -------
  Normal  Scheduled              9m    default-scheduler                                        Successfully assigned k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a to MY-NODE
  Normal  SuccessfulMountVolume  9m    kubelet, MY-NODE  MountVolume.SetUp succeeded for volume "default-token-ms8jc"
  Normal  Pulling                9m    kubelet, MY-NODE  pulling image "gocd/gocd-agent-docker-dind:v18.6.0"
  Normal  Pulled                 9m    kubelet, MY-NODE  Successfully pulled image "gocd/gocd-agent-docker-dind:v18.6.0"
  Normal  Created                9m    kubelet, MY-NODE  Created container
  Normal  Started                9m    kubelet, MY-NODE  Started container

And the output of kubectl get po -n infra k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a -o yaml, which correctly outputs my pod configuration template in the annotation, but is missing the actual mount in the spec down below:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    Elastic-Agent-Job-Identifier: '{"pipeline_name":"hello_world_ssh","pipeline_counter":1,"pipeline_label":"1","stage_name":"default_stage","stage_counter":"1","job_name":"default_job","job_id":7}'
    Environment: ""
    Image: gocd/gocd-agent-docker-dind:v18.6.0
    MaxCPU: ""
    MaxMemory: ""
    PodConfiguration: |-
      apiVersion: v1
      kind: Pod
      metadata:
        name: pod-name-prefix-{{ POD_POSTFIX }}
        labels:
          app: web
      spec:
        containers:
          - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
            image: {{ GOCD_AGENT_IMAGE }}:{{ LATEST_VERSION }}
            securityContext:
              privileged: true
            volumeMounts:
              - name: ssh
                mountPath: /home/go/.ssh
                readOnly: true
        volumes:
          - name: ssh
            secret:
              secretName: gocd-ssh-key
    Privileged: "true"
    SpecifiedUsingPodConfiguration: "false"
  creationTimestamp: 2018-06-28T13:08:10Z
  labels:
    Elastic-Agent-Created-By: cd.go.contrib.elasticagent.kubernetes
    Elastic-Agent-Job-Id: "7"
    kind: kubernetes-elastic-agent
  name: k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
  namespace: infra
  resourceVersion: "2879120"
  selfLink: /api/v1/namespaces/infra/pods/k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
  uid: 4ea4b29d-7ad4-11e8-940f-02118ba64a3e
spec:
  containers:
  - env:
    - name: GO_EA_SERVER_URL
      value: https://gocd-server:8154/go
    - name: GO_EA_AUTO_REGISTER_KEY
      value: 1df623f2-601f-45ec-8578-711a2ca9ba2a
    - name: GO_EA_AUTO_REGISTER_ELASTIC_AGENT_ID
      value: k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
    - name: GO_EA_AUTO_REGISTER_ELASTIC_PLUGIN_ID
      value: cd.go.contrib.elasticagent.kubernetes
    image: gocd/gocd-agent-docker-dind:v18.6.0
    imagePullPolicy: IfNotPresent
    name: k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-ms8jc
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: MY-NODE
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-ms8jc
    secret:
      defaultMode: 420
      secretName: default-token-ms8jc
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-06-28T13:08:10Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-06-28T13:08:23Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-06-28T13:08:10Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://179c413dec10dd689e19af2794acba14c47a998fbb322628d5809501ffd4fe14
    image: gocd/gocd-agent-docker-dind:v18.6.0
    imageID: docker-pullable://gocd/gocd-agent-docker-dind@sha256:90521ed917de7c6535c072eae8432870e7d9004e0f08100a0dc7aa01b01107ac
    lastState: {}
    name: k8s-ea-77f0a914-7877-494f-9fb6-872044ab3b5a
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-06-28T13:08:23Z
  hostIP: MY-IP
  phase: Running
  podIP: 100.96.0.22
  qosClass: BestEffort
  startTime: 2018-06-28T13:08:10Z

cannot list all namespaces in the cluster

When trying to configure the plugin I get this error:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://mycluster:8443/api/v1/namespaces . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:myelasticagentsproject:myuseraccount" cannot list namespaces at the cluster scope: User "system:serviceaccount: myelasticagentsproject: myuseraccount" cannot list all namespaces in the cluster.

Is it a requirement for the service account to have access to all namespace in the target cluster ?
I'm trying to configure the plugin targeting a shared cluster where I do not have a sa with cluster reader access.

Provide ability to configure "max-idle-timeout" for an elastic agent pod

There should be a configuration option while configuring an elastic agent profile to control "max-idle-timeout" for an agent pod. Currently, this defaults to 10 mins. This would help in

Not have multiple agents show up when running multiple jobs and clutter the UI.
Idle agents tend to confuse users, in case they are around to pick new jobs. While the current behavior is one agent per job.

Provide an option to download a pod spec from a repository

Currently, the pod spec that one can configure for the k8s elastic agent profile is done using the text area. If an option to download the pod yaml (or json) from another location is provided, then users may use a pod spec that's been checked in somewhere. The reason someone would want to check in the pod spec is the same as having pipelines in config repos. Although, it might seem that the pod spec is not likely to change as often as pipelines that is needs versioning, this can still be useful especially when a platform supporting k8s provides an option to download the pod spec as yaml or json.

Additionally, this can be used for gocd to define some pod templates in a repository for people to use?

@sheroy , @arvindsv - WDYT?

Agents are killed after 10 minutes

GOCD: 20.3.0
kubernetes-elastic-agents-plugin: 3.7.0-217
Hi, I am following the setup in of k8s elastic agent in documentation, and I noticed that if the job runs more than 10 min the agent is killed even though the job was not finished.
I tried to setup a simple job that prints to console then sleeps for 30sec and the behaviour seems consistent all elastic agents are killed after 10 min mark.

This really affects us from migrating all projects to GOCD as most of them have a lifespan of more than 10 min.

Support for multiple clusters

I have a scenario where I need different gocd agents for different environments due to networking policies (as in, qa agents only see qa resources, same for staging and production). Is support for multiple clusters planned? Is there any way to support this scenario with the current plugin version?

Passing environment variables to Elastic Agents from templates

Hello guys,

I'm using GO.CD Elastic Agents and everything works fine except for passing environment variables from Templates.

When passing varibles from Elastic Agent configuration everything works but I would like to pass encrypted variables from my Template definition. Is there a way to pass environment varibles to Elastic Agent Pods from Templates?

Thanks in advance,

Ronal

EKS configuration

We are experiencing the same issue running GoCD and the Elastic Agents plugin under EKS as described in
#81

We have GoCD installed via helm and have configured:

aws-alb-ingress-controller
a service account in the gocd namespace with cluster-admin role bindings
external-dns via cloudflare

using the following values

server:
  ingress:
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internet-facing
      external-dns.alpha.kubernetes.io/hostname: [removed]
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/certificate-arn: [arn removed]
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    hosts:
    - [removed]

and settings

--set rbac.roleRef=cluster-admin --set serviceAccount.create=true --set serviceAccount.name=gocd --set server.persistence.existingClaim=gocd-pv-claim

Trying to save the plugin settings shows "Failed validation of plugin settings. The reasons could be - Cluster Url is configured incorrectly or the service account token might not have enough permissions to get namespaces or incorrect CA certificate.Please check the plugin log for more details."

and the logs show

2019-04-09 11:17:13,334 ERROR [156@MessageListener for ServerPingListener] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:128 [plugin-cd.go.contrib.elasticagent.kubernetes] - Failed to handle request cd.go.elastic-agent.server-ping
java.lang.RuntimeException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/api/v1/namespaces/gocd/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.

GoCD Server URL, Cluster URL, Namespace and Security token are set. Cluster ca certificate data is not.

Setting those attributes by following the steps in notes shows the same error.

Any help would be very welcome. Thank you in advance

Provide a support for private docker registry

Currently, the plugin is pulling docker images from hub.docker.com. Provide way to pull a docker image from private registry.

Certificate issue in the agent container for docker pulling from public repository

There is an issue about invalid certificate for docker pulling in the Agent pod container

!!|12:50:37.464 [go] Task: /bin/bash -c "docker pull php:7.2-fpm-alpine3.8"
&2|12:50:37.594 Error response from daemon: Get https://registry-1.docker.io/v2/: x509: certificate has expired or is not yet valid

But it works fine, if I run the docker container manually in the Node

$ docker run --privileged --rm -it gocd/gocd-agent-docker-dind:v19.7.0 sh
$ exec /usr/local/sbin/tini -g -- sh
/ $ /bin/bash -c "docker pull php:7.2-fpm-alpine3.8"
7.2-fpm-alpine3.8: Pulling from library/php
c87736221ed0: Pull complete

Stack:
GoCD - 19.7.0 installing through helm to k8s cluster
Kubernetes Elastic Agent Plugin - 3.0.0-156

based on gocd/gocd-agent-docker-dind:v19.7.0
In the agent container Docker version 19.03.1, build 74b1e89e8a

Question: Which rule is the plugin using when it choose the k8s node for the agent?

I'm asking this because all my agent pods are always scheduled on the same worker node.
If I run a pipeline which has a stage with 10 jobs, 10 agent are scheduled on the same worker.

Intermediate NullPointerException during create agent request

Issue Type

Bug Report

Summary

Kubernetes Elastic Agent plugin intermediately throwing java.lang.NullPointerException: null during create-agent request.

Log snippets

2019-09-05 13:33:33,633 ERROR [5221@MessageListener for ServerPingListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.server-ping
java.lang.NullPointerException: null
	at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:168)
	at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForCluster(KubernetesPlugin.java:139)
	at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForAllClusters(KubernetesPlugin.java:122)
	at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:90)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
	at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.serverPing(ElasticAgentExtensionV5.java:152)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:64)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:52)
	at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:31)
	at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:21)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
	at java.lang.Thread.run(Thread.java:748)

Getting status report for a non running agent leaves a huge stack trace in the log file

The plugin prints following error and stack trace in the plugin log:

2019-06-12 10:44:53,376 ERROR [qtp821513849-70757] KubernetesPlugin:128 - Error while generating status report: Pod is not running.
cd.go.contrib.elasticagent.reports.StatusReportGenerationException: Pod is not running.
	at cd.go.contrib.elasticagent.reports.StatusReportGenerationException.noRunningPod(StatusReportGenerationException.java:42)
	at cd.go.contrib.elasticagent.executors.AgentStatusReportExecutor.findPodUsingJobIdentifier(AgentStatusReportExecutor.java:87)
	at cd.go.contrib.elasticagent.executors.AgentStatusReportExecutor.execute(AgentStatusReportExecutor.java:65)
	at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:105)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:153)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:304)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:246)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:150)
	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
	at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.getAgentStatusReport(ElasticAgentExtensionV5.java:192)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.getAgentStatusReport(ElasticAgentExtension.java:104)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.getAgentStatusReport(ElasticAgentPluginRegistry.java:72)
	at com.thoughtworks.go.server.service.ElasticAgentPluginService.getAgentStatusReport(ElasticAgentPluginService.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:425)
	at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:292)
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:28)
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:90)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:301)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:82)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:522)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:103)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
	at org.jruby.RubyClass.finvoke(RubyClass.java:525)
	at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1752)
	at org.jruby.RubyKernel.send(RubyKernel.java:2182)
	at org.jruby.RubyKernel$INVOKER$s$send.call(RubyKernel$INVOKER$s$send.gen)
	at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:105)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.basic_implicit_render.invokeSuper2:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/basic_implicit_render.rb:6)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.basic_implicit_render.RUBY$method$send_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/basic_implicit_render.rb:6)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:72)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.invokeOther0:send_action(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:194)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:194)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rendering.invokeSuper6:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rendering.rb:30)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rendering.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rendering.rb:30)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.zSuperSplatArgs(IRRuntimeHelpers.java:1127)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.invokeSuper2:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:42)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.RUBY$block$process_action$1(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:42)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
	at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:85)
	at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:463)
	at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:119)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.callbacks.RUBY$method$run_callbacks$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/callbacks.rb:132)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:355)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:180)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.invokeOther6:run_callbacks(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:41)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:41)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rescue.invokeSuper0:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rescue.rb:22)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rescue.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rescue.rb:22)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.zSuperSplatArgs(IRRuntimeHelpers.java:1127)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.invokeSuper3:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:38)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.RUBY$block$process_action$1(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:38)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:459)
	at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:92)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.RUBY$block$instrument$1(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:459)
	at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:92)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.instrumenter.RUBY$method$instrument$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications/instrumenter.rb:23)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:111)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:209)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:218)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.invokeOther5:instrument(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.RUBY$method$instrument$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:111)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:209)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:218)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.invokeOther43:instrument(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:32)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:32)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.params_wrapper.invokeSuper20:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/params_wrapper.rb:256)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.params_wrapper.RUBY$method$process_action$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/params_wrapper.rb:256)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:72)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.invokeOther11:process_action(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:134)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.RUBY$method$process$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:134)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionview_minus_5_dot_2_dot_2_dot_1.lib.action_view.rendering.invokeSuper5:-unknown-super-target-(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionview-5.2.2.1/lib/action_view/rendering.rb:32)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionview_minus_5_dot_2_dot_2_dot_1.lib.action_view.rendering.RUBY$method$process$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionview-5.2.2.1/lib/action_view/rendering.rb:32)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:98)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:344)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.invokeOther2:process(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:191)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.RUBY$method$dispatch$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:191)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:127)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:219)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:388)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:232)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.invokeOther10:dispatch(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:252)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.RUBY$method$dispatch$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:252)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:127)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:219)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:388)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:232)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther0:dispatch(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:52)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$dispatch$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:52)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$dispatch$0$__VARARGS__(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther6:dispatch(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:34)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$serve$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:34)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.invokeOther27:serve(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:52)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.RUBY$block$serve$1(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:52)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.RubyArray.each(RubyArray.java:1801)
	at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:156)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.invokeOther37:each(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:35)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.RUBY$method$serve$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:35)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther5:serve(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:840)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:840)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.invokeOther63:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:39)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.RUBY$method$_call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:39)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.invokeOther1:_call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:17)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:17)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.tempfile_reaper.invokeOther8:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.tempfile_reaper.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.etag.invokeOther5:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.etag.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.conditional_get.invokeOther26:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:25)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.conditional_get.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:25)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.head.invokeOther5:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.head.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.http.content_security_policy.invokeOther2:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/http/content_security_policy.rb:18)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.http.content_security_policy.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/http/content_security_policy.rb:18)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at uri_3a_classloader_3a_.jruby.rack.session_store.invokeOther2:call(uri:classloader:/jruby/rack/session_store.rb:79)
	at uri_3a_classloader_3a_.jruby.rack.session_store.RUBY$method$context$0(uri:classloader:/jruby/rack/session_store.rb:79)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:98)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.session.abstract.id.invokeOther0:context(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.session.abstract.id.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.cookies.invokeOther2:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/cookies.rb:670)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.cookies.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/cookies.rb:670)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.invokeOther1:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:28)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.RUBY$block$call$1(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:28)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
	at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:85)
	at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:463)
	at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:119)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.callbacks.RUBY$method$run_callbacks$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/callbacks.rb:98)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:178)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.invokeOther5:run_callbacks(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:26)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:26)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.debug_exceptions.invokeOther2:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.debug_exceptions.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.show_exceptions.invokeOther2:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.show_exceptions.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.invokeOther11:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:38)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.RUBY$method$call_app$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:38)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:114)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:199)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.invokeOther10:call_app(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:28)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:28)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.remote_ip.invokeOther7:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/remote_ip.rb:81)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.remote_ip.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/remote_ip.rb:81)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.request_id.invokeOther7:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/request_id.rb:27)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.request_id.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/request_id.rb:27)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.method_override.invokeOther9:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.method_override.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.runtime.invokeOther2:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.runtime.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.cache.strategy.local_cache_middleware.invokeOther6:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.cache.strategy.local_cache_middleware.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.executor.invokeOther4:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/executor.rb:14)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.executor.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/executor.rb:14)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.static.invokeOther13:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/static.rb:127)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.static.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/static.rb:127)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.sendfile.invokeOther13:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.sendfile.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.engine.invokeOther3:call(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/engine.rb:524)
	at go_minus_working_minus_dir.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.engine.RUBY$method$call$0(/go-working-dir/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/engine.rb:524)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
	at uri_3a_classloader_3a_.rack.handler.servlet.invokeOther4:call(uri:classloader:/rack/handler/servlet.rb:22)
	at uri_3a_classloader_3a_.rack.handler.servlet.RUBY$method$call$0(uri:classloader:/rack/handler/servlet.rb:22)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.RubyClass.finvoke(RubyClass.java:911)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:356)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:369)
	at org.jruby.javasupport.JavaEmbedUtils$1.callMethod(JavaEmbedUtils.java:117)
	at org.jruby.rack.DefaultRackApplication.call(DefaultRackApplication.java:64)
	at org.jruby.rack.AbstractRackDispatcher.process(AbstractRackDispatcher.java:33)
	at org.jruby.rack.AbstractServlet.service(AbstractServlet.java:34)
	at org.jruby.rack.AbstractServlet.service(AbstractServlet.java:40)
	at com.thoughtworks.go.rackhack.DelegatingServlet.service(DelegatingServlet.java:48)
	at com.thoughtworks.go.rackhack.DelegatingServlet.service(DelegatingServlet.java:53)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:614)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:227)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:78)
	at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:39)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:53)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:58)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:67)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.AbstractBasicAuthenticationFilter.doFilterInternal(AbstractBasicAuthenticationFilter.java:57)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.thoughtworks.go.server.newsecurity.filters.AssumeAnonymousUserFilter.doFilterInternal(AssumeAnonymousUserFilter.java:65)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.thoughtworks.go.server.newsecurity.filters.AbstractReAuthenticationFilter.doFilterInternal(AbstractReAuthenticationFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.thoughtworks.go.server.newsecurity.filters.InvalidateAuthenticationOnSecurityConfigChangeFilter.doFilterInternal(InvalidateAuthenticationOnSecurityConfigChangeFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.RememberLastRequestUrlFilter.doFilterInternal(RememberLastRequestUrlFilter.java:39)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:41)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:81)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:77)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at com.thoughtworks.go.server.web.BackupFilter.doFilter(BackupFilter.java:90)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:50)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

Ideally, a plugin should just print a warning message instead of printing unnecessary stack trace.

Fix text alignment in header

Duplicate of: gocd-contrib/docker-swarm-elastic-agent-plugin#51

Elastic agents finishing before all logs are sent to GoCD after upgrade to 2.0.0-113

Hello,

So I've been using this plugin for a while and it generally works great.

After I've upgraded to the latest GoCD and the latest version of this plugin (kubernetes-elastic-agent-2.0.0-113.jar), my EAs have been dying without giving me all the logs from the container

As an example, in GoCD this is the last line of log I get:

[INFO] [INFO] Compiling 7 source files to /godata/pipelines/EventDocumenter/event-documenter/target/classes****

And on the gcloud console of the container I have a lot more stuff

2018-11-15 15:32:57,277 [stdout] - 2018-11-15 15:32:57,272 INFO  [Thread-24] CommandLine:51 - [INFO] [INFO] Compiling 7 source files to /godata/pipelines/EventDocumenter/event-documenter/target/classes     **<=== LAST LINE I SAW ON GoCD Web Console **
2018-11-15 15:33:01,116 [stdout] - 2018-11-15 15:33:01,099 INFO  [pool-2-thread-1] HttpService:138 - Got back 200 from server
.
.
.
Lot's of other stuff, close to 100 lines of output
.
.
.
2018-11-15 15:33:16,895 [stdout] - 2018-11-15 15:33:16,895 INFO  [Thread-24] CommandLine:51 - [INFO] BUILD FAILURE
2018-11-15 15:33:17,706 [stdout] - 2018-11-15 15:33:17,691 INFO  [scheduler-3] DefaultGoPublisher:99 - Agent [gocd-jdk11-dind-991552d2-b280-4ff3-b533-3314cd061a6a, 10.48.2.117, 816045bb-7ead-44c8-8cd6-2e3fca968cae] is reporting build result [Failed] to Go Server for Build [EventDocumenter/39/release/3/release/616
2018-11-15 15:33:17,844 [stdout] - 2018-11-15 15:33:17,834 INFO  [scheduler-3] DefaultGoPublisher:94 - Agent [gocd-jdk11-dind-991552d2-b280-4ff3-b533-3314cd061a6a, 10.48.2.117, 816045bb-7ead-44c8-8cd6-2e3fca968cae] is reporting status [Completing] to Go Server for Build [EventDocumenter/39/release/3/release/616
2018-11-15 15:33:17,934 [stdout] - 2018-11-15 15:33:17,927 INFO  [scheduler-3] ArtifactsPublisher:69 - Pluggable metadata folder is empty.
2018-11-15 15:33:17,936 [stdout] - 2018-11-15 15:33:17,935 INFO  [scheduler-3] DefaultGoPublisher:104 - Agent [gocd-jdk11-dind-991552d2-b280-4ff3-b533-3314cd061a6a, 10.48.2.117, 816045bb-7ead-44c8-8cd6-2e3fca968cae] is reporting build result [Failed] to Go Server for Build [EventDocumenter/39/release/3/release/616
2018-11-15 15:33:19,533 INFO  [ShutdownThread-5] AgentProcessParentImpl$Shutdown:185 - Shutdown hook invoked. Shutting down Process[pid=69, exitValue="not exited"]

Do you guys know what could be the issue here?

I am using a GoCD agent derived from the Debian 9 one.

Provide an option to always pull the elastic agent docker image

Usually, docker images used with the elastic agents are versioned. So, if a new image needs to be used, the elastic profile is updated with the right tag and the jobs start to use the new image. If the tag is not versioned, the newer image is not pulled down.

Have image pull policies to either pull the elastic agent docker image always and to pull the image if not present on the machine.

Validation of HTTPs url

I think it's time to disable URL validation on the plugin and allow passing in http:// URLs instead of https:// URLs for the server url field.

Use the defaults for autoregister timeout and pending pods if they are not provided.

Currently, a null pointer exception is logged in the plugin log if the agent autoregister timeout and pending pods aren't provided.

2018-04-13 15:15:11,713 ERROR [qtp843512726-28] KubernetesPlugin:128 - Failed to handle request go.plugin-settings.validate-configuration
com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: empty String
	at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:227)
	at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:217)
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:129)
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:220)
	at com.google.gson.Gson.fromJson(Gson.java:887)
	at com.google.gson.Gson.fromJson(Gson.java:852)
	at com.google.gson.Gson.fromJson(Gson.java:801)
	at com.google.gson.Gson.fromJson(Gson.java:773)

In version 2.0.0 console log is often lost.

After update to plugin v 2.0.0 (with gocd 18.10) consolo output is often lost or truncated

Kubernetes Plugin fails with NullPointerException when Pod name is set using generateName

When following elastic agent profile pod YAML is used:

apiVersion: v1
kind: Pod
metadata:
  generateName: pod-name-prefix
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: gocd/gocd-agent-alpine-3.10:v19.7.0

The plugin fails to create a new elastic agent with the following error:

2019-09-05 14:06:39,917 ERROR [5705@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
java.lang.NullPointerException: null
	at cd.go.contrib.elasticagent.KubernetesInstanceFactory.setPodNameIfNecessary(KubernetesInstanceFactory.java:287)
	at cd.go.contrib.elasticagent.KubernetesInstanceFactory.createUsingPodYaml(KubernetesInstanceFactory.java:240)
	at cd.go.contrib.elasticagent.KubernetesInstanceFactory.create(KubernetesInstanceFactory.java:61)
	at cd.go.contrib.elasticagent.KubernetesAgentInstances.createKubernetesInstance(KubernetesAgentInstances.java:90)
	at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:68)
	at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:39)
	at cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:44)
	at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
	at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
	at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
	at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
	at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:60)
	at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:43)
	at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
	at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
	at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
	at java.lang.Thread.run(Thread.java:748)

Cannot add new Cluster Profile

Hello
I've been testing goCD, and I'm getting stuck with some issues.

Scenario

One dev K8s cluster with goCD installed via helm
One staging k8s cluster that I'm trying to connect as Elastic Cluster profile.

Steps to Reproduce

Install goCD via helm in the dev cluster

helm install stable/gocd --name gocd --namespace gocd

Go to Staging Cluster and create service account

kubectl create sa gocd

Create Clusterrole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
  labels:
    app: gocd
    chart: gocd-1.19.3
    heritage: Tiller
    release: gocd
  name: gocd
  resourceVersion: "13566932"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/gocd
  uid: ec92d205-0d13-11ea-b5b7-42010a840176
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get

Create a Clusterrolebinding for new ClusterRole

k create clusterrolebinding gocd --clusterrole gocd --serviceaccount <namespace>:gocd

Go to Admin > Elastic Profiles > Add Cluster Profile
Add a new id (ie. Test)
Add the Go Server URL
Add cluster IP getting it with kubectl cluster-info
Namespace: configured in the clusterrolebinding
Security Token: The one from the secret in k describe sa gocd
Cluster CA: Same as above

Once I save it and try to Click "Status Report" I get the following message:

Failure executing: GET at: https://XX.XX.XX.XX/api/v1/nodes. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.

If I try to

kubectl auth can-i list nodes --namespace <namespace> --as=system:serviceaccount:<namespace>:gocd I get as result yes

I'm a bit lost, and I would gladly accept some helm on that :)

Thanks in advance!

PS: Every time I try to update the cluster information I got this Error:
Someone has modified the configuration for Cluster Profile 'stg'. Please update your copy of the config with the changes and try again.

Don't know if it's related to configuration or just another random issue.

gocd / kubernetes-elastic-agents Goto Github PK

kubernetes-elastic-agents's Introduction

Kubernetes Elastic agent plugin for GoCD

Table of Contents

Building the code base

Documentation

License

kubernetes-elastic-agents's People

Contributors

Stargazers

Watchers

Forkers

kubernetes-elastic-agents's Issues

Issue Type

Description

Possible Fix

Issue Type

Summary

Possible Fix

Environment

Deployment

GoCD Agent Deployments

ServiceAccount

ClusterRole

ClusterRoleBinding

Deployments

Logs

Issue Type

Summary

Environment

Basic environment details

Additional Environment Details

Steps to Reproduce

Expected Results

Actual Results

Possible Fix

Log snippets

Code snippets/Screenshots

Any other info

Issue Type

Summary

Description

Issue Type

Summary

Environment

Basic environment details

Expected Results

Actual Results

Any other info

Issue Type

Summary

Log snippets

Recommend Projects

Recommend Topics

Recommend Org