Unusual error breaking kubernetes connection about kubernetes-elastic-agents HOT 11 CLOSED

@Torgon could you help with a screenshot of your Cluster Profile and Elastic Profile you have configured?

Any environment details would also be helpful. The helm chart is deployed in minikube, EKS, GCP etc?

from kubernetes-elastic-agents.

Sure
Elastic agent profile https://gyazo.com/80ff8cef49f87f82e408c1f0e114b702
Deployed on a DigitalOcean kubernetes cluster v1.15.3

from kubernetes-elastic-agents.

@Torgon I tried helm install of the latest GoCD helm chart on a minikube cluster with

minikube version: v1.4.0
commit: 7969c25a98a018b94ea87d949350f3271e9d64b6

Kubectl version

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T23:49:20Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

I could see the status report

2019-10-15 10:35:38,140 INFO  [qtp1152224728-36] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2019-10-15 10:35:38,145 INFO  [qtp1152224728-36] KubernetesPlugin:72 - [status-report] Generating status report.
2019-10-15 10:35:38,153 INFO  [qtp1152224728-36] KubernetesPlugin:72 - Running kubernetes nodes 1
2019-10-15 10:35:38,158 INFO  [qtp1152224728-36] KubernetesPlugin:72 - Running pods 0

What happens when you do the following for your DigitalOcean kubernetes cluster?

kubectl get nodes

#pick a node where your GoCD server is deployed and run

kubectl describe nodes <name-of-the-node> 

from kubernetes-elastic-agents.

@Torgon also have you made any changes to GoCD helm chart values file? If yes can you let me know what are the changes, might help in replicating this

from kubernetes-elastic-agents.

Name: pool-prod-01-wo4l
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=s-4vcpu-8gb
beta.kubernetes.io/os=linux
doks.digitalocean.com/node-id=66734b22-1e11-46cc-b529-5ddecfe04dc3
doks.digitalocean.com/node-pool=pool-prod-01
doks.digitalocean.com/node-pool-id=49823cd5-d74e-44e3-9b84-361994428487
doks.digitalocean.com/version=1.15.3-do.1
failure-domain.beta.kubernetes.io/region=fra1
kubernetes.io/arch=amd64
kubernetes.io/hostname=pool-prod-01-wo4l
kubernetes.io/os=linux
region=fra1
Annotations: csi.volume.kubernetes.io/nodeid: {"dobs.csi.digitalocean.com":"161214164","io.rancher.longhorn":"pool-prod-01-wo4l"}
field.cattle.io/creatorId:
io.cilium.network.ipv4-cilium-host: 10.244.2.1
io.cilium.network.ipv4-health-ip: 10.244.2.74
io.cilium.network.ipv4-pod-cidr: 10.244.2.0/24
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 01 Oct 2019 08:42:59 +0000
Taints:
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

NetworkUnavailable False Tue, 01 Oct 2019 08:43:14 +0000 Tue, 01 Oct 2019 08:43:14 +0000 CiliumIsUp Cilium is running on this node
MemoryPressure False Tue, 15 Oct 2019 11:09:34 +0000 Tue, 01 Oct 2019 08:42:59 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 15 Oct 2019 11:09:34 +0000 Tue, 01 Oct 2019 08:42:59 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 15 Oct 2019 11:09:34 +0000 Tue, 01 Oct 2019 08:42:59 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 15 Oct 2019 11:09:34 +0000 Tue, 01 Oct 2019 08:43:09 +0000 KubeletReady kubelet is posting ready status
Addresses:
Hostname: pool-prod-01-wo4l
InternalIP: 10.135.233.58
ExternalIP: 206.81.21.21
Capacity:
attachable-volumes-csi-dobs.csi.digitalocean.com: 7
cpu: 4
ephemeral-storage: 165105408Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8170040Ki
pods: 110
Allocatable:
attachable-volumes-csi-dobs.csi.digitalocean.com: 7
cpu: 4
ephemeral-storage: 165105408Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 6694Mi
pods: 110
System Info:
Machine ID: 96dc6ede0ebd41bca73d4fcc38e016e3
System UUID: 96dc6ede-0ebd-41bc-a73d-4fcc38e016e3
Boot ID: cb256851-77b1-4780-b608-5b2d90200d5c
Kernel Version: 4.19.0-0.bpo.5-amd64
OS Image: Debian GNU/Linux 9 (stretch)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.2
Kubelet Version: v1.15.3
Kube-Proxy Version: v1.15.3
PodCIDR: 10.244.2.0/24
ProviderID: digitalocean://161214164
Non-terminated Pods: (27 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE

... skipped list of running pods

gocd gocd-server-5bdd76689f-zwpbt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18h

... skipped list of running pods

Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

cpu 727m (18%) 1402m (35%)
memory 1287Mi (19%) 570Mi (8%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-csi-dobs.csi.digitalocean.com 0 0
Events:

Helm set variable:
server.service.type=ClusterIP

from kubernetes-elastic-agents.

@Torgon thank you for the information. As I understand this line here is responsible for converting the Allocatable Memory into the Long format.

For one of your nodes, the Allocatable memory is returned with different units than expected. This looks like a bug and we will take a look at it and publish a fix.

from kubernetes-elastic-agents.

thats great, thank you !

from kubernetes-elastic-agents.

@Torgon the team has made a fix for the issue you were facing. We have an experimental release available here, can you help us verify if this works in your environment?

To use this experimental in your Helm chart you can do the following

1. Download the values.yaml file for GoCD helm chart. You can then edit the url here to point to the experimental build and use this values file to install your helm chart. The command to do so would be

helm install --name gocd-app --namespace gocd stable/gocd -f values.yaml

from kubernetes-elastic-agents.

@adityasood that looks a whole lot better, seems to be fixed
Thank you !

from kubernetes-elastic-agents.

Minor follow up, though: the pod link in this screenshot is not working

https://gocdserver.../go/admin/status_reports/cd.go.contrib.elasticagent.kubernetes/gocd-agent-cac2d33d-abd0-4114-9609-ea237938d645
Pod is up and running

from kubernetes-elastic-agents.

@Torgon can you please share the go-server and Kubernetes plugin logs for the same?

from kubernetes-elastic-agents.