Summary
I am trying to create the kuma control plane on a GKE cluster by following the standalone instructions as detailed here although after everything has completed it seems kuma is stuck at creating the default mesh. The control plane logs indicates
with INFO defaults trying to create default Mesh
repeating.
I have tried to creating a mesh manually by applying the follow with kubectl -f mesh.yml:
mesh.yml
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
name: my-mesh
spec:
mtls:
enabledBackend: test
backends:
- name: test
type: builtin
enabled: true
but this resulted in the following error:
Error from server (InternalError): error when creating "test.yml": Internal error occurred: failed calling webhook "mesh.defaulter.kuma-admission.kuma.io": Post https://kuma-control-plane.kuma-system.svc:443/default-kuma-io-v1alpha1-mesh?timeout=30s: context deadline exceeded
Steps To Reproduce
- Setup a GKE cluster using the following parameters
gcloud beta container clusters create test \
--no-enable-basic-auth \
--cluster-version "1.17.17-gke.1101" \
--machine-type "n1-standard-2" \
--image-type "COS" \
--disk-type "pd-standard" \
--disk-size "20" \
--metadata disable-legacy-endpoints=true, \
--service-account limited-gke-service-account \
--max-pods-per-node "30" \
--num-nodes "3" \
--enable-stackdriver-kubernetes \
--enable-private-nodes \
--master-ipv4-cidr "$MASTER_IPV4" \
--enable-ip-alias \
--default-max-pods-per-node "110" \
--enable-autoscaling \
--min-nodes "0" \
--max-nodes "3" \
--enable-network-policy \
--enable-master-authorized-networks \
--master-authorized-networks 0.0.0.0/0 \
--addons HorizontalPodAutoscaling,HttpLoadBalancing \
--enable-autoupgrade \
--enable-autorepair \
--enable-shielded-nodes
The limited-gke-service-account
has a Kubernetes Engine Admin IAM role as well as the Metric Writer role.
- Use the 1.1.1 release of kuma, and deploy the control plane as described here
- Monitor the control plane logs and see the repeat logging of the default mesh trying to be created. When you access the UI you will also note that there isn't a default mesh displayed.
- While this is occurring you may try and deploy the mesh.yml above to create the my-mesh mesh which should result in the following error:
Error from server (InternalError): error when creating "test.yml": Internal error occurred: failed calling webhook "mesh.defaulter.kuma-admission.kuma.io": Post https://kuma-control-plane.kuma-system.svc:443/default-kuma-io-v1alpha1-mesh?timeout=30s: context deadline exceeded
Additional Details & Logs
Tested on both kuma 1.1.1 and 1.1.0
There doesn't seem to be any other logs indicating error besides the repeating of the trying to create default Mesh
INFO entries
Update: leaving the control plane in this state after a while results in this error message:
ERROR mesh-insight-resyncer component terminated with an error {"generationID": 1, "error": "stop channel was closed", "errorVerbose": "stop channel was closed\ngithub.com/kumahq/kuma/pkg/events.(*reader).Recv\n\t/go/src/github.com/kumahq/kuma/pkg/events/eventbus.go:57\ngithub.com/kumahq/kuma/pkg/insights.(*resyncer).Start\n\t/go/src/github.com/kumahq/kuma/pkg/insights/resyncer.go:101\ngithub.com/kumahq/kuma/pkg/core/runtime/component.(*resilientComponent).Start.func1\n\t/go/src/github.com/kumahq/kuma/pkg/core/runtime/component/resilient.go:43\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}
Although it could be due to GKE restarting the pod.
Default quick start, no custom changes.
- Platform and Operating System
GKE
- Installation Method (Helm, kumactl, AWS CloudFormation, etc.)
This occurs in the latest helm charts as well as with the kumactl client.