bug: apisix-etcd pod CrashLoopBackOff about apisix HOT 6 OPEN

Can you check the reason about why etcd apisix-etcd-0 not running ?

from apisix.

this is the guide from apisix
due to this error, I have reinstalled with this guide but all pods is not running

https://apisix.apache.org/blog/2021/12/15/deploy-apisix-in-kubernetes/
using kubectl describe pod pod_name --n namespace_name , I got this error

using kubectl logs pod_name --n namespace_name, I got this error

from apisix.

Can you check the reason about why etcd apisix-etcd-0 not running ?
Could you please check the errors ? It is a little challenge from me. I hope you help me on this

from apisix.

Yes, the etcd cluster is not running normal.
So you can scale etcd replicas from 3 to 1 .
Or you can try preference etcd cluster config like this:

      ETCD_ENABLE_V2:                    true
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_NAME:                         cooper-apisix-etcd-1 (v1:metadata.name)
      POD_IP:                             (v1:status.podIP)
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(POD_IP):2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(ETCD_NAME).cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_INITIAL_CLUSTER:              cooper-apisix-etcd-0=http://cooper-apisix-etcd-0.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-1=http://cooper-apisix-etcd-1.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-2=http://cooper-apisix-etcd-2.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_INITIAL_CLUSTER_TOKEN:        apisix-etcd-cluster
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_DEBUG:                        true

from apisix.

ETCD_ADVERTISE_CLIENT_URLS

Yes, the etcd cluster is not running normal. So you can scale etcd replicas from 3 to 1 . Or you can try preference etcd cluster config like this:

      ETCD_ENABLE_V2:                    true
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_NAME:                         cooper-apisix-etcd-1 (v1:metadata.name)
      POD_IP:                             (v1:status.podIP)
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(POD_IP):2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(ETCD_NAME).cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_INITIAL_CLUSTER:              cooper-apisix-etcd-0=http://cooper-apisix-etcd-0.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-1=http://cooper-apisix-etcd-1.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-2=http://cooper-apisix-etcd-2.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_INITIAL_CLUSTER_TOKEN:        apisix-etcd-cluster
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_DEBUG:                        true

Could you explain the difference between standard and your configuration ?

git clone https://github.com/zaunist/apisix-on-kubernetes.git
I have download the etcd.yaml and changed

• name: ETCD_ADVERTISE_CLIENT_URLS
  value: >-
  http://$(MY_POD_IP):2379
  like this
  for other configuration, I am not sure it works on my side
  Because when I check ETCD_DEBUG and ETCD_ENABLE_V2, it didn't work

from apisix.

this is my pod yaml using kubectl edit command for the pod
but it seems different with your config

• name: etcd
  image: docker.io/bitnami/etcd:3.5.7-debian-11-r14
  ports:
  - name: client
  containerPort: 2379
  protocol: TCP
  - name: peer
  containerPort: 2380
  protocol: TCP
  env:
  - name: BITNAMI_DEBUG
  value: 'false'
  - name: MY_POD_IP
  valueFrom:
  fieldRef:
  apiVersion: v1
  fieldPath: status.podIP
  - name: MY_POD_NAME
  valueFrom:
  fieldRef:
  apiVersion: v1
  fieldPath: metadata.name
  - name: MY_STS_NAME
  value: apisix-etcd
  - name: ETCDCTL_API
  value: '3'
  - name: ETCD_ON_K8S
  value: 'yes'
  - name: ETCD_START_FROM_SNAPSHOT
  value: 'no'
  - name: ETCD_DISASTER_RECOVERY
  value: 'no'
  - name: ETCD_NAME
  value: $(MY_POD_NAME)
  - name: ETCD_DATA_DIR
  value: /bitnami/etcd/data
  - name: ETCD_LOG_LEVEL
  value: info
  - name: ALLOW_NONE_AUTHENTICATION
  value: 'yes'
  - name: ETCD_AUTH_TOKEN
  value: >-
  jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
  - name: ETCD_ADVERTISE_CLIENT_URLS
  value: >-
  http://$(MY_POD_NAME).apisix-etcd-headless.api-gateway.svc.cluster.local:2379,http://apisix-etcd.api-gateway.svc.cluster.local:2379
  - name: ETCD_LISTEN_CLIENT_URLS
  value: http://0.0.0.0:2379
  - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
  value: >-
  http://$(MY_POD_NAME).apisix-etcd-headless.api-gateway.svc.cluster.local:2380
  - name: ETCD_LISTEN_PEER_URLS
  value: http://0.0.0.0:2380
  - name: ETCD_INITIAL_CLUSTER_TOKEN
  value: etcd-cluster-k8s
  - name: ETCD_INITIAL_CLUSTER_STATE
  value: new
  - name: ETCD_INITIAL_CLUSTER
  value: >-
  apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.api-gateway.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.api-gateway.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.api-gateway.svc.cluster.local:2380
  - name: ETCD_CLUSTER_DOMAIN
  value: apisix-etcd-headless.api-gateway.svc.cluster.local

from apisix.