nanit / kubernetes-rabbitmq-cluster Goto Github PK

Hello,
could you please advise how to enable TLS support? I'm trying to define environment variables in statefulset manifest:

• name: RABBITMQ_SSL_CERTFILE
  value: /etc/ssl/certs/server.pem
• name: RABBITMQ_SSL_KEYFILE
  value: /etc/ssl/certs/server.key
• name: RABBITMQ_SSL_CACERTFILE
  value: /etc/ssl/certs/ca.pem

and according to docker-endpoint.sh it should work, but it does not.
Thank you.

Sometimes the pod is ready but still the exec command fails
Should probably use rabbitmqctl wait

Currently, running the deployment command again solves the problem

Error:

+ echo 'Waiting for RabbitMQ pod to be ready....'
Waiting for RabbitMQ pod to be ready....
++ kubectl get pods -n test
++ grep rabbitmq-0
++ grep Running
+ [[ -n '' ]]
+ echo 'RabbitMQ pod still not ready...'
RabbitMQ pod still not ready...
+ sleep 5

Steps to Reproduce

1. git clone
2. export the required environment variables
3. make deploy.

The above error just keeps repeating for indefinite period. All the K8s pods seem to be working fine though. Any help would be deeply appreciated.

Hi, I have a huge amount of worker.heartbeat messages in my cluster (around 6 millions), which seems to came progressively since a few hours (cluster works from ~10 days).

Do you know where it could come from ?

Exchange	celeryev
Routing Key	worker.heartbeat
Redelivered	●
Properties	
priority:	0
delivery_mode:	2
headers:	
hostname:	[email protected]
content_encoding:	utf-8
content_type:	application/json
Payload
309 bytes
Encoding: string
{"sw_sys": "Linux", "clock": 9033333, "timestamp": 1498082536.210317, "hostname": "[email protected]", "pid": 34, "sw_ver": "3.1.18", "utcoffset": -2, "loadavg": [1.26, 1.3, 1.47], "processed": 0, "active": 0, "freq": 2.0, "type": "worker-heartbeat", "sw_ident": "py-celery"}

Hi there, when i try to apply the statefulset, my pods fire up but error out after a few seconds.

When i check the k8s logs on the pods, I get the following:

/set_cluster_nodes.sh: line 20: NODES[@]: unbound variable

Can you maybe point me in any direction to fix this?

TIA

hello
so following this per the instructions , after i run make deploy it craps out with the error logs listed below. i think it does not detect all the exports
Some hints.. ? thanks

make deploy
docker pull nanit/rabbitmq:133aedd || ( docker build -t nanit/rabbitmq:133aedd docker && docker push nanit/rabbitmq:133aedd)
Trying to pull repository docker.io/nanit/rabbitmq ...
133aedd: Pulling from docker.io/nanit/rabbitmq
Digest: sha256:392c129dee79739488e15cdfc195d9c2d0babb8aabcab186433f6b546428b5e8
kubectl get ns default || kubectl create ns default
NAME STATUS AGE
default Active 11d
kubectl get svc -n default rabbitmq || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rabbitmq/g' kube/svc.yml | kubectl create -n default -f -
Error from server (NotFound): services "rabbitmq" not found
service "rabbitmq" created
kubectl get svc -n default rmq-cluster || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rmq-cluster/g' kube/svc.headless.yml | kubectl create -n default -f -
Error from server (NotFound): services "rmq-cluster" not found
service "rmq-cluster" created
if [ "TRUE" = "TRUE" ]; then kubectl get svc -n default rabbitmq-management || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rabbitmq-management/g' kube/svc.management.yml | kubectl create -n default -f - ; fi
Error from server (NotFound): services "rabbitmq-management" not found
service "rabbitmq-management" created
if [ -z "5" ]; then echo "ERROR: RABBITMQ_REPLICAS is empty!"; exit 1; fi
if [ -z "dimtheo" ]; then echo "ERROR: RABBITMQ_DEFAULT_USER is empty!"; exit 1; fi
if [ -z "123456" ]; then echo "ERROR: RABBITMQ_DEFAULT_PASS is empty!"; exit 1; fi
if [ -z "secret" ]; then echo "ERROR: RABBITMQ_ERLANG_COOKIE is empty!"; exit 1; fi
sed -e 's/{{SVC_NAME}}/rmq-cluster/g;s/{{APP_NAME}}/rabbitmq/g;s,{{IMAGE_NAME}},nanit/rabbitmq:133aedd,g;s/{{REPLICAS}}/5/g;s/{{RABBITMQ_DEFAULT_USER}}/dimtheo/g;s/{{RABBITMQ_DEFAULT_PASS}}/123456/g;s/{{RABBITMQ_ERLANG_COOKIE}}/secret/g' kube/stateful.set.yml | kubectl apply -n default -f -
error: unable to decode "STDIN": [pos 305]: json: expect char '"' but got char '1'
make: *** [deploy-rabbitmq] Error 1

This policy may not be desirable for everyone, and is currently being reapplied every time the deployment restarts.

Hello,

after upgrading Kubernetes cluster to 1.8.2 pods are crashing. I can see next error in logs:
curl: (22) The requested URL returned error: 403 Forbidden

I suspect the problem is in the next string of set_cluster_nodes.sh script:

STATEFUL_SETS=$(curl -f -k https://${KUBERNETES_SERVICE_HOST}/apis/apps/v1beta1/statefulsets -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

For some reason the url is not accessible.

in rabbitmq-0 logs getting error
curl: (22) The requested URL returned error: 403 Forbidden
would you have an idea of the cause

[root@node01 kubernetes-rabbitmq-cluster]# kubectl get pod |grep rabbit
rabbitmq-0 1/1 Running 0 18m
rabbitmq-1 0/1 CrashLoopBackOff 8 18m
rabbitmq-2 1/1 Running 2 18m

[root@node01 kubernetes-rabbitmq-cluster]# kubectl get svc |grep rab
rabbitmq ClusterIP 10.111.1.199 5672/TCP,4369/TCP,25672/TCP 19m
rabbitmq-management LoadBalancer 10.108.55.45 15672:31449/TCP 19m

[root@node01 kubernetes-rabbitmq-cluster]# kubectl logs rabbitmq-1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16652 0 16652 0 0 246k 0 --:--:-- --:--:-- --:--:-- 246k
ERROR: epmd error for host rabbitmq-1.rmq-cluster: nxdomain (non-existing domain)

[root@node01 kubernetes-rabbitmq-cluster]# kubectl exec rabbitmq-0 -- rabbitmqctl cluster_status
Cluster status of node '[email protected]' ...
[{nodes,[{disc,['[email protected]']}]},
{running_nodes,['[email protected]']},
{cluster_name,<<"[email protected]">>},
{partitions,[]},
{alarms,[{'[email protected]',[]}]}]

Hello,

When I deployed the rabbitmq in kubernetes v1.8, rabbitmq-0 pod returned this error:

/etc/rabbitmq/rabbitmq.config: Permision denied

The same configuration worked well in kubernetes v1.7.

Potential fix is to update the Dockerfile to add the following line after add of clusterer.config:

RUN chmod 666 /etc/rabbitmq/*

Hi i have enabled two contexts in kube prod and dev. In prod i have installed and properly working rabbitmq.

Now i trying to deploy rabbitmq in clear dev namespace, but without success.

My steps:

export NAMESPACE=dev &&
export DOCKER_REPOSITORY=nanit &&
export RABBITMQ_REPLICAS=3 &&
export RABBITMQ_DEFAULT_USER=username &&
export RABBITMQ_DEFAULT_PASS=password &&
export RABBITMQ_ERLANG_COOKIE=devrabbit &&
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE &&
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer &&
export RABBITMQ_HA_POLICY='{"ha-mode":"all"}' &&
export RABBITMQ_LOG_LEVEL=info &&
export RABBITMQ_ADDITIONAL_YAML="" &&
export SUDO="" &&
make deploy

.....
rabbitmq-0 pod ready ,setting ha policy: {"ha-mode":"all"}

• sleep 20
• kubectl exec -n dev rabbitmq-0 -- rabbitmqctl set_policy ha-all '.*' '{"ha-mode":"all"}' --apply-to queues
  error: unable to upgrade connection: container not found ("rabbitmq")
  Makefile:55: recipe for target 'deploy-rabbitmq' failed
  make: *** [deploy-rabbitmq] Error 1

all 3 replicas in Error state:

NAME READY STATUS RESTARTS AGE
rabbitmq-0 0/1 Error 1 14s
rabbitmq-1 0/1 Error 1 12s
rabbitmq-2 0/1 Error 1 11s

$kubectl logs rabbitmq-0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 47828 0 47828 0 0 834k 0 --:--:-- --:--:-- --:--:-- 819k
/set_cluster_nodes.sh: line 12: ((: 5
3: syntax error in expression (error token is "3")

all free pods - same error:

/set_cluster_nodes.sh: line 12: ((: 5
3: syntax error in expression (error token is "3")

The example run command says to run:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_EXPOSE_MANAGEMENT=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export SUDO="" && \
make deploy

This is broken, since it repeats the RABBITMQ_EXPOSE_MANAGEMENT twice, with the second time overwriting TRUE with LoadBalancer which is incorrect. The proper var to use is RABBITMQ_MANAGEMENT_SERVICE_TYPE

The proper sample should be:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export SUDO="" && \
make deploy

Upgraded cluster from 1.15 to 1.16 pods are failing with exit code 22 error.

Please provide guidance on this issue?

Hey,

Is there a to create rabbitmq service as a LoadBalancer type (in case the 'ha' policy is set to all).
In addition, how are we suppose to access rabbitmq service from the outside, if its type is 'CLUSTER_IP' type?

Thanks for any help

If you change the value of export NAMESPACE=default to something other than default this curl command fails and the whole deployment fails. After I deleted everything and ran it again with namespace set to default it all worked fine.

Perhaps you might have to specify a hostname that includes the namespace?

https://stackoverflow.com/questions/30690186/how-do-i-access-the-kubernetes-api-from-within-a-pod-container#comment75562617_33109739

Does this have data persistence? I want to create a new rabbitqm user and it has to survive pod termination. I don't see any persistent volume claims in the kubernetes yaml files. Please correct me if I'm wrong. Thank you.

i running this turn.

Deployment:

1. Clone this repository
2. Run:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export RABBITMQ_LOG_LEVEL=info && \
export RABBITMQ_ADDITIONAL_YAML="" && \
export SUDO="" && \
make deploy

but. i'm not success.
i don't know.  why i have this error?  
how i do i it? 

[root@hyperm-1 kubernetes-rabbitmq-cluster-master]# make deploy
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
sed -i -e 's/{{RABBITMQ_HA_POLICY}}/{\"ha-mode\":\"all\"}/g' docker/set_ha.sh
docker pull nanit/rabbitmq: || ( docker build -t nanit/rabbitmq: docker &&  docker push nanit/rabbitmq:)
invalid reference format
invalid argument "nanit/rabbitmq:" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
make: *** [docker-rabbitmq] Error 125