Git Product home page Git Product logo

Comments (11)

baluchicken avatar baluchicken commented on September 4, 2024

@xiongmaodada I will try to reproduce your error, can you please also share the operators log. Thanks

from koperator.

xiongmaodada avatar xiongmaodada commented on September 4, 2024

Thank you for your replys quickly.

kubectl logs kafka-operator-operator-0 -c manager -n kafka -f

{"level":"info","ts":1566990081.2534606,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990081.4579751,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990081.4906683,"logger":"controller","msg":"resource created","Request.Namespace":"kafka","Request.Name":"kafka","component":"kafka","kind":"*v1.Pod"}
{"level":"info","ts":1566990081.5584164,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990083.35322,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990098.9545534,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990099.957768,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990100.9538174,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990102.053241,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990121.5538023,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990121.753047,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990121.7804203,"logger":"controller","msg":"resource created","Request.Namespace":"kafka","Request.Name":"kafka","component":"kafka","kind":"*v1.Pod"}
{"level":"info","ts":1566990121.9516146,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990123.6931918,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990137.050859,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990138.0411665,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990139.1509326,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}
{"level":"info","ts":1566990140.1538386,"logger":"controller","msg":"Reconciling KafkaCluster","Request.Namespace":"kafka","Request.Name":"kafka"}

from koperator.

baluchicken avatar baluchicken commented on September 4, 2024

@xiongmaodada I managed to reproduce your error.
The first broker get's OOMKilled.

The used example configuration kubectl create -n kafka -f config/samples/banzaicloud_v1alpha1_kafkacluster.yaml configures the first broker's container to use only 300M memory which is simply not enough.
I will create a PR which comments the referenced lines out. (It is placed there to show all the available configurations.)
Please remove the referenced block from the CR.

On the other hand I would also suggest to increase your Minikube Memory and CPU size to at least 4 CPUs and 6GB RAM.

from koperator.

xiongmaodada avatar xiongmaodada commented on September 4, 2024

@baluchicken thank you, I try it.

Is there a document that test the above Kafka cluster? such as Send and receive messages Part. I don't know how to connect to kafka cluster from external of minikube k8s cluster.

from koperator.

baluchicken avatar baluchicken commented on September 4, 2024

We have something called Spotguides. You can read more about the concept here. We have a Kafka Spotguide which uses this Operator.

Spotguide contains documentation which considers your configuration.

I just copied the relevant part for you:

kubectl create -n kafka -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kafka-test
spec:
  containers:
  - name: kafka-test
    image: solsson/kafkacat
    # Just spin & wait forever
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]
    volumeMounts:
    - name: sslcerts
      mountPath: "/ssl/certs"
  volumes:
  - name: sslcerts
    secret:
      secretName: test-kafka-operator
EOF

Then exec into the container and produce and consume some messages:

kubectl exec -it -n kafka kafka-test bash

Produce some message

kafkacat -P -b kafka-headless:29092 -t kafka-test \
-X security.protocol=SSL \ 
-X ssl.key.location=/ssl/certs/clientKey \ 
-X ssl.certificate.location=/ssl/certs/clientCert \ 
-X ssl.ca.location=/ssl/certs/caCert

Consume them

kafkacat -C -b kafka-headless:29092 -t kafka-test \ 
-X security.protocol=SSL \
-X ssl.key.location=/ssl/certs/clientKey \ 
-X ssl.certificate.location=/ssl/certs/clientCert \ 
-X ssl.ca.location=/ssl/certs/caCert

from koperator.

xiongmaodada avatar xiongmaodada commented on September 4, 2024

@baluchicken i got it.

how to create topic at external of minikube k8s cluster? for example, there is a bin/kafka-topics.sh at the other machine that is at external of minikube k8s cluster, how to create topic by bin/kafka-topics.sh command?

/bin/kafka-topics.sh --create --zookeeper ip:port --replication-factor 1 --partitions 1 --topic my-kafka-topic

what is zookeeper ip:prt ?

producer:

/bin/kafka-console-producer.sh --broker-list nodeip:port --topic my-kafka-topic

what is zookeeper nodeip:port ?

from koperator.

baluchicken avatar baluchicken commented on September 4, 2024

Unfortunately, ZK is not accessable from outside. It is provisioned by a third party operator which as far as I know does not support this feature yet.

You can use the following command to create topics from inside the cluster.

kubectl -n kafka run kafka-topics -it --image=wurstmeister/kafka:2.12-2.1.0 --rm=true --restart=Never -- /opt/kafka/bin/kafka-topics.sh --zookeeper example-zookeepercluster-client.zookeeper:2181 --topic my-topic --create --partitions 1 --replication-factor 1

from koperator.

xiongmaodada avatar xiongmaodada commented on September 4, 2024

@baluchicken
Thanks a lot, It's ok!

I have another question, how to use /bin/kafka-console-producer.sh or /bin/kafka-console-consumer.sh command to send and receive messages?

from koperator.

baluchicken avatar baluchicken commented on September 4, 2024

@xiongmaodada I just created a simple docs about how to produce/consume messages on a freshly deployed Kafka Cluster.
Regarding the java producer/consumer because your cluster is using SSL I recommend to follow the official documentation on keystone/trustrore creation.

from koperator.

xiongmaodada avatar xiongmaodada commented on September 4, 2024

@baluchicken It's just what I need, Thank you very much!

from koperator.

bechhansen avatar bechhansen commented on September 4, 2024

Hi @xiongmaodada

Did you ever find the solution for this issue?

When enabling Prometheus annotations for the Kafka nodes (using the operator) one of my nodes become unstable and is terminated after ~30-40 seconds. I don't think this issue is a resource issue as I use the default resource requirements/limits that looks okay.

Its only one node that becomes unstable, and it looks like the operator is gracefully terminating the node.

My issue:
#659

from koperator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.