Comments (11)
bjrke
commented on August 29, 2024
2
its working! thx @aesteve and @ybyzek
from cp-all-in-one.
aesteve
commented on August 29, 2024
1
Facing the same issue that I first discovered using librdkafka, but the same happens with a Java producer, too.
What's happening is the API version (v3) request gets "cut" before actually receiving a response.
From the broker container: netstat -ano -p
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 0.0.0.0:9101 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 0.0.0.0:41777 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 127.0.0.11:46301 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 127.0.0.1:9092 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 192.168.48.2:29092 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 192.168.48.2:29093 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 192.168.48.2:29092 192.168.48.3:45354 ESTABLISHED - keepalive (6592.04/0/0)
tcp 0 0 192.168.48.2:29092 192.168.48.3:45348 ESTABLISHED - keepalive (6591.66/0/0)
tcp 0 0 192.168.48.2:29092 192.168.48.3:45360 ESTABLISHED - keepalive (6592.52/0/0)
tcp 0 0 192.168.48.2:29093 192.168.48.2:37916 TIME_WAIT - timewait (52.04/0/0)
tcp 0 0 192.168.48.2:29093 192.168.48.2:37902 ESTABLISHED - keepalive (6587.84/0/0)
tcp 0 0 192.168.48.2:29092 192.168.48.3:45350 ESTABLISHED - keepalive (6591.73/0/0)
tcp 0 0 192.168.48.2:37902 192.168.48.2:29093 ESTABLISHED - keepalive (6587.83/0/0)
udp 0 0 127.0.0.11:32893 0.0.0.0:* - off (0.00/0/0)
Whereas if running the standard image with zookeeper I'm getting this result:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 0.0.0.0:9092 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 0.0.0.0:29092 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 0.0.0.0:34405 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 0.0.0.0:9101 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 127.0.0.11:37105 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 192.168.80.3:41014 52.85.187.45:443 TIME_WAIT - timewait (27.25/0/0)
tcp 0 0 192.168.80.3:50126 192.168.80.3:29092 ESTABLISHED - keepalive (7159.45/0/0)
tcp 0 0 192.168.80.3:50106 192.168.80.3:29092 TIME_WAIT - timewait (18.20/0/0)
tcp 0 0 192.168.80.3:50138 192.168.80.3:29092 TIME_WAIT - timewait (19.78/0/0)
tcp 0 0 192.168.80.3:50096 192.168.80.3:29092 ESTABLISHED - keepalive (7155.10/0/0)
tcp 0 0 192.168.80.3:50160 192.168.80.3:29092 TIME_WAIT - timewait (20.39/0/0)
tcp 0 0 192.168.80.3:29092 192.168.80.4:35756 ESTABLISHED - keepalive (7159.82/0/0)
tcp 0 0 192.168.80.3:50116 192.168.80.3:29092 TIME_WAIT - timewait (18.42/0/0)
tcp 0 0 192.168.80.3:29092 192.168.80.3:50090 ESTABLISHED - keepalive (7156.31/0/0)
tcp 0 0 192.168.80.3:29092 192.168.80.3:50204 ESTABLISHED - keepalive (7170.32/0/0)
tcp 0 0 192.168.80.3:50098 192.168.80.3:29092 TIME_WAIT - timewait (17.51/0/0)
tcp 0 0 192.168.80.3:41016 52.85.187.45:443 TIME_WAIT - timewait (27.37/0/0)
tcp 0 0 192.168.80.3:29092 192.168.80.3:50096 ESTABLISHED - keepalive (7156.32/0/0)
tcp 0 0 192.168.80.3:29092 192.168.80.3:50126 ESTABLISHED - keepalive (7159.45/0/0)
tcp 0 0 192.168.80.3:50164 192.168.80.3:29092 TIME_WAIT - timewait (20.53/0/0)
tcp 0 0 192.168.80.3:50112 192.168.80.3:29092 TIME_WAIT - timewait (18.34/0/0)
tcp 0 0 192.168.80.3:50158 192.168.80.3:29092 TIME_WAIT - timewait (20.39/0/0)
What gets my attention is:
tcp 0 0 127.0.0.1:9092 0.0.0.0:* LISTEN - off (0.00/0/0)
vs.
tcp 0 0 0.0.0.0:9092 0.0.0.0:* LISTEN - off (0.00/0/0)
I'm guessing that could be part of the problem (I remember having troubles when running HTTP server in docker for instance, and having to use 0.0.0.0 as listen host).
So I tried and changed my config to:
broker:
image: confluentinc/cp-kafka:7.0.1
hostname: broker
container_name: broker
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: localhost
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_NODE_ID: 1
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
KAFKA_LOG4J_LOGGERS: "kafka.controller=TRACE,kafka.server=TRACE,kafka.broker=TRACE,kafka.server.IncrementalFetchContext=WARN"
volumes:
- ./update_run.sh:/tmp/update_run.sh
command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"the change being:
KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'And it looks ok so far, need to experiment more and make sure everything is still working both from the inside of docker compose and from the outside.
For now:
- topic creation is OK (from a kafka-setup container inside docker-compose)
- a producer (from the outside, local machine) can connect using
localhost:9092and produce messages - control center shows everything properly, including the messages sent by the producer
- connect instance looks ok so far
- ksqldb still to test
Hopefully this helps. Can file a PR if this is indeed the appropriate solution.
from cp-all-in-one.
pmeister
commented on August 29, 2024
Same here! I'm trying to connect with kafka-topics:
$ kafka-topics --bootstrap-server localhost:9092 --list
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2021-10-18 15:38:56,506] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
from cp-all-in-one.
bjrke
commented on August 29, 2024
Same Issue here tested with the v7.0.0 container
I am able to run kafka-topics via docker exec but my kafka client on the host is not able to connect and produces the same error.
Interestingly the console consumer (kafka-console-consumer.sh) running on my host machine is able to connect, no idea why. Maybe the error is not logged.
from cp-all-in-one.
pmeister
commented on August 29, 2024
@bjrke Can you please clarify what you mean by "the console consume"?
from cp-all-in-one.
bjrke
commented on August 29, 2024
the consoleConsumer provided with kafka itself, sorry for the typo, I will edit my comment
from cp-all-in-one.
mbreevoort
commented on August 29, 2024
Does this fix help?
#84
from cp-all-in-one.
pmeister
commented on August 29, 2024
@mbreevoort Not for me. Made no difference, unfortunately.
from cp-all-in-one.
ybyzek
commented on August 29, 2024
@aesteve thank you for tracking down the issue. I have verified in my environment that kafka-topics --bootstrap-server localhost:9092 --list fails with the current config and works with the proposed changed. If you could please file a PR, that would be excellent! Note: please base/merge on 6.2.0-post (not latest 7.0.1-post) since the problem exists there. Once PR is merged, I'll propagate the fix to all recent branches.
from cp-all-in-one.
ybyzek
commented on August 29, 2024
Now fixed in latest release: https://github.com/confluentinc/cp-all-in-one/blob/7.0.1-post/cp-all-in-one-kraft/docker-compose.yml#L25
from cp-all-in-one.
pmeister
commented on August 29, 2024
Thanks much for your fix, @aesteve, and for the quick incorporation of that fix, @ybyzek.
from cp-all-in-one.
Related Issues (20)
- Add ksqlDB-server and the CLI into KRaft cp-all-in-one-kraft HOT 2
- Kraft mode docker wont start HOT 1
- KRAFT mode does not work in cluster HOT 3
- No Connect Clusters Found HOT 1
- docker-compose file does not contain volumes spec HOT 5
- How to run the contents of update_run and /etc/confluent/docker/run inline without file HOT 1
- README for cp-all-in-one-community has wrong. link HOT 1
- KRaft - Kafka nodes continuously generating metadata deltas from snapshots HOT 4
- image given in cp-all-in-one-kraft is not found, docker-compose up fails
- Can't connect/consume from a different machine on the same network HOT 1
- enable.auto.commit HOT 1
- Docker Compose - ERROR Broker HOT 7
- cp-all-in-one-cloud Connect admin calls timing out
- A 3-broker version of "cp-all-in-one" docker-compose.yaml file ... HOT 4
- Fix for start cp-all-in-one-kraft kafka without zookeeper
- Port 2181 bind error on Zookeeper startup
- Docker compose fails on Fedora 35 with podman
- Starting environment fails on Mac M1 HOT 1
- Containers stop with Exit code 1 HOT 1
- Control-Center not able to connect to Kafka-Connect cluster HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cp-all-in-one.