CrashLoopBackOff caused by Exiting because log truncation is not allowed about kubernetes-kafka HOT 3 CLOSED

Re-assigned partitions of the two topics to the two working brokers, using #95 with BROKERS env set to 0,1. The job pod logged:

# reassign-topics.json
{"topics":[
 {"topic":"test-produce-consume"},
 {"topic":"test-kafkacat"}
]}
# proposed-reassignment.json
{"version":1,"partitions":[{"topic":"test-kafkacat","partition":0,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"test-produce-consume","partition":0,"replicas":[0,1],"log_dirs":["any","any"]}]}
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test-kafkacat","partition":0,"replicas":[2,0],"log_dirs":["any","any"]},{"topic":"test-produce-consume","partition":0,"replicas":[1,2],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

Now kafka-2 starts but load is unbalanced. Did another reassign, with BROKERS set to 0,1,2 and got proposed reassignment {"version":1,"partitions":[{"topic":"test-kafkacat","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"test-produce-consume","partition":0,"replicas":[2,0],"log_dirs":["any","any"]}]}.

Still seeing this log message though: WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:13, offset:2404}, Current: {epoch:14, offset1852} for Partition: test-produce-consume-0 (kafka.server.epoch.LeaderEpochFileCache).

from kubernetes-kafka.

Pod restart got rid of the warning, but showed ERROR [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition test-produce-consume-0 to broker 0:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread).

Used maintenance/preferred-replica-election-job.yml.

from kubernetes-kafka.

Hi @solsson, could u figure it out why happens the above WARN message exactly?
"WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order."

from kubernetes-kafka.