After rebalancing kafka error

[kspmmitt]

Hello,

As soon as kafka cluster is rebalanced, there is error seen as below and sometimes kafka PODs are re-started as well. Can you please give inputs if any?

{"version": "1.2.0", "timestamp": "2025-08-02T02:57:28.405+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "[Broker id=0] Received LeaderAndIsrRequest with correlation id 1 from controller 2 epoch 1 for partition internal-readiness-check_eric-bss-em-kb-operand-emoperand-kafka-0-0 (last update controller epoch 1) but cannot become follower since the new leader -1 is unavailable.", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-0"}, "extra_data": {"file": "Logging.scala", "line": "74", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

Rgds,
Kamal

[kyguy]

This error could be caused by a couple different things like if the broker that was the elected leader for this partition was down or was unreachable. Usually these kinds of issues sort themselves out however, I am not sure why the Kafka brokers would restarted. If you are able to capture the full broker logs when this happens again and upload them as attachments along with the full broker configurations I could take a closer look if you would like.

and sometimes kafka PODs

With the use of the word "PODs" I assume you are running Cruise Control in Kubernetes, is that correct? If so, it might be worth chasing up the question in a forum dedicated to that operator since the operator configurations and logs may shed so light on the underlying cause of the error as well.

Regardless, if you upload the related logs here, I will be best to diagnose and if I cannot, I will redirect you to someone who can.

[kspmmitt]

Got below logs, I changed the topic replication order and new leader was elected by cruise control but as soon as triggered rebalancing then Kafka pods got restarted.

{"version": "1.2.0", "timestamp": "2025-08-05T04:21:29.019+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "[LogLoader partition=strimzi.cruisecontrol.modeltrainingsamples-7, dir=/var/lib/kafka/data/kafka-log0] Could not find offset index file corresponding to log file /var/lib/kafka/data/kafka-log0/strimzi.cruisecontrol.modeltrainingsamples-7.227ed526ca0247c5a114ebf50612c24d-delete/00000000000000000000.log, recovering segment and rebuilding index files...", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-0"}, "extra_data": {"file": "Logging.scala", "line": "74", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

{"version": "1.2.0", "timestamp": "2025-08-05T04:21:31.767+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "[LogLoader partition=strimzi.cruisecontrol.modeltrainingsamples-3, dir=/var/lib/kafka/data/kafka-log1] Could not find offset index file corresponding to log file /var/lib/kafka/data/kafka-log1/strimzi.cruisecontrol.modeltrainingsamples-3.ad1aec9ec7f1485eba0054de2d266793-delete/00000000000000000000.log, recovering segment and rebuilding index files...", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-1"}, "extra_data": {"file": "Logging.scala", "line": "74", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

After above errors, same errors are repeating as in first post above. Some more errors below -

{"version": "1.2.0", "timestamp": "2025-08-05T04:59:07.673+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Error while creating ephemeral at /brokers/ids/2, node already exists and owner '0x203c71dfe4e0011' does not match current session '0x303c73d9be90020'", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-2"}, "extra_data": {"file": "Logging.scala", "line": "74", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}
{"version": "1.2.0", "timestamp": "2025-08-05T04:59:07.681+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "[KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists|\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:126)|\tat kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:2189)|\tat kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:2127)|\tat kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:2094)|\tat kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:106)|\tat kafka.server.KafkaServer.startup(KafkaServer.scala:365)|\tat kafka.Kafka$.main(Kafka.scala:113)|\tat kafka.Kafka.main(Kafka.scala)", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-2"}, "extra_data": {"file": "Logging.scala", "line": "82", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

{"version": "1.2.0", "timestamp": "2025-08-05T04:59:07.979+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Exiting Kafka due to fatal exception during startup. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists|\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:126)|\tat kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:2189)|\tat kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:2127)|\tat kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:2094)|\tat kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:106)|\tat kafka.server.KafkaServer.startup(KafkaServer.scala:365)|\tat kafka.Kafka$.main(Kafka.scala:113)|\tat kafka.Kafka.main(Kafka.scala)", "metadata": {"container_name": "kafka", "pod_name": "eric-bss-em-kb-operand-emoperand-kafka-2"}, "extra_data": {"file": "Kafka.scala", "line": "29", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

From cruise control pod -

{"version": "1.2.0", "timestamp": "2025-08-05T04:22:26.797+00:00", "severity": "warning", "service_id": "eric-data-messagebus-kf", "message": "Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=KafkaCruiseControlSampleStoreProducer|\tat java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)|\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1865)|\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:960)|\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895)|\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)|\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523)|\tat org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64)|\tat org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:436)|\tat org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:292)|\tat org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:319)|\tat org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:304)|\tat com.linkedin.kafka.cruisecontrol.monitor.sampling.AbstractKafkaSampleStore.createProducer(AbstractKafkaSampleStore.java:55)|\tat com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore.configure(KafkaSampleStore.java:141)|\tat com.linkedin.kafka.cruisecontrol.config.KafkaCruiseControlConfig.getConfiguredInstance(KafkaCruiseControlConfig.java:82)|\tat com.linkedin.kafka.cruisecontrol.monitor.task.LoadMonitorTaskRunner.(LoadMonitorTaskRunner.java:111)|\tat com.linkedin.kafka.cruisecontrol.monitor.task.LoadMonitorTaskRunner.(LoadMonitorTaskRunner.java:81)|\tat com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.(LoadMonitor.java:173)|\tat com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.(LoadMonitor.java:124)|\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.(KafkaCruiseControl.java:126)|\tat com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.(AsyncKafkaCruiseControl.java:34)|\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControlApp.(KafkaCruiseControlApp.java:36)|\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControlServletApp.(KafkaCruiseControlServletApp.java:32)|\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControlUtils.getCruiseControlApp(KafkaCruiseControlUtils.java:920)|\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:37)", "metadata": {"container_name": "cruise-control", "pod_name": "eric-bss-em-kb-operand-emoperand-cruise-control-7476cd496cfkt9d"}, "extra_data": {"file": "AppInfoParser.java", "line": "68", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

[kyguy]

Got below logs, I changed the topic replication order and new leader was elected by cruise control but as soon as triggered rebalancing then Kafka pods got restarted.

I am not sure I understand what "topic replication order" is, do you mean the replication factor a topic was changed? A replication factor change shouldn't normally trigger any broker restarts but it could happen if there was a misconfiguration of the Kafka brokers or Cruise Control.

If you post the full configuration of the following:

• Kafka brokers
• Cruise Control

I can take a closer look if you would like. That being said, if you are using an Kubernetes Operator I would advise chasing up this question in a forum that is dedicated to that operator since the error seems to be a configuration issue. The full log of that operator will likely contain information important to diagnosing any possible configuration issues.

[kspmmitt]

Used below command inside pod - partition replica order changed which will cause preferred leader goal to violate as preferred leader should be first in replica order right?

/usr/bin/kafka-reassign-partitions.sh --bootstrap-server eric-bss-em-kb-operand-emoperand-kafka-brokers:9092 --execute --reassignment-json-file leaderchange.json

cat < /tmp/leaderchange.json
{
"partitions": [
{
"topic": "inputjson",
"partition": 0,
"replicas": [2,0,1]
}
],
"version": 1
}
EOF

[kyguy]

Used below command inside pod - partition replica order changed which will cause preferred leader goal to violate as preferred leader should be first in replica order right?

The PreferredLeaderElectionGoal would be violated if Cruise Control was unable to generate a partition reassignment plan that would elect the preferred leaders as leaders as part of the rebalance.

On a side note, be careful about executing administrative commands manually like this while using a Kubernetes operator as it is generally not supported. Most Kubernetes operators provide an interface to preform such administrative tasks safely without interfering with other ongoing operations.