Monitor producer/consumer stop time

[kspmmitt]

Hello,

Is there anyway or some metrices available in kafka/cruise control which can help to check the "pause times" for producer/consumer during rebalancing?

Is there any possibility as well that producer/consumer can lose connection and loss offset?

Rgds,
Kamal

[kyguy]

Is there anyway or some metrices available in kafka/cruise control which can help to check the "pause times" for producer/consumer during rebalancing?

Assuming we are talking about consumer group rebalances (which if we are this question would probably better answered in a forum specific to Apache Kafka than this Cruise Control forum, regardless, I will do my best to answer here):

Apache Kafka exposes several metrics [1] that could help indicated pauses, such as:

• rebalance-latency-avg
• fetch-latency-avg
• records-lag
• records-consumed-rate

Monitoring these metrics can help detect and quantify pauses caused by consumer group rebalances.

Is there any possibility as well that producer/consumer can lose connection and loss offset?

For consumers, yes, it is possible to lose connection during consumer group rebalances but losing committed offsets is not likely if the consumer commits offsets properly and Kafka is configured correctly.

[1] https://kafka.apache.org/documentation/#consumer_monitoring