Kafka consumer group rebalancing timeout during disaster recovery failover

Problem Description

During a planned disaster‑recovery (DR) failover from the primary Kafka cluster to a secondary replica, the AI model‑training data ingestion pipeline stalls. Batch jobs that consume from Kafka topics exceed their SLA and are terminated with a Job completion timeout. The underlying failure manifests as consumer‑group rebalancing timeouts and growing offset lag.

Typical log excerpts

org.apache.kafka.clients.consumer.RebalanceInProgressException: Rebalance in progress, consumer poll timed out
org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to rebalance in progress
Consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms
OffsetOutOfRangeException: Offset 1245789 is out of range for partition topic-0

Operational impact includes:

• Data ingestion pipelines stop pulling new training samples.
• Spark/Flink streaming jobs accumulate unprocessed records, leading to SLA breach.
• Downstream model‑training jobs time out, requiring manual restarts.

Root Cause Analysis

The failure is a convergence of three mechanisms that are tightly coupled during a DR switch:

1. GroupCoordinator unavailability – When the primary cluster goes down, the GroupCoordinator for the consumer group disappears. The secondary cluster must elect a new coordinator, but the client’s bootstrap.servers list still points primarily to the failed brokers, causing repeated GroupCoordinatorNotAvailableException until DNS or load‑balancer failover completes.
2. Rebalance protocol timeout mismatch – Consumer configuration defaults (session.timeout.ms=10000, max.poll.interval.ms=300000) assume a stable coordinator. During failover, the coordinator handshake stalls, and the client exceeds session.timeout.ms. The broker aborts the member’s heartbeat, triggering a rebalance that never completes because the new coordinator cannot respond within the default rebalance.timeout.ms (30 s). This aligns with the Apache Kafka documentation on consumer group rebalancing.
3. Offset synchronization lag – MirrorMaker 2 (or Confluent Replicator) replicates topic data across clusters, but offset metadata is only synced after successful commit. If the primary cluster crashes before the offset sync completes, the secondary cluster’s consumer starts from the last replicated offset, which can be far behind. The resulting OffsetOutOfRangeException forces the consumer to seek to earliest or latest, further delaying job completion.

In summary, the consumer group cannot complete a rebalance because the coordinator is unreachable, the rebalance timeout is too short for a DR transition, and offset lag amplifies the processing delay.

Investigation and Debugging

Below is a step‑by‑step diagnostic workflow that was used in the Netflix and Uber incidents referenced in the evidence package.

1. Verify coordinator availability

kafka-broker-api-versions.sh --bootstrap-server secondary.kafka.example.com:9092 --command-config client.properties

Expected output shows a list of broker IDs and their supported APIs. If the command hangs or returns ERROR: Unable to connect to any broker, the client is still pointing to the failed primary.

2. Inspect consumer group state

kafka-consumer-groups.sh --bootstrap-server secondary.kafka.example.com:9092 \
  --describe --group ai-training-consumers

Group	State	Protocol Type	Members
ai-training-consumers	Stable	consumer	5

If the state shows PreparingRebalance or Empty for an extended period, the coordinator is not responding.

3. Capture network traffic during rebalance

tcpdump -i eth0 port 9092 -w rebalance.pcap

Filter for JoinGroupRequest and HeartbeatRequest. A missing response pattern confirms coordinator outage.

4. Review broker logs for rebalance timeouts

journalctl -u kafka -n 200 | grep -i "rebalance"

Typical entries during failure:

[2026-09-16 14:23:45,123] WARN [GroupCoordinator 0] Failed to complete rebalance for group ai-training-consumers due to timeout (rebalance.timeout.ms)

5. Check offset sync status

kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list secondary.kafka.example.com:9092 \
  --topic training-data \
  --time -1

Compare the returned offsets with the last committed offsets in the primary cluster (if still accessible) to quantify lag.

Resolution

The fix consists of three coordinated changes: client configuration, broker‑side timeout tuning, and offset‑sync handling.

1. Enable static membership and increase rebalance timeout

Static membership prevents the group from being torn down when a consumer loses its heartbeat, allowing it to rejoin quickly after the coordinator recovers.

// Before (default)
props.put("group.id", "ai-training-consumers");
props.put("session.timeout.ms", "10000");
props.put("max.poll.interval.ms", "300000");

// After – static membership & extended timeouts
props.put("group.id", "ai-training-consumers");
props.put("group.instance.id", "consumer-${HOSTNAME}-${PID}");
props.put("session.timeout.ms", "30000");          // 30 s
props.put("max.poll.interval.ms", "900000");      // 15 min
props.put("rebalance.timeout.ms", "120000");      // 2 min (Kafka 2.4+)
props.put("heartbeat.interval.ms", "10000");

2. Raise broker‑side rebalance timeout

On each broker in the secondary cluster, update server.properties:

# Before
rebalance.max.timeout.ms=30000

# After – give DR failover ample time
rebalance.max.timeout.ms=180000

Restart the brokers or apply the change via dynamic config:

kafka-configs.sh --bootstrap-server secondary.kafka.example.com:9092 \
  --alter --entity-type brokers --entity-name 0 \
  --add-config rebalance.max.timeout.ms=180000

3. Enable offset sync on the replicator

For MirrorMaker 2, set the sync.topic.offsets.enabled flag to true and configure a retry back‑off to avoid spikes.

// MirrorMaker2 connector config
"tasks.max": "4",
"topics": "training-data",
"sync.topic.offsets.enabled": true,
"offset.syncs.topic.replication.factor": 3,
"offset.syncs.topic.partitions": 3,
"offset.syncs.topic.retention.ms": 86400000,
"retry.backoff.ms": 5000

4. Update DNS or load‑balancer failover order

Ensure that the bootstrap.servers list resolves to the secondary cluster within the DR window. A typical pattern is to use a CNAME that points to the active region’s broker pool.

Validation

After applying the changes, verify each layer:

Consumer group stability

kafka-consumer-groups.sh --bootstrap-server secondary.kafka.example.com:9092 \
  --describe --group ai-training-consumers | grep Stable

Output should show State: Stable and no members in PreparingRebalance for at least 5 minutes.

Rebalance latency

Measure the time between JoinGroupRequest and SyncGroupResponse in the broker logs. It should be < 30 s.

Offset lag

kafka-consumer-groups.sh --bootstrap-server secondary.kafka.example.com:9092 \
  --describe --group ai-training-consumers --members

The LAG column should be within acceptable thresholds (e.g., < 10 k records) after the failover.

Job completion

Run a controlled ingestion job and confirm it finishes within its SLA (e.g., 30 min). Check the job logs for absence of Job completion timeout messages.

Operational Best Practices and Prevention

• Static group membership – Always configure group.instance.id for long‑running consumers in multi‑region deployments.
• Graceful DR orchestration – Sequence the failover: (1) redirect DNS, (2) verify broker connectivity, (3) increase client timeouts, (4) start consumers.
• Monitoring – Alert on:
  • Consumer lag > 5 min (Prometheus query kafka_consumer_group_lag{group="ai-training-consumers"} > 300000)
  • GroupCoordinatorNotAvailableException rate spikes
  • Rebalance duration metric exceeding rebalance.timeout.ms
• Offset sync health check – Periodically compare primary and secondary offset checkpoints via the Replicator’s internal topics.
• Testing – Run automated DR drills that trigger a controlled broker outage and validate that consumers remain in Stable state.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the consumer keep throwing RebalanceInProgressException after the failover?
   Because the client’s heartbeat interval exceeds session.timeout.ms while the new coordinator is still electing. Extending the session and rebalance timeouts and using static membership allows the client to survive the coordinator transition.
2. Can I avoid offset lag without increasing offset.syncs.topic.replication.factor?
   Yes. Enable sync.topic.offsets.enabled and configure a reasonable offset.syncs.topic.retention.ms. This forces the replicator to publish offset checkpoints more frequently, reducing the window of divergence.
3. Is increasing max.poll.interval.ms safe?
   It is safe as long as the processing time per poll does not exceed the new limit. The setting only controls the maximum time between successive poll() calls; setting it too high can mask genuine processing stalls.
4. What is the difference between rebalance.timeout.ms and rebalance.max.timeout.ms?
   rebalance.timeout.ms is a client‑side limit for how long a member will wait for the coordinator to complete the rebalance. rebalance.max.timeout.ms is a broker‑side upper bound that caps the total time the coordinator will spend on a rebalance. Both must be increased for DR scenarios.
5. How can I confirm which broker is acting as the GroupCoordinator?
   Run kafka-broker-api-versions.sh --bootstrap-server ... and look for the line GroupCoordinator: brokerId=2 in the broker logs, or query the Zookeeper/metadata path /brokers/topics/_consumer_offsets/partitions/0/state for the current leader.