kafka

Kafka consumer group JVM heap exhaustion during peak tensor feature streaming

by
a large sculpture in front of a building

Problem Description During peak AI inference traffic, Kafka consumer pods in a Kubernetes cluster begin to terminate with the following errors: java.lang.OutOfMemoryError: Java heap space at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:267) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1152) … java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:698) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:112) at org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:124) … [Consumer clientId=consumer-1, groupId=feature-stream] Failed to allocate memory for record batch Operational impact includes: Consumer … Read more

Kafka consumer fails to process messages due to dimension mismatch

by
a large sculpture in front of a building

Problem – Consumer Crashes on Embedding Dimension Mismatch In a staging deployment of an AI inference pipeline (Kafka 3.3 brokers, Confluent Schema Registry 7.x), the downstream consumer that reads nlp-embeddings messages began throwing SerializationException and shutting down its poll loop. The failure manifested as: org.apache.kafka.common.errors.SerializationException: Size of data 3072 does not match expected fixed size 2048 for … Read more

Kafka HPA scaling issues with incorrect CPU metrics

by
a couple of statues sitting on top of a stone ground

Problem – Inconsistent Kafka HPA Scaling Due to Mis‑configured CPU Metrics In a Kubernetes‑based telemetry pipeline ingesting IoT sensor data, the Horizontal Pod Autoscaler (HPA) for Kafka consumer pods either adds ~30 % extra replicas during low‑traffic periods or fails to add enough pods during traffic spikes. The symptoms manifest as: Over‑provisioned pods at night, inflating … Read more

Kafka consumer group rebalancing fails during GPU worker restart

by
a couple of statues sitting on top of a stone ground

Kafka consumer group rebalancing fails during GPU worker restart Problem Description In a Kubernetes‑based distributed training pipeline, each GPU worker runs a Kafka consumer that streams training batches. When a pod is terminated (pre‑empted, rolling update, or node failure), the consumer group experiences a rebalance that never completes, leading to errors such as: org.apache.kafka.clients.consumer.CommitFailedException: Commit … Read more

Kafka container crash loop on premises with low CPU

by
a black and white photo of a mountain

Problem In an on‑premises data‑pipeline node that hosts a Kafka broker container, the pod repeatedly enters a CrashLoopBackOff state. The container exits after a few seconds, the orchestrator restarts it, and the cycle continues. The environment is a bare‑metal server with 4 vCPU and 8 GiB RAM, shared with other AI‑training services. The symptom manifests as: Kafka … Read more