kafka

Model loading timeout in Docker after Kafka consumer lag increase

by
a group of people walking down a street next to tall buildings

Problem The AI inference service runs inside a Docker container and consumes model artifact messages from a Kafka topic. After a period of increasing consumer lag, the service logs a Model loading timeout error and fails to load the new model version. The container remains healthy, but inference requests continue using the stale model. Root … Read more

Kafka client token refresh failure inside Docker after container restart

by
white and brown concrete building

Problem After a Docker container running a Kafka client is restarted, the client repeatedly fails to authenticate with the broker. The failure manifests as token‑refresh errors and results in Authentication failed: token expired messages. New connections cannot be established until the container is recreated from scratch. Root Cause The Kafka client is configured to use … Read more

GPU OOM in inference service after blue‑green deployment on Kubernetes

by
white concrete building under blue sky during daytime

Problem The AI inference microservice runs on GPU‑enabled nodes in a Kubernetes cluster and consumes a high‑throughput Kafka topic. After a blue‑green deployment rollout, the new (green) pods start reporting CUDA out of memory (OOM) errors, causing request failures and a cascade of consumer lag in Kafka. Typical log excerpt from a failing pod: 2026-09-28 … Read more

Kafka producer OAuth2 token acquisition failure after client secret rotation

by
a group of people walking down a street next to tall buildings

Problem After a scheduled rotation of the OAuth2 client secret used by Kafka producers and consumers, the applications began failing to acquire access tokens from the authorization server. The resulting authentication errors caused the following observable symptoms: Kafka client logs contain org.apache.kafka.common.errors.SaslAuthenticationException with messages such as Failed to acquire OAuth token: 401 Unauthorized. Producer metrics … Read more

Kafka consumer group rebalancing timeout during disaster recovery failover

by
white and brown concrete building

Problem Description During a planned disaster‑recovery (DR) failover from the primary Kafka cluster to a secondary replica, the AI model‑training data ingestion pipeline stalls. Batch jobs that consume from Kafka topics exceed their SLA and are terminated with a Job completion timeout. The underlying failure manifests as consumer‑group rebalancing timeouts and growing offset lag. Typical … Read more

Kafka vector index corruption after blue‑green deployment traffic switch

by
a couple of statues sitting on top of a stone ground

Problem – Vector Index Corruption After Blue‑Green Traffic Switch During a blue‑green rollout of a streaming analytics pipeline, the new version of the service began writing vector index files (Avro‑encoded embeddings) to a compacted Kafka topic while the previous version was still running. After the traffic cut‑over, downstream consumers started failing with deserialization errors such … Read more

Kafka video/LiDAR timestamp drift after intermittent network on edge node

by
a group of people walking down a street next to tall buildings

Problem – Timestamp Drift Between Video and LiDAR Streams on an Edge Kafka Node In a low‑power edge deployment a local Kafka broker aggregates two high‑rate topics: video‑frames – 30 fps H.264 payloads, timestamps generated by the camera firmware. lidar‑points – 10 Hz point‑cloud packets, timestamps generated by the LiDAR driver. During intermittent Wi‑Fi outages the following … Read more

Kafka topic creation fails with TopicExistsException after docker‑compose up

by
white concrete building under blue sky during daytime

Problem: Kafka topic creation fails with TopicExistsException after docker‑compose up During integration testing of an AI model inference pipeline, the test harness attempts to create a Kafka topic named inference-logs via the AdminClient. The call consistently returns an error similar to: org.apache.kafka.common.errors.TopicExistsException: Topic ‘inference-logs’ already exists. Despite the exception, the topic is not visible in … Read more

Kafka consumer group JVM heap exhaustion during peak tensor feature streaming

by
a large sculpture in front of a building

Problem Description During peak AI inference traffic, Kafka consumer pods in a Kubernetes cluster begin to terminate with the following errors: java.lang.OutOfMemoryError: Java heap space at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:267) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1152) … java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:698) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:112) at org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:124) … [Consumer clientId=consumer-1, groupId=feature-stream] Failed to allocate memory for record batch Operational impact includes: Consumer … Read more

Kafka consumer fails to process messages due to dimension mismatch

by
a large sculpture in front of a building

Problem – Consumer Crashes on Embedding Dimension Mismatch In a staging deployment of an AI inference pipeline (Kafka 3.3 brokers, Confluent Schema Registry 7.x), the downstream consumer that reads nlp-embeddings messages began throwing SerializationException and shutting down its poll loop. The failure manifested as: org.apache.kafka.common.errors.SerializationException: Size of data 3072 does not match expected fixed size 2048 for … Read more