kafka

Kafka authentication errors after secret rotation in GPU cluster

by
a large sculpture in front of a building

Problem – Kafka authentication failures after secret rotation in a GPU compute cluster After rotating the Kafka SASL credentials stored in the cluster‑wide secret, all producers and consumers running on the A100/H100 GPU nodes start reporting authentication errors. The symptoms include connection timeouts, org.apache.kafka.common.errors.SaslAuthenticationException, and intermittent drops of streaming data used for distributed training jobs. … Read more

Kafka producer OAuth2 authentication fails after internet outage in air‑gapped cluster

by
white concrete building under blue sky during daytime

Kafka producer OAuth2 authentication fails after internet outage in an air‑gapped cluster Problem In a production Kafka cluster that is completely air‑gapped, producers and consumers obtain OAuth2 access tokens from an external authorization server (e.g., Keycloak, Auth0). When the corporate internet link goes down, the token acquisition and refresh flow cannot reach the token endpoint, … Read more

Model loading timeout in Docker after Kafka consumer lag increase

by
a group of people walking down a street next to tall buildings

Problem The AI inference service runs inside a Docker container and consumes model artifact messages from a Kafka topic. After a period of increasing consumer lag, the service logs a Model loading timeout error and fails to load the new model version. The container remains healthy, but inference requests continue using the stale model. Root … Read more

Kafka client token refresh failure inside Docker after container restart

by
white and brown concrete building

Problem After a Docker container running a Kafka client is restarted, the client repeatedly fails to authenticate with the broker. The failure manifests as token‑refresh errors and results in Authentication failed: token expired messages. New connections cannot be established until the container is recreated from scratch. Root Cause The Kafka client is configured to use … Read more

GPU OOM in inference service after blue‑green deployment on Kubernetes

by
white concrete building under blue sky during daytime

Problem The AI inference microservice runs on GPU‑enabled nodes in a Kubernetes cluster and consumes a high‑throughput Kafka topic. After a blue‑green deployment rollout, the new (green) pods start reporting CUDA out of memory (OOM) errors, causing request failures and a cascade of consumer lag in Kafka. Typical log excerpt from a failing pod: 2026-09-28 … Read more

Kafka producer OAuth2 token acquisition failure after client secret rotation

by
a group of people walking down a street next to tall buildings

Problem After a scheduled rotation of the OAuth2 client secret used by Kafka producers and consumers, the applications began failing to acquire access tokens from the authorization server. The resulting authentication errors caused the following observable symptoms: Kafka client logs contain org.apache.kafka.common.errors.SaslAuthenticationException with messages such as Failed to acquire OAuth token: 401 Unauthorized. Producer metrics … Read more

Kafka consumer group rebalancing timeout during disaster recovery failover

by
white and brown concrete building

Problem Description During a planned disaster‑recovery (DR) failover from the primary Kafka cluster to a secondary replica, the AI model‑training data ingestion pipeline stalls. Batch jobs that consume from Kafka topics exceed their SLA and are terminated with a Job completion timeout. The underlying failure manifests as consumer‑group rebalancing timeouts and growing offset lag. Typical … Read more

Kafka vector index corruption after blue‑green deployment traffic switch

by
a couple of statues sitting on top of a stone ground

Problem – Vector Index Corruption After Blue‑Green Traffic Switch During a blue‑green rollout of a streaming analytics pipeline, the new version of the service began writing vector index files (Avro‑encoded embeddings) to a compacted Kafka topic while the previous version was still running. After the traffic cut‑over, downstream consumers started failing with deserialization errors such … Read more

Kafka video/LiDAR timestamp drift after intermittent network on edge node

by
a group of people walking down a street next to tall buildings

Problem – Timestamp Drift Between Video and LiDAR Streams on an Edge Kafka Node In a low‑power edge deployment a local Kafka broker aggregates two high‑rate topics: video‑frames – 30 fps H.264 payloads, timestamps generated by the camera firmware. lidar‑points – 10 Hz point‑cloud packets, timestamps generated by the LiDAR driver. During intermittent Wi‑Fi outages the following … Read more

Kafka topic creation fails with TopicExistsException after docker‑compose up

by
white concrete building under blue sky during daytime

Problem: Kafka topic creation fails with TopicExistsException after docker‑compose up During integration testing of an AI model inference pipeline, the test harness attempts to create a Kafka topic named inference-logs via the AdminClient. The call consistently returns an error similar to: org.apache.kafka.common.errors.TopicExistsException: Topic ‘inference-logs’ already exists. Despite the exception, the topic is not visible in … Read more