Kafka authentication errors after secret rotation in GPU cluster

Problem – Kafka authentication failures after secret rotation in a GPU compute cluster

After rotating the Kafka SASL credentials stored in the cluster‑wide secret, all producers and consumers running on the A100/H100 GPU nodes start reporting authentication errors. The symptoms include connection timeouts, org.apache.kafka.common.errors.SaslAuthenticationException, and intermittent drops of streaming data used for distributed training jobs.

Root Cause – Mismatch between rotated secret and client configuration reload

The GPU workloads mount the kafka‑creds secret as a volume in the pod. The Kafka client library reads the credentials at process start and caches them for the lifetime of the JVM. When the secret is updated, Kubernetes performs a atomic volume update, but the running Java processes never re‑read the files. Consequently, the client continues to present the old password/key, while the Kafka brokers have already been re‑configured to accept the new credentials. The authentication handshake therefore fails.

Additional contributing factors:

• Sidecar init containers that copy the secret to /etc/kafka/creds only run on pod start.
• Pod security policies prevent in‑place file modification detection (no inotify watches).
• Some training jobs use kafka‑client version 2.8.0 which does not support automatic credential refresh.

Debug – Investigation steps and evidence

1. Log inspection

2026-10-18 14:32:07,123 ERROR [Producer clientId=producer-1] org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: Invalid username or password
2026-10-18 14:32:07,124 INFO  [Consumer clientId=consumer-2] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Failed to join group due to authentication error

2. Verify secret content on a node

# Inside a running pod
cat /etc/kafka/creds/username
cat /etc/kafka/creds/password

The output shows the previous credentials, while the secret object in Kubernetes contains the new values:

kubectl get secret kafka-creds -n ml-platform -o yaml
...
data:
  username: bXl1c2Vy   # base64 "myuser"
  password: bmV3cGFzcw==   # base64 "newpass"

3. Check broker configuration

# On a broker node
grep -A2 "sasl.jaas.config" /etc/kafka/server.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="myuser" \
  password="newpass";

4. Confirm that the client library does not reload credentials

# Enable DEBUG logging for the client
export KAFKA_LOG4J_OPTS="-Dlog4j.logger.org.apache.kafka=DEBUG"
# Restart the producer (without pod restart) and watch logs – no attempt to re‑read the secret file.

5. Observe pod restart behavior

kubectl rollout restart deployment/ml-training-job -n ml-platform
# After restart, authentication succeeds.

Solution – Enable credential refresh and ensure pods restart on secret change

Option A – Use Kubernetes secret volume with subPath and restartPolicy: OnFailure

Modify the deployment to add an annotation that triggers a rolling restart when the secret’s resourceVersion changes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-training-job
  annotations:
    secret.reloader.stakater.com/reload: "kafka-creds"
spec:
  template:
    spec:
      containers:
      - name: trainer
        image: myorg/trainer:latest
        volumeMounts:
        - name: kafka-creds
          mountPath: /etc/kafka/creds
          readOnly: true
      volumes:
      - name: kafka-creds
        secret:
          secretName: kafka-creds

The secret-reloader sidecar watches the secret and patches the deployment, causing a rolling restart. New pods read the updated files at startup, fixing authentication.

Option B – Enable SASL credential refresh in the client

Upgrade to kafka-clients ≥ 3.0.0 and configure sasl.login.refresh.window.seconds and sasl.login.refresh.min.period.seconds so the client periodically reloads the JAAS file.

Properties props = new Properties();
props.put("bootstrap.servers", "kafka-broker:9092");
props.put("security.protocol", "SASL_PLAINTEXT");
props.put("sasl.mechanism", "PLAIN");
props.put("sasl.jaas.config",
    "org.apache.kafka.common.security.plain.PlainLoginModule required " +
    "username=\"${file:/etc/kafka/creds/username}\" " +
    "password=\"${file:/etc/kafka/creds/password}\";");
props.put("sasl.login.refresh.window.seconds", "300");
props.put("sasl.login.refresh.min.period.seconds", "60");

With the ${file:} placeholder, the client reads the file on each refresh cycle, automatically picking up rotated secrets without pod restarts.

Before / After comparison of deployment manifest

containers:
- name: trainer
  envFrom:
  - secretRef:
      name: kafka-creds

containers:
- name: trainer
  volumeMounts:
  - name: kafka-creds
    mountPath: /etc/kafka/creds
    readOnly: true
  env:
  - name: KAFKA_JAAS_CONFIG
    value: |
      org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="${file:/etc/kafka/creds/username}" \
      password="${file:/etc/kafka/creds/password}";

Verify – Confirm that authentication succeeds after the fix

1. Check pod logs for successful connection

2026-10-18 14:45:12,001 INFO [Producer clientId=producer-1] org.apache.kafka.clients.producer.KafkaProducer - Producer started successfully
2026-10-18 14:45:12,015 INFO [Consumer clientId=consumer-2] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Successfully joined group

2. Run a health‑check script

#!/usr/bin/env bash
set -e
kafka-console-producer.sh --broker-list kafka-broker:9092 \
  --topic healthcheck --producer.config /etc/kafka/producer.properties \
  <<< "ping"
kafka-console-consumer.sh --bootstrap-server kafka-broker:9092 \
  --topic healthcheck --from-beginning --max-messages 1 \
  --consumer.config /etc/kafka/consumer.properties
echo "Kafka auth OK"

Output should be:

ping
Kafka auth OK

3. Verify metric kafka_client_auth_success_total increments

curl -s http://prometheus:9090/api/v1/query?query=kafka_client_auth_success_total

Prevent – Operational safeguards for future secret rotations

• Automated secret‑rotation pipeline: Use kubectl rollout restart or a secret‑reloader sidecar to trigger rolling restarts automatically.
• Client library version policy: Pin Kafka client libraries to a version that supports ${file:} placeholders and credential refresh.
• Monitoring: Alert on SaslAuthenticationException rate exceeding a threshold for more than 2 minutes.
• Testing: Include a CI job that simulates secret rotation in a staging namespace and verifies that producers/consumers reconnect without manual pod restarts.
• Documentation: Record the secret name, mount path, and required client configuration in the platform runbook.

FAQ – Common follow‑up questions

1. Why do some pods authenticate successfully after rotation while others fail? Pods that were restarted (e.g., due to a deployment update) read the new secret at startup, while long‑running pods keep the old cached credentials.
2. Can I avoid rolling restarts for every secret change? Yes, by using the ${file:} syntax together with sasl.login.refresh.* properties (Kafka client ≥ 3.0) or by deploying a sidecar that watches the secret and triggers an in‑process credential reload.
3. What if I cannot upgrade the Kafka client library? As a fallback, wrap the Java process with a script that watches the secret files (using inotifywait) and sends a SIGHUP to the JVM, causing the client to re‑initialize its producer/consumer objects.
4. Do TLS certificates need rotation at the same time? TLS and SASL are independent; however, rotating both simultaneously can mask the root cause. Rotate one component at a time and validate before proceeding.
5. How do I test secret rotation in a non‑production environment? Create a separate namespace, deploy a minimal producer/consumer pair, rotate the secret with kubectl edit secret, and observe whether authentication succeeds without pod restarts.

Related Topic Hub: Data Infrastructure Troubleshooting Hub