Kubernetes secret rotation failure for multi-GPU training pods

Problem: Kubernetes Secret Rotation Failure for Multi‑GPU Training Pods

In a production AI training platform, each training job runs in a pod that requests multiple GPUs. Access to the GPU resource provider (e.g., NVIDIA GPU Cloud, a proprietary licensing server, or a cloud‑native GPU quota service) is gated by short‑lived credentials stored in a Secret. The platform rotates these credentials every 24 hours via a controller that updates the Secret and triggers a rolling restart of the pods.

When rotation occurs, the pods continue to use the previous credential set. The GPU provider rejects the request with an authentication error, causing the pod to fail to acquire GPUs and abort the training run.

Observed Symptoms

• Training pod logs contain GPUProviderError: authentication failed after a secret rotation window.
• Pod events show FailedMount or CrashLoopBackOff without any change to the pod spec.
• Metrics: sudden drop in gpu_allocation_success_total and spike in gpu_allocation_failure_total at the rotation time.
• kubectl describe pod shows the secret volume mounted but the file content matches the previous version.

Sample Log Excerpts

2024-06-25T03:12:45Z ERROR trainer[pid=12345]: GPUProviderError: authentication failed (code=401)
2024-06-25T03:12:45Z INFO  trainer[pid=12345]: Attempting to acquire 4 GPUs from provider
2024-06-25T03:12:46Z WARN  kubelet[pid=6789]: Container runtime failed: containerd: failed to create task for container "trainer": failed to mount secret "gpu-credentials"

Root Cause

The failure stems from the interaction between the Secret update mechanism and the ProjectedVolume cache inside the kubelet. When the controller patches the Secret, the API server updates the object, but the kubelet only reloads the volume content when the pod is restarted or when the secret volume is explicitly refreshed. Multi‑GPU pods are typically long‑running (hours to days) and are not restarted during rotation, so they continue to read the stale file from the pod’s in‑memory filesystem.

Additional contributing factors:

• Pod spec uses volumeMounts with readOnly: true and does not set subPath, causing the entire directory to be cached.
• The rotation controller patches the Secret without setting the metadata.resourceVersion annotation that triggers a Secret volume refresh.
• GPU driver sidecar containers cache the credential file path at start‑up and never re‑read it.

Debug: Investigation Steps

1. Confirm secret update:

   
   kubectl get secret gpu-credentials -o yaml
   

   Check metadata.resourceVersion before and after rotation; it should increment.

2. Inspect pod volume mounts:

   
   kubectl exec -it trainer-pod -- cat /etc/gpu/credentials.json
   

   Compare the file content with the latest secret data (base64‑decoded).

3. Check kubelet volume cache logs:

   
   journalctl -u kubelet | grep "secret volume" | tail -n 20
   

   Look for messages like Skipping update for secret gpu-credentials (already mounted).

4. Verify sidecar credential loading:

   
   kubectl logs trainer-pod -c gpu-driver-sidecar | grep "Loading credentials"
   

   If the sidecar logs “Credentials loaded at startup”, it never re‑reads the file.

5. Simulate manual rotation: Delete the pod to force a restart and observe if the new secret is used successfully.

   
   kubectl delete pod trainer-pod
   kubectl wait --for=condition=Ready pod/trainer-pod --timeout=120s
   kubectl exec -it trainer-pod -- cat /etc/gpu/credentials.json
   

   Success after restart confirms the secret itself is correct.

Solution: Reliable Secret Rotation for Long‑Running Multi‑GPU Pods

1. Enable Automatic Volume Refresh

Add the secret volume annotation kubernetes.io/secret-reload (supported in K8s 1.22+) so the kubelet watches for resourceVersion changes and updates the mounted file without pod restart.

apiVersion: v1
kind: Pod
metadata:
  name: trainer-pod
spec:
  containers:
  - name: trainer
    image: myorg/trainer:latest
    volumeMounts:
    - name: gpu-cred
      mountPath: /etc/gpu
  volumes:
  - name: gpu-cred
    secret:
      secretName: gpu-credentials
      optional: false
    # New annotation to trigger reload
    annotations:
      kubernetes.io/secret-reload: "true"

2. Use Init Container to Copy Credentials into a Shared EmptyDir

Copy the secret into an emptyDir at pod start and have the sidecar watch the file for changes (e.g., via inotify). This decouples the driver from the secret cache.

apiVersion: v1
kind: Pod
metadata:
  name: trainer-pod
spec:
  initContainers:
  - name: copy-cred
    image: busybox
    command: ["/bin/sh", "-c", "cp /secret/credentials.json /shared/credentials.json"]
    volumeMounts:
    - name: gpu-cred
      mountPath: /secret
    - name: shared
      mountPath: /shared
  containers:
  - name: trainer
    image: myorg/trainer:latest
    volumeMounts:
    - name: shared
      mountPath: /etc/gpu
  - name: gpu-driver-sidecar
    image: myorg/gpu-driver:latest
    env:
    - name: CRED_PATH
      value: /etc/gpu/credentials.json
    volumeMounts:
    - name: shared
      mountPath: /etc/gpu
  volumes:
  - name: gpu-cred
    secret:
      secretName: gpu-credentials
  - name: shared
    emptyDir: {}

3. Patch the Rotation Controller to Add an Annotation Update

When the controller updates the secret, also add a dummy annotation that forces the pod’s projected volume to refresh.

# Pseudo‑code for the controller
secret = k8s_client.read_namespaced_secret(name="gpu-credentials", namespace="ml")
secret.data["token"] = new_token_base64
# Force a pod restart via annotation (optional if using secret-reload)
secret.metadata.annotations = {"rotation-timestamp": str(int(time.time()))}
k8s_client.replace_namespaced_secret(name="gpu-credentials", namespace="ml", body=secret)

4. Ensure Sidecar Reacts to File Changes

Modify the sidecar entrypoint to watch the credential file and reload the driver without exiting.

#!/bin/sh
CRED=/etc/gpu/credentials.json
while true; do
  inotifywait -e modify "$CRED"
  echo "$(date) Reloading GPU driver credentials"
  /usr/local/bin/gpu-driver --reload-credentials "$CRED"
done

Verification: Confirm Rotation Works End‑to‑End

1. Trigger a rotation manually:

   
   kubectl patch secret gpu-credentials -p '{"data":{"token":"'"$(echo -n newtoken | base64)"'"}}'
   

2. Observe the pod’s credential file update:

   
   kubectl exec -it trainer-pod -- cat /etc/gpu/credentials.json
   

   Output should reflect the new token without pod restart.

3. Check driver sidecar logs for reload events:

   
   kubectl logs trainer-pod -c gpu-driver-sidecar | grep "Reloading GPU driver credentials"
   

4. Validate GPU allocation succeeds:

   
   kubectl logs trainer-pod -c trainer | grep "GPU allocation successful"
   

5. Confirm metrics reset:

   
   curl -s http://prometheus:9090/api/v1/query?query=gpu_allocation_failure_total
   

   Failure count should remain unchanged after rotation.

Prevention: Operational Guardrails

• Enable secret‑reload annotation by default on all pods that consume rotating credentials.
• Adopt a sidecar pattern that watches credential files and reloads dependent services.
• Instrument Prometheus alerts on gpu_allocation_failure_total spikes combined with secret_rotation_success_total to catch regressions early.
• Automated integration test that simulates a secret rotation in a staging cluster and verifies that long‑running pods pick up the new values.
• Document the rotation contract (TTL, required annotations, sidecar behavior) in the platform runbook.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the secret update succeed in the API server but the pod still sees the old value?
   Because the kubelet caches the mounted secret file until the pod restarts or the secret-reload annotation triggers a refresh. Without either, the in‑memory view remains stale.
2. Can I use kubectl rollout restart on a Deployment to force a secret refresh?
   Yes, but it disrupts all running training jobs. The sidecar‑watcher pattern or secret‑reload annotation provides a non‑disruptive path.
3. Do I need to restart the GPU driver process after a credential change?
   Only if the driver does not support hot‑reloading. The provided sidecar script uses the driver’s --reload-credentials flag; otherwise a process restart is required.
4. What Kubernetes version introduced the kubernetes.io/secret-reload annotation?
   The feature graduated to GA in Kubernetes 1.22. Earlier versions require a manual pod restart or a custom controller.
5. How can I test that my sidecar correctly detects credential changes?
   Inject a temporary change to the secret (e.g., change a dummy key) and watch the sidecar logs for the inotify‑triggered reload message.