Kubernetes Pod Crash Loop After Canary Deploying Hugging Face Transformers Model

Q: Problem Description

A canary rollout of a Hugging Face transformers model on a Kubernetes cluster results in the newly created pods entering CrashLoopBackOff. The failure manifests during the model initialization phase, before the inference server becomes ready. Typical log excerpts include:

Q: 3. Extend Probe Timings

Increase initialDelaySeconds to allow the model to load, and raise timeoutSeconds to accommodate the first inference request.

In this article

Table of Contents

Problem Description

A canary rollout of a Hugging Face transformers model on a Kubernetes cluster results in the newly created pods entering CrashLoopBackOff. The failure manifests during the model initialization phase, before the inference server becomes ready. Typical log excerpts include:


2024-08-30T12:15:42.123Z ERROR  [model_loader] RuntimeError: CUDA out of memory. Tried to allocate 12.34 GiB. This is a common symptom when the pod’s memory limit is lower than the model’s RAM requirement.
2024-08-30T12:15:42.130Z ERROR  [model_loader] FileNotFoundError: tokenizer_config.json not found in /model directory
2024-08-30T12:15:45.001Z ERROR  [server] Readiness probe failed: HTTP 500 Internal Server Error
2024-08-30T12:15:46.000Z FATAL  [torch] SIGSEGV (segmentation fault) in libtorch, likely due to GPU driver mismatch.

The pod lifecycle, as described in the Kubernetes documentation, shows the container repeatedly terminated, causing the controller to restart it.

Root Cause Analysis

1. Resource Mismatch

The official Transformers Model Serving Guide recommends allocating at least 2 × model_size RAM for CPU‑only containers and 1.5 × model_size GPU memory when using CUDA. In the canary deployment, the new model (e.g., gpt-2-12b) required ~30 GiB of GPU memory, but the pod spec only requested 16 GiB. This mismatch triggers the CUDA out of memory error and forces the container to exit.

2. Incomplete Model Artifacts

Community reports (Stack Overflow, GitHub issue) show that a canary rollout that updates only the model weights without synchronizing tokenizer files leads to a FileNotFoundError. The init container that pulls the model checkpoint did not copy the tokenizer_config.json and related vocab files, causing the server to abort during startup.

3. Probe Timing

The default liveness/readiness probes in the Hugging Face Docker image perform a health check against /health after a 5 s initial delay. Large models can take 30 s–2 min to load. If the probe timeout is too short (e.g., 10 s), Kubernetes kills the container before the model finishes loading, as documented in the Kubernetes pod lifecycle guide.

4. GPU Driver Incompatibility

One incident involved a node running NVIDIA driver 525 while the Hugging Face Docker image was built against CUDA 11.8. The resulting SIGSEGV in libtorch matches the pattern described in the community issue “GPU driver version mismatch” (GitHub issue).

Investigation and Debugging Steps

Inspect pod events and status

kubectl describe pod hf-canary-abc123 -n ml-serving

Expected output snippet:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Pulling    2m    kubelet, node-1    Pulling image "huggingface/transformers:latest"
  Normal  Pulled     2m    kubelet, node-1    Successfully pulled image
  Warning BackOff    1m    kubelet, node-1    Back-off restarting failed container
  Warning OOMKilled  30s   kubelet, node-1    Container hf-server was OOMKilled

Collect container logs
```
kubectl logs hf-canary-abc123 -c hf-server --previous -n ml-serving
```
Look for the error patterns listed in the problem description.

Check resource allocation

kubectl get pod hf-canary-abc123 -o jsonpath='{.spec.containers[0].resources}' -n ml-serving

Compare against model size requirements from the official guide.

Validate model artifact completeness
```
kubectl exec -it hf-canary-abc123 -c hf-server -- ls /model
```
Missing tokenizer_config.json or vocab.txt indicates an artifact sync issue.
Verify GPU driver compatibility
```
kubectl exec -it hf-canary-abc123 -c hf-server -- nvidia-smi
```
Ensure the driver version matches the CUDA runtime inside the container (check cat /usr/local/cuda/version.txt).
Review probe configuration
```
kubectl get deployment hf-canary -o yaml -n ml-serving | grep -A5 readinessProbe
```
Check initialDelaySeconds, periodSeconds, and timeoutSeconds.

Resolution

1. Adjust Resource Requests and Limits

Update the deployment to request sufficient GPU memory and add a memory limit that matches the model’s footprint.

# Before (insufficient)
resources:
  requests:
    nvidia.com/gpu: "1"
    memory: "16Gi"
  limits:
    memory: "16Gi"

# After (aligned with gpt-2-12b)
resources:
  requests:
    nvidia.com/gpu: "1"
    memory: "32Gi"
  limits:
    memory: "32Gi"

2. Ensure Complete Model Artifacts

Modify the init container to copy the entire model directory, including tokenizer files, or use the --model-repo flag that pulls a complete snapshot.

# init container snippet
initContainers:
- name: model-fetch
  image: huggingface/transformers:latest
  command: ["sh", "-c"]
  args:
    - |
      huggingface-cli download $MODEL_REPO /model --include "pytorch_model.bin tokenizer_config.json vocab.txt"
  volumeMounts:
    - name: model-volume
      mountPath: /model

3. Extend Probe Timings

Increase initialDelaySeconds to allow the model to load, and raise timeoutSeconds to accommodate the first inference request.

# Before
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 2

# After
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 60   # give up to 1 min for model load
  periodSeconds: 15
  timeoutSeconds: 10

4. Align GPU Driver Versions

Rebuild the Docker image with the same CUDA version as the node drivers, or upgrade the node drivers to match the container. Example Dockerfile adjustment:

# Dockerfile fragment
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
# Install transformers and torch compiled for CUDA 11.8
RUN pip install torch==2.2.0+cu118 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip install transformers[torch]==4.40.0

Verification

Redeploy the canary and watch pod status:
```
kubectl rollout status deployment/hf-canary -n ml-serving
```
The rollout should complete without CrashLoopBackOff.

Confirm the readiness probe passes:

kubectl get pod -l app=hf-canary -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

Output should be True for all pods.

Run a test inference:

curl -X POST http://:8080/infer -d '{"inputs":"Hello world"}' -H "Content-Type: application/json"

Response should contain a generated token sequence without HTTP 500 errors.

Monitor memory usage:
```
kubectl top pod -l app=hf-canary -n ml-serving
```
Memory consumption should stay below the defined limit (e.g., ~28 GiB for a 30 GiB model with overhead).

Operational Best Practices and Prevention

Resource Profiling: Before a canary rollout, benchmark the model locally (e.g., using torch.cuda.memory_allocated()) to determine realistic memory requirements.
Artifact Versioning: Store model checkpoints and tokenizer files together in a version‑controlled repository (e.g., Hugging Face Hub) and reference the same tag in both the main and canary deployments.
Gradual Probe Warm‑up: Use initialDelaySeconds proportional to model size (e.g., model_size_GiB × 5 seconds).
GPU Driver Auditing: Automate a CI check that extracts the CUDA version from the Dockerfile and validates it against the node pool’s driver matrix.
Canary Health Checks: Deploy a sidecar that periodically queries the inference endpoint and emits a custom metric (e.g., hf_model_load_success) to Prometheus. Alert on consecutive failures before the rollout proceeds.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the pod crash only during the canary phase and not for existing replicas?
The canary uses a newer model checkpoint that is larger and lacks the tokenizer files. Existing replicas continue to run the previous, smaller model with all required artifacts, so they remain healthy.
How can I determine the exact GPU memory needed for a given model?
Run a short script inside the container that loads the model and prints torch.cuda.max_memory_allocated(). Add a 20 % safety margin before setting the pod memory limit.
Is it safe to set initialDelaySeconds to a very high value?
Yes, but it delays detection of genuine startup failures. A better approach is to implement a custom readiness endpoint that returns 200 only after the model load flag is set.
What should I do if the driver version mismatch persists after rebuilding the image?
Verify the node’s driver version with nvidia-smi, then rebuild the image using the exact CUDA_VERSION environment variable (e.g., FROM nvidia/cuda:${CUDA_VERSION}-runtime-ubuntu22.04). Ensure the CI pipeline pins the driver version.
Can I use CPU‑only pods for large models to avoid OOM?
CPU inference for multi‑billion‑parameter models is typically impractical due to latency and memory pressure. If GPU resources are unavailable, consider model sharding or using a smaller distilled variant for the canary.