photo of computer cables

vLLM inference failure after model weights update

by

vLLM Inference Failure After Model Weights Update

Problem Description

After a scheduled model checkpoint rollout, a fleet of vLLM workers began returning errors during request handling. Typical symptoms observed across the cluster were:

  • Log excerpt: 
    
[2024-08-07 10:12:03] ERROR vllm.engine.engine: Failed to load model: weight shape mismatch
Traceback (most recent call last):
  File ".../vllm/engine/engine.py", line 312, in _load_model
    self.model = AutoModelForCausalLM.from_pretrained(self.model_path, **self.load_kwargs)
  File ".../transformers/modeling_utils.py", line 1025, in from_pretrained
    raise RuntimeError(f"Unexpected shape for weight tensor (expected {expected}, got {actual})")
RuntimeError: Unexpected shape for weight tensor (expected [4096, 12288], got [4096, 12320])
  • Segmentation faults on some workers: 
    
[2024-08-07 10:13:41] FATAL vllm.engine.engine: Segmentation fault (core dumped)
```
... Failed to allocate memory for weight cache
```
  • CUDA device‑side asserts: 
    
CUDA error: device-side assert triggered
```
... kernel launch failed after loading checkpoint with dtype bf16 while engine compiled for fp16
```
  • Latency spike of ~30 % as the fallback model was used (production incident on Kubernetes).

These errors match the “Common errors” listed in the evidence package, notably “RuntimeError: Unexpected shape for weight tensor” and “Segmentation fault … Failed to allocate memory for weight cache”.

Root Cause Analysis

The failure originates from a mismatch between the newly deployed checkpoint and the runtime expectations of the vLLM engine:

  1. Model version incompatibility – The new checkpoint was trained with a newer transformer architecture (e.g., additional attention heads) that changes tensor shapes. vLLM 0.2.5, as documented in the Version Compatibility Matrix, only supports checkpoints that match the model configuration used at compile time.
  2. Stale weight cache – vLLM keeps an in‑process weight cache to avoid re‑loading tensors on every request. When the checkpoint files are overwritten without clearing the cache, workers attempt to reuse the old memory layout, leading to “Segmentation fault … stale weight cache” (see GitHub Issue #1983).
  3. CUDA kernel dtype mismatch – The new checkpoint uses bf16 tensors while the engine was started with --dtype fp16. The runtime tries to launch kernels compiled for fp16, causing device‑side asserts (observed in the managed SageMaker incident).
  4. Filesystem race – In edge deployments (vLLM 0.2.5), the service started before the new files were fully synced, resulting in “weight file not found” errors (see the edge deployment incident).

In summary, the update introduced three independent violations of vLLM’s loading contract: shape mismatch, stale cache, and dtype incompatibility.

Investigation and Debugging Steps

Below is a reproducible debugging workflow that was used across the incidents:

1. Verify the checkpoint metadata


$ python -c "
import json, pathlib
meta = json.load(open('model_dir/config.json'))
print('model_type:', meta.get('model_type'))
print('num_attention_heads:', meta.get('num_attention_heads'))
print('hidden_size:', meta.get('hidden_size'))
"

Compare the output with the values used when the engine was launched (see --tensor-parallel-size and --dtype arguments).

2. Inspect vLLM logs for cache state


$ journalctl -u vllm.service -n 200 | grep -i "weight cache"
2024-08-07 10:12:02 vllm[1234]: INFO Weight cache initialized, 8 GiB allocated
2024-08-07 10:12:03 vllm[1234]: ERROR Failed to load model: weight shape mismatch

3. Check CUDA kernel compatibility


$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

$ cat /proc/$(pgrep -f vllm)/cmdline | tr '\0' ' '
/usr/bin/python3 -m vllm.entrypoints.api_server --model model_dir --dtype fp16

4. Capture a packet trace (optional)

When inference crashes after a request, a short tcpdump can confirm whether the client receives a 500 response or the connection is reset.


$ sudo tcpdump -i eth0 -nn -s 0 -w /tmp/vllm.pcap port 8000

5. Reproduce the failure locally

Spin up a single‑GPU container with the same command line and replace the model directory with the new checkpoint. The container logs will surface the same RuntimeError, confirming that the issue is not a cluster‑specific artifact.

Resolution

The fix consists of three coordinated actions: align model metadata, purge the weight cache, and restart the engine with matching dtype.

Step 1 – Align Engine Parameters with the New Checkpoint

Update the launch script to reflect the checkpoint’s configuration. Example before/after:

Before (failing configuration):


#!/bin/bash
# launch_vllm.sh
MODEL_DIR=/models/llama-7b
python -m vllm.entrypoints.api_server \
    --model $MODEL_DIR \
    --tensor-parallel-size 4 \
    --dtype fp16 \
    --max-num-batched-tokens 8192

After (aligned with new checkpoint that uses bf16 and 8 attention heads):


#!/bin/bash
# launch_vllm.sh
MODEL_DIR=/models/llama-7b-v2
python -m vllm.entrypoints.api_server \
    --model $MODEL_DIR \
    --tensor-parallel-size 4 \
    --dtype bf16 \   # changed
    --max-num-batched-tokens 8192 \
    --trust-remote-code   # if needed for custom config

Step 2 – Clear the In‑Process Weight Cache

vLLM stores the cache in shared memory segments identified by VLLM_WEIGHT_CACHE_PATH. Remove the directory before restart:


$ rm -rf /dev/shm/vllm_weight_cache/*
$ systemctl restart vllm.service

Alternatively, set --disable-weight-cache for a one‑off rollout to guarantee a cold load.

Step 3 – Ensure Filesystem Consistency

When overwriting checkpoints on a shared volume, use an atomic rename:


# Deploy new checkpoint to a staging directory
$ cp -r new_checkpoint /mnt/models/llama-7b.tmp

# Atomically replace the live directory
$ mv /mnt/models/llama-7b.tmp /mnt/models/llama-7b

This prevents the “weight file not found” race observed in the edge deployment incident.

Verification

After applying the three steps, verify success with the following checks:

  1. Log health check – No error lines related to weight loading should appear: 
    
$ journalctl -u vllm.service -n 20 | grep -i "Failed"
(no output)
  2. Functional request – A simple curl should return a valid completion: 
    
$ curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"llama-7b-v2","prompt":"Hello"}'
{
  "id":"cmpl-123",
  "object":"text_completion",
  "choices":[{"text":" world!"}]
}
  3. GPU memory usage – Confirm that the weight cache occupies the expected amount (e.g., 12 GiB) without allocation failures: 
    
$ nvidia-smi --query-gpu=memory.used,memory.total --format=csv
0, 12288 MiB, 24576 MiB
  4. Latency baseline – Compare request latency before and after the fix; the 30 % spike should disappear.

Prevention and Best Practices

  • Version pinning – Record the exact vLLM release that was used to train the checkpoint and enforce it in CI/CD pipelines. Refer to the Version Compatibility Matrix.
  • Cache invalidation policy – Automate weight‑cache cleanup on every model rollout. A Kubernetes postStart hook that removes /dev/shm/vllm_weight_cache is a reliable pattern.
  • Atomic checkpoint swaps – Use a staging directory and mv to replace the live model directory, avoiding partial visibility.
  • Health‑check endpoint – Enable vLLM’s /health route and configure alerts for non‑200 responses. This catches load failures early.
  • Monitoring dtype and shape – Export model metadata (e.g., via a side‑car container) and compare it against the engine’s launch arguments using a simple validation script.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the error say “Unexpected shape for weight tensor” instead of “model version mismatch”?
    The engine validates tensor shapes against the configuration derived from the checkpoint’s config.json. If the shapes differ, the lower‑level check raises a RuntimeError before the version compatibility layer is consulted.
  2. Can I keep the weight cache across checkpoint updates?
    Only if the new checkpoint is binary‑compatible (identical architecture, dtype, and tensor shapes). Otherwise the cache must be cleared; otherwise you will see segmentation faults as described in GitHub Issue #1983.
  3. What is the safest way to roll out a new checkpoint in a Kubernetes deployment?
    Use a RollingUpdate strategy with an init container that copies the new checkpoint to a temporary volume, then atomically renames it inside the pod’s shared volume. Include a postStart hook that removes any existing weight‑cache files.
  4. How do I know which dtype the engine is using?
    The engine logs the dtype at startup (e.g., “Running with dtype=fp16”). You can also inspect the process command line with ps -f $(pgrep -f vllm) or query the /info endpoint if enabled.
  5. Is there a way to automatically detect shape mismatches before the service starts?
    Yes. Write a pre‑flight script that loads the checkpoint with transformers.AutoModelForCausalLM.from_pretrained using the same --dtype and compares the resulting model.config against the expected values. Exit with non‑zero status to abort the deployment.