VLLM Token Refresh Fails After Cluster Restoration From Snapshot

Q: Investigation and Debugging Steps

Follow these steps to reproduce the symptom and isolate the failing component.

Q: 1. Inspect vLLM logs for token errors

kubectl logs -n vllm $(kubectl get pods -n vllm -l app=vllm -o jsonpath='{.items[0].metadata.name}') | grep -E "TokenRefreshError|Invalid token" Typical output:

Q: 2. Verify the persisted token cache file

# Exec into the pod kubectl exec -it -n vllm $(kubectl get pod -n vllm -l app=vllm -o jsonpath='{.items[0].metadata.name}') -- bash # Show file timestamp and size ls -l /var/lib/vllm/token_cache.db If the file timestamp predates the snapshot restore, it is likely stale.

Q: 3. Query the session state store directly

# Example for Redis redis-cli -h redis-service -p 6379 HGETALL vllm:session:0x1a2b3c Expected fields include access_token, refresh_token, and expires_at. Compare expires_at with the current time:

Q: 4. Test the refresh endpoint manually

curl -X POST https://vllm.example.com/api/v1/refresh_token \ -H "Authorization: Bearer $(redis-cli -h redis-service HGET vllm:session:0x1a2b3c refresh_token)" \ -d '{"client_id":"svc-vllm"}' -i Typical error response after restore:

Q: Resolution

The fix consists of two coordinated actions: invalidate the stale token cache on start‑up and force a regeneration of session tokens after a snapshot restore.

Q: 1. Add a post‑restore init container to purge token_cache.db

apiVersion: v1 kind: Pod metadata: name: vllm-init spec: initContainers: - name: purge-token-cache image: busybox command: ['sh', '-c', 'rm -f /var/lib/vllm/token_cache.db'] volumeMounts: - name: vllm-data mountPath: /var/lib/vllm containers: - name: vllm image: vllm:latest volumeMounts: - name: vllm-data mountPath: /var/lib/vllm volumes: - name: vllm-data persistentVolumeClaim: claimName: vllm-pvc This ensures the on‑disk cache is rebuilt from fresh credentials on every restore.

Q: 2. Enable automatic session state invalidation

Update the vLLM configuration (vLLM Configuration Reference) to set session_state_store.refresh_on_startup=true. This triggers a sweep that removes any session entries whose expires_at is older than the current time.

In this article

Table of Contents

Problem: vLLM token refresh fails after cluster restoration from snapshot

During disaster‑recovery drills and production failovers, a vLLM cluster restored from a persistent snapshot repeatedly logs errors such as:


TokenRefreshError: token expired
Invalid token in session state
Failed to refresh token: authentication failed
Session token mismatch after restore
Token cache not found or corrupted

All active API requests that depend on refresh_token return HTTP 401, causing downstream inference jobs to stall. The issue appears only after the cluster is started from a snapshot; a fresh deployment works without incident.

Root Cause Analysis

vLLM’s token management follows the lifecycle described in the Token Management Guide. Tokens are cached in two places:

session_state_store – a durable key‑value store (e.g., Redis or etcd) that persists the token associated with each client session.
token_cache – an on‑disk SQLite file (token_cache.db) used by the vLLM process for fast lookup and refresh scheduling.

When a snapshot is taken, both the session_state_store and token_cache.db are persisted. However, the authentication backend (OAuth provider, IAM service, etc.) may issue new tokens after the snapshot is created. Upon restore:

The session_state_store still contains the old token values.


The token_cache.db file is restored verbatim, preserving the original expiry timestamps.

The vLLM process reads the stale entries, attempts a refresh, and the auth service rejects the request because the refresh token has been rotated or revoked.


Official documentation (Disaster Recovery & Snapshot Restore) warns that “stateful components that depend on external credential lifecycles must be re‑initialized after restore.” The community discussion in GitHub Issue #2749 confirms that the stale token cache is the primary failure point.
Investigation and Debugging Steps
Follow these steps to reproduce the symptom and isolate the failing component.
1. Inspect vLLM logs for token errors

kubectl logs -n vllm $(kubectl get pods -n vllm -l app=vllm -o jsonpath='{.items[0].metadata.name}') | grep -E "TokenRefreshError|Invalid token"

Typical output:

2024-07-15T03:12:47Z ERROR TokenRefreshError: token expired for session 0x1a2b3c
2024-07-15T03:12:48Z WARN Invalid token in session state (session_id=0x1a2b3c)
2024-07-15T03:12:49Z ERROR Failed to refresh token: authentication failed (client_id=svc-vllm)

2. Verify the persisted token cache file

# Exec into the pod
kubectl exec -it -n vllm $(kubectl get pod -n vllm -l app=vllm -o jsonpath='{.items[0].metadata.name}') -- bash

# Show file timestamp and size
ls -l /var/lib/vllm/token_cache.db

If the file timestamp predates the snapshot restore, it is likely stale.
3. Query the session state store directly

# Example for Redis
redis-cli -h redis-service -p 6379 HGETALL vllm:session:0x1a2b3c

Expected fields include access_token, refresh_token, and expires_at. Compare expires_at with the current time:

> HGET vllm:session:0x1a2b3c expires_at
"2024-07-10T12:00:00Z"

When the timestamp is in the past, the refresh logic will inevitably fail.
4. Test the refresh endpoint manually

curl -X POST https://vllm.example.com/api/v1/refresh_token \
  -H "Authorization: Bearer $(redis-cli -h redis-service HGET vllm:session:0x1a2b3c refresh_token)" \
  -d '{"client_id":"svc-vllm"}' -i

Typical error response after restore:

HTTP/1.1 401 Unauthorized
{
  "error": "invalid_grant",
  "error_description": "Refresh token has been revoked"
}

Resolution
The fix consists of two coordinated actions: invalidate the stale token cache on start‑up and force a regeneration of session tokens after a snapshot restore.
1. Add a post‑restore init container to purge token_cache.db

apiVersion: v1
kind: Pod
metadata:
  name: vllm-init
spec:
  initContainers:
  - name: purge-token-cache
    image: busybox
    command: ['sh', '-c', 'rm -f /var/lib/vllm/token_cache.db']
    volumeMounts:
    - name: vllm-data
      mountPath: /var/lib/vllm
  containers:
  - name: vllm
    image: vllm:latest
    volumeMounts:
    - name: vllm-data
      mountPath: /var/lib/vllm
  volumes:
  - name: vllm-data
    persistentVolumeClaim:
      claimName: vllm-pvc

This ensures the on‑disk cache is rebuilt from fresh credentials on every restore.
2. Enable automatic session state invalidation
Update the vLLM configuration (vLLM Configuration Reference) to set session_state_store.refresh_on_startup=true. This triggers a sweep that removes any session entries whose expires_at is older than the current time.

# vllm.yaml
session_state_store:
  type: redis
  address: redis-service:6379
  refresh_on_startup: true

token_cache:
  path: /var/lib/vllm/token_cache.db
  max_size: 1000

3. Deploy the updated manifest and restart the cluster

kubectl apply -f vllm-deployment.yaml
kubectl rollout restart deployment/vllm -n vllm

After the rollout, the init container removes the stale cache, and the vLLM process repopulates it using fresh refresh tokens obtained from the auth backend.
Validation
Confirm that token refresh succeeds and the error messages disappear.
Log verification

kubectl logs -n vllm -l app=vllm | grep "TokenRefreshError"

Expected: No lines returned.
Session state sanity check

redis-cli -h redis-service HGETALL vllm:session:0x1a2b3c | grep expires_at

Output should show a future timestamp, e.g., "2024-07-20T14:30:00Z".
API health probe

curl -s -o /dev/null -w "%{http_code}" https://vllm.example.com/healthz

Should return 200.
Prevention and Best Practices

Stateless token cache: Prefer an external cache (Redis) over on‑disk SQLite when snapshots are part of the DR workflow.
Cache invalidation hook: Configure session_state_store.refresh_on_startup or implement a custom init container that always clears token_cache.db after a restore.
Snapshot policy: Exclude token_cache.db from snapshots if the auth backend rotates refresh tokens more frequently than the snapshot interval.
Monitoring: Alert on log patterns “TokenRefreshError” or “Invalid token in session state” with a severity that triggers immediate DR verification.
Testing: Include a “restore‑from‑snapshot” stage in CI pipelines that runs the validation steps above.

Related Topic Hub: Model Serving Troubleshooting Hub
FAQ

Why does the token refresh succeed in a fresh deployment but fail after restore?

   The fresh deployment generates a new token_cache.db and stores fresh tokens. A restored snapshot reuses the old cache, which contains expired or revoked refresh tokens, leading to authentication failures.
Can I keep the token cache in the snapshot if I rotate refresh tokens less often?

   Only if you also snapshot the credential store at the same point in time and guarantee no token rotation occurs between snapshot and restore. Otherwise, invalidate the cache on start‑up.
Is there a way to programmatically detect a stale token cache without restarting the pod?

   Yes. Implement a health‑check endpoint that calls the refresh_token API with a test session. If the response is 401, trigger a pod restart or cache purge.
What error codes does the refresh_token endpoint return for stale tokens?

   According to the API reference, it returns HTTP 401 with {"error":"invalid_grant"} when the refresh token is revoked or expired.
Do other vLLM components (e.g., model cache) need similar handling?

   Model artifacts are immutable and safe to snapshot. Only components that depend on external mutable credentials—such as session_state_store and token_cache—require explicit invalidation.