woman in black tank top covering face with hands

Hugging Face Transformers endpoint slices return stale data during A/B testing

by

Problem Description

During a production A/B test that routes traffic to two model variants (v1 and v2) via Hugging Face Inference Endpoints, a subset of requests returns predictions from the previous model version. The symptom manifests as:

  • Inconsistent inference results for identical inputs.
  • Log entries such as ModelVersionMismatchError: Expected version 'v2', but serving 'v1'.
  • Cache‑related warnings: CacheStaleError: Response served from stale cache; TTL expired.
  • Observed impact: a fintech company reported a 15 % error rate in sentiment scores for 10 minutes after a rollout.

Root Cause Analysis

The stale responses are caused by a combination of:

  1. Endpoint slice caching. Each slice (a pod behind the load balancer) holds an in‑memory model and an HTTP cache layer. When a new model version is deployed, the slice updates its model files but does not automatically purge the HTTP cache. The Inference API documentation states that cached responses are retained until the Cache‑Control TTL expires.
  2. Routing rule propagation delay. Model versioning and routing rules are stored in the Hub metadata. Changes propagate to the load balancer asynchronously. As documented in the Inference Endpoints quick‑start, routing updates may take up to a minute per slice. During this window, some slices continue to receive traffic for the old version.
  3. Warm‑up replicas. Horizontal scaling creates new pods that load the new checkpoint while existing pods continue serving the previous checkpoint until they are gracefully restarted. This behavior matches the e‑commerce incident where 20 % of traffic hit a warm‑up replica still loading the old model.

Therefore, the stale data originates from stale HTTP cache entries combined with incomplete model version rollout across all endpoint slices.

Investigation and Debugging Steps

Follow these steps to reproduce and isolate the problem:

  1. Inspect routing rules. Retrieve the current routing configuration via the Hub API:
curl -H "Authorization: Bearer $HF_TOKEN" \
     https://huggingface.co/api/endpoints/your-org/your-endpoint/routing

Typical output (JSON) showing traffic split:

{
  "rules": [
    {"model": "v1", "weight": 0.5},
    {"model": "v2", "weight": 0.5}
  ]
}

If the weight for v2 is present but logs still show v1 serving, proceed.

  1. Check slice health and model version. List pods (slices) and query their loaded model hash:
kubectl get pods -n hf-inference -l endpoint=your-endpoint -o jsonpath="{.items[*].metadata.name}"
for pod in $(kubectl get pods -n hf-inference -l endpoint=your-endpoint -o name); do
  echo "$pod:"
  kubectl exec -n hf-inference $pod -- curl -s http://localhost:8080/health
done

Expected health JSON contains "model_hash":"new_hash". Stale pods will still report the old hash.

  1. Validate HTTP cache entries. Capture a request that returns a stale response and inspect the response headers:
curl -v -H "Cache-Control: max-age=0" \
     -X POST https://api-inference.huggingface.co/models/your-org/your-endpoint \
     -d '{"inputs":"Hello world"}'

Look for headers such as:

Cache-Control: max-age=3600
X-Cache-Status: HIT

A HIT indicates the response was served from the endpoint slice cache rather than the newly loaded model.

  1. Examine logs for cache invalidation events. Search the slice logs for the LoadBalancerCacheHit message:
kubectl logs -n hf-inference $(kubectl get pods -n hf-inference -l endpoint=your-endpoint -o name | head -n1) \
    | grep "LoadBalancerCacheHit"

Sample log line:

2024-06-12T14:03:27Z INFO LoadBalancerCacheHit: Served model from previous deployment (model_hash=abc123)

Resolution

The fix consists of three coordinated actions: purge the HTTP cache, enforce immediate routing rule propagation, and ensure graceful model reload across all slices.

1. Force cache invalidation

Use the Hub API to send a cache‑purge request for the endpoint slice:

curl -X POST -H "Authorization: Bearer $HF_TOKEN" \
     https://huggingface.co/api/endpoints/your-org/your-endpoint/purge-cache

Alternatively, set a short Cache-Control TTL in the request headers during the rollout window:

curl -H "Cache-Control: no-cache" ...

2. Apply routing rule update with force flag

Update the routing configuration and request immediate propagation:

# Before (stale)
{
  "rules": [
    {"model": "v1", "weight": 0.5},
    {"model": "v2", "weight": 0.5}
  ]
}
# After (force propagation)
{
  "rules": [
    {"model": "v1", "weight": 0.0},
    {"model": "v2", "weight": 1.0}
  ],
  "propagation": "force"
}

Apply with:

curl -X PATCH -H "Authorization: Bearer $HF_TOKEN" \
     -H "Content-Type: application/json" \
     -d @routing.json \
     https://huggingface.co/api/endpoints/your-org/your-endpoint/routing

3. Graceful rolling restart of slices

Trigger a rolling restart so each pod reloads the new model and clears its in‑memory cache:

kubectl rollout restart deployment/your-endpoint -n hf-inference

This ensures no pod continues to serve the old checkpoint after the restart completes.

Validation

After applying the fixes, perform the following checks:

  1. Confirm routing rules. Re‑run the routing‑rule curl command; the response should show the updated weights and "propagation":"force".
  2. Verify cache headers. A new request should now return X-Cache-Status: MISS and a fresh model_hash matching v2.
  3. Health endpoint consistency. All pods’ /health responses must report the new model hash.
  4. Functional test. Run a deterministic input through the endpoint multiple times; the output should be identical and correspond to the new model version.

Sample successful request:

curl -v -H "Cache-Control: no-cache" \
     -X POST https://api-inference.huggingface.co/models/your-org/your-endpoint \
     -d '{"inputs":"The quick brown fox"}'

Expected header snippet:

Cache-Control: max-age=0
X-Cache-Status: MISS
X-Model-Hash: def456   # matches v2 checkpoint

Prevention and Best Practices

  • Set explicit cache‑control defaults. In production, configure the endpoint to emit Cache-Control: max-age=30 or no-cache during A/B test windows.
  • Use versioned model tags. Deploy models under immutable tags (e.g., v1.0.0, v1.1.0) to avoid “in‑place updates” that can confuse slice caches, as warned in the Hub documentation on updating models in place.
  • Automate rolling restarts. Include a post‑deployment hook that triggers kubectl rollout restart for the endpoint deployment.
  • Monitor cache hit ratio. Create a Grafana panel on the hf_endpoint_cache_hits_total metric; alert if the ratio exceeds a threshold after a version change.
  • Validate routing propagation. After any routing rule change, query the routing API from multiple geographic regions to ensure the update is globally visible within the expected SLA.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why do some requests still hit the old model after I updated the routing rules?
    Because the slice’s HTTP cache was not purged and the load balancer kept routing a fraction of traffic to pods that had not yet reloaded the new model.
  2. Can I disable caching for an entire endpoint?
    Yes. Set the endpoint’s default Cache-Control header to no-cache via the Hub API or include it in every client request during rollout.
  3. How long does it take for routing rule changes to propagate to all slices?
    Typically < 60 seconds, but the exact time depends on the number of slices and the underlying load‑balancer cache. Using the “propagation”:”force” flag forces immediate propagation.
  4. What log entry indicates that a pod is still serving an old model?
    Look for LoadBalancerCacheHit: Served model from previous deployment (model_hash=...) in the slice logs.
  5. Is there a way to verify which model version a given request used?
    Include the X-Model-Hash response header (available when Cache-Control: no-cache is set) and compare it against the expected hash of the target version.