photo of computer cables

GPT-3.5 API slice returns stale responses after DR failover

by

Problem

During a planned disaster‑recovery (DR) failover, traffic for the GPT‑3.5 service was rerouted from the primary region (us-east-1) to the backup region (us-west-2). The API slice that serves the text‑completion model continued to return responses that were identical to those generated before the failover, even though the underlying model version had been updated in the backup region. Clients observed:

  • Repeated outputs for distinct prompts.
  • Stale token probabilities when inspecting the logprobs field.
  • Inconsistent behavior between the primary and backup deployments.

Root Cause

The stale responses were traced to a three‑layer caching architecture that persisted across the failover:

  1. Edge CDN cache – configured with a max‑age=86400 header for the /v1/engines/gpt-3.5-turbo/completions path.
  2. Regional Redis shard – used as a request‑level memoization store keyed by prompt+model_version. The shard was replicated from the primary region but the replication lag caused the backup region to read outdated entries.
  3. In‑process LRU cache inside the API gateway (written in Go) that retained the last 10 000 request hashes for performance.

When the DR switch occurred, the edge CDN continued to serve cached responses for up to 24 hours, the Redis replica had not yet synchronized the new model version hash, and the in‑process cache was never flushed because the gateway process was not restarted. Consequently, the request path resolved to the stale cached payload before reaching the fresh model inference service.

Debug

Investigation steps that isolated each caching layer:

1. Log inspection


2026-09-22T14:03:12Z INFO  gateway: incoming request id=7f3a2b9c prompt="Explain quantum tunneling"
2026-09-22T14:03:12Z DEBUG cache: hit (edge) key=hash(7f3a2b9c)
2026-09-22T14:03:12Z DEBUG cache: hit (redis) key=prompt_hash:7f3a2b9c:ver=2026-09-01
2026-09-22T14:03:12Z INFO  response: cached=true latency=12ms

The cached=true flag confirmed that the response never reached the inference backend.

2. CDN header verification


$ curl -I https://api.backup.example.com/v1/engines/gpt-3.5-turbo/completions
HTTP/2 200 
content-type: application/json
cache-control: public, max-age=86400
x-cache: HIT

The x-cache: HIT header indicated a CDN hit.

3. Redis replication lag check


$ redis-cli -h redis-backup.us-west-2.internal INFO replication
# Replication
role:slave
master_host:redis-primary.us-east-1.internal
master_link_status:up
master_last_io_seconds_ago:85

Replication lag of ~85 seconds meant the new model version key (model_version=2026-09-15) had not propagated.

4. In‑process cache dump


$ curl -s http://gateway.internal/debug/cache_dump | jq '.size'
10000

The cache size was at its maximum, confirming no room for new entries.

Solution

The fix required coordinated invalidation of all three caching layers and a change to the cache‑control policy to prevent future stale reads.

1. Edge CDN purge


# Purge all objects under the completions path
curl -X POST "https://cdn.example.com/api/purge" \
     -H "Authorization: Bearer $CDN_TOKEN" \
     -d '{"paths":["/v1/engines/gpt-3.5-turbo/completions*"]}'

2. Redis key eviction

Delete all memoization entries that reference the old model version.


$ redis-cli -h redis-backup.us-west-2.internal --scan --pattern "prompt_hash:*:ver=2026-09-01" | xargs -L1 redis-cli -h redis-backup.us-west-2.internal del

3. Restart API gateway to flush in‑process LRU


# Systemd service restart
systemctl restart gpt-gateway

4. Update cache‑control configuration

Replace the 24‑hour max‑age with a short‑lived directive and add a must‑revalidate directive.

Before After 
cache-control: public, max-age=86400
 
cache-control: public, max-age=60, must-revalidate

5. Deploy a version‑aware cache key

Include the model version hash in the cache key to avoid cross‑version collisions.


// Go snippet (gateway middleware)
func cacheKey(req *http.Request) string {
    body, _ := io.ReadAll(req.Body)
    var payload struct{ Prompt string `json:"prompt"` }
    json.Unmarshal(body, &payload)
    version := os.Getenv("MODEL_VERSION") // e.g. "2026-09-15"
    hash := sha256.Sum256([]byte(payload.Prompt + version))
    return fmt.Sprintf("prompt_hash:%x:ver=%s", hash[:], version)
}

Verify

After applying the changes, the following checks confirmed that fresh responses were being generated:

1. CDN header now shows a miss


$ curl -I https://api.backup.example.com/v1/engines/gpt-3.5-turbo/completions
HTTP/2 200 
content-type: application/json
cache-control: public, max-age=60, must-revalidate
x-cache: MISS

2. Redis no longer contains old version keys


$ redis-cli -h redis-backup.us-west-2.internal --scan --pattern "*:ver=2026-09-01"
(integer) 0

3. Gateway logs show cache miss and backend call


2026-09-22T14:15:03Z INFO  gateway: incoming request id=9c2d4e1a prompt="Explain quantum tunneling"
2026-09-22T14:15:03Z DEBUG cache: miss (edge)
2026-09-22T14:15:03Z DEBUG cache: miss (redis)
2026-09-22T14:15:04Z INFO  inference: model_version=2026-09-15 latency=420ms
2026-09-22T14:15:04Z INFO  response: cached=false latency=432ms

4. Functional test confirms freshness


$ curl -s -X POST https://api.backup.example.com/v1/engines/gpt-3.5-turbo/completions \
     -H "Authorization: Bearer $TOKEN" \
     -d '{"model":"gpt-3.5-turbo","prompt":"What is the current UTC time?","max_tokens":5}'
{
  "id":"cmpl-7a9b...",
  "object":"text_completion",
  "created":1695473702,
  "model":"gpt-3.5-turbo-2026-09-15",
  "choices":[{"text":"2026-09-23T14:15:02Z","index":0,"logprobs":null,"finish_reason":"stop"}]
}

Prevent

To avoid recurrence, the following guardrails were added:

  • Cache versioning: All cache keys now embed the MODEL_VERSION environment variable.
  • Short TTL with revalidation: Edge CDN max‑age reduced to 60 seconds; must-revalidate forces a backend check on each request after TTL expiry.
  • Automated purge on DR events: A CI/CD step that runs the CDN purge and Redis eviction scripts automatically when a failover flag is set.
  • Health‑check endpoint that reports cache synchronization status, e.g.: 
    
GET /health/cache
{
  "cdn_status":"miss",
  "redis_lag_seconds":12,
  "gateway_cache_size":342
}
  • Monitoring alerts for:
    • CDN x-cache: HIT rate > 5 % over a 5‑minute window.
    • Redis replication lag > 30 seconds.
    • Gateway cache hit ratio > 80 % after a model version bump.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why did the CDN continue to serve stale responses after the failover?

    The CDN was configured with a 24‑hour max‑age. Because the origin never sent a Cache‑Control: no‑cache header on model version changes, the CDN considered the cached objects still fresh.

  2. Can I rely on Redis replication to propagate model version updates instantly?

    No. Even in the same cloud provider, network partitions or burst traffic can cause replication lag. Always treat the replica as eventually consistent and design cache keys to be version‑aware.

  3. Is restarting the API gateway sufficient to clear stale in‑process caches?

    Restarting flushes the in‑memory LRU, but it does not affect external caches. A full remediation must address edge and Redis layers as well.

  4. How can I test that the cache‑busting logic works before a real DR event?

    Deploy a temporary model version (e.g., gpt-3.5-turbo-test) and issue a request. Verify that the x-cache header switches from HIT to MISS and that the model_version field in the response matches the test version.

  5. What monitoring metric should I watch to detect stale responses in production?

    Track the ratio of response.model_version to the expected version label from your deployment pipeline. A sudden drop indicates that cached responses are being served.