DeepSeek tool-use parsing failure after Kubernetes autoscaling deployment

Problem

When DeepSeek is deployed on a Kubernetes‑based inference platform that uses horizontal pod autoscaling (HPA) and a middleware layer for routing tool calls, the model intermittently fails to parse tool‑use commands embedded in user prompts. The result is either a malformed API request or the complete omission of the tool invocation, causing downstream failures in the application that relies on those tool calls.

Root Cause

The parsing failure originates from a race condition introduced by the autoscaling lifecycle:

  • The middleware receives a request payload, extracts [[TOOL:...] ] tokens, and stores the parsed representation in an in‑memory cache.
  • When HPA scales the deployment, new pods start with a fresh cache while old pods are terminated.
  • The cache implementation uses a non‑thread‑safe map[string]string without proper synchronization. Concurrent reads/writes during pod termination lead to partial or corrupted entries.
  • Corrupted cache entries cause the parser to emit incomplete token streams, which downstream code interprets as a normal text response, dropping the tool call entirely.

Additionally, the pod readiness probe only checks HTTP 200 on the health endpoint, not the health of the parsing subsystem, allowing a pod to become “ready” while its cache is in an inconsistent state.

Debug

1. Observe the symptom in logs


2026-11-14T08:12:31.842Z WARN  parser.go:124  Tool token parsing failed: unexpected EOF while reading token
2026-11-14T08:12:31.845Z INFO  request_handler.go:58  Received user prompt: "Find the latest sales report [[TOOL:fetch_report id=1234]]"
2026-11-14T08:12:31.846Z INFO  middleware.go:212  Forwarding request to model pod: deepseek-infer-7f9c9d5c5b-9kz2l
2026-11-14T08:12:31.849Z ERROR model_server.go:87  Model response missing tool invocation
2026-11-14T08:12:31.850Z INFO  response_builder.go:45  Sending fallback text response to client

2. Correlate with autoscaling events


kubectl get hpa deepseek-infer -n ai-platform
NAME            REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
deepseek-infer  Deployment/deepseek-infer  85%/80%   2         10        5          12h

kubectl describe hpa deepseek-infer -n ai-platform | grep -A3 "Events"
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 5; reason: cpu utilization above target
  Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  New size: 6; reason: cpu utilization above target

3. Inspect the in‑memory cache state


# Attach to a running pod
kubectl exec -it deepseek-infer-7f9c9d5c5b-9kz2l -n ai-platform -- bash

# Run a tiny Go program that prints the cache size (for illustration)
go run /opt/deepseek/tools/cache_inspector.go
Cache entries: 1243 (expected ~1250)
Corrupted entries: 7

4. Verify that the cache implementation lacks synchronization


// middleware/cache.go (before fix)
type TokenCache struct {
    store map[string]string // NOT thread‑safe
}

// Get token
func (c *TokenCache) Get(key string) (string, bool) {
    val, ok := c.store[key]
    return val, ok
}

// Set token
func (c *TokenCache) Set(key, value string) {
    c.store[key] = value
}

Solution

1. Replace the unsafe map with a concurrent safe structure


// middleware/cache.go (after fix)
import "sync"

type TokenCache struct {
    mu    sync.RWMutex
    store map[string]string
}

// NewCache creates a thread‑safe cache
func NewCache() *TokenCache {
    return &TokenCache{
        store: make(map[string]string),
    }
}

// Get token – read lock
func (c *TokenCache) Get(key string) (string, bool) {
    c.mu.RLock()
    defer c.mu.RUnlock()
    val, ok := c.store[key]
    return val, ok
}

// Set token – write lock
func (c *TokenCache) Set(key, value string) {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.store[key] = value
}

2. Add graceful shutdown handling to flush the cache


func (c *TokenCache) Flush() {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.store = make(map[string]string)
}

// In main.go – capture SIGTERM
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)
go func() {
    <-stop
    tokenCache.Flush()
    os.Exit(0)
}()

3. Extend the readiness probe to verify cache health


# /etc/kubernetes/healthz.sh
#!/bin/bash
# Simple health check that also validates cache entry count
CACHE_COUNT=$(curl -s http://localhost:8080/internal/cache/size)
if [[ "$CACHE_COUNT" -gt 0 && "$CACHE_COUNT" -lt 2000 ]]; then
    exit 0
else
    exit 1
fi

Update the pod spec:


readinessProbe:
  exec:
    command: ["/bin/bash", "/etc/kubernetes/healthz.sh"]
  initialDelaySeconds: 5
  periodSeconds: 10

Verify

Functional verification


curl -X POST http://deepseek-api.ai-platform/v1/chat \
  -d '{"prompt":"Summarize the quarterly results [[TOOL:fetch_report id=5678]]"}' \
  -H "Content-Type: application/json"

# Expected fragment in response
{
  "answer": "...",
  "tool_result": {
    "report_id": "5678",
    "content": "..."
  }
}

Log confirmation


2026-11-14T09:03:12.210Z INFO  parser.go:45  Parsed tool token: fetch_report id=5678
2026-11-14T09:03:12.215Z INFO  tool_executor.go:78  Invoked fetch_report API, received 200 OK

Load test after scaling


# Simulate a burst that triggers HPA to scale to 8 replicas
kubectl apply -f load.yaml

# After scaling, repeat the functional request on multiple pods
for pod in $(kubectl get pods -l app=deepseek-infer -o name); do
    kubectl exec $pod -- curl -s http://localhost:8080/healthz
done
# All pods return HTTP 200, indicating healthy cache

Prevent

  • Always use concurrency‑safe containers (e.g., sync.RWMutex, sync.Map) for shared mutable state.
  • Include health checks that validate internal subsystem state, not just HTTP reachability.
  • Configure HPA termination grace period (>30s) to allow middleware to flush caches on SIGTERM.
  • Instrument cache hit/miss and corruption metrics (e.g., Prometheus counters) and set alerts for sudden spikes in cache_corruption_total.
  • Run integration tests that simulate pod churn: start/stop pods while sending tool‑use prompts to verify parser stability.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the parsing error appear only after a scale‑out event? The new pods start with empty caches, while terminating pods may still be processing requests. The unsafe map allows concurrent writes during this transition, corrupting entries that the parser later reads.
  2. Can I keep the original map implementation and avoid the race? Only if you add external synchronization (e.g., a global lock) or move the cache to an external store (Redis, etc.) that guarantees atomicity. The simplest fix is to use a built‑in thread‑safe structure.
  3. Do I need to change the HPA configuration? No direct change is required, but increasing terminationGracePeriodSeconds and ensuring the readiness probe checks cache health reduces the window where corrupted state can be served.
  4. How can I monitor for future cache corruption? Export a Prometheus gauge that reports the number of entries flagged as corrupted during parsing. Alert when the rate exceeds a threshold (e.g., >1 per minute).
  5. Is the issue specific to DeepSeek, or does it affect other LLM deployments? Any deployment that caches parsed tool tokens in a non‑thread‑safe container will exhibit similar symptoms under rapid pod churn. The pattern is generic to stateful middleware in autoscaling environments.