DeepSeek Token Refresh Failure During Rolling Update Of Inference Pods

In this article

Table of Contents

Problem – Token Refresh Failure During Rolling Update of DeepSeek Inference Pods

During a rolling update of the DeepSeek inference service in a Kubernetes cluster, newly scheduled pods attempt to obtain a fresh OAuth2 access token from the internal token‑issuer service. The request fails, resulting in authentication errors (HTTP 401) and dropped inference requests. The issue is intermittent: only pods that start during the update window experience the failure, while existing pods continue to serve traffic correctly.

Observed Symptoms

Load balancer health checks report 503 Service Unavailable for a subset of pods.
Application logs contain:


2026-10-29T12:04:15.321Z ERROR token_refresh.go:87 failed to refresh token: request failed with status 500: internal server error
2026-10-29T12:04:15.322Z WARN inference_handler.go:45 request dropped due to missing authentication

Metrics show a spike in token_refresh_failure_total during the update.
After the rollout completes, the failure rate drops back to zero.

Root Cause – Race Condition Between Pod Startup and Token‑Issuer Readiness

The DeepSeek inference container starts a background goroutine that immediately calls /token/refresh on the deepseek-token-issuer service. During a rolling update, Kubernetes creates new pods before the token‑issuer deployment has finished rolling out its own updated pods. The service discovery (via DNS) resolves to the cluster IP of the token‑issuer Service, which load‑balances across the current set of endpoint pods.

Because the token‑issuer pods are still initializing, a subset of the Service endpoints return HTTP 500 (or are not yet listening). The inference pod receives a failure response and does not retry, leaving it without a valid token. Subsequent inference requests are rejected by the downstream authentication middleware.

Key factors:

Immediate token refresh on container start without exponential back‑off.
Load balancer does not consider pod readiness probes when routing internal Service traffic.
Readiness probe for the token‑issuer is misconfigured to succeed before the token endpoint is fully operational.

Debug – Investigation Steps

Confirm failing pods:

kubectl get pods -l app=deepseek-inference -o wide | grep -i CrashLoopBackOff

Inspect token‑issuer endpoints during rollout:

kubectl get endpoints deepseek-token-issuer -o yaml

Expected output shows a mix of ready and not‑ready addresses:

subsets:
- addresses:
  - ip: 10.1.2.5
  - ip: 10.1.2.6
  notReadyAddresses:
  - ip: 10.1.2.7
  ports:
  - port: 443

Check token‑issuer logs for 500 responses:

kubectl logs -l app=deepseek-token-issuer -c token-issuer --since=5m | grep "500"

Sample log:

2026-10-29T12:04:14.987Z ERROR token_handler.go:112 failed to load signing key: file not found

Capture network traffic from a newly started inference pod:

kubectl exec -ti $(kubectl get pod -l app=deepseek-inference -o name | head -n1) -- tcpdump -i any -nn -s 0 port 443 -c 5 -w /tmp/capture.pcap
kubectl cp $(kubectl get pod -l app=deepseek-inference -o name | head -n1):/tmp/capture.pcap .

Analyzing the pcap shows HTTP 500 responses from the token‑issuer IPs that are not ready.

Validate readiness probe configuration:

kubectl get deployment deepseek-token-issuer -o yaml | grep -A5 readinessProbe

Current probe:

readinessProbe:
  httpGet:
    path: /healthz
    port: 443
  initialDelaySeconds: 5
  periodSeconds: 10

The /healthz endpoint returns 200 before the /token/refresh endpoint is fully initialized.

Solution – Make Token Refresh Resilient and Align Service Readiness

1. Add Exponential Back‑off and Retry Logic

Modify the token refresh routine to retry on 5xx responses with jitter.

// token_refresh.go (before)
func refreshToken() error {
    resp, err := http.Post(tokenURL, "application/json", nil)
    if err != nil || resp.StatusCode != 200 {
        return fmt.Errorf("failed to refresh token: %v", err)
    }
    // ...
}

// token_refresh.go (after)
func refreshToken() error {
    backoff := 1 * time.Second
    for i := 0; i < 5; i++ {
        resp, err := http.Post(tokenURL, "application/json", nil)
        if err == nil && resp.StatusCode == 200 {
            // success
            return nil
        }
        // log and wait
        log.Printf("token refresh attempt %d failed (status %d, err %v); retrying in %s",
            i+1, resp.StatusCode, err, backoff)
        time.Sleep(backoff + time.Duration(rand.Intn(500))*time.Millisecond)
        backoff *= 2
    }
    return fmt.Errorf("exhausted retries for token refresh")
}

2. Delay Initial Refresh Until Token‑Issuer Readiness

Introduce a start‑up hook that blocks the inference container until the token‑issuer Service reports all endpoints ready.

# wait-for-token-issuer.sh
#!/usr/bin/env bash
set -euo pipefail

SERVICE=deepseek-token-issuer
NAMESPACE=default
RETRIES=30
INTERVAL=5

for ((i=1;i<=RETRIES;i++)); do
    READY=$(kubectl get endpoints $SERVICE -n $NAMESPACE -o jsonpath='{.subsets[0].addresses[*].ip}')
    NOT_READY=$(kubectl get endpoints $SERVICE -n $NAMESPACE -o jsonpath='{.subsets[0].notReadyAddresses[*].ip}')
    if [[ -z "$NOT_READY" && -n "$READY" ]]; then
        echo "All token‑issuer endpoints ready"
        exit 0
    fi
    echo "Waiting for token‑issuer readiness ($i/$RETRIES)..."
    sleep $INTERVAL
done

echo "Timeout waiting for token‑issuer"
exit 1

Add this script as an initContainer in the inference pod spec.

3. Fix Token‑Issuer Readiness Probe

Update the readiness probe to check the actual token endpoint.

# deployment.yaml (before)
readinessProbe:
  httpGet:
    path: /healthz
    port: 443
  initialDelaySeconds: 5

# deployment.yaml (after)
readinessProbe:
  httpGet:
    path: /token/refresh
    port: 443
    httpHeaders:
    - name: X-Health-Check
      value: true
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 3

4. Adjust Rolling Update Strategy

Ensure the token‑issuer rollout completes before inference pods are replaced.

# deepseek-token-issuer deployment
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 0
    maxUnavailable: 1

Set maxSurge: 0 so new token‑issuer pods are added only after old ones are terminated, guaranteeing at least one ready endpoint at all times.

Verify – Confirm the Fix Works

Deploy the updated token‑issuer and inference manifests.
Trigger a rolling update of the inference deployment:

kubectl rollout restart deployment deepseek-inference

Monitor pod status:

kubectl get pods -l app=deepseek-inference -w

All pods should reach Running without CrashLoopBackOff.

Check token refresh metrics:

curl -s http://prometheus:9090/api/v1/query?query=token_refresh_failure_total

The value should remain at zero throughout the rollout.

Inspect logs for successful refresh messages:

kubectl logs -l app=deepseek-inference | grep "token refreshed successfully"

Run an end‑to‑end inference request:

curl -X POST https://deepseek.example.com/v1/infer -d '{"prompt":"Hello"}' -H "Authorization: Bearer $(cat /var/run/secrets/token)"

Response should be a valid inference payload (HTTP 200).

Prevent – Operational Guardrails and Best Practices

Health‑check alignment: Always base readiness probes on the same endpoint that downstream services depend on.
Graceful startup: Use init containers or start‑up scripts to verify critical dependencies before the main process begins.
Retry policies: Implement exponential back‑off for any external call made during container start‑up.
Rolling update ordering: Define maxSurge: 0 for services that act as authentication providers to avoid transient unavailability.
Observability: Export a token_refresh_success_total and token_refresh_failure_total metric; set alerts on failure spikes.
Testing: Include integration tests that simulate a rolling update of the token‑issuer while asserting that a client can still obtain a token.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the token refresh succeed after the rollout but fail during it? The inference pod contacts the token‑issuer Service before all token‑issuer pods are ready. The Service load‑balances to a not‑ready pod that returns 500, causing the refresh to fail.
Can I rely on Kubernetes Service DNS to filter out not‑ready endpoints? No. Service DNS returns all pod IPs; only the kube‑proxy load‑balancer respects readiness. Internal client libraries that bypass kube‑proxy (e.g., direct IP selection) may see not‑ready pods.
Is increasing initialDelaySeconds on the inference pod’s readiness probe sufficient? It reduces the window but does not guarantee that the token‑issuer is fully operational. A dedicated dependency check (init container) is more reliable.
What if the token‑issuer uses mutual TLS? Ensure the client certificate is loaded before the first refresh attempt and that the readiness probe also validates the TLS handshake.
How should I monitor for similar race conditions in other services? Track metrics for external calls made at start‑up, instrument retries, and set alerts on sudden spikes in failure counters during deployments.