DeepSeek API Server Unreachable During High‑concurrency Model Evaluation

Q: 1. Concurrency throttling

Replace unbounded thread pools or multiprocessing.Pool with a bounded semaphore or an async semaphore.

Q: 2. Exponential back‑off with jitter

Implement retries according to the DeepSeek Rate Limiting Guide, respecting the Retry-After header.

Q: Operational Experience

During the initial investigation we assumed the failure was purely a server‑side outage because the DeepSeek SLA reports 99.9 % uptime. However, the pattern of failures correlated tightly with the number of concurrent sockets, a detail that was missed until ss output was examined. Another misleading symptom was the intermittent ENOTFOUND error, which turned out to be a DNS cache issue rather than a network partition.

In this article

Table of Contents

Problem Description

During large‑scale model evaluation runs, scripts that invoke the https://api.deepseek.com/v1/chat/completions endpoint start failing after a few seconds of sustained traffic. The most common error messages observed are:


Error: connect ETIMEDOUT https://api.deepseek.com/v1/chat/completions
HTTP 502 Bad Gateway – received from DeepSeek load balancer during peak load
HTTP 504 Gateway Timeout – request exceeded server timeout threshold
HTTP 429 Too Many Requests – Rate limit exceeded, retry-after: 30
NetworkError: Failed to fetch – underlying socket closed unexpectedly

Impact includes aborted benchmark suites, incomplete hyper‑parameter sweeps, and CI/CD pipelines terminating with “API server unreachable”. The failure typically appears when the number of concurrent HTTP calls exceeds 200‑300.

Technical Background

DeepSeek provides a public inference API protected by API keys. Key operational characteristics are:

Rate limiting: The service enforces a per‑IP request quota (e.g., 1200 req/min) and a burst limit (e.g., 200 concurrent connections). See the DeepSeek Rate Limiting Guide.
Connection limits: The load balancer caps simultaneous TCP connections per client IP. Exceeding this limit results in connection resets or 502/504 responses (GitHub issue #124).
DNS resolution: DeepSeek uses a global Anycast load balancer. Short TTLs (30 s) are recommended; stale caches can cause ENOTFOUND errors under burst traffic (Reddit discussion 2024‑07‑15).
SDK recommendations: The official SDK suggests using a shared requests.Session (or httpx.AsyncClient) and implementing exponential back‑off with jitter for retries (DeepSeek SDK Usage).

Root Cause Analysis

The “API server unreachable” symptom is a convergence of three independent limits:

Per‑IP concurrent connection ceiling: DeepSeek’s load balancer drops new TCP handshakes once the active connection count for the client IP exceeds the configured burst limit (≈200). This manifests as ETIMEDOUT or 502 Bad Gateway errors.
Rate‑limit exhaustion: The request quota is tracked per minute. When the aggregate request rate of the evaluation job exceeds the quota, the service returns 429 Too Many Requests. Many scripts treat this as a fatal error instead of retrying, causing the “unreachable” message.
DNS TTL misconfiguration: In environments with aggressive DNS caching (e.g., corporate resolvers with TTL = 5 min), stale IP addresses for the Anycast endpoint persist after a load‑balancer rotation, leading to ENOTFOUND or connection resets during the burst window.

These factors combine under high concurrency, as demonstrated by the real incident where a Kubernetes pod generated >300 simultaneous calls and hit DeepSeek’s per‑IP connection limit after ~30 seconds.

Investigation and Debugging Steps

Collect client‑side logs – capture HTTP status codes and exception traces.
Inspect network sockets – use ss -s or netstat -anp to count ESTABLISHED connections to api.deepseek.com:443.
Measure request rate – query Prometheus or use curl -w "%{time_total}" in a loop to compute QPS.
Check DNS TTL – run dig +nocmd api.deepseek.com +noall +answer and note the TTL value.
Validate SDK usage – ensure a single session/client is reused across threads/processes.

Example log excerpt from a failing evaluation run:


2024-09-06T12:15:42.317Z ERROR request_id=7f9c3a2b - connect ETIMEDOUT https://api.deepseek.com/v1/chat/completions
2024-09-06T12:15:42.319Z WARN  request_id=7f9c3a2b - retrying after 30s (attempt 3/5)
2024-09-06T12:15:45.001Z ERROR request_id=8a2d5e1f - HTTP 429 Too Many Requests (retry-after: 28)

Network socket snapshot during the spike:


$ ss -anp | grep 443 | wc -l
312

Resolution

The fix consists of three coordinated changes: limiting concurrency, adding robust retry logic, and ensuring fresh DNS resolution.

1. Concurrency throttling

Replace unbounded thread pools or multiprocessing.Pool with a bounded semaphore or an async semaphore.

# Before (unbounded concurrency)
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=500)
futures = [executor.submit(call_deepseek, payload) for payload in jobs]

# After (bounded to 150 concurrent calls)
import threading
semaphore = threading.Semaphore(150)

def limited_call(payload):
    with semaphore:
        return call_deepseek(payload)

with ThreadPoolExecutor(max_workers=150) as executor:
    futures = [executor.submit(limited_call, p) for p in jobs]

2. Exponential back‑off with jitter

Implement retries according to the DeepSeek Rate Limiting Guide, respecting the Retry-After header.

# Before (single retry, no back‑off)
response = client.post(url, json=data)
if response.status_code != 200:
    raise RuntimeError("API failed")

# After (robust retry)
import time, random, httpx

def post_with_retry(url, json, max_attempts=5):
    attempt = 0
    while attempt < max_attempts:
        resp = httpx.post(url, json=json, timeout=30.0)
        if resp.status_code == 200:
            return resp
        if resp.status_code == 429:
            retry_after = int(resp.headers.get("Retry-After", "30"))
            wait = retry_after + random.uniform(0, 5)
        else:
            wait = (2 ** attempt) + random.uniform(0, 1)
        time.sleep(wait)
        attempt += 1
    raise RuntimeError(f"Failed after {max_attempts} attempts")

3. DNS cache control

Force a short TTL for the DeepSeek endpoint in the resolver configuration or use dns.resolver to re‑resolve before each batch.

# Example using python-dns to refresh IP every minute
import dns.resolver, time

def resolve_deepseek():
    answer = dns.resolver.resolve('api.deepseek.com', 'A')
    return str(answer[0])

current_ip = resolve_deepseek()
while True:
    # use current_ip in HTTP client base URL
    time.sleep(60)
    current_ip = resolve_deepseek()

Validation

After applying the changes, verify the following:

Connection count: ss -anp | grep 443 | wc -l should stay below the burst limit (e.g., < 150).
HTTP status distribution: No 502/504 responses; 429 may appear briefly but is followed by successful retries.
Latency: Average request latency remains within SLA (< 2 s) as measured by curl -w "%{time_total}" in a loop.
Successful evaluation run: All evaluation jobs complete without “API server unreachable” errors.

Sample successful log excerpt:


2024-09-06T12:18:03.112Z INFO  request_id=9b3e7f4c - response 200 OK (latency 1.42s)
2024-09-06T12:18:03.115Z INFO  request_id=9b3e7f4d - response 200 OK (latency 1.38s)
...

Operational Experience

During the initial investigation we assumed the failure was purely a server‑side outage because the DeepSeek SLA reports 99.9 % uptime. However, the pattern of failures correlated tightly with the number of concurrent sockets, a detail that was missed until ss output was examined. Another misleading symptom was the intermittent ENOTFOUND error, which turned out to be a DNS cache issue rather than a network partition.

Best Practices and Prevention

Limit concurrent API calls to ≤ 150 per client IP; use a semaphore or rate‑limiting library.
Implement exponential back‑off with jitter and honor Retry-After headers.
Reuse a single HTTP client/session across threads/processes to reduce TCP handshake overhead.
Configure DNS resolvers with a TTL ≤ 30 s for api.deepseek.com or programmatically refresh the endpoint IP.
Monitor http_requests_total and http_request_duration_seconds metrics; set alerts on spikes in 5xx or 429 responses.
Include a circuit‑breaker (e.g., pybreaker) to pause traffic when error rates exceed a threshold.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why do I see HTTP 502 only under load? DeepSeek’s load balancer enforces a per‑IP concurrent connection limit. Exceeding it forces the balancer to drop new connections, resulting in 502 responses.
How can I tell which request triggered the rate limit? The response includes a Retry-After header. Log the request ID together with the status code; the first 429 in a burst indicates the limit breach.
Is increasing the API key quota a solution? No. The quota is per‑IP, not per‑key. You must reduce concurrency or distribute traffic across multiple egress IPs (e.g., NAT gateways).
Do async batch inference calls avoid the issue? Not automatically. If the batch creates many parallel HTTP streams, the same connection ceiling applies. Use a bounded async semaphore to limit parallelism.
What DNS settings should I use? Ensure your resolver respects the 30 s TTL advertised by DeepSeek. If you cannot control the resolver, programmatically re‑resolve the endpoint before each batch.