Success Criteria No HTTP 500 responses in the test run. Metrics show request_queue_length staying below 1 000. Indexer thread‑pool active count < 80 % of threadPoolSize. Database connection usage < 75 % of maximumPoolSize. Verification Commands

# Verify no 500s in logs journalctl -u deepseek --since "5 minutes ago" | grep "500" # Confirm HPA scaled up kubectl get hpa deepseek # Check gateway queue curl -s http://gateway-admin:8001/services/deepseek/metrics | grep request_queue_length Prevention and Best Practices Capacity Planning: Align request_queue_size, thread‑pool, and DB pool sizes with expected peak RPS plus a safety margin (30‑50 %). Rate Limiting at the Gateway: Enforce per‑client or per‑service limits to avoid sudden bursts overwhelming downstream services. Observability: Export DeepSeek’s custom metrics (indexer_thread_pool_active, db_pool_in_use) to Prometheus and set alerts when they exceed 80 % of capacity. Graceful Degradation: Return a 429 (Too Many Requests) instead of 500 when the gateway queue is full; configure DeepSeek to map Rate limit exceeded to 429 in the error_handler middleware. Autoscaling Warm‑up: Use pre‑emptive scaling (e.g., scheduled HPA) during known traffic windows to reduce lag. Related Topic Hub: LLM Systems Troubleshooting Hub

DeepSeek Collection Creation Fails With 500 Error Under High Traffic

Q: Problem Description

During peak traffic periods the DeepSeek POST /v1/collections endpoint returns intermittent HTTP 500 responses. The failures manifest as timeouts and generic error bodies such as:

Q: Root Cause Analysis

The DeepSeek official API reference states that collection creation triggers a synchronous write to the metadata database and an asynchronous indexing job. Under high concurrency the following resource constraints collide:

Q: Investigation and Debugging

Below is a reproducible debugging workflow that isolates each layer.

Q: 1. Verify API Gateway Queue Metrics

# Example for Kong gateway curl -s http://gateway-admin:8001/services/deepseek/metrics | grep request_queue Expected output when healthy:

Q: 2. Inspect DeepSeek Indexer Thread‑pool

# DeepSeek exposes /metrics via Prometheus curl -s http://deepseek:9090/metrics | grep indexer_thread_pool_active Look for sustained values at the max threshold (e.g., indexer_thread_pool_active 64 when indexer.threadPoolSize=64).

Q: 3. Check Database Connection‑pool Usage

# Assuming PostgreSQL psql -U deepseek -c "SELECT count(*) FROM pg_stat_activity WHERE state='active';" Compare the count against the pool max size defined in application.yml (e.g., maxPoolSize: 30).

Q: 4. Capture a Spike with tcpdump

sudo tcpdump -i eth0 -w /tmp/spike.pcap port 443 and host api-gateway Analyze the pcap to confirm request bursts and any TCP retransmissions.

Q: 5. Review DeepSeek logs around the failure window

journalctl -u deepseek -f | grep -E "CreateCollection|Timeout|Connection pool" Typical interleaved logs:

Q: Resolution

The fix consists of three coordinated changes: increase API gateway capacity, enlarge DeepSeek’s worker and DB pools, and add proactive autoscaling.

Q: 4. Deploy Changes

# Apply gateway config kong reload # Restart DeepSeek with new config kubectl rollout restart deployment/deepseek # Update HPA kubectl apply -f deepseek-hpa.yaml Validation After the rollout, perform a controlled load test that mimics the production spike (e.g., 2 000 concurrent POST /v1/collections requests).

In this article

Table of Contents

Problem Description

During peak traffic periods the DeepSeek POST /v1/collections endpoint returns intermittent HTTP 500 responses. The failures manifest as timeouts and generic error bodies such as:


HTTP/1.1 500 Internal Server Error
Content-Type: application/json

{
  "error": "Internal Server Error: collection creation failed"
}

Observed symptoms include:

Spike in ERROR - CreateCollection - Timeout after 30s log entries.
Log lines indicating resource exhaustion, e.g. Connection pool exhausted – unable to acquire DB connection for collection creation.
Brief periods (5‑30 seconds) where all collection creation requests fail, while read‑only indexing and retrieval continue to operate.
Impact: client applications receive 500 errors during indexing bursts, causing retries, back‑pressure on the API gateway, and degraded overall throughput.

Root Cause Analysis

The DeepSeek official API reference states that collection creation triggers a synchronous write to the metadata database and an asynchronous indexing job. Under high concurrency the following resource constraints collide:

API Gateway Queue Overflow – The gateway’s request queue is bounded (often 1 000–2 000 pending requests). When the queue fills, excess requests are dropped or proxied with a 500 fallback, as reported in the fintech incident where “the API gateway’s request queue overflow caused DeepSeek to return HTTP 500”.
Thread‑pool Exhaustion in the Indexing Service – DeepSeek’s indexing workers are limited by indexer.threadPoolSize. When thousands of POST /v1/collections calls arrive, the pool saturates, leading to log entries such as ERROR - CreateCollection - Timeout after 30s (see GitHub issue #1567).
Database Connection‑pool Saturation – Each collection creation opens a DB transaction to store metadata. The default connection pool (e.g., HikariCP max 30) cannot keep up with the burst, causing Connection pool exhausted – unable to acquire DB connection for collection creation errors.
Autoscaling Lag – In Kubernetes deployments, the Horizontal Pod Autoscaler (HPA) may take 30‑60 seconds to spin up additional pods. During that window the existing pods are overloaded, matching the “autoscaling lag” incident.

These factors combine to produce intermittent 500 responses that appear random but are tightly correlated with traffic spikes and resource limits.

Investigation and Debugging

Below is a reproducible debugging workflow that isolates each layer.

1. Verify API Gateway Queue Metrics


# Example for Kong gateway
curl -s http://gateway-admin:8001/services/deepseek/metrics | grep request_queue

Expected output when healthy:


request_queue_length 0

If the value approaches the configured limit (e.g., 1500), the gateway is the bottleneck.

2. Inspect DeepSeek Indexer Thread‑pool


# DeepSeek exposes /metrics via Prometheus
curl -s http://deepseek:9090/metrics | grep indexer_thread_pool_active

Look for sustained values at the max threshold (e.g., indexer_thread_pool_active 64 when indexer.threadPoolSize=64).

3. Check Database Connection‑pool Usage


# Assuming PostgreSQL
psql -U deepseek -c "SELECT count(*) FROM pg_stat_activity WHERE state='active';"

Compare the count against the pool max size defined in application.yml (e.g., maxPoolSize: 30).

4. Capture a Spike with tcpdump


sudo tcpdump -i eth0 -w /tmp/spike.pcap port 443 and host api-gateway

Analyze the pcap to confirm request bursts and any TCP retransmissions.

5. Review DeepSeek logs around the failure window


journalctl -u deepseek -f | grep -E "CreateCollection|Timeout|Connection pool"

Typical interleaved logs:


2024-07-04T12:03:14.321Z ERROR CreateCollection - Timeout after 30s
2024-07-04T12:03:14.322Z WARN  DBPool - Connection pool exhausted – unable to acquire DB connection for collection creation
2024-07-04T12:03:14.323Z ERROR Indexer - Thread pool exhausted, rejecting collection creation request

Resolution

The fix consists of three coordinated changes: increase API gateway capacity, enlarge DeepSeek’s worker and DB pools, and add proactive autoscaling.

1. Expand API Gateway Queue and Enable Rate‑limiting

Before (Kong example):


# /etc/kong/kong.conf
request_queue_size = 1000

After:


# /etc/kong/kong.conf
request_queue_size = 5000
# Enable per‑service rate limit to smooth spikes
plugins = bundled,rate-limiting

2. Raise DeepSeek Thread‑pool and DB Connection Pool

Before (application.yml):


indexer:
  threadPoolSize: 32
datasource:
  hikari:
    maximumPoolSize: 30

After:


indexer:
  threadPoolSize: 96   # 3× increase, matches CPU cores
datasource:
  hikari:
    maximumPoolSize: 120   # 4× increase, ensure DB can handle
    connectionTimeout: 60000

3. Configure HPA to React Faster

Before:


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: deepseek
spec:
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

After (add custom metric for request queue length and lower cooldown):


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: deepseek
spec:
  minReplicas: 5
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: request_queue_length
      target:
        type: AverageValue
        averageValue: 1000
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 30
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Pods
        value: 4
        periodSeconds: 15

4. Deploy Changes


# Apply gateway config
kong reload

# Restart DeepSeek with new config
kubectl rollout restart deployment/deepseek

# Update HPA
kubectl apply -f deepseek-hpa.yaml

Validation

After the rollout, perform a controlled load test that mimics the production spike (e.g., 2 000 concurrent POST /v1/collections requests).

Success Criteria

No HTTP 500 responses in the test run.
Metrics show request_queue_length staying below 1 000.
Indexer thread‑pool active count < 80 % of threadPoolSize.
Database connection usage < 75 % of maximumPoolSize.

Verification Commands


# Verify no 500s in logs
journalctl -u deepseek --since "5 minutes ago" | grep "500"

# Confirm HPA scaled up
kubectl get hpa deepseek

# Check gateway queue
curl -s http://gateway-admin:8001/services/deepseek/metrics | grep request_queue_length

Prevention and Best Practices

Capacity Planning: Align request_queue_size, thread‑pool, and DB pool sizes with expected peak RPS plus a safety margin (30‑50 %).
Rate Limiting at the Gateway: Enforce per‑client or per‑service limits to avoid sudden bursts overwhelming downstream services.
Observability: Export DeepSeek’s custom metrics (indexer_thread_pool_active, db_pool_in_use) to Prometheus and set alerts when they exceed 80 % of capacity.
Graceful Degradation: Return a 429 (Too Many Requests) instead of 500 when the gateway queue is full; configure DeepSeek to map Rate limit exceeded to 429 in the error_handler middleware.
Autoscaling Warm‑up: Use pre‑emptive scaling (e.g., scheduled HPA) during known traffic windows to reduce lag.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the 500 error only appear during traffic spikes?
Because the underlying resources (gateway queue, thread‑pool, DB connections) are exhausted only when the concurrent request count exceeds their limits.
Can I safely increase the database connection pool without affecting the DB server?
Only if the database instance has sufficient CPU and memory to handle the additional connections. Verify with SELECT max_conn FROM pg_settings; and monitor pg_stat_activity after the change.
What metric should I alert on to catch this issue before it impacts users?
Set alerts on request_queue_length > 80 % of request_queue_size, indexer_thread_pool_active > 85 % of threadPoolSize, and db_pool_in_use > 80 % of maximumPoolSize.
Is returning HTTP 429 preferable to 500 for overloaded collection creation?
Yes. A 429 signals the client to back‑off, while 500 suggests an internal bug. Configure DeepSeek’s error handling middleware to map overload conditions to 429.
Do I need to modify the DeepSeek client SDK when scaling the backend?
No code change is required, but ensure the SDK respects Retry-After headers and implements exponential back‑off to avoid amplifying spikes.