A close up of a cell phone on a table

Intermittent Mistral AI query latency spikes during high concurrency batch jobs

by

Problem: Intermittent Mistral AI Query Latency Spikes During High‑Concurrency Batch Jobs

During nightly batch processing, a fleet of GPU‑accelerated inference servers receives up to 10,000 concurrent requests per batch. The SLA for a single query is ≤ 150 ms, yet certain batches exhibit response times of 600‑800 ms, a 4‑5× increase. The latency spikes are non‑deterministic: the same batch run twice can show normal latency on one run and severe spikes on the next.

Observed Symptoms

  • Latency histogram shows a long tail only when concurrency > 8,000.
  • Load balancer (Envoy) logs report upstream_response_time of 0.6s‑0.8s for a subset of requests.
  • GPU utilization hovers at 70‑80 % even during spikes; CPU usage rises to 95 % on the inference host.
  • Application logs contain occasional torch.cuda.OutOfMemoryError warnings that are caught and retried.
  • Prometheus metric mistral_inference_latency_seconds_bucket shows a sudden shift in the 0.5‑s bucket during spikes.

Root Cause

The latency spikes originate from a combination of:

  1. GPU memory fragmentation caused by variable‑length input tensors. When many requests arrive simultaneously, the CUDA allocator cannot find contiguous blocks, triggering internal memory compaction that stalls the kernel launch pipeline.
  2. Thread‑pool starvation on the inference server. The default torch::jit::ThreadPool size is tied to the number of CPU cores (8 on the instance). Under high concurrency the request dispatcher queues tasks faster than the pool can execute them, leading to queuing delays that appear as latency spikes.
  3. Load balancer connection‑reuse limits. Envoy’s max_requests_per_connection is set to 1000. When the limit is reached, Envoy tears down the TCP connection, forcing a new TLS handshake for the next request. The handshake adds ~150 ms latency, which aggregates into the observed tail.

These three factors interact: memory fragmentation slows kernel execution, which keeps CPU threads occupied longer, increasing the chance that Envoy reaches its connection limit, thereby adding extra network latency.

Debug: Investigation Steps

1. Capture latency distribution

curl -s http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99, sum(rate(mistral_inference_latency_seconds_bucket[5m])) by (le))

Result shows 99th‑percentile latency jumping from 0.13s to 0.68s when concurrency exceeds 8k.

2. Examine GPU memory usage

nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv -l 5

Typical output during a spike:

2026 MiB, 1024 MiB, 3072 MiB
2028 MiB, 1022 MiB, 3072 MiB
...

Notice the “used” value fluctuates without reaching the total, indicating fragmentation rather than exhaustion.

3. Enable CUDA memory diagnostics

export TORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:64
python -c "import torch; torch.cuda.memory_summary(device=0, abbreviated=False)"

The summary reports “large allocation failed” warnings that correspond to the timestamps of latency spikes.

4. Verify thread‑pool saturation

ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -n 10

During spikes, the inference process shows %CPU 92 with many threads in R (running) state, confirming thread‑pool bottleneck.

5. Inspect Envoy connection metrics

curl -s http://envoy-admin:9901/stats | grep max_requests_per_connection

Output:

listener.0.0.0.0_443.downstream_cx_total: 124578
listener.0.0.0.0_443.downstream_cx_destroy_local_with_active_rq: 312

High “destroy_local_with_active_rq” indicates forced connection teardown.

Solution: Mitigations and Code Changes

1. Reduce GPU memory fragmentation

Enable PyTorch’s memory_format=torch.channels_last and pre‑allocate a fixed buffer pool.

Before:

def infer(request):
    input_tensor = torch.tensor(request["data"]).to(device)
    return model(input_tensor).cpu().numpy()

After:

import torch
from torch.cuda import amp

# Pre‑allocate a pool of 256‑MiB buffers
BUFFER_POOL = [torch.empty((1, 1024), dtype=torch.float16, device="cuda") for _ in range(64)]

def infer(request):
    # Reuse a buffer to avoid new allocations
    buf = BUFFER_POOL.pop()
    try:
        # Copy data without allocating new memory
        buf.copy_(torch.tensor(request["data"], dtype=torch.float16, device="cuda"))
        with amp.autocast():
            out = model(buf)
        return out.cpu().numpy()
    finally:
        BUFFER_POOL.append(buf)

This eliminates per‑request allocations, keeping the CUDA allocator from fragmenting.

2. Increase inference server thread pool

Modify the server start‑up script to set OMP_NUM_THREADS and TORCH_NUM_THREADS to the number of physical cores × 2.

Before:

python serve.py --model mistral-7b

After:

export OMP_NUM_THREADS=16
export TORCH_NUM_THREADS=16
python serve.py --model mistral-7b --worker_threads=32

The --worker_threads flag expands the internal request dispatcher pool, allowing more concurrent kernel launches.

3. Adjust Envoy connection limits

Update the Envoy listener configuration:

{
  "name": "listener_0",
  "address": {"socket_address": {"address": "0.0.0.0", "port_value": 443}},
  "filter_chains": [...],
  "listener_filters": [...],
  "per_connection_buffer_limit_bytes": 32768,
  "max_requests_per_connection": 5000   // increased from 1000
}

Reload Envoy without downtime:

curl -X POST http://envoy-admin:9901/quitquitquit
# Or use hot‑restart:
envoy -c /etc/envoy.yaml --restart-epoch 1 --service-cluster my‑cluster

4. Enable request‑level back‑pressure

Introduce a token‑bucket limiter in the batch orchestrator to cap concurrent requests at 8,000.

import asyncio
from aiolimiter import AsyncLimiter

limiter = AsyncLimiter(max_rate=8000, time_period=1)

async def send_batch(requests):
    async with limiter:
        await asyncio.gather(*[client.post("/infer", json=req) for req in requests])

This prevents the system from exceeding the threshold where the three failure modes align.

Verify: Post‑Fix Validation

  1. Latency histogram – Re‑run the same batch and query Prometheus: 
    curl -s http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99, sum(rate(mistral_inference_latency_seconds_bucket[5m])) by (le))

    Expected 99th‑percentile ≤ 0.18s.

  2. GPU memory fragmentation – Run torch.cuda.memory_summary before and after a batch; the “largest free block” should stay > 200 MiB.
  3. Thread‑pool metrics – top -H -p $(pgrep -f serve.py) should show average %CPU ~60 % with no long‑running queues.
  4. Envoy connection churn – curl http://envoy-admin:9901/stats | grep downstream_cx_destroy should drop to near zero.
  5. End‑to‑end test – Execute a synthetic load test (e.g., hey -c 8000 -n 20000 http://api.mistral/v1/infer) and confirm no request exceeds 200 ms.

Prevent: Operational Guardrails

  • Monitoring: Set alerts on mistral_inference_latency_seconds_bucket{le="0.5"} < 0.90 and on envoy.listener.0.0.0.0_443.downstream_cx_destroy_local_with_active_rq > 10 per minute.
  • Capacity planning: Keep the max_concurrent_requests below 80 % of the GPU memory pool size (e.g., 7,500 for a 3 GB GPU).
  • Configuration drift detection: Use a CI pipeline to validate that max_requests_per_connection and thread‑pool sizes match the production baseline.
  • Periodic stress test: Run a nightly locust script that ramps to 10k concurrent requests and verifies latency stays within SLA.
  • Graceful degradation: Implement a fallback path that returns a cached response when the inference server reports GPU OOM or thread‑pool saturation.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the latency spike only under certain batches and not others?

    The spike occurs when three thresholds are crossed simultaneously: GPU memory fragmentation, thread‑pool saturation, and Envoy connection limit. Small batches stay below at least one of those thresholds, so they remain fast.

  2. Can increasing GPU memory solve the problem?

    Adding memory reduces fragmentation risk but does not address the root cause of per‑request allocations. The recommended fix is to reuse buffers and limit concurrency.

  3. Is the max_requests_per_connection setting safe to increase arbitrarily?

    Higher values reduce TLS handshake overhead but increase the risk of a single faulty connection affecting many requests. A value of 5,000 balances reuse with isolation; any increase should be validated with load testing.

  4. How do I know if the thread‑pool size is still a bottleneck after the change?

    Inspect the metric process_cpu_seconds_total and the per‑thread CPU usage. If the average queue length in the dispatcher remains near zero and CPU utilization stays below 80 %, the pool is sufficient.

  5. Do I need to change the client library that sends requests?

    No code change is required on the client side, but applying a client‑side rate limiter (as shown) helps keep the server within safe operating bounds.