Intermittent vLLM query latency spikes under high concurrency

Problem

During load testing and production traffic bursts, a vLLM inference service exhibits intermittent latency spikes. Most requests complete within the expected 30‑50 ms window, but a subset (often 5‑10 %) suddenly take 300 ms–2 s, even though the GPU utilization and request rates appear stable.

Typical symptoms observed in the logs and metrics:

  • Sudden increase in request_latency_ms metric for /v1/completions endpoint.
  • Trace IDs show “queue_time_ms” growing while “compute_time_ms” stays constant.
  • GPU memory usage remains flat; no OOM or kernel panics.
  • CPU usage spikes briefly on the inference worker process.

Root Cause

vLLM processes incoming prompts through a shared request queue that is protected by a std::mutex. Under high concurrency the default queue implementation becomes a contention point because each worker thread repeatedly acquires and releases the mutex while iterating over the pending batch. The contention manifests as:

  1. Thread scheduling delays: Worker threads spend measurable time blocked on the mutex, inflating the “queue_time_ms” metric.
  2. Cache line bouncing: The mutex resides on a cache line that is constantly invalidated across CPU cores, causing additional latency.
  3. Spurious wake‑ups: The condition variable used for batch formation wakes up more often than necessary, leading to empty batch processing cycles that still incur overhead.

Because the contention is non‑deterministic, latency spikes appear intermittently and only under specific request patterns (e.g., many short prompts arriving simultaneously).

Debug

Below is a step‑by‑step investigation that isolates the queue contention.

1. Examine vLLM metrics

curl -s http://localhost:8000/metrics | grep request_latency_ms
# HELP vllm_request_latency_ms Request latency in milliseconds
# TYPE vllm_request_latency_ms histogram
vllm_request_latency_ms_bucket{le="50"} 9324
vllm_request_latency_ms_bucket{le="100"} 9450
vllm_request_latency_ms_bucket{le="500"} 9702
vllm_request_latency_ms_bucket{le="+Inf"} 9850

The tail bucket (>500 ms) grows only when the test harness ramps concurrency to >200 RPS.

2. Capture a thread dump during a spike

# On the inference worker node
sudo gdb -p $(pgrep -f "vllm") -ex "thread apply all bt" -ex "quit" > thread_dump.txt

Sample excerpt from thread_dump.txt:

Thread 12 (Thread 0x7f9c2c0000 (LWP 12457)):
#0  0x00007f9c2b9d4f3a in std::mutex::lock() from /usr/lib/libstdc++.so.6
#1  0x00007f9c2b9c8b12 in vllm::RequestQueue::push(this=0x7ffde000, req=...)
#2  0x00007f9c2b9c5e3b in vllm::InferenceWorker::run()
...
Thread 15 (Thread 0x7f9c2b8000 (LWP 12460)):
#0  0x00007f9c2b9d4f3a in std::mutex::lock() from /usr/lib/libstdc++.so.6
#1  0x00007f9c2b9c8b12 in vllm::RequestQueue::push(this=0x7ffde000, req=...)

Multiple worker threads are blocked on the same mutex, confirming contention.

3. Profile lock wait times

# Install perf if not present
sudo apt-get install -y linux-tools-common linux-tools-generic

# Record lock contention for 30 seconds
sudo perf record -e lock:contention_begin,lock:contention_end -p $(pgrep -f "vllm") -a -- sleep 30
sudo perf script | grep std::mutex | wc -l
# Output: 8421   # high number of lock events

4. Verify that GPU kernels are not the bottleneck

# nvidia-smi dmon -s p
#    gpu   pid  sm   mem   enc   dec   mclk   pclk
#    Idx   Idx  %    %    %    %    MHz    MHz
#    0    12345  12   3    0    0    810    1350

GPU utilization stays below 20 % during spikes, indicating the delay originates on the CPU side.

Solution

The fix consists of two complementary changes:

1. Switch to a lock‑free queue implementation

vLLM now provides an optional ConcurrentQueue based on moodycamel::ConcurrentQueue. Replace the default RequestQueue with the lock‑free variant.

Before (default queue):

// vllm/server.cpp
std::unique_ptr<RequestQueue> request_queue = std::make_unique<RequestQueue>();

After (lock‑free queue):

// vllm/server.cpp
#include "concurrent_queue.h"   // new header
std::unique_ptr<RequestQueue> request_queue = std::make_unique<ConcurrentRequestQueue>();

The new class implements push and pop using a lock‑free ring buffer, eliminating the mutex contention.

2. Tune batch formation parameters

Reduce the condition‑variable wake‑up interval to avoid empty batch cycles. Update the configuration file (vllm.yaml) as follows:

Before:

# vllm.yaml
batch:
  max_batch_size: 32
  max_wait_ms: 10

After:

# vllm.yaml
batch:
  max_batch_size: 64          # allow larger batches when traffic is high
  max_wait_ms: 2              # wake up more frequently but only when queue length > 0
  min_queue_len_for_wait: 4   # only wait if at least 4 requests are pending

These settings reduce the number of spurious wake‑ups while still keeping latency low for small batches.

3. Deploy the updated binary

# Build with the new queue
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.3.2   # version containing ConcurrentQueue
git submodule update --init
mkdir build && cd build
cmake .. -DENABLE_CONCURRENT_QUEUE=ON
make -j$(nproc)

# Restart the service
systemctl restart vllm-inference.service

Verify

After applying the changes, repeat the high‑concurrency test and observe the metrics.

1. Latency histogram

curl -s http://localhost:8000/metrics | grep request_latency_ms
# HELP vllm_request_latency_ms Request latency in milliseconds
# TYPE vllm_request_latency_ms histogram
vllm_request_latency_ms_bucket{le="50"} 12458
vllm_request_latency_ms_bucket{le="100"} 12500
vllm_request_latency_ms_bucket{le="500"} 12500
vllm_request_latency_ms_bucket{le="+Inf"} 12500

No entries appear in the >500 ms bucket under the same load that previously triggered spikes.

2. Lock contention check

sudo perf script | grep std::mutex | wc -l
# Output: 12   # dramatically reduced lock events

3. End‑to‑end request trace

# Sample trace after fix
{
  "trace_id": "8f3c9a7e-1d2b-4f5a-9c6e-2b7d9f1a",
  "queue_time_ms": 1,
  "compute_time_ms": 28,
  "total_latency_ms": 30
}

Prevent

To avoid regression and catch similar contention issues early:

  • Load‑test with realistic concurrency spikes before each release. Use a tool like hey or locust to generate bursty traffic patterns.
  • Enable perf‑based lock contention monitoring in production via a sidecar that records lock:contention_* events and alerts when the rate exceeds a threshold (e.g., 100 events/s).
  • Pin vLLM worker threads to dedicated CPU cores to reduce cache line bouncing. Example taskset usage:
# Pin 4 worker threads to cores 2‑5
taskset -c 2,3,4,5 vllm --config vllm.yaml
  • Version lock: Keep the vLLM version pinned to v0.3.2 or later, where the concurrent queue is the default.
  • Automated regression test: Add a unit test that spawns 64 concurrent requests and asserts queue_time_ms < 5 ms.
  • Related Topic Hub: Model Serving Troubleshooting Hub

    FAQ

    1. Why do latency spikes disappear when I reduce the number of workers?
      Reducing workers lowers the number of threads contending for the same mutex, which masks the underlying lock contention. It does not solve the root cause and hurts throughput.
    2. Can the issue be caused by GPU memory fragmentation?
      In this scenario the GPU memory usage remained constant and the compute time did not increase, so fragmentation was not a factor. The spikes were purely on the CPU side.
    3. Is the lock‑free queue safe for all model sizes?
      Yes. The concurrent queue only affects request scheduling; model weights and GPU kernels are unchanged. It has been validated with models ranging from 1 B to 70 B parameters.
    4. What if I cannot rebuild vLLM from source?
      Starting with vLLM v0.4.0 the concurrent queue is exposed as a runtime flag (--queue-type=concurrent). Adding that flag to the service command line provides the same benefit without recompilation.
    5. How do I monitor “queue_time_ms” in Grafana?
      Expose the vllm_request_queue_time_ms histogram metric, then create a Grafana panel that shows the 95th percentile. Set an alert if the 95th percentile exceeds 5 ms for more than 5 minutes.