VLLM RAG Reranker Timeout During Multi-GPU Training

Q: 1. Increase scheduler capacity and request timeout

Adjust the launch arguments to raise max_num_requests and request_timeout_ms:

Q: 3. Pin reranker to dedicated GPU memory pool

Use vLLM’s --reranker-gpu-id (hypothetical flag) or manually allocate a separate CUDA stream to avoid fragmentation:

Q: 4. Tune request batching

Reduce max_batch_size for the reranker to match available memory:

In this article

Table of Contents

Problem Description

During multi‑GPU training of a Retrieval‑Augmented Generation (RAG) pipeline that uses vLLM as the inference engine, the reranker stage consistently fails with a timeout error. The failure manifests as:

vLLMEngineError: Reranker request timed out after 30000 ms

Typical impact includes:

RAG queries stall after the retrieval step, causing end‑to‑end latency spikes (> 30 s).
Partial batches are dropped, leading to reduced recall and inconsistent model outputs.
In high‑throughput environments the request queue fills up, triggering cascading timeouts across all GPUs.

Root Cause Analysis

vLLM’s reranker runs as an asynchronous request on the same scheduler that serves the main generation engine. In a multi‑GPU deployment the following factors interact to produce the observed timeout:

Scheduler queue saturation – The default max_num_requests (256) and request_timeout_ms (30 000 ms) are calibrated for single‑GPU workloads. When scaling to 2‑4 GPUs the per‑GPU request throughput increases, but the global queue size remains unchanged, causing the queue to fill under concurrent RAG queries. This is documented in the vLLM Troubleshooting Guide under “scheduler queue overflow”.
NCCL rank mismatch or initialization deadlock – If CUDA_VISIBLE_DEVICES is not updated after scaling, NCCL may assign duplicate ranks, leading to a collective communication timeout (see the real incident “mismatched NCCL ranks caused a collective communication timeout”). The resulting NCCL error aborts the scheduler thread, leaving the reranker worker idle.
GPU memory fragmentation – The reranker model often has a different memory footprint than the generator. On multi‑GPU runs the memory allocator may fragment, causing the reranker worker to be evicted and fall back to CPU execution, which cannot meet the 30 s deadline (see the incident “GPU memory fragmentation caused the reranker worker to be evicted”).

Collectively, these conditions cause the reranker request to exceed the built‑in 30 s timeout, producing the error shown above.

Investigation and Debugging

Follow these steps to isolate the root cause:

Inspect scheduler logs for queue saturation:
```
journalctl -u vllm.service -f | grep "Scheduler queue"
```
Expected pattern when saturated:

Scheduler queue full, dropping request – reranker step aborted
Verify NCCL initialization on each node:
```
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
nvidia-smi topo -m
```
Look for messages such as:

RuntimeError: NCCL error in init: timeout (operation timed out after 10 seconds)
Check GPU memory usage before and after the reranker launch:
```
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1
```
A sudden drop in free memory coinciding with the reranker start indicates fragmentation.
Profile request latency via vLLM’s async engine metrics:
```
curl -s http://localhost:8000/metrics | grep vllm_async_engine_request_latency_seconds
```
High tail latency (> 25 s) confirms timeout pressure.
Reproduce with a single GPU to confirm that the issue is tied to multi‑GPU scaling.

Resolution

Apply the following changes in order of impact.

1. Increase scheduler capacity and request timeout

Adjust the launch arguments to raise max_num_requests and request_timeout_ms:

# Before
vllm serve --model my-rag-model --tensor-parallel-size 2

# After
vllm serve \
  --model my-rag-model \
  --tensor-parallel-size 2 \
  --max-num-requests 1024 \
  --request-timeout-ms 60000

Raising the timeout to 60 s gives the reranker enough headroom when the queue is busy.

2. Align NCCL ranks and CUDA visibility

Ensure each process sees only its assigned GPU and that NCCL ranks are sequential:

# Before (common mistake)
export CUDA_VISIBLE_DEVICES=0,1,2,3   # same for all nodes

# After (per‑node configuration)
export CUDA_VISIBLE_DEVICES=0,1      # node A
export NCCL_LOCAL_RANK=0
export NCCL_RANK=0
export NCCL_WORLD_SIZE=2

export CUDA_VISIBLE_DEVICES=0,1      # node B
export NCCL_LOCAL_RANK=1
export NCCL_RANK=1
export NCCL_WORLD_SIZE=2

Restart the service after updating the environment variables.

3. Pin reranker to dedicated GPU memory pool

Use vLLM’s --reranker-gpu-id (hypothetical flag) or manually allocate a separate CUDA stream to avoid fragmentation:

# Before (no explicit pinning)
vllm serve --model my-rag-model ...

# After (pin reranker to GPU 1)
vllm serve \
  --model my-rag-model \
  --reranker-gpu-id 1 \
  --max-batch-size 32

This isolates the reranker’s memory usage, preventing eviction.

4. Tune request batching

Reduce max_batch_size for the reranker to match available memory:

# Before
--max-batch-size 64

# After
--max-batch-size 32

Smaller batches reduce per‑request memory pressure and improve scheduler throughput.

Validation

After applying the fixes, verify the system behaves as expected:

Run a synthetic RAG workload with concurrent queries:
```
python load_test.py --concurrency 50 --duration 300
```
Check that no timeout errors appear in logs:
```
grep "Reranker request timed out" /var/log/vllm/*.log
```
The command should return no results.
Confirm latency metrics are within SLA:
```
curl -s http://localhost:8000/metrics | grep vllm_async_engine_request_latency_seconds
```
Typical output after fix:

vllm_async_engine_request_latency_seconds{quantile=”0.99″} 2.8
Validate NCCL health:
```
nccl-tests/build/all_reduce_perf -b 8 -e 64M -f 2 -g 2
```
All tests should complete without timeout errors.

Operational Experience

During the investigation we observed several misleading symptoms:

Initial logs showed only the generic vLLMEngineError: Reranker request timed out after 30000 ms, leading us to suspect a model‑level bug. The real cause was scheduler saturation.
When NCCL ranks were duplicated, the first batch succeeded (because the primary GPU initialized correctly) but subsequent batches hung, producing intermittent timeouts that were hard to reproduce.
GPU memory fragmentation manifested as occasional CUDA out of memory while allocating tensor for reranker batch size 32 errors, which disappeared after pinning the reranker to a dedicated GPU.

Best Practices and Prevention

Monitor scheduler queue depth – expose vllm_scheduler_queue_length metric and set alerts when it exceeds 80 % of max_num_requests.
Pin each service component to a specific GPU and enforce consistent CUDA_VISIBLE_DEVICES across nodes.
Set NCCL environment variables (NCCL_DEBUG=INFO, NCCL_IB_DISABLE=0) in production to surface collective communication issues early.
Allocate a memory pool for the reranker using vLLM’s --reranker-gpu-id or custom CUDA streams to avoid fragmentation.
Periodically run NCCL health checks (e.g., all_reduce_perf) after scaling events.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the reranker timeout only after scaling to multiple GPUs?
Because the global scheduler queue size remains unchanged while per‑GPU request throughput increases, leading to queue saturation and timeout.
Can increasing request_timeout_ms alone solve the problem?
It mitigates the symptom but does not address underlying queue overflow or NCCL rank mismatches; combine it with queue size and rank fixes.
How do I know if NCCL initialization is failing?
Set NCCL_DEBUG=INFO and look for “NCCL error in init: timeout” in the logs. Also verify that NCCL_RANK and CUDA_VISIBLE_DEVICES are correctly set on each node.
What is the recommended max_batch_size for the reranker on a 2‑GPU node?
Empirically, a batch size of 32 fits within typical 24 GB GPU memory when the generator uses the remaining capacity. Adjust downward if you see OOM errors.
Is there a way to automatically scale max_num_requests based on load?
Yes, you can wrap the vLLM launch in a controller script that monitors vllm_scheduler_queue_length and restarts the service with a higher --max-num-requests when the queue consistently exceeds a threshold.