a rack of servers in a server room

RAG reranker timeout in LlamaIndex Docker container under load

by

Problem

During load testing and in production traffic spikes, the RAG reranker component of LlamaIndex running inside a Docker container consistently throws timeout errors. The failure manifests as:


2024-09-28 12:34:56,789 ERROR  reranker - RerankerTimeoutError: Reranking request timed out after 30.0 seconds
Traceback (most recent call last):
  File "/app/llama_index/reranker.py", line 212, in rank
    response = self.client.post(payload, timeout=self.timeout)
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in request
    resp = session.request(method=method, url=url, **kwargs)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='reranker-service', port=443): Read timed out. (read timeout=30)

Symptoms observed under load:

  • Increased 5xx responses from the API endpoint that forwards queries to LlamaIndex.
  • CPU throttling in the container (observed via docker stats).
  • Spike in RerankerTimeoutError logs while other LlamaIndex components (retriever, generator) continue to respond.

Root Cause

The timeout originates from three interacting factors:

  1. Insufficient container CPU quota: The Docker run command limits the container to 0.5 CPUs. Under load the Python ThreadPoolExecutor used by the reranker cannot schedule new tasks, causing request queuing and eventual timeout.
  2. Low default reranker timeout: LlamaIndex sets reranker_timeout_seconds=30 by default. The external reranker service (a hosted transformer model) exhibits 20‑25 s latency when the GPU is saturated, leaving a narrow margin for queuing delays.
  3. Blocking I/O in the request path: The reranker client uses requests synchronously. When the container hits the ulimit -n file‑descriptor ceiling (default 1024), new outbound connections are blocked, further extending latency.

Combined, these conditions cause the reranker request to exceed the 30 s deadline, triggering the observed timeout.

Debug

Below is the step‑by‑step investigation performed in the failing environment.

1. Verify container resource limits


$ docker inspect rag_container --format='{{json .HostConfig}}' | jq .CpuQuota
# Output
100000   # 0.1 CPU (100 ms of CPU time per 100 ms period)

Result: the container was unintentionally limited to 0.1 CPU.

2. Observe live CPU usage


$ docker stats rag_container --no-stream
CONTAINER ID   NAME          CPU %     MEM USAGE / LIMIT
a1b2c3d4e5f6    rag_container  98.73%    512Mi / 1Gi

CPU saturation correlates with the spike in timeout errors.

3. Capture request latency to the reranker service


$ curl -w "\n%{time_total}\n" -o /dev/null -s https://reranker-service/v1/rank \
    -d '{"queries":["test"]}' -H "Content-Type: application/json"
# Example output during load
0.00
24.73

Latency approaches 25 s, leaving only 5 s for queuing before the 30 s deadline.

4. Check file‑descriptor limits


$ docker exec rag_container sh -c 'ulimit -n'
1024

During load the process reaches the limit:


2024-09-28 12:35:02,112 WARNING  reranker - ResourceWarning: too many open files

5. Review LlamaIndex reranker configuration


# settings.py (original)
RERANKER_TIMEOUT_SECONDS = 30
MAX_WORKERS = 4

Solution

The fix consists of three coordinated changes: increase container CPU, raise the reranker timeout, and adjust the client to use a non‑blocking HTTP pool.

1. Update Docker run configuration

Before:


docker run -d --name rag_container \
  -p 8080:8080 \
  --cpu-quota=100000 \   # 0.1 CPU
  --memory=1g \
  myorg/llamaindex-rag:latest

After (allocate 2 full CPUs and raise file‑descriptor limit):


docker run -d --name rag_container \
  -p 8080:8080 \
  --cpus="2.0" \          # 2 CPUs
  --memory=4g \
  --ulimit nofile=65535:65535 \
  myorg/llamaindex-rag:latest

2. Extend reranker timeout in LlamaIndex configuration

Before:


# config.yaml
reranker:
  timeout_seconds: 30
  max_workers: 4

After (increase to 60 s and double workers):


# config.yaml
reranker:
  timeout_seconds: 60
  max_workers: 8

3. Switch to an asynchronous HTTP client

Replace the synchronous requests call with httpx.AsyncClient to avoid blocking the thread pool.

Before:


import requests

def rank(payload):
    resp = requests.post(
        url=self.endpoint,
        json=payload,
        timeout=self.timeout,
    )
    resp.raise_for_status()
    return resp.json()

After:


import httpx
import asyncio

class AsyncReranker:
    def __init__(self, endpoint, timeout):
        self.endpoint = endpoint
        self.timeout = timeout
        self.client = httpx.AsyncClient(timeout=timeout)

    async def rank(self, payload):
        resp = await self.client.post(self.endpoint, json=payload)
        resp.raise_for_status()
        return resp.json()

# Usage in async context
async def rank_query(payload):
    reranker = AsyncReranker("https://reranker-service/v1/rank", 60)
    return await reranker.rank(payload)

Deploy the updated code and rebuild the Docker image.

Verify

After applying the changes, run a controlled load test (e.g., wrk with 200 concurrent connections for 2 minutes) and observe the following:

1. No timeout logs


2024-09-28 13:12:04,321 INFO  reranker - Ranking completed in 31.4 seconds

2. CPU usage stays below 80 %


$ docker stats rag_container --no-stream
CONTAINER ID   NAME          CPU %     MEM USAGE / LIMIT
b7c8d9e0f1a2    rag_container  45.12%    1.2Gi / 4Gi

3. File‑descriptor count remains stable


$ docker exec rag_container sh -c 'lsof | wc -l'
342

4. End‑to‑end latency meets SLA


$ wrk -t12 -c200 -d120s http://localhost:8080/query
Latency   512ms   95% <= 1.2s
Requests/sec:  845.23

Prevent

  • Monitoring: Export reranker_request_duration_seconds and reranker_timeout_total to Prometheus; set alerts on 95th‑percentile > 45 s.
  • Resource Guardrails: Define Kubernetes resources.limits.cpu and resources.requests.cpu that match the observed load profile; use HorizontalPodAutoscaler to scale out when cpu_utilization exceeds 70 %.
  • Connection Pooling: Keep the httpx.AsyncClient instance alive for the container lifetime to reuse TCP connections.
  • Timeout Margin: Configure the reranker timeout to be at least 2× the 99th‑percentile latency of the external model service.
  • File‑Descriptor Limits: Set nofile ulimit to ≥ 65535 in production containers.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

  1. Why does the reranker timeout only under load and not in development?
    Development runs with unlimited host resources and a low query rate, so the synchronous requests client never queues. Production containers are CPU‑capped and hit the file‑descriptor ceiling, causing queuing delays that exceed the 30 s timeout.
  2. Can I keep the original 30 s timeout and still avoid errors?
    Only if you guarantee the external reranker service stays below ~15 s latency and the container has enough CPU to process requests without queuing. In practice, increasing the timeout and adding async I/O is more robust.
  3. Is switching to httpx.AsyncClient mandatory?
    Not mandatory, but it eliminates thread‑pool blockage caused by synchronous network I/O. If you retain requests, you must increase max_workers and CPU, which may still lead to scaling limits.
  4. How do I know the appropriate max_workers value?
    Start with max_workers = min(2 * CPU_cores, 8). Adjust upward while monitoring ThreadPoolExecutor._work_queue.qsize() and CPU usage; avoid exceeding the container’s CPU quota.
  5. What alert thresholds should I set for reranker latency?
    - Warning: 95th‑percentile latency > 40 s
    - Critical: timeout count > 5 per minute or 95th‑percentile latency > 55 s