LlamaIndex inference queue backlog during A/B testing

Problem – Inference Queue Backlog During A/B Testing

During a recent high‑traffic A/B test two LlamaIndex variants (A and B) were deployed side‑by‑side in a Kubernetes cluster. Users of the test variant experienced:

• Latency spikes from ~200 ms to >30 s.
• HTTP 503 responses with the log line Inference queue full: max_queue_size reached.
• Increasing numbers of “Worker pool exhausted – cannot schedule new inference task” entries in the pod logs.

The backlog persisted even though CPU and memory metrics stayed well below the autoscaling thresholds, leading to the impression that the service was “healthy” while the request queue grew unchecked.

Root Cause Analysis

The LlamaIndex async service context uses a bounded ThreadPoolExecutor (or Ray worker pool) to drive inference calls. By default the max_concurrent_requests parameter is set to 10 and the internal queue size to 1000 (see LlamaIndex Documentation – Async Retrieval & Generation). When the A/B test doubled the request rate from 200 rps to 800 rps, the following sequence occurred:

1. Both variants shared the same Redis queue backend (as described in the async docs). The queue length grew past 10 k entries, triggering the “Inference queue full” error.
2. Because the worker pool size remained at the default (2 Celery workers in the AWS ECS incident, or the default thread pool size in Kubernetes), the CPU usage never spiked, so the Horizontal Pod Autoscaler (HPA) never fired (LlamaIndex Deployment Guide – Scaling with Kubernetes).
3. The static max_concurrent_requests value in the global service context prevented the pool from expanding to meet the burst, causing back‑pressure to accumulate in the Redis queue.

In short, the queue backlog was a classic case of static concurrency limits + insufficient autoscaling metrics, exacerbated by running multiple test variants that compete for the same queue resources.

Investigation and Debugging

Below is a reproducible debugging workflow that isolates the bottleneck.

1. Inspect LlamaIndex logs

2026-06-16 14:02:31,842 INFO  inference_worker.py:124 - Inference queue full: max_queue_size reached (1000)
2026-06-16 14:02:31,845 WARN  inference_worker.py:130 - Worker pool exhausted - cannot schedule new inference task
2026-06-16 14:02:31,847 ERROR inference_worker.py:145 - Task timed out after 30 seconds

2. Check Redis queue length

$ redis-cli -p 6379 LLEN llamaindex:inference_queue
10023

3. Verify HPA metrics

$ kubectl get hpa -n test-a
NAME          REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
llamaindex-a  Deployment/llamaindex-a  12%/80%   2         10        2          15m

CPU is at 12 % while the queue length is >10 k, confirming the metric mismatch.

4. Examine service context configuration

from llama_index import ServiceContext, set_global_service_context

# Current (default) configuration
svc_ctx = ServiceContext.from_defaults()
set_global_service_context(svc_ctx)

The default max_concurrent_requests is 10, and request_timeout is 30 s.

5. Trace request flow with kubectl exec and ss

$ kubectl exec -it llamaindex-a-0 -- ss -ltnp | grep 8080
LISTEN 0      128    0.0.0.0:8080      0.0.0.0:*    users:(("uvicorn",pid=23,fd=5))

Connection counts remained stable, confirming the bottleneck is internal queueing, not network saturation.

Resolution – Scaling the Inference Pipeline

The fix consists of three coordinated changes:

1. Increase the async service context limits

Adjust max_concurrent_requests and max_queue_size to values that match the expected traffic burst.

Before:

from llama_index import ServiceContext, set_global_service_context

svc_ctx = ServiceContext.from_defaults()  # max_concurrent_requests=10, max_queue_size=1000
set_global_service_context(svc_ctx)

After:

from llama_index import ServiceContext, set_global_service_context

svc_ctx = ServiceContext.from_defaults(
    max_concurrent_requests=100,   # allow 10× more parallel inferences
    max_queue_size=10000,          # raise queue capacity to absorb spikes
    request_timeout=120           # give longer time for heavy prompts
)
set_global_service_context(svc_ctx)

This change is documented in the Async Retrieval & Generation section and mirrors the solution discussed in Stack Overflow question 78543219.

2. Deploy a dedicated worker pool per variant

Use Ray or Celery workers scoped to each namespace so that variant A cannot starve variant B.

# Example Celery worker deployment (Kubernetes manifest snippet)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llamaindex-a-worker
  namespace: test-a
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llamaindex-a-worker
  template:
    metadata:
      labels:
        app: llamaindex-a-worker
    spec:
      containers:
      - name: worker
        image: llamaindex/worker:latest
        env:
        - name: CELERY_CONCURRENCY
          value: "20"          # 20 processes per pod
        - name: QUEUE_NAME
          value: "llamaindex-a-queue"

Increasing the replica count triggers the HPA based on custom.metrics.k8s.io/queue_length (see the scaling guide). The custom metric can be exported via a sidecar that reads the Redis queue length.

3. Adjust HPA to use queue‑length metrics instead of CPU

Deploy a ExternalMetric that reflects the Redis queue size.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: llamaindex-a-hpa
  namespace: test-a
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llamaindex-a
  minReplicas: 2
  maxReplicas: 15
  metrics:
  - type: External
    external:
      metric:
        name: redis_queue_length
        selector:
          matchLabels:
            queue: llamaindex-a-queue
      target:
        type: AverageValue
        averageValue: "5000"

When the queue exceeds 5 k entries, the HPA will add pods, ensuring CPU usage rises and the autoscaler reacts appropriately.

Validation – Verifying the Fix

After applying the changes, perform the following checks:

1. Queue length drops:

   
   $ redis-cli LLEN llamaindex:inference_queue
   124
   

2. Latency returns to baseline:

   
   $ curl -s -w "%{time_total}\\n" -o /dev/null https://api.example.com/v1/infer
   0.215
   

3. HPA scales up during a traffic spike:

   
   $ kubectl get hpa -n test-a
   NAME          REFERENCE               TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
   llamaindex-a  Deployment/llamaindex-a  12%/80% 5/5000   2         15        6          3m
   

4. No “Inference queue full” errors in logs:

   
   2026-06-16 15:12:04,231 INFO  inference_worker.py:124 - Processed request in 0.38s
   2026-06-16 15:12:04,235 INFO  inference_worker.py:124 - Queue length: 78
   

Prevention – Operational Guardrails

• Metric‑driven autoscaling: Prefer queue‑length or custom request‑rate metrics over CPU alone for inference services.
• Per‑variant isolation: Deploy separate Redis namespaces or queue prefixes for each A/B variant to avoid cross‑contamination.
• Dynamic configuration: Store max_concurrent_requests in a ConfigMap and reload via rolling updates when traffic patterns change.
• Alerting: Create alerts on:
  • Redis queue length > 8 k.
  • Log pattern “Inference queue full”.
  • Request latency > 5 s.
• Load testing: Prior to each A/B rollout, run a load test that ramps to the expected peak RPS and verifies that queue length stays under the configured max_queue_size.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

1. Why does increasing CPU limits not solve the backlog?
   Because the bottleneck is the fixed thread‑pool size and queue capacity, not CPU utilization. Autoscaling on CPU never triggers when workers are idle but the queue is full.
2. Can I keep the default max_concurrent_requests and just add more pods?
   Adding pods helps only if each pod’s worker pool can process requests. With the default limit of 10 concurrent requests per pod, you would need many pods to match the traffic, which is inefficient. Raising the per‑pod limit reduces pod count and improves latency.
3. How do I monitor the Redis queue length in CloudWatch or Prometheus?
   Export the LLEN value via a sidecar exporter (e.g., redis_exporter) and create a Prometheus rule: queue_length{queue="llamaindex-a-queue"} > 5000. Then forward the metric to CloudWatch if needed.
4. Is Ray a better backend than Celery for scaling?
   Ray provides automatic worker scaling and can handle heterogeneous workloads. If you already use Ray in your stack, replace the Celery worker deployment with a Ray cluster and set ray.init(..., num_cpus=...). The underlying issue (static limits) still needs to be addressed by configuring max_concurrent_requests appropriately.
5. What timeout should I set for request_timeout?
   Set it slightly above the 99th‑percentile latency observed under load (e.g., 120 s if the 99th‑percentile is 80 s). This prevents premature task failures while still protecting against runaway requests.