Black cables and wires connected to the back of modular LED display panels

LlamaIndex batch ingestion timeout under high‑traffic API gateway

by

Problem – Batch Ingestion Timeout in LlamaIndex Behind a High‑Traffic API Gateway

When a large document batch is submitted to an endpoint that forwards the request to a LlamaIndex ingestion service, the request frequently ends with a 504 Gateway Timeout or a client‑side deadline exceeded error. Under sustained traffic the ingestion job is aborted, leading to partial data loading and eventual inconsistency in the vector store.

Typical symptoms observed in production:

  • API gateway logs contain ERROR: request timed out after 30s.
  • LlamaIndex process logs show INFO: Starting batch ingestion of 10,000 documents followed by ERROR: Ingestion aborted: context deadline exceeded.
  • CPU spikes to 95 % and memory usage climbs to >80 % during the failure window.
  • Partial vectors appear in the downstream vector DB (e.g., Pinecone, Weaviate) – queries return some but not all newly added documents.

Root Cause – Interaction Between API Gateway Timeouts, Synchronous Ingestion, and Resource Contention

The failure is not a single mis‑configuration but a cascade of constraints:

  1. Gateway timeout limits. The API gateway (e.g., Kong, AWS API Gateway, or NGINX) enforces a hard request timeout (commonly 30 s). The ingestion endpoint processes the batch synchronously, so the request thread remains blocked for the entire duration of the batch.
  2. Synchronous batch processing. LlamaIndex’s index.ingest_documents() runs in the request thread, iterating over the entire batch, building embeddings, and writing to the vector store. For large batches (≥5 k docs) this can easily exceed the gateway timeout.
  3. Resource contention under load. When the gateway receives many concurrent ingestion requests, the Python worker pool (e.g., Gunicorn workers) competes for CPU and memory. The per‑worker process becomes CPU‑bound, extending the wall‑clock time of each batch.
  4. Default client‑side deadline. The SDK used to call the gateway (e.g., requests or httpx) also sets a default timeout (often 10 s). The client aborts before the gateway does, causing the server to log “context deadline exceeded”.

In short: the ingestion path is too heavyweight for a request‑/response cycle that is subject to strict timeout policies.

Debug – Step‑by‑Step Investigation

Below is a reproducible debugging workflow that isolates each factor.

1. Capture gateway timeout configuration

# Example for NGINX
cat /etc/nginx/conf.d/api_gateway.conf | grep -i timeout
proxy_connect_timeout 30s;
proxy_read_timeout    30s;
proxy_send_timeout    30s;

2. Verify LlamaIndex ingestion duration

# Run a local benchmark with the same batch size
python - <<'PY'
import time, json
from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex, ServiceContext
docs = SimpleDirectoryReader('sample_docs').load_data()
svc = ServiceContext.from_defaults()
start = time.time()
index = GPTVectorStoreIndex.from_documents(docs, service_context=svc)
print("Ingestion took", time.time() - start, "seconds")
PY

Typical output on a 4‑core VM:

Ingestion took 48.7 seconds

3. Observe process resource usage during ingestion

# While a batch is running, in another terminal:
ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -n 10

Sample output:

  PID  PPID CMD                         %CPU %MEM
 3124  3101 python3 ./ingest_service.py  97.3 78.4
 3125  3101 python3 ./ingest_service.py  93.1 77.9
 ...

4. Check client‑side timeout

# Example using requests
import requests
response = requests.post(
    "https://api.example.com/ingest",
    json={"batch_id": "2024-09-01"},
    timeout=10  # seconds
)
print(response.status_code)

If the request aborts with requests.exceptions.ReadTimeout, the client timeout is a factor.

5. Correlate logs

# Gateway log snippet (NGINX)
2024/09/24 12:34:56 [error] 1456#0: *12345 upstream timed out (30s) while reading response header from upstream, client: 203.0.113.42, server: api.example.com, request: "POST /ingest HTTP/1.1", upstream: "http://127.0.0.1:8000/ingest", host: "api.example.com"

# Ingestion service log snippet
2024-09-24 12:34:55,123 INFO  ingestion_worker.py:45 Starting batch ingestion of 10000 documents (batch_id=2024-09-01)
2024-09-24 12:35:27,890 ERROR ingestion_worker.py:78 Ingestion aborted: context deadline exceeded (batch_id=2024-09-01)

Solution – Decouple Ingestion from the Request Path and Tune Timeouts

The reliable fix consists of two complementary changes:

1. Convert the endpoint to an asynchronous job submission

Instead of processing the batch inline, the API should enqueue a job (e.g., via Celery, RQ, or a cloud task queue) and return a job identifier immediately.

Before (synchronous endpoint):

# ingest_service.py
from fastapi import FastAPI, HTTPException
from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, ServiceContext

app = FastAPI()

@app.post("/ingest")
def ingest(batch: dict):
    try:
        docs = SimpleDirectoryReader(batch["path"]).load_data()
        index = GPTVectorStoreIndex.from_documents(docs, service_context=ServiceContext.from_defaults())
        return {"status": "completed"}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

After (asynchronous job submission):

# ingest_service.py
from fastapi import FastAPI, HTTPException
from celery import Celery
from uuid import uuid4

app = FastAPI()
celery_app = Celery(broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@celery_app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_batch(self, batch_path):
    from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, ServiceContext
    try:
        docs = SimpleDirectoryReader(batch_path).load_data()
        index = GPTVectorStoreIndex.from_documents(docs, service_context=ServiceContext.from_defaults())
        return {"status": "completed"}
    except Exception as exc:
        raise self.retry(exc=exc)

@app.post("/ingest")
def ingest(batch: dict):
    job_id = str(uuid4())
    process_batch.apply_async(args=[batch["path"]], task_id=job_id)
    return {"job_id": job_id, "status": "queued"}

Now the request returns within milliseconds, well under any gateway timeout.

2. Increase and align timeout settings for the remaining short‑lived request

If the endpoint must still perform a quick validation step, adjust the gateway and client timeouts to a safe margin (e.g., 5 s).

Component Current Setting Recommended Setting
NGINX proxy_read_timeout 30 s 5 s (or remove if only health‑check)
API Gateway (AWS) timeoutInMillis 30 000 ms 5 000 ms
Python requests timeout 10 s 5 s

3. Tune LlamaIndex batch size and worker pool

Even with asynchronous processing, a single worker can become a bottleneck under heavy load. Split large batches into smaller chunks (e.g., 1 k documents) and increase the Celery concurrency.

# celery configuration (celeryconfig.py)
worker_concurrency = 8          # number of parallel workers
task_acks_late = True
worker_prefetch_multiplier = 1

Chunking logic example:

# chunked_ingest.py
def chunked_ingest(batch_path, chunk_size=1000):
    from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex, ServiceContext
    all_docs = SimpleDirectoryReader(batch_path).load_data()
    for i in range(0, len(all_docs), chunk_size):
        chunk = all_docs[i:i+chunk_size]
        GPTVectorStoreIndex.from_documents(chunk, service_context=ServiceContext.from_defaults())

Verify – Confirm the Fix Works End‑to‑End

  1. Submit a test batch. Use curl to POST a small payload and verify an immediate 202 Accepted with a job_id. 
    curl -X POST https://api.example.com/ingest \
  -H "Content-Type: application/json" \
  -d '{"path":"/data/batch_20240901"}' -i
HTTP/1.1 202 Accepted
Content-Type: application/json
...
{"job_id":"c3f9e2b4-7d1a-4f3c-9a2e-5b6d8f9a1c2d","status":"queued"}
  2. Poll the job status. Query the Celery backend or expose a /jobs/{id} endpoint until the status becomes completed. Verify the elapsed time is well under the previous 30 s gateway limit.
  3. Check vector store integrity. Run a simple similarity query for a known document from the batch and confirm it returns the expected result.
  4. Monitor resource usage. During the background job, htop or docker stats should show CPU usage spread across the configured number of workers, without a single process saturating the host.

Prevent – Operational Guardrails for Future Deployments

  • Metrics & alerts. Emit ingestion_job_duration_seconds and queue_length metrics. Alert if average duration exceeds a threshold (e.g., 60 s) or if queue depth > 100.
  • Back‑pressure at the gateway. Enable rate‑limiting (e.g., 10 requests per second) on the ingestion endpoint to avoid overwhelming the worker pool.
  • Health‑check endpoint. Separate health checks from job submission; expose /healthz that returns 200 instantly.
  • Automated batch sizing. Implement a function that estimates optimal chunk size based on current CPU load (e.g., chunk = max(500, int(available_cpus * 2000))).
  • Graceful shutdown. Ensure Celery workers finish in‑flight jobs before termination to avoid partial ingestion.

FAQ – Common Follow‑Up Questions

  1. Why does the timeout only appear under high traffic and not during low‑volume testing?
    Because low traffic leaves a single worker idle, allowing the synchronous request to finish before the gateway timeout. Under load multiple workers compete for CPU, extending the wall‑clock time beyond the timeout limit.
  2. Can I keep the endpoint synchronous and just raise the gateway timeout?
    Increasing the timeout mitigates the symptom but does not solve the underlying resource contention. It also ties up gateway connections, reducing overall throughput and risking denial‑of‑service under spikes.
  3. How do I choose an appropriate chunk size for large batches?
    Start with 1 k documents per chunk, then profile CPU usage. If workers stay below 80 % CPU, increase the chunk size; if they saturate, reduce it. Automate the adjustment based on psutil.cpu_percent() at job start.
  4. Is Celery the only way to decouple ingestion?
    No. Alternatives include AWS SQS + Lambda, Google Cloud Tasks, Kubernetes Jobs, or a simple “fire‑and‑forget” background thread. The key is to return a response quickly and let a separate worker handle the heavy lifting.
  5. Do I need to modify the LlamaIndex client configuration when using a task queue?
    Only if you change the embedding model endpoint (e.g., to a hosted OpenAI service). In that case, set a higher request_timeout in the OpenAIEmbedding client to avoid premature failures inside the worker.

Related Topic Hub: RAG Systems Troubleshooting Hub