Haystack Pipeline Worker Timeout During Heavy Document Ingestion

Q: Problem Description

The Haystack /document_ingestion endpoint returns a 504 Gateway Timeout when ingesting large batches of documents under load. The orchestrator logs show:

Q: Resolution

Three complementary changes eliminated the timeout:

Q: 1. Increase Celery soft time limit

Modify celeryconfig.py to give the task enough headroom:

Q: 2. Raise Haystack pipeline max_execution_time

In the pipeline definition (e.g., pipeline.yaml), set a larger timeout:

Q: 3. Adjust Docker Compose resource limits

Provide the worker with more GPU memory and a larger RAM ceiling:

Q: 4. Reduce network‑induced heartbeat loss

Increase the orchestrator heartbeat interval (default 30 s) to tolerate 200 ms inter‑service latency:

Q: Validation

After applying the changes, run a full‑scale ingestion test (10 000 PDFs) and verify:

In this article

Table of Contents

Problem Description

The Haystack /document_ingestion endpoint returns a 504 Gateway Timeout when ingesting large batches of documents under load. The orchestrator logs show:


Orchestrator log: Job 42 terminated due to timeout (504 Gateway Timeout)
TaskTimeoutError: Task exceeded time limit of 300 seconds
worker_1 | celery.exceptions.SoftTimeLimitExceeded
ERROR haystack.pipeline.base: Execution exceeded max_execution_time

Typical impact:

Batch indexing jobs for 10 000 PDFs abort after ~5 minutes.
Client API receives { "error": "Job timed out", "code": 504 }.
Subsequent ingestion attempts are queued, leading to growing backlog.
Inter‑service latency (~200 ms) and GPU memory pressure exacerbate the failure.

Root Cause Analysis

Haystack’s asynchronous pipeline runs as a Celery task. Two independent timeout mechanisms are in play:

Celery soft time limit – configured via task_soft_time_limit (default 300 s). When the limit is hit, Celery raises SoftTimeLimitExceeded, aborts the task, and propagates a TaskTimeoutError to the orchestrator.
Haystack pipeline execution timeout – documented in Haystack Pipelines: Execution and timeout settings. The pipeline checks max_execution_time and logs Execution exceeded max_execution_time when exceeded.

In the on‑premise Docker Compose deployment, the following conditions align to trigger both limits:

Heavy document ingestion (10 000 PDFs) requires GPU‑accelerated embedding generation. The Docker Compose file caps GPU memory to 4 GB (mem_limit: 4g), causing occasional OOM and slowdown.
Concurrent ingestion requests increase CPU contention, extending the total processing time beyond the 300 s soft limit.
Network latency between the API gateway and the worker (~200 ms) delays heartbeat messages; the orchestrator interprets missed heartbeats as a timeout.

Thus, the orchestrator terminates the job and returns a 504 before the pipeline can finish indexing.

Investigation and Debugging

Step‑by‑step debugging performed on a replica of the production environment:

Confirm timeout source – tail the worker and orchestrator logs:


docker compose logs -f worker
docker compose logs -f orchestrator

Observed the SoftTimeLimitExceeded message from Celery and the Haystack log entry Execution exceeded max_execution_time.

Check Celery configuration – inspect docker-compose.yml and celeryconfig.py:


# celeryconfig.py (default)
task_soft_time_limit = 300
task_time_limit = 600

Measure GPU memory usage – use nvidia-smi while a batch runs:


docker exec -it worker nvidia-smi

GPU memory peaked at 3.9 GB, leaving < 0.2 GB for the embedding model, leading to frequent memory swapping.

Profile pipeline duration – instrument the pipeline with timestamps:


from haystack.pipeline import Pipeline
import time, logging

class TimedPipeline(Pipeline):
    def run(self, *args, **kwargs):
        start = time.time()
        result = super().run(*args, **kwargs)
        elapsed = time.time() - start
        logging.info(f"Pipeline execution time: {elapsed:.2f}s")
        return result

Typical batch of 10 000 PDFs took ~420 s on the constrained node.

Resolution

Three complementary changes eliminated the timeout:

1. Increase Celery soft time limit

Modify celeryconfig.py to give the task enough headroom:


# Before
task_soft_time_limit = 300   # 5 minutes

# After
task_soft_time_limit = 900   # 15 minutes
task_time_limit = 1200       # 20 minutes (hard limit)

Rationale: The soft limit is raised to exceed the observed 420 s execution while keeping a hard limit to prevent runaway jobs.

2. Raise Haystack pipeline `max_execution_time`

In the pipeline definition (e.g., pipeline.yaml), set a larger timeout:


# Before
max_execution_time: 300  # seconds

# After
max_execution_time: 900

Reference: Haystack Pipelines: Execution and timeout settings.

3. Adjust Docker Compose resource limits

Provide the worker with more GPU memory and a larger RAM ceiling:

Parameter	Before	After
`mem_limit`	4g	8g
`deploy.resources.reservations.devices`	1 GPU, 4 GB	1 GPU, 8 GB
`environment` – `CUDA_VISIBLE_DEVICES`	0	0


# docker-compose.yml snippet (worker service)
services:
  worker:
    image: deepset/haystack-worker:latest
    deploy:
      resources:
        limits:
          memory: 8g
        reservations:
          devices:
            - capabilities: [gpu]
              driver: nvidia
              count: 1
    environment:
      - CUDA_VISIBLE_DEVICES=0
    mem_limit: 8g

Result: GPU memory pressure dropped, OOM events disappeared, and overall batch time fell to ~350 s.

4. Reduce network‑induced heartbeat loss

Increase the orchestrator heartbeat interval (default 30 s) to tolerate 200 ms inter‑service latency:


# orchestrator config (orchestrator.yaml)
heartbeat_interval: 60   # seconds
heartbeat_timeout: 180   # seconds

Validation

After applying the changes, run a full‑scale ingestion test (10 000 PDFs) and verify:

Job status – API returns 200 OK with "status":"completed" after ~340 s.
Logs – No SoftTimeLimitExceeded or Execution exceeded max_execution_time entries.
GPU usage – nvidia-smi shows stable memory ~3.5 GB, no OOM.
Metrics – Prometheus metric haystack_pipeline_duration_seconds reports values below the new 900 s threshold.


curl -X POST http://api.local/document_ingestion \
  -H "Content-Type: application/json" \
  -d '{"batch_id":"test-001","files": [...]}'

# Expected response
{
  "job_id": "abc123",
  "status": "completed",
  "documents_indexed": 10000,
  "duration_seconds": 342
}

Best Practices and Prevention

Set timeouts based on realistic batch sizes. Use a benchmark ingestion run to determine a safe max_execution_time and Celery limits.
Monitor GPU memory and OOM events. Alert on nvidia-smi memory usage > 90 % or Docker OOM kills.
Separate heavy ingestion workloads. Deploy a dedicated worker node with higher GPU RAM for bulk indexing.
Configure heartbeats conservatively. Align heartbeat_interval and heartbeat_timeout with observed network latency.
Use Celery task retries. Enable autoretry_for=(SoftTimeLimitExceeded,) with exponential back‑off for transient spikes.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the timeout only appear under heavy load?
Because the default 300 s soft limit is calibrated for small batches. Large batches increase GPU processing time and contention, pushing execution past the limit.
Can I keep the default timeout and still process 10 000 documents?
Only by splitting the ingestion into smaller chunks (e.g., 2 000‑document batches) or by provisioning more GPU memory so each batch finishes faster.
Do I need to change both Celery and Haystack timeouts?
Yes. Celery aborts the task first; Haystack’s internal check provides a second safety net. Both must be increased consistently.
What if I cannot increase GPU memory on the host?
Consider swapping to a CPU‑only embedding model for bulk ingestion, or offload embedding generation to a separate inference service with dedicated resources.
How can I detect future timeout regressions early?
Add a Prometheus alert on haystack_pipeline_duration_seconds exceeding 80 % of the configured limit, and on Celery task_soft_time_limit_exceeded_total metric.