Haystack inference queue backlog during CI/CD pipeline execution

Problem – Inference Queue Backlog During CI/CD Pipeline Execution

Symptoms and Impact

During automated testing and deployment steps, Haystack inference calls start to time out. Typical log excerpts look like:

2024-08-24 12:03:45,212 [celery.worker] ERROR QueueFullError: Inference queue is full (maxsize=100)
2024-08-24 12:03:48,019 [haystack.client] ERROR TimeoutError: Inference request timed out after 30 seconds
2024-08-24 12:04:01,877 [celery.worker] ERROR WorkerLostError: Celery worker worker-1 lost connection

Consequences observed in the CI pipeline:

• Test steps that depend on model answers fail with TimeoutError.
• Overall pipeline duration increases by 30‑40 seconds per batch of parallel tests.
• In extreme cases the pipeline aborts with a non‑zero exit code, leaving the environment partially torn down.

Root Cause – Why the Queue Backlog Happens

Interaction of CI Orchestration and Haystack Async Pipelines

Haystack’s async pipeline architecture (see Haystack Documentation – “Async Pipelines and Queue Management”) relies on a Redis broker and a pool of Celery workers. Each inference request is serialized into a Redis list; workers pop jobs, run the model, and push results back.

In CI environments the following factors converge:

1. Burst of parallel requests. CI steps often launch dozens of integration tests simultaneously (e.g., 20 parallel tests). Each test creates a separate inference request, filling the Redis list faster than workers can consume it.
2. Static worker pool size. The default Celery configuration starts only 2–4 workers (see Scaling Inference with Celery and Redis). When CPU resources are limited (Docker Compose on a single node, Jenkins agents, or GitHub Actions runners), workers are throttled and cannot scale up quickly enough.
3. Queue size limit. Haystack’s maxsize defaults to 100 (documented in Configuration of Inference Timeout and Retries). Once the Redis list reaches this limit, further push operations raise QueueFullError, causing the client to abort early.
4. Inference timeout mismatch. The client timeout (default 30 s) is shorter than the time needed for the queue to drain under load, leading to TimeoutError even though workers eventually finish the jobs.

Combined, these conditions produce the observed backlog and timeouts.

Debug – Investigation Process

Step‑by‑step diagnostics

1. Inspect Redis queue length. Use redis-cli to query the list size while the pipeline runs.

redis-cli -p 6379 LLEN haystack_inference_queue
# Expected output during idle: 0
# During CI burst: 102

Check Celery worker count and CPU usage.

celery -A haystack_app status
# Sample output:
# celery@worker-1: OK
# celery@worker-2: OK
# Active workers: 2

top -b -n1 | grep celery
# Shows each worker consuming ~100% CPU, indicating saturation.

Verify timeout configuration. Look at haystack_config.yaml or environment variables.

# haystack_config.yaml
inference:
  timeout: 30  # seconds
  max_retries: 3
  queue_maxsize: 100

Capture a short packet trace. Confirm that requests reach the Redis broker.

tcpdump -i any port 6379 -c 20 -w /tmp/redis_capture.pcap

Review CI logs for container readiness. Look for messages such as “ConnectionError: Unable to connect to inference service”.

2024-08-24 12:02:58,003 [ci.runner] INFO Starting Haystack inference container...
2024-08-24 12:03:02,411 [ci.runner] ERROR ConnectionError: Unable to connect to inference service at http://localhost:8000

Solution – Eliminating the Backlog

1. Increase Worker Pool Dynamically

Configure Celery to start more workers and enable autoscaling when running in CI. Add the following to docker-compose.ci.yml (or the Kubernetes pod spec used by the pipeline):

# Before (single worker)
services:
  haystack-worker:
    image: deepset/haystack-worker:latest
    command: celery -A haystack_app worker --loglevel=info
    deploy:
      resources:
        limits:
          cpus: "1"

# After (autoscaling to 8 workers, max 2 CPUs each)
services:
  haystack-worker:
    image: deepset/haystack-worker:latest
    command: celery -A haystack_app worker --loglevel=info --autoscale=8,2
    deploy:
      resources:
        limits:
          cpus: "8"

2. Raise Queue Capacity

Adjust queue_maxsize to accommodate the burst size observed in CI (e.g., 500).

# Before
queue_maxsize: 100

# After
queue_maxsize: 500

3. Align Client Timeout with Expected Drain Time

Increase the inference timeout to a value that covers the worst‑case queue drain time (e.g., 120 s).

# Before
timeout: 30

# After
timeout: 120

4. Pre‑warm Workers Before Test Execution

Insert a “warm‑up” step in the CI script that sends a lightweight dummy request and waits for a successful response. This ensures the workers are alive and the Redis queue is empty before the real load starts.

# CI script snippet
curl -X POST http://localhost:8000/api/v1/predict \
     -H "Content-Type: application/json" \
     -d '{"queries": ["warm up"]}' \
     --max-time 10

5. Resource Allocation for CI Runners

When using Docker Compose on a single‑node Jenkins agent, allocate more CPU/memory to the Docker daemon or run the inference stack on a dedicated runner with at least 4 vCPU.

Verify – Confirming the Fix

1. Re‑run the CI pipeline with the updated configuration.
2. Monitor Redis queue length; it should stay below the new maxsize (e.g., max 120 during burst).
3. Check that no QueueFullError or TimeoutError appears in the logs.
4. Validate that all integration tests complete within the expected time window (e.g., < 5 min total).

Sample successful log excerpt after the fix:

2024-08-24 12:15:02,104 [celery.worker] INFO celery@worker-1 ready.
2024-08-24 12:15:02,108 [celery.worker] INFO celery@worker-2 ready.
2024-08-24 12:15:05,321 [haystack.client] INFO Inference request completed in 2.3s
2024-08-24 12:15:05,322 [haystack.client] INFO Inference request completed in 2.1s
# No QueueFullError or TimeoutError observed.

Prevent – Best Practices for Future CI Runs

• Capacity planning. Estimate the maximum parallel inference calls in a pipeline and size queue_maxsize and worker autoscaling accordingly.
• Health checks. Add a pre‑flight health endpoint that verifies Redis connectivity and worker availability before test execution.
• Metrics and alerts. Export celery.worker.active and Redis queue length to Prometheus; set alerts for queue length > 80% of maxsize.
• Separate CI environment. Run Haystack inference in a dedicated namespace or Docker network to avoid resource contention with other CI services.
• Graceful shutdown. Ensure CI pipelines wait for all pending jobs to finish (e.g., celery -A app control shutdown) before tearing down the stack.

FAQ – Related Questions

1. Why does the backlog only appear in CI and not in local development? CI runners typically have fewer CPU cores and launch many tests in parallel, causing a higher request burst than a single developer’s workstation.
2. Can I keep the default queue size and still avoid timeouts? Yes, by increasing the number of Celery workers or enabling autoscaling so that the processing rate matches the request arrival rate.
3. What is the recommended timeout value for CI pipelines? Set the client timeout to at least twice the expected maximum queue drain time. For a burst of 500 requests with 8 workers, 120 seconds is a safe default.
4. How do I detect a “WorkerLostError” early? Enable Celery’s worker_send_task_events and monitor the celery.worker.lost metric; trigger a pipeline abort if the loss count exceeds 1.
5. Is Redis persistence required for CI runs? No. Use a non‑persistent Redis instance (e.g., redis:alpine) to keep startup fast; persistence only adds I/O latency that can worsen the backlog.

Related Topic Hub: RAG Systems Troubleshooting Hub