Haystack Streaming Response Interruption After Model Inference Service Disconnect

Q: Root Cause

The pipeline uses a TransformersGenerator that communicates with a separate model inference service (e.g., vLLM or Triton) over a Unix domain socket (UDS) or TCP loopback. In an air‑gapped environment the inference service is container‑isolated and relies on a systemd socket‑activated service. When the inference container experiences a cgroup OOM kill or a network namespace teardown (triggered by a periodic docker system prune job), the socket is silently closed. Haystack’s streaming generator does not detect the closure until the next token is requested, at which point it raises a ConnectionError and aborts the stream.

Q: Debug 1. Capture streaming logs

2024-10-27 14:02:13,421 INFO haystack.pipeline - Starting pipeline: "question_answering" 2024-10-27 14:02:13,425 DEBUG haystack.nodes.generator - Sending request to inference service at http://127.0.0.1:8000/generate 2024-10-27 14:02:13,428 INFO haystack.nodes.generator - Received token: "The" 2024-10-27 14:02:13,530 INFO haystack.nodes.generator - Received token: "capital" 2024-10-27 14:02:13,632 INFO haystack.nodes.generator - Received token: "of" 2024-10-27 14:02:13,734 INFO haystack.nodes.generator - Received token: "France" 2024-10-27 14:02:13,836 INFO haystack.nodes.generator - Received token: "is" 2024-10-27 14:02:13,938 ERROR haystack.nodes.generator - gRPC stream closed: StatusCode.UNAVAILABLE 2024-10-27 14:02:13,938 TRACE haystack.pipeline - Pipeline aborted due to generator error 2. Verify inference service health $ curl -s http://127.0.0.1:8000/healthz {"status":"healthy"} The endpoint reports healthy because it only checks the process PID, not the socket state.

Q: 3. Inspect container resource limits

$ docker inspect inference-service --format '{{json .HostConfig.Memory}}' 1024MiB $ docker stats inference-service --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT a1b2c3d4e5f6 inference-service 0.00% 1.02GiB / 1.02GiB The memory usage hovers at the limit, indicating an imminent OOM kill.

Q: 4. Check systemd journal for OOM events

$ journalctl -u inference-service -b | grep -i oom Oct 27 14:02:13 hostname systemd[1]: inference-service.service: Main process exited, code=exited, status=137/Killed Oct 27 14:02:13 hostname systemd[1]: inference-service.service: Failed with result 'exit-code'. 5. Reproduce with a minimal script Run a local generator that streams tokens from a mock inference service. When the mock process is killed after a few seconds, the same interruption occurs.

Q: 2. Enable gRPC keep‑alive and explicit timeout

Configure the Haystack TransformersGenerator to use a keep‑alive ping every 30 seconds and a 120‑second overall timeout.

Q: Verify 1. Confirm memory headroom

$ docker stats inference-service --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT a1b2c3d4e5f6 inference-service 0.02% 2.1GiB / 4.0GiB 2. Validate keep‑alive behavior $ grpcurl -plaintext -proto generate.proto localhost:8000 Generate/StreamTokens # Observe ping frames every 30 s in the client trace 3. Run an end‑to‑end streaming request >>> curl -N -X POST http://localhost:8000/api/pipeline \ -H "Content-Type: application/json" \ -d '{"query":"What is the capital of France?"}' The capital of France is Paris. The response completes without interruption.

Problem

In an on‑premises Haystack deployment that runs inside an air‑gapped network, the StreamingResponse returned by a Haystack pipeline intermittently stops mid‑generation. The client receives a truncated answer and the HTTP connection is closed without an explicit error payload.

Typical symptoms observed in the client console:


>>> Answer: The capital of France is
>>> ConnectionError: stream closed unexpectedly

Server‑side logs show the inference worker losing its gRPC connection to the model server while the generator is still producing tokens.

Root Cause

The pipeline uses a TransformersGenerator that communicates with a separate model inference service (e.g., vLLM or Triton) over a Unix domain socket (UDS) or TCP loopback. In an air‑gapped environment the inference service is container‑isolated and relies on a systemd socket‑activated service. When the inference container experiences a cgroup OOM kill or a network namespace teardown (triggered by a periodic docker system prune job), the socket is silently closed. Haystack’s streaming generator does not detect the closure until the next token is requested, at which point it raises a ConnectionError and aborts the stream.

Key factors that make the issue hard to spot:

Both Haystack and the inference service run on the same host, so no external network latency is visible.
Health‑check probes succeed because they query the inference service’s /ready endpoint, which remains up until the OOM event.
The default timeout for the gRPC channel is None, so the client waits indefinitely, masking the underlying socket failure.

Debug

1. Capture streaming logs


2024-10-27 14:02:13,421 INFO  haystack.pipeline - Starting pipeline: "question_answering"
2024-10-27 14:02:13,425 DEBUG haystack.nodes.generator - Sending request to inference service at http://127.0.0.1:8000/generate
2024-10-27 14:02:13,428 INFO  haystack.nodes.generator - Received token: "The"
2024-10-27 14:02:13,530 INFO  haystack.nodes.generator - Received token: "capital"
2024-10-27 14:02:13,632 INFO  haystack.nodes.generator - Received token: "of"
2024-10-27 14:02:13,734 INFO  haystack.nodes.generator - Received token: "France"
2024-10-27 14:02:13,836 INFO  haystack.nodes.generator - Received token: "is"
2024-10-27 14:02:13,938 ERROR haystack.nodes.generator - gRPC stream closed: StatusCode.UNAVAILABLE
2024-10-27 14:02:13,938 TRACE haystack.pipeline - Pipeline aborted due to generator error

2. Verify inference service health


$ curl -s http://127.0.0.1:8000/healthz
{"status":"healthy"}

The endpoint reports healthy because it only checks the process PID, not the socket state.

3. Inspect container resource limits


$ docker inspect inference-service --format '{{json .HostConfig.Memory}}'
1024MiB
$ docker stats inference-service --no-stream
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT
a1b2c3d4e5f6   inference-service  0.00%     1.02GiB / 1.02GiB

The memory usage hovers at the limit, indicating an imminent OOM kill.

4. Check systemd journal for OOM events


$ journalctl -u inference-service -b | grep -i oom
Oct 27 14:02:13 hostname systemd[1]: inference-service.service: Main process exited, code=exited, status=137/Killed
Oct 27 14:02:13 hostname systemd[1]: inference-service.service: Failed with result 'exit-code'.

5. Reproduce with a minimal script

Run a local generator that streams tokens from a mock inference service. When the mock process is killed after a few seconds, the same interruption occurs.


# mock_generator.py
import time, sys
for token in ["The", "capital", "of", "France", "is", "Paris"]:
    print(token, end=' ', flush=True)
    time.sleep(0.1)
    if token == "is":
        # Simulate service death
        sys.exit(1)

Solution

1. Increase memory headroom for the inference container

Allocate at least 2 × the peak model memory consumption and set a soft limit to avoid hard OOM kills.


# Before (insufficient)
docker run -d --name inference-service \
  --memory=1024m \
  myrepo/vllm:latest

# After (adequate)
docker run -d --name inference-service \
  --memory=4096m \
  --memory-swap=8192m \
  myrepo/vllm:latest

2. Enable gRPC keep‑alive and explicit timeout

Configure the Haystack TransformersGenerator to use a keep‑alive ping every 30 seconds and a 120‑second overall timeout.


# before.py
generator = TransformersGenerator(
    model_name_or_path="local-model",
    device="cuda",
    use_gpu=True,
)

# after.py
generator = TransformersGenerator(
    model_name_or_path="local-model",
    device="cuda",
    use_gpu=True,
    generation_kwargs={
        "grpc_options": [
            ("grpc.keepalive_time_ms", 30000),
            ("grpc.keepalive_timeout_ms", 120000),
        ],
        "timeout": 120,
    },
)

3. Switch from Unix domain socket to TCP with SO_REUSEPORT

TCP sockets survive container restarts better when the host network namespace is preserved.


# docker-compose.yml snippet (before)
services:
  inference:
    image: myrepo/vllm:latest
    network_mode: "none"
    volumes:
      - /tmp/vllm.sock:/tmp/vllm.sock

# after
services:
  inference:
    image: myrepo/vllm:latest
    ports:
      - "8000:8000"
    environment:
      - GRPC_SERVER_ADDRESS=0.0.0.0:8000
    sysctls:
      - net.ipv4.tcp_tw_reuse=1

4. Add a watchdog sidecar that restarts the inference service on socket failure

The sidecar monitors the gRPC health endpoint and triggers a container restart if the health check fails three consecutive times.


# watchdog.sh
while true; do
  if ! curl -s http://127.0.0.1:8000/healthz | grep -q healthy; then
    echo "$(date) - inference unhealthy, restarting..."
    docker restart inference-service
    sleep 5
  fi
  sleep 30
done

Verify

1. Confirm memory headroom


$ docker stats inference-service --no-stream
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT
a1b2c3d4e5f6   inference-service  0.02%     2.1GiB / 4.0GiB

2. Validate keep‑alive behavior


$ grpcurl -plaintext -proto generate.proto localhost:8000 Generate/StreamTokens
# Observe ping frames every 30 s in the client trace

3. Run an end‑to‑end streaming request


>>> curl -N -X POST http://localhost:8000/api/pipeline \
    -H "Content-Type: application/json" \
    -d '{"query":"What is the capital of France?"}'
The capital of France is Paris.

The response completes without interruption.

4. Simulate a failure and ensure watchdog recovery


# Kill the inference process manually
$ docker exec inference-service pkill -SIGKILL python

# Watchdog logs
2024-10-27 14:15:02,112 INFO watchdog - inference unhealthy, restarting...
2024-10-27 14:15:07,423 INFO watchdog - inference service back online

# Re‑run the streaming request – it succeeds

Prevent

Resource Planning: Allocate at least 150 % of the model’s peak VRAM + system RAM to the inference container.
Health‑check Granularity: Use gRPC health checking (e.g., grpc.health.v1.Health) instead of a simple HTTP ping.
Observability: Export container OOM metrics to Prometheus and set alerts on container_memory_failcnt > 0.
Deployment Guardrails: Pin the inference service version and lock the Docker image digest to avoid accidental upgrades that change socket behavior.
Network Isolation: Prefer TCP loopback with explicit SO_REUSEPORT over Unix sockets in air‑gapped deployments where container restarts are common.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the streaming response work locally but fail after deployment? Locally the inference process runs with unlimited host memory, so OOM kills never occur. In the containerized deployment memory limits trigger a hard kill, which silently closes the gRPC stream.
Can I rely on the /healthz endpoint for streaming reliability? No. /healthz only checks process liveness. Use the gRPC health service to detect socket‑level failures.
Is increasing the gRPC timeout enough? It prevents premature client timeouts but does not solve the underlying socket closure. Combine timeout adjustments with keep‑alive and sufficient resources.
Do I need to change the Haystack pipeline definition? Only the generator configuration (keep‑alive, timeout) needs updating; the pipeline topology remains unchanged.
How do I monitor for intermittent OOM events? Export container_memory_failcnt and container_memory_usage_bytes to Prometheus, and create an alert when failcnt > 0 over a 5‑minute window.