Haystack document retrieval failures after cloud storage API intermittent timeouts during peak traffic

Problem

During peak traffic periods the API gateway that fronts a Haystack‑based search service starts returning 504 Gateway Timeout, 502 Bad Gateway and occasional 429 Too Many Requests errors. The downstream Haystack document store logs show repeated failures when trying to fetch source documents from the cloud storage provider (e.g., AWS S3, GCP Cloud Storage, Azure Blob). The symptoms are:

• Intermittent ReadTimeout exceptions from the storage SDK.
• HTTP 5xx responses from the storage endpoint.
• Rate‑limit headers (e.g., Retry-After) appearing only under load.
• Overall search latency spikes from < 100 ms to > 5 s.

These failures cascade: Haystack cannot retrieve the document text, the query processor aborts, and the API gateway propagates the error to the client.

Root Cause

The cloud storage provider enforces per‑client request quotas that are dynamically adjusted based on observed traffic patterns. Under sustained high QPS the provider:

1. Detects a burst of GET Object calls from a single IP range.
2. Enters a throttling state, returning 503 Service Unavailable with a Retry-After header.
3. If the client does not respect the back‑off, the provider escalates to 429 Too Many Requests and eventually drops connections, causing socket timeouts.

Haystack’s default DocumentStore implementation uses a synchronous HTTP client with a fixed 2 s timeout and no built‑in retry/back‑off logic. When the storage API starts throttling, the client immediately times out, producing the observed errors.

Debug

Below is a typical log excerpt from the Haystack worker when the issue occurs:

2026-10-02 14:12:37,842 - haystack.document_stores.s3 - ERROR - S3 GET failed for key docs/12345.txt: ReadTimeoutError: HTTPSConnectionPool(host=s3.amazonaws.com, port=443): Read timed out. (timeout=2)
2026-10-02 14:12:38,001 - haystack.nodes.retriever - INFO - Retrieval aborted after 1 attempts
2026-10-02 14:12:38,003 - api.gateway - ERROR - 504 Gateway Timeout: upstream service failed

Key observations from the logs:

• Timeout value is hard‑coded to 2 seconds.
• No retry attempts are logged.
• The gateway error correlates with the storage timeout.

Diagnostic steps performed:

1. Network trace with tcpdump on the worker node showed repeated SYN retransmissions to the storage endpoint during the spike.
2. Metrics inspection in Prometheus revealed a sharp rise in http_requests_total{service="s3"}[1m] from 500 RPS to > 3 000 RPS.
3. SDK instrumentation enabled detailed debug output for the storage client, confirming receipt of 503 and 429 responses.

Sample command to capture the storage responses:

sudo tcpdump -i eth0 -s 0 -w /tmp/s3.pcap host s3.amazonaws.com and port 443

After filtering the capture with tshark the following HTTP status codes were observed:

HTTP/1.1 503 Service Unavailable
Retry-After: 30

HTTP/1.1 429 Too Many Requests
Retry-After: 60

Solution

The fix consists of three coordinated changes:

1. Introduce exponential back‑off with jitter

Replace the raw SDK call with a wrapper that respects Retry-After and performs bounded retries.

# before (simplified)
def fetch_document(key):
    return s3_client.get_object(Bucket=BUCKET, Key=key)['Body'].read()

# after
import time
import random
import botocore.exceptions

MAX_RETRIES = 5
BASE_DELAY = 0.5  # seconds

def fetch_document(key):
    attempt = 0
    while True:
        try:
            resp = s3_client.get_object(Bucket=BUCKET, Key=key)
            return resp['Body'].read()
        except botocore.exceptions.ClientError as e:
            status = e.response['ResponseMetadata']['HTTPStatusCode']
            if status in (429, 503):
                retry_after = e.response.get('RetryAfter')
                delay = float(retry_after) if retry_after else BASE_DELAY * (2 ** attempt)
                jitter = random.uniform(0, delay * 0.1)
                time.sleep(delay + jitter)
                attempt += 1
                if attempt > MAX_RETRIES:
                    raise
            else:
                raise

2. Increase the client timeout and enable connection pooling

Configure the underlying HTTP adapter to use a larger timeout (e.g., 10 s) and keep‑alive connections.

# before
s3_client = boto3.client('s3', config=botocore.client.Config(connect_timeout=2, read_timeout=2))

# after
s3_client = boto3.client(
    's3',
    config=botocore.client.Config(
        connect_timeout=5,
        read_timeout=10,
        retries={'max_attempts': 0},  # disable SDK retries, we handle them ourselves
        max_pool_connections=50
    )
)

3. Apply request rate limiting at the API gateway

Introduce a token‑bucket limiter that caps the outbound S3 request rate per worker to a safe value (e.g., 1 200 RPS). Below is an example using envoy rate limit filter:

http_filters:
- name: envoy.filters.http.ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
    domain: s3_rate_limit
    request_type: both
    timeout: 0.025s
    rate_limit_service:
      grpc_service:
        envoy_grpc:
          cluster_name: rate_limit_cluster

Corresponding rate_limit_cluster points to a lightweight in‑process limiter that tracks per‑worker counters.

Verify

After deploying the changes, perform the following validation steps:

1. Smoke test a single document fetch with curl against the API gateway; response time should be < 200 ms.
2. Load test using hey or wrk at 3 000 RPS for 5 minutes. Monitor:
   • Haystack logs – no longer see ReadTimeoutError.
   • Prometheus metric http_requests_total{service="s3"}[5m] should stay below the configured limit.
   • Storage provider metrics – absence of 429/503 responses.
3. Health check endpoint /healthz should return 200 with a sub‑check confirming successful S3 connectivity.

Sample successful log entry after the fix:

2026-10-02 14:45:12,311 - haystack.document_stores.s3 - INFO - S3 GET succeeded for key docs/98765.txt after 2 retries (total latency 4.3s)

Prevent

To avoid recurrence, implement the following guardrails:

• Monitoring: Alert on http_requests_total{service="s3"}[1m] > 1,200 and on any 5xx or 429 responses from the storage endpoint.
• Circuit breaker: Deploy a per‑worker circuit breaker that temporarily halts S3 calls if error rate exceeds 5 % over a 30‑second window.
• Capacity planning: Coordinate with the cloud provider to obtain the exact request quota for the account and request a higher limit if needed.
• Configuration as code: Store the retry/back‑off parameters and rate‑limit thresholds in a version‑controlled config.yaml and enforce CI linting.
• Graceful degradation: Cache frequently accessed documents in an in‑memory store (e.g., Redis) to reduce peak S3 load.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

• Why does increasing the timeout alone not solve the problem? The storage service actively throttles during spikes; a longer timeout only prolongs the waiting period and can still result in 5xx responses. Without back‑off, the client continues hammering the endpoint, worsening the throttle.
• Can I rely on the SDK’s built‑in retry mechanism? The SDK retries are limited to idempotent errors and use a fixed back‑off that does not honor Retry-After. A custom wrapper gives deterministic control and respects provider‑specified delays.
• What if the provider’s quota is insufficient even after back‑off? Request a quota increase or introduce a secondary storage tier (e.g., multi‑region replication) to spread the load.
• How do I differentiate between network latency spikes and provider throttling? Throttling responses include HTTP 429/503 with Retry-After. Pure latency spikes return 200 but with high response times; correlating with the storage service’s status page can confirm throttling events.
• Is it safe to cache documents indefinitely? Cache staleness must be bounded. Use a TTL aligned with the source data update frequency (e.g., 5 minutes) and invalidate on write‑through events.