Haystack RAG reranker timeout after autoscaling event
Problem
During query processing the Haystack RAG pipeline fails with a timeout error from the Reranker component. The error message returned to the client is:
RerankerError: query processing exceeded 30.0 seconds (timeout)
Symptoms observed after a horizontal pod autoscaling (HPA) event:
- Spike in
Rerankerlatency (often > 45 s) - Readiness probe failures for the reranker pod
- CPU usage temporarily drops to < 10 % while memory stays near the limit
- Subsequent queries are routed to the new pod before the old pod has fully initialized, causing a cascade of timeouts.
Root Cause
The reranker service is deployed as a single‑replica Deployment with a CPU request of 500m and a memory limit of 1Gi. The HPA is configured to scale the haystack‑pipeline Deployment (which includes the retriever, generator, and reranker containers) based on CPU utilization.
When the cluster scales out, Kubernetes creates a new pod and terminates the existing one after the terminationGracePeriodSeconds expires (default 30 s). The reranker container performs a heavy model warm‑up (loading a sentence‑transformers model) that takes ~40 s. Because the pod is killed before the model finishes loading, the next pod receives traffic while its own warm‑up is still in progress. The readiness probe (which checks /health) returns 200 as soon as the HTTP server starts, not when the model is ready, so the service load‑balancer routes queries to a pod that cannot answer within the configured 30‑second timeout.
In summary:
- Autoscaling triggers pod replacement.
- Reranker model warm‑up exceeds the termination grace period.
- Readiness probe signals “ready” too early.
- Incoming queries hit a partially‑initialized reranker, causing timeout errors.
Debug & Investigation
Step‑by‑step diagnostics performed in the affected cluster:
- Inspect pod events and termination timestamps
kubectl get pods -n haystack -l app=haystack-reranker -o wide
kubectl describe pod haystack-reranker-5d9f6c9f9b-7kz9m -n haystack
Relevant excerpt:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m deployment-controller Scaled up replica set haystack-reranker-5d9f6c9f9b to 2
Normal Killing 1m kubelet, gke-node-1 Killing container with id docker://reranker:Need to kill pod
Normal SuccessfulDelete 55s replicaset-controller Deleted pod: haystack-reranker-5d9f6c9f9b-7kz9m
- Check reranker container logs for warm‑up duration
kubectl logs haystack-reranker-5d9f6c9f9b-7kz9m -c reranker -n haystack
2024-11-18 10:12:03,001 INFO Loading sentence‑transformers model...
2024-11-18 10:12:45,237 INFO Model loaded in 42.2 seconds
2024-11-18 10:12:45,238 INFO Reranker ready to serve requests
The log shows that the model load takes ~42 s, longer than the default terminationGracePeriodSeconds (30 s).
- Validate readiness probe definition
kubectl get deployment haystack-reranker -n haystack -o yaml | grep -A5 readinessProbe
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
The probe only checks the HTTP endpoint, which becomes available before the model is fully loaded.
- Measure query latency before and after scaling
curl -s -w "\n%{time_total}\n" http://haystack-reranker.haystack.svc.cluster.local/query -d '{"query":"What is the capital of France?"}'
{"answers":[...]}
0.842
After scaling:
{"error":"Reranker timeout after 30.0 seconds"}
30.001
Solution
The fix consists of three coordinated changes:
- Increase the termination grace period to exceed the model warm‑up time.
- Make the readiness probe wait for the model to be fully loaded.
- Add a side‑car init container that pre‑loads the model into a shared volume so that subsequent pods can start serving instantly.
1. Extend terminationGracePeriodSeconds
Update the Deployment manifest:
Before
apiVersion: apps/v1
kind: Deployment
metadata:
name: haystack-reranker
spec:
replicas: 1
template:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: reranker
image: haystack/reranker:latest
resources:
limits:
cpu: "500m"
memory: "1Gi"
After
apiVersion: apps/v1
kind: Deployment
metadata:
name: haystack-reranker
spec:
replicas: 1
template:
spec:
terminationGracePeriodSeconds: 90 # > model load time
containers:
- name: reranker
image: haystack/reranker:latest
resources:
limits:
cpu: "500m"
memory: "1Gi"
2. Refine readiness probe to check model status
Modify the application to expose /ready only after the model is loaded. Then adjust the probe:
Before
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
After
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 12 # allows up to 60 s before marking unhealthy
Application snippet (Python FastAPI example):
model_ready = False
@app.on_event("startup")
async def load_model():
global model_ready
# heavy loading
await load_sentence_transformer()
model_ready = True
@app.get("/ready")
def readiness():
if model_ready:
return {"status":"ready"}
raise HTTPException(status_code=503, detail="model loading")
3. Add init container to pre‑warm model
Store the downloaded model in an emptyDir volume shared between the init container and the main container.
Deployment snippet
spec:
volumes:
- name: model-cache
emptyDir: {}
initContainers:
- name: model-warmup
image: haystack/reranker:latest
command: ["python", "-c"]
args:
- |
from sentence_transformers import SentenceTransformer
SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2',
cache_folder='/model')
volumeMounts:
- name: model-cache
mountPath: /model
containers:
- name: reranker
image: haystack/reranker:latest
env:
- name: TRANSFORMERS_CACHE
value: /model
volumeMounts:
- name: model-cache
mountPath: /model
The init container performs the download once; subsequent pod restarts load the model from the local cache in <~5 s, well within the readiness window.
Verification
After applying the manifest changes, perform the following checks:
- Confirm pod lifecycle
kubectl get pod -l app=haystack-reranker -w
Observe that when a new pod is created the old pod stays alive for ~90 s, allowing the model to finish loading.
- Validate readiness endpoint
curl -s http://haystack-reranker.haystack.svc.cluster.local/ready
{"status":"ready"}
- Run a load test before and after scaling
for i in {1..10}; do
time curl -s -X POST http://haystack-reranker.haystack.svc.cluster.local/query \
-d '{"query":"Explain quantum entanglement"}' -o /dev/null
done
All requests should complete under 2 seconds, and no Reranker timeout errors appear in the logs.
Prevention
- Set
terminationGracePeriodSecondsto at leastmodel_load_time + 10 sfor any service that performs heavyweight initialization. - Expose a dedicated readiness endpoint that reflects actual application readiness, not just HTTP server start.
- Use init containers or shared caches to amortize model download costs across pod restarts.
- Configure HPA to scale based on request latency (e.g.,
custom.metrics.k8s.io/response_time) rather than pure CPU, reducing unnecessary restarts during traffic spikes. - Monitor
container_start_time_secondsandcontainer_restart_countmetrics; alert if restarts exceed a threshold within a short window.
FAQ
- Why does the timeout only appear after an autoscaling event?
Because the new pod is considered ready before the model finishes loading, so incoming queries are routed to a cold instance that cannot answer within the 30‑second limit. - Can I keep the original readiness probe and just increase the timeout?
Increasing the timeout masks the problem but does not guarantee service availability; queries will still experience high latency. A proper readiness check ensures traffic only reaches fully‑initialized pods. - Is the
emptyDirvolume sufficient for large models?
For models < 1 Gi,emptyDiron node local SSD is fine. For larger artifacts consider ahostPathor a persistent volume backed by fast storage. - How do I know the exact model warm‑up time for sizing
terminationGracePeriodSeconds?
Measure the duration in a controlled pod (e.g.,time python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('model')") and add a safety margin of 10–15 seconds. - Will the init container increase pod startup latency for every replica?
The init container runs only once per node; subsequent pods reuse the cached model, reducing warm‑up to a few seconds.
Related Topic Hub: RAG Systems Troubleshooting Hub