Problem – RAG Reranker Timeout After Rolling Update of Anthropic Claude Service
The Retrieval‑Augmented Generation (RAG) pipeline uses a reranker microservice that forwards candidate passages to the Anthropic Claude LLM for scoring. After a rolling update of the Claude service across a 6‑node fleet, the reranker began returning HTTP 504 timeout errors for POST /rerank calls. The symptom manifested as incomplete responses in the downstream chat API and a measurable increase in end‑to‑end latency from the usual 200 ms to > 5 s.
Root Cause – Inconsistent TLS Handshake Timeout During Staggered Rollout
Claude nodes are fronted by an internal Envoy load balancer that terminates TLS and forwards traffic over HTTP/2 to the model server. The rolling update introduced a new version of the model binary that, due to a regression in the OpenSSL configuration, only advertised TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher suites. Existing client pods (the reranker) were built with a stricter SSLContext that only allowed TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. During the rollout, the load balancer gradually shifted traffic to the upgraded nodes, causing a subset of requests to hit a node that rejected the client’s cipher suite. The TLS handshake therefore failed after the client‑side 2‑second timeout, which the reranker propagated as a 504.
Key points:
- Only the upgraded nodes exhibited the mismatch; older nodes continued to work.
- The mismatch was not visible in the service health checks because they use a permissive curl command that falls back to a supported cipher.
- The rolling update did not pause traffic draining, so some in‑flight requests were forced onto the new nodes before the client pods were redeployed.
Debug – Investigation Steps
1. Capture Reranker Logs
2026-11-06T12:03:41.112Z INFO reranker: Received query id=7f3c2a1b
2026-11-06T12:03:41.115Z DEBUG reranker: Forwarding 5 candidates to Claude (node=claude-03)
2026-11-06T12:03:41.117Z ERROR reranker: TLS handshake timeout after 2000ms (node=claude-03)
2026-11-06T12:03:41.119Z WARN reranker: Returning partial result after timeout
2. Verify TLS Handshake Failure with OpenSSL
$ openssl s_client -connect claude-03.internal:443 -tls1_2 -cipher ECDHE-ECDSA-AES128-GCM-SHA256
CONNECTED(00000003)
...
handshake failure
Expected output on a healthy node shows the certificate chain and a successful handshake.
3. Inspect Envoy Listener Configuration
listeners:
- name: listener_claude
address:
socket_address:
address: 0.0.0.0
port_value: 443
filter_chains:
- filter_chain_match:
transport_protocol: "tls"
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/claude/tls.crt
private_key:
filename: /etc/claude/tls.key
tls_params:
tls_minimum_protocol_version: TLSv1_2
tls_maximum_protocol_version: TLSv1_3
cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
The cipher_suites field was unintentionally overridden by the new deployment manifest.
4. Check Rolling Update Strategy
kubectl get deployment claude -o yaml | grep strategy -A5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
With maxUnavailable: 0, no pods were terminated before the new pods became ready, but the readiness probe only checked HTTP 200 on /healthz, which succeeded despite the TLS mismatch.
Solution – Align Cipher Suites and Adjust Rolling Update Guardrails
1. Restore Compatible Cipher Suite List
Update the Envoy configuration to include the cipher suite required by the reranker.
Before (faulty snippet):
tls_params:
cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
After (compatible snippet):
tls_params:
cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
Deploy the corrected ConfigMap and restart Envoy sidecars.
2. Tighten Readiness Probe to Validate TLS Handshake
readinessProbe:
exec:
command:
- /usr/bin/openssl
- s_client
- -connect
- localhost:443
- -tls1_2
- -cipher
- ECDHE-ECDSA-AES128-GCM-SHA256
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
This ensures a pod is only marked ready when the required cipher suite is actually usable.
3. Modify Rolling Update Strategy to Drain Traffic
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1 # allow one pod to be unavailable during rollout
Combined with the stricter readiness probe, this guarantees that traffic is never sent to a pod that fails the TLS test.
4. Redeploy Reranker with Relaxed Cipher List (optional)
If supporting multiple cipher suites on the client side is acceptable, adjust the Python SSL context:
import ssl
ctx = ssl.create_default_context()
ctx.set_ciphers('ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384')
This change is defensive and should be applied only after confirming the server side is stable.
Verify – Confirming the Fix
Functional Test
$ curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
https://claude-prod.internal/rerank -d '{"query":"..."}' -H "Content-Type: application/json"
200 0.212
Latency returns to ~200 ms and HTTP 200 is observed consistently across all nodes.
Log Inspection
2026-11-06T12:45:02.003Z INFO reranker: Forwarding 5 candidates to Claude (node=claude-04)
2026-11-06T12:45:02.210Z INFO reranker: Received ranking response (node=claude-04)
No TLS timeout entries appear for a 10‑minute observation window.
Metrics Dashboard
| Metric | Pre‑Fix | Post‑Fix |
|---|---|---|
| rerank_http_504_total | 152 | 0 |
| rerank_latency_seconds (p95) | 4.8 | 0.22 |
| claude_tls_handshake_failure_total | 147 | 0 |
Prevent – Operational Guardrails
- Version‑locked TLS profiles: Store the allowed cipher suite list in a ConfigMap versioned alongside the model binary. Enforce a CI lint rule that checks for mismatches between client and server profiles.
- Readiness probes that exercise TLS: Use an exec probe with
openssl s_clientrather than a simple HTTP health check. - Canary rollout with traffic shadowing: Deploy the new Claude version to a dedicated canary namespace, route a fixed 5 % of traffic, and monitor
tls_handshake_failure_totalbefore scaling. - Alert on TLS handshake failures: Create a Prometheus alert on
claude_tls_handshake_failure_total > 0with a 5‑minute evaluation window. - Automated compatibility test: In the CI pipeline, spin up a temporary Envoy sidecar with the new config and run a matrix of
openssl s_clientcipher tests against the client binary.
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why did the health check not catch the TLS mismatch? The default
/healthzendpoint only verifies HTTP 200 on the Envoy listener, which succeeds before TLS negotiation. Adding an exec probe that performs a real TLS handshake surfaces the incompatibility. - Can I keep the old cipher suite and still use the new model binary? Yes, the model binary does not depend on a specific cipher; the regression was in the Envoy config template. Restoring the full cipher list resolves the issue without rolling back the model.
- What is the safest rolling update strategy for services that terminate TLS? Use
maxUnavailable: 1(or higher) combined with a readiness probe that validates TLS. This forces the scheduler to keep at least one healthy pod serving traffic while the new pod is verified. - How do I debug similar issues if the client library does not expose TLS errors? Enable
SSLKEYLOGFILEon the client container and capture the handshake withtcpdump -i any -w capture.pcap port 443. Analyzing the pcap with Wireshark will reveal the Alert (Handshake Failure) message. - Is there a way to make the reranker fallback to a different LLM if Claude times out? Implement a circuit‑breaker pattern around the Claude call; on timeout, route the request to a secondary model endpoint (e.g., OpenAI GPT) and merge the scores. This provides resilience but adds latency, so it should be used as a last resort.