a rack of servers in a server room

Meta LLaMA webhook timeout during canary deployment on Kubernetes

by

Problem – Webhook Timeout During Canary Deployment of Meta LLaMA

During a canary rollout on an Amazon EKS cluster, 10 % of traffic was routed to a new version of the Meta LLaMA inference service. The service invokes an external monitoring endpoint via a REST webhook after each inference request. Operators observed a surge in failed inference calls and incomplete metric logs. Typical error messages included:

  • context deadline exceeded – logged by the LLaMA inference container.
  • client.TimeoutError: Get "https://monitoring.example.com/metrics": net/http: request canceled (timeout exceeded) – Go client stack trace.
  • Webhook request timed out (HTTP 504) – recorded in the Kubernetes API server logs.
  • Readiness probe failed: Get http://localhost:8080/healthz: dial tcp 127.0.0.1:8080: i/o timeout – indicating the webhook pod was not ready.

The failure manifested as:

  • HTTP 504 responses to downstream clients.
  • Missing entries in the external monitoring system.
  • Elevated latency metrics on the llama-inference service.

Root Cause Analysis

Interaction Between Canary Traffic, Inference Processing, and Webhook Calls

The LLaMA inference server (official deployment guide) processes a request, then synchronously posts metrics to the monitoring webhook using a configurable --request-timeout flag (default 30 s). During a canary rollout, the new model version introduced a 15 % increase in per‑request compute time. When combined with the existing 30 s webhook timeout, the total latency for some requests exceeded the allowed window, causing the Go client to emit context deadline exceeded.

Resource Saturation

Evidence from a fintech incident shows that a 10 % traffic canary can increase webhook latency by up to 40 % when the webhook service pod count is insufficient. In our cluster, the webhook deployment had a replica count of 1, leading to a saturated request queue as the canary traffic spiked.

Network Policy & Ingress Idle Timeout

A separate AWS EKS case demonstrated that a network policy inadvertently blocked outbound traffic from the inference pod to the monitoring endpoint, resulting in repeated “context deadline exceeded” errors. Additionally, the ALB ingress controller’s default idle timeout of 30 s conflicted with the webhook’s 45 s processing time (AWS canary best practices), causing systematic 504 responses.

Investigation and Debugging

Log Inspection


2026-05-31T12:45:23Z ERROR inference: context deadline exceeded while calling webhook
2026-05-31T12:45:23Z DEBUG webhook: POST https://monitoring.example.com/metrics (duration: 32.7s)
2026-05-31T12:45:23Z INFO  kube-apiserver: webhook "monitoring-webhook" timed out after 30s

Metrics Review

Using kubectl top pods revealed the webhook pod CPU at 95 % and memory at 80 % during the canary window.


NAME                         CPU(cores)   MEMORY(bytes)
monitoring-webhook-5d9f7c9   950m         820Mi

Network Trace

Running a packet capture from the inference pod confirmed outbound TCP SYN retransmissions to monitoring.example.com:443 after the first 30 s.


$ tcpdump -i eth0 host monitoring.example.com and port 443 -w webhook.pcap
...
12:45:23.123456 IP pod-1234.56789 > monitoring.example.com.https: Flags [S], seq 0, win 29200, length 0
12:45:53.124001 IP pod-1234.56789 > monitoring.example.com.https: Flags [S], seq 0, win 29200, length 0
...

Admission Webhook Configuration Check

The webhook was registered with the default failurePolicy=Fail and timeoutSeconds=30 (Kubernetes Admission Webhook docs). This forced the API server to treat any timeout as a fatal error.

Resolution

Increase Webhook Request Timeout

Adjust the inference server’s --request-timeout flag to accommodate the longer processing time of the new model version.

Before:

containers:
- name: llama-inference
  image: llama:2.0
  args: ["--model=/models/v2", "--request-timeout=30"]

After:

containers:
- name: llama-inference
  image: llama:2.0
  args: ["--model=/models/v2", "--request-timeout=60"]

Scale the Monitoring Webhook Deployment

Increase replicas from 1 to 3 and enable horizontal pod autoscaling based on CPU utilization.

Before:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitoring-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: monitoring-webhook
  template:
    metadata:
      labels:
        app: monitoring-webhook
    spec:
      containers:
      - name: webhook
        image: monitoring/webhook:1.2

After:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitoring-webhook
spec:
  replicas: 3
  selector:
    matchLabels:
      app: monitoring-webhook
  template:
    metadata:
      labels:
        app: monitoring-webhook
    spec:
      containers:
      - name: webhook
        image: monitoring/webhook:1.2
        resources:
          requests:
            cpu: "200m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: monitoring-webhook-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: monitoring-webhook
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Adjust Ingress Idle Timeout

Configure the ALB listener to use a 90 s idle timeout, matching the new webhook processing window.


aws elbv2 modify-load-balancer-attributes \
    --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/llama-alb/abcd1234 \
    --attributes idle_timeout.timeout_seconds=90

Update Admission Webhook Timeout

Set timeoutSeconds to 60 s in the ValidatingWebhookConfiguration to give the API server a larger window.


apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: monitoring-webhook
webhooks:
- name: webhook.monitoring.example.com
  clientConfig:
    service:
      name: monitoring-webhook
      namespace: default
      path: /validate
    caBundle: 
  rules:
  - apiGroups: ["*"]
    apiVersions: ["v1"]
    operations: ["CREATE","UPDATE"]
    resources: ["pods"]
  failurePolicy: Ignore
  timeoutSeconds: 60

Validation

Functional Test

Issue a test inference request that triggers the webhook and verify a 200 OK response from the monitoring endpoint.


curl -X POST https://llama.example.com/v1/infer \
    -d '{"prompt":"Hello world"}' -w "\nHTTP %{http_code}\n"

Expected output:


{"result":"Hello world ..."}
HTTP 200

Log Confirmation

Search the inference pod logs for the absence of timeout messages.


kubectl logs -l app=llama-inference -c inference | grep "timeout"

Result: No matches.

Metrics Dashboard

Confirm that the webhook_latency_seconds histogram shows < 30 s for 99 % of calls, and that the ALB 504 count drops to zero.

Operational Experience & Prevention

  • Misleading Symptom: Initial alerts pointed to the inference pod CPU saturation, but the actual blocker was the outbound webhook latency.
  • Assumption Failure: The default 30 s webhook timeout was considered sufficient based on the original model’s latency profile; the new version altered that baseline.
  • Edge Case: NetworkPolicy rules that allow egress only to specific CIDR ranges caused intermittent “context deadline exceeded” spikes when the monitoring service scaled its IP pool.
  • Lesson Learned: Treat external webhook calls as part of the service’s SLA and provision both compute and network capacity accordingly during canary rollouts.

Best Practices and Prevention

  • Define separate ReadinessProbe and LivenessProbe timeouts that exceed the maximum expected webhook processing time.
  • Enable failurePolicy=Ignore for non‑critical monitoring webhooks to prevent API server request rejection during transient spikes.
  • Configure Horizontal Pod Autoscaling for webhook services based on both CPU and request latency metrics.
  • Monitor apiserver_admission_webhook_latency_seconds and set alerts for latency > 40 s.
  • Use connection draining on the ALB during version upgrades to avoid abrupt termination of in‑flight webhook calls.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the webhook timeout only after a canary rollout? The new model version increases per‑request processing time, pushing the total latency beyond the static 30 s webhook timeout. A canary introduces a traffic burst that quickly saturates a single‑replica webhook pod.
  2. How can I determine the optimal --request-timeout value? Benchmark the end‑to‑end latency of the new model version under realistic load, add a safety margin (e.g., 20 %), and set the flag accordingly. Use wrk or hey to generate load and measure the 99th‑percentile latency.
  3. What is the impact of changing the AdmissionWebhook failurePolicy to Ignore? The API server will treat webhook failures as non‑blocking, allowing the primary request to proceed. This prevents 504 errors caused by temporary webhook latency spikes, but you lose strict validation for those calls.
  4. Can service mesh idle timeouts cause similar issues? Yes. If using Istio or Linkerd, ensure the outbound egress timeout is greater than the webhook processing time; otherwise, the mesh will abort the call and surface a 504.
  5. Do I need to adjust TLS handshake settings for multi‑region deployments? Inconsistent TLS handshake latency can add several seconds. Verify that all clusters share the same cipher suite list and that the CA bundle is refreshed across regions to avoid intermittent spikes.