Hugging Face Transformers query latency spike after Kubernetes autoscale

Problem Description

During peak traffic (≈10 000 req/min) the Hugging Face Transformers inference service, deployed behind an NGINX Ingress controller on a Kubernetes cluster with Horizontal Pod Autoscaler (HPA) enabled, exhibited a sudden latency increase:

• Baseline latency: ~50 ms per request.
• Observed tail latency: 1 s – 2 s, often resulting in client‑side timeouts.
• NGINX logs showed repeated 504 Gateway Timeout errors:

2024/06/19 12:45:23 [error] 12#12: *1023 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.2.3.45, server: api.example.com, request: "POST /v1/predict HTTP/1.1", upstream: "http://10.244.2.17:8080/v1/predict", host: "api.example.com"

Additional symptoms from the pod logs:

2024-06-19 12:44:58,721 INFO  [torchserve.entrypoint] Model loading started for model_id=distilbert-base-uncased
2024-06-19 12:45:31,842 INFO  [torchserve.entrypoint] Model loading completed (elapsed=33.1s)
2024-06-19 12:46:07,014 WARN  [torchserve.worker] Worker timeout: request took longer than 30 seconds

The latency spike coincided with HPA scaling events that added new pods to the deployment.

Root Cause Analysis

Multiple interacting factors caused the latency regression:

1. Cold‑start model loading – New pods started by HPA must load the transformer model from remote storage (S3/EFS). As documented in the Transformers Inference Guide, model loading can take 500‑800 ms, during which the pod cannot serve traffic. This matches the “Cold‑start latency after HPA adds new pods” incident.
2. Insufficient CPU resources on start‑up – Autoscaled pods inherit the default limit of cpu: 500m. Tokenization and preprocessing run on CPU; with limited CPU the pod queues requests, inflating latency (see the “CPU throttling on pod startup” incident).
3. Readiness probe misconfiguration – The probe checks /health immediately after container start. Because the model isn’t loaded yet, the probe fails, causing the pod to be marked Ready only after the model is loaded. During the window when the pod is registered by the service but not ready, NGINX forwards traffic, leading to 504 errors (evidence from “Readiness probe failure” logs).
4. NGINX upstream timeout and buffering – Default proxy_read_timeout 60s and limited proxy_buffering cause request buffering to fill quickly under burst traffic, resulting in “upstream timed out” errors (see NGINX Ingress Controller docs).
5. Aggressive scaling policy – HPA’s short scaleDownDelay (30 s) caused pods to be terminated before they could warm‑up, leading to repeated cold starts and “Pod thrashing” (incident reference).

Collectively, these factors produced a latency tail >1 s and intermittent 504 errors during traffic spikes.

Investigation and Debugging

Step‑by‑step diagnostics performed:

1. Inspect HPA events to confirm scaling frequency:

kubectl describe hpa transformer-api
Events:
  Type    Reason             Age   From               Message
  ----    ------             ----  ----               -------
  Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 8; reason: cpu utilization above target
  Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  New size: 12; reason: cpu utilization above target

2. Check pod startup logs for model load time:

kubectl logs -f deployment/transformer-api -c torchserve --tail=20
...
2024-06-19 12:44:58,721 INFO  Model loading started for model_id=distilbert-base-uncased
2024-06-19 12:45:31,842 INFO  Model loading completed (elapsed=33.1s)

3. Measure CPU throttling:

kubectl exec -it $(kubectl get pod -l app=transformer-api -o jsonpath="{.items[0].metadata.name}") -- \
  cat /sys/fs/cgroup/cpu/cpu.stat
nr_periods nr_throttled nr_time_throttled
1000        850          1200000000

4. Review NGINX Ingress status:

kubectl exec -n ingress-nginx $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].metadata.name}") -- \
  curl -s http://localhost:10254/metrics | grep nginx_upstream
nginx_upstream_response_time_seconds_sum{...} 125.4
nginx_upstream_response_time_seconds_count{...} 2000

5. Validate readiness probe timing:

kubectl get pod -l app=transformer-api -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP            NODE
transformer-api-7f8c9d9d5b-abcde   0/1     Running   0          30s   10.244.2.17   ip-10-0-0-1

Pod remained 0/1 until the model finished loading.

Resolution

The fix involved three coordinated changes: pre‑warming, resource allocation, and ingress tuning.

1. Pre‑warm pods using a InitContainer

Load the model into a shared volume before the main container starts, eliminating per‑pod load latency.

Before:

containers:
- name: torchserve
  image: huggingface/transformers:latest
  command: ["torchserve", "--start"]
  resources:
    limits:
      cpu: "500m"
      memory: "2Gi"

After:

initContainers:
- name: model-warmup
  image: huggingface/transformers:latest
  command: ["python", "-c", "from transformers import AutoModel; AutoModel.from_pretrained('/model')"]
  volumeMounts:
  - name: model-volume
    mountPath: /model
containers:
- name: torchserve
  image: huggingface/transformers:latest
  command: ["torchserve", "--start"]
  volumeMounts:
  - name: model-volume
    mountPath: /model
  resources:
    limits:
      cpu: "500m"
      memory: "2Gi"
volumes:
- name: model-volume
  persistentVolumeClaim:
    claimName: model-pvc

The InitContainer ensures the model is cached on the node’s local storage before the inference container becomes ready.

2. Increase CPU limits and enable burstable QoS

Allocate sufficient CPU for tokenization and avoid throttling.

Before:

resources:
  limits:
    cpu: "500m"
    memory: "2Gi"
  requests:
    cpu: "250m"
    memory: "1Gi"

After:

resources:
  limits:
    cpu: "2000m"
    memory: "4Gi"
  requests:
    cpu: "1000m"
    memory: "2Gi"

These values were chosen based on observed CPU usage (~1.4 cores) during tokenization (see Performance Guide).

3. Adjust NGINX Ingress timeout and buffering

Increase proxy_read_timeout and enable proxy_buffering to accommodate occasional warm‑up delays.

Before (default):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"

After:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"

4. Stabilize HPA scaling policy

Introduce a scaleDownDelay of 5 minutes and raise the cpuUtilizationTarget to 80 % to reduce rapid churn.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: transformer-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: transformer-api
  minReplicas: 4
  maxReplicas: 20
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Validation

After applying the changes, the following checks confirmed the resolution:

1. Latency measurement:

ab -n 5000 -c 100 -p payload.json -T application/json http://api.example.com/v1/predict
...
Time per request:       58.3 [ms] (mean)

Latency returned to the baseline ~55 ms**.

2. NGINX error rate:

kubectl logs -n ingress-nginx $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[0].metadata.name}") | grep "504"
# No output – 504 errors eliminated

3. Pod readiness timing:

kubectl get pods -l app=transformer-api -o jsonpath="{range .items[*]}{.metadata.name}:{.status.containerStatuses[0].ready}{'\n'}{end}"
transformer-api-7f8c9d9d5b-abcde:true
transformer-api-7f8c9d9d5b-fghij:true

4. CPU throttling metrics:

kubectl top pod -l app=transformer-api
NAME                               CPU(cores)   MEMORY(bytes)
transformer-api-7f8c9d9d5b-abcde   1.2          3.1Gi

CPU usage stays within the allocated limit, no throttling observed.

Operational Experience

• Initial assumption that the latency spike was purely network‑related proved false; the real bottleneck was model warm‑up on newly created pods.
• Readiness probes that only check a health endpoint can mask the fact that the model isn’t loaded yet. Extending the probe to run a lightweight inference request (e.g., a dummy tokenization) provides a more accurate readiness signal.
• Aggressive HPA scaling saved resources under low load but introduced thrashing under bursty traffic. Adding a stabilization window dramatically reduced churn without sacrificing elasticity.
• Enabling proxy_buffering prevented NGINX from prematurely closing connections when upstream pods were still initializing.

Best Practices and Prevention

• Warm‑up strategy: Use an InitContainer or a sidecar that loads the model into a shared cache before traffic is accepted.
• Resource sizing: Allocate enough CPU for tokenization; benchmark tokenizers on CPU to set realistic limits.
• Readiness probes: Probe an endpoint that performs a minimal inference (e.g., /healthz?warmup=1) to ensure the model is fully loaded.
• HPA tuning: Set scaleDownDelay ≥ 3 minutes and avoid scaling below the minimum number of warm pods needed for the expected burst traffic.
• Ingress configuration: Increase proxy_read_timeout and enable buffering; monitor nginx_upstream_queue_length metrics for early detection of back‑pressure.
• Monitoring: Track model load duration (custom metric from TorchServe logs), CPU throttling (cpu.throttled.time), and NGINX queue length to catch regressions before they affect users.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does latency only increase after an autoscaling event?
   Because each new pod must load the transformer model from remote storage, which takes 500‑800 ms. Until the model is loaded, the pod cannot serve requests, causing request queuing and timeouts.
2. Can I avoid cold starts without increasing the number of always‑ready pods?
   Yes. Use an InitContainer to preload the model into a shared volume, or employ a sidecar that keeps the model in memory across pod restarts. This reduces per‑pod load time to a few milliseconds.
3. What CPU limit should I set for tokenization‑heavy workloads?
   Benchmark your tokenizer; most BERT‑style tokenizers consume ~1.2 CPU cores per instance under load. A safe limit is 2 CPU (2000m) with a request of 1 CPU to avoid throttling.
4. How do I differentiate between NGINX timeouts caused by upstream latency vs. downstream pod unavailability?
   Check NGINX logs for “upstream timed out” and correlate with pod readiness events. If pods are in Running but not Ready, the issue is upstream latency. If pods are missing or terminated, it’s a routing problem.
5. Is it safe to increase the HPA’s maxReplicas arbitrarily?
   Increasing maxReplicas alone does not solve latency; you must also ensure each replica can start quickly (warm‑up) and has sufficient resources. Over‑provisioning can lead to resource contention and OOM errors.