Kubernetes AI Inference Latency Spike During Peak Hours

Q: Debug – Systematic Investigation Steps 1. Verify Ingress Queueing

# Inspect NGINX status endpoint (if enabled) curl -s http://:10254/metrics | grep nginx_worker_connections Expected output (default):

Q: 2. Check HPA Scaling Timeline

# Show recent HPA events kubectl describe hpa inference-service Look for events such as:

Q: 3. Examine Conntrack Utilization

# On a node running inference pods sysctl net.netfilter.nf_conntrack_count sysctl net.netfilter.nf_conntrack_max Typical output during normal load:

Q: 4. Correlate GPU Allocation Errors

# Stream container runtime logs kubectl logs -f -c inference-container Sample error:

Q: 5. Capture Network Latency

# tcpdump on the ingress node for a few seconds sudo tcpdump -i eth0 -nn -s 0 -w /tmp/ingress.pcap 'port 80 or port 443' Analysis with wireshark will show increased TCP retransmissions and longer SYN‑ACK RTTs when the conntrack table is full.

Q: 4. Reduce Node Provisioning Latency for GPU Nodes

Enable node auto‑upgrade with pre‑warmed GPU nodes in the cloud provider’s AI platform, or use a node pool with preemptible GPU VMs kept in Ready state. This eliminates the 30‑second “node becomes Ready” window reported in the incident “Autoscaler scale‑out lag on a managed GKE cluster”.

In this article

Table of Contents

Problem – Latency Spike in Kubernetes‑Hosted AI Inference Service

During peak traffic windows the inference endpoint that serves ~100 ms predictions suddenly starts responding in 1 s +. The spike is repeatable, lasts for the duration of the load burst, and then returns to baseline once traffic subsides.

Key observations:

CPU and memory usage on GPU‑accelerated pods stay within 30‑40 % of limits.
GPU utilization reported by nvidia-smi remains stable around 55 %.
Metrics‑server (Kubernetes Documentation – Resource Metrics API) shows no abnormal resource pressure.

Ingress logs contain entries such as:

nginx: [error] 12345#0: *6789 upstream timed out (110: Connection timed out) while reading response header from upstream

Pod logs occasionally show:

Failed to allocate GPU device: device not available

Root Cause – Confluence of Ingress Queueing, Autoscaling Lag, and Conntrack Saturation

The latency spike is not caused by a single component but by three interacting factors that become visible only under high request concurrency:

NGINX Ingress worker‑connection limit. The default worker_connections (≈1024) is insufficient for the burst of concurrent inference requests. When the limit is reached NGINX queues new connections, adding ~800 ms of wait time (Ingress‑NGINX performance guide). This matches the real incident where “NGINX Ingress controller hit its default ‘worker_connections’ limit during peak load”.
Horizontal Pod Autoscaler (HPA) reaction time. The HPA polls the metrics-server every 30 s (default) and scales out only after the metric window is satisfied (HPA design). During a sudden traffic surge the existing GPU pods cannot absorb the extra load, but the autoscaler does not create new pods quickly enough (HPA scaling lag issue).
Conntrack table saturation in the CNI plugin. Calico’s default nf_conntrack_max can be exhausted by the high number of short‑lived TCP flows generated by inference requests. When the table fills, new connections are dropped or delayed, which manifests as increased round‑trip times and occasional “client body buffer size” errors (Network policies documentation).

Individually each symptom could be dismissed (CPU looks fine, GPU appears idle), but together they create a feedback loop: ingress queues → request latency → HPA perceives high latency but not high CPU → scaling lag → more queuing.

Debug – Systematic Investigation Steps

1. Verify Ingress Queueing

# Inspect NGINX status endpoint (if enabled)
curl -s http://:10254/metrics | grep nginx_worker_connections

Expected output (default):

nginx_worker_connections 1024

During the spike the metric stays at the limit while nginx_connections_active climbs to the same value, indicating saturation.

2. Check HPA Scaling Timeline

# Show recent HPA events
kubectl describe hpa inference-service

Look for events such as:

ScalingReplicaSet from 3 to 6 at 2026-06-25T14:02:30Z

If the ScaleUp event occurs >30 s after the traffic burst started, the autoscaler is the bottleneck.

3. Examine Conntrack Utilization

# On a node running inference pods
sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max

Typical output during normal load:

net.netfilter.nf_conntrack_count = 842
net.netfilter.nf_conntrack_max = 65536

During the spike you may see:

net.netfilter.nf_conntrack_count = 64200

Approaching the limit confirms table saturation.

4. Correlate GPU Allocation Errors

# Stream container runtime logs
kubectl logs -f  -c inference-container

Sample error:

Failed to allocate GPU device: device not available

These messages appear when the node’s GPU device plugin cannot satisfy a new pod request because the driver is still initializing the GPU after node provisioning.

5. Capture Network Latency

# tcpdump on the ingress node for a few seconds
sudo tcpdump -i eth0 -nn -s 0 -w /tmp/ingress.pcap 'port 80 or port 443'

Analysis with wireshark will show increased TCP retransmissions and longer SYN‑ACK RTTs when the conntrack table is full.

Solution – Multi‑Layer Mitigation

1. Increase NGINX Ingress Worker Connections

Modify the ConfigMap used by the NGINX controller:

# before (default)
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  worker-connections: "1024"

# after (scaled for AI workload)
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  worker-connections: "8192"

Reload the controller (it watches the ConfigMap and updates workers automatically).

2. Tune HPA for Faster Reaction

Set --horizontal-pod-autoscaler-sync-period=10s on the controller manager (if self‑managed) or use the cloud provider’s custom metric interval.
Use behavior.scaleUp.stabilizationWindowSeconds: 0 to eliminate the default stabilization delay.
Add a custom metric based on request latency (e.g., http_request_duration_seconds) so the HPA can scale on latency directly.

Example HPA manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-deployment
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_request_duration_seconds
      target:
        type: AverageValue
        averageValue: 200ms
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Pods
        value: 4
        periodSeconds: 10

3. Expand Conntrack Capacity

On each node, adjust the kernel parameters (via a DaemonSet or node‑pool startup script):

# before
sysctl -w net.netfilter.nf_conntrack_max=65536

# after (example for high‑throughput AI inference)
sysctl -w net.netfilter.nf_conntrack_max=262144

Persist the change in /etc/sysctl.d/99-conntrack.conf:

net.netfilter.nf_conntrack_max = 262144

4. Reduce Node Provisioning Latency for GPU Nodes

Enable node auto‑upgrade with pre‑warmed GPU nodes in the cloud provider’s AI platform, or use a node pool with preemptible GPU VMs kept in Ready state. This eliminates the 30‑second “node becomes Ready” window reported in the incident “Autoscaler scale‑out lag on a managed GKE cluster”.

5. Adjust Ingress TLS Settings (if TLS termination is used)

Increase the session cache size and enable session tickets to avoid renegotiation storms:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  ssl-session-cache: "shared:SSL:10m"
  ssl-session-timeout: "1d"
  ssl-prefer-server-ciphers: "true"

Verification – Confirming the Fix

Ingress metrics. After applying the ConfigMap, query the status endpoint again:

curl -s http://:10254/metrics | grep nginx_worker_connections
nginx_worker_connections 8192

Latency baseline. Run a load test (e.g., hey -c 200 -n 5000 http://service/api/predict) and verify the 95th percentile stays < 150 ms.
HPA activity. Observe that new pods appear within 10 s of a sustained latency increase:
```
kubectl get hpa inference-service -w
```
Conntrack health. Check nf_conntrack_count remains well below the new max during peak:
```
net.netfilter.nf_conntrack_count = 12000
net.netfilter.nf_conntrack_max = 262144
```
GPU allocation logs. Ensure no “device not available” messages appear after the node pool is pre‑warmed.

Prevention – Operational Guardrails

Monitoring. Create alerts for:
- NGINX worker_connections utilization > 80 %.
- HPA scaling latency > 20 s.
- Conntrack nf_conntrack_count > 70 % of nf_conntrack_max.
Capacity planning. Periodically run a stress test that simulates peak request concurrency and validates that all three layers (Ingress, Autoscaler, CNI) stay within thresholds.
Infrastructure as Code. Store the tuned ConfigMaps, DaemonSet for sysctl, and HPA manifest in version‑controlled manifests; enforce review of changes to worker-connections and nf_conntrack_max.
Node pool management. Keep a minimum number of ready GPU nodes (e.g., 2) to absorb sudden spikes without waiting for node provisioning.
Ingress timeout tuning. Align proxy-read-timeout and proxy-send-timeout with the SLA of the inference model (e.g., 5 s) to avoid premature 504s.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does CPU/Memory look normal while latency spikes? The bottleneck is in the network path (Ingress queueing, conntrack saturation) and in the autoscaling control loop, not in pod compute resources.
Can increasing only the HPA limits solve the problem? No. If the ingress controller cannot accept new connections, additional pods will sit idle. Both layers must be sized appropriately.
What is the safe value for worker_connections? It depends on the expected concurrent connections. A rule of thumb is max_concurrent_requests * 2. For a 5 k RPS AI service, 8192–16384 is typical.
How do I know if conntrack is the culprit? When nf_conntrack_count stays near nf_conntrack_max and you see kernel messages like “nf_conntrack: table full, dropping packet”.
Is TLS termination a factor? Yes. Mis‑configured TLS can cause frequent renegotiations that amplify queueing. Ensure session caching and appropriate buffer sizes are set.