HPA not scaling HAProxy pods in dev sandbox despite CPU spike

Problem

In a development sandbox (a local kind cluster) the haproxy deployment is configured with a Horizontal Pod Autoscaler (HPA) that should react to both CPU utilization and a custom request‑latency metric. During load testing of an AI model serving pipeline the following symptoms appear:

• Average request latency climbs from ~30 ms to >500 ms.
• CPU usage on each haproxy pod spikes to 85 % (target is 70 %).
• The HPA reports desiredReplicas: 1 even though the metrics exceed the thresholds.
• No new haproxy pods appear; kubectl get hpa shows a stable replica count.

Root Cause

The HPA relies on two metric sources:

1. Resource metrics – CPU usage reported by the metrics-server.
2. Custom metrics – request latency exported by a PrometheusAdapter as haproxy_request_duration_seconds.

Investigation revealed two independent failures:

Component	Issue	Why it mattered
metrics‑server	Running version 0.5.0 with --kubelet-insecure-tls flag disabled.	CPU metrics from the kubelet could not be fetched, causing the HPA to see 0 CPU usage.
Prometheus Adapter	Metric name mismatch – HPA requested haproxy_request_latency_seconds while the adapter exposed haproxy_request_duration_seconds.	The custom metric query returned an empty series, so the HPA ignored latency entirely.

Because both metric sources returned 0, the HPA calculation never crossed the targetAverageUtilization or targetValue thresholds, leaving the replica count unchanged.

Debug

Step‑by‑step diagnostics that reproduced the failure:

1. Check HPA status:

kubectl get hpa haproxy -o yaml

status:
  currentMetrics:
  - resource:
      name: cpu
      currentAverageUtilization: 0
    type: Resource
  - type: External
    external:
      metric:
        name: haproxy_request_latency_seconds
      currentValue: 0
  desiredReplicas: 1
  currentReplicas: 1

Verify CPU metrics from metrics-server:

kubectl top pod -l app=haproxy

NAME               CPU(cores)   MEMORY(bytes)
haproxy-6d5f7b9c9b-8k2zv   0m          50Mi
haproxy-6d5f7b9c9b-9l3wq   0m          48Mi

Zero CPU indicates the metrics API is not returning data.

Inspect metrics-server logs for TLS errors:

kubectl logs -n kube-system deploy/metrics-server

error: failed to get node metrics: Get "https://10.96.0.1:10250/stats/summary": dial tcp 10.96.0.1:10250: i/o timeout

Query the custom metric directly from Prometheus:

curl -s "http://prometheus:9090/api/v1/query?query=haproxy_request_latency_seconds"

{"status":"success","data":{"result":[]}}

No series returned – metric name mismatch.

Solution

Two corrective actions were required.

1. Fix metrics‑server TLS configuration

Update the metrics-server deployment to allow insecure kubelet connections (acceptable in a sandbox) or provide the proper CA bundle.

# Before
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0
        args:
        - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        # missing --kubelet-insecure-tls

# After
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0
        args:
        - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        - --kubelet-insecure-tls

Apply the manifest and restart the pod:

kubectl apply -f metrics-server.yaml
kubectl rollout restart deployment/metrics-server -n kube-system

2. Align custom metric name with Prometheus adapter

Update the HPA definition to request the correct metric name (haproxy_request_duration_seconds) and ensure the adapter mapping is present.

# Before (HPA)
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: haproxy
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: haproxy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: haproxy_request_latency_seconds
      target:
        type: Value
        value: 200ms

# After (HPA)
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: haproxy
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: haproxy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: haproxy_request_duration_seconds
      target:
        type: Value
        value: 200ms

Redeploy the HPA:

kubectl apply -f haproxy-hpa.yaml

Verify

Confirm that both metric sources now report non‑zero values and that the HPA reacts.

1. Generate load (e.g., hey or wrk) to push CPU >70 % and latency >200 ms.
2. Watch the HPA:

kubectl get hpa haproxy -w

NAME      REFERENCE            TARGETS                     MINPODS   MAXPODS   REPLICAS   AGE
haproxy   Deployment/haproxy   85%/70%, 350ms/200ms        1         5         1          2m
haproxy   Deployment/haproxy   85%/70%, 350ms/200ms        1         5         3          3m

CPU and latency metrics now appear, and the replica count scales to 3.

Validate that the new pods are serving traffic:

kubectl get pods -l app=haproxy -o wide

Optionally, query Prometheus for the latency series to ensure values are being scraped:

curl -s "http://prometheus:9090/api/v1/query?query=haproxy_request_duration_seconds"

Prevent

• Metrics‑server health checks: Deploy a PodDisruptionBudget and a ReadinessProbe that verifies /metrics endpoint returns data. Alert on metrics-server pod restarts or scrape failures.
• Custom metric contract: Keep a single source of truth (e.g., a ConfigMap) that lists metric names used by HPAs and adapters. Include a CI lint step that validates the HPA YAML against the adapter’s rules config.
• Sandbox isolation: In local clusters, explicitly enable --kubelet-insecure-tls or provide the node’s CA bundle to avoid silent metric loss.
• Observability guardrails: Add a Grafana dashboard that plots HPA desired replicas vs. underlying metrics. Set alerts when desiredReplicas stays at minReplicas while any metric exceeds its target for >5 minutes.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the HPA show currentMetrics with zero values instead of the actual CPU usage?
   Because the metrics-server could not reach the kubelet over TLS. Enabling --kubelet-insecure-tls (or providing the correct CA) restores the CPU metric stream.
2. Can I use a Prometheus query directly in the HPA without an adapter?
   No. The HPA only consumes metrics exposed via the Kubernetes custom‑metrics API. The Prometheus Adapter translates PromQL queries into that API.
3. What if the custom metric name is correct but the HPA still ignores it?
   Check the adapter’s rules ConfigMap for a matching seriesQuery. A mismatch in label selectors or metric type will cause the adapter to drop the series.
4. Is it safe to run --kubelet-insecure-tls in production?
   Generally not. In production you should mount the kubelet’s CA bundle into the metrics-server pod and configure --kubelet-certificate-authority instead.
5. How do I debug a situation where the HPA scales down unexpectedly?
   Inspect the status.currentMetrics fields, verify that both resource and external metrics are still above their targets, and ensure no conflicting HPAs or PodDisruptionBudgets are forcing a lower replica count.