HPA not scaling pods with high CPU usage in Kubernetes

Problem Description

The Horizontal Pod Autoscaler (HPA) attached to a Deployment that serves Hugging Face Transformers inference never creates additional replicas, even when the pods report CPU utilization above the configured target (e.g., 80%). The symptom manifests as:

• CPU usage in kubectl top pod consistently >90%.
• Inference latency spikes and occasional request time‑outs.
• HPA status shows desiredReplicas: 1 and currentReplicas: 1 despite high load.
• Controller logs contain messages such as:

  
  failed to get cpu utilization metric for pod "my-model-abc123": metric not available
  HPA scaling failed: observed CPU utilization (0%) is lower than target (80%) – likely due to missing resource requests
  

Root Cause Analysis

Several intertwined factors can prevent CPU‑based scaling in a Kubernetes cluster running Hugging Face models:

1. Missing or zero CPU requests – The HPA calculates utilization as currentUsage / requestedCPU. If requests.cpu is 0 or omitted, the denominator becomes zero, yielding 0 % utilization. This matches the real incident where a BERT service had requests.cpu: 0m and the HPA never triggered.
2. Metrics Server unavailability or stale data – The Metrics Server must expose per‑pod CPU metrics. Known issues (e.g., kubernetes/kubernetes#98173) show that a crashed or memory‑starved Metrics Server returns time‑outs:

   
   metrics-server: unable to fetch metrics from kubelet: Get https://10.96.0.1:10250/stats/summary: i/o timeout
   

   When metrics are missing, the HPA logs “failed to get cpu utilization metric…”.

3. RBAC restrictions on the Metrics Server – If the Metrics Server ServiceAccount lacks permission to read pods/metrics, the API returns 403 Forbidden, causing the same “metric not available” error.
4. Node‑level CPU throttling – Aggressive cpu.limit combined with cgroup quotas can cause the kernel to throttle the pod, reporting low usage to the Metrics Server while the container is actually constrained. The HPA then believes utilization is low.
5. Target utilization mismatch – Setting targetCPUUtilizationPercentage too low (e.g., 50 %) while the pod’s cpu.limit is close to the request can make the pod appear “ready” to the HPA even though it is throttled.

Investigation and Debugging

Follow these steps to isolate the failure mode.

1. Verify pod resource specifications

kubectl get deployment my-model -o yaml | grep -A5 resources

Expected snippet (problematic):

resources:
  limits:
    cpu: "2"
  requests:
    cpu: "0"

2. Check Metrics Server health

kubectl get pods -n kube-system -l k8s-app=metrics-server
kubectl logs -n kube-system deployment/metrics-server
kubectl top pod -n default

Look for errors such as:

metrics-server: unable to fetch metrics from kubelet: Get https://10.96.0.1:10250/stats/summary: i/o timeout

3. Confirm RBAC for Metrics Server

kubectl auth can-i get --raw="/apis/metrics.k8s.io/v1beta1/nodes" -n kube-system --as=system:serviceaccount:kube-system:metrics-server

The command should return yes. If no, adjust the ClusterRoleBinding.

4. Inspect HPA status and events

kubectl describe hpa my-model-hpa

Typical relevant sections:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 1; reason: cpu utilization below target

5. Capture a packet trace (optional)

kubectl exec -n kube-system $(kubectl get pod -n kube-system -l k8s-app=metrics-server -o name) -- \
  tcpdump -i any -nn port 443 and host 10.96.0.1

Verify that the Metrics Server can reach the kubelet API.

6. Review node‑level CPU throttling

kubectl top node
kubectl describe node  | grep -i thrott

If cpu.throttled is high, the node is over‑committed.

Resolution

Apply the fixes that address the identified root causes.

1. Set realistic CPU requests

For a typical BERT inference container, a request of 500m and a limit of 2 (2 CPU) is a good starting point.

Before:

resources:
  limits:
    cpu: "2"
  requests:
    cpu: "0"

After:

resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "2"

2. Ensure Metrics Server is healthy

Increase the Metrics Server memory limits and enable --kubelet-insecure-tls only if needed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.4
        args:
        - --kubelet-insecure-tls
        resources:
          limits:
            memory: "200Mi"
          requests:
            cpu: "100m"
            memory: "100Mi"

3. Fix RBAC if missing

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

4. Adjust HPA target utilization

If the pod frequently hits its CPU limit, raise the target to 70 % or use custom metrics that reflect request latency.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-model
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Validation

After applying the changes, verify that the HPA reacts as expected.

1. Generate load (e.g., hey -c 50 -z 2m http://service.my-namespace.svc.cluster.local/predict).
2. Monitor CPU usage:

   
   kubectl top pod -l app=my-model
   

3. Observe HPA scaling:

   
   kubectl get hpa my-model-hpa -w
   

   Expected output after load:

   
   NAME           REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
   my-model-hpa   Deployment/my-model  85%/70%   1         10        3          5m
   

4. Confirm that inference latency returns to baseline (< 200 ms for BERT) using your monitoring system or a simple curl loop.

Prevention and Best Practices

• Always specify non‑zero CPU requests. HPA cannot compute utilization without them.
• Deploy Metrics Server with sufficient resources. A memory limit below 100 MiB often leads to crashes under load.
• Enable proper RBAC. Verify that the Metrics Server ServiceAccount can read pods/metrics and nodes/metrics.
• Monitor Metrics Server health. Alert on metrics-server pod restarts or on kubectl top failures.
• Consider custom metrics. For transformer inference, request latency or queue length may be more indicative than raw CPU.
• Test scaling in a staging environment. Use load generators to confirm that the HPA reacts before production rollout.

FAQ

1. Why does the HPA show 0 % CPU utilization even though kubectl top pod reports 90 %?
   Because the pod’s requests.cpu is set to 0 m. Utilization is calculated as usage / request, resulting in 0 %.
2. My Metrics Server pod is running, but kubectl top pod returns “error: metrics not available”.
   Check the Metrics Server logs for kubelet connectivity errors or RBAC denials. Also verify that the APIService for v1beta1.metrics.k8s.io is Available.
3. Can I use a custom metric (e.g., request latency) instead of CPU for scaling transformer inference?
   Yes. Deploy a custom metrics adapter (e.g., Prometheus Adapter) and define an HPA that references the latency metric. This avoids CPU request mismatches.
4. After fixing CPU requests, the HPA still doesn’t scale. What else should I check?
   Ensure that the node isn’t throttling CPU (high cpu.throttled), and verify that the HPA’s maxReplicas is high enough. Also confirm that the HPA controller logs contain no errors.
5. Is there a way to see which CPU limit the pod is hitting?
   Run kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us. A non‑‑1 value indicates a limit imposed by the container runtime.

Related Topic Hub: Model Serving Troubleshooting Hub

Related Articles

• Kubernetes ReplicaSet not scaling with high CPU utilization
• Kubernetes ReplicaSet scaling issue with GPU resources available