Kubernetes ReplicaSet Not Scaling With High CPU Utilization

Q: Problem – ReplicaSet for Qwen Fails to Scale Under High CPU Load

The production Qwen service runs behind a HorizontalPodAutoscaler (HPA) that watches CPU utilization and a custom latency metric. During traffic spikes the following symptoms were observed:

Q: 1. Verify HPA status and metric availability

kubectl get hpa qwen-hpa -n qwen-prod -o yaml Sample output:

Q: 2. Inspect readiness probe behavior

kubectl describe pod qwen-7f9c9d5c5-abcde -n qwen-prod Relevant event snippet:

Q: 3. Review PodDisruptionBudget

kubectl get pdb qwen-pdb -n qwen-prod -o yaml Output:

Q: 2. Correct HPA CPU target and ensure it respects 100 % ceiling

Set targetCPUUtilizationPercentage to a realistic value (e.g., 70) and remove the invalid >100 setting.

Q: 3. Reduce metric latency lag Increase scrape_interval for the Qwen pod metrics in prometheus.yml from 30s to 10s. Configure the Prometheus Adapter to cache for 30s instead of 60s. # prometheus.yml excerpt scrape_configs: - job_name: 'qwen' scrape_interval: 10s static_configs: - targets: ['qwen-service.qwen-prod.svc.cluster.local:8080'] 4. Relax PodDisruptionBudget to allow scaling while pods restart

apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: qwen-pdb spec: maxUnavailable: 1 # permits one pod to be unavailable during scaling/restarts selector: matchLabels: app: qwen 5. Apply changes atomically kubectl apply -f rs-qwen.yaml kubectl apply -f hpa-qwen.yaml kubectl apply -f pdb-qwen.yaml Verification – Confirm Scaling Works as Expected Generate load with hey or wrk to push CPU > 80 %: hey -c 200 -n 100000 http://qwen-prod.example.com/inference Watch HPA calculations: kubectl get hpa qwen-hpa -w -n qwen-prod Expected output after a few seconds:

In this article

Table of Contents

Problem – ReplicaSet for Qwen Fails to Scale Under High CPU Load

The production Qwen service runs behind a HorizontalPodAutoscaler (HPA) that watches CPU utilization and a custom latency metric. During traffic spikes the following symptoms were observed:

CPU usage on existing pods consistently exceeds 80%.
HPA reports a desired replica count (e.g., 8) but the ReplicaSet reports only 3 ready pods.
Client requests receive 502 Bad Gateway errors.

Pod events contain messages such as:


Readiness probe failed: connection refused
HorizontalPodAutoscaler sync failed: unable to compute desired replica count
failed to get cpu utilization: metric not available

Metrics server logs show a 90‑second lag for the custom latency metric.

Root Cause Analysis

Multiple intertwined misconfigurations prevent the HPA from materializing the desired replica count:

Inconsistent readiness probes – The Qwen container image ships with a HTTP readiness probe that checks /healthz. Under load the model loading thread blocks the HTTP server, causing intermittent probe failures. According to the Kubernetes ReplicaSet and Pod readiness/liveness probes documentation, a pod that never becomes Ready is not counted toward the ReplicaSet’s availableReplicas, so the HPA sees no capacity increase.
Mis‑configured targetCPUUtilizationPercentage – The HPA was set to 150% (see the Stack Overflow question 76184231). Since the HPA caps the target at 100 %, the controller treats the current 80 % usage as within limits and does not trigger scaling.
Stale custom metric data – The Prometheus Adapter for the latency_seconds metric lagged by ~90 seconds during spikes (see the incident “Custom latency metric server lagged by 90 seconds”). The HPA therefore bases its decision on outdated CPU values, delaying scaling.
PodDisruptionBudget (PDB) deadlock – A PDB with maxUnavailable: 0 prevented the creation of new pods while older pods were being restarted for config changes. This matches the “ReplicaSet stuck at 0 pods after HPA trigger” community case.

Investigation and Debugging Steps

The following systematic approach reproduced the failure and isolated each factor.

1. Verify HPA status and metric availability


kubectl get hpa qwen-hpa -n qwen-prod -o yaml

Sample output:


apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: qwen-rs
  minReplicas: 3
  maxReplicas: 15
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 150   # <-- mis‑configured
  - type: Pods
    pods:
      metric:
        name: latency_seconds
      target:
        type: AverageValue
        averageValue: 200ms

Check metric server health:


kubectl top pods -l app=qwen -n qwen-prod
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/qwen-prod/pods/*/latency_seconds"

Typical error:


custom.metrics.k8s.io metric latency_seconds not found

2. Inspect readiness probe behavior


kubectl describe pod qwen-7f9c9d5c5-abcde -n qwen-prod

Relevant event snippet:


Events:
  Type    Reason             Age   From               Message
  ----    ------             ----  ----               -------
  Normal  Pulling            2m    kubelet, node-1    Pulling image "registry.cn-hangzhou.aliyuncs.com/qwen/qwen:latest"
  Normal  Pulled             2m    kubelet, node-1    Successfully pulled image
  Normal  Created            2m    kubelet, node-1    Created container qwen
  Normal  Started            2m    kubelet, node-1    Started container qwen
  Warning Readiness probe failed  30s (x5 over 2m)  kubelet, node-1  Readiness probe failed: connection refused

3. Review PodDisruptionBudget


kubectl get pdb qwen-pdb -n qwen-prod -o yaml

Output:


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: qwen-pdb
spec:
  maxUnavailable: 0   # prevents any additional pod from being created while others are terminating
  selector:
    matchLabels:
      app: qwen

4. Correlate metric latency with scaling decision

Capture a short‑term time series from Prometheus:


promtool query range 'rate(container_cpu_user_seconds_total{pod=~"qwen-.*"}[1m])' \
  --start=$(date -d '5 minutes ago' +%s) --end=$(date +%s) --step=15s

Observe a ~90 s gap between load spike and metric availability.

Resolution – Align Probes, Autoscaling Policy, Metrics, and PDB

1. Fix readiness probe to tolerate brief model load delays

Update the Deployment (or directly the ReplicaSet) to use an initialDelaySeconds and a higher failureThreshold:


apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: qwen-rs
spec:
  template:
    spec:
      containers:
      - name: qwen
        image: registry.cn-hangzhou.aliyuncs.com/qwen/qwen:latest
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 6   # allow up to 60 s of failure before marking Unready

2. Correct HPA CPU target and ensure it respects 100 % ceiling

Set targetCPUUtilizationPercentage to a realistic value (e.g., 70) and remove the invalid >100 setting.


apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen-hpa
spec:
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70   # <-- corrected

3. Reduce metric latency lag

Increase scrape_interval for the Qwen pod metrics in prometheus.yml from 30s to 10s.
Configure the Prometheus Adapter to cache for 30s instead of 60s.


# prometheus.yml excerpt
scrape_configs:
  - job_name: 'qwen'
    scrape_interval: 10s
    static_configs:
      - targets: ['qwen-service.qwen-prod.svc.cluster.local:8080']

4. Relax PodDisruptionBudget to allow scaling while pods restart


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: qwen-pdb
spec:
  maxUnavailable: 1   # permits one pod to be unavailable during scaling/restarts
  selector:
    matchLabels:
      app: qwen

5. Apply changes atomically


kubectl apply -f rs-qwen.yaml
kubectl apply -f hpa-qwen.yaml
kubectl apply -f pdb-qwen.yaml

Verification – Confirm Scaling Works as Expected

Generate load with hey or wrk to push CPU > 80 %:


hey -c 200 -n 100000 http://qwen-prod.example.com/inference

Watch HPA calculations:


kubectl get hpa qwen-hpa -w -n qwen-prod

Expected output after a few seconds:


NAME       REFERENCE               TARGET   CURRENT   MIN   MAX   RECOMMENDATION   AGE
qwen-hpa   ReplicaSet/qwen-rs      70%      85%       3     15    9                1m

Verify that new pods become Ready:


kubectl get pods -l app=qwen -n qwen-prod -w

All new pods should show READY 1/1 within the initialDelaySeconds + failureThreshold * periodSeconds window.

Check request success rate:


kubectl logs -l app=qwen -n qwen-prod --tail=100 | grep "HTTP 200"

Confirm metrics are up‑to‑date:


kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/qwen-prod/pods/*/latency_seconds"

Prevention – Operational Guardrails and Best Practices

Probe robustness: Use generous initialDelaySeconds and failureThreshold for workloads with heavy model initialization.
CPU target sanity check: Keep targetCPUUtilizationPercentage ≤ 100 % and align with the container’s resource requests/limits as recommended in the Qwen Model Deployment Guide.
Metric freshness: Set scrape_interval ≤ 15 s for latency‑critical services; monitor metrics-server and prometheus-adapter health.
PDB design: Allow at least one pod to be unavailable (maxUnavailable: 1) unless strict SLA mandates otherwise.
Observability: Create alerts for:
- HPA desired replica count ≠ ready replica count for >30 s.
- Readiness probe failures exceeding threshold.
- Stale custom metric errors (“metric not available”).

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the HPA ignore high CPU usage?
Because the HPA’s targetCPUUtilizationPercentage was set to 150 %, which exceeds the 100 % ceiling, causing the controller to treat the current usage as within bounds.
Can a failing readiness probe prevent scaling?
Yes. Pods that never become Ready are not counted as available, so the ReplicaSet’s availableReplicas stays low and the HPA sees no capacity increase, as documented in the Kubernetes readiness/liveness probe reference.
How do metric server delays affect scaling?
If the custom latency metric lags (e.g., 90 s), the HPA bases its decision on stale data, delaying scaling until after the load subsides. Reducing the scrape interval and adapter cache time mitigates this.
What role does a PodDisruptionBudget play in scaling deadlocks?
A PDB with maxUnavailable: 0 blocks the creation of new pods while older ones are terminating, creating a deadlock where the HPA cannot increase replica count.
Should I use both CPU and latency metrics?
Combining them is valid, but ensure both metric sources are reliable and have comparable update frequencies; otherwise one metric may dominate the scaling decision and hide issues in the other.