Pink claude logo on a golden background with clear elements

HPA not scaling Claude inference pods despite CPU threshold in production

by

Problem

The Horizontal Pod Autoscaler (HPA) for the Claude inference service never scales out, even though the cpu utilization metric consistently exceeds the configured target (80%). Pods remain at the minimum replica count (2), CPU usage stays above 90%, and request latency grows from ~150 ms to >2 s during traffic spikes.

Root Cause

Investigation revealed three intertwined issues:

  1. Metrics Server version mismatch: The cluster ran metrics-server v0.5.0, which does not expose resource.cpu for pods using the runtimeClassName: "anthropic" runtime. Consequently, the HPA received stale or zero values for those pods.
  2. Incorrect selector in the HPA: The HPA referenced the label app=claude-inference, but the Deployment used app.kubernetes.io/name=claude. The selector never matched any pods, so the HPA operated on an empty pod set.
  3. Missing metrics.k8s.io API aggregation: The cluster’s API aggregation layer was not configured to allow the HPA controller to read custom metrics from the custom-metrics-apiserver, which is required for the containerResource metric used by the HPA.

Because of these misconfigurations, the HPA controller continuously reported currentReplicas=2 and desiredReplicas=2, ignoring the actual CPU load.

Debug

Step‑by‑step diagnostics performed in the production cluster:

1. Verify HPA status

kubectl get hpa claude-inference-hpa -n inference -o yaml

Sample output:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: claude-inference-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: claude-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
status:
  currentReplicas: 2
  desiredReplicas: 2
  currentMetrics:
  - type: Resource
    resource:
      name: cpu
      currentAverageUtilization: 0
      currentAverageValue: "0m"

The currentAverageUtilization: 0 indicates the metric source is not providing data.

2. Check Metrics Server health

kubectl get pods -n kube-system -l k8s-app=metrics-server -o wide
NAME                                 READY   STATUS    RESTARTS   AGE
metrics-server-6c9d8b9d7c-5x9zv       1/1     Running   0          12d
kubectl logs deployment/metrics-server -n kube-system | grep -i error

No errors, but the server logs a warning about unsupported runtime classes:

W0123 10:15:23.123456    12345 metrics_server.go:215] Pod runtimeClassName "anthropic" is not supported for resource metrics

3. Confirm selector mismatch

kubectl get deployment claude-inference -n inference -o jsonpath='{.spec.selector.matchLabels}'
{ "app.kubernetes.io/name": "claude" }
kubectl get hpa claude-inference-hpa -n inference -o jsonpath='{.spec.scaleTargetRef.name}'
claude-inference

The HPA selector defaults to the Deployment’s selector; however, the explicit selector field in the HPA YAML (if present) was set to app=claude-inference, which does not match any pods.

4. Verify custom‑metrics API registration

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "resources": []
}

The empty list confirms the custom‑metrics API is not registered.

Solution

Apply three corrective actions:

1. Upgrade and configure Metrics Server

Upgrade to metrics-server v0.6.3, which supports runtimeClassName filtering, and add the --kubelet-insecure-tls flag if needed for the node configuration.

# Before (metrics-server deployment snippet)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server:v0.5.0
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

# After (metrics-server deployment snippet)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server:v0.6.3
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-insecure-tls
        - --runtime-config=metrics.k8s.io/v1beta1=true

After applying the new manifest, the server begins exposing CPU metrics for pods with the anthropic runtime class.

2. Align HPA selector with Deployment labels

Remove the explicit mismatched selector from the HPA and let it inherit the Deployment’s selector, or update it to match the Deployment’s labels.

# Before (HPA snippet with wrong selector)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: claude-inference-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: claude-inference
  selector:
    matchLabels:
      app: claude-inference   # <-- incorrect
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

# After (HPA without explicit selector)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: claude-inference-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: claude-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

3. Enable custom‑metrics API aggregation (if using custom metrics)

Deploy the custom-metrics-apiserver and register it via an APIService object.

# custom-metrics-apiserver deployment (simplified)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-metrics-apiserver
  namespace: custom-metrics
spec:
  selector:
    matchLabels:
      app: custom-metrics-apiserver
  template:
    metadata:
      labels:
        app: custom-metrics-apiserver
    spec:
      containers:
      - name: apiserver
        image: k8s.gcr.io/custom-metrics-apiserver:v0.4.0
        args:
        - --secure-port=443
        - --cert-dir=/tmp

# APIService registration
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

After these components are running, the HPA can retrieve both standard and custom metrics.

Verify

Confirm that the HPA now reacts to CPU pressure:

# Generate load (example using a simple HTTP load generator)
kubectl run load-generator --image=busybox --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://claude-inference.inference.svc.cluster.local/v1/predict; done"

Watch the HPA status:

kubectl get hpa claude-inference-hpa -n inference -w

Expected output after load spikes:

NAME                     REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
claude-inference-hpa    Deployment/claude-inference   85%/80%   2         10        2          5m
... (after ~30s) ...
claude-inference-hpa    Deployment/claude-inference   85%/80%   2         10        5          6m

Additional validation steps:

  • Check pod CPU usage with kubectl top pods -n inference -l app.kubernetes.io/name=claude – values should drop below the target after scaling.
  • Inspect request latency via the existing Prometheus query histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="claude-inference"}[1m])) by (le)) – latency should return to <150 ms.
  • Confirm no HPA events indicate metric collection failures: kubectl describe hpa claude-inference-hpa -n inference.

Prevent

Implement guardrails to avoid recurrence:

  • Automated HPA health check: Add a PrometheusRule that fires if kube_hpa_status_current_replicas{hpa="claude-inference-hpa"} == kube_hpa_status_desired_replicas{hpa="claude-inference-hpa"} and kube_hpa_status_current_replicas stays constant for >5 min while node_cpu_utilisation exceeds 80%.
  • Metrics Server version pinning: Enforce metrics-server version ≥ 0.6.0 via GitOps and CI validation.
  • Label consistency lint: Use a Kustomize or OPA policy that validates HPA selectors match the target Deployment’s selector labels.
  • RuntimeClass awareness: Document that any new RuntimeClass must be added to the metrics server’s --runtime-config whitelist.
  • Periodic HPA sanity test: Run a nightly job that spikes CPU on a canary deployment and asserts desiredReplicas > currentReplicas within a configurable window.

FAQ

  1. Why does the HPA show currentAverageUtilization: 0 even though pods are busy?
    Because the metrics server cannot expose CPU metrics for pods that use an unsupported runtimeClassName. Upgrading the server and enabling the runtime class resolves the issue.
  2. Can I use custom metrics instead of CPU for Claude inference scaling?
    Yes. Deploy a custom-metrics-apiserver that emits inference‑specific metrics (e.g., requests_per_second) and reference them in the HPA spec. Ensure the APIService is registered and the HPA controller has RBAC to read them.
  3. Is it safe to remove the explicit selector from the HPA?
    Removing the selector makes the HPA inherit the Deployment’s selector, which is the recommended pattern. It eliminates mismatches caused by label drift.
  4. What if I need to scale based on GPU utilization?
    GPU metrics are not provided by the default metrics server. Install a GPU exporter (e.g., NVIDIA DCGM Exporter) that exposes custom.metrics.k8s.io and reference resource: nvidia.com/gpu in the HPA.
  5. How do I know the HPA controller itself is healthy?
    Check the controller manager logs: kubectl logs -n kube-system -l component=controller-manager -c kube-controller-manager. Look for messages like “Successfully synced HPA” and ensure the pod is not restarting.

Related Topic Hub: LLM Systems Troubleshooting Hub

Related Articles