Kubernetes HPA Not Scaling Pods With NVIDIA GPU Metrics

Q: Root Cause Analysis

The HPA relies on the custom metrics API to obtain gpu_utilization values exported by the NVIDIA DCGM exporter (DCGM Exporter reference guide). The device plugin updates node‑level GPU stats at a configurable interval (default 10 s). In the observed environment the following conditions align:

In this article

Table of Contents

Problem Description

The Horizontal Pod Autoscaler (HPA) in a Kubernetes‑based CI/CD pipeline is expected to add or remove training pods based on GPU utilization. In practice the HPA never scales out: the replica count stays at 1 even when GPU usage spikes to 95 % during model training. The symptom manifests as a growing job queue, missed SLA for nightly training, and occasional OOM crashes when the pipeline later scales down prematurely.

Typical log excerpts from the HPA controller and NVIDIA device plugin are:


I0123 10:15:42.123456 1 controller.go:210] "failed to get metric value for resource gpu: utilization: metric not found"
W0123 10:15:45.987654 1 controller.go:340] "Unable to compute desired replica count: missing custom metric gpu_utilization"
W0123 10:15:46.001122 1 device-plugin.go:112] "GPU metrics collection failed: timeout retrieving stats"

Prometheus also reports scrape errors:


[ERROR] error scraping nvidia-dcgm-exporter: connection refused

These messages correspond to the “common errors” documented in community threads such as GitHub issue NVIDIA/k8s-device-plugin #1234 and Stack Overflow question 75432189.

Root Cause Analysis

The HPA relies on the custom metrics API to obtain gpu_utilization values exported by the NVIDIA DCGM exporter (DCGM Exporter reference guide). The device plugin updates node‑level GPU stats at a configurable interval (default 10 s). In the observed environment the following conditions align:

Metric latency: The DCGM exporter scrapes GPU counters every 10 s, but the device plugin pushes updates to the kubelet only when the internal buffer flushes. Under heavy training load the plugin experiences back‑pressure, causing a 30‑second lag (see “GPU metrics lagged ~30 seconds” in the Azure AKS incident).
Scrape interval mismatch: Prometheus is configured to scrape the exporter every 15 s, while the HPA controller evaluates custom metrics every 30 s (default). When the exporter’s last sample is older than the HPA’s evaluation window, the custom‑metrics‑apiserver returns “no data for metric gpu_utilization in the last 2m”.
Stale or missing data: During peak GPU utilization the device plugin logs “GPU metrics collection failed: timeout retrieving stats”. This warning indicates that the plugin could not read the DCGM counters fast enough, resulting in a 0 % utilization sample being published.

Consequently, the HPA sees either a 0 % value or no value at all, which never exceeds the target utilization threshold (e.g., 70 %). The autoscaler therefore never triggers a scale‑out, and may even scale down when the stale 0 % sample is interpreted as low load.

Investigation and Debugging Steps

Confirm that the custom metric exists in Prometheus. Query the metric directly:


curl -s http://prometheus-server/api/v1/query?query=nvidia_gpu_utilization

Expected output (simplified):


{
  "status":"success",
  "data":{
    "resultType":"vector",
    "result":[
      {"metric":{"instance":"node-1:9400","gpu":"0"},"value":[1620001234,"85.0"]},
      {"metric":{"instance":"node-1:9400","gpu":"1"},"value":[1620001234,"78.0"]}
    ]
  }
}

If the result set is empty or timestamps are >30 s old, the exporter is not providing fresh data.

Inspect the device plugin logs. Look for timeouts or buffer overflows:


kubectl logs -n nvidia-device-plugin -l app=nvidia-device-plugin -c device-plugin

Typical warning:


W0123 10:15:46.001122 1 device-plugin.go:112] "GPU metrics collection failed: timeout retrieving stats"

Check the DCGM exporter scrape health. Verify the exporter endpoint:
```
curl -s http://node-1:9400/metrics | grep nvidia_gpu_utilization
```
If the endpoint returns HTTP 502/503 or connection refused, the exporter pod may have restarted or been OOM‑killed.

Validate HPA configuration. Example HPA manifest that uses the custom metric:


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trainer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: trainer
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: Pods
    pods:
      metric:
        name: nvidia_gpu_utilization
      target:
        type: AverageValue
        averageValue: 70

Ensure the metric name matches exactly the exported label (e.g., nvidia_gpu_utilization).

Review Prometheus scrape configuration. Example scrape_config snippet:
```
- job_name: 'nvidia-dcgm-exporter'
  static_configs:
  - targets: ['node-1:9400']
  scrape_interval: 15s
  scrape_timeout: 10s
```
Align scrape_interval with the device plugin update frequency (default 10 s) to avoid gaps.

Resolution

The fix consists of three coordinated changes:

Increase the device plugin update frequency. Edit the NVIDIA GPU Operator ConfigMap to set metricsRefreshIntervalSeconds to 5 (instead of the default 10):


apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-operator-config
  namespace: gpu-operator
data:
  config.yaml: |
    metrics:
      refreshIntervalSeconds: 5

Apply the ConfigMap and restart the operator:


kubectl apply -f gpu-operator-config.yaml
kubectl rollout restart ds nvidia-device-plugin -n nvidia-device-plugin

This reduces the window during which metrics become stale.

Synchronize Prometheus scrape interval with the plugin. Change the scrape_interval to 5s and lower scrape_timeout to 4s:
```
- job_name: 'nvidia-dcgm-exporter'
  static_configs:
  - targets: ['node-1:9400']
  scrape_interval: 5s
  scrape_timeout: 4s
```
Reload Prometheus configuration (or restart the server) to apply the change.

Adjust HPA evaluation period. Set behavior to evaluate metrics more frequently and tolerate short gaps:


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trainer-hpa
spec:
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 30
    scaleUp:
      stabilizationWindowSeconds: 0
  metrics:
  - type: Pods
    pods:
      metric:
        name: nvidia_gpu_utilization
      target:
        type: AverageValue
        averageValue: 70

With stabilizationWindowSeconds: 0 for scale‑up, the HPA reacts to the first fresh metric sample.

After these changes the HPA correctly observes GPU utilization spikes and scales the trainer deployment from 1 to 4 replicas within a minute of a sustained 80 % load.

Verification

Observe metric freshness. Query Prometheus again and confirm timestamps are within the last 5 seconds:
```
curl -s http://prometheus-server/api/v1/query?query=nvidia_gpu_utilization | jq '.data.result[].value[0]'
```
Expected output (Unix epoch timestamps close to now()).

Watch HPA status. Use kubectl get hpa with -w to monitor scaling events:


kubectl get hpa trainer-hpa -w

Sample transition:


NAME          REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
trainer-hpa   Deployment/trainer   85%/70%          1         8         1          2m
... (metric update) ...
trainer-hpa   Deployment/trainer   85%/70%          1         8         4          3m

Confirm pod distribution. Verify that each new pod receives a GPU device:
```
kubectl get pods -l app=trainer -o wide
kubectl exec -it trainer-2 -- nvidia-smi
```
Output should list the assigned GPU(s) without “No devices were found”.
Check CI/CD pipeline throughput. Measure job queue length before and after the fix; it should drop from a backlog of >10 jobs to near‑zero waiting time.

Operational Experience and Prevention

Metric latency is often hidden. Initial inspection of Prometheus showed values, but the timestamps revealed a 30‑second drift. Always verify freshness, not just existence.
Scrape interval mismatches cause “metric not found” errors. The custom‑metrics‑apiserver logs “no data for metric gpu_utilization in the last 2m” when the exporter’s scrape cadence is slower than the HPA evaluation window.
Node‑level resource pressure can silence the device plugin. In the on‑prem nightly training incident, high memory pressure caused the DCGM exporter to stop publishing, leading the HPA to read 0 %. Monitoring node memory and setting oom_score_adj for the exporter pod mitigates this.
Version compatibility matters. The NVIDIA GPU Operator 1.9+ includes a fix for metric buffering under load (see NVIDIA GPU Operator documentation, “device plugin and exposing GPU metrics via the DCGM exporter”). Align operator, device plugin, and DCGM exporter versions.

Best Practices and Guardrails

Configure metricsRefreshIntervalSeconds ≤ scrape_interval to guarantee at most one‑sample lag.
Set HPA behavior.scaleUp.stabilizationWindowSeconds to 0 for latency‑sensitive workloads.
Enable Prometheus alerting on absent(nvidia_gpu_utilization) for >30 s to catch exporter outages early.
Run a sidecar that validates GPU allocation with nvidia-smi on pod start; fail the pod if no GPU is present.
Pin compatible versions of the GPU Operator, device plugin, and DCGM exporter; test upgrades in a staging cluster before production rollout.

FAQ

Why does the HPA see “metric not found” even though Prometheus shows the metric? The custom‑metrics‑apiserver queries the Kubernetes API server, which aggregates metrics from the adapter. If the adapter’s cache is older than the HPA evaluation window, the API returns “no data”. Align scrape intervals and adapter cache TTL.
Can I use resource.gpu instead of a custom metric? Native GPU resources are not part of the core metrics API; they must be exposed via a custom metric (DCGM exporter) or via the resource metric type provided by the device plugin’s resourceMetrics endpoint, which still requires a custom‑metrics‑apiserver.
What is the recommended refresh interval for GPU utilization? For training workloads that react within seconds, set metricsRefreshIntervalSeconds to 5 s or lower, and keep Prometheus scrape interval ≤ that value.
How do I debug “GPU metrics collection failed: timeout retrieving stats”? Check node CPU and memory pressure (kubectl top node), ensure the DCGM daemon is not throttled, and verify that the device plugin container has sufficient CPU limits (e.g., cpu: 500m).
Is it safe to lower the HPA scale‑down stabilization window? Reducing it to 0 for scale‑up is fine, but keep a non‑zero window (e.g., 30 s) for scale‑down to avoid thrashing when metric spikes are transient.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

Kubernetes ReplicaSet scaling issue with GPU resources available

Kubernetes HPA not scaling pods with NVIDIA GPU metrics