Grafana dashboard rendering fails with high CPU and memory usage

Problem – Grafana Dashboard Rendering Fails Under Load

In a GPU‑focused LLM training cluster (64 × A100, 48 × H100 for inference) operators observed the following symptoms after adding per‑GPU DCGM metrics and reducing the dashboard refresh interval to 5s:

• Grafana pod repeatedly OOMKilled:
  

  panic: runtime: out of memory

• HTTP 504 Gateway Timeout responses from the Grafana UI.
• CPU usage on the Grafana node spiking to >90 % during peak monitoring periods.
• Prometheus logs showing query timeouts:
  

  datasource.prometheus: query timeout (30s)

• Grafana logs throttling concurrent queries:
  

  Too many concurrent queries (limit: 20)

The failure manifests as blank panels, missing graphs, and occasional loss of the entire dashboard view, severely impacting observability of GPU utilization and memory pressure.

Root Cause Analysis

The core issue is a combination of unbounded Prometheus query complexity and excessive dashboard refresh rates that exhaust Grafana’s CPU and memory resources.

1. High‑cardinality time series: Each GPU node exports dozens of DCGM metrics (e.g., dcgm_gpu_memory_used_bytes, dcgm_gpu_sm_utilization) with a gpu_uuid label. With 64 nodes, the cardinality exceeds 200 k series per node, as described in the Prometheus best‑practice guide.
2. Unrestricted PromQL expressions: Panels used raw rate() over node_cpu_seconds_total[5m] and sum by (instance) across the entire cluster without label filters. This forces Prometheus to scan the full metric set on each refresh.
3. Refresh interval too low: Grafana’s default per‑panel refresh inherits the dashboard setting. A 5‑second interval caused dashboard refresh to fire dozens of concurrent queries, exceeding the datasource max_concurrent_requests (default 20) and triggering the “Too many concurrent queries” error.
4. Insufficient query timeout configuration: Prometheus’ query_timeout default (2 min) is longer than Grafana’s datasource timeout (30 s). Long‑running scans therefore hit Grafana’s timeout, producing 504 errors.
5. Resource limits on the Grafana pod: The pod was allocated 2 CPU and 2 GiB RAM. The combination of heavy query load and rapid refresh caused the Go runtime to allocate memory faster than the limit, resulting in OOM kills.

Investigation and Debugging

Step‑by‑step diagnostics that reproduced the issue:

1. Inspect Grafana pod logs for OOM and throttling messages:

kubectl logs -n monitoring grafana-7c9d5f9b9f-9kz2l
...
panic: runtime: out of memory
...
Too many concurrent queries (limit: 20)

2. Check Prometheus query duration via the /api/v1/query_range endpoint:

curl -G 'http://prometheus.monitoring.svc:9090/api/v1/query' \
  --data-urlencode 'query=sum(rate(node_cpu_seconds_total[5m]))' \
  --data-urlencode 'time=1650000000'

Response contained "duration": 42.7 seconds, exceeding Grafana’s 30 s timeout.

3. Capture a packet trace to verify the volume of HTTP requests from Grafana to Prometheus during a refresh burst:

tcpdump -i any port 9090 -w /tmp/grafana-prometheus.pcap &
sleep 30
kill %1

The pcap showed ~250 GET requests per second during a 5 s refresh window.

4. Examine Prometheus series count to confirm high cardinality:

curl http://prometheus.monitoring.svc:9090/api/v1/label/__name__/values | \
  jq '.data | length'
# Returns 342 metric names

curl http://prometheus.monitoring.svc:9090/api/v1/series?match[]=dcgm_gpu_memory_used_bytes | \
  jq '.data | length'
# Returns 128,960 series (64 nodes × 2 GPUs × 1000 labels)

5. Validate Grafana datasource settings:

kubectl -n monitoring exec -it grafana-7c9d5f9b9f-9kz2l -- \
  cat /etc/grafana/provisioning/datasources/prometheus.yaml

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus.monitoring.svc:9090
    jsonData:
      httpMethod: GET
      queryTimeout: 30s
      maxConcurrentRequests: 20

Resolution – Reducing Load and Optimizing Queries

The fix consists of three orthogonal actions: throttling refresh rates, tightening PromQL, and scaling Grafana resources.

1. Adjust Dashboard Refresh Settings

Increase the global refresh interval to a sane default (30 s) and enforce a per‑panel minimum of 15 s.

# Before (dashboard JSON snippet)
"refresh": "5s",
"panels": [
  {
    "type": "graph",
    "refresh": "5s",
    ...
  }
]

# After
"refresh": "30s",
"panels": [
  {
    "type": "graph",
    "refresh": "15s",
    ...
  }
]

2. Apply PromQL Filters and Recording Rules

Limit the query scope to GPU nodes only and pre‑aggregate heavy calculations.

# Original panel query
sum(rate(node_cpu_seconds_total[5m]))

# Revised query with label filter
sum(rate(node_cpu_seconds_total{instance=~"gpu-node-.*"}[5m]))

Introduce recording rules to cache the result:

# prometheus.yml snippet
rule_files:
  - "recording_rules.yml"

# recording_rules.yml
groups:
  - name: gpu_node_cpu
    interval: 1m
    rules:
      - record: job:gpu_node_cpu:rate5m
        expr: sum(rate(node_cpu_seconds_total{instance=~"gpu-node-.*"}[5m])) by (instance)

Dashboard panels can now query the pre‑computed series:

job:gpu_node_cpu:rate5m

3. Increase Grafana Pod Resources and Concurrency Limits

Update the Helm values (or Kubernetes manifest) to allocate more CPU/RAM and raise the datasource concurrency ceiling.

# Before
resources:
  limits:
    cpu: "2"
    memory: "2Gi"
datasources:
  prometheus:
    jsonData:
      maxConcurrentRequests: 20

# After
resources:
  limits:
    cpu: "4"
    memory: "4Gi"
datasources:
  prometheus:
    jsonData:
      maxConcurrentRequests: 50
      queryTimeout: 60s

4. Enable Prometheus Query Concurrency Controls

Set --query.max-concurrency=30 in the Prometheus startup flags to prevent a single client from overwhelming the engine.

# prometheus deployment args
- --query.max-concurrency=30
- --query.timeout=60s

Verification – Confirming the Fix

1. Grafana health endpoint should return 200 OK with low CPU:

curl -s http://grafana.monitoring.svc:3000/api/health | jq .
{
  "commit": "8a9f5c3",
  "database": "ok",
  "version": "9.5.2",
  "status": "OK"
}

Grafana pod metrics (via kubectl top pod) now show CPU 0.5 cores and Memory 800Mi under load.

2. No OOM events in pod lifecycle:

kubectl get pod grafana-7c9d5f9b9f-9kz2l -n monitoring -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Output: Completed (no OOMKilled)

3. Prometheus query latency reduced below Grafana timeout:

curl -G 'http://prometheus.monitoring.svc:9090/api/v1/query' \
  --data-urlencode 'query=job:gpu_node_cpu:rate5m' \
  --data-urlencode 'time=1650000000' | jq '.data.result[0].value'
# Returns ["1650000000","0.45"]

4. Dashboard panels render within 2 s (measured via browser dev tools).

Prevention – Operational Guardrails

• Enforce minimum refresh intervals via Grafana provisioning: set min_refresh_interval = "15s" in grafana.ini.
• Limit high‑cardinality metrics ingestion by applying relabeling rules in Prometheus to drop rarely used labels.
• Use recording rules for any aggregate over the entire GPU fleet. This caps query execution time regardless of UI refresh rate.
• Monitor Grafana resource usage with alerts on CPU > 80 % or memory > 75 % of request.
• Set datasource concurrency limits lower than Prometheus --query.max-concurrency to provide headroom.
• Automated testing of dashboard load using grafana-toolkit or k6 scripts that simulate concurrent users.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the dashboard work fine with a 30 s refresh but fails at 5 s?
   Because each refresh spawns a new set of Prometheus queries. At 5 s the concurrency limit (default 20) is exceeded, causing Grafana to throttle and Prometheus to queue, leading to timeouts and OOM.
2. Can increasing Prometheus query_timeout alone solve the issue?
   Only partially. Longer timeouts let queries run longer, but they still consume CPU and memory. The root problem is query volume; reducing refresh rate and optimizing queries is required.
3. What is the recommended way to handle per‑GPU metrics without blowing up cardinality?
   Use instance=~"gpu-node-.*" filters to limit scope, drop unnecessary labels via relabel_configs, and create recording rules for aggregated views (e.g., total memory usage per node).
4. How do I know if Grafana’s max_concurrent_requests is too low?
   Grafana logs will emit “Too many concurrent queries (limit: X)”. Raise the value in the datasource JSON and ensure Prometheus --query.max-concurrency is higher.
5. Is it safe to run Grafana with cpu=4 and memory=8Gi in production?
   Yes, provided you monitor actual usage. Over‑provisioning prevents OOM but incurs cost; the goal is to size based on observed peak load after applying the above optimizations.