grafana

Grafana dashboard query latency spikes over 8000ms with PromQL aggregations

by
a close-up of a server room

Problem Description Grafana dashboards that visualize real‑time model inference latency and GPU utilization are experiencing intermittent rendering latency spikes that exceed 8000 ms. The symptoms observed in the UI and logs are: Panel data request timed out after 8000 ms (Grafana UI). Grafana logs contain datasource query timeout and Failed to query datasource: context deadline exceeded. Prometheus … Read more

Grafana dashboard rendering fails with high CPU and memory usage

by
a rack of servers in a server room

Problem – Grafana Dashboard Rendering Fails Under Load In a GPU‑focused LLM training cluster (64 × A100, 48 × H100 for inference) operators observed the following symptoms after adding per‑GPU DCGM metrics and reducing the dashboard refresh interval to 5s: Grafana pod repeatedly OOMKilled: panic: runtime: out of memory HTTP 504 Gateway Timeout responses from the Grafana UI. CPU … Read more

Grafana metric spikes causing LLM context window overflow

by
cable network

Problem Description An AI‑driven observability agent receives raw Prometheus query results and Loki log streams from Grafana during a blue‑green traffic shift. The ingestion pipeline builds a single prompt for an OpenAI LLM. When metric spikes and duplicate log streams are unbounded, the prompt exceeds the model’s token limit, producing errors such as: { “error”: … Read more

Grafana dashboard time-series and log data misalignment issue

by
cable network

Problem – Misaligned Time‑Series and Log Visualizations In a Docker‑compose sandbox the Grafana dashboard shows a noticeable gap between Prometheus metrics and Loki logs. The time‑series panel displays metric points at the expected timestamps, while the log panel appears shifted (commonly 5 minutes, 1 hour, or a few seconds). The visual gap makes correlation impossible and leads … Read more

Grafana tokenizer encoding errors with non-UTF-8 log data

by
img IX mining rig inside white and gray room

Problem Description – Tokenizer Encoding Errors in Grafana In a multi‑tenant machine‑learning monitoring platform hosted on AWS Managed Grafana, engineers observed intermittent failures when visualizing log and trace data sourced from Loki. The Log panel rendered a generic error message such as: panic: runtime error: invalid UTF-8 sequence in log tokenization goroutine 112 [running]: pkg/logql/lexer.go:123 … Read more

Grafana RAG metadata filter excluding relevant data sources

by
a close-up of a server room

Problem Description When developing locally with Grafana’s Retrieval Augmented Generation (RAG) component, engineers often observe that queries return incomplete or empty results. The RAG backend logs messages such as: 2024-05-12T14:23:07Z WARN rag.filter – RAG metadata filter: excluded source ‘my_test_ds’ due to missing tag ‘rag_enabled’ 2024-05-12T14:23:07Z WARN rag.filter – RAG filter applied, 0 of 5 candidate … Read more