grafana

Grafana dashboard time-series and log data misalignment issue

by
cable network

Problem – Misaligned Time‑Series and Log Visualizations In a Docker‑compose sandbox the Grafana dashboard shows a noticeable gap between Prometheus metrics and Loki logs. The time‑series panel displays metric points at the expected timestamps, while the log panel appears shifted (commonly 5 minutes, 1 hour, or a few seconds). The visual gap makes correlation impossible and leads … Read more

Grafana tokenizer encoding errors with non-UTF-8 log data

by
img IX mining rig inside white and gray room

Problem Description – Tokenizer Encoding Errors in Grafana In a multi‑tenant machine‑learning monitoring platform hosted on AWS Managed Grafana, engineers observed intermittent failures when visualizing log and trace data sourced from Loki. The Log panel rendered a generic error message such as: panic: runtime error: invalid UTF-8 sequence in log tokenization goroutine 112 [running]: pkg/logql/lexer.go:123 … Read more

Grafana RAG metadata filter excluding relevant data sources

by
a close-up of a server room

Problem Description When developing locally with Grafana’s Retrieval Augmented Generation (RAG) component, engineers often observe that queries return incomplete or empty results. The RAG backend logs messages such as: 2024-05-12T14:23:07Z WARN rag.filter – RAG metadata filter: excluded source ‘my_test_ds’ due to missing tag ‘rag_enabled’ 2024-05-12T14:23:07Z WARN rag.filter – RAG filter applied, 0 of 5 candidate … Read more