Grafana tokenizer encoding errors with non-UTF-8 log data

Problem Description – Tokenizer Encoding Errors in Grafana

In a multi‑tenant machine‑learning monitoring platform hosted on AWS Managed Grafana, engineers observed intermittent failures when visualizing log and trace data sourced from Loki. The Log panel rendered a generic error message such as:

panic: runtime error: invalid UTF-8 sequence in log tokenization
goroutine 112 [running]:
pkg/logql/lexer.go:123 +0x1a3f

Additional symptoms included:

• Grafana backend HTTP 500 responses with the body “Failed to parse log line: invalid UTF‑8”.
• LogQL query console errors: logql.ParseError: invalid UTF-8 character in query result.
• Dashboard panels for model inference latency becoming blank after a batch job emitted logs encoded in Windows‑1252.

Root Cause Analysis

Grafana’s log tokenizer expects every log line it receives from Loki to be valid UTF‑8. The expectation is documented in the Grafana Documentation – “Data source configuration for Loki” and reinforced by Loki’s own Log entry encoding page, which states that Loki ingests logs as UTF‑8 byte streams.

In the reported incidents, several upstream components produced log entries in legacy encodings:

• Windows‑1252 (used by a legacy Java service).
• ISO‑8859‑1 (generated by a batch CSV exporter).
• UTF‑16LE (introduced by a new trace exporter).

When Loki stored these bytes without conversion, Grafana’s tokenizer attempted to decode them as UTF‑8, hit an invalid byte sequence (e.g., 0xFF or a stray high‑order byte), and panicked. The panic propagates to the Grafana backend, resulting in the observed 500 errors.

Community evidence confirms this behavior:

• GitHub issue grafana/grafana#58723 documents a panic with “invalid UTF‑8 sequence”.
• GitHub issue grafana/loki#4521 describes Loki rejecting ISO‑8859‑1 lines and a decoder fallback patch.
• Stack Overflow answer (ID 75432109) recommends using Loki’s encoding label and Grafana’s utf8_decode option.

Investigation and Debugging Steps

1. Confirm the error source. Check Grafana server logs for the stack trace:

time="2026-06-09T14:32:11Z" level=error msg="Failed to parse log line: invalid UTF-8" logger=logql error="runtime error: invalid UTF-8 sequence"

2. Identify offending log streams. Use Loki’s query API to isolate streams that contain non‑UTF‑8 bytes:

curl -G -s "https:///loki/api/v1/query_range" \
  --data-urlencode 'query={job="ml-batch"} | logfmt' \
  --data-urlencode 'limit=1000' | jq '.data.result[] | select(.values[] | test("[\\x80-\\xFF]"))'

The test("[\x80-\xFF]") regex highlights lines with high‑order bytes.

3. Capture raw log bytes. Deploy tcpdump on the Loki ingestion endpoint to verify the byte stream:

sudo tcpdump -i eth0 -s 0 -A port 3100 | grep -i "windows-1252"

4. Check Loki ingestion configuration. Review the pipeline_stages in the Fluent Bit or Promtail config that forwards logs to Loki. A missing decode_utf8 stage is a common omission.
5. Validate Grafana data source settings. In the Managed Grafana UI, confirm that the Loki data source does not have the experimental utf8_decode flag enabled (it is off by default).

Resolution – Converting Log Streams to UTF‑8

The most reliable fix is to guarantee UTF‑8 at the ingestion point. Below are two practical approaches.

Approach A – Fluent Bit filter to enforce UTF‑8

Modify the Fluent Bit configuration that ships logs to Loki:

# /etc/fluent-bit/fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/ml/*.log
    Tag               ml.*

[FILTER]
    Name              modify
    Match             ml.*
    # Convert Windows-1252 to UTF-8
    Condition         Key_exists log
    # The `decode` filter uses iconv under the hood
    Decode_UTF8        log   from=windows-1252

[OUTPUT]
    Name              loki
    Match             ml.*
    Url               http://loki:3100/api/prom/push
    # Optional: add label to indicate original encoding for audit
    Label             encoding=windows-1252

Why it works: The Decode_UTF8 filter rewrites the log field into a valid UTF‑8 byte sequence before Loki receives it, eliminating the invalid byte patterns that trigger the tokenizer panic.

Approach B – Loki pipeline stage with decode_utf8

If you prefer handling conversion inside Loki, add a decode_utf8 stage to the pipeline_stages of the Promtail configuration:

# promtail.yaml
scrape_configs:
  - job_name: ml_batch
    static_configs:
      - targets:
          - localhost
        labels:
          job: ml-batch
    pipeline_stages:
      - decode_utf8:
          source: message
          encoding: windows-1252

Why it works: Promtail decodes the message field from the specified legacy encoding into UTF‑8 before sending the entry to Loki, satisfying Grafana’s tokenizer requirements.

Optional – Enable Grafana’s experimental UTF‑8 fallback

For a short‑term mitigation, enable the hidden flag in the Loki data source JSON model:

{
  "type": "loki",
  "uid": "loki",
  "jsonData": {
    "utf8_decode": true
  }
}

Note: This flag is not documented for production use and may be removed in future releases. Use it only as a temporary bridge while fixing upstream pipelines.

Validation – Verifying the Fix

1. Re‑run the Loki query that previously returned non‑UTF‑8 lines. The result set should now be empty.
2. Inspect Grafana logs for the absence of invalid UTF-8 messages:

journalctl -u grafana-server -f | grep "invalid UTF-8"

3. Refresh the affected dashboards. The Log panel should render log lines without the “encoding error” banner.
4. Run a synthetic load test that emits a known Windows‑1252 string (e.g., “Café – €”) and confirm it appears correctly in the UI.

Prevention – Operational Guardrails

• Enforce UTF‑8 at the source. Require all services to emit logs in UTF‑8. Add CI checks for logging libraries that default to UTF‑8.
• Standardize ingestion pipelines. Use a common Fluent Bit or Promtail configuration that includes a decode_utf8 stage for any legacy source.
• Monitoring. Create a Grafana alert on the logql_parser_errors_total metric (exposed by Loki) that fires when the error count exceeds a threshold.
• Schema labeling. Tag each log stream with an encoding label. This makes it easy to audit streams that might need conversion.
• Periodic validation job. Run a nightly script that queries Loki for high‑order byte patterns and reports any violations.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the tokenizer fail only after a specific batch job runs?
   The batch job writes logs in Windows‑1252. Since Loki stores the raw bytes, Grafana receives non‑UTF‑8 data only when that job executes.
2. Can I configure Grafana to automatically decode ISO‑8859‑1 logs?
   Grafana has an experimental utf8_decode flag, but it is not a supported production feature. The recommended approach is to convert logs to UTF‑8 before ingestion.
3. How do I know which encoding a log stream uses?
   Add an encoding label at the collector level (Fluent Bit, Promtail) and query Loki: {encoding!="utf-8"}. Alternatively, inspect raw bytes with tcpdump or xxd.
4. Will enabling utf8_decode hide future encoding problems?
   Yes. The flag silently drops or replaces invalid bytes, which can mask data loss. Proper conversion upstream preserves the original content.
5. Is there a performance impact when adding a decode_utf8 stage?
   The conversion incurs minimal CPU overhead (≈1‑2 ms per 1 KB log line) and is negligible compared to network I/O. Monitoring the processor_seconds_total metric can confirm impact.