Prometheus Tokenizer Encoding Error With Non-standard Characters

Q: Debugging Steps 1. Identify offending metric lines

# View recent scrape logs (journalctl example) journalctl -u prometheus -f | grep "error parsing metric" Typical output:

Q: 2. Pull raw metric exposition from the exporter

curl -s http://experiment-exporter:9100/metrics | grep -A2 "experiment" Example raw output (malformed):

Q: 3. Verify byte encoding

curl -s http://experiment-exporter:9100/metrics | hexdump -C | head Hexdump snippet showing a non‑UTF‑8 byte (0xA9 in Latin‑1) for the © symbol:

Q: Option 2 – Use relabel_configs to strip characters during scrape

Update prometheus.yml to apply a regex replacement on the offending label.

Q: Option 3 – Upgrade exporter to emit UTF‑8 explicitly

Ensure the HTTP response header includes Content-Type: text/plain; charset=utf-8 and that all strings are encoded with UTF‑8.

Problem: Prometheus tokenizer encoding error with non‑standard characters

During the rollout of an A/B testing experiment, the Prometheus server began reporting scrape failures. The logs contained messages such as:

error parsing metric: invalid UTF-8 string in label value "variant=🧪"
error parsing metric: invalid UTF-8 string in label value "experiment=beta©"
failed to ingest metric: label value contains non-ASCII characters
scrape failed: unexpected character '\x00' in metric line

These errors caused the exporter responsible for emitting experiment metrics to be dropped from the target list, resulting in missing data on the experiment dashboards.

Root Cause Analysis

Allowed character set

According to the Prometheus documentation – “Metric and label naming”, label values must be valid UTF‑UTF‑8 strings without control characters. The text exposition format further mandates that each metric line be UTF‑8 encoded (Text format spec).

Violation introduced by the experiment pipeline

The A/B testing framework concatenates user‑supplied experiment identifiers directly into a label value (e.g., experiment=<experiment_name>).
One new experiment used the © symbol (U+00A9) and another used an emoji (U+1F9EA). These characters are valid Unicode code points, but the exporter emitted them as raw bytes in the system’s default encoding (often Latin‑1 or Windows‑1252) instead of UTF‑8.
In a separate incident, a newline character (\n) was introduced into a label value by a malformed JSON log entry, creating a control character that the tokenizer rejects.

These observations match the community reports:

GitHub issue #8425 – “Invalid UTF‑8 label value causes scrape failure”.
GitHub issue #1023 – “Metric label contains malformed UTF‑8, scraper rejects metric”.
Real‑world incident at Shopify where a label containing the “®” symbol stopped metric ingestion.

Why the tokenizer rejects the line

The Prometheus parser reads each metric line byte‑by‑byte. When it encounters a byte sequence that does not form a valid UTF‑8 code point, it aborts parsing and emits a invalid UTF-8 string in label value error. Control characters (e.g., newline, NUL) are also disallowed because they break the line‑oriented exposition format.

Debugging Steps

1. Identify offending metric lines

# View recent scrape logs (journalctl example)
journalctl -u prometheus -f | grep "error parsing metric"

Typical output:

2023-11-12T08:15:42Z level=error msg="error parsing metric: invalid UTF-8 string in label value \"variant=🧪\""
2023-11-12T08:15:45Z level=error msg="failed to ingest metric: label value contains non-ASCII characters \"experiment=beta©\""

2. Pull raw metric exposition from the exporter

curl -s http://experiment-exporter:9100/metrics | grep -A2 "experiment"

Example raw output (malformed):

# HELP experiment_active_total Number of active users per experiment variant
# TYPE experiment_active_total gauge
experiment_active_total{experiment=beta©,variant=control} 1234
experiment_active_total{experiment=beta🧪,variant=treatment} 567

3. Verify byte encoding

curl -s http://experiment-exporter:9100/metrics | hexdump -C | head

Hexdump snippet showing a non‑UTF‑8 byte (0xA9 in Latin‑1) for the © symbol:

00000000  23 20 48 45 4c 50 20 65  78 70 65 65 72 69 6d 65  |# HELP experime|
00000010  6e 74 5f 74 6f 73 74 5f  63 6f 75 6e 74 5f 74 6f  |nt_total_count_to|
... 
00000120  20 65 62 6c 61 73 63 68  5f 74 6f 74 61 6c 7d 20  | escape_total} |
00000130  31 31 31 31 0a 65 78 70  65 72 69 6d 65 65 6e 74  |1111.experiment|
00000140  5f 61 63 74 69 69 76 65  5f 74 61 6c 6b 20 7b 65  |_active_talk {e|
00000150  78 70 65 65 72 69 6d 65  6e 74 3d 62 65 74 61 a9  |xperiment=beta.|

4. Check client library handling

If the exporter uses the Go client, the Unicode handling guidelines require explicit UTF‑8 conversion before assigning label values.

Solution

Option 1 – Sanitize labels at the source (recommended)

Modify the experiment metric emission code to enforce UTF‑8 and strip disallowed characters.

Before (Python client example):

from prometheus_client import Gauge

experiment_gauge = Gauge('experiment_active_total',
                         'Number of active users per experiment variant',
                         ['experiment', 'variant'])

def emit_metrics(exp_name, variant, count):
    experiment_gauge.labels(experiment=exp_name, variant=variant).set(count)

Problem: exp_name may contain raw bytes or control characters.

After – with sanitization:

import unicodedata
import re

def sanitize_label(value):
    # Ensure UTF‑8, replace invalid bytes
    if isinstance(value, bytes):
        value = value.decode('utf-8', errors='replace')
    # Normalize to NFC, strip control chars, replace whitespace
    value = unicodedata.normalize('NFC', value)
    value = re.sub(r'[\x00-\x1F\x7F]', '', value)  # remove C0 controls
    # Optionally replace non‑ASCII with fallback
    value = re.sub(r'[^\x20-\x7E]', '_', value)
    return value

def emit_metrics(exp_name, variant, count):
    exp_name = sanitize_label(exp_name)
    variant = sanitize_label(variant)
    experiment_gauge.labels(experiment=exp_name, variant=variant).set(count)

Option 2 – Use `relabel_configs` to strip characters during scrape

Update prometheus.yml to apply a regex replacement on the offending label.

scrape_configs:
  - job_name: 'experiment_exporter'
    static_configs:
      - targets: ['experiment-exporter:9100']
    metric_relabel_configs:
      - source_labels: [experiment]
        regex: '(.+)'
        target_label: experiment
        replacement: '${1}'
        action: replace
      - source_labels: [experiment]
        regex: '[^\x20-\x7E]+'
        replacement: '_'
        action: replace

This configuration removes any non‑ASCII characters from the experiment label before the metric is stored.

Option 3 – Upgrade exporter to emit UTF‑8 explicitly

Ensure the HTTP response header includes Content-Type: text/plain; charset=utf-8 and that all strings are encoded with UTF‑8.

# Example in Go
w.Header().Set("Content-Type", "text/plain; charset=utf-8")

Verification

1. Confirm scraper health

curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.scrapeUrl=="http://experiment-exporter:9100/metrics")'

Look for "health":"up" and no recent error fields.

2. Validate metric values

curl -s http://prometheus:9090/api/v1/query?query=experiment_active_total | jq .

Sample successful response:

{
  "status":"success",
  "data":{
    "resultType":"vector",
    "result":[
      {
        "metric":{"experiment":"beta_c","variant":"control"},
        "value":[166.0,"1234"]
      },
      {
        "metric":{"experiment":"beta_c","variant":"treatment"},
        "value":[166.0,"567"]
      }
    ]
  }
}

3. Check logs for residual errors

journalctl -u prometheus -u experiment-exporter -f | grep "error parsing metric"

No output indicates the issue is resolved.

Prevention and Best Practices

Validate label values at ingestion. Use a shared sanitization library across all exporters.
Enforce UTF‑8 at the HTTP layer. Set charset=utf-8 in the Content-Type header.
Run a lint step in CI. Include a test that scrapes the exporter locally and asserts no invalid UTF-8 errors.
Monitor scraper error metrics. Alert on scrape_duration_seconds spikes or scrape_error counters.
Document allowed characters. Align experiment naming conventions with Prometheus label restrictions (ASCII printable, no control chars).
Use metric_relabel_configs as a safety net. Apply a whitelist regex that only permits [a-zA-Z0-9_:] for label values.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the error appear only after a new experiment is added? The new experiment introduced a label value containing a character outside the ASCII printable range (e.g., © or an emoji). Existing metrics were clean, so the scraper only failed when it encountered the offending line.
Can I keep Unicode characters in labels if I need them? Yes, but they must be valid UTF‑8 bytes. Ensure the exporter encodes strings as UTF‑8 and that no control characters are present. Using the Go client’s MustNewConstMetric with properly encoded strings works.
What is the difference between relabel_configs and metric_relabel_configs? relabel_configs runs before the scrape, transforming target metadata. metric_relabel_configs runs after metrics are collected, allowing you to modify or drop individual metric labels.
How do I detect hidden control characters in log‑derived labels? Pipe the metric output through cat -v or use od -c to reveal non‑printable bytes. Example: curl .../metrics | cat -v | grep -i "experiment".
Is there a built‑in Prometheus metric that counts label sanitization failures? Not directly, but the scrape_error_total counter increments for any parse error, which can be filtered by the instance label to isolate problematic exporters.