Loki query parse error from AI query builder in Grafana

Problem

In a development sandbox the AI‑powered query builder integrated with Grafana produces Loki queries that fail at execution time. The UI shows a “filter expression parse error” and the Loki server returns syntax‑related error messages such as:

parse error at line 1: unexpected token “=” near “app=frontend”

The failure occurs only for queries generated by the AI component; manually typed LogQL expressions work correctly.

Root Cause

The AI builder constructs the filter clause by concatenating user‑provided key/value pairs without applying LogQL‑specific quoting or escaping rules. Two specific mistakes were identified:

• Missing label selector braces: The builder emits app=frontend instead of the required {app="frontend"} syntax.
• Improper handling of special characters: Values containing spaces, quotes, or regex meta‑characters are inserted verbatim, breaking the parser (e.g., message=User login failed becomes message=User login failed which Loki interprets as two separate tokens).

Because Loki parses the filter string before any runtime validation, the malformed expression triggers a parse error, preventing the query from reaching the storage layer.

Debug

The investigation followed these steps:

1. Inspect Grafana server logs for the request payload:

2024-10-11 14:23:07.123 [info] plugin.ai-query-builder: Received query payload:
{
  "expr": "log | filter app=frontend and level=error",
  "start": "2024-10-10T00:00:00Z",
  "end": "2024-10-11T00:00:00Z"
}

2. Capture the HTTP request sent to Loki using tcpdump on the sandbox host:

sudo tcpdump -i any -s 0 -w /tmp/loki_req.pcap port 3100

3. Decode the captured request (or use curl to reproduce):

curl -G -s "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query=log | filter app=frontend and level=error' \
  --data-urlencode 'start=1728556800' \
  --data-urlencode 'end=1728643200'

Response:

{
  "status":"error",
  "errorType":"bad_data",
  "error":"parse error at line 1: unexpected token \"=\" near \"app=frontend\""
}

4. Compare with a hand‑crafted valid query:

curl -G -s "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={app="frontend",level="error"} |= ""' \
  --data-urlencode 'start=1728556800' \
  --data-urlencode 'end=1728643200'

This request returns a proper data.result array, confirming that the syntax is the only problem.

Solution

The AI builder was updated to generate a syntactically correct LogQL filter string. The key changes are:

Label selector construction

// Before (buggy)
function buildFilter(filters) {
  return filters.map(f => `${f.key}=${f.value}`).join(' and ');
}

// After (fixed)
function escapeValue(v) {
  // Escape backslashes and double quotes for LogQL string literals
  return v.replace(/\\/g, '\\\\').replace(/"/g, '\\"');
}

function buildFilter(filters) {
  const parts = filters.map(f => {
    const escaped = escapeValue(f.value);
    return `${f.key}="${escaped}"`;
  });
  // Wrap in curly braces to form a label selector
  return `{${parts.join(',')}}`;
}

Integration into the full query

// Before (buggy)
const query = `log | filter ${buildFilter(userFilters)} ${userExpression}`;

// After (fixed)
const selector = buildFilter(userFilters);
const query = `${selector} ${userExpression}`; // e.g. `{app="frontend",level="error"} |= "timeout"`

With these changes the AI builder now emits queries that Loki can parse:

{app="frontend",level="error"} |= "timeout"

Verify

After redeploying the updated AI service, perform the following checks:

1. In Grafana Explore, run a query generated by the AI builder. The result pane should display log streams without parse errors.
2. Inspect Loki’s /metrics endpoint for a decrease in logql_parse_errors_total:

curl -s http://loki:3100/metrics | grep logql_parse_errors_total
# Expected output: logql_parse_errors_total 0

3. Run the same curl command used during debugging; the JSON response should contain "status":"success".

Prevent

To avoid regression:

• Unit tests for buildFilter covering:
  • Values with spaces, quotes, backslashes.
  • Empty or missing values.
  • Special characters that require regex escaping.
• Integration test that sends the generated query to a Loki test instance and asserts status == "success".
• Schema validation in the AI service: reject any filter that does not match /^[a-zA-Z_][a-zA-Z0-9_]*$/ for keys and enforce string quoting for values.
• Monitoring: alert on logql_parse_errors_total rate > 0 for more than 5 minutes.
• Feature flag: keep the legacy builder as a fallback until the AI component passes a full production smoke test.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the error only appear in the sandbox and not in production?
   The sandbox uses a newer version of Loki (2.9) that enforces stricter LogQL parsing. Production runs an older 2.7 release that tolerated missing braces, masking the bug.
2. Can I see which part of the generated query caused the parse error?
   Enable logql_parser_debug=true in Loki’s config; the server will log the offending token and its position, which appears in the HTTP response error field.
3. Do I need to escape regular‑expression characters in label values?
   Label values are always treated as exact string matches, so they must be quoted and escaped as JSON strings. Regexes belong in the |~ operator, not in the selector.
4. Is there a way to validate a LogQL expression before sending it to Loki?
   Use the /loki/api/v1/query_range?query=...&debug=true endpoint; Loki returns a status:"success" with an empty result set if the syntax is valid.
5. What should I do if the AI builder needs to support complex pipelines (e.g., multiple |= stages)?
   Construct the selector first, then concatenate pipeline stages as separate strings. Ensure each stage is validated independently and that the final assembled query respects LogQL ordering rules.