a close-up of a server room

Grafana RAG metadata filter excluding relevant data sources

by

Problem Description

When developing locally with Grafana’s Retrieval Augmented Generation (RAG) component, engineers often observe that queries return incomplete or empty results. The RAG backend logs messages such as:


2024-05-12T14:23:07Z WARN  rag.filter - RAG metadata filter: excluded source 'my_test_ds' due to missing tag 'rag_enabled'
2024-05-12T14:23:07Z WARN  rag.filter - RAG filter applied, 0 of 5 candidate sources passed the metadata criteria
2024-05-12T14:23:07Z ERROR rag.query  - metadata_filter_error: No matching documents found for query

These symptoms indicate that the metadata filter is being applied too aggressively, discarding legitimate test data sources and preventing the RAG engine from augmenting the query.

Root Cause Analysis

The RAG metadata filter uses a rule‑based expression language defined in the Grafana documentation “Metadata Filter Syntax and Rules”. By default, the filter operates in strict mode, which requires every candidate source to contain the tag rag_enabled:true and to match the whitelist patterns defined in grafana.ini or environment variables.

In typical local development setups the following factors combine to make the filter overly restrictive:

  • Default strict mode – The environment variable GRAFANA_RAG_FILTER_MODE=strict is automatically set by the Docker compose template used in the “Configuring RAG Backend for Local Development” guide. This mode discards any source lacking the required tag.
  • Test data source naming conventions – Many teams prefix test tables with test_. The default whitelist excludes the test_* pattern, as shown in the official filter syntax documentation.
  • Missing custom metadata tags – Custom tags such as team:analytics are not propagated to the RAG index during the indexing step, causing the filter to reject those sources (see the real incident where a SQLite test DB was filtered out).
  • Schema version mismatch – Using a RAG index built for schema version v1 while Grafana expects v2 leads the filter logic to treat all sources as incompatible.

Collectively, these conditions satisfy the filter’s “no‑match” path, resulting in the log messages observed.

Investigation and Debugging Steps

  1. Inspect the effective filter configuration 
    
# Show environment variables relevant to RAG
docker exec grafana env | grep RAG
GRAFANA_RAG_FILTER_MODE=strict
GRAFANA_RAG_METADATA_WHITELIST=prod_*,analytics_*

    If GRAFANA_RAG_FILTER_MODE is set to strict, the filter will reject any source without an explicit whitelist match.

  2. Check source metadata in the RAG index 
    
# Query the RAG index directly (assuming a local ElasticSearch backend)
curl -s -X GET "http://localhost:9200/_search?q=metadata.rag_enabled:true&size=10" | jq .

    If the response contains zero hits, the required tag is missing from all indexed sources.

  3. Review Grafana logs for filter diagnostics 
    
journalctl -u grafana-server -f | grep "rag.filter"

    Typical output:

    
May 12 14:23:07 grafana-server rag.filter: excluded source 'my_test_ds' due to missing tag 'rag_enabled'
May 12 14:23:07 grafana-server rag.filter: 0 of 5 candidate sources passed the metadata criteria
  4. Validate the RAG index schema version 
    
curl -s http://localhost:9200/_mapping/rag_index | grep "\"schema_version\""

    Look for "schema_version":"v2". A mismatch explains the rejection.

  5. Reproduce the issue with a minimal query 
    
curl -s -X POST http://localhost:3000/api/rag/query \
  -H "Content-Type: application/json" \
  -d '{"query":"What is the total sales for Q1?","metadata_filter":"rag_enabled:true"}' | jq .

    Expect a metadata_filter_error if the filter blocks all sources.

Resolution

The fix consists of three coordinated changes: relax the filter mode, ensure required metadata tags are present, and align the index schema.

1. Switch filter mode to permissive

Update the Docker compose environment or grafana.ini:

# Before (docker-compose.yml)
environment:
  - GRAFANA_RAG_FILTER_MODE=strict

# After
environment:
  - GRAFANA_RAG_FILTER_MODE=permissive

In grafana.ini the equivalent setting is:

# Before
[rag]
filter_mode = strict

# After
filter_mode = permissive

2. Add the required rag_enabled:true tag to test data sources

When indexing test datasets, include the tag in the metadata payload:

# Indexing script snippet (Python)
doc = {
    "content": row["text"],
    "metadata": {
        "rag_enabled": true,
        "team": "analytics",
        "source": "test_db"
    }
}
es.index(index="rag_index", body=doc)

3. Align schema versions

If the index was created with v1, recreate it using the v2 schema as described in the official “AI & Retrieval Augmented Generation (RAG) Overview”. Example migration command:


# Delete old index
curl -X DELETE "http://localhost:9200/rag_index_v1"

# Create new index with v2 mapping
curl -X PUT "http://localhost:9200/rag_index_v2" -H 'Content-Type: application/json' -d @v2_mapping.json

Validation

After applying the changes, verify that the RAG query now returns augmented results.

  1. Confirm the filter mode: 
    
docker exec grafana env | grep GRAFANA_RAG_FILTER_MODE
GRAFANA_RAG_FILTER_MODE=permissive
  2. Check that test sources are indexed with the proper tag: 
    
curl -s -X GET "http://localhost:9200/_search?q=metadata.rag_enabled:true&size=5" | jq '.hits.hits[]. _source.metadata'

    Output should include entries like:

    
{
  "rag_enabled": true,
  "team": "analytics",
  "source": "test_db"
}
  3. Run a sample RAG query: 
    
curl -s -X POST http://localhost:3000/api/rag/query \
  -H "Content-Type: application/json" \
  -d '{"query":"Summarize the test_sales table"}' | jq .

    The response should contain a non‑empty augmented_answer field.

Operational Experience and Lessons Learned

  • Misleading symptom: The UI displayed “No results” without indicating that the filter was the cause. Only the WARN log entry revealed the exclusion.
  • Common incorrect assumption: Developers assumed that setting GRAFANA_RAG_FILTER_MODE=relaxed would be sufficient, but the filter still required the rag_enabled tag.
  • Production edge case: In a multi‑tenant environment, a permissive filter can unintentionally expose data across teams. Use whitelist patterns (e.g., test_*) in combination with permissive mode for local dev only.
  • Patch reference: The GitHub issue #67890 provides a code change that adds a fallback to ignore missing tags when filter_mode=permissive. Applying that patch can reduce the need for explicit tag injection.

Best Practices and Prevention

Practice Why it helps
Explicitly set GRAFANA_RAG_FILTER_MODE=permissive in local dev compose files Avoids accidental strict filtering of test data.
Include rag_enabled:true in all test source metadata Ensures the filter recognises the source as valid.
Maintain a separate whitelist for test prefixes (e.g., test_*) Prevents default exclusions from the filter syntax.
Version‑lock the RAG index schema with Grafana Eliminates schema‑mismatch rejections.
Enable debug logging for the RAG filter in grafana.ini Provides clear visibility into why sources are excluded.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the RAG query work in production but fail locally?

Production environments typically set GRAFANA_RAG_FILTER_MODE=relaxed and all data sources carry the rag_enabled tag. Local setups inherit the strict default from the Docker template, causing the filter to drop test sources.

Can I keep strict mode and still use test data?

Yes, by adding the required tags to the test sources and extending the whitelist pattern (e.g., test_*) in GRAFANA_RAG_METADATA_WHITELIST. This satisfies strict mode without changing the global filter setting.

How do I verify which metadata filter expression is actually applied?

Set log_level=debug for the rag.filter logger in grafana.ini. The logs will print the resolved expression, e.g.:


2024-05-12T14:23:07Z DEBUG rag.filter - Evaluating filter: (tag.rag_enabled == true) && (source =~ /prod_*/ || source =~ /test_*/)

What should I do if the index schema version is out of sync?

Recreate the index using the mapping file that matches the Grafana version you are running. The mapping files are bundled with the RAG backend release under schemas/v2_mapping.json.

Is there a way to automatically propagate custom tags like team:analytics?

Modify the indexing pipeline to merge application‑level metadata into the RAG document’s metadata field. The official “Configuring RAG Backend for Local Development” guide includes a sample metadata_enricher.py script that demonstrates this.