RabbitMQ RAG Metadata Filter Dropping Relevant Docs After Filter Tweak

Q: Root Cause

The filter logic was tightened to require both a source:trusted tag and a confidence >= 0.9 field. The original implementation used a OR condition, allowing documents that satisfied either criterion. The new AND clause unintentionally excluded:

Q: Before (v1.4)

# rag_consumer/metadata_filter.py def should_forward(msg): return (msg["source"] == "trusted") and (msg["confidence"] >= 0.9) After (v1.5) # rag_consumer/metadata_filter.py WHITELIST_SOURCES = {"internal_wiki", "partner_feed"} def should_forward(msg): # Allow any whitelisted source regardless of confidence if msg["source"] in WHITELIST_SOURCES: return True # For all other sources keep the original strict rule return (msg["source"] == "trusted") or (msg["confidence"] >= 0.9) Deploy the updated consumer:

In this article

Table of Contents

Problem

The Retrieval‑Augmented Generation (RAG) pipeline uses RabbitMQ to stream document metadata to a GPU‑accelerated inference service. After a recent change to the metadata filter, the pipeline began discarding documents that should have been included in the context set. The LLM responses became fragmented, missing critical facts, and the generation_quality_score metric dropped from 0.87 to 0.62.

Observed symptoms:

Log entries such as INFO dropped_message: metadata filter mismatch appear for a large fraction of incoming doc_meta messages.
Downstream context_builder reports context_size=0 for 30‑40% of queries.
GPU inference pods show increased GPUUtil idle time because fewer tokens are generated.
Alert RAG_IncompleteContext fires after the filter tweak.

Root Cause

The filter logic was tightened to require both a source:trusted tag and a confidence >= 0.9 field. The original implementation used a OR condition, allowing documents that satisfied either criterion. The new AND clause unintentionally excluded:

High‑confidence documents from less‑trusted sources (e.g., internal wikis).
Trusted sources that only provide a confidence estimate of 0.8 (common for newer models).

Because the filter runs inside the RabbitMQ consumer before the message is forwarded to the context_aggregator, any document that fails the filter is silently dropped, leaving no trace in the downstream logs except the generic “metadata filter mismatch”. This explains why the issue manifested only after the filter tweak and why the symptom appears as missing context rather than a RabbitMQ connectivity problem.

Debug

Step‑by‑step investigation performed in the production environment:

Inspect RabbitMQ queue depth and dead‑letter exchange.
```
rabbitmqctl list_queues name messages_ready messages_unacknowledged
# Output
metadata_queue    0    0
```
No backlog, confirming messages are being consumed.

Enable debug logging for the consumer.

kubectl -n rag set env deployment/rag-consumer LOG_LEVEL=debug
# After restart, tail logs
kubectl -n rag logs -f deployment/rag-consumer

Sample log excerpt:

2024-09-30T12:14:03.212Z DEBUG filter_evaluate: doc_id=42c1a7b3 match=False
  tags=[internal, confidential] confidence=0.85 source=trusted
2024-09-30T12:14:03.215Z INFO  dropped_message: doc_id=42c1a7b3 reason=metadata_filter_mismatch

Reproduce the filter decision locally.

python - <<'PY'
msg = {
    "doc_id": "42c1a7b3",
    "tags": ["internal", "confidential"],
    "confidence": 0.85,
    "source": "trusted"
}
def filter(msg):
    return (msg["source"] == "trusted") and (msg["confidence"] >= 0.9)
print(filter(msg))  # False
PY

Compare original and updated filter expressions.

Version Expression

v1.3 (pre‑tweak) (source == "trusted") or (confidence >= 0.9)

v1.4 (post‑tweak) (source == "trusted") and (confidence >= 0.9)

Version	Expression
v1.3 (pre‑tweak)	`(source == "trusted") or (confidence >= 0.9)`
v1.4 (post‑tweak)	`(source == "trusted") and (confidence >= 0.9)`

Solution

The filter must allow documents that satisfy either condition, but also give priority to a whitelist of sources that are known to produce lower confidence scores. The fix introduces a configurable whitelist and reverts the logical operator to OR while preserving the stricter rule for non‑whitelisted sources.

Before (v1.4)

# rag_consumer/metadata_filter.py
def should_forward(msg):
    return (msg["source"] == "trusted") and (msg["confidence"] >= 0.9)

After (v1.5)

# rag_consumer/metadata_filter.py
WHITELIST_SOURCES = {"internal_wiki", "partner_feed"}

def should_forward(msg):
    # Allow any whitelisted source regardless of confidence
    if msg["source"] in WHITELIST_SOURCES:
        return True
    # For all other sources keep the original strict rule
    return (msg["source"] == "trusted") or (msg["confidence"] >= 0.9)

Deploy the updated consumer:

kubectl -n rag set image deployment/rag-consumer rag-consumer=registry.example.com/rag-consumer:1.5.0
kubectl -n rag rollout status deployment/rag-consumer

Why it works:

Documents from internal_wiki and partner_feed are always forwarded, ensuring internal knowledge bases are not lost.
All other sources retain the original “trusted OR high confidence” rule, preventing overly permissive inclusion of noisy data.
The whitelist is externalized to a config map, making future adjustments without code changes possible.

Verify

Validation steps after the rollout:

Check that the metadata_filter debug log no longer reports mismatches for whitelisted sources.
Run a controlled query that previously returned an empty context.

curl -s -X POST http://rag-gateway/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"What is the latest internal policy on data retention?"}' | jq .response
# Expected: contains policy excerpt from internal_wiki

Monitor the generation_quality_score metric for a 15‑minute window.

promtool query instant --query='generation_quality_score{job="rag"}' now
# Should be >= 0.85

Confirm queue depth remains stable and no messages are dead‑lettered.

rabbitmqctl list_queues name messages_ready messages_unacknowledged
# Expected: metadata_queue 0 0

Prevent

To avoid regression:

Unit test the filter logic. Add a table‑driven test covering whitelist, trusted, and confidence scenarios.
Configuration‑driven thresholds. Store confidence_threshold and whitelist in a ConfigMap; reload without redeploy.
Alert on drop rate. Create a Prometheus alert that fires when metadata_filter_dropped_total exceeds 5% of metadata_filter_processed_total over 5 minutes.
Canary deployment. Deploy the new filter to 10% of consumer pods first, compare context size metrics before full rollout.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why did the filter start dropping documents only after the code change?
The logical operator switched from OR to AND, making the condition stricter. Documents that satisfied only one of the two criteria were no longer forwarded.
Can I see which documents are being filtered in real time?
Enable LOG_LEVEL=debug for the consumer and watch the dropped_message logs. They include the full payload, allowing you to spot patterns.
Is there a way to make the whitelist dynamic without redeploy?
Store the whitelist in a ConfigMap and mount it as a volume. The consumer watches the file for changes (e.g., using fsnotify) and reloads the list on the fly.
What impact does a high drop rate have on GPU utilization?
Fewer context tokens mean the model generates shorter sequences, leading to lower GPUUtil and higher idle time, which reduces overall throughput.
How do I differentiate between a filter‑related drop and a RabbitMQ delivery failure?
Delivery failures surface as nack or requeue events in RabbitMQ logs, while filter drops are logged by the consumer with the dropped_message tag. Correlate the timestamps to identify the source.