Problem – Empty RAG Answers from RabbitMQ Payload in Docker
Symptoms
- When the RAG service consumes a message from the
rag_requestsqueue, the downstream LLM returns an empty string. - Application logs contain entries such as:
2026-11-17 14:03:12,874 ERROR rag.consumer - Answer extraction failed: payload missing 'question' field
2026-11-17 14:03:12,876 INFO rag.consumer - Sending empty response to client
- RabbitMQ management UI shows messages with a body size of
0 Bor a truncated JSON object. - The issue appears only in the local Docker‑compose environment; production (K8s) works as expected.
Impact
The API endpoint that wraps the RAG pipeline returns HTTP 200 with an empty answer field, causing downstream UI components to display “No answer found”. This breaks the developer feedback loop and masks genuine retrieval failures.
Root Cause
The consumer expects a JSON payload with the following schema:
| Field | Type | Description |
|---|---|---|
| request_id | string | Correlation identifier |
| question | string | User query to be answered |
| metadata | object | Optional context |
In the Docker‑compose setup the producer serialises the message using pickle instead of JSON, while the consumer always deserialises with json.loads. Because pickle produces a binary blob, RabbitMQ stores it as a application/octet-stream payload. When the consumer reads the message, json.loads raises a JSONDecodeError, which is caught and logged as “payload missing ‘question’ field”. The exception handling path then forwards an empty answer.
Additional contributing factors:
- Docker‑compose mounts the
./datavolume with a different line‑ending convention, causing the producer to read an empty file when constructing the payload. - The
RABBITMQ_DEFAULT_USERandRABBITMQ_DEFAULT_PASSenvironment variables are mismatched between producer and consumer containers, leading to a fallback to theguestaccount that has restricted virtual‑host permissions. The producer silently fails to publish the full payload, leaving only the message properties.
Debug – Investigation Steps
1. Verify the raw message in the queue
# Enter the RabbitMQ container
docker exec -it rag_rabbitmq bash
# List messages in the rag_requests queue (requires rabbitmqadmin)
rabbitmqadmin get queue=rag_requests requeue=false count=5
Typical output observed:
{
"payload": "",
"properties": {
"content_type": "application/json",
"delivery_mode": 2,
"correlation_id": "c3f9a1b2"
}
}
2. Inspect producer logs
2026-11-17 13:58:41,102 INFO rag.producer - Publishing request_id=7a9c3d
2026-11-17 13:58:41,103 ERROR rag.producer - Failed to serialize payload: TypeError: Object of type bytes is not JSON serializable
The error is swallowed by a generic except Exception block, allowing the publish call to succeed with an empty body.
3. Reproduce the failure locally
# Run a minimal consumer script to dump the raw body
python - <<'PY'
import pika, json, sys
conn = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
ch = conn.channel()
for method, properties, body in ch.consume('rag_requests', inactivity_timeout=1):
if body is None:
break
print('RAW BODY:', body)
try:
print('JSON:', json.loads(body))
except Exception as e:
print('JSON error:', e)
ch.basic_ack(method.delivery_tag)
PY
Result shows RAW BODY: b'' and a JSON error, confirming the empty payload.
4. Check environment variable consistency
# In docker-compose.yml
services:
producer:
environment:
- RABBITMQ_DEFAULT_USER=rag_user
- RABBITMQ_DEFAULT_PASS=secret
consumer:
environment:
- RABBITMQ_DEFAULT_USER=rag_user
- RABBITMQ_DEFAULT_PASS=secret
In the broken setup the consumer used RABBITMQ_DEFAULT_PASS=wrong, causing a permission fallback.
Solution – Code and Configuration Fixes
1. Align serialization format (JSON)
Update the producer to use json.dumps and ensure the content type is set correctly.
Before (producer.py)
import pika, pickle
def publish(request):
connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq'))
channel = connection.channel()
payload = pickle.dumps(request) # <-- binary payload
channel.basic_publish(
exchange='',
routing_key='rag_requests',
body=payload,
properties=pika.BasicProperties(content_type='application/json')
)
After (producer.py)
import json, pika
def publish(request):
connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq'))
channel = connection.channel()
payload = json.dumps(request).encode('utf-8') # JSON bytes
channel.basic_publish(
exchange='',
routing_key='rag_requests',
body=payload,
properties=pika.BasicProperties(content_type='application/json')
)
2. Harden consumer deserialization
Add explicit validation and fallback logging.
Before (consumer.py)
payload = json.loads(body)
question = payload['question']
After (consumer.py)
import json, logging
def handle_message(body):
try:
payload = json.loads(body)
except json.JSONDecodeError as exc:
logging.error('Invalid JSON payload: %s', exc)
return None
if not isinstance(payload, dict) or 'question' not in payload:
logging.error('Payload missing required field: %s', payload)
return None
return payload['question']
3. Ensure consistent RabbitMQ credentials
Update docker-compose.yml to share a single set of credentials and grant the correct permissions.
services:
rabbitmq:
image: rabbitmq:3-management
environment:
RABBITMQ_DEFAULT_USER: rag_user
RABBITMQ_DEFAULT_PASS: secret
ports:
- "5672:5672"
- "15672:15672"
producer:
build: ./producer
environment:
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_USER=rag_user
- RABBITMQ_PASS=secret
consumer:
build: ./consumer
environment:
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_USER=rag_user
- RABBITMQ_PASS=secret
4. Add schema validation (optional but recommended)
# schema.py
import jsonschema
RAG_REQUEST_SCHEMA = {
"type": "object",
"required": ["request_id", "question"],
"properties": {
"request_id": {"type": "string"},
"question": {"type": "string"},
"metadata": {"type": "object"},
},
}
# consumer.py snippet
from schema import RAG_REQUEST_SCHEMA
from jsonschema import validate, ValidationError
def handle_message(body):
try:
payload = json.loads(body)
validate(instance=payload, schema=RAG_REQUEST_SCHEMA)
except (json.JSONDecodeError, ValidationError) as exc:
logging.error('Payload validation error: %s', exc)
return None
return payload['question']
Verify – Confirming the Fix
Functional test
# Run the producer with a known question
python -m producer '{"request_id":"test-123","question":"What is the capital of France?"}'
# Consume the message
python -m consumer --once
Expected log output:
2026-11-17 14:20:05,112 INFO rag.consumer - Received request_id=test-123
2026-11-17 14:20:05,115 INFO rag.consumer - Extracted answer: "Paris"
2026-11-17 14:20:05,118 INFO rag.consumer - Sent response to client
Queue inspection
docker exec -it rag_rabbitmq rabbitmqadmin get queue=rag_requests requeue=false count=1
Output now shows a non‑empty JSON payload:
{
"payload": "{\"request_id\":\"test-123\",\"question\":\"What is the capital of France?\"}",
"properties": {
"content_type": "application/json",
"delivery_mode": 2,
"correlation_id": "test-123"
}
}
Prevent – Operational Guardrails
- Schema enforcement: Integrate
jsonschemavalidation in both producer and consumer pipelines; fail fast on schema mismatch. - Health endpoint: Expose
/healthzthat publishes a test message and verifies successful deserialization. - Monitoring: Alert on
consumer.errors.payload_missingorconsumer.errors.json_decodecounters exceeding a threshold. - CI checks: Add unit tests that serialize a sample request and deserialize it using the consumer code path.
- Docker compose lint: Use
docker-compose configand a linter to ensure environment variables are not duplicated or overridden.
Related Topic Hub: Data Infrastructure Troubleshooting Hub
FAQ
- Why does the production environment work while local Docker fails?
Production uses a Kafka‑based transport that serialises JSON, whereas the local stack mistakenly usedpickle. The mismatch only appears in the Docker compose configuration. - Can I keep using
pickleif I add a custom deserializer?
Technically yes, butpickleis insecure for untrusted data and breaks language‑agnostic consumers. Switching to JSON (or protobuf) is the recommended approach. - What if the message size exceeds the default RabbitMQ frame limit?
Increaseframe_maxon both producer and consumer connections or split large contexts into multiple messages. The empty‑payload symptom is unrelated but worth monitoring. - How do I debug similar empty‑payload issues with other queues?
Userabbitmqadmin getto inspect raw bodies, verifycontent_type, and compare producer serialization code with consumer deserialization logic. - Is there a way to automatically retry failed deserialization?
Wrap the consumer handling in a dead‑letter exchange; on validation failure, republish to arag_failedqueue for manual inspection.