RAG answer extraction returns empty response from RabbitMQ payload in Docker

Problem – Empty RAG Answers from RabbitMQ Payload in Docker

Symptoms

  • When the RAG service consumes a message from the rag_requests queue, the downstream LLM returns an empty string.
  • Application logs contain entries such as:
2026-11-17 14:03:12,874 ERROR rag.consumer - Answer extraction failed: payload missing 'question' field
2026-11-17 14:03:12,876 INFO  rag.consumer - Sending empty response to client
  • RabbitMQ management UI shows messages with a body size of 0 B or a truncated JSON object.
  • The issue appears only in the local Docker‑compose environment; production (K8s) works as expected.

Impact

The API endpoint that wraps the RAG pipeline returns HTTP 200 with an empty answer field, causing downstream UI components to display “No answer found”. This breaks the developer feedback loop and masks genuine retrieval failures.

Root Cause

The consumer expects a JSON payload with the following schema:

Field Type Description
request_id string Correlation identifier
question string User query to be answered
metadata object Optional context

In the Docker‑compose setup the producer serialises the message using pickle instead of JSON, while the consumer always deserialises with json.loads. Because pickle produces a binary blob, RabbitMQ stores it as a application/octet-stream payload. When the consumer reads the message, json.loads raises a JSONDecodeError, which is caught and logged as “payload missing ‘question’ field”. The exception handling path then forwards an empty answer.

Additional contributing factors:

  • Docker‑compose mounts the ./data volume with a different line‑ending convention, causing the producer to read an empty file when constructing the payload.
  • The RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS environment variables are mismatched between producer and consumer containers, leading to a fallback to the guest account that has restricted virtual‑host permissions. The producer silently fails to publish the full payload, leaving only the message properties.

Debug – Investigation Steps

1. Verify the raw message in the queue

# Enter the RabbitMQ container
docker exec -it rag_rabbitmq bash

# List messages in the rag_requests queue (requires rabbitmqadmin)
rabbitmqadmin get queue=rag_requests requeue=false count=5

Typical output observed:

{
    "payload": "",
    "properties": {
        "content_type": "application/json",
        "delivery_mode": 2,
        "correlation_id": "c3f9a1b2"
    }
}

2. Inspect producer logs

2026-11-17 13:58:41,102 INFO  rag.producer - Publishing request_id=7a9c3d
2026-11-17 13:58:41,103 ERROR rag.producer - Failed to serialize payload: TypeError: Object of type bytes is not JSON serializable

The error is swallowed by a generic except Exception block, allowing the publish call to succeed with an empty body.

3. Reproduce the failure locally

# Run a minimal consumer script to dump the raw body
python - <<'PY'
import pika, json, sys
conn = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
ch = conn.channel()
for method, properties, body in ch.consume('rag_requests', inactivity_timeout=1):
    if body is None:
        break
    print('RAW BODY:', body)
    try:
        print('JSON:', json.loads(body))
    except Exception as e:
        print('JSON error:', e)
    ch.basic_ack(method.delivery_tag)
PY

Result shows RAW BODY: b'' and a JSON error, confirming the empty payload.

4. Check environment variable consistency

# In docker-compose.yml
services:
  producer:
    environment:
      - RABBITMQ_DEFAULT_USER=rag_user
      - RABBITMQ_DEFAULT_PASS=secret
  consumer:
    environment:
      - RABBITMQ_DEFAULT_USER=rag_user
      - RABBITMQ_DEFAULT_PASS=secret

In the broken setup the consumer used RABBITMQ_DEFAULT_PASS=wrong, causing a permission fallback.

Solution – Code and Configuration Fixes

1. Align serialization format (JSON)

Update the producer to use json.dumps and ensure the content type is set correctly.

Before (producer.py)

import pika, pickle

def publish(request):
    connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq'))
    channel = connection.channel()
    payload = pickle.dumps(request)   # <-- binary payload
    channel.basic_publish(
        exchange='',
        routing_key='rag_requests',
        body=payload,
        properties=pika.BasicProperties(content_type='application/json')
    )

After (producer.py)

import json, pika

def publish(request):
    connection = pika.BlockingConnection(pika.ConnectionParameters('rabbitmq'))
    channel = connection.channel()
    payload = json.dumps(request).encode('utf-8')   # JSON bytes
    channel.basic_publish(
        exchange='',
        routing_key='rag_requests',
        body=payload,
        properties=pika.BasicProperties(content_type='application/json')
    )

2. Harden consumer deserialization

Add explicit validation and fallback logging.

Before (consumer.py)

payload = json.loads(body)
question = payload['question']

After (consumer.py)

import json, logging

def handle_message(body):
    try:
        payload = json.loads(body)
    except json.JSONDecodeError as exc:
        logging.error('Invalid JSON payload: %s', exc)
        return None

    if not isinstance(payload, dict) or 'question' not in payload:
        logging.error('Payload missing required field: %s', payload)
        return None

    return payload['question']

3. Ensure consistent RabbitMQ credentials

Update docker-compose.yml to share a single set of credentials and grant the correct permissions.

services:
  rabbitmq:
    image: rabbitmq:3-management
    environment:
      RABBITMQ_DEFAULT_USER: rag_user
      RABBITMQ_DEFAULT_PASS: secret
    ports:
      - "5672:5672"
      - "15672:15672"

  producer:
    build: ./producer
    environment:
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_USER=rag_user
      - RABBITMQ_PASS=secret

  consumer:
    build: ./consumer
    environment:
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_USER=rag_user
      - RABBITMQ_PASS=secret

4. Add schema validation (optional but recommended)

# schema.py
import jsonschema

RAG_REQUEST_SCHEMA = {
    "type": "object",
    "required": ["request_id", "question"],
    "properties": {
        "request_id": {"type": "string"},
        "question": {"type": "string"},
        "metadata": {"type": "object"},
    },
}
# consumer.py snippet
from schema import RAG_REQUEST_SCHEMA
from jsonschema import validate, ValidationError

def handle_message(body):
    try:
        payload = json.loads(body)
        validate(instance=payload, schema=RAG_REQUEST_SCHEMA)
    except (json.JSONDecodeError, ValidationError) as exc:
        logging.error('Payload validation error: %s', exc)
        return None
    return payload['question']

Verify – Confirming the Fix

Functional test

# Run the producer with a known question
python -m producer '{"request_id":"test-123","question":"What is the capital of France?"}'

# Consume the message
python -m consumer --once

Expected log output:

2026-11-17 14:20:05,112 INFO  rag.consumer - Received request_id=test-123
2026-11-17 14:20:05,115 INFO  rag.consumer - Extracted answer: "Paris"
2026-11-17 14:20:05,118 INFO  rag.consumer - Sent response to client

Queue inspection

docker exec -it rag_rabbitmq rabbitmqadmin get queue=rag_requests requeue=false count=1

Output now shows a non‑empty JSON payload:

{
    "payload": "{\"request_id\":\"test-123\",\"question\":\"What is the capital of France?\"}",
    "properties": {
        "content_type": "application/json",
        "delivery_mode": 2,
        "correlation_id": "test-123"
    }
}

Prevent – Operational Guardrails

  • Schema enforcement: Integrate jsonschema validation in both producer and consumer pipelines; fail fast on schema mismatch.
  • Health endpoint: Expose /healthz that publishes a test message and verifies successful deserialization.
  • Monitoring: Alert on consumer.errors.payload_missing or consumer.errors.json_decode counters exceeding a threshold.
  • CI checks: Add unit tests that serialize a sample request and deserialize it using the consumer code path.
  • Docker compose lint: Use docker-compose config and a linter to ensure environment variables are not duplicated or overridden.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

  1. Why does the production environment work while local Docker fails?
    Production uses a Kafka‑based transport that serialises JSON, whereas the local stack mistakenly used pickle. The mismatch only appears in the Docker compose configuration.
  2. Can I keep using pickle if I add a custom deserializer?
    Technically yes, but pickle is insecure for untrusted data and breaks language‑agnostic consumers. Switching to JSON (or protobuf) is the recommended approach.
  3. What if the message size exceeds the default RabbitMQ frame limit?
    Increase frame_max on both producer and consumer connections or split large contexts into multiple messages. The empty‑payload symptom is unrelated but worth monitoring.
  4. How do I debug similar empty‑payload issues with other queues?
    Use rabbitmqadmin get to inspect raw bodies, verify content_type, and compare producer serialization code with consumer deserialization logic.
  5. Is there a way to automatically retry failed deserialization?
    Wrap the consumer handling in a dead‑letter exchange; on validation failure, republish to a rag_failed queue for manual inspection.