Redis RAG Context Injection Failure After Serialization Error In Pipeline

Q: Root Cause – Incompatible Serialization + Expiration Policy Conflict

The pipeline serializes Document objects (containing embeddings, metadata, and raw text) using json.dumps(). The default json encoder cannot handle NumPy float32 arrays that store embeddings, causing a SerializationError. The error is caught, but the code proceeds to write an empty string to Redis.

Q: Debug – Step‑by‑Step Investigation 1. Reproduce the failure locally

# Run the ingestion script with a single document python ingest.py --doc-id 12345 Expected output: successful SET with a JSON payload containing embedding array.

Q: 2. Inspect the stored key

$ redis-cli GET rag:ctx:12345 "" $ redis-cli TTL rag:ctx:12345 -1 # indicates no expiration (key persisted) 3. Verify serialization path >>> from pipeline import serialize_document >>> doc = Document(id="12345", text="Paris is ...", embedding=np.random.rand(768).astype('float32')) >>> serialize_document(doc) Traceback (most recent call last): File ".../pipeline.py", line 42, in serialize_document return json.dumps(doc.__dict__) TypeError: Object of type ndarray is not JSON serializable 4. Check TTL enforcement job $ ps aux | grep ttl_enforcer redis-cli --scan --pattern 'rag:ctx:*' | while read key; do redis-cli PERSIST "$key" done The job runs every 5 minutes and removes TTL from any key that already exists, causing the later SET EX 3600 to be ignored.

Q: Solution – Robust Serialization and Consistent TTL Management 1. Use a serializer that handles NumPy arrays

Replace the naive json.dumps() with orjson or a custom encoder that converts ndarray to a list of floats.

Q: Verify – Confirming the Fix 1. Unit test serialization

def test_serialize_document(): doc = Document(id="test", text="sample", embedding=np.zeros(768, dtype='float32')) payload = serialize_document(doc) assert isinstance(payload, str) data = json.loads(payload) assert isinstance(data["embedding"], list) assert len(data["embedding"]) == 768 2. End‑to‑end pipeline run $ python ingest.py --doc-id 99999 INFO:pipeline:Serialized document size=12456 bytes INFO:redis_client:Stored key rag:ctx:99999 with TTL=3600 Check Redis:

Q: 3. Inference service prompt

2026-10-14 00:02:11,043 INFO inference - Prompt built: {"question":"What is the capital of France?","context":"Paris is the capital ..."} Prevent – Operational Guardrails Schema validation: Use pydantic models for the payload before writing to Redis; reject objects that fail validation. Monitoring: Emit a Prometheus counter pipeline_serialization_errors_total and set an alert on non‑zero values. TTL consistency check: Periodic job that verifies all rag:ctx:* keys have TTL > 0 and logs mismatches. CI gate: Add integration tests that simulate a failed serialization and assert that no key is created. Dependency lock: Pin orjson and numpy versions to avoid silent API changes. Related Topic Hub: Data Infrastructure Troubleshooting Hub

In this article

Table of Contents

Problem – Missing or Corrupted RAG Context after Pipeline Serialization

During the nightly data pipeline that populates a Redis vector store for Retrieval‑Augmented Generation (RAG), the downstream language‑model inference service receives prompts that lack the expected retrieved documents. The prompts contain empty context fields or truncated JSON blobs, resulting in hallucinations or incomplete answers.

Typical symptoms observed in the logs of the ingestion service and the inference service are:


2026-10-13 02:15:42,871 ERROR pipeline - SerializationError: Object of type Document is not JSON serializable
2026-10-13 02:15:43,102 WARN redis_client - Failed to set key rag:ctx:12345: TTL mismatch, existing TTL=0, new TTL=3600
2026-10-13 02:15:44,210 INFO inference - Prompt built: {"question":"What is the capital of France?","context":""}

Root Cause – Incompatible Serialization + Expiration Policy Conflict

The pipeline serializes Document objects (containing embeddings, metadata, and raw text) using json.dumps(). The default json encoder cannot handle NumPy float32 arrays that store embeddings, causing a SerializationError. The error is caught, but the code proceeds to write an empty string to Redis.

Simultaneously, the Redis client sets a TTL of 3600 seconds on the context key, while a background redis-cli --scan job that enforces a global no‑expire policy for keys matching rag:ctx:* overwrites the TTL to 0. When the key already exists with TTL=0, the SET command with EX 3600 fails silently, leaving the stale (empty) value in place.

Debug – Step‑by‑Step Investigation

1. Reproduce the failure locally


# Run the ingestion script with a single document
python ingest.py --doc-id 12345

Expected output: successful SET with a JSON payload containing embedding array.

Observed output: exception trace shown above and an empty value stored.

2. Inspect the stored key


$ redis-cli GET rag:ctx:12345
""
$ redis-cli TTL rag:ctx:12345
-1   # indicates no expiration (key persisted)

3. Verify serialization path


>>> from pipeline import serialize_document
>>> doc = Document(id="12345", text="Paris is ...", embedding=np.random.rand(768).astype('float32'))
>>> serialize_document(doc)
Traceback (most recent call last):
  File ".../pipeline.py", line 42, in serialize_document
    return json.dumps(doc.__dict__)
TypeError: Object of type ndarray is not JSON serializable

4. Check TTL enforcement job


$ ps aux | grep ttl_enforcer
redis-cli --scan --pattern 'rag:ctx:*' | while read key; do
    redis-cli PERSIST "$key"
done

The job runs every 5 minutes and removes TTL from any key that already exists, causing the later SET EX 3600 to be ignored.

Solution – Robust Serialization and Consistent TTL Management

1. Use a serializer that handles NumPy arrays

Replace the naive json.dumps() with orjson or a custom encoder that converts ndarray to a list of floats.


# before (pipeline.py)
def serialize_document(doc):
    return json.dumps(doc.__dict__)

# after (pipeline.py)
import orjson
def _encode_ndarray(obj):
    if isinstance(obj, np.ndarray):
        return obj.tolist()
    raise TypeError

def serialize_document(doc):
    return orjson.dumps(doc.__dict__, default=_encode_ndarray).decode()

2. Guard against partial writes

Only write to Redis when serialization succeeds. Use a transaction (MULTI/EXEC) to ensure the key is set with the correct TTL atomically.


# before (redis_client.py)
def store_context(key, payload, ttl):
    client.set(key, payload, ex=ttl)

# after (redis_client.py)
def store_context(key, payload, ttl):
    pipe = client.pipeline()
    pipe.set(key, payload)
    pipe.expire(key, ttl)
    pipe.execute()

3. Align TTL policy

Remove the conflicting “persist‑all” job or scope it to a different key pattern. If persistence is required for a subset, modify the script to skip rag:ctx: keys.


# before (ttl_enforcer.sh)
redis-cli --scan --pattern 'rag:*' | while read key; do redis-cli PERSIST "$key"; done

# after (ttl_enforcer.sh)
redis-cli --scan --pattern 'rag:cache:*' | while read key; do redis-cli PERSIST "$key"; done

Verify – Confirming the Fix

1. Unit test serialization


def test_serialize_document():
    doc = Document(id="test", text="sample", embedding=np.zeros(768, dtype='float32'))
    payload = serialize_document(doc)
    assert isinstance(payload, str)
    data = json.loads(payload)
    assert isinstance(data["embedding"], list)
    assert len(data["embedding"]) == 768

2. End‑to‑end pipeline run


$ python ingest.py --doc-id 99999
INFO:pipeline:Serialized document size=12456 bytes
INFO:redis_client:Stored key rag:ctx:99999 with TTL=3600

Check Redis:


$ redis-cli GET rag:ctx:99999 | wc -c
12456
$ redis-cli TTL rag:ctx:99999
3599

3. Inference service prompt


2026-10-14 00:02:11,043 INFO inference - Prompt built: {"question":"What is the capital of France?","context":"Paris is the capital ..."}

Prevent – Operational Guardrails

Schema validation: Use pydantic models for the payload before writing to Redis; reject objects that fail validation.
Monitoring: Emit a Prometheus counter pipeline_serialization_errors_total and set an alert on non‑zero values.
TTL consistency check: Periodic job that verifies all rag:ctx:* keys have TTL > 0 and logs mismatches.
CI gate: Add integration tests that simulate a failed serialization and assert that no key is created.
Dependency lock: Pin orjson and numpy versions to avoid silent API changes.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the pipeline sometimes succeed and sometimes fail? The failure occurs only when the document contains a NumPy float32 embedding. If the embedding is already a Python list (e.g., after a previous conversion), json.dumps() succeeds.
Can I keep the “persist‑all” job and still have TTL for RAG context? Yes, by excluding the rag:ctx: prefix from the pattern or by adding a conditional check inside the script to skip those keys.
Is orjson the only serializer that works? Any serializer that can convert ndarray to a JSON‑compatible type works (e.g., jsonpickle with custom handlers). orjson is preferred for speed and deterministic output.
What if I need to store binary embeddings instead of JSON? Store the raw bytes in a separate Redis key (e.g., rag:vec:12345) using SETEX with base64 encoding, and keep only metadata in the JSON context key.
How can I detect stale context keys that were written with an empty payload? Query Redis for keys where STRLEN is less than a threshold (e.g., STRLEN key < 100) and raise an alert.