RAG Document Loader Segfault After Redis Batch Load During Benchmark

Q: Root Cause

The loader uses a custom C++ extension (redis_loader.cpp) that streams documents from a local file system into Redis via the hiredis async API. Two interacting problems cause the crash:

In this article

Table of Contents

RAG Document Loader Segfault After Redis Batch Load During Benchmark

Problem

During a high‑throughput performance benchmark of a Retrieval‑Augmented Generation (RAG) pipeline, the document loader crashes with a segmentation fault (SIGSEGV) or an out‑of‑memory (OOM) kill. The failure occurs immediately after a bulk INSERT of thousands of documents into Redis, just before the first query‑time retrieval is issued.

Typical symptoms observed in the benchmark run:

Benchmark harness aborts with Segmentation fault (core dumped) on the loader process.
System OOM killer logs sometimes appear, indicating a sudden memory spike.
Redis logs show a burst of SET commands completing successfully, followed by a pause of several seconds.
Container or pod restarts after the crash, causing the benchmark to lose progress.

Root Cause

The loader uses a custom C++ extension (redis_loader.cpp) that streams documents from a local file system into Redis via the hiredis async API. Two interacting problems cause the crash:

Unbounded buffer allocation. The loader pre‑allocates a std::vector sized to the total byte count of the batch (often > 2 GB). When the benchmark scales the batch size, the allocation exceeds the container memory limit, triggering an OOM kill or corrupting the heap.
Race condition in the async callback. The async redisAsyncCommand callback frees the buffer that was passed as void* user data before the Redis server has finished processing the command. Under high concurrency, the callback runs earlier than expected, causing the loader to dereference freed memory, leading to a segmentation fault.

Both issues stem from the assumption that the Redis client library will copy the payload synchronously, which is false for the async API. The combination of large payloads and aggressive concurrency amplifies the problem during benchmarking.

Debug & Investigation

Step‑by‑step debugging that reproduced the failure:

Reproduce locally with reduced batch size. Running the loader with --batch-size 1000 succeeded, while --batch-size 50000 crashed.
Inspect core dump. Using gdb on the core file revealed the crash point inside redisAsyncHandleWrite:


(gdb) bt
#0  0x00007f8c2a1b6f3a in __memcpy_chk () from /lib64/libc.so.6
#1  0x00007f8c2a1b6f3a in redisAsyncCommand (c=0x7f8c2a2c8000, fmt=0x5555555590c0 "...", ...) at hiredis.c:1245
#2  0x000055555555c3e2 in load_batch (this=0x7fffd5a3c0c0, batch=..., size=52428800) at redis_loader.cpp:87
#3  0x000055555555b9a1 in main (argc=3, argv=0x7fffd5a3c2e8) at redis_loader.cpp:42

Check memory usage. docker stats showed the loader container jumping from 300 MiB to >2 GiB just before the crash.
Enable hiredis tracing. Setting HIREDIS_DEBUG=1 produced the following log snippet:


[DEBUG] async command queued, user data ptr=0x7fffd5a3c0c0, size=52428800
[DEBUG] async callback invoked, freeing user data ptr=0x7fffd5a3c0c0
[DEBUG] async write attempted on freed buffer -> segfault

Validate Redis server health. redis-cli INFO memory showed normal memory usage; the server was not the source of OOM.

Solution

The fix consists of two parts: safe buffer handling and limiting concurrency.

1. Stream documents in chunks instead of a single massive buffer

Replace the monolithic allocation with a fixed‑size sliding window (e.g., 4 MiB) that is reused for each SET command.

Before:


std::vector payload;
payload.reserve(total_batch_bytes);
for (auto &doc : docs) {
    payload.insert(payload.end(), doc.begin(), doc.end());
}
redisAsyncCommand(context, async_cb, payload.data(), "SET %s %b", key.c_str(),
                  payload.data(), payload.size());

After:


constexpr size_t CHUNK_SIZE = 4 * 1024 * 1024; // 4 MiB
std::vector chunk(CHUNK_SIZE);
size_t offset = 0;

for (auto &doc : docs) {
    if (offset + doc.size() > CHUNK_SIZE) {
        // send current chunk
        redisAsyncCommand(context, async_cb, nullptr,
                          "SET %s %b", key.c_str(),
                          chunk.data(), offset);
        offset = 0;
    }
    std::memcpy(chunk.data() + offset, doc.data(), doc.size());
    offset += doc.size();
}

// send any remaining data
if (offset > 0) {
    redisAsyncCommand(context, async_cb, nullptr,
                      "SET %s %b", key.c_str(),
                      chunk.data(), offset);
}

The callback no longer frees the user data because nullptr is passed; the loader manages the memory lifecycle.

2. Serialize async writes or use a bounded semaphore

Introduce a semaphore that caps the number of in‑flight async commands (e.g., 64). The callback releases the semaphore, ensuring the loader never overwhelms the client library.


std::binary_semaphore inflight{64};

void async_cb(redisAsyncContext *c, void *r, void *priv) {
    // No payload to free
    inflight.release();
}

for (auto &chunk : chunks) {
    inflight.acquire(); // blocks when limit reached
    redisAsyncCommand(context, async_cb, nullptr,
                      "SET %s %b", key.c_str(),
                      chunk.data(), chunk.size());
}

3. Add explicit OOM guard

Before allocating a chunk, verify that the process RSS plus the new allocation stays below cgroup limits.


size_t rss = get_rss_kb() * 1024;
if (rss + CHUNK_SIZE > MAX_RSS_ALLOWED) {
    throw std::runtime_error("Memory limit exceeded before next chunk");
}

Verification

After applying the changes, run the same benchmark with the original batch size (50 k documents, ~2 GB total).

Process memory. docker stats shows a steady ~350 MiB RSS, no spikes.
Loader logs. Sample output:


[INFO] Starting batch load: 50000 docs, total 2.1GB
[INFO] Chunk 1/525 sent (4MiB)
[INFO] Chunk 2/525 sent (4MiB)
...
[INFO] All chunks sent, awaiting acknowledgements
[INFO] Load completed in 12.4s, 4.2k docs/s

Redis INFO stats reports total_commands_processed matching the number of chunks, and latency remains under 2 ms per command.

Prevention

Never pass raw buffers to async Redis callbacks without guaranteeing their lifetime. Use nullptr for user data or allocate per‑command structures that are freed inside the callback.
Enforce a maximum in‑flight command count. A semaphore or token bucket prevents burst‑induced heap pressure.
Cap batch size or stream documents. Even when the underlying storage can hold the whole batch, the loader should process in bounded chunks.
Monitor process RSS and Redis client queue depth. Alert when RSS exceeds 80 % of the container limit or when the async write queue length grows beyond a threshold.
Include unit tests that simulate high‑concurrency async writes. Use valgrind --leak-check=full or ASAN to catch use‑after‑free bugs early.

FAQ

Why does the crash only appear during benchmarks and not in normal operation?
Benchmarks increase both batch size and concurrency, exposing the unbounded buffer allocation and the race condition in the async callback. Production traffic typically processes smaller batches, staying below the memory threshold.
Can I keep the async API and still avoid the segfault?
Yes, by ensuring the payload’s lifetime extends until the callback runs. Pass a heap‑allocated struct as the privdata argument and free it inside the callback after confirming the command succeeded.
Is the issue specific to hiredis or does it affect other Redis clients?
The root cause—using a buffer that is freed before the client finishes sending—can appear in any asynchronous client that does not copy the payload. The fix (buffer lifetime management and throttling) applies universally.
How can I detect similar memory‑related bugs before they hit production?
Enable address sanitizer (ASAN) in CI builds, run load tests with exaggerated batch sizes, and monitor the process’s VmRSS via /proc/self/status during tests.
Do I need to adjust Redis server configuration after fixing the loader?
No server‑side changes are required. However, consider setting client-output-buffer-limit to a reasonable value (e.g., 64 MiB) to protect the server from misbehaving clients that might still send oversized payloads.

Related Topic Hub: Data Infrastructure Troubleshooting Hub