Segmentation fault in RAG document loader during Pinecone batch upsert

Problem – Segmentation fault in RAG document loader during Pinecone batch upsert

When running the Retrieval‑Augmented Generation (RAG) pipeline in a sandbox environment, the document loader crashes with a SIGSEGV (segmentation fault) while attempting to batch‑upsert vectors into a Pinecone test index. The failure stops the index creation process and leaves the API container in a non‑responsive state.

Root Cause

The crash originates from the native faiss library used to generate embeddings before the upsert. In the Docker image the faiss binary is compiled against glibc 2.28, while the base image provides glibc 2.31. The mismatch causes an illegal memory access when faiss::Index::add processes a batch larger than 500 vectors. The Pinecone client itself is not at fault; it receives a malformed request payload because the embedding generation aborts mid‑execution.

Debug – Investigation Steps

  1. Reproduce the crash locally
    docker compose up -d api pinecone-mock
    python run_rag.py --docs ./sample_corpus --batch-size 1000
    

    Output:

    2026-11-10 14:32:01,234 INFO  rag.loader Starting document ingestion
    2026-11-10 14:32:01,240 DEBUG rag.loader Generating embeddings for batch 1 (1000 docs)
    2026-11-10 14:32:01,245 FATAL faiss[12345]: Segmentation fault (core dumped)
    
  2. Inspect core dump with gdb
    gdb -c core
    (gdb) bt
    #0  0x00007f8c5a1c3b2a in faiss::Index::add (this=0x7fffdc0000, n=1000, x=0x0) at Index.cpp:215
    #1  0x00007f8c5a1d8f10 in rag.loader._embed_batch (self=..., batch=...) at loader.py:112
    #2  0x00007f8c5a1e2c50 in rag.loader.load_documents (self=..., path=...) at loader.py:78
    

    The backtrace shows faiss::Index::add receiving a null pointer for the embedding matrix when the batch size exceeds the internal buffer limit.

  3. Check library versions inside the container
    # inside api container
    ldd $(python -c "import faiss; print(faiss.__file__)") | grep GLIBC
    	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (GLIBC_2.28)
    

    The host provides GLIBC_2.31, confirming the ABI mismatch.

  4. Validate Pinecone client request payload
    curl -X POST http://localhost:8181/v1/upsert \
      -H "Content-Type: application/json" \
      -d '{"vectors": [{"id":"doc-1","values":null}]}' -v
    

    Response shows 400 Bad Request because values is null, confirming that the embedding step failed before the HTTP call.

Solution – Aligning glibc and limiting batch size

Two complementary changes resolve the segmentation fault:

1. Rebuild the faiss wheel against the container’s glibc

Update the Dockerfile to install glibc from the base image before compiling faiss:

# Dockerfile (before)
FROM python:3.11-slim
RUN pip install faiss-cpu==1.7.4

# Dockerfile (after)
FROM python:3.11-slim
# Install build dependencies and match host glibc version
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libopenblas-dev \
    wget \
    && wget -qO- https://ftp.gnu.org/gnu/libc/glibc-2.31.tar.gz | tar xz -C /tmp \
    && cd /tmp/glibc-2.31 && mkdir build && cd build && \
       ../configure --prefix=/usr && make -j$(nproc) && make install \
    && rm -rf /tmp/glibc-2.31 \
    && pip install --no-binary :all: faiss-cpu==1.7.4

Rebuilding ensures the faiss binary links against GLIBC_2.31, eliminating the illegal memory access.

2. Enforce a safe batch size in the loader

Modify loader.py to cap the batch size at 500 vectors, which is the maximum safe size for the default IndexFlatIP configuration.

# loader.py (before)
BATCH_SIZE = 1000

def _embed_batch(docs):
    vectors = model.encode([d.text for d in docs])
    return vectors

# loader.py (after)
MAX_SAFE_BATCH = 500

def _embed_batch(docs):
    # Split large batches into safe chunks
    vectors = []
    for i in range(0, len(docs), MAX_SAFE_BATCH):
        chunk = docs[i:i+MAX_SAFE_BATCH]
        vectors.extend(model.encode([d.text for d in chunk]))
    return vectors

The chunking logic prevents faiss::Index::add from ever receiving more than 500 vectors at once.

Verify – Confirming the fix

  1. Rebuild and redeploy
    docker compose build api
    docker compose up -d api pinecone-mock
    
  2. Run the ingestion script with the original batch size
    python run_rag.py --docs ./sample_corpus --batch-size 1000
    

    Expected log excerpt:

    2026-11-10 15:02:13,112 INFO  rag.loader Starting document ingestion
    2026-11-10 15:02:13,118 DEBUG rag.loader Generating embeddings for batch 1 (500 docs)
    2026-11-10 15:02:13,125 INFO  rag.loader Upserting batch 1 to Pinecone (500 vectors)
    2026-11-10 15:02:13,210 INFO  rag.loader Batch 1 upsert successful
    2026-11-10 15:02:13,215 DEBUG rag.loader Generating embeddings for batch 2 (500 docs)
    ...
    2026-11-10 15:02:14,001 INFO  rag.loader Document ingestion completed
    
  3. Validate Pinecone index contents
    curl -X POST http://localhost:8181/v1/query \
      -H "Content-Type: application/json" \
      -d '{"vector": [0.01,0.02,...], "topK": 5}' | jq .
    

    Response returns a list of matching document IDs, confirming that vectors were stored correctly.

Prevent – Operational safeguards

  • Pin the faiss wheel to a version built for the exact base image or compile it in‑container as part of the CI pipeline.
  • Enforce MAX_SAFE_BATCH as a configurable constant and expose it via environment variable (FAISS_MAX_BATCH) to avoid accidental oversizing.
  • Add a health‑check endpoint that runs a tiny embedding + upsert round‑trip; alert if the request fails or returns a non‑200 status.
  • Monitor container logs for SIGSEGV or faiss stack traces and trigger an automatic rollback.
  • Include a unit test that generates a batch of 1,000 dummy documents and asserts that the loader completes without raising an exception.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the crash only appear when using the sandbox Docker image?

    The sandbox image is based on python:3.11-slim which ships with glibc 2.31. The pre‑compiled faiss-cpu wheel on PyPI is built against glibc 2.28. In environments where the host glibc matches the wheel (e.g., Ubuntu 20.04), the mismatch does not surface.

  2. Can I keep the original batch size of 1,000?

    Only if you rebuild faiss against the container’s glibc and also increase the internal buffer limit of the index (e.g., by using IndexIVFFlat with a larger nlist). The simplest production‑ready approach is to respect the 500‑vector safe limit.

  3. Is the segmentation fault related to Pinecone’s SDK version?

    No. The SDK receives a well‑formed HTTP request. The fault occurs earlier, inside the embedding generation step. Updating the SDK alone does not affect the crash.

  4. How can I detect similar ABI mismatches for other native libraries?

    Run ldd on the library file inside the container and compare the reported GLIBC version with the host’s ldd --version. CI pipelines should fail if the versions diverge.

  5. Will this fix work on ARM‑based runners?

    The same principle applies, but you must rebuild faiss for the target architecture (e.g., aarch64) and ensure the appropriate glibc version is present. Using the faiss-cpu source distribution instead of the binary wheel avoids cross‑architecture issues.