Problem – Segmentation fault in RAG document loader during Pinecone batch upsert
When running the Retrieval‑Augmented Generation (RAG) pipeline in a sandbox environment, the document loader crashes with a SIGSEGV (segmentation fault) while attempting to batch‑upsert vectors into a Pinecone test index. The failure stops the index creation process and leaves the API container in a non‑responsive state.
Root Cause
The crash originates from the native faiss library used to generate embeddings before the upsert. In the Docker image the faiss binary is compiled against glibc 2.28, while the base image provides glibc 2.31. The mismatch causes an illegal memory access when faiss::Index::add processes a batch larger than 500 vectors. The Pinecone client itself is not at fault; it receives a malformed request payload because the embedding generation aborts mid‑execution.
Debug – Investigation Steps
- Reproduce the crash locally
docker compose up -d api pinecone-mock python run_rag.py --docs ./sample_corpus --batch-size 1000Output:
2026-11-10 14:32:01,234 INFO rag.loader Starting document ingestion 2026-11-10 14:32:01,240 DEBUG rag.loader Generating embeddings for batch 1 (1000 docs) 2026-11-10 14:32:01,245 FATAL faiss[12345]: Segmentation fault (core dumped) - Inspect core dump with gdb
gdb -c core (gdb) bt #0 0x00007f8c5a1c3b2a in faiss::Index::add (this=0x7fffdc0000, n=1000, x=0x0) at Index.cpp:215 #1 0x00007f8c5a1d8f10 in rag.loader._embed_batch (self=..., batch=...) at loader.py:112 #2 0x00007f8c5a1e2c50 in rag.loader.load_documents (self=..., path=...) at loader.py:78The backtrace shows
faiss::Index::addreceiving a null pointer for the embedding matrix when the batch size exceeds the internal buffer limit. - Check library versions inside the container
# inside api container ldd $(python -c "import faiss; print(faiss.__file__)") | grep GLIBC libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (GLIBC_2.28)The host provides
GLIBC_2.31, confirming the ABI mismatch. - Validate Pinecone client request payload
curl -X POST http://localhost:8181/v1/upsert \ -H "Content-Type: application/json" \ -d '{"vectors": [{"id":"doc-1","values":null}]}' -vResponse shows
400 Bad Requestbecausevaluesis null, confirming that the embedding step failed before the HTTP call.
Solution – Aligning glibc and limiting batch size
Two complementary changes resolve the segmentation fault:
1. Rebuild the faiss wheel against the container’s glibc
Update the Dockerfile to install glibc from the base image before compiling faiss:
# Dockerfile (before)
FROM python:3.11-slim
RUN pip install faiss-cpu==1.7.4
# Dockerfile (after)
FROM python:3.11-slim
# Install build dependencies and match host glibc version
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libopenblas-dev \
wget \
&& wget -qO- https://ftp.gnu.org/gnu/libc/glibc-2.31.tar.gz | tar xz -C /tmp \
&& cd /tmp/glibc-2.31 && mkdir build && cd build && \
../configure --prefix=/usr && make -j$(nproc) && make install \
&& rm -rf /tmp/glibc-2.31 \
&& pip install --no-binary :all: faiss-cpu==1.7.4
Rebuilding ensures the faiss binary links against GLIBC_2.31, eliminating the illegal memory access.
2. Enforce a safe batch size in the loader
Modify loader.py to cap the batch size at 500 vectors, which is the maximum safe size for the default IndexFlatIP configuration.
# loader.py (before)
BATCH_SIZE = 1000
def _embed_batch(docs):
vectors = model.encode([d.text for d in docs])
return vectors
# loader.py (after)
MAX_SAFE_BATCH = 500
def _embed_batch(docs):
# Split large batches into safe chunks
vectors = []
for i in range(0, len(docs), MAX_SAFE_BATCH):
chunk = docs[i:i+MAX_SAFE_BATCH]
vectors.extend(model.encode([d.text for d in chunk]))
return vectors
The chunking logic prevents faiss::Index::add from ever receiving more than 500 vectors at once.
Verify – Confirming the fix
- Rebuild and redeploy
docker compose build api docker compose up -d api pinecone-mock - Run the ingestion script with the original batch size
python run_rag.py --docs ./sample_corpus --batch-size 1000Expected log excerpt:
2026-11-10 15:02:13,112 INFO rag.loader Starting document ingestion 2026-11-10 15:02:13,118 DEBUG rag.loader Generating embeddings for batch 1 (500 docs) 2026-11-10 15:02:13,125 INFO rag.loader Upserting batch 1 to Pinecone (500 vectors) 2026-11-10 15:02:13,210 INFO rag.loader Batch 1 upsert successful 2026-11-10 15:02:13,215 DEBUG rag.loader Generating embeddings for batch 2 (500 docs) ... 2026-11-10 15:02:14,001 INFO rag.loader Document ingestion completed - Validate Pinecone index contents
curl -X POST http://localhost:8181/v1/query \ -H "Content-Type: application/json" \ -d '{"vector": [0.01,0.02,...], "topK": 5}' | jq .Response returns a list of matching document IDs, confirming that vectors were stored correctly.
Prevent – Operational safeguards
- Pin the
faisswheel to a version built for the exact base image or compile it in‑container as part of the CI pipeline. - Enforce
MAX_SAFE_BATCHas a configurable constant and expose it via environment variable (FAISS_MAX_BATCH) to avoid accidental oversizing. - Add a health‑check endpoint that runs a tiny embedding + upsert round‑trip; alert if the request fails or returns a non‑200 status.
- Monitor container logs for
SIGSEGVorfaissstack traces and trigger an automatic rollback. - Include a unit test that generates a batch of 1,000 dummy documents and asserts that the loader completes without raising an exception.
Related Topic Hub: Vector Databases Troubleshooting Hub
FAQ
- Why does the crash only appear when using the sandbox Docker image?
The sandbox image is based on
python:3.11-slimwhich ships withglibc 2.31. The pre‑compiledfaiss-cpuwheel on PyPI is built againstglibc 2.28. In environments where the host glibc matches the wheel (e.g., Ubuntu 20.04), the mismatch does not surface. - Can I keep the original batch size of 1,000?
Only if you rebuild
faissagainst the container’s glibc and also increase the internal buffer limit of the index (e.g., by usingIndexIVFFlatwith a largernlist). The simplest production‑ready approach is to respect the 500‑vector safe limit. - Is the segmentation fault related to Pinecone’s SDK version?
No. The SDK receives a well‑formed HTTP request. The fault occurs earlier, inside the embedding generation step. Updating the SDK alone does not affect the crash.
- How can I detect similar ABI mismatches for other native libraries?
Run
lddon the library file inside the container and compare the reported GLIBC version with the host’sldd --version. CI pipelines should fail if the versions diverge. - Will this fix work on ARM‑based runners?
The same principle applies, but you must rebuild
faissfor the target architecture (e.g.,aarch64) and ensure the appropriate glibc version is present. Using thefaiss-cpusource distribution instead of the binary wheel avoids cross‑architecture issues.