Weaviate Segmentation Fault During Batch PDF Ingestion In Docker Compose

Q: Problem – Segmentation Fault in Weaviate RAG Loader During Batch PDF Ingestion

During staging tests a Weaviate instance (v1.22.0) deployed via docker‑compose crashes with a segmentation fault (SIGSEGV) when the RAG document loader attempts to ingest a batch of large PDF files (each > 30 MB). The failure occurs after the first few documents are indexed and manifests as an immediate container restart, breaking the simulated user query workload.

Q: Root Cause – Native PDF Parsing Library Exhausts Stack in the Multi‑Threaded Worker

The RAG loader uses pdfminer.six (via langchain) to extract text. In the Docker image the library is compiled against libpoppler provided by ubuntu:22.04. When Weaviate processes a batch, it spawns multiple ingestion workers (default --batch-size=100, --workers=4). Each worker loads the native poppler shared library. For PDFs that contain complex object streams, poppler allocates large stack frames. The default Docker container stack size (8 MiB) is insufficient for the concurrent workers, causing the native code to write past the stack and trigger a segmentation fault.

Q: Debug – Reproducing and Isolating the Failure 1. Observe Container Crash

$ docker compose logs weaviate weaviate_1 | 2024-10-22T14:12:31Z INFO [batch] Received batch of 120 documents weaviate_1 | 2024-10-22T14:12:31Z INFO [loader] Starting ingestion workers=4 weaviate_1 | 2024-10-22T14:12:33Z ERROR [loader] Worker 2 crashed: Segmentation fault (core dumped) weaviate_1 | 2024-10-22T14:12:33Z FATAL [loader] Process terminated by signal 11 weaviate_1 | 2024-10-22T14:12:33Z INFO [docker] Container exiting with code 139 2. Core Dump Inspection (host) # ls -l /var/lib/docker/containers/*/core /var/lib/docker/containers/abcd1234/core # core file present # gdb -c core /usr/local/bin/weaviate (gdb) bt #0 0x00007f9c5d2a1b3f in poppler::PDFDoc::open (this=0x7fffdc0000) at PDFDoc.cc:215 #1 0x00007f9c5d2a5e2a in pdfminer::PDFParser::parse (this=0x7fffdc1000) at parser.cpp:87 #2 0x00007f9c5d2b1c10 in ingest_worker (arg=0x7fffdc2000) at loader.py:112 ... 3. Verify Stack Size Limitation # docker exec -it weaviate_1 cat /proc/self/limits | grep "stack size" Max stack size 8192 kB unlimited bytes 4. Reproduce with Single Worker # curl -X POST http://localhost:8080/v1/batch \ -H "Content-Type: application/json" \ -d '{"batchSize":100,"workers":1,"documents":[...large PDFs...]}' -w "\n%{http_code}" 200 Ingestion completes without a crash, confirming the worker concurrency as the trigger.

Q: Solution – Increase Stack Size and Limit Worker Concurrency 1. Adjust Docker Compose to Raise ulimit for stack

Set ulimit -s to 64 MiB for the Weaviate container.

Q: 2. Reduce Parallel Workers in the RAG Loader Configuration

Update the loader’s environment variable INGEST_WORKERS to 2 (or 1 for safest operation).

Q: 3. Rebuild and Redeploy

# docker compose down -v # docker compose up -d --build After the changes, the same batch of PDFs processes successfully with no container restarts.

Q: Verify – Confirming the Fix Works 1. Successful Ingestion Log

$ docker compose logs rag-loader -f rag-loader_1 | 2024-10-22T14:45:12Z INFO Starting ingestion: batchSize=100 workers=2 rag-loader_1 | 2024-10-22T14:45:15Z INFO Worker 0 processed 30 documents rag-loader_1 | 2024-10-22T14:45:16Z INFO Worker 1 processed 30 documents rag-loader_1 | 2024-10-22T14:45:18Z INFO All 120 documents indexed successfully 2. Health Check Endpoint # curl -s http://localhost:8080/v1/.well-known/ready {"status":"ready"} 3. Monitoring Metrics Prometheus scrape shows weaviate_ingestion_success_total incremented by 120 and no spikes in process_resident_memory_bytes beyond expected limits.

In this article

Table of Contents

Problem – Segmentation Fault in Weaviate RAG Loader During Batch PDF Ingestion

During staging tests a Weaviate instance (v1.22.0) deployed via docker‑compose crashes with a segmentation fault (SIGSEGV) when the RAG document loader attempts to ingest a batch of large PDF files (each > 30 MB). The failure occurs after the first few documents are indexed and manifests as an immediate container restart, breaking the simulated user query workload.

Root Cause – Native PDF Parsing Library Exhausts Stack in the Multi‑Threaded Worker

The RAG loader uses pdfminer.six (via langchain) to extract text. In the Docker image the library is compiled against libpoppler provided by ubuntu:22.04. When Weaviate processes a batch, it spawns multiple ingestion workers (default --batch-size=100, --workers=4). Each worker loads the native poppler shared library. For PDFs that contain complex object streams, poppler allocates large stack frames. The default Docker container stack size (8 MiB) is insufficient for the concurrent workers, causing the native code to write past the stack and trigger a segmentation fault.

Key points:

Segfault originates in libpoppler.so, not in Python code.
Only appears with large PDFs and when workers > 1.
Weaviate’s batch endpoint forwards the request to the loader without adjusting the container’s ulimit.

Debug – Reproducing and Isolating the Failure

1. Observe Container Crash

$ docker compose logs weaviate
weaviate_1  | 2024-10-22T14:12:31Z INFO  [batch] Received batch of 120 documents
weaviate_1  | 2024-10-22T14:12:31Z INFO  [loader] Starting ingestion workers=4
weaviate_1  | 2024-10-22T14:12:33Z ERROR [loader] Worker 2 crashed: Segmentation fault (core dumped)
weaviate_1  | 2024-10-22T14:12:33Z FATAL  [loader] Process terminated by signal 11
weaviate_1  | 2024-10-22T14:12:33Z INFO  [docker] Container exiting with code 139

2. Core Dump Inspection (host)

# ls -l /var/lib/docker/containers/*/core
/var/lib/docker/containers/abcd1234/core  # core file present

# gdb -c core /usr/local/bin/weaviate
(gdb) bt
#0  0x00007f9c5d2a1b3f in poppler::PDFDoc::open (this=0x7fffdc0000) at PDFDoc.cc:215
#1  0x00007f9c5d2a5e2a in pdfminer::PDFParser::parse (this=0x7fffdc1000) at parser.cpp:87
#2  0x00007f9c5d2b1c10 in ingest_worker (arg=0x7fffdc2000) at loader.py:112
...

3. Verify Stack Size Limitation

# docker exec -it weaviate_1 cat /proc/self/limits | grep "stack size"
Max stack size            8192 kB            unlimited            bytes

4. Reproduce with Single Worker

# curl -X POST http://localhost:8080/v1/batch \
  -H "Content-Type: application/json" \
  -d '{"batchSize":100,"workers":1,"documents":[...large PDFs...]}' -w "\n%{http_code}"
200

Ingestion completes without a crash, confirming the worker concurrency as the trigger.

Solution – Increase Stack Size and Limit Worker Concurrency

1. Adjust Docker Compose to Raise `ulimit` for stack

Set ulimit -s to 64 MiB for the Weaviate container.

# docker-compose.yml (excerpt before)
services:
  weaviate:
    image: semitechnologies/weaviate:1.22.0
    environment:
      - QUERY_DEFAULTS_LIMIT=20
    ports:
      - "8080:8080"
    depends_on:
      - postgres

# docker-compose.yml (excerpt after)
services:
  weaviate:
    image: semitechnologies/weaviate:1.22.0
    ulimits:
      stack: 65536  # 64 MiB
    environment:
      - QUERY_DEFAULTS_LIMIT=20
    ports:
      - "8080:8080"
    depends_on:
      - postgres

2. Reduce Parallel Workers in the RAG Loader Configuration

Update the loader’s environment variable INGEST_WORKERS to 2 (or 1 for safest operation).

# docker-compose.yml (loader service)
services:
  rag-loader:
    image: myorg/rag-loader:latest
    environment:
      - WEAVIATE_URL=http://weaviate:8080
      - INGEST_BATCH_SIZE=100
      - INGEST_WORKERS=2   # reduced from default 4
    depends_on:
      - weaviate

3. Rebuild and Redeploy

# docker compose down -v
# docker compose up -d --build

After the changes, the same batch of PDFs processes successfully with no container restarts.

Verify – Confirming the Fix Works

1. Successful Ingestion Log

$ docker compose logs rag-loader -f
rag-loader_1  | 2024-10-22T14:45:12Z INFO Starting ingestion: batchSize=100 workers=2
rag-loader_1  | 2024-10-22T14:45:15Z INFO Worker 0 processed 30 documents
rag-loader_1  | 2024-10-22T14:45:16Z INFO Worker 1 processed 30 documents
rag-loader_1  | 2024-10-22T14:45:18Z INFO All 120 documents indexed successfully

2. Health Check Endpoint

# curl -s http://localhost:8080/v1/.well-known/ready
{"status":"ready"}

3. Monitoring Metrics

Prometheus scrape shows weaviate_ingestion_success_total incremented by 120 and no spikes in process_resident_memory_bytes beyond expected limits.

Prevent – Operational Guardrails and Best Practices

Set explicit stack limits for any container loading native libraries that may allocate deep recursion (e.g., ulimits.stack ≥ 64 MiB).
Cap ingestion workers based on PDF size distribution; start with workers=1 and increase only after stress testing.
Enable core dump analysis in staging to catch native crashes early (add --security-opt seccomp=unconfined if needed).
Automated regression test that ingests a representative set of >30 MB PDFs and asserts container uptime.
Monitor container exit codes (e.g., alert on exit code 139) to surface segmentation faults promptly.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the segmentation fault only appear with large PDFs?
Large PDFs trigger deeper parsing recursion in poppler, which consumes more stack per worker. The default 8 MiB stack is insufficient, causing overflow.
Can I keep workers=4 and avoid the crash?
Yes, by raising the stack limit (e.g., ulimits.stack: 131072 for 128 MiB) or by using a custom poppler build with reduced recursion depth. However, increasing stack size is the simplest mitigation.
Is the issue specific to the Ubuntu base image?
The stack size default is inherited from the Docker runtime, not the distro. Any base image with the same default will exhibit the same behavior unless ulimit is overridden.
Do I need to rebuild the Weaviate image?
No. Adjusting ulimits in docker‑compose.yml is sufficient because the stack limit is applied at container start time.
How can I detect similar native crashes in production?
Configure a health‑check that watches for container exit code 139, and ship core dumps to a central analysis bucket. Pair this with a Prometheus alert on process_cpu_seconds_total spikes during batch ingestion.