a close-up of a server room

LlamaIndex controller manager crash during model evaluation

by

Problem Description

The LlamaIndex controller manager process crashes with a segmentation fault (SIGSEGV) when an automated evaluation script attempts to load a custom‑trained language model. The failure occurs inside a Docker container used for local development and stops the entire evaluation pipeline.

Typical log excerpt (Docker container stdout):


2026-06-29 14:12:03,212 INFO  controller_manager - Starting model evaluation
2026-06-29 14:12:03,215 DEBUG torch.jit - Loading TorchScript model from /models/custom_llama.pt
2026-06-29 14:12:03,218 ERROR torch.jit - SIGSEGV at 0x00007f8a9c5d0000 in torch::jit::load
2026-06-29 14:12:03,219 FATAL controller_manager - Process terminated by signal 11 (Segmentation fault)

Other observed symptoms include:

  • Docker logs report Failed to allocate shared memory: /dev/shm is too small shortly before the crash.
  • When the same model is evaluated on a host without Docker, the script completes successfully.
  • Switching to a different model (e.g., gpt2) does not trigger the crash.

Root Cause Analysis

The controller manager is responsible for orchestrating model lifecycle events (loading, inference, evaluation) as described in the LlamaIndex Getting Started guide. During model deserialization it calls torch::jit::load, which in turn relies on native C++ extensions and shared memory buffers.

Three overlapping factors commonly trigger the observed SIGSEGV:

  1. Insufficient shared memory (/dev/shm) – The Docker deployment guide recommends --shm-size=1g for large models. When the container defaults to 64 MiB, the model loader fails to allocate the required memory region, leading to a segmentation fault (see real incident where CI pipeline ran out of shared memory).
  2. PyTorch / C++ extension version mismatch – GitHub Issue #913 documents that PyTorch 2.2.0 introduced ABI changes that broke the binary wheels used by LlamaIndex’s controller manager. Loading a model compiled with a newer torch version against an older runtime triggers illegal memory accesses.
  3. Quantization library incompatibility – When the model is quantized with bitsandbytes, the compiled shared objects may be linked against a different libc version. This manifested as intermittent crashes after the first forward pass in production‑grade pipelines.

In the presented scenario the primary trigger is the shared memory limit, compounded by a PyTorch version incompatibility (the container uses torch==2.2.0 while the model was exported with torch==2.3.0).

Investigation and Debugging Steps

Follow this checklist to isolate the failure:

  1. Inspect container resource limits: 
    docker inspect $(docker ps -qf "name=llamaindex_eval") \
  --format='{{json .HostConfig.ShmSize}}'

    Expected output: 1073741824 (1 GiB). If the value is 67108864 (64 MiB), the shared memory size is insufficient.

  2. Verify PyTorch versions inside the container: 
    python -c "import torch, sys; print(torch.__version__); print(sys.version)"

    Compare the printed version with the version used to export the model (check model_info.json if present).

  3. Capture a core dump for post‑mortem analysis (requires --ulimit core=-1): 
    docker run --rm \
  --ulimit core=-1 \
  -e PYTHONFAULTHANDLER=1 \
  -v $(pwd):/app llamaindex_eval

    After the crash, locate core.* in /app and run:

    gdb -c core.* $(which python)
(gdb) bt

    The backtrace typically points to torch::jit::load when /dev/shm allocation fails.

  4. Check CUDA driver compatibility if using GPU: 
    nvidia-smi

    Mismatch between driver and torch CUDA build can surface as torch.cuda.CudaError: device-side assert triggered.

  5. Review model export flags:
    If the model was exported with torch.compile or bitsandbytes, ensure the target container has matching libraries (e.g., bitsandbytes==0.41.1 built for the same GLIBC version).

Resolution

Apply the following changes to eliminate the crash.

1. Increase shared memory allocation

Update the Docker run command or docker-compose.yml to set an explicit shm_size of at least 1 GiB.

# Before
docker run -d --name llamaindex_eval llamaindex_image

# After
docker run -d --name llamaindex_eval \
  --shm-size=2g \
  llamaindex_image

2. Align PyTorch versions

Rebuild the container with the exact PyTorch version used during model export (e.g., torch==2.3.0).

# Dockerfile snippet before
RUN pip install torch==2.2.0

# Dockerfile snippet after
RUN pip install torch==2.3.0

Re‑install any dependent C++ extensions after the version change:

pip uninstall -y llama-index && pip install llama-index

3. Ensure compatible quantization libraries (optional)

If the model uses bitsandbytes, install the matching binary for the container’s CUDA and GLIBC:

# Before
pip install bitsandbytes

# After (specify version built for CUDA 12.1)
pip install bitsandbytes==0.41.1+cu121 -f https://github.com/jllllll/bitsandbytes-wheels/releases

4. Validate container health after changes

Run a short sanity check before the full evaluation:

python - <<'PY'
from llama_index import load_index_from_storage
index = load_index_from_storage("storage/")
print("Index loaded successfully")
PY

Successful output confirms that the controller manager can load the model without crashing.

Verification

Confirm that the issue is resolved using the following methods:

  • Log inspection – No SIGSEGV entries should appear after the model load step.
  • Health endpoint – If the controller manager exposes /healthz, a 200 OK response indicates the process is alive.
  • Metrics – Observe controller_manager_up{status="up"} in Prometheus (if instrumented).
  • Functional test – Execute the full evaluation script and verify that it completes with the expected EvaluationResult JSON.

Prevention and Best Practices

  • Always set --shm-size to ≥ 1 GiB for models larger than 500 MiB. Document this requirement in the Docker deployment guide.
  • Pin the PyTorch version in both model export pipelines and runtime containers. Use a requirements.txt that locks the exact version.
  • When using quantization or TorchScript, rebuild the container after any library upgrade to avoid binary ABI drift.
  • Enable PYTHONFAULTHANDLER=1 and ulimit -c unlimited in CI to capture core dumps for early detection.
  • Add a health check in docker-compose.yml that attempts a lightweight model load; failures will cause container restarts before they affect downstream jobs.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

  1. Why does the controller manager segfault only when running inside Docker?
    Docker containers default to a 64 MiB /dev/shm size. Large model tensors require more shared memory; the allocation fails and the native C++ loader dereferences a null pointer, producing a segmentation fault.
  2. Can I use a smaller shm-size if I disable model quantization?
    Even unquantized models allocate several hundred megabytes of shared memory for the weight tensors. Reducing shm-size below the model’s memory footprint will still cause crashes.
  3. How do I know which PyTorch version was used to export the model?
    Check the model_info.json generated by torch.save or inspect the torch.__version__ recorded in the training script’s logs. Align the runtime container to that exact version.
  4. Is the issue related to CUDA driver mismatches?
    A mismatched driver can surface as torch.cuda.CudaError: device-side assert triggered, which is a different failure mode. The segmentation fault described here occurs before any CUDA call, indicating a CPU‑side memory allocation problem.
  5. What monitoring alerts should I add to catch this early?
    Create an alert on the Prometheus metric process_resident_memory_bytes exceeding 80 % of the container’s memory limit, and on the container exit code = 139 (SIGSEGV). Pair with a log‑based alert for “SIGSEGV” in the controller manager logs.