Weaviate controller manager crashes with segfault during high load inference

Problem – Controller‑manager crashes with segmentation fault under high‑load inference

In a multi‑node Weaviate cluster running on Kubernetes with GPU‑accelerated inference, the controller‑manager pod repeatedly terminates with a SIGSEGV during spikes of concurrent /v1/objects or /v1/graphql queries. The failure manifests as:

Segmentation fault (core dumped)
[signal SIGSEGV: segmentation fault] at 0x7f... in libtorch.so
panic: runtime error: invalid memory address or nil pointer dereference
    goroutine 12 [running]:
    github.com/weaviate/weaviate/modules/text2vec-contextionary/...

Typical operational impact includes:

• API endpoints become unavailable for seconds to minutes.
• Load balancers restart the pod, causing a brief warm‑up latency.
• Downstream services (e.g., batch vectorisation jobs) fail with “controller manager aborts with fatal error”.

Root Cause – Interaction of GPU driver, libtorch ABI, and resource limits

The crash is not a generic Go panic; it originates from the native libtorch library loaded via cgo. Multiple evidence sources point to two tightly coupled failure modes:

1. CUDA / libtorch ABI mismatch – A production GKE cluster (4‑node, NVIDIA A100) experienced SIGSEGV after a spike of ~2000 concurrent queries. The root cause was libtorch 1.13 compiled against CUDA 11.6 while the node driver reported CUDA 11.2 (GitHub #3124). The mismatch triggers illegal memory accesses inside libtorch.so when the inference engine allocates GPU buffers.
2. Insufficient memory & OOM‑induced segfault – Helm charts with the default memoryLimit: 256Mi cause the controller‑manager to be OOM‑killed (GitHub #2987). The kernel then delivers a SIGSEGV when the process later dereferences a GPU buffer that has already been freed.
3. Race between vectoriser and PostgreSQL connection pool – A nightly batch job that refreshes embeddings can race with the request dispatcher, leading to a nil‑pointer dereference in the controller‑manager’s dispatcher (Community Forum).

These conditions violate assumptions documented in the official “Modules & Inference” guide, which states that the controller‑manager expects a compatible CUDA driver (≥ 11.6 for libtorch 1.13) and sufficient pod resources to hold GPU buffers for the duration of a request.

Investigation – Step‑by‑step debugging

1. Capture pod state and logs

# Get the crashing pod name
kubectl get pods -n weaviate -l app=weaviate-controller-manager

# Stream logs with timestamps
kubectl logs -n weaviate weaviate-controller-manager-abcde -c controller-manager \
    --since=5m --timestamps

Typical log excerpt:

2024-06-28T14:12:03.421Z WARN  controller-manager: Failed to allocate GPU buffer: out of memory (CUDA error 2)
2024-06-28T14:12:03.422Z FATAL controller-manager: Segmentation fault (core dumped) at 0x7f3a9c2b4000 in libtorch.so

2. Verify CUDA driver version vs. libtorch ABI

# On a node where the pod was scheduled
kubectl exec -it -n weaviate weaviate-controller-manager-abcde -- nvidia-smi

# Expected output (example)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

# Inside the container, inspect libtorch linkage
kubectl exec -it -n weaviate weaviate-controller-manager-abcde -- \
    ldd /usr/local/lib/libtorch.so | grep cuda
    libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f3a9c2b4000)

If the driver reports CUDA 11.2 while libtorch expects ≥ 11.6, the mismatch is confirmed.

3. Check pod resource limits

kubectl get pod weaviate-controller-manager-abcde -n weaviate -o yaml \
    | grep -A4 resources
      resources:
        limits:
          cpu: "2"
          memory: 256Mi
          nvidia.com/gpu: "1"
        requests:
          cpu: "1"
          memory: 128Mi
          nvidia.com/gpu: "1"

Memory limits of 256 Mi are far below the typical GPU buffer footprint (≈ 1‑2 Gi per inference batch).

4. Examine PostgreSQL connection pool exhaustion

kubectl exec -it -n weaviate weaviate-controller-manager-abcde -- \
    curl -s http://localhost:8080/v1/.well-known/readyz
# Returns 503 if pool exhausted

Log line indicating pool exhaustion:

2024-06-28T14:12:05.001Z ERROR controller-manager: postgres: connection pool exhausted – max open connections reached

5. Core dump analysis (optional)

# Enable core dumps in the pod spec (if not already)
spec:
  securityContext:
    allowPrivilegeEscalation: true
    sysctls:
    - name: kernel.core_pattern
      value: /tmp/core.%e.%p.%t

# After crash, copy core file
kubectl cp weaviate-controller-manager-abcde:/tmp/core.controller-manager.* ./core.dump

# Analyze with gdb
gdb /usr/local/bin/controller-manager ./core.dump -ex "bt" -ex "quit"

The backtrace typically points to at::cuda::CUDAGraph::launch() inside libtorch, confirming the native layer failure.

Solution – Align CUDA stack, increase resources, and serialize vectoriser access

1. Pin compatible CUDA driver and libtorch version

Update the node image or install the required driver version (≥ 11.6 for libtorch 1.13). The official Weaviate Helm chart allows overriding the driver version via values.yaml:

helm upgrade weaviate weaviate/weaviate \
  -f values.yaml \
  --namespace weaviate \
  --set image.tag=v1.23.0

# Generate 2000 concurrent requests using wrk
wrk -t12 -c2000 -d30s -s ./weaviate_infer.lua http://weaviate.example.com/v1/graphql

kubectl get pods -n weaviate -w | grep controller-manager
# No RESTARTS column increment

kubectl logs -n weaviate weaviate-controller-manager-abcde -c controller-manager \
    | grep -i "Segmentation fault"
# No matches

kubectl exec -it -n weaviate weaviate-controller-manager-abcde -- \
    nvidia-smi
# Should show no errors and the correct driver version

curl -s http://weaviate.example.com/v1/.well-known/readyz | grep "postgres"
# Should return "postgres: OK"