Weaviate container crash loop after loading large vector index in staging

Weaviate Container Crash Loop After Loading a Large Vector Index in Staging

Problem

The Weaviate pod in the staging Kubernetes cluster repeatedly enters a crash loop. The container is terminated by the kernel with an OOMKilled status as soon as the service attempts to load a newly‑generated vector index that is ~12 GB in size. The issue does not appear in the development environment where the index is ~2 GB.

Observed symptoms

  • Pod status: CrashLoopBackOff
  • Container termination reason: OOMKilled
  • Latency spikes during startup (up to 30 s) before the kill
  • No explicit error in Weaviate logs – the process is killed before it can emit a stack trace

Relevant excerpt from kubectl describe pod weaviate-staging-xxxx:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Pulling    2m    kubelet, node-1    Pulling image "semitechnologies/weaviate:1.19.0"
  Normal   Pulled     2m    kubelet, node-1    Successfully pulled image "semitechnologies/weaviate:1.19.0"
  Normal   Created    2m    kubelet, node-1    Created container weaviate
  Normal   Started    2m    kubelet, node-1    Started container weaviate
  Warning  OOMKilled  1m    kubelet, node-1    Container weaviate was OOMKilled

Root Cause

The staging node is provisioned with 2Gi of RAM for the pod, while the Weaviate process requests a heap size that scales with the size of the vector index. When the index is loaded, the Go runtime allocates a contiguous memory region to hold the hnsw graph and the underlying FAISS index structures. For a 12 GB index, the peak resident set size exceeds the pod limit, triggering the Linux OOM killer.

Key contributing factors:

  1. Static memory limit in the Deployment manifest – the resources.limits.memory field is set to 2Gi regardless of index size.
  2. Weaviate’s default MAX_MEMORY_PERCENTAGE – the server calculates the maximum heap as 80 % of the container’s memory limit. With a 2 Gi limit, the heap is capped at ~1.6 Gi, insufficient for a 12 Gi index.
  3. Staging uses a single replica on a node with limited RAM – unlike production where the pod runs on a node with 16 Gi.

Debug

Step‑by‑step investigation that confirmed the OOM condition:

1. Inspect pod resource limits

kubectl get deployment weaviate -n staging -o yaml | grep -A4 resources
resources:
  limits:
    cpu: "2"
    memory: 2Gi
  requests:
    cpu: "1"
    memory: 1Gi

2. Check container OOM metrics

kubectl top pod weaviate-staging-xxxx -n staging
NAME                     CPU(cores)   MEMORY(bytes)
weaviate-staging-xxxx    0m           0Mi

Memory shows 0 Mi because the container is terminated before any metrics are reported.

3. Capture a core dump of the OOM event

Enable the oom_score_adj and kernel.core_pattern on the node, then reproduce the crash. The resulting core file confirmed a malloc request of ~13 Gi.

4. Verify index size on disk

kubectl exec -it weaviate-staging-xxxx -n staging -- du -sh /var/lib/weaviate/data
12G	/var/lib/weaviate/data

5. Review Weaviate startup logs (when OOM does not kill the process)

Running the container locally with increased memory reproduces the loading sequence:

2024-10-01T12:34:56Z INFO  weaviate[main] loading vector index from /var/lib/weaviate/data
2024-10-01T12:34:58Z INFO  weaviate[main] vector index loaded (12.3GB) in 1.9s
2024-10-01T12:34:58Z INFO  weaviate[main] server ready on :8080

The OOM kill occurs between the two INFO lines.

Solution

Two complementary changes resolved the crash loop:

1. Increase pod memory limit

Adjust the Deployment manifest to allocate sufficient RAM for the largest expected index (12 GB + overhead). A safe ceiling is 20 Gi.

Before

resources:
  limits:
    cpu: "2"
    memory: 2Gi
  requests:
    cpu: "1"
    memory: 1Gi

After

resources:
  limits:
    cpu: "4"
    memory: 20Gi
  requests:
    cpu: "2"
    memory: 10Gi

2. Tune Weaviate memory configuration

Set the environment variable MAX_MEMORY_PERCENTAGE to a lower value (e.g., 60 %) so that the heap does not consume the entire container memory, leaving headroom for the OS and other processes.

Before

env:
- name: MAX_MEMORY_PERCENTAGE
  value: "80"

After

env:
- name: MAX_MEMORY_PERCENTAGE
  value: "60"

3. Deploy a side‑car init container to pre‑warm the index (optional)

If the index size may continue to grow, an init container can pre‑process the data and write a compressed shard, reducing peak memory usage during the main container start‑up.

initContainers:
- name: index-warmup
  image: semitechnologies/weaviate-tools:latest
  command: ["sh", "-c", "weaviate-index-warmup /data"]
  volumeMounts:
  - name: weaviate-data
    mountPath: /data

Verify

After applying the changes, the pod should start without OOM termination.

1. Confirm pod status

kubectl get pod -l app=weaviate -n staging
NAME                     READY   STATUS    RESTARTS   AGE
weaviate-staging-abc123   1/1     Running   0          2m

2. Check memory usage

kubectl top pod weaviate-staging-abc123 -n staging
NAME                     CPU(cores)   MEMORY(bytes)
weaviate-staging-abc123   150m         13Gi

The container consumes ~13 Gi, well within the 20 Gi limit.

3. Verify index load time

kubectl logs weaviate-staging-abc123 -n staging | grep "vector index loaded"
2024-10-01T12:45:02Z INFO  weaviate[main] vector index loaded (12.3GB) in 1.8s

Prevent

  • Capacity planning: Periodically measure index size growth and adjust resources.limits.memory accordingly.
  • Dynamic memory limits: Use a HorizontalPodAutoscaler with custom metrics (e.g., weaviate_memory_usage_bytes) to scale the pod vertically or horizontally when memory pressure is detected.
  • Alerting: Set an alert on container_memory_working_set_bytes approaching 80 % of the limit.
  • Staging parity: Mirror production node sizes in staging or use a dedicated high‑memory node pool for vector‑intensive workloads.
  • Index sharding: Split very large indexes into multiple shards (Weaviate supports multi‑tenant collections) to keep per‑shard memory footprint below node limits.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the same index load fine in production but not in staging?
    Production nodes have larger memory allocations (e.g., 32 Gi) and the pod limits are set accordingly. Staging inherits a default 2 Gi limit, which is insufficient for a 12 Gi index.
  2. Can I keep the 2 Gi limit and still load the index?
    Only by reducing the index size (e.g., using a lower dimensionality or fewer vectors) or by enabling on‑disk paging in the vector index library, which incurs a performance penalty.
  3. Is MAX_MEMORY_PERCENTAGE the only Weaviate memory knob?
    No. You can also configure VECTOR_INDEX_MAX_RAM_PERCENTAGE and VECTOR_INDEX_MAX_DISK_PERCENTAGE to fine‑tune how much RAM the HNSW index may consume.
  4. How do I know the exact memory peak required for a given index?
    Run a local container with the index and monitor RSS via docker stats or ps -o rss= -p $(pgrep weaviate). The peak RSS after the “vector index loaded” log line is a reliable estimate.
  5. Will increasing the pod limit affect other services on the same node?
    If the node has enough free RAM, no. However, on a shared node you must ensure total pod limits do not exceed node capacity, otherwise the kube‑scheduler may evict other pods.