Weaviate Container Crash Loop After Loading a Large Vector Index in Staging
Problem
The Weaviate pod in the staging Kubernetes cluster repeatedly enters a crash loop. The container is terminated by the kernel with an OOMKilled status as soon as the service attempts to load a newly‑generated vector index that is ~12 GB in size. The issue does not appear in the development environment where the index is ~2 GB.
Observed symptoms
- Pod status:
CrashLoopBackOff - Container termination reason:
OOMKilled - Latency spikes during startup (up to 30 s) before the kill
- No explicit error in Weaviate logs – the process is killed before it can emit a stack trace
Relevant excerpt from kubectl describe pod weaviate-staging-xxxx:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 2m kubelet, node-1 Pulling image "semitechnologies/weaviate:1.19.0"
Normal Pulled 2m kubelet, node-1 Successfully pulled image "semitechnologies/weaviate:1.19.0"
Normal Created 2m kubelet, node-1 Created container weaviate
Normal Started 2m kubelet, node-1 Started container weaviate
Warning OOMKilled 1m kubelet, node-1 Container weaviate was OOMKilled
Root Cause
The staging node is provisioned with 2Gi of RAM for the pod, while the Weaviate process requests a heap size that scales with the size of the vector index. When the index is loaded, the Go runtime allocates a contiguous memory region to hold the hnsw graph and the underlying FAISS index structures. For a 12 GB index, the peak resident set size exceeds the pod limit, triggering the Linux OOM killer.
Key contributing factors:
- Static memory limit in the Deployment manifest – the
resources.limits.memoryfield is set to2Giregardless of index size. - Weaviate’s default
MAX_MEMORY_PERCENTAGE– the server calculates the maximum heap as 80 % of the container’s memory limit. With a 2 Gi limit, the heap is capped at ~1.6 Gi, insufficient for a 12 Gi index. - Staging uses a single replica on a node with limited RAM – unlike production where the pod runs on a node with 16 Gi.
Debug
Step‑by‑step investigation that confirmed the OOM condition:
1. Inspect pod resource limits
kubectl get deployment weaviate -n staging -o yaml | grep -A4 resources
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 1Gi
2. Check container OOM metrics
kubectl top pod weaviate-staging-xxxx -n staging
NAME CPU(cores) MEMORY(bytes)
weaviate-staging-xxxx 0m 0Mi
Memory shows 0 Mi because the container is terminated before any metrics are reported.
3. Capture a core dump of the OOM event
Enable the oom_score_adj and kernel.core_pattern on the node, then reproduce the crash. The resulting core file confirmed a malloc request of ~13 Gi.
4. Verify index size on disk
kubectl exec -it weaviate-staging-xxxx -n staging -- du -sh /var/lib/weaviate/data
12G /var/lib/weaviate/data
5. Review Weaviate startup logs (when OOM does not kill the process)
Running the container locally with increased memory reproduces the loading sequence:
2024-10-01T12:34:56Z INFO weaviate[main] loading vector index from /var/lib/weaviate/data
2024-10-01T12:34:58Z INFO weaviate[main] vector index loaded (12.3GB) in 1.9s
2024-10-01T12:34:58Z INFO weaviate[main] server ready on :8080
The OOM kill occurs between the two INFO lines.
Solution
Two complementary changes resolved the crash loop:
1. Increase pod memory limit
Adjust the Deployment manifest to allocate sufficient RAM for the largest expected index (12 GB + overhead). A safe ceiling is 20 Gi.
Before
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 1Gi
After
resources:
limits:
cpu: "4"
memory: 20Gi
requests:
cpu: "2"
memory: 10Gi
2. Tune Weaviate memory configuration
Set the environment variable MAX_MEMORY_PERCENTAGE to a lower value (e.g., 60 %) so that the heap does not consume the entire container memory, leaving headroom for the OS and other processes.
Before
env:
- name: MAX_MEMORY_PERCENTAGE
value: "80"
After
env:
- name: MAX_MEMORY_PERCENTAGE
value: "60"
3. Deploy a side‑car init container to pre‑warm the index (optional)
If the index size may continue to grow, an init container can pre‑process the data and write a compressed shard, reducing peak memory usage during the main container start‑up.
initContainers:
- name: index-warmup
image: semitechnologies/weaviate-tools:latest
command: ["sh", "-c", "weaviate-index-warmup /data"]
volumeMounts:
- name: weaviate-data
mountPath: /data
Verify
After applying the changes, the pod should start without OOM termination.
1. Confirm pod status
kubectl get pod -l app=weaviate -n staging
NAME READY STATUS RESTARTS AGE
weaviate-staging-abc123 1/1 Running 0 2m
2. Check memory usage
kubectl top pod weaviate-staging-abc123 -n staging
NAME CPU(cores) MEMORY(bytes)
weaviate-staging-abc123 150m 13Gi
The container consumes ~13 Gi, well within the 20 Gi limit.
3. Verify index load time
kubectl logs weaviate-staging-abc123 -n staging | grep "vector index loaded"
2024-10-01T12:45:02Z INFO weaviate[main] vector index loaded (12.3GB) in 1.8s
Prevent
- Capacity planning: Periodically measure index size growth and adjust
resources.limits.memoryaccordingly. - Dynamic memory limits: Use a
HorizontalPodAutoscalerwith custom metrics (e.g.,weaviate_memory_usage_bytes) to scale the pod vertically or horizontally when memory pressure is detected. - Alerting: Set an alert on
container_memory_working_set_bytesapproaching 80 % of the limit. - Staging parity: Mirror production node sizes in staging or use a dedicated high‑memory node pool for vector‑intensive workloads.
- Index sharding: Split very large indexes into multiple shards (Weaviate supports multi‑tenant collections) to keep per‑shard memory footprint below node limits.
Related Topic Hub: Vector Databases Troubleshooting Hub
FAQ
- Why does the same index load fine in production but not in staging?
Production nodes have larger memory allocations (e.g., 32 Gi) and the pod limits are set accordingly. Staging inherits a default 2 Gi limit, which is insufficient for a 12 Gi index. - Can I keep the 2 Gi limit and still load the index?
Only by reducing the index size (e.g., using a lower dimensionality or fewer vectors) or by enabling on‑disk paging in the vector index library, which incurs a performance penalty. - Is
MAX_MEMORY_PERCENTAGEthe only Weaviate memory knob?
No. You can also configureVECTOR_INDEX_MAX_RAM_PERCENTAGEandVECTOR_INDEX_MAX_DISK_PERCENTAGEto fine‑tune how much RAM the HNSW index may consume. - How do I know the exact memory peak required for a given index?
Run a local container with the index and monitorRSSviadocker statsorps -o rss= -p $(pgrep weaviate). The peak RSS after the “vector index loaded” log line is a reliable estimate. - Will increasing the pod limit affect other services on the same node?
If the node has enough free RAM, no. However, on a shared node you must ensure total pod limits do not exceed node capacity, otherwise the kube‑scheduler may evict other pods.