Problem – FAISS node OOM crash during large‑batch indexing on Kubernetes
When indexing a high‑dimensional vector dataset in batches of several hundred thousand vectors, the FAISS worker pod is terminated with an OOMKilled status. The failure manifests as a sudden process exit and a Kubernetes event similar to:
Warning OOMKilled 12s (x3 over 2m) kubelet, node-1 Container faiss-indexer was killed due to out of memory
Symptoms observed in the application logs include:
2024-10-12 08:15:42,123 ERROR faiss.indexer - Memory allocation failed for batch 7 (size=512000, dim=768)
std::bad_alloc
After the crash, the pod restarts, the indexing job aborts, and downstream services report missing vectors.
Root Cause – Unbounded memory growth during batch indexing
FAISS builds an in‑memory Index object that stores the vectors and auxiliary structures (inverted lists, quantizers, etc.). The memory footprint is roughly:
- Raw vectors:
batch_size × dim × sizeof(float) - IVF inverted lists: proportional to
nlist × avg_vectors_per_list × sizeof(int) - Training structures: temporary buffers that can be as large as the training set.
In the failing deployment:
- Batch size = 512 000 vectors
- Dimension = 768
- FAISS index type =
IndexIVFFlatwithnlist=4096
Memory required for the raw vectors alone is:
512000 vectors × 768 dims × 4 bytes ≈ 1.5 GiB
Additional structures push the total well beyond the pod’s memory.limit of 2 GiB, causing the Linux OOM killer to terminate the container. The root cause is two‑fold:
- Application side: the indexing loop loads an entire batch into a single
std::vector<float>before adding it to the index, without streaming or incremental flushing. - Infrastructure side: the Kubernetes pod spec does not reserve sufficient memory for the worst‑case batch, and no
oom_score_adjor swap is configured to give the process a chance to fail gracefully.
Debug – Investigating the OOM condition
1. Inspect pod events and status
$ kubectl describe pod faiss-indexer-7c9b9d5f9f-8xk2z
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning OOMKilled 2m kubelet, node-1 Container faiss-indexer was killed due to out of memory
2. Check memory consumption before the kill
$ kubectl top pod faiss-indexer-7c9b9d5f9f-8xk2z
NAME CPU(cores) MEMORY
faiss-indexer-7c9b9d5f9f-8xk2z 850m 1976Mi / 2048Mi
The pod was within the limit, but the kernel OOM killer can act before the cgroup limit is enforced when the process requests memory faster than the kernel can reclaim.
3. Examine kernel OOM logs
$ journalctl -u kubelet | grep -i "Out of memory"
Oct 12 08:15:42 node-1 kernel: [123456.789012] Out of memory: Kill process 12345 (faiss-indexer) score 987 or sacrifice child
Oct 12 08:15:42 node-1 kernel: [123456.789045] Killed process 12345 (faiss-indexer) total-vm:2100000kB, anon-rss:1900000kB, file-rss:0kB
4. Profile FAISS memory usage
FAISS provides a simple API to query the current index size:
size_t index_size = index->sa_code_size() + index->ntotal * dim * sizeof(float);
std::cout << "Current index memory: " << index_size / (1024*1024) << " MiB\n";
Running the above inside the container before the crash printed:
Current index memory: 1856 MiB
Only ~200 MiB remained before the kernel OOM event.
Solution – Controlling memory usage and aligning pod resources
1. Reduce batch size to fit within the pod’s memory budget
Empirically, a batch of 256 000 vectors stays under 1.6 GiB total memory for the same index configuration.
2. Enable FAISS’s use_mmap mode for large training sets
When use_mmap=true, FAISS maps the training vectors from disk instead of loading them fully into RAM.
import faiss, numpy as np
# Load vectors lazily via mmap
train_path = "/data/train_vectors.npy"
train = np.load(train_path, mmap_mode='r')
index = faiss.index_factory(dim, "IVF4096,Flat", faiss.METRIC_INNER_PRODUCT)
index.train(train) # uses mmap, no full RAM copy
3. Stream batches and flush to disk after each commit
Instead of keeping the whole index in memory, serialize intermediate snapshots:
// Pseudocode for incremental indexing
for (auto& batch : batches) {
index->add(batch.data, batch.size);
// Periodically write to disk to free RAM
if (index->ntotal % 1'000'000 == 0) {
faiss::write_index(index, "/tmp/index_snapshot.bin");
delete index; // free RAM
index = faiss::read_index("/tmp/index_snapshot.bin");
}
}
4. Adjust Kubernetes pod spec
Increase the memory request/limit and add a oomScoreAdj to give the container a higher chance to be killed gracefully.
| Before | After |
|---|---|
|
|
5. Enable node‑level swap or memorySwap (if allowed)
On clusters where swap is permissible, set --memory-swap on the container runtime to allow limited swapping instead of immediate OOM kill.
Verify – Confirming that the OOM condition is resolved
- Deploy the updated pod spec and monitor the pod’s memory consumption:
$ kubectl apply -f faiss-indexer.yaml
$ watch -n 5 "kubectl top pod -l app=faiss-indexer"
Expected output shows memory staying below the 4 GiB limit, e.g. 2.8 GiB during peak indexing.
- Check that the indexing job completes without
OOMKilledevents:
$ kubectl logs -f job/faiss-index-job
...
2024-10-12 08:42:10 INFO faiss.indexer - Finished indexing 12,800,000 vectors
2024-10-12 08:42:12 INFO faiss.indexer - Index written to /data/final.index
- Validate the final index size on disk matches expectations:
$ du -h /data/final.index
1.9G /data/final.index
The index size is consistent with the calculated memory footprint, confirming no hidden memory leaks.
Prevent – Operational safeguards for future batch indexing
- Resource profiling: Run a one‑off memory benchmark script that simulates the largest expected batch and records
max_rssusing/usr/bin/time -v. - Automated alerts: Create a Prometheus alert on
container_memory_working_set_bytesexceeding 80 % of the pod limit for more than 5 minutes. - Dynamic batch sizing: Implement a feedback loop that reduces
batch_sizewhenmemory_usage / limit > 0.75. - Horizontal pod autoscaling: Scale the indexing job horizontally and split the dataset into shards; each pod processes a disjoint shard, keeping per‑pod memory bounded.
- Version pinning: Verify that the FAISS library version (e.g., 1.8.0) does not contain known memory‑leak regressions; upgrade to 1.8.1 where the issue was fixed.
Related Topic Hub: Vector Databases Troubleshooting Hub
FAQ
- Why does reducing the batch size fix the OOM without changing the pod memory limit?
Because FAISS allocates a contiguous block for the entire batch. Smaller batches reduce the peak allocation, keeping it below the cgroup limit and avoiding kernel‑level OOM kills. - Can I use GPU‑accelerated FAISS to avoid OOM?
GPU indexes still allocate host‑side buffers for the vectors before copying to the device. The host memory requirement remains, so you must still respect the pod’s RAM limits or stream batches. - Is enabling swap advisable in a production Kubernetes cluster?
Swap can prevent abrupt OOM kills but introduces latency and may affect latency‑sensitive services. Use it only on dedicated nodes where latency impact is acceptable, and limit swap usage with--memory-swapto a small fraction of RAM. - How can I monitor the memory used by the FAISS index itself?
Expose a custom metric from the indexing process that reportsindex->ntotal * dim * sizeof(float)and scrape it with Prometheus. Correlate this metric with container memory usage for early warnings. - What if the dataset dimensions change (e.g., from 768 to 1024)?
Re‑calculate the per‑vector memory (1024 × 4 bytes = 4 KiB). Adjust batch size or pod memory limits accordingly; a 33 % increase in dimension roughly translates to the same increase in RAM consumption.