FAISS node OOM crash during large batch indexing on Kubernetes

Problem – FAISS node OOM crash during large‑batch indexing on Kubernetes

When indexing a high‑dimensional vector dataset in batches of several hundred thousand vectors, the FAISS worker pod is terminated with an OOMKilled status. The failure manifests as a sudden process exit and a Kubernetes event similar to:


Warning  OOMKilled  12s (x3 over 2m)  kubelet, node-1  Container faiss-indexer was killed due to out of memory

Symptoms observed in the application logs include:


2024-10-12 08:15:42,123 ERROR faiss.indexer - Memory allocation failed for batch 7 (size=512000, dim=768)
std::bad_alloc

After the crash, the pod restarts, the indexing job aborts, and downstream services report missing vectors.

Root Cause – Unbounded memory growth during batch indexing

FAISS builds an in‑memory Index object that stores the vectors and auxiliary structures (inverted lists, quantizers, etc.). The memory footprint is roughly:

  • Raw vectors: batch_size × dim × sizeof(float)
  • IVF inverted lists: proportional to nlist × avg_vectors_per_list × sizeof(int)
  • Training structures: temporary buffers that can be as large as the training set.

In the failing deployment:

  • Batch size = 512 000 vectors
  • Dimension = 768
  • FAISS index type = IndexIVFFlat with nlist=4096

Memory required for the raw vectors alone is:


512000 vectors × 768 dims × 4 bytes ≈ 1.5 GiB

Additional structures push the total well beyond the pod’s memory.limit of 2 GiB, causing the Linux OOM killer to terminate the container. The root cause is two‑fold:

  1. Application side: the indexing loop loads an entire batch into a single std::vector<float> before adding it to the index, without streaming or incremental flushing.
  2. Infrastructure side: the Kubernetes pod spec does not reserve sufficient memory for the worst‑case batch, and no oom_score_adj or swap is configured to give the process a chance to fail gracefully.

Debug – Investigating the OOM condition

1. Inspect pod events and status


$ kubectl describe pod faiss-indexer-7c9b9d5f9f-8xk2z
...
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Warning  OOMKilled  2m    kubelet, node-1   Container faiss-indexer was killed due to out of memory

2. Check memory consumption before the kill


$ kubectl top pod faiss-indexer-7c9b9d5f9f-8xk2z
NAME                         CPU(cores)   MEMORY
faiss-indexer-7c9b9d5f9f-8xk2z   850m         1976Mi / 2048Mi

The pod was within the limit, but the kernel OOM killer can act before the cgroup limit is enforced when the process requests memory faster than the kernel can reclaim.

3. Examine kernel OOM logs


$ journalctl -u kubelet | grep -i "Out of memory"
Oct 12 08:15:42 node-1 kernel: [123456.789012] Out of memory: Kill process 12345 (faiss-indexer) score 987 or sacrifice child
Oct 12 08:15:42 node-1 kernel: [123456.789045] Killed process 12345 (faiss-indexer) total-vm:2100000kB, anon-rss:1900000kB, file-rss:0kB

4. Profile FAISS memory usage

FAISS provides a simple API to query the current index size:


size_t index_size = index->sa_code_size() + index->ntotal * dim * sizeof(float);
std::cout << "Current index memory: " << index_size / (1024*1024) << " MiB\n";

Running the above inside the container before the crash printed:


Current index memory: 1856 MiB

Only ~200 MiB remained before the kernel OOM event.

Solution – Controlling memory usage and aligning pod resources

1. Reduce batch size to fit within the pod’s memory budget

Empirically, a batch of 256 000 vectors stays under 1.6 GiB total memory for the same index configuration.

2. Enable FAISS’s use_mmap mode for large training sets

When use_mmap=true, FAISS maps the training vectors from disk instead of loading them fully into RAM.


import faiss, numpy as np

# Load vectors lazily via mmap
train_path = "/data/train_vectors.npy"
train = np.load(train_path, mmap_mode='r')
index = faiss.index_factory(dim, "IVF4096,Flat", faiss.METRIC_INNER_PRODUCT)
index.train(train)   # uses mmap, no full RAM copy

3. Stream batches and flush to disk after each commit

Instead of keeping the whole index in memory, serialize intermediate snapshots:


// Pseudocode for incremental indexing
for (auto& batch : batches) {
    index->add(batch.data, batch.size);
    // Periodically write to disk to free RAM
    if (index->ntotal % 1'000'000 == 0) {
        faiss::write_index(index, "/tmp/index_snapshot.bin");
        delete index;               // free RAM
        index = faiss::read_index("/tmp/index_snapshot.bin");
    }
}

4. Adjust Kubernetes pod spec

Increase the memory request/limit and add a oomScoreAdj to give the container a higher chance to be killed gracefully.

Before After

resources:
  requests:
    cpu: "1"
    memory: "2Gi"
  limits:
    cpu: "2"
    memory: "2Gi"

resources:
  requests:
    cpu: "2"
    memory: "4Gi"
  limits:
    cpu: "4"
    memory: "4Gi"
securityContext:
  privileged: false
  runAsUser: 1000
  oomScoreAdj: 500

5. Enable node‑level swap or memorySwap (if allowed)

On clusters where swap is permissible, set --memory-swap on the container runtime to allow limited swapping instead of immediate OOM kill.

Verify – Confirming that the OOM condition is resolved

  1. Deploy the updated pod spec and monitor the pod’s memory consumption:

$ kubectl apply -f faiss-indexer.yaml
$ watch -n 5 "kubectl top pod -l app=faiss-indexer"

Expected output shows memory staying below the 4 GiB limit, e.g. 2.8 GiB during peak indexing.

  1. Check that the indexing job completes without OOMKilled events:

$ kubectl logs -f job/faiss-index-job
...
2024-10-12 08:42:10 INFO  faiss.indexer - Finished indexing 12,800,000 vectors
2024-10-12 08:42:12 INFO  faiss.indexer - Index written to /data/final.index
  1. Validate the final index size on disk matches expectations:

$ du -h /data/final.index
1.9G    /data/final.index

The index size is consistent with the calculated memory footprint, confirming no hidden memory leaks.

Prevent – Operational safeguards for future batch indexing

  • Resource profiling: Run a one‑off memory benchmark script that simulates the largest expected batch and records max_rss using /usr/bin/time -v.
  • Automated alerts: Create a Prometheus alert on container_memory_working_set_bytes exceeding 80 % of the pod limit for more than 5 minutes.
  • Dynamic batch sizing: Implement a feedback loop that reduces batch_size when memory_usage / limit > 0.75.
  • Horizontal pod autoscaling: Scale the indexing job horizontally and split the dataset into shards; each pod processes a disjoint shard, keeping per‑pod memory bounded.
  • Version pinning: Verify that the FAISS library version (e.g., 1.8.0) does not contain known memory‑leak regressions; upgrade to 1.8.1 where the issue was fixed.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does reducing the batch size fix the OOM without changing the pod memory limit?
    Because FAISS allocates a contiguous block for the entire batch. Smaller batches reduce the peak allocation, keeping it below the cgroup limit and avoiding kernel‑level OOM kills.
  2. Can I use GPU‑accelerated FAISS to avoid OOM?
    GPU indexes still allocate host‑side buffers for the vectors before copying to the device. The host memory requirement remains, so you must still respect the pod’s RAM limits or stream batches.
  3. Is enabling swap advisable in a production Kubernetes cluster?
    Swap can prevent abrupt OOM kills but introduces latency and may affect latency‑sensitive services. Use it only on dedicated nodes where latency impact is acceptable, and limit swap usage with --memory-swap to a small fraction of RAM.
  4. How can I monitor the memory used by the FAISS index itself?
    Expose a custom metric from the indexing process that reports index->ntotal * dim * sizeof(float) and scrape it with Prometheus. Correlate this metric with container memory usage for early warnings.
  5. What if the dataset dimensions change (e.g., from 768 to 1024)?
    Re‑calculate the per‑vector memory (1024 × 4 bytes = 4 KiB). Adjust batch size or pod memory limits accordingly; a 33 % increase in dimension roughly translates to the same increase in RAM consumption.