FAISS Vision Token Overflow after Multi‑GPU PyTorch DDP Training
Problem
During a nightly pipeline that trains a vision encoder on 8 × NVIDIA A100 GPUs with torch.nn.parallel.DistributedDataParallel (DDP), the downstream step that builds a FAISS index from the generated image embeddings fails with a token overflow error:
2026-11-13 02:14:37.821 | ERROR | faiss.index_factory: Token overflow: number of vectors (2147483648) exceeds maximum allowed (2^31‑1)
The failure stops the index construction, causing downstream nearest‑neighbor queries to be unavailable for the new model version.
Root Cause
The overflow originates from an unchecked accumulation of embeddings across all DDP ranks. Each GPU processes a shard of the dataset and streams its float32 embeddings to a central faiss.IndexFlatL2 instance via a shared memory queue. The queue implementation concatenates the incoming batches without resetting the internal ntotal counter after each training run. When the cumulative ntotal exceeds 2^31‑1 (the maximum 32‑bit signed integer used by FAISS for vector counts), the library throws the token overflow exception.
Two contributing factors amplify the problem:
- Persistent index objects: The training script reuses the same FAISS index object across multiple epochs without calling
reset()or recreating it. - Mixed‑precision accumulation: Embeddings are stored as
torch.float16on the GPU, cast tofloat32only when enqueued. The cast does not reduce the number of vectors, only their memory footprint, so the overflow condition remains.
Debug
Step‑by‑step investigation that reproduced the overflow:
- Inspect FAISS index statistics after each epoch.
>>> index.ntotal
2147483647 # after epoch 7, just below the limit
>>> index.ntotal
2147483648 # epoch 8 triggers overflow
- Check the size of the shared memory queue.
$ ls -lh /dev/shm/faiss_queue*
total 0
-rw------- 1 user user 8.0G Sep 13 02:10 faiss_queue_0
-rw------- 1 user user 8.0G Sep 13 02:10 faiss_queue_1
...
- Verify that the index object is not reset.
# In the training script
print(f"Rank {rank}: index.ntotal before training = {index.ntotal}")
# Output shows non‑zero value from previous run
Rank 3: index.ntotal before training = 2147483640
- Capture the exact exception stack trace.
Traceback (most recent call last):
File "build_faiss.py", line 112, in
index.add(batch_embeddings)
File "/usr/local/lib/python3.8/site-packages/faiss/__init__.py", line 254, in add
self._add_vectors(np.ascontiguousarray(vectors, dtype=np.float32))
faiss.swigfaiss.IndexError: token overflow: number of vectors (2147483648) exceeds maximum allowed (2^31-1)
Solution
The fix consists of three coordinated changes:
1. Explicit index reset at the start of each training run
# Before
index = faiss.index_factory(d, "Flat", faiss.METRIC_L2)
# After – add a reset guard
def get_faiss_index(dim):
idx = faiss.index_factory(dim, "Flat", faiss.METRIC_L2)
idx.reset() # ensures ntotal = 0
return idx
# In the main entry point
index = get_faiss_index(embedding_dim)
2. Use 64‑bit counters by enabling the FAISS FAISS_ENABLE_GPU flag and constructing a IndexIVFFlat with faiss.IndexIVFFlat that supports int64 IDs.
# Before (32‑bit only)
index = faiss.IndexFlatL2(embedding_dim)
# After – switch to IVF with 64‑bit IDs
quantizer = faiss.IndexFlatL2(embedding_dim)
index = faiss.IndexIVFFlat(quantizer, embedding_dim, 4096, faiss.METRIC_L2)
index.train(embedding_dataset) # train once per model version
index.nprobe = 10
# Enable 64‑bit IDs
faiss.ParameterSpace().set_index_parameter(index, "use_int64_ids", True)
3. Clear the shared memory queue after each epoch to avoid stale buffers.
# queue_cleanup.py
import os, glob, shutil
def clean_faiss_queues():
for path in glob.glob("/dev/shm/faiss_queue_*"):
try:
os.remove(path)
except OSError as e:
print(f"Failed to remove {path}: {e}")
if __name__ == "__main__":
clean_faiss_queues()
Invoke clean_faiss_queues() from the training driver after index.add() completes.
4. Guard against accidental reuse in DDP workers
# In the DDP training loop (each rank)
if rank == 0:
# Only rank 0 owns the FAISS index
index = get_faiss_index(embedding_dim)
else:
index = None
# Broadcast the index handle (or use a manager) after reset
torch.distributed.barrier()
Verify
After applying the changes, run the full training‑to‑index pipeline on a subset of the data and confirm that the overflow no longer occurs.
$ python train_and_index.py --epochs 10 --batch-size 256
...
[Epoch 10] Added 2,560,000 vectors – index.ntotal = 2,560,000
Index construction completed in 3m12s
FAISS index saved to /models/faiss/v2.index
Additional verification steps:
- Check
index.ntotalafter the final epoch – it should match the expected number of images. - Run a sanity nearest‑neighbor query:
>> D, I = index.search(query_embedding, k=5)
>>> print(I)
[[ 12345 67890 23456 98765 34567]]
Successful retrieval indicates that the index is functional.
Prevent
To avoid recurrence in future training cycles:
| Preventive Action | Why it Helps |
|---|---|
Always call index.reset() before reusing an index object. |
Guarantees ntotal starts at zero, preventing hidden accumulation. |
Prefer FAISS indexes that support 64‑bit IDs (e.g., IndexIVFFlat with use_int64_ids). |
Eliminates the 2³¹‑1 hard limit. |
| Encapsulate queue cleanup in a post‑epoch hook. | Prevents stale shared‑memory files from inflating the perceived vector count. |
Instrument a metric for index.ntotal and alert when > 90 % of 2^31‑1. |
Early detection before the exception is raised. |
| Document the ownership model of the FAISS index in DDP (rank 0 only). | Reduces accidental concurrent modifications. |
FAQ
- Why does the overflow happen only when using DDP and not during single‑GPU runs?
Because DDP aggregates embeddings from multiple ranks into a single index without resetting the counter. A single GPU processes fewer batches, staying below the 2³¹‑1 threshold. - Can I keep using
IndexFlatL2if I enable 64‑bit IDs?
No.IndexFlatL2is hard‑coded to 32‑bit counters. Switching to an IVF‑based index is required for 64‑bit support. - Is there a performance penalty for using IVF instead of Flat?
IVF introduces a quantization step but typically improves query latency for large collections. Training time increases modestly during thetrain()phase, which is a one‑time cost per model version. - How can I monitor
index.ntotalin production?
Exposeindex.ntotalvia a custom Prometheus gauge in the indexing service, e.g.:faiss_index_size.set(index.ntotal)Set an alert at 80 % of
2^31‑1. - What if I need to index more than 2³¹‑1 vectors?
FAISS supports 64‑bit IDs only on GPU or IVF indexes. For > 2³¹ vectors, useIndexIVFPQorIndexIVFSQwithuse_int64_ids=True, and store the index on disk withfaiss.write_indexto avoid memory pressure.
Related Topic Hub: Vector Databases Troubleshooting Hub