FAISS vision token overflow after multi‑GPU PyTorch DDP training

FAISS Vision Token Overflow after Multi‑GPU PyTorch DDP Training

Problem

During a nightly pipeline that trains a vision encoder on 8 × NVIDIA A100 GPUs with torch.nn.parallel.DistributedDataParallel (DDP), the downstream step that builds a FAISS index from the generated image embeddings fails with a token overflow error:

2026-11-13 02:14:37.821 | ERROR | faiss.index_factory: Token overflow: number of vectors (2147483648) exceeds maximum allowed (2^31‑1)

The failure stops the index construction, causing downstream nearest‑neighbor queries to be unavailable for the new model version.

Root Cause

The overflow originates from an unchecked accumulation of embeddings across all DDP ranks. Each GPU processes a shard of the dataset and streams its float32 embeddings to a central faiss.IndexFlatL2 instance via a shared memory queue. The queue implementation concatenates the incoming batches without resetting the internal ntotal counter after each training run. When the cumulative ntotal exceeds 2^31‑1 (the maximum 32‑bit signed integer used by FAISS for vector counts), the library throws the token overflow exception.

Two contributing factors amplify the problem:

  • Persistent index objects: The training script reuses the same FAISS index object across multiple epochs without calling reset() or recreating it.
  • Mixed‑precision accumulation: Embeddings are stored as torch.float16 on the GPU, cast to float32 only when enqueued. The cast does not reduce the number of vectors, only their memory footprint, so the overflow condition remains.

Debug

Step‑by‑step investigation that reproduced the overflow:

  1. Inspect FAISS index statistics after each epoch.
>>> index.ntotal
2147483647  # after epoch 7, just below the limit
>>> index.ntotal
2147483648  # epoch 8 triggers overflow
  1. Check the size of the shared memory queue.
$ ls -lh /dev/shm/faiss_queue*
total 0
-rw------- 1 user user 8.0G Sep 13 02:10 faiss_queue_0
-rw------- 1 user user 8.0G Sep 13 02:10 faiss_queue_1
...
  1. Verify that the index object is not reset.
# In the training script
print(f"Rank {rank}: index.ntotal before training = {index.ntotal}")
# Output shows non‑zero value from previous run
Rank 3: index.ntotal before training = 2147483640
  1. Capture the exact exception stack trace.
Traceback (most recent call last):
  File "build_faiss.py", line 112, in 
    index.add(batch_embeddings)
  File "/usr/local/lib/python3.8/site-packages/faiss/__init__.py", line 254, in add
    self._add_vectors(np.ascontiguousarray(vectors, dtype=np.float32))
faiss.swigfaiss.IndexError: token overflow: number of vectors (2147483648) exceeds maximum allowed (2^31-1)

Solution

The fix consists of three coordinated changes:

1. Explicit index reset at the start of each training run

# Before
index = faiss.index_factory(d, "Flat", faiss.METRIC_L2)

# After – add a reset guard
def get_faiss_index(dim):
    idx = faiss.index_factory(dim, "Flat", faiss.METRIC_L2)
    idx.reset()               # ensures ntotal = 0
    return idx

# In the main entry point
index = get_faiss_index(embedding_dim)

2. Use 64‑bit counters by enabling the FAISS FAISS_ENABLE_GPU flag and constructing a IndexIVFFlat with faiss.IndexIVFFlat that supports int64 IDs.

# Before (32‑bit only)
index = faiss.IndexFlatL2(embedding_dim)

# After – switch to IVF with 64‑bit IDs
quantizer = faiss.IndexFlatL2(embedding_dim)
index = faiss.IndexIVFFlat(quantizer, embedding_dim, 4096, faiss.METRIC_L2)
index.train(embedding_dataset)   # train once per model version
index.nprobe = 10
# Enable 64‑bit IDs
faiss.ParameterSpace().set_index_parameter(index, "use_int64_ids", True)

3. Clear the shared memory queue after each epoch to avoid stale buffers.

# queue_cleanup.py
import os, glob, shutil

def clean_faiss_queues():
    for path in glob.glob("/dev/shm/faiss_queue_*"):
        try:
            os.remove(path)
        except OSError as e:
            print(f"Failed to remove {path}: {e}")

if __name__ == "__main__":
    clean_faiss_queues()

Invoke clean_faiss_queues() from the training driver after index.add() completes.

4. Guard against accidental reuse in DDP workers

# In the DDP training loop (each rank)
if rank == 0:
    # Only rank 0 owns the FAISS index
    index = get_faiss_index(embedding_dim)
else:
    index = None

# Broadcast the index handle (or use a manager) after reset
torch.distributed.barrier()

Verify

After applying the changes, run the full training‑to‑index pipeline on a subset of the data and confirm that the overflow no longer occurs.

$ python train_and_index.py --epochs 10 --batch-size 256
...
[Epoch 10] Added 2,560,000 vectors – index.ntotal = 2,560,000
Index construction completed in 3m12s
FAISS index saved to /models/faiss/v2.index

Additional verification steps:

  • Check index.ntotal after the final epoch – it should match the expected number of images.
  • Run a sanity nearest‑neighbor query:
>> D, I = index.search(query_embedding, k=5)
>>> print(I)
[[ 12345  67890  23456  98765  34567]]

Successful retrieval indicates that the index is functional.

Prevent

To avoid recurrence in future training cycles:

Preventive Action Why it Helps
Always call index.reset() before reusing an index object. Guarantees ntotal starts at zero, preventing hidden accumulation.
Prefer FAISS indexes that support 64‑bit IDs (e.g., IndexIVFFlat with use_int64_ids). Eliminates the 2³¹‑1 hard limit.
Encapsulate queue cleanup in a post‑epoch hook. Prevents stale shared‑memory files from inflating the perceived vector count.
Instrument a metric for index.ntotal and alert when > 90 % of 2^31‑1. Early detection before the exception is raised.
Document the ownership model of the FAISS index in DDP (rank 0 only). Reduces accidental concurrent modifications.

FAQ

  1. Why does the overflow happen only when using DDP and not during single‑GPU runs?
    Because DDP aggregates embeddings from multiple ranks into a single index without resetting the counter. A single GPU processes fewer batches, staying below the 2³¹‑1 threshold.
  2. Can I keep using IndexFlatL2 if I enable 64‑bit IDs?
    No. IndexFlatL2 is hard‑coded to 32‑bit counters. Switching to an IVF‑based index is required for 64‑bit support.
  3. Is there a performance penalty for using IVF instead of Flat?
    IVF introduces a quantization step but typically improves query latency for large collections. Training time increases modestly during the train() phase, which is a one‑time cost per model version.
  4. How can I monitor index.ntotal in production?
    Expose index.ntotal via a custom Prometheus gauge in the indexing service, e.g.:

    faiss_index_size.set(index.ntotal)

    Set an alert at 80 % of 2^31‑1.

  5. What if I need to index more than 2³¹‑1 vectors?
    FAISS supports 64‑bit IDs only on GPU or IVF indexes. For > 2³¹ vectors, use IndexIVFPQ or IndexIVFSQ with use_int64_ids=True, and store the index on disk with faiss.write_index to avoid memory pressure.

Related Topic Hub: Vector Databases Troubleshooting Hub