Hugging Face RAG retrieval empty results on A100 GPU cluster

Problem Description – Empty Retrieval Results on A100/H100 GPU Cluster

When executing a Retrieval‑Augmented Generation (RAG) pipeline on a multi‑node GPU cluster (A100/H100), the RAGRetriever consistently returns an empty document list despite:

• Valid natural‑language queries.
• A fully populated FAISS index built from the knowledge base.
• Successful generation of query embeddings on the GPU.

Typical log output on each worker:

[2026-07-17 12:03:45] [RAGRetriever] No documents retrieved for query "What is the latency of the new API?"

In addition, occasional errors appear:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu
FAISS error: "Index is not trained"

The issue manifests only when the RAG model runs under torch.distributed.DistributedDataParallel (DDP) across multiple nodes.

Root Cause Analysis

The empty results stem from a mismatch between the location and state of the FAISS index and the distributed execution context. Three inter‑related factors are identified from the evidence package:

1. Index placement on a single rank – In the production incident on an 8‑node A100 cluster, the FAISS index was built and loaded only on rank 0. Other ranks queried an uninitialized index, which by default contains no vectors, leading to the log message above.
2. Missing GPU transfer per rank – FAISS indexes created on CPU must be explicitly moved to each GPU with faiss.index_cpu_to_gpu. A Stack Overflow discussion shows that omitting this call on every rank results in a CPU‑only index that cannot accept GPU query embeddings, triggering the “Expected all tensors to be on the same device” runtime error.
3. Improper barrier synchronization – A GitHub issue reports that placing torch.distributed.barrier() after the retrieval step allows some workers to query before the index is fully loaded and transferred, causing intermittent empty results.

Combined, these factors cause the retriever to see an empty or incorrectly located index, so retriever.retrieve() returns an empty list.

Investigation and Debugging

Below is a step‑by‑step debugging workflow that reproduces the failure and isolates the cause.

1. Verify index existence on each rank

import torch
import faiss, os

rank = torch.distributed.get_rank()
index_path = "/shared/faiss_index.bin"

if os.path.exists(index_path):
    index = faiss.read_index(index_path)
    print(f"Rank {rank}: Index loaded, nb_vectors={index.ntotal}")
else:
    print(f"Rank {rank}: Index file missing")

Typical output on rank 1 (failed case):

Rank 1: Index loaded, nb_vectors=0

2. Check device placement of the index

print(f"Rank {rank}: Index is on GPU? {faiss.get_num_gpus() > 0}")

If the result is False, the index is still on CPU while query embeddings are on GPU.

3. Inspect the query embedding tensor

query_emb = model.question_encoder(**inputs).pooler_output
print(query_emb.device)

Expected output:

cuda:0

4. Capture a short packet trace (optional)

sudo tcpdump -i eth0 -w rag_trace.pcap port 443 &
# Run a single query, then stop capture
sudo pkill -INT tcpdump

Ensures that network traffic reaches the FAISS service when using a remote index server.

5. Confirm barrier ordering

# Incorrect ordering (common bug)
retrieved = retriever.retrieve(query)
torch.distributed.barrier()
# Correct ordering
torch.distributed.barrier()
retrieved = retriever.retrieve(query)

Resolution – Correct Index Distribution and Synchronization

The fix consists of three coordinated changes:

1. Build the index once on a designated rank and broadcast it to all ranks

# rank 0 builds the index
if torch.distributed.get_rank() == 0:
    index = faiss.IndexFlatIP(d)                     # d = embedding dimension
    index.add(knowledge_embeddings)                 # knowledge_embeddings is a numpy array
    faiss.write_index(index, "/shared/faiss_index.bin")

# Ensure all ranks see the file
torch.distributed.barrier()

2. Load and move the index to the local GPU on every rank

import faiss
import torch

rank = torch.distributed.get_rank()
device = torch.device(f"cuda:{rank % torch.cuda.device_count()}")

# Load CPU index
cpu_index = faiss.read_index("/shared/faiss_index.bin")
print(f"Rank {rank}: CPU index nb_vectors={cpu_index.ntotal}")

# Transfer to GPU
gpu_res = faiss.StandardGpuResources()
gpu_index = faiss.index_cpu_to_gpu(gpu_res, device.index, cpu_index)
print(f"Rank {rank}: GPU index nb_vectors={gpu_index.ntotal}")

3. Synchronize after the GPU transfer and before any retrieval

torch.distributed.barrier()   # All ranks have a valid GPU index now
retrieved = retriever.retrieve(query)

Before vs. After Comparison

Aspect	Before Fix	After Fix
Index location	CPU on rank 0 only	GPU on every rank
Barrier placement	After retrieval	Before retrieval
Retrieval result	Empty list	Non‑empty document list
Log output	“No documents retrieved”	“Retrieved 5 documents”

Validation – Confirming Correct Operation

1. Run a single‑node sanity test:

python run_rag.py --query "Explain the RAG architecture"
# Expected log snippet:
[RAGRetriever] Retrieved 5 documents for query

2. Execute the distributed job and inspect each rank’s logs for the same message.
3. Metric verification (optional):

# Prometheus metric example
rag_retrieval_success_total{job="rag_service"} 8

4. Functional test via curl against the inference endpoint:

curl -X POST http://cluster-endpoint/v1/rag -d '{"query":"What is the latency of the new API?"}'
# Response should contain a "context" field with retrieved passages.

Operational Experience – Lessons Learned

• Misleading symptom: The “RuntimeError: Expected all tensors to be on the same device” often appears only after a few batches, making it easy to attribute the failure to CUDA memory pressure rather than index placement.
• Common incorrect assumption: Developers assume that faiss.read_index automatically respects the current CUDA device. In reality, the index remains on CPU until faiss.index_cpu_to_gpu is invoked per process.
• Production edge case: When using mixed‑precision (fp16) embeddings, the index dimension must match exactly; otherwise, FAISS silently drops vectors, yielding an empty index on some ranks.
• Barrier timing: Placing torch.distributed.barrier() after retrieval caused intermittent empty results in a real incident because rank 0 finished loading the index earlier than the others.

Best Practices and Prevention

• Always broadcast the FAISS index file after it is written and call torch.distributed.barrier() before any rank accesses it.
• Wrap the GPU transfer in a helper that runs on every rank:

def load_faiss_gpu(index_path, device_id):
    cpu = faiss.read_index(index_path)
    res = faiss.StandardGpuResources()
    return faiss.index_cpu_to_gpu(res, device_id, cpu)

• Validate index.is_trained and index.ntotal on each rank after loading; abort early if the index is empty.
• Instrument a health check endpoint that returns the number of vectors in the in‑memory index; alert if ntotal == 0.
• When scaling to more than one GPU per node, create a separate StandardGpuResources per device and shard the index accordingly (see FAISS GPU documentation).

Related Questions (FAQ)

1. Why does the RAG retriever work on a single GPU but return empty results in a multi‑node DDP run?
   Because the FAISS index is only loaded on the rank that built it. Other ranks query an empty CPU index, leading to no documents.
2. How can I verify that the FAISS index is correctly placed on each GPU?
   Print faiss.get_num_gpus() and gpu_index.ntotal after index_cpu_to_gpu. Both should be >0 on every rank.
3. What is the correct order of torch.distributed.barrier() when initializing the retriever?
   Place the barrier after the index has been transferred to GPU on all ranks and before any call to retriever.retrieve().
4. Can mixed‑precision embeddings cause the index to appear empty?
   Yes. If the embedding dimension of the query (e.g., fp16) does not match the index dimension (fp32), FAISS will reject the vectors, resulting in zero matches.
5. Is it necessary to call faiss.index_cpu_to_gpu on each rank even when the index file resides on a shared filesystem?
   Absolutely. The index file is stored on CPU; each process must create its own GPU copy. Sharing the GPU pointer across processes is not supported.

Related Topic Hub: Model Serving Troubleshooting Hub