Haystack Query Decomposition Error With FAISS Index On A100 GPU

Q: 3. Check GPU memory consumption before query

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader Sample output on an A100:

Q: 4. Capture a minimal failing request

curl -X POST http://localhost:8000/query \ -H "Content-Type: application/json" \ -d '{"queries": ["What is the capital of France?"]}' \ --max-time 30 Logs (excerpt from haystack.log) show the exception stack trace shown earlier.

Q: 5. Examine NCCL health (relevant for distributed setups)

cat /var/log/nvidia-nccl.log | grep -i error Any “unhandled system error” lines point to communication failures that can abort the query_decomposition step.

Q: 2. Configure FAISS GPU resources to limit pre‑allocation

Use faiss.GpuResources with a custom memory pool.

In this article

Table of Contents

Problem Description

In a distributed Retrieval‑Augmented Generation (RAG) pipeline built with Haystack 1.19.x, the query_decomposition step crashes when the FAISS index is placed on an A100/H100 GPU. Typical error messages observed in the logs are:

Traceback (most recent call last):
  File ".../haystack/pipeline/pipeline.py", line 312, in run
    results = component.run(**inputs)
  File ".../haystack/nodes/retriever/faiss.py", line 421, in run
    query_embeddings = self.embedding_model.embed_queries(queries)
  File ".../haystack/modeling/model.py", line 215, in embed_queries
    embeddings = self.model(**inputs).pooler_output
  File ".../torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: CUDA out of memory. Tried to allocate 2.13 GiB (GPU 0; 40.00 GiB total capacity; 38.50 GiB already allocated; 0.00 GiB free; 38.50 GiB reserved in total)

ValueError: Expected tensor of shape [batch, 768] but got [1, 1024]
    at haystack/modeling/model.py:237
    during cosine_similarity computation between query embedding and FAISS index vectors.

Impact includes:

Pipeline stalls before any passage is retrieved.
Down‑stream generation nodes never receive context, causing empty answers.
High GPU memory pressure leads to cascading OOM failures across the inference cluster.

Root Cause Analysis

Two intertwined factors cause the failure:

Tensor shape mismatch: The BERT retriever model outputs 768‑dimensional embeddings, while the FAISS index was built with 1024‑dimensional vectors after a recent model upgrade (e.g., switching to roberta‑large). Haystack validates that the query embedding dimension matches the index dimension before invoking faiss.Index.search. When they differ, the cosine_similarity helper raises the ValueError shown above. This aligns with the Haystack FAISS integration docs which state that index.d must equal embedding_dim.
GPU memory allocation on A100/H100: Even when dimensions match, the default faiss.GpuIndexFlatIP allocates a contiguous buffer for the entire index. On a 4‑node A100 cluster with 40 GB per GPU, loading a 5 M‑vector index (each 768 float32) exceeds available memory, triggering the CUDA out of memory error. The FAISS GPU documentation notes that GpuIndex uses pinned memory and may require faiss::gpu::StandardGpuResources configuration to limit pre‑allocation.

Both issues are documented in community sources: GitHub issue #2987 (shape mismatch) and the Stack Overflow thread 78543219 (OOM on A100).

Investigation and Debugging

Below is a reproducible debugging workflow that isolates the two failure modes.

1. Verify embedding dimension of the retriever

python - <<'PY'
from haystack.nodes import EmbeddingRetriever
retriever = EmbeddingRetriever(
    model_name_or_path="deepset/bert-base-cased-sentence-transformer"
)
print("Embedding dimension:", retriever.embedding_model.embedding_dim)
PY

Expected output (for a 768‑dim model):

Embedding dimension: 768

2. Inspect FAISS index dimension

python - <<'PY'
import faiss, numpy as np
index = faiss.read_index("faiss_index.faiss")
print("Index dimension:", index.d)
PY

If the output is 1024, the dimensions are inconsistent.

3. Check GPU memory consumption before query

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader

Sample output on an A100:

40 GiB, 38.5 GiB, 0 GiB

4. Capture a minimal failing request

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"queries": ["What is the capital of France?"]}' \
  --max-time 30

Logs (excerpt from haystack.log) show the exception stack trace shown earlier.

5. Examine NCCL health (relevant for distributed setups)

cat /var/log/nvidia-nccl.log | grep -i error

Any “unhandled system error” lines point to communication failures that can abort the query_decomposition step.

Resolution

Fixes address both the shape mismatch and the memory pressure.

1. Align embedding dimensions

Re‑create the FAISS index with the same dimension as the retriever, or downgrade the retriever model.

Before (incorrect index creation):

from haystack.document_stores import FAISSDocumentStore
doc_store = FAISSDocumentStore(
    sql_url="sqlite:///faiss.db",
    embedding_dim=1024,          # ← mismatched dimension
    faiss_index_factory_str="Flat"
)

After (dimension aligned to 768):

from haystack.document_stores import FAISSDocumentStore
doc_store = FAISSDocumentStore(
    sql_url="sqlite:///faiss.db",
    embedding_dim=768,           # matches BERT output
    faiss_index_factory_str="Flat"
)
# Re‑index documents
doc_store.write_documents(docs)
doc_store.update_embeddings(retriever)

2. Configure FAISS GPU resources to limit pre‑allocation

Use faiss.GpuResources with a custom memory pool.

import faiss
res = faiss.StandardGpuResources()
# Limit the temporary memory pool to 8 GiB
res.setTempMemory(8 * 1024 ** 3)

gpu_index = faiss.GpuIndexFlatIP(res,
                                 d=768,
                                 gpu_id=0,
                                 faiss_cfg=faiss.GpuIndexFlatConfig())
# Transfer the CPU index to GPU
cpu_index = faiss.read_index("faiss_index.faiss")
gpu_index.copyFrom(cpu_index)

In Haystack, inject the GPU index via the FAISSDocumentStore constructor:

doc_store = FAISSDocumentStore(
    sql_url="sqlite:///faiss.db",
    embedding_dim=768,
    faiss_index_factory_str="Flat",
    index_path="faiss_index.faiss",
    use_gpu=True,
    gpu_device=0,
    gpu_res=res               # custom resources
)

3. Reduce batch size and enable mixed‑precision safely

If FP16 is required, ensure the retriever outputs FP32 embeddings before FAISS search, as FAISS on GPU does not accept FP16 for inner‑product indexes.

# Disable FP16 for embedding generation
retriever = EmbeddingRetriever(
    model_name_or_path="deepset/bert-base-cased-sentence-transformer",
    use_gpu=True,
    device="cuda",
    embedder_kwargs={"torch_dtype": torch.float32}
)

4. Adjust NCCL environment for multi‑node inference

export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=eth0
export NCCL_NET_GDR_LEVEL=2

These settings mitigate “unhandled system error” messages observed in the distributed inference discussion thread.

Validation

After applying the fixes, run the following checks:

Dimension sanity:

python -c "import faiss; idx=faiss.read_index('faiss_index.faiss'); print(idx.d)"
# Expected: 768

GPU memory footprint (should stay below 30 GiB on a 40 GiB A100):
```
nvidia-smi --query-gpu=memory.used --format=csv,noheader
```

Successful query:

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"queries": ["Explain quantum entanglement."]}'

Expected JSON response contains a non‑empty answers array with retrieved passages.

Health endpoint:
```
curl http://localhost:8000/health
```
Should return {"status":"healthy"} with faiss_index_on_gpu:true in the payload (if exposed).

Operational Experience

During the incident we observed several misleading symptoms:

Initial logs showed only OOM; the underlying shape mismatch was hidden because the OOM occurred during index loading, not during the similarity computation.
Mixed‑precision inference appeared to reduce memory usage, yet FAISS rejected FP16 tensors, leading to the ValueError: Expected tensor of shape [batch, 768] but got [1, 1024] after the index was rebuilt with a larger dimension.
In a 4‑node cluster, one node with a slightly older driver (470 vs 525) caused NCCL handshake failures, which manifested as random query_decomposition crashes. Aligning driver versions resolved the “unhandled system error”.

Best Practices and Prevention

Version pinning: Keep faiss==1.7.4 (or the version compatible with your Haystack release) until the breaking change in 1.8.0 is fully vetted. See the internal incident note about required faiss.GpuIndexFlatIP conversion.
Embedding‑index dimension audit: Automate a CI check that compares retriever.embedding_dim against faiss_index.d after any model upgrade.
GPU memory budgeting: Reserve at most 80 % of GPU memory for FAISS. Use faiss.StandardGpuResources.setTempMemory to enforce the limit.
Batch size control: Configure the retriever’s batch_size (default 16) to a value that fits comfortably within the allocated memory, especially when using large batch inference on A100/H100.
Monitoring: Track torch.cuda.memory_allocated and faiss_gpu_memory_total via Prometheus exporters. Alert when free memory drops below 5 GiB.
Distributed inference hygiene: Ensure homogeneous CUDA driver versions across nodes and set NCCL environment variables as shown above.

Haystack query decomposition error with FAISS index on A100 GPU