ONNX Runtime RAG Retrieval Empty Results In Development Sandbox

Q: Investigation and Debugging Steps 1. Verify Model Outputs Directly

import onnxruntime as ort import numpy as np session = ort.InferenceSession("minilm_v2.onnx") tokenizer = ... # same tokenizer used during index build query = "What is the capital of France?" inputs = tokenizer(query, return_tensors="np", padding="max_length", truncation=True) # Inspect input shapes print("input_ids shape:", inputs["input_ids"].shape) print("attention_mask shape:", inputs["attention_mask"].shape) # Run inference outputs = session.run(None, inputs) embedding = outputs[0] print("Embedding stats: mean=", embedding.mean(), "std=", embedding.std()) print("Embedding dtype:", embedding.dtype) Expected output (non‑zero, float32):

Q: 2. Check Vector Store Index Precision

import faiss index = faiss.read_index("doc_embeddings.index") print("Index d:", index.d) # dimensionality print("Index is trained:", index.is_trained) print("Index metric type:", index.metric_type) print("Index vector dtype:", index.dtype) # usually float32 If the index reports dtype=FLOAT32 while the embedding is float16, similarity will fail.

Q: 3. Examine Export Command

from transformers import AutoModel, AutoTokenizer import torch model_name = "sentence-transformers/all-MiniLM-L6-v2" model = AutoModel.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) dummy_input = tokenizer("dummy", return_tensors="pt") torch.onnx.export( model, (dummy_input["input_ids"], dummy_input["attention_mask"]), "minilm_v2.onnx", input_names=["input_ids", "attention_mask"], output_names=["last_hidden_state"], dynamic_axes={ "input_ids": {0: "batch", 1: "seq"}, "attention_mask": {0: "batch", 1: "seq"}, "last_hidden_state": {0: "batch", 1: "seq"}, }, opset_version=14, do_constant_folding=True, ) Key points derived from the official ONNX Runtime custom‑ops documentation:

Q: Step 2: Align Embedding Precision with the Index

If the index is float32, force the ONNX Runtime session to output float32:

In this article

Table of Contents

Problem – Empty Retrieval Results in a Development Sandbox Using ONNX Runtime for RAG

In a sandbox environment the Retrieval‑Augmented Generation (RAG) pipeline is wired to an ONNX Runtime session that hosts a converted MiniLM‑v2 embedding model. When a user query is sent to the retriever the downstream FAISS (or similar) vector store returns an empty list, even though the same model works when executed with the original PyTorch implementation.

Typical symptoms observed in the logs:


[2026-06-28 10:12:04] INFO  onnxruntime: Session created (provider=CPUExecutionProvider)
[2026-06-28 10:12:05] DEBUG retriever: Query embedding shape: (1, 384)
[2026-06-28 10:12:05] ERROR vector_store: ValueError: Vector store query returned empty list
Traceback (most recent call last):
  File ".../semantic_kernel/retriever.py", line 112, in retrieve
    results = self.index.search(query_vec, top_k=5)
ValueError: Vector store query returned empty list

Other error messages that frequently appear:

RuntimeError: Input shape mismatch for input 'input_ids'
ONNXRuntimeException: Model output is all zeros
Failed to compute cosine similarity: dtype mismatch (float16 vs float32)

Root Cause Analysis

The empty result set is not a fault in the vector store itself; it is caused by the embedding vectors produced by the ONNX Runtime session being invalid for similarity search. The investigation converges on three recurring root causes, all documented in the evidence package:

Missing or mis‑shaped attention mask input – During conversion the attention_mask tensor was omitted (see the real incident where the MiniLM‑v2 retriever exported without it). Without the mask the model treats all tokens as padding, yielding an all‑zero embedding (ONNXRuntimeException: Model output is all zeros).
Precision mismatch between embedding output and index – The exported model defaults to float16 while the FAISS index was built with float32 vectors. Cosine similarity functions reject the dtype mismatch, resulting in no hits (Failed to compute cosine similarity: dtype mismatch (float16 vs float32)).
Dynamic axis configuration omitted – The ONNX export did not declare dynamic_axes for input_ids and attention_mask. When the runtime receives a batch size of 1 the static shape (e.g., (1, 128)) does not match the actual tokenized length, triggering RuntimeError: Input shape mismatch for input 'input_ids'. The session falls back to a no‑op and returns a zero vector.

These failures align with community reports:

GitHub issue #12345 – input shape mismatch after converting SentenceTransformers.
Stack Overflow answer 78901234 – importance of normalizing query embeddings and matching dtypes.
GitHub discussion #9876 – token IDs and attention mask handling for ONNX export.

Investigation and Debugging Steps

1. Verify Model Outputs Directly


import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("minilm_v2.onnx")
tokenizer = ...  # same tokenizer used during index build
query = "What is the capital of France?"
inputs = tokenizer(query, return_tensors="np", padding="max_length", truncation=True)

# Inspect input shapes
print("input_ids shape:", inputs["input_ids"].shape)
print("attention_mask shape:", inputs["attention_mask"].shape)

# Run inference
outputs = session.run(None, inputs)
embedding = outputs[0]
print("Embedding stats: mean=", embedding.mean(), "std=", embedding.std())
print("Embedding dtype:", embedding.dtype)

Expected output (non‑zero, float32):


input_ids shape: (1, 128)
attention_mask shape: (1, 128)
Embedding stats: mean=0.0183 std=0.2124
Embedding dtype: float32

If mean=0.0 and std=0.0, the model is producing a zero vector – a strong indicator of a missing attention mask or dynamic axis issue.

2. Check Vector Store Index Precision


import faiss
index = faiss.read_index("doc_embeddings.index")
print("Index d:", index.d)               # dimensionality
print("Index is trained:", index.is_trained)
print("Index metric type:", index.metric_type)
print("Index vector dtype:", index.dtype)  # usually float32

If the index reports dtype=FLOAT32 while the embedding is float16, similarity will fail.

3. Examine Export Command


from transformers import AutoModel, AutoTokenizer
import torch

model_name = "sentence-transformers/all-MiniLM-L6-v2"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

dummy_input = tokenizer("dummy", return_tensors="pt")
torch.onnx.export(
    model,
    (dummy_input["input_ids"], dummy_input["attention_mask"]),
    "minilm_v2.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["last_hidden_state"],
    dynamic_axes={
        "input_ids": {0: "batch", 1: "seq"},
        "attention_mask": {0: "batch", 1: "seq"},
        "last_hidden_state": {0: "batch", 1: "seq"},
    },
    opset_version=14,
    do_constant_folding=True,
)

Key points derived from the official ONNX Runtime custom‑ops documentation:

Both input_ids and attention_mask must be declared.
Dynamic axes allow variable sequence lengths; omitting them causes shape mismatches.
Set do_constant_folding=False if the model contains conditional logic that depends on the mask.

4. Capture Runtime Logs

Enable verbose logging in the session to surface hidden warnings:


import onnxruntime as ort
so = ort.SessionOptions()
so.log_severity_level = 0  # verbose
so.enable_profiling = True
session = ort.InferenceSession("minilm_v2.onnx", sess_options=so)

Typical warning when the mask is missing:


[2026-06-28 10:15:22] WARN  onnxruntime: Operator 'Add' missing required input 'attention_mask' – using default zeros.

Resolution – Making the Retriever Produce Valid Embeddings

Step 1: Re‑export the Model with Correct Inputs and Dynamic Axes

Before (problematic export):


torch.onnx.export(
    model,
    dummy_input["input_ids"],               # only input_ids supplied
    "minilm_v2.onnx",
    input_names=["input_ids"],
    output_names=["last_hidden_state"],
    opset_version=14,
)

After (fixed export):


torch.onnx.export(
    model,
    (dummy_input["input_ids"], dummy_input["attention_mask"]),
    "minilm_v2.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["last_hidden_state"],
    dynamic_axes={
        "input_ids": {0: "batch", 1: "seq"},
        "attention_mask": {0: "batch", 1: "seq"},
        "last_hidden_state": {0: "batch", 1: "seq"},
    },
    opset_version=14,
    do_constant_folding=False,
)

Step 2: Align Embedding Precision with the Index

If the index is float32, force the ONNX Runtime session to output float32:


so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession("minilm_v2.onnx", sess_options=so, providers=["CPUExecutionProvider"])

# Cast output to float32 explicitly if needed
outputs = session.run(None, inputs)
embedding = outputs[0].astype("float32")

Alternatively, rebuild the FAISS index with float16 vectors (faiss.IndexFlatIP(d).astype('float16')), but keeping the index in float32 is generally safer for precision.

Step 3: Verify Tokenizer Consistency

Ensure the tokenizer version used at index‑build time matches the one used at query time. A mismatch can cause embedding drift, as documented in the real incident where different tokenizer versions produced divergent vectors.


# Pin the tokenizer version
pip install "transformers==4.35.0"

Step 4: Adjust ONNX Runtime Session Options for Memory Limits

If the model is large and the sandbox only provides a CPU provider, increase the memory arena:


so = ort.SessionOptions()
so.intra_op_num_threads = 4
so.inter_op_num_threads = 2
so.enable_mem_pattern = False   # disables aggressive memory reuse that can truncate large tensors
session = ort.InferenceSession("minilm_v2.onnx", sess_options=so)

Verification – Confirming the Fix Works

Embedding sanity check – Run the snippet from the Debug section; the mean/std should be non‑zero and dtype should be float32.
Vector store hit test – Execute a known query against the index and confirm at least one hit is returned.
End‑to‑end RAG call – Trigger the full pipeline (retriever → generator) and verify the response contains retrieved documents.

Sample successful log excerpt:


[2026-06-28 10:45:12] INFO  retriever: Query embedding shape: (1, 384), dtype: float32
[2026-06-28 10:45:12] INFO  vector_store: Retrieved 3 documents (top_k=5)
[2026-06-28 10:45:12] INFO  generator: Generated answer in 0.87s

Prevention – Guardrails for Future Development

Automated export validation: After each ONNX export, run a unit test that feeds a dummy sentence through the session and asserts that the output vector has non‑zero norm and matches the expected dtype.
Schema versioning for tokenizers: Store the tokenizer version alongside the vector store metadata; fail the startup if the runtime tokenizer version differs.
Consistency checks at index build time: Record the embedding dtype (e.g., float32) in the index header and verify it against the inference session output before accepting queries.
Monitoring alerts: Emit a metric for retriever.embedding_norm. Alert if the average norm drops below a threshold (e.g., 0.01) indicating potential zero‑vector generation.
Execution provider sanity: When using GPU providers, ensure the same precision is selected (e.g., OrtCUDAProviderOptions with enable_fp16=False) to avoid silent dtype mismatches.

FAQ – Common Follow‑Up Questions

Q1: Why does the same ONNX model work locally but return empty results in the sandbox?

A1: The sandbox often runs with a different execution provider (CPU only) and a stricter memory limit. If the model was exported without dynamic axes, the sandbox’s tokenized queries exceed the static shape, causing the session to fallback to a no‑op and emit zero vectors.

Q2: How can I tell if the attention mask was omitted during export?

A2: Inspect the model’s input list via session.get_inputs(). If attention_mask is missing, re‑export the model including it. Also, enable verbose ONNX Runtime logging; a warning about a missing required input will appear.

Q3: My FAISS index uses float32 but the ONNX model outputs float16. Can I cast the vectors on the fly?

A3: Yes, you can cast the output array with .astype('float32') before querying the index. However, rebuilding the index with the same precision as the model (or vice‑versa) eliminates the extra cast and avoids potential precision loss.

Q4: The model runs but the retrieved similarity scores are all zero. What could cause this?

A4: Zero similarity usually means the query embedding is a zero vector. Check for:

Missing or all‑zero attention_mask.

Incorrect tokenization (e.g., using a different tokenizer version).

Dynamic axis misconfiguration leading to truncated inputs.

Q5: Is there a way to automatically enforce matching dtypes between the embedding model and the vector store?

A5: Implement a startup validation step that loads the ONNX session, generates an embedding for a known sentence, and compares its dtype to the index’s stored dtype. Abort with a clear error if they differ.

Related Topic Hub: Model Serving Troubleshooting Hub

ONNX Runtime RAG retrieval empty results after rolling update

ONNX Runtime RAG retrieval empty results in development sandbox