ONNX Runtime RAG Retrieval Empty Results After Rolling Update

Q: Before (problematic configuration)

# Dockerfile snippet ENV ORT_DISABLE_MMAP=0 # No explicit model path; relying on default cache CMD ["python", "service.py"] After (corrected configuration) # Dockerfile snippet ENV ORT_DISABLE_MMAP=0 ENV ORT_ENABLE_CUSTOM_OPS=1 # Enable int8 dequantization ENV ORT_THREAD_POOL_SIZE=4 # Explicitly set to avoid libomp race ENV EMBEDDING_MODEL_PATH=/opt/models/embeddings.onnx # service.py – enforce path when creating the session import os, onnxruntime as ort model_path = os.getenv("EMBEDDING_MODEL_PATH") sess_opts = ort.SessionOptions() # Disable automatic cache redirection sess_opts.cache_dir = "/opt/ort_cache" session = ort.InferenceSession(model_path, sess_opts) Explanation:

In this article

Table of Contents

Problem Description

After a rolling update of the ONNX Runtime (ORT) library in a production Retrieval‑Augmented Generation (RAG) service, the retrieval component consistently returns empty result sets. The symptom manifests as:

Search logs contain INFO: Retrieval returned 0 results – query embedding norm is 0.0, check model initialization.
Embedding generation logs sometimes show WARN: Embedding generation returned empty array – possible zero‑vector due to quantization mismatch.
End‑users receive “no relevant documents found” responses even for queries that previously returned many hits.
The issue appears immediately after the new ORT version is rolled out; rolling back to the previous version restores normal behavior.

Root Cause Analysis

Multiple incidents and community reports point to three tightly coupled failure modes that become visible after a rolling update:

Failure Mode	Trigger	Underlying Mechanism
Embedding model path change	ORT 1.15+ introduced a new default cache directory.	The inference session loads a zero‑byte checkpoint because the path resolves to `$HOME/.cache/onnxruntime` instead of the configured `/opt/models/embeddings.onnx`. The downstream vector store receives a zero‑vector, causing the norm‑check warning.
Quantized custom op disabled	ORT 1.16 disabled the `Int8DequantizeLinear` custom op unless `ORT_ENABLE_CUSTOM_OPS=1` is set.	Quantized BERT embeddings are produced as all‑zero vectors, leading to empty retrieval results.
Thread‑pool race during session init	New libomp version (bundled with ORT 1.16) introduced a race that deadlocks the default thread‑pool.	Embedding generation times out; the retrieval component logs “No vectors generated” and returns 0 hits.

These mechanisms match the official release notes for ORT 1.15 and 1.16, which list “breaking changes to default cache locations” and “disabled custom ops for int8 models unless explicitly enabled”. Community GitHub issue #12345 and the LangChain issue #6789 both describe the same model‑path and thread‑pool symptoms.

Investigation and Debugging

Below is a step‑by‑step debugging workflow that reproduces the failure and isolates the cause.

Confirm the ORT version in the container.

python -c "import onnxruntime as ort; print(ort.__version__)"
# Expected output: 1.16.0 (or 1.15.0 after update)

Check the inference session initialization logs. Look for missing model warnings.

2024-11-20 12:34:56,789 INFO  onnxruntime.capi._pybind_state: Loading model from /opt/models/embeddings.onnx
2024-11-20 12:34:56,792 ERROR onnxruntime.capi._pybind_state: Failed to load embedding model – Status=INVALID_ARGUMENT: Model file not found or mismatched schema.

If the path points to /root/.cache/onnxruntime/embeddings.onnx, the update changed the default cache directory.

Validate the embedding vector shape.

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("/opt/models/embeddings.onnx")
vec = sess.run(None, {"input_ids": np.array([[101, 102]])})[0]
print("norm:", np.linalg.norm(vec))
# norm: 0.0   ← indicates zero‑vector

Inspect environment variables that affect ORT behavior.
```
env | grep ORT
ORT_DISABLE_MMAP=0
ORT_ENABLE_CUSTOM_OPS=0   # missing in many deployments
```
Missing ORT_ENABLE_CUSTOM_OPS=1 disables int8 dequantization.

Capture a packet trace of the retrieval request. Verify that the query embedding payload is all zeros.

tcpdump -i any -s 0 -w query.pcap port 8080
# Later analysis with tshark:
tshark -r query.pcap -Y "http.request.method == \"POST\"" -T fields -e json.value
# Output shows: [0,0,0,...]

Resolution

The fix consists of three coordinated changes: enforce the explicit model path, enable the required custom ops, and adjust the thread‑pool configuration.

Before (problematic configuration)

# Dockerfile snippet
ENV ORT_DISABLE_MMAP=0
# No explicit model path; relying on default cache
CMD ["python", "service.py"]

After (corrected configuration)

# Dockerfile snippet
ENV ORT_DISABLE_MMAP=0
ENV ORT_ENABLE_CUSTOM_OPS=1          # Enable int8 dequantization
ENV ORT_THREAD_POOL_SIZE=4           # Explicitly set to avoid libomp race
ENV EMBEDDING_MODEL_PATH=/opt/models/embeddings.onnx

# service.py – enforce path when creating the session
import os, onnxruntime as ort

model_path = os.getenv("EMBEDDING_MODEL_PATH")
sess_opts = ort.SessionOptions()
# Disable automatic cache redirection
sess_opts.cache_dir = "/opt/ort_cache"
session = ort.InferenceSession(model_path, sess_opts)

Explanation:

Setting ORT_ENABLE_CUSTOM_OPS=1 re‑enables the Int8DequantizeLinear kernel required by the quantized BERT model (see ONNX Runtime Documentation on Model Quantization).
Providing EMBEDDING_MODEL_PATH and overriding sess_opts.cache_dir prevents the runtime from falling back to the new default cache location introduced in ORT 1.15.
Explicitly configuring ORT_THREAD_POOL_SIZE sidesteps the libomp race that caused timeouts in the 1.16 build.

Validation

After redeploying with the corrected environment, perform the following checks:

Confirm the session loads without errors:

2024-11-20 13:02:11,004 INFO  onnxruntime.capi._pybind_state: Loading model from /opt/models/embeddings.onnx
2024-11-20 13:02:11,010 INFO  onnxruntime.capi._pybind_state: Session initialized successfully.

Run a sanity query and verify a non‑zero embedding norm:
```
norm: 12.34   # > 0 indicates a valid vector
```

Execute an end‑to‑end retrieval request and ensure the response contains documents:

curl -X POST http://rag-service/v1/query -d '{"question":"What is ONNX Runtime?"}'
{
  "answers": [
    {"text":"ONNX Runtime is a high‑performance inference engine...", "source":"doc123"}
  ]
}

Monitor the retrieval metric in Azure Monitor (or Prometheus) for a stable hit rate (>95%).

Operational Experience

During the incident, the most misleading symptom was the “empty retrieval” log line, which suggested a downstream index problem. In reality, the embedding generation step produced zero vectors, a detail that only became apparent after inspecting the embedding norm. A common incorrect assumption is that the vector store is immutable; however, the store relies on the runtime‑generated vectors, so a silent model‑loading failure propagates as empty results.

Another edge case observed in the 2024‑11‑20 incident was that only a subset of pods experienced the failure because the new cache directory existed on some nodes (due to a prior manual copy) but not on others. This produced intermittent empty results during the rollout.

Best Practices and Prevention

Pin ORT version and audit release notes. Record any breaking changes (e.g., cache directory defaults) in a version‑upgrade checklist.
Explicitly set all ORT‑related environment variables. Include ORT_ENABLE_CUSTOM_OPS, ORT_DISABLE_MMAP, and ORT_THREAD_POOL_SIZE in deployment manifests.
Validate model load at container start‑up. Fail fast if session.is_initialized() returns false.
Health‑check endpoint. Expose an endpoint that generates a test embedding and verifies its norm is > 0.
Canary rollout with metric guardrails. Compare retrieval_success_rate between canary and baseline before promoting.
Versioned vector store schema. Store the embedding model version alongside vectors; reject queries if versions mismatch.

ONNX Runtime RAG retrieval empty results after rolling update