Problem Description
In a hybrid‑cloud deployment of a Retrieval‑Augmented Generation (RAG) pipeline that uses vLLM for inference, the retrieval step consistently returns an empty list or null despite:
- Valid user queries arriving at the API endpoint.
- Document embeddings successfully indexed in the vector store (e.g., Milvus or Pinecone).
- No errors reported by the vLLM server itself.
Typical log excerpts observed during inference:
2024-06-12 14:03:27,812 INFO langchain.retriever - No results found for query
2024-06-12 14:03:27,815 WARNING milvus_sdk - Vector search returned 0 hits
2024-06-12 14:03:27,818 DEBUG vllm.custom_requests - Retrieval callback returned None
These messages match the pattern reported in community sources such as the GitHub issue “vLLM RAG returns empty results” and the Stack Overflow question about empty retrieval lists.
Root Cause Analysis
The empty‑result symptom can be traced to three broad categories that frequently appear in hybrid‑cloud environments:
- Connectivity / Network Isolation
– On‑premise firewalls block outbound traffic to the managed vector DB (e.g., Pinecone). The SDK logs “Failed to connect to vector database: connection refused”, but the higher‑level LangChain retriever swallows the exception and returns an empty list. This mirrors the real incident where a firewall caused silent query failures. - Metadata / Index Mismatch
– The collection name used in the retriever does not exist in the cloud store (e.g., “docs” vs. “documents”). The Milvus SDK raises “Index not found: collection ‘documents’ does not exist”, which LangChain again translates to “No results found for query”. - Embedding Dimensionality Drift
– The on‑premise encoder produces 768‑dimensional vectors while the cloud index expects 1024 dimensions. Milvus rejects the query with “Embedding dimension mismatch: expected 1024, got 768”. The error is logged, but the RAG pipeline proceeds with an empty hit list. - Stale / Replicated Index
– Cross‑region replication lag leaves the query‑target replica without the latest documents. The query succeeds but finds zero matches, a scenario documented in the “Hybrid cloud vector store returns no hits” discussion.
All of these root causes are referenced in the official vLLM custom request handler documentation and the LangChain vLLM integration guide, which emphasize that the retriever must surface connection and schema errors explicitly.
Investigation and Debugging
Follow this step‑by‑step checklist to isolate the cause.
1. Verify vLLM request handling
# Example custom request handler registration (vLLM)
from vllm import LLMEngine, RequestHandler
def rag_handler(prompt, **kwargs):
docs = retriever.retrieve(prompt)
if not docs:
# Log explicit warning
logger.warning("RAG retrieval returned empty list for prompt: %s", prompt)
# Append retrieved context to prompt...
return docs
engine = LLMEngine(...)
engine.register_request_handler("rag", RequestHandler(rag_handler))
Check that the warning appears in engine.log. If it does, the problem is downstream of vLLM.
2. Test vector store connectivity from the on‑premise node
# Using Milvus Python SDK
from pymilvus import connections, utility
connections.connect(
alias="default",
host="milvus.cloud.example.com",
port="19530",
user="my_user",
password="my_secret"
)
print("Connected:", connections.has_connection("default"))
Expected output:
Connected: True
If the command hangs or raises ConnectionError: connection refused, investigate firewall/NAT rules.
3. Confirm collection existence and schema
from pymilvus import Collection
try:
coll = Collection("documents")
print("Collection loaded, dim:", coll.schema.fields[0].params["dim"])
except Exception as e:
print("Error:", e)
Typical error messages:
Error: Index not found: collection 'documents' does not exist
4. Validate embedding dimensionality
# Generate a single embedding with the on‑prem encoder
embedding = encoder.encode("sample query")
print("Embedding dim:", len(embedding))
If the output is 768 while the collection schema reports dim=1024, the mismatch is the cause.
5. Check replication lag (cloud provider console)
Inspect the vector store’s replication metrics. Look for a non‑zero “replication delay” or “stale replica” flag. If lag exceeds the query latency window, recent documents will not be searchable.
6. Capture a network trace (optional)
sudo tcpdump -i eth0 host milvus.cloud.example.com and port 19530 -w /tmp/milvus.pcap
Inspect the capture for SYN‑RETRANSMIT patterns indicating blocked outbound traffic.
Resolution
Apply the fixes that correspond to the identified root cause. Below are before/after snippets for the most common scenarios.
Scenario A – Firewall Blocking Outbound Traffic
Before (no egress rule):
# iptables -L (simplified)
Chain OUTPUT (policy DROP)
target prot opt source destination
After (allow Milvus endpoint):
# iptables -A OUTPUT -p tcp -d milvus.cloud.example.com --dport 19530 -j ACCEPT
# iptables -L OUTPUT
Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- anywhere milvus.cloud.example.com tcp dpt:19530
Result: connections.connect(...) now returns True, and retrieval returns hits.
Scenario B – Wrong Collection Name
Before (incorrect name):
retriever = MilvusRetriever(collection_name="docs", ...)
After (correct name):
retriever = MilvusRetriever(collection_name="documents", ...)
Result: LangChain no longer logs “Index not found” and returns actual documents.
Scenario C – Embedding Dimension Mismatch
Before (on‑prem encoder 768‑dim):
# encoder model: sentence‑transformers/all-MiniLM-L6-v2 (768)
After (align dimensions):
# Option 1: Re‑index with a 768‑dim collection
milvus_client.create_collection(name="documents", dim=768, ...)
# Option 2: Upgrade encoder to 1024‑dim model
from sentence_transformers import SentenceTransformer
encoder = SentenceTransformer("sentence-transformers/all-mpnet-base-v2") # 1024 dim
Result: Milvus accepts the query vectors and returns nearest‑neighbor hits.
Scenario D – Replication Lag
Mitigation steps:
- Configure the retriever to target the primary region for reads.
- Enable “read‑after‑write” consistency in the vector DB client (e.g.,
consistency_level="Strong"). - If immediate consistency is required, add a post‑indexing sync call:
# Force sync after bulk upsert (Milvus example)
milvus_client.flush(["documents"])
milvus_client.wait_for_index_build_progress("documents", 100, timeout=30)
Result: Subsequent queries see the newly indexed vectors.
Validation
After applying the fix, perform the following checks:
- Run a sanity query through the RAG endpoint and verify that
retriever.retrieve()returns a non‑empty list. - Inspect the vLLM logs for the absence of “No results found for query” warnings.
- Query the vector store directly:
results = retriever.retrieve("What is the SLA for service X?")
print("Retrieved docs:", len(results))
Expected output (example):
Retrieved docs: 3
Additionally, monitor the retrieval_latency_seconds metric (if exported) to confirm that latency remains within SLA bounds.
Operational Experience
- Initial suspicion often falls on the LLM side because vLLM logs show no errors; however, the retriever silently consumes lower‑level failures.
- In hybrid deployments, the same code works on a pure cloud testbed but fails on‑prem due to missing egress rules – a classic “works in dev, breaks in prod” scenario.
- Embedding dimension mismatches are easy to miss because both encoder and index creation succeed independently; only a runtime query reveals the incompatibility.
- Replication lag manifested as “zero hits” only for queries issued immediately after a bulk upsert; adding a short
sleep(2)masked the problem in early tests.
Best Practices and Prevention
| Area | Recommendation |
|---|---|
| Network | Whitelist vector store endpoints in both inbound and outbound firewall rules; verify with nc -zv host port from every compute node. |
| Schema Management | Store collection name, dimension, and metric type in a version‑controlled config file; validate at service start‑up. |
| Embedding Consistency | Pin encoder model version and enforce matching dim in the vector DB schema; add a CI test that compares len(encoder.encode("test")) to the stored dimension. |
| Replication | Use strong consistency reads for RAG queries; monitor replication lag metrics and set alerts for >5 seconds delay. |
| Observability | Expose retrieval‑specific metrics (hit count, latency, error codes) and create alerts on “hit count == 0” for consecutive requests. |
Related Topic Hub: Model Serving Troubleshooting Hub
FAQ
- Why does the RAG pipeline return empty results only in the hybrid‑cloud environment?
Because the on‑premise network cannot reach the managed vector store, causing the SDK to fall back to an empty result set. Verify egress rules and DNS resolution. - How can I confirm that the embedding dimensions match the vector store schema?
Print the length of a sample embedding and compare it to the collection’sdimfield via the SDK. Align them by re‑creating the collection or switching the encoder model. - What does “Vector search returned 0 hits” mean in Milvus logs?
It indicates that the query executed successfully but found no vectors within the similarity threshold. Common causes are wrong collection name, dimension mismatch, or querying a replica that lacks the data. - Can replication lag cause intermittent empty results?
Yes. If a query hits a replica that has not yet received the latest upserts, the similarity search will return zero hits. Use strong consistency reads or target the primary region. - Is there a way to make the retriever raise an exception instead of returning an empty list?
Wrap the retriever call in a try/except block and re‑raise on empty results, or setraise_on_empty=Trueif using LangChain’sMilvusRetriever(available in recent releases).