ONNX Runtime RAG Vector Store Connection Timeout During Batch Inference

Q: Problem – ONNX Runtime RAG Vector Store Connection Timeout During Batch Inference

Symptoms observed in production batch runs:

Q: Root Cause – Mismatch Between ONNX Runtime Client Settings and Vector Store Service

The RAG pipeline uses the onnxruntime‑extensions package to call a remote vector database (e.g., Qdrant, Milvus, or Pinecone) via HTTPS. Two independent misconfigurations caused the timeout:

Q: Debug – Investigation and Diagnostic Steps 1. Verify network connectivity from the pod

# Inside the failing pod curl -v --max-time 5 https://vectordb.internal:443/v1/health Expected output:

Q: 2. Inspect Kubernetes NetworkPolicy

kubectl get networkpolicy vector-db-egress -n rag-pipeline -o yaml Snippet of the policy:

Q: 3. Check authentication token lifecycle

# Locate the cached token file used by the extension cat /tmp/onnx_rag_token.json { "access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...", "exp":1699171200 } Current epoch time at failure:

Q: 4. Review ONNX Runtime extension configuration

# onnx_rag_config.yaml vector_store: endpoint: "https://vectordb.internal:443/v1/query" auth: method: "bearer" token_path: "/tmp/onnx_rag_token.json" refresh_interval_seconds: 0 #

Q: 3. Redeploy the inference service

kubectl rollout restart deployment rag-inference -n rag-pipeline Why the fix works Allowing outbound TCP 443 removes the packet drop that caused the client’s connect timeout. Setting a non‑zero refresh_interval_seconds forces the extension to re‑authenticate before the cached JWT expires, preventing “token expired” errors. Using a credential provider abstracts token management and reduces the risk of stale credentials in long‑running jobs. Verify – Validation Steps After Applying the Fix Confirm network connectivity: curl -v https://vectordb.internal:443/v1/health Expected output contains 200 OK within 100 ms.

In this article

Table of Contents

Problem – ONNX Runtime RAG Vector Store Connection Timeout During Batch Inference

Symptoms observed in production batch runs:

Batch inference jobs exit with RuntimeError: Retrieval failed – connection timeout.
Log entries from the ONNX Runtime host show repeated authentication failures:


2026-11-05 02:13:41,872 [ERROR] onnxruntime.capi._pybind_state: RetrievalError: 
    Failed to query vector store at https://vectordb.internal:443/v1/query 
    - error: ConnectTimeoutError: Connection timed out after 5000ms
2026-11-05 02:13:41,874 [ERROR] onnxruntime.capi._pybind_state: RetrievalError: 
    Authentication failed: token expired or invalid

Impact:

Retrieval‑augmented generation (RAG) steps return empty context, causing the language model to generate unrelated or generic answers.
Overall batch latency increases by 30‑45 seconds per 1 000 records due to retry loops.
Downstream SLAs for response quality are missed, triggering alert fatigue.

Root Cause – Mismatch Between ONNX Runtime Client Settings and Vector Store Service

The RAG pipeline uses the onnxruntime‑extensions package to call a remote vector database (e.g., Qdrant, Milvus, or Pinecone) via HTTPS. Two independent misconfigurations caused the timeout:

Network egress restriction: The Linux host runs inside a Kubernetes pod with NetworkPolicy that only allowed outbound traffic to ports 80/443 on a whitelist. The vector store service was reachable only through an internal service mesh endpoint (vectordb.internal:443) that was not included, causing packets to be dropped and the client to hit its 5 s connect timeout.
Stale authentication token: The ONNX Runtime extension cached a JWT token at pod start. The token’s exp claim was 1 hour, but the batch job ran for 4 hours. The client never refreshed the token because auth_refresh_interval was set to 0, so every request after the first hour failed with “token expired”.

Both issues manifested as the same generic “connection timeout / authentication error” because the client aborts early on TLS handshake failures caused by dropped packets.

Debug – Investigation and Diagnostic Steps

1. Verify network connectivity from the pod

# Inside the failing pod
curl -v --max-time 5 https://vectordb.internal:443/v1/health

Expected output:

*   Trying 10.2.5.12:443...
* Connected to vectordb.internal (10.2.5.12) port 443 (#0)
* TLS 1.3 ...
> GET /v1/health HTTP/1.1
...
< 200 OK

Observed output:

curl: (28) Connection timed out after 5000 milliseconds

2. Inspect Kubernetes NetworkPolicy

kubectl get networkpolicy vector-db-egress -n rag-pipeline -o yaml

Snippet of the policy:

spec:
  podSelector:
    matchLabels:
      app: rag-inference
  egress:
  - to:
    - ipBlock:
        cidr: 10.1.0.0/16
    ports:
    - protocol: TCP
      port: 80

Port 443 was missing, confirming the egress block.

3. Check authentication token lifecycle

# Locate the cached token file used by the extension
cat /tmp/onnx_rag_token.json

{
  "access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "exp":1699171200
}

Current epoch time at failure:

date +%s
1699185600

The token had expired 4 hours earlier.

4. Review ONNX Runtime extension configuration

# onnx_rag_config.yaml
vector_store:
  endpoint: "https://vectordb.internal:443/v1/query"
  auth:
    method: "bearer"
    token_path: "/tmp/onnx_rag_token.json"
    refresh_interval_seconds: 0   # <-- problematic
  timeout_seconds: 5

Solution – Configuration Corrections and Token Refresh Logic

1. Update Kubernetes NetworkPolicy to allow HTTPS egress

# networkpolicy.yaml (updated)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vector-db-egress
  namespace: rag-pipeline
spec:
  podSelector:
    matchLabels:
      app: rag-inference
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.1.0.0/16
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443   # added

Apply the change:

kubectl apply -f networkpolicy.yaml

2. Enable periodic token refresh in the ONNX Runtime extension

Modify onnx_rag_config.yaml:

# Before
refresh_interval_seconds: 0

# After
refresh_interval_seconds: 300   # refresh every 5 minutes

Alternatively, switch to the built‑in credential provider that automatically fetches a fresh token from the auth service:

# onnx_rag_config.yaml (using provider)
auth:
  method: "provider"
  provider: "aws_sts"
  role_arn: "arn:aws:iam::123456789012:role/RagVectorAccess"
  refresh_interval_seconds: 300

3. Redeploy the inference service

kubectl rollout restart deployment rag-inference -n rag-pipeline

Why the fix works

Allowing outbound TCP 443 removes the packet drop that caused the client’s connect timeout.
Setting a non‑zero refresh_interval_seconds forces the extension to re‑authenticate before the cached JWT expires, preventing “token expired” errors.
Using a credential provider abstracts token management and reduces the risk of stale credentials in long‑running jobs.

Verify – Validation Steps After Applying the Fix

Confirm network connectivity:

curl -v https://vectordb.internal:443/v1/health

Expected output contains 200 OK within 100 ms.

Run a short batch inference (e.g., 10 records) and inspect logs for successful retrieval:

2026-11-05 02:45:12,301 [INFO] onnxruntime.capi._pybind_state: Retrieval succeeded – 10 vectors returned
2026-11-05 02:45:12,305 [INFO] onnxruntime.capi._pybind_state: Batch inference completed in 1.2s

Check that the token file is being refreshed:

stat -c %Y /tmp/onnx_rag_token.json   # modification epoch
date +%s                               # current epoch

The difference should be less than refresh_interval_seconds.

Monitor the pod’s netstat for established connections to the vector store:

ss -tnp | grep vectordb.internal

Sample output:

ESTAB 0 0 10.2.5.12:54321 10.2.5.13:443 users:(("python",pid=3124,fd=5))

Prevent – Best Practices and Ongoing Guardrails

NetworkPolicy Audits: Include port 443 in any egress whitelist for services that require TLS. Automate policy diff checks in CI.
Credential Rotation: Use short‑lived tokens (≤15 min) and enforce a non‑zero refresh interval. Prefer cloud‑native providers (AWS STS, GCP IAM) over static token files.
Health Probes: Add a lightweight health endpoint (/v1/health) to the vector store and configure ONNX Runtime to ping it before batch start.
Observability: Export onnxruntime_retrieval_latency_seconds and onnxruntime_retrieval_errors_total to Prometheus. Alert on error rate > 0.1 % or latency spikes > 2× baseline.
Retry Strategy: Configure exponential back‑off with jitter for vector store queries. Ensure the total retry window exceeds the maximum expected token refresh time.

FAQ – Common Follow‑Up Questions

Why does the connection succeed locally but time out in the Kubernetes pod?
Because the local machine is not subject to the pod’s NetworkPolicy. The policy blocks outbound TCP 443, causing the pod’s connections to be dropped.
Can I keep refresh_interval_seconds set to zero and still avoid token expiry?
Only if you implement an external process that rewrites the token file before it expires. The built‑in refresh mechanism is the safest approach.
What if the vector store uses mutual TLS (mTLS) instead of bearer tokens?
Add the client certificate and key to the ONNX Runtime config and ensure the pod’s service account has permission to mount the secret. Also open port 443 for egress.
How do I debug intermittent timeouts that occur only under load?
Capture a packet trace with tcpdump -i eth0 host vectordb.internal and port 443 -w /tmp/capture.pcap, then analyze for SYN retransmissions or TCP window stalls. Correlate with pod CPU/memory pressure.
Is there a way to test the token refresh logic without redeploying?
Yes. Manually delete the cached token file and trigger a new inference request; the extension will fetch a fresh token using the configured provider.

Related Topic Hub: Model Serving Troubleshooting Hub

ONNX Runtime RAG vector store connection timeout during batch inference