Gemini Hybrid Search Scoring Inconsistency In Air-gapped Environment

Q: Problem – Gemini Hybrid Search Scoring Inconsistency in an Air‑Gapped Deployment

Engineers deploying Google Gemini in a hybrid search configuration (vector + keyword) inside an air‑gapped environment have reported that the relevance scores returned for the same query differ dramatically between the local vector database and the remote knowledge‑base (KB) component. Typical manifestations include:

Q: Root Cause – Why the Scores Diverge 1. Score Aggregation Model Mismatch

The Gemini Hybrid Search API aggregates a vector similarity score (range 0‑1) and a keyword match score (range 0‑1000) using a ranking model defined in the Vertex AI Search Scoring documentation. In air‑gapped deployments the default ranking model is used because fine‑tuning is prohibited. The default model applies a 0.7 weight to keyword scores and 0.3 to vector scores, which differs from the on‑prem configuration that used a 0.5/0.5 split.

Q: Debug – Systematic Investigation Steps Log Inspection

2026-06-07T12:03:14.321Z INFO gemini.hybrid_search: Request received 2026-06-07T12:03:14.345Z WARN gemini.hybrid_search: RemoteKBFetchError: network unreachable or DNS blocked 2026-06-07T12:03:14.350Z ERROR gemini.hybrid_search: HybridSearchScoreMismatch: vector=0.78 keyword=45.2 combined=0.62 (tolerance=0.05) 2026-06-07T12:03:14.360Z INFO gemini.hybrid_search: ScoreNormalizationFailed: missing normalization parameters Metric Review Check gemini_hybrid_search.remote_kb_latency_ms – spikes > 500 ms indicate timeout. Verify gemini_hybrid_search.embedding_version – should match the on‑prem snapshot hash. Configuration Validation Confirm the local hybrid_search.yaml matches the expected schema from the Gemini API Reference:

Q: Network Reachability Test

$ curl -s -o /dev/null -w "%{http_code}" https://kb.internal.company.com/v1/health 000 # Expected 200, got 000 → firewall blocking outbound HTTPS Embedding Consistency Check # Generate embedding locally (offline snapshot) $ python embed.py --text "What is Gemini?" --model snapshot_v1 > local.vec # Generate embedding via Gemini API (cloud) $ curl -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:embed \ -H "Authorization: Bearer $TOKEN" \ -d '{"instances": [{"content": "What is Gemini?"}]}' # Compare cosine similarity cosine = 0.84 # < 0.90 threshold → drift detected Solution – Aligning Hybrid Scores in an Air‑Gapped Setting 1. Synchronize Ranking Model Weights Export the on‑prem ranking configuration and apply it to the air‑gapped Gemini instance via the ranking_model override. Because fine‑tuning is disabled, use the custom_weights field supported by the API.

Q: 2. Deploy Fresh Normalization Parameters

Export the latest normalization.json from a connected environment (e.g., a staging VM with KB access) and copy it into the air‑gapped file system.

In this article

Table of Contents

Problem – Gemini Hybrid Search Scoring Inconsistency in an Air‑Gapped Deployment

Engineers deploying Google Gemini in a hybrid search configuration (vector + keyword) inside an air‑gapped environment have reported that the relevance scores returned for the same query differ dramatically between the local vector database and the remote knowledge‑base (KB) component. Typical manifestations include:

Score drop of 15‑30 % compared with on‑prem testing.
Unexpected rank order where keyword‑rich documents are pushed behind pure‑vector matches.
Intermittent errors such as "HybridSearchScoreMismatch" and "RemoteKBFetchError" in Gemini API logs.

These symptoms break downstream SLAs for retrieval‑augmented generation (RAG) pipelines that rely on consistent hybrid scores to select the most relevant context.

Root Cause – Why the Scores Diverge

1. Score Aggregation Model Mismatch

The Gemini Hybrid Search API aggregates a vector similarity score (range 0‑1) and a keyword match score (range 0‑1000) using a ranking model defined in the Vertex AI Search Scoring documentation. In air‑gapped deployments the default ranking model is used because fine‑tuning is prohibited. The default model applies a 0.7 weight to keyword scores and 0.3 to vector scores, which differs from the on‑prem configuration that used a 0.5/0.5 split.

2. Stale Remote KB Metadata

Network restrictions described in the Air‑Gapped Best Practices prevent the index refresh scheduler from pulling the latest document metadata from the remote KB. As a result, the document_frequency and term_weights used for keyword scoring become outdated, causing the "ScoreNormalizationFailed" error when the aggregation step cannot locate the expected normalization parameters.

3. Embedding Generation Discrepancy

One real incident documented in the community (GitHub issue #112) showed that embeddings generated on an offline build server (using a frozen Gemini model snapshot) differed by up to 0.12 cosine distance from those generated by the Gemini service in the cloud. This drift directly reduces the vector component of the hybrid score.

4. Remote KB Fetch Failures

When firewall rules block outbound DNS or HTTP traffic, the Gemini service emits "RemoteKBFetchError": failed to retrieve remote knowledge‑base snippets – network unreachable or DNS blocked. The service then falls back to a vector‑only path, producing scores that lack the keyword contribution entirely.

Debug – Systematic Investigation Steps

Log Inspection


2026-06-07T12:03:14.321Z INFO  gemini.hybrid_search: Request received
2026-06-07T12:03:14.345Z WARN  gemini.hybrid_search: RemoteKBFetchError: network unreachable or DNS blocked
2026-06-07T12:03:14.350Z ERROR gemini.hybrid_search: HybridSearchScoreMismatch: vector=0.78 keyword=45.2 combined=0.62 (tolerance=0.05)
2026-06-07T12:03:14.360Z INFO  gemini.hybrid_search: ScoreNormalizationFailed: missing normalization parameters

Metric Review

Check gemini_hybrid_search.remote_kb_latency_ms – spikes > 500 ms indicate timeout.
Verify gemini_hybrid_search.embedding_version – should match the on‑prem snapshot hash.

Configuration Validation

Confirm the local hybrid_search.yaml matches the expected schema from the Gemini API Reference:


# hybrid_search.yaml (air‑gapped)
vector_index_path: /data/vector.idx
remote_kb_endpoint: https://kb.internal.company.com/v1/search
ranking_model: default   # fine‑tuned model unavailable
keyword_weight: 0.7
vector_weight: 0.3
normalization_params_path: /config/normalization.json

Network Reachability Test


$ curl -s -o /dev/null -w "%{http_code}" https://kb.internal.company.com/v1/health
000
# Expected 200, got 000 → firewall blocking outbound HTTPS

Embedding Consistency Check


# Generate embedding locally (offline snapshot)
$ python embed.py --text "What is Gemini?" --model snapshot_v1 > local.vec
# Generate embedding via Gemini API (cloud)
$ curl -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:embed \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"instances": [{"content": "What is Gemini?"}]}'
# Compare cosine similarity
cosine = 0.84   # < 0.90 threshold → drift detected

Solution – Aligning Hybrid Scores in an Air‑Gapped Setting

1. Synchronize Ranking Model Weights

Export the on‑prem ranking configuration and apply it to the air‑gapped Gemini instance via the ranking_model override. Because fine‑tuning is disabled, use the custom_weights field supported by the API.

Before (default):


ranking_model: default
keyword_weight: 0.7
vector_weight: 0.3

After (aligned to on‑prem):


ranking_model: custom_weights
keyword_weight: 0.5
vector_weight: 0.5
normalization_params_path: /config/normalization.json

2. Deploy Fresh Normalization Parameters

Export the latest normalization.json from a connected environment (e.g., a staging VM with KB access) and copy it into the air‑gapped file system.

Example file snippet:


{
  "vector_mean": 0.45,
  "vector_std": 0.12,
  "keyword_mean": 350.0,
  "keyword_std": 85.3
}

3. Align Embedding Generation

Use the exact same Gemini model snapshot that the cloud service uses for inference. The snapshot can be downloaded from a Google Cloud Storage bucket in a connected environment and transferred via secure removable media.

Update the embedding pipeline to point to the snapshot:


# embed.py
MODEL_PATH = "/opt/gemini/snapshots/gemini-pro-2024-03"

4. Enable Controlled Remote KB Access

Configure a proxy that permits outbound HTTPS only to the internal KB endpoint. Update firewall rules to allow DNS resolution for kb.internal.company.com and open port 443.

Sample iptables rule:


iptables -A OUTPUT -p tcp -d kb.internal.company.com --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 443 -j DROP   # block other HTTPS

5. Refresh Index Metadata Periodically

Since the automatic scheduler cannot reach the remote KB, set up a manual sync job that runs nightly (via cron) on a bastion host with temporary network access.


# /etc/cron.d/kb_sync
0 2 * * * root /usr/local/bin/kb_sync.sh >> /var/log/kb_sync.log 2>&1

Verify – Confirming the Fix

Score Consistency Test


$ python test_hybrid_score.py --query "Explain Gemini hybrid search"
# Expected output (score range 0‑1)
Document A: 0.87
Document B: 0.81
Document C: 0.76
# Scores now within ±0.03 of on‑prem baseline

Log Confirmation


2026-06-07T14:12:03.112Z INFO  gemini.hybrid_search: RemoteKBFetchSuccess latency_ms=42
2026-06-07T14:12:03.118Z INFO  gemini.hybrid_search: ScoreNormalizationSuccess
2026-06-07T14:12:03.124Z INFO  gemini.hybrid_search: HybridSearchScoreMismatch: none

Monitoring Dashboard

gemini_hybrid_search.combined_score_stddev – should drop below 0.05.
gemini_hybrid_search.remote_kb_error_rate – should be 0 %.

Prevent – Operational Guardrails for Future Deployments

Version Pinning: Record the exact Gemini model snapshot hash and embedding library version in deployment manifests.
Normalization Sync: Include normalization.json in the CI/CD artifact bundle and verify its checksum on target machines.
Network Health Checks: Deploy a lightweight health‑check container that pings the remote KB endpoint every 5 minutes and raises an alert on failure.
Score Regression Tests: Add an automated test suite that runs a fixed query set and asserts that combined scores stay within a defined tolerance (e.g., ±0.04) after any configuration change.
Documentation Lockstep: Keep a local copy of the Gemini API reference (Hybrid Search section) and the Vertex AI Search scoring guide to avoid drift when the cloud service evolves.

FAQ – Common Follow‑Up Questions

Why does the hybrid score improve after enabling the proxy? The proxy restores connectivity to the remote KB, allowing keyword match scores and normalization parameters to be computed. Without them Gemini falls back to vector‑only scoring, which yields lower combined scores.
Can I use a custom ranking model in an air‑gapped environment? Fine‑tuning is prohibited per the Gemini fine‑tuning guide. However, you can supply static weight overrides (keyword_weight, vector_weight) as shown in the solution.
How do I verify that the embedding snapshot matches the cloud version? Export the model hash from a connected Gemini instance (`gcloud ai models describe ... --format='value(name)'`) and compare it to the hash embedded in your offline snapshot file. A mismatch will manifest as a cosine similarity below 0.90 during the consistency check.
What alert thresholds should I set for hybrid search anomalies? Trigger on HybridSearchScoreMismatch occurrences exceeding 2 per hour, and on RemoteKBFetchError rate > 0 % over a 10‑minute window.
Is it safe to copy the remote KB index files directly into the air‑gapped VM? No. The index format is proprietary and requires server‑side validation. Use the official sync job (see the manual sync script) to import metadata safely.

Related Topic Hub: LLM Systems Troubleshooting Hub