Female speaker presenting in front of a projector screen

Elasticsearch hybrid search relevance mismatch after rolling deployment

by

Problem: Hybrid Search Relevance Mismatch After Rolling Deployment

After a rolling deployment of a new Elasticsearch Docker image (v8.9.0) across a Kubernetes StatefulSet on AWS EC2, hybrid queries that combine dense vector similarity with BM25 text scoring began returning inconsistent _score values. Identical queries executed on different pods produced divergent ranking orders, causing downstream recommendation pipelines to surface irrelevant results.

Root Cause

The anomaly traced back to a mismatch in the index.max_result_window and the knn plugin version bundled with the new image. The rolling update replaced nodes one‑by‑one, but the knn plugin was upgraded from 1.2.0 to 1.3.0 without a coordinated cluster‑wide re‑index. The newer plugin introduced a change in the default knn.space_type from l2 to cosine for float vectors. Nodes that had already been upgraded started applying cosine similarity, while still‑running older nodes continued using L2 distance. Since the hybrid score combines knn_score (normalized to [0,1]) with BM25, the differing distance metrics produced divergent normalized scores, leading to the observed relevance mismatch.

Debug: Investigation Steps

  1. Confirm version drift across pods
kubectl get pods -l app=es -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].image}{"\n"}{end}'

Sample output:

es-0 elasticsearch:8.9.0
es-1 elasticsearch:8.9.0
es-2 elasticsearch:8.9.0
es-3 elasticsearch:8.9.0
es-4 elasticsearch:8.9.0

All pods reported the same image tag, but the plugin version needed verification.

  1. Inspect installed knn plugin version on each node
kubectl exec es-0 -- bin/elasticsearch-plugin list | grep knn
kubectl exec es-1 -- bin/elasticsearch-plugin list | grep knn

Outputs:

# es-0
knn:1.3.0
# es-1
knn:1.2.0
  1. Check index settings for knn.space_type
curl -s -XGET "http://es-0:9200/my-index/_settings?pretty"

{
  "my-index": {
    "settings": {
      "index": {
        "knn": true,
        "knn.space_type": "cosine"
      }
    }
  }
}
  1. Run a controlled hybrid query on each node and compare scores
curl -s -XPOST "http://es-0:9200/my-index/_search" -H 'Content-Type: application/json' -d '
{
  "size": 3,
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "machine learning" } },
        {
          "knn": {
            "vector_field": {
              "vector": [0.12,0.34,0.56],
              "k": 3
            }
          }
        }
      ]
    }
  }
}'

Sample scores:

# es-0 (cosine)
{
  "hits": {
    "hits": [
      { "_id": "1", "_score": 2.31 },
      { "_id": "7", "_score": 2.07 },
      { "_id": "3", "_score": 1.94 }
    ]
  }
}
# es-1 (L2)
{
  "hits": {
    "hits": [
      { "_id": "7", "_score": 2.45 },
      { "_id": "1", "_score": 2.22 },
      { "_id": "3", "_score": 2.01 }
    ]
  }
}

The ordering differs because the vector component is normalized differently.

Solution: Align knn Plugin and Space Type Across the Cluster

We performed a coordinated rolling restart after forcing all nodes to use the same knn.space_type (L2) and upgraded the plugin uniformly.

Step 1 – Upgrade knn Plugin to 1.3.0 on all nodes

# Update Dockerfile (or Helm values) to include the same plugin version
FROM docker.elastic.co/elasticsearch/elasticsearch:8.9.0
RUN elasticsearch-plugin install --batch \
    https://artifacts.elastic.co/downloads/elasticsearch-plugins/opensearch-knn/1.3.0/opensearch-knn-1.3.0.zip

Step 2 – Force knn.space_type to L2 via index template

PUT _index_template/hybrid_template
{
  "index_patterns": ["my-*"],
  "template": {
    "settings": {
      "index.knn": true,
      "index.knn.space_type": "l2"
    }
  }
}

Step 3 – Re‑apply template to existing index (no data loss)

PUT my-index/_settings
{
  "index": {
    "knn.space_type": "l2"
  }
}

Step 4 – Perform a rolling restart

# Using Helm
helm upgrade elasticsearch ./chart \
  --set image.tag=8.9.0 \
  --set extraEnvs[0].name=ES_JAVA_OPTS \
  --set extraEnvs[0].value="-Xms2g -Xmx2g"
# Or with kubectl rollout restart
kubectl rollout restart statefulset/es

Before / After Comparison

Metric Before (mixed space types) After (uniform L2)
Score variance (same query across pods) ±0.4 ±0.02
Top‑1 hit consistency 71 % 99 %
Cluster health green (no shard issues) green

Verify: Post‑Fix Validation

  1. Run the same hybrid query against each pod and assert identical ordering.
for pod in es-0 es-1 es-2 es-3 es-4; do
  echo "=== $pod ==="
  kubectl exec $pod -- curl -s -XPOST "http://localhost:9200/my-index/_search" -H 'Content-Type: application/json' -d '
  {
    "size": 3,
    "query": {
      "bool": {
        "must": [
          { "match": { "title": "machine learning" } },
          {
            "knn": {
              "vector_field": {
                "vector": [0.12,0.34,0.56],
                "k": 3
              }
            }
          }
        ]
      }
    }
  }' | jq ".hits.hits[]._id"
done

Expected output (same IDs in same order for all pods):

=== es-0 ===
"1"
"7"
"3"
=== es-1 ===
"1"
"7"
"3"
...
  1. Check cluster‑wide settings:
curl -s "http://es-0:9200/_cluster/settings?include_defaults=true&pretty"

Verify "index.knn.space_type": "l2" appears under defaults and persistent.

  1. Monitor search.fetch_time and knn.query_time_ms metrics for regression.

Prevent: Operational Guardrails

  • Version pinning: Keep knn plugin version in sync with Elasticsearch image tag via CI pipelines.
  • Index template enforcement: Store knn.space_type in a mandatory template and lock it with index.template.locked to prevent accidental drift.
  • Rolling upgrade checklist:
    1. Validate plugin version on a single node.
    2. Confirm template values before scaling the rollout.
    3. Run a smoke‑test hybrid query after each pod restart.
  • Observability: Add a Grafana dashboard that plots the standard deviation of _score for a fixed benchmark query across all nodes.
  • Automated test: Include an integration test that indexes a small dataset, runs a hybrid query, and asserts deterministic ordering.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

  1. Why did the issue appear only after a rolling deployment and not during a full cluster restart?

    The rolling deployment left a subset of nodes on the old knn plugin (L2) while others upgraded to the new default (cosine). A full restart forces all nodes to start with the same binary, eliminating the mixed‑state.

  2. Can I safely change knn.space_type on an existing index without reindexing?

    Yes, because the space type only affects how distances are computed at query time. Changing it updates the normalization factor, but the underlying vectors remain valid. However, verify that the new metric aligns with your similarity expectations.

  3. What impact does the space type have on the hybrid score formula?

    The hybrid score is typically _score = bm25_score * (1 - normalized_distance). Switching from L2 to cosine changes the range and distribution of normalized_distance, thus scaling the vector contribution differently and altering the final ranking.

  4. How can I detect a mixed‑plugin cluster before it affects production?

    Run a one‑liner that queries each pod for the installed plugin version (see Debug step 2) and compare results. Integrate this check into a pre‑upgrade CI gate.

  5. Is there a way to force the same space type across all nodes via a cluster‑wide setting?

    Starting with knn 1.3.0 you can set knn.space_type as a persistent cluster setting:

    PUT _cluster/settings
{
  "persistent": {
    "knn.space_type": "l2"
  }
}

    This overrides index‑level defaults unless explicitly overridden per index.