Ingress Routing Sends A/B Test Queries To Wrong Elasticsearch Index

Q: Root Cause – Mis‑aligned Ingress path/host rules and missing header‑based routing

The Kubernetes Ingress object used nginx‑ingress with path‑based routing only. The A/B test relied on a custom HTTP header X‑Test‑Variant (values v1 or v2) that the application set before forwarding the request to Elasticsearch. The original Ingress configuration did not include a map block to route based on that header, so all requests matching /search were sent to the default backend (search‑v1). Additionally, the rewrite-target annotation unintentionally stripped the variant suffix from the URL, causing the downstream service to resolve the default index.

Q: Debug – Investigation steps and evidence 1. Log inspection

$ kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -c controller | grep "search" 2024-10-12T08:15:23Z "GET /search?q=foo HTTP/1.1" 200 1234 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200" 2024-10-12T08:15:24Z "GET /search?q=bar HTTP/1.1" 200 1234 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200" The upstream address is always search‑v1, even when the request header X‑Test‑Variant: v2 is present (verified later with curl -H).

Q: 2. Header verification

$ curl -s -D - -H "X-Test-Variant: v2" "https://search.example.com/search?q=test" | grep X-Test-Variant X-Test-Variant: v2 The header reaches the Ingress controller, but no routing rule consumes it.

Q: 3. Ingress configuration snapshot (before)

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: search-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: search.example.com http: paths: - path: /search pathType: Prefix backend: service: name: search-v1 port: number: 9200 The configuration only defines a single backend (search‑v1) and lacks any conditional routing.

Q: 4. Elasticsearch query logs

[2024-10-12T08:15:23,456][INFO ][o.e.c.m.MetaDataMappingService] [node-1] index [search-v1] created [2024-10-12T08:15:24,112][INFO ][o.e.c.m.MetaDataMappingService] [node-1] index [search-v1] created No queries are observed against search‑v2, confirming the routing issue.

Q: Solution – Correct Ingress configuration with header‑based routing

We introduce an nginx.ingress.kubernetes.io/server-snippet that creates an map to select the backend based on X‑Test‑Variant. The rewrite target is also removed because the downstream service expects the original path.

Q: Before (problematic) snippet

annotations: nginx.ingress.kubernetes.io/rewrite-target: / After – Updated Ingress manifest apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: search-ingress annotations: nginx.ingress.kubernetes.io/server-snippet: | map $http_x_test_variant $search_backend { default search-v1.default.svc.cluster.local:9200; "v1" search-v1.default.svc.cluster.local:9200; "v2" search-v2.default.svc.cluster.local:9200; } nginx.ingress.kubernetes.io/configuration-snippet: | proxy_pass http://$search_backend; spec: rules: - host: search.example.com http: paths: - path: /search pathType: Prefix backend: service: name: placeholder # required field, not used port: number: 80 Key changes:

Q: Verify – Confirm that traffic is now correctly split 1. Ingress controller reload

$ kubectl rollout restart deployment -n ingress-nginx ingress-nginx-controller deployment.apps/ingress-nginx-controller restarted 2. Functional test with both variants $ curl -s -H "X-Test-Variant: v1" https://search.example.com/search?q=alpha | jq '.hits.total' { "value": 1245, "relation": "eq" } $ curl -s -H "X-Test-Variant: v2" https://search.example.com/search?q=alpha | jq '.hits.total' { "value": 987, "relation": "eq" } The two calls return different hits.total values, matching the data seeded in search‑v1 and search‑v2 respectively.

Q: 3. Ingress logs after fix

2024-10-12T09:02:11Z "GET /search?q=alpha HTTP/1.1" 200 1245 "-" "curl/7.68.0" upstream: "search-v2.default.svc.cluster.local:9200" 2024-10-12T09:02:13Z "GET /search?q=beta HTTP/1.1" 200 987 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200" The upstream now reflects the variant header.

In this article

Table of Contents

Problem – Ingress routing sends A/B test queries to the wrong Elasticsearch index

In a production environment that runs A/B tests on the search pipeline, traffic is supposed to be split between two Elasticsearch clusters (or two indices within the same cluster): search‑v1 and search‑v2. After deploying the new version, a subset of queries that should hit search‑v2 were observed returning results from search‑v1. The symptom manifested as inconsistent search results across the two test variants, causing flaky A/B test metrics.

Root Cause – Mis‑aligned Ingress path/host rules and missing header‑based routing

The Kubernetes Ingress object used nginx‑ingress with path‑based routing only. The A/B test relied on a custom HTTP header X‑Test‑Variant (values v1 or v2) that the application set before forwarding the request to Elasticsearch. The original Ingress configuration did not include a map block to route based on that header, so all requests matching /search were sent to the default backend (search‑v1). Additionally, the rewrite-target annotation unintentionally stripped the variant suffix from the URL, causing the downstream service to resolve the default index.

Debug – Investigation steps and evidence

1. Log inspection


$ kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -c controller | grep "search"
2024-10-12T08:15:23Z "GET /search?q=foo HTTP/1.1" 200 1234 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200"
2024-10-12T08:15:24Z "GET /search?q=bar HTTP/1.1" 200 1234 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200"

The upstream address is always search‑v1, even when the request header X‑Test‑Variant: v2 is present (verified later with curl -H).

2. Header verification


$ curl -s -D - -H "X-Test-Variant: v2" "https://search.example.com/search?q=test" | grep X-Test-Variant
X-Test-Variant: v2

The header reaches the Ingress controller, but no routing rule consumes it.

3. Ingress configuration snapshot (before)


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: search-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: search.example.com
    http:
      paths:
      - path: /search
        pathType: Prefix
        backend:
          service:
            name: search-v1
            port:
              number: 9200

The configuration only defines a single backend (search‑v1) and lacks any conditional routing.

4. Elasticsearch query logs


[2024-10-12T08:15:23,456][INFO ][o.e.c.m.MetaDataMappingService] [node-1] index [search-v1] created
[2024-10-12T08:15:24,112][INFO ][o.e.c.m.MetaDataMappingService] [node-1] index [search-v1] created

No queries are observed against search‑v2, confirming the routing issue.

Solution – Correct Ingress configuration with header‑based routing

We introduce an nginx.ingress.kubernetes.io/server-snippet that creates an map to select the backend based on X‑Test‑Variant. The rewrite target is also removed because the downstream service expects the original path.

Before (problematic) snippet


annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /

After – Updated Ingress manifest


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: search-ingress
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      map $http_x_test_variant $search_backend {
          default          search-v1.default.svc.cluster.local:9200;
          "v1"             search-v1.default.svc.cluster.local:9200;
          "v2"             search-v2.default.svc.cluster.local:9200;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_pass http://$search_backend;
spec:
  rules:
  - host: search.example.com
    http:
      paths:
      - path: /search
        pathType: Prefix
        backend:
          service:
            name: placeholder  # required field, not used
            port:
              number: 80

Key changes:

Define a map that selects the upstream based on the X‑Test‑Variant header.
Replace the default proxy_pass with a dynamic reference to $search_backend.
Remove the rewrite rule to preserve the original request URI.
Keep a placeholder backend to satisfy the Ingress schema; it is never used because the custom snippet overrides the upstream.

Verify – Confirm that traffic is now correctly split

1. Ingress controller reload


$ kubectl rollout restart deployment -n ingress-nginx ingress-nginx-controller
deployment.apps/ingress-nginx-controller restarted

2. Functional test with both variants


$ curl -s -H "X-Test-Variant: v1" https://search.example.com/search?q=alpha | jq '.hits.total'
{
  "value": 1245,
  "relation": "eq"
}
$ curl -s -H "X-Test-Variant: v2" https://search.example.com/search?q=alpha | jq '.hits.total'
{
  "value": 987,
  "relation": "eq"
}

The two calls return different hits.total values, matching the data seeded in search‑v1 and search‑v2 respectively.

3. Ingress logs after fix


2024-10-12T09:02:11Z "GET /search?q=alpha HTTP/1.1" 200 1245 "-" "curl/7.68.0" upstream: "search-v2.default.svc.cluster.local:9200"
2024-10-12T09:02:13Z "GET /search?q=beta HTTP/1.1" 200 987 "-" "curl/7.68.0" upstream: "search-v1.default.svc.cluster.local:9200"

The upstream now reflects the variant header.

Prevent – Guardrails and monitoring to avoid regression

Ingress validation CI: Add a Helm test that sends a request with X‑Test‑Variant: v2 and asserts the response originates from search‑v2 (e.g., by checking a custom header added by the downstream service).
Metric alert: Export a Prometheus metric nginx_ingress_upstream_response_time_seconds{upstream="search-v1"} and create an alert if the proportion of search‑v2 traffic drops below the expected split (e.g., 45‑55%).
Configuration drift detection: Use kubectl diff or Argo CD health checks to ensure the server-snippet map remains present after any Helm upgrade.
Documentation: Store the routing logic in a version‑controlled ConfigMap and reference it in the Ingress annotation to make future changes auditable.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why did the rewrite‑target annotation break the routing?
The rewrite stripped the original path, causing the downstream service to default to its primary index (search‑v1). Removing the rewrite preserves the intended URI, allowing the header‑based map to select the correct backend.
Can I use a separate Ingress resource per variant instead of header routing?
Yes, but it introduces DNS or host‑based complexity and doubles the number of Ingress objects. Header‑based routing keeps a single entry point and scales better for additional variants.
What if the client does not send X‑Test‑Variant?
The map defaults to search‑v1. Ensure your A/B test harness injects the header for all traffic; otherwise, the request will fall back to the control variant.
Is the placeholder backend required for all Ingress controllers?
For nginx‑ingress the backend.service field is mandatory in the spec, even when a custom proxy_pass overrides it. The placeholder can be a dummy service that returns 404; it will never be hit.
How do I debug similar routing issues with other protocols (e.g., gRPC)?
Replace the HTTP header map with a grpc_set_header directive or use nginx.ingress.kubernetes.io/backend-protocol: "GRPC" and route based on grpc metadata. The investigation steps—log inspection, upstream verification, and configuration diff—remain the same.