Milvus DNS resolution failure during A/B testing deployment

Problem: Milvus DNS resolution failure during A/B testing deployment

During an A/B testing rollout, two versions of a recommendation model query Milvus simultaneously. The Milvus client libraries intermittently report DNS resolution failed, preventing connections to the Milvus service endpoints (milvus-frontend, milvus-data, etc.). The failure appears only in the A/B testing namespace and manifests as increased request latency, time‑outs, and occasional pod restarts.

Root Cause

The underlying cause is a combination of:

  • Kubernetes CoreDNS cache saturation: The A/B testing namespace creates 100+ short‑lived pods per minute, each performing a DNS lookup for the same Milvus service FQDN. CoreDNS default cache size (1000 entries) fills quickly, causing cache evictions and increased lookup latency.
  • Incorrect dnsPolicy on the test pods: Pods inherit the node’s DNS configuration (ClusterFirst) but the test runner overrides it with dnsPolicy: Default, forcing the pods to use the host’s /etc/resolv.conf, which points to an external DNS server that does not have the .svc.cluster.local zone.
  • Service name mismatch in the A/B test manifest: The canary deployment references the service as milvus-frontend-canary, while the actual Service object is still named milvus-frontend. CoreDNS receives a query for a non‑existent name, returns NXDOMAIN, and the client interprets it as a resolution failure.

Debug: Investigation and Diagnostics

  1. Inspect pod DNS configuration

    kubectl exec -n ab-test test-pod-abc123 -- cat /etc/resolv.conf
    

    nameserver 8.8.8.8
    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5

    The presence of an external nameserver (8.8.8.8) confirms dnsPolicy: Default is in effect.

  2. Check CoreDNS logs for cache pressure

    kubectl logs -n kube-system -l k8s-app=kube-dns -c coredns --tail=100
    

    [INFO] 2024-10-12T14:22:31Z cache: cache size 1024, entries 1023, hits 15000, misses 3000
    [WARN] 2024-10-12T14:22:35Z cache: eviction triggered, cache full
    [ERROR] 2024-10-12T14:22:36Z forward: upstream DNS server timed out

  3. Validate service name resolution from a test pod

    kubectl run -i --tty dns-test --image=busybox --restart=Never --rm -- nslookup milvus-frontend-canary.default.svc.cluster.local
    

    Server: 10.96.0.10
    Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
    **server can’t find milvus-frontend-canary.default.svc.cluster.local: NXDOMAIN**

    Switching to the correct name resolves successfully:

    nslookup milvus-frontend.default.svc.cluster.local
    

    Name: milvus-frontend.default.svc.cluster.local
    Address: 10.244.2.5

  4. Monitor DNS query latency

    kubectl exec -n kube-system -c coredns $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath="{.items[0].metadata.name}") -- \
      curl -s http://localhost:9153/metrics | grep coredns_dns_request_duration_seconds_bucket
    

    coredns_dns_request_duration_seconds_bucket{le=”0.025″} 12000
    coredns_dns_request_duration_seconds_bucket{le=”0.05″} 15000
    coredns_dns_request_duration_seconds_bucket{le=”0.1″} 18000
    coredns_dns_request_duration_seconds_bucket{le=”+Inf”} 25000

    Notice a shift toward higher latency buckets after the test deployment starts.

Solution: Configuration Corrections and CoreDNS Tuning

1. Align pod DNS policy with cluster DNS

Update the A/B test Deployment manifest to use dnsPolicy: ClusterFirst (or omit the field, as it defaults to ClusterFirst).

# Before
apiVersion: apps/v1
kind: Deployment
metadata:
  name: recommendation-test
spec:
  template:
    spec:
      dnsPolicy: Default
      containers:
      - name: model
        image: registry.example.com/reco:test
# After
apiVersion: apps/v1
kind: Deployment
metadata:
  name: recommendation-test
spec:
  template:
    spec:
      # dnsPolicy omitted – defaults to ClusterFirst
      containers:
      - name: model
        image: registry.example.com/reco:test

2. Correct service name in canary manifest

Replace the erroneous service reference with the actual Service name.

# Before (canary manifest)
apiVersion: v1
kind: Service
metadata:
  name: milvus-frontend-canary
spec:
  selector:
    app: milvus-frontend
# After
apiVersion: v1
kind: Service
metadata:
  name: milvus-frontend
spec:
  selector:
    app: milvus-frontend

3. Increase CoreDNS cache size and enable loop plugin to reduce eviction churn

Edit the coredns ConfigMap:

# Before
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 1000
        loop
        reload
        loadbalance
    }
# After (increase cache, add ttl)
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 5000 ttl 30
        loop
        reload
        loadbalance
    }

Apply the changes and roll the CoreDNS deployment:

kubectl apply -f coredns-configmap.yaml
kubectl rollout restart deployment coredns -n kube-system

4. Throttle DNS query rate from test pods

Introduce a sidecar DNS cache (e.g., dnsmasq) or configure the client SDK to reuse connections instead of creating a new resolver per request.

Verify: Post‑fix Validation Steps

  1. Confirm pod DNS policy:

    kubectl get pod -n ab-test -l app=recommendation-test -o jsonpath="{.items[0].spec.dnsPolicy}"
    

    Expected output: ClusterFirst

  2. Validate service name resolution:

    kubectl exec -n ab-test test-pod-xyz -- nslookup milvus-frontend.default.svc.cluster.local
    

    Expected output contains an Address line with the Milvus service IP.

  3. Check CoreDNS cache metrics for reduced evictions:

    kubectl exec -n kube-system -c coredns $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath="{.items[0].metadata.name}") -- \
      curl -s http://localhost:9153/metrics | grep cache_evictions
    

    Metric should stay at 0 during a sustained load test.

  4. Run an end‑to‑end load test against Milvus (e.g., hey or wrk) and verify latency stays below the SLA (e.g., p99 < 200ms).

Prevent: Operational Guardrails

  • Enforce DNS policy via admission controller: Reject pods that set dnsPolicy: Default in production namespaces.
  • Automated service name linting: CI pipeline checks that service references match existing Service objects.
  • CoreDNS scaling policy: Deploy a HorizontalPodAutoscaler for CoreDNS based on coredns_cache_size and coredns_dns_request_duration_seconds metrics.
  • Metrics alerting: Create alerts for
    • CoreDNS cache eviction rate > 5/min
    • DNS lookup latency > 100ms for milvus-* services
    • Pod restarts due to DNS_PROBE_FINISHED_NXDOMAIN errors
  • Sidecar DNS cache for high‑frequency test workloads: Deploy dnsmasq as a lightweight sidecar to reduce CoreDNS pressure.

FAQ

  1. Why does the DNS failure only appear during A/B testing and not in production?
    The A/B test spawns many short‑lived pods that each perform a fresh DNS lookup. Production services use long‑lived pods with cached entries, keeping CoreDNS cache well below its limit.
  2. Can I keep dnsPolicy: Default and still resolve .svc.cluster.local names?
    Only if you manually add the cluster DNS server (e.g., 10.96.0.10) to /etc/resolv.conf. Using ClusterFirst is the recommended and less error‑prone approach.
  3. Is increasing the CoreDNS cache size enough?
    It mitigates cache pressure but does not address the root cause of incorrect DNS policy and service name mismatches. Both must be fixed.
  4. How do I know if a DNS query is being served from cache or forwarded upstream?
    Enable the log plugin in CoreDNS and look for the cache tag in the log line, e.g., cache: answer from cache. Metrics coredns_cache_hits_total vs coredns_cache_misses_total also indicate cache usage.
  5. Will adding a dnsmasq sidecar affect TLS termination or other middleware?
    No. The sidecar only intercepts DNS queries on 127.0.0.1:53. Application traffic continues to flow unchanged; ensure the pod’s dnsConfig.nameservers points to the sidecar.

Related Topic Hub: Vector Databases Troubleshooting Hub