Problem: Milvus DNS resolution failure during A/B testing deployment
During an A/B testing rollout, two versions of a recommendation model query Milvus simultaneously. The Milvus client libraries intermittently report DNS resolution failed, preventing connections to the Milvus service endpoints (milvus-frontend, milvus-data, etc.). The failure appears only in the A/B testing namespace and manifests as increased request latency, time‑outs, and occasional pod restarts.
Root Cause
The underlying cause is a combination of:
- Kubernetes CoreDNS cache saturation: The A/B testing namespace creates
100+short‑lived pods per minute, each performing a DNS lookup for the same Milvus service FQDN. CoreDNS default cache size (1000 entries) fills quickly, causing cache evictions and increased lookup latency. - Incorrect
dnsPolicyon the test pods: Pods inherit the node’s DNS configuration (ClusterFirst) but the test runner overrides it withdnsPolicy: Default, forcing the pods to use the host’s/etc/resolv.conf, which points to an external DNS server that does not have the.svc.cluster.localzone. - Service name mismatch in the A/B test manifest: The canary deployment references the service as
milvus-frontend-canary, while the actual Service object is still namedmilvus-frontend. CoreDNS receives a query for a non‑existent name, returnsNXDOMAIN, and the client interprets it as a resolution failure.
Debug: Investigation and Diagnostics
-
Inspect pod DNS configuration
kubectl exec -n ab-test test-pod-abc123 -- cat /etc/resolv.confnameserver 8.8.8.8
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5The presence of an external nameserver (8.8.8.8) confirms
dnsPolicy: Defaultis in effect. -
Check CoreDNS logs for cache pressure
kubectl logs -n kube-system -l k8s-app=kube-dns -c coredns --tail=100[INFO] 2024-10-12T14:22:31Z cache: cache size 1024, entries 1023, hits 15000, misses 3000
[WARN] 2024-10-12T14:22:35Z cache: eviction triggered, cache full
[ERROR] 2024-10-12T14:22:36Z forward: upstream DNS server timed out -
Validate service name resolution from a test pod
kubectl run -i --tty dns-test --image=busybox --restart=Never --rm -- nslookup milvus-frontend-canary.default.svc.cluster.localServer: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
**server can’t find milvus-frontend-canary.default.svc.cluster.local: NXDOMAIN**Switching to the correct name resolves successfully:
nslookup milvus-frontend.default.svc.cluster.localName: milvus-frontend.default.svc.cluster.local
Address: 10.244.2.5 -
Monitor DNS query latency
kubectl exec -n kube-system -c coredns $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath="{.items[0].metadata.name}") -- \ curl -s http://localhost:9153/metrics | grep coredns_dns_request_duration_seconds_bucketcoredns_dns_request_duration_seconds_bucket{le=”0.025″} 12000
coredns_dns_request_duration_seconds_bucket{le=”0.05″} 15000
coredns_dns_request_duration_seconds_bucket{le=”0.1″} 18000
coredns_dns_request_duration_seconds_bucket{le=”+Inf”} 25000Notice a shift toward higher latency buckets after the test deployment starts.
Solution: Configuration Corrections and CoreDNS Tuning
1. Align pod DNS policy with cluster DNS
Update the A/B test Deployment manifest to use dnsPolicy: ClusterFirst (or omit the field, as it defaults to ClusterFirst).
# Before
apiVersion: apps/v1
kind: Deployment
metadata:
name: recommendation-test
spec:
template:
spec:
dnsPolicy: Default
containers:
- name: model
image: registry.example.com/reco:test
# After
apiVersion: apps/v1
kind: Deployment
metadata:
name: recommendation-test
spec:
template:
spec:
# dnsPolicy omitted – defaults to ClusterFirst
containers:
- name: model
image: registry.example.com/reco:test
2. Correct service name in canary manifest
Replace the erroneous service reference with the actual Service name.
# Before (canary manifest)
apiVersion: v1
kind: Service
metadata:
name: milvus-frontend-canary
spec:
selector:
app: milvus-frontend
# After
apiVersion: v1
kind: Service
metadata:
name: milvus-frontend
spec:
selector:
app: milvus-frontend
3. Increase CoreDNS cache size and enable loop plugin to reduce eviction churn
Edit the coredns ConfigMap:
# Before
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 1000
loop
reload
loadbalance
}
# After (increase cache, add ttl)
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 5000 ttl 30
loop
reload
loadbalance
}
Apply the changes and roll the CoreDNS deployment:
kubectl apply -f coredns-configmap.yaml
kubectl rollout restart deployment coredns -n kube-system
4. Throttle DNS query rate from test pods
Introduce a sidecar DNS cache (e.g., dnsmasq) or configure the client SDK to reuse connections instead of creating a new resolver per request.
Verify: Post‑fix Validation Steps
-
Confirm pod DNS policy:
kubectl get pod -n ab-test -l app=recommendation-test -o jsonpath="{.items[0].spec.dnsPolicy}"Expected output:
ClusterFirst -
Validate service name resolution:
kubectl exec -n ab-test test-pod-xyz -- nslookup milvus-frontend.default.svc.cluster.localExpected output contains an
Addressline with the Milvus service IP. -
Check CoreDNS cache metrics for reduced evictions:
kubectl exec -n kube-system -c coredns $(kubectl get pod -n kube-system -l k8s-app=kube-dns -o jsonpath="{.items[0].metadata.name}") -- \ curl -s http://localhost:9153/metrics | grep cache_evictionsMetric should stay at
0during a sustained load test. -
Run an end‑to‑end load test against Milvus (e.g.,
heyorwrk) and verify latency stays below the SLA (e.g.,p99 < 200ms).
Prevent: Operational Guardrails
- Enforce DNS policy via admission controller: Reject pods that set
dnsPolicy: Defaultin production namespaces. - Automated service name linting: CI pipeline checks that service references match existing Service objects.
- CoreDNS scaling policy: Deploy a HorizontalPodAutoscaler for CoreDNS based on
coredns_cache_sizeandcoredns_dns_request_duration_secondsmetrics. - Metrics alerting: Create alerts for
- CoreDNS cache eviction rate > 5/min
- DNS lookup latency > 100ms for
milvus-*services - Pod restarts due to
DNS_PROBE_FINISHED_NXDOMAINerrors
- Sidecar DNS cache for high‑frequency test workloads: Deploy
dnsmasqas a lightweight sidecar to reduce CoreDNS pressure.
FAQ
- Why does the DNS failure only appear during A/B testing and not in production?
The A/B test spawns many short‑lived pods that each perform a fresh DNS lookup. Production services use long‑lived pods with cached entries, keeping CoreDNS cache well below its limit. - Can I keep
dnsPolicy: Defaultand still resolve.svc.cluster.localnames?
Only if you manually add the cluster DNS server (e.g.,10.96.0.10) to/etc/resolv.conf. UsingClusterFirstis the recommended and less error‑prone approach. - Is increasing the CoreDNS cache size enough?
It mitigates cache pressure but does not address the root cause of incorrect DNS policy and service name mismatches. Both must be fixed. - How do I know if a DNS query is being served from cache or forwarded upstream?
Enable thelogplugin in CoreDNS and look for thecachetag in the log line, e.g.,cache: answer from cache. Metricscoredns_cache_hits_totalvscoredns_cache_misses_totalalso indicate cache usage. - Will adding a dnsmasq sidecar affect TLS termination or other middleware?
No. The sidecar only intercepts DNS queries on127.0.0.1:53. Application traffic continues to flow unchanged; ensure the pod’sdnsConfig.nameserverspoints to the sidecar.
Related Topic Hub: Vector Databases Troubleshooting Hub