red and blue cargo ship on sea during daytime

Docker container DNS resolution fails intermittently in multi-region Kubernetes

by

Problem – Intermittent DNS Resolution Failures in Docker Containers Across Multi‑Region Kubernetes

AI inference services deployed as Docker containers on Amazon EKS clusters in several AWS regions (e.g., us-east-1, eu-central-1, ap-south-1) experience sporadic failures when resolving external endpoints such as model registries, third‑party APIs, or telemetry services. The failures manifest as:

  • Go errors: dial tcp: lookup api.example.com: no such host
  • Node.js/Python errors: getaddrinfo ENOTFOUND api.example.com
  • Container logs: lookup api.example.com on 127.0.0.11:53 failed: read udp 127.0.0.11:53: i/o timeout

The issue is intermittent: a pod may resolve the same hostname successfully seconds later, but a burst of parallel requests (e.g., batch inference jobs) can trigger timeouts across many pods in a region. The impact includes delayed model loading, failed health checks, and cascading back‑pressure on the global load balancer.

Root Cause – Interaction of Docker’s Embedded DNS, CoreDNS Cache, and AWS VPC Resolver Limits

Three tightly coupled mechanisms create the failure window:

  1. Docker’s embedded DNS (127.0.0.11) forwards queries to the node’s /etc/resolv.conf nameserver list. When the host’s resolver is 169.254.169.253 (AWS metadata DNS), Docker injects that address into each container’s /etc/resolv.conf (Docker Engine docs).
  2. CoreDNS cache plugin (default in EKS) holds recent answers. Under rapid scaling, cache entries are evicted and CoreDNS must forward queries upstream. High QPS (>10 k queries/s) can saturate the VPC resolver, which throttles at roughly 10 k queries/s per ENI (EKS VPC CNI guide).
  3. AWS VPC resolver instability during inter‑region traffic spikes (e.g., PrivateLink calls) can cause UDP timeouts (GitHub issue). When the resolver drops packets, Docker’s forwarder receives no reply, and the container reports the “read udp … i/o timeout” error.

The combination of a rotating /etc/resolv.conf on the host (see GitHub issue #4532) and CoreDNS cache churn creates a narrow window where containers query an overloaded or unreachable upstream DNS server, leading to the observed intermittent failures.

Debug – Systematic Investigation Steps

1. Capture the symptom inside a failing pod

kubectl exec -it $(kubectl get pod -l app=ai-model -n prod -o name | head -n1) -- sh -c 'cat /etc/resolv.conf'

Typical output during failure:

# Generated by Docker Desktop
nameserver 127.0.0.11
options ndots:0

Inspect the Docker daemon’s view of the host resolver:

cat /etc/resolv.conf

During a spike you may see:

# Generated by DHCP client
nameserver 169.254.169.253
options timeout:2 attempts:3

2. Verify CoreDNS health

kubectl -n kube-system get pods -l k8s-app=kube-dns

Check logs for timeouts or cache evictions:

kubectl -n kube-system logs -l k8s-app=kube-dns -c coredns --tail=100 | grep -i timeout

Example log line from the June 2024 incident:

2024-06-12T14:23:07Z [error] plugin/cache: cache miss for "api.example.com." (type A) - forwarding

3. Measure VPC resolver latency

kubectl exec -it $(kubectl get pod -l app=ai-model -n prod -o name | head -n1) -- dig @169.254.169.253 api.example.com +time=2 +tries=1

Typical successful response:

; <<>> DiG 9.16.1-Ubuntu <<>> @169.254.169.253 api.example.com +time=2 +tries=1
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; QUERY TIME: 32 msec

During a failure you may see:

; <<>> DiG 9.16.1-Ubuntu <<>> @169.254.169.253 api.example.com +time=2 +tries=1
;; connection timed out; no servers could be reached

4. Correlate scaling events with DNS metrics

In CloudWatch, chart the coredns_forward_requests_total and coredns_cache_hits_total counters. Spikes in forward_requests_total coinciding with node‑autoscaling events confirm cache eviction pressure.

5. Reproduce the race condition locally

On a test node, restart the Docker daemon while a container is performing DNS lookups:

# In one terminal
docker run --rm -it alpine sh -c 'while true; do nslookup api.example.com; sleep 0.1; done'

# In another terminal (after a few seconds)
sudo systemctl restart docker

Observe intermittent “connection refused” or “i/o timeout” messages, mirroring the production pattern described in the Stack Overflow answer (73582112).

Solution – Stabilize DNS Path and Harden CoreDNS Cache

1. Pin the node’s /etc/resolv.conf to a reliable DNS server

Configure the EKS node group to use AmazonProvidedDNS (172.20.0.10) instead of the metadata resolver:

# user-data for the launch template
cat <<'EOF' > /etc/dhcp/dhclient.conf
supersede domain-name-servers 172.20.0.10;
EOF
systemctl restart dhclient

After the change, cat /etc/resolv.conf on the host should show:

# Generated by DHCP client
nameserver 172.20.0.10
options timeout:2 attempts:3

2. Override Docker’s DNS configuration for the daemon

Add a /etc/docker/daemon.json entry to force a static upstream DNS (e.g., the VPC resolver) and disable the automatic injection of the host’s resolv.conf:

{
  "dns": ["172.20.0.10"],
  "dns-opts": ["ndots:2"],
  "dns-search": ["svc.cluster.local", "cluster.local"]
}

Restart Docker:

sudo systemctl restart docker

Resulting container /etc/resolv.conf now contains the static nameserver, eliminating the race with host DHCP updates.

3. Tune CoreDNS cache and forward plugins

Update the CoreDNS ConfigMap (coredns in kube-system) to increase cache TTL and limit forward QPS:

kubectl -n kube-system edit configmap coredns

Replace the Corefile section with:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    cache 30 {
        success 9980
        denial 9980
        maxsize 1000000
        ttl 300
    }
    loop
    reload
    loadbalance
    forward . 172.20.0.10 {
        max_fails 3
        max_concurrent 100
        policy random
    }
}

Key changes:

  • cache TTL 300s reduces forward load during rapid scaling.
  • max_concurrent 100 throttles outbound queries, preventing VPC resolver overload.
  • policy random spreads queries across multiple ENI DNS resolvers when available.

4. Deploy a sidecar DNS cache (optional for high‑QPS workloads)

Run dnsmasq as a DaemonSet that caches external lookups locally, pointing containers to 127.0.0.1 instead of CoreDNS. This eliminates the UDP round‑trip to the VPC resolver for repeated external domains.

5. Apply a PodDisruptionBudget for CoreDNS

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coredns-pdb
  namespace: kube-system
spec:
  minAvailable: 2
  selector:
    matchLabels:
      k8s-app: kube-dns

Ensures at least two CoreDNS replicas stay alive during node autoscaling, reducing eviction‑induced cache misses.

Verify – Confirming the Fix

  1. Check container resolv.conf – it should now list the static VPC DNS only:
    2. nameserver 172.20.0.10
options ndots:2
  2. Run a sustained DNS load test inside a pod:
    3. kubectl run dnsload -i --rm --image=alpine --restart=Never -- sh -c "while true; do dig @172.20.0.10 api.example.com +short; sleep 0.05; done"

    Observe stable responses without timeouts for at least 10 minutes.

  3. Validate CoreDNS metrics – coredns_cache_hits_total should increase, and coredns_forward_requests_total should stay flat during scaling events.
  4. Monitor CloudWatch alarms – ensure no new “i/o timeout” errors appear in container logs after the next autoscaling wave.

Prevent – Operational Guardrails and Ongoing Monitoring

  • Enforce static DNS for nodes via launch template or Managed Node Group settings; disable DHCP overrides.
  • Set a CloudWatch metric alarm on coredns_forward_requests_total that triggers when QPS exceeds 8 k, indicating potential resolver saturation.
  • Enable CoreDNS health checks (kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide) and configure a PodDisruptionBudget as shown.
  • Periodically audit Docker daemon DNS config after OS patches; a regression can re‑introduce the host resolv.conf race.
  • Consider regional DNS forwarders (e.g., Route 53 Resolver endpoints) to reduce cross‑region latency and avoid the single VPC resolver bottleneck.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does DNS resolution fail only under heavy load?
    Because CoreDNS cache entries are evicted quickly, forcing forward queries to the VPC resolver, which throttles at ~10 k QPS per ENI. The resulting UDP timeouts surface as “i/o timeout” errors.
  2. Can I keep Docker’s default 127.0.0.11 DNS and still fix the issue?
    Yes, but you must ensure the host’s /etc/resolv.conf points to a stable DNS (e.g., 172.20.0.10) and that Docker’s daemon JSON pins that server. Otherwise the embedded DNS will inherit transient host changes.
  3. Do I need to restart all running pods after changing CoreDNS?
    CoreDNS config changes are applied instantly to new queries. Existing pods will pick up the new behavior without restart, but flushing the local DNS cache (e.g., restarting the container) helps clear stale negative entries.
  4. What is the impact of increasing the CoreDNS cache TTL?
    A higher TTL reduces forward traffic and VPC resolver load, but stale records may persist longer after DNS changes. For AI services that query stable third‑party APIs, a 300 s TTL is a safe trade‑off.
  5. How can I detect when the VPC resolver becomes unreachable?
    Enable a CloudWatch metric filter on container logs for the pattern “read udp .*:53: i/o timeout”. Trigger an alarm to open a ticket before the issue spreads.