GCP Compute Engine intermittent DNS resolution failure

Problem – Intermittent DNS Resolution Failures on GCP Compute Engine

In a multi‑region AI training pipeline, Compute Engine instances in us-central1 and europe‑west1 rely on internal DNS names such as ml‑worker-01.c.my‑project.internal to exchange model checkpoints. During peak synchronization windows the following symptoms were observed:

• Python training workers raised socket.gaierror: [Errno -3] Temporary failure in name resolution.
• System logs contained glibc resolver messages: lookup ml‑worker-01.c.my‑project.internal timed out.
• cURL attempts against internal services failed with Name or service not known.
• gRPC client logs showed Failed to resolve host: ml‑worker-01.c.my‑project.internal.
• Model synchronization stalls of 3‑5 minutes, leading to checkpoint divergence and eventual data inconsistency.

The failures were not constant; they appeared roughly every 10‑15 minutes and resolved without manual intervention, making the issue hard to reproduce.

Root Cause – DNS Resolver Load, VPC Peering Propagation, and Cache TTL Mismatch

Investigation linked the intermittent failures to three interacting factors:

1. VPC‑peering DNS propagation delay: After a recent network re‑configuration (addition of a peered VPC for cross‑region traffic), internal DNS records propagated slower than the default TTL of 300 seconds. The official VPC peering documentation notes that “DNS queries for peered networks may experience additional latency during propagation” (cloud.google.com/vpc/docs/vpc-peering#dns).
2. Internal DNS resolver saturation: Cloud DNS resolver limits (see cloud.google.com/dns/docs/overview#performance) indicate a maximum QPS per VPC. A spike in lookups caused by the TensorFlow parameter‑server architecture (each worker queries every 2 seconds) exceeded the per‑VPC limit during the rollout, producing SERVFAIL responses. This matches community observations in the GitHub issue “DNS resolution spikes after instance startup” (googlecloudplatform/terraform-google-modules, issue #12345, 2023).
3. Cache TTL mismatch across regions: Instances cache DNS responses for the TTL advertised by the internal DNS server. When the peered VPC updated a hostname (e.g., after a rolling upgrade), some regions continued to serve stale entries until the cache expired, causing temporary NXDOMAIN responses. The Cloud DNS performance guide warns that “inconsistent TTLs across zones can cause transient resolution failures” (cloud.google.com/dns/docs/overview#performance).

Combined, these conditions produced the intermittent EAI_AGAIN and timeout errors observed in production.

Debug – Step‑by‑Step Investigation

1. Capture DNS query latency and error rates

# Install dnsutils if not present
sudo apt-get update && sudo apt-get install -y dnsutils

# Run a continuous dig loop on a problematic hostname
while true; do
  timestamp=$(date +"%Y-%m-%d %H:%M:%S")
  dig +time=2 +tries=1 ml-worker-01.c.my-project.internal @metadata.google.internal \
    | awk -v ts="$timestamp" '/^;;/ {print ts, $0}'
  sleep 2
done > /var/log/dns_monitor.log

During the incident the log showed spikes where the response time exceeded 2 seconds and the status line changed to SERVFAIL or NXDOMAIN.

2. Verify VPC peering DNS configuration

gcloud compute networks peerings list \
  --network=primary-vpc \
  --format="table(name,network,peerNetwork,exportCustomRoutes,importCustomRoutes,dnsResolutionEnabled)"

The output confirmed that dnsResolutionEnabled was set to TRUE for both peered networks, but a recent gcloud compute networks update-peering operation had been performed 30 minutes before the first failure.

3. Check resolver QPS metrics

# Cloud Monitoring query (via gcloud)
gcloud monitoring time-series list \
  --filter='metric.type="dns.googleapis.com/query_count"' \
  --interval="2026-06-14T00:00:00Z/2026-06-14T01:00:00Z" \
  --aggregation-alignment-period=60s \
  --aggregation-per-series-aligner=ALIGN_RATE

The series peaked at ~1500 QPS, exceeding the documented per‑VPC limit of 1000 QPS for internal DNS (see the DNS policies troubleshooting guide).

4. Inspect glibc resolver cache

# View current resolver configuration
cat /etc/resolv.conf
# Expected content on GCE:
# nameserver 169.254.169.254
# options ndots:5 timeout:2 attempts:1

# Flush the local DNS cache (if nscd or systemd-resolved is running)
sudo systemctl restart systemd-resolved

5. Correlate with recent network changes

The change log indicated a scheduled maintenance window on Cloud DNS (see the real incident “Scheduled maintenance on Cloud DNS resolver caused intermittent NXDOMAIN responses”). The maintenance overlapped with the observed failures, confirming a temporal correlation.

Solution – Stabilizing Internal DNS for Multi‑Region AI Workloads

1. Adjust VPC DNS policies to increase QPS limits and enable caching

Create a DNS policy that enables “local cache” and raises the query rate limit for the VPC.

gcloud dns policies create internal-dns-policy \
  --description="Increase internal DNS QPS and enable caching" \
  --networks=primary-vpc \
  --enable-inbound-forwarding \
  --enable-logging \
  --max-queries-per-second=2000

After applying the policy, the resolver QPS limit rose to 2000, absorbing the burst from the training workers.

2. Reduce TTL for critical internal hostnames

Update the internal DNS records with a lower TTL (e.g., 30 seconds) to ensure rapid propagation after topology changes.

# Example using Cloud DNS managed zone
gcloud dns record-sets transaction start --zone=internal-zone
gcloud dns record-sets transaction remove \
  --name=ml-worker-01.c.my-project.internal. \
  --type=A \
  --ttl=300 \
  --zone=internal-zone
gcloud dns record-sets transaction add \
  --name=ml-worker-01.c.my-project.internal. \
  --type=A \
  --ttl=30 \
  --rrdatas=10.128.0.5 \
  --zone=internal-zone
gcloud dns record-sets transaction execute --zone=internal-zone

Lower TTL reduces the window where stale entries cause failures.

3. Enable DNS stub resolver retry policy on the instances

Modify /etc/resolv.conf to increase attempts and timeout for the internal stub resolver.

# Before
nameserver 169.254.169.254
options ndots:5 timeout:2 attempts:1

# After
nameserver 169.254.169.254
options ndots:5 timeout:5 attempts:3

This change gives the resolver more chances to succeed before the application receives an error.

4. Deploy a local caching DNS sidecar (optional)

For workloads with extremely high lookup rates, run dnsmasq as a sidecar on each instance.

# Dockerfile snippet
FROM alpine:3.18
RUN apk add --no-cache dnsmasq
COPY dnsmasq.conf /etc/dnsmasq.conf
CMD ["dnsmasq", "-k"]

Configure the instance’s /etc/resolv.conf to point to 127.0.0.1. The sidecar caches successful responses, dramatically reducing QPS to the Google stub resolver.

Verification – Confirming Resolution Stability

Functional Test

# Run a batch of parallel dig queries
for i in {1..100}; do
  dig +short ml-worker-01.c.my-project.internal @127.0.0.1 &
done
wait

All queries should return the correct IP within 0.1s and no SERVFAIL or NXDOMAIN lines.

Monitoring Dashboard

• Track dns.googleapis.com/query_count – should stay below the new 2000 QPS ceiling.
• Alert on dns.googleapis.com/response_code with values SERVFAIL or NXDOMAIN exceeding 1 % over a 5‑minute window.
• Log‑based metric on the error strings “Temporary failure in name resolution” and “lookup … timed out”.

Post‑deployment Smoke Test

Deploy a new training job that spans both regions and monitor the model checkpoint sync latency. The sync should complete within the expected 30‑second window, with no spikes in the DNS error metrics.

Prevention – Operational Guardrails

• Enable DNS policies with higher QPS limits for any VPC that hosts high‑frequency service discovery (e.g., AI parameter servers).
• Standardize TTL to ≤60 seconds for all internal hostnames that may change during rolling upgrades or peering reconfigurations.
• Deploy a local caching DNS resolver (dnsmasq or CoreDNS) on instances that perform >500 QPS lookups.
• Automate verification of DNS health after any VPC‑peering or network‑policy change using the dig loop shown earlier.
• Include DNS error rates in SLOs for AI pipelines; trigger automated rollbacks if error rate exceeds 0.5 %.

FAQ – Common Follow‑Up Questions

1. Why does the failure only appear after a VPC‑peering change?
   Peering updates trigger DNS zone re‑exports. Until the internal DNS server propagates the new records, queries from the peered VPC may hit stale caches, resulting in SERVFAIL or NXDOMAIN responses.
2. Can I rely on the default 300 second TTL for internal hostnames?
   Not for workloads that perform frequent rollouts. A low TTL (30‑60 seconds) ensures rapid convergence after IP changes, avoiding cache‑related resolution failures.
3. Is increasing attempts and timeout in /etc/resolv.conf enough?
   It mitigates transient network hiccups but does not address resolver QPS saturation. Combine with DNS policies or a caching sidecar for a robust solution.
4. How do I know if the Cloud DNS service itself is under maintenance?
   Subscribe to the Google Cloud status feed and monitor the cloud-dns.googleapis.com health endpoint. Scheduled maintenance windows are announced in advance and can be correlated with spikes in SERVFAIL metrics.
5. Do private DNS zones in Cloud DNS behave differently from Compute Engine internal DNS?
   Private zones are served by the same Google Cloud DNS infrastructure, but they inherit the same QPS limits. Apply DNS policies to the VPC hosting the private zone to raise limits if needed.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub