a purple and black background with a purple and black logo

PyTorch benchmark DNS resolution failure on remote object storage

by

Problem

When running distributed PyTorch performance benchmarks on a multi‑node GPU cluster, the benchmark processes abort with errors such as:


2026-10-24 10:12:45,321 - torch.distributed.elastic.multiprocessing.api - ERROR - Failed to resolve hostname "datasets.example.com": Name or service not known
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 112, in _launch
    torchrun_main()
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 78, in torchrun_main
    main()
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 215, in main
    args = parser.parse_args()
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 172, in _parse_args
    raise ValueError(f"Could not resolve {args.master_addr}")
ValueError: Could not resolve master_addr: "metrics.prometheus.internal"

Similar DNS failures appear when the benchmark attempts to download training data from an object storage bucket (e.g., s3://datasets.example.com/imagenet) or when it pushes metrics to a Prometheus pushgateway.

Root Cause

The failure originates from the cluster’s DNS resolution path:

  • Node‑level resolv.conf points to an internal DNS server that is unreachable from the compute nodes due to a recent network policy change.
  • Kubernetes pod DNSPolicy is set to Default, causing pods to inherit the host’s broken resolver instead of the cluster DNS (kube-dns/CoreDNS).
  • Object‑storage SDK (e.g., boto3) performs a synchronous getaddrinfo() call; when the resolver times out (default 5 s), the SDK raises a EndpointConnectionError, which propagates up to the PyTorch data loader.
  • Prometheus pushgateway address is defined as a short name (pushgw) without a search domain, so the resolver appends the node’s search suffix from /etc/resolv.conf. The suffix is stale, leading to an NXDOMAIN response.

In short, the DNS configuration inside the benchmark containers does not match the expected cluster DNS topology, causing all external hostname lookups to fail.

Debug

Follow these steps to isolate the failure:

1. Inspect container DNS configuration

# Inside a benchmark pod
cat /etc/resolv.conf


# Expected (Cluster DNS)
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

If the file shows the host’s DNS server (e.g., nameserver 192.168.1.1) the pod is not using the cluster DNS.

2. Verify DNS server reachability


# From the node
nslookup kube-dns.kube-system.svc.cluster.local 10.96.0.10


Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kube-dns.kube-system.svc.cluster.local
Address: 10.96.0.10

If the query times out or returns SERVFAIL, the DNS service itself is unreachable.

3. Test external hostname resolution


nslookup datasets.example.com
dig +short datasets.example.com


;; connection timed out; no servers could be reached

A timeout indicates that the resolver cannot reach the upstream DNS (often the corporate DNS forwarder).

4. Check network policies / firewall rules


# List iptables rules that might block UDP/53
iptables -L -v -n | grep 53

Look for DROP rules targeting the node’s network interface.

5. Review pod spec for DNSPolicy


apiVersion: v1
kind: Pod
metadata:
  name: benchmark
spec:
  dnsPolicy: Default   # <-- problematic
  containers:
  - name: runner
    image: myorg/pytorch-bench:latest

6. Examine SDK error logs


2026-10-24 10:13:02,112 - boto3 - ERROR - Unable to locate credentials
Traceback (most recent call last):
  File ".../boto3/session.py", line 123, in _get_credentials
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
...
2026-10-24 10:13:02,115 - botocore.endpoint - ERROR - Endpoint connection error: Could not resolve host: datasets.example.com

Solution

Apply the following changes to restore reliable DNS resolution for benchmark workloads.

1. Align pod DNSPolicy with the cluster

Update the benchmark pod (or the Helm chart/Job template) to use ClusterFirst or ClusterFirstWithHostNet when host networking is required.


# Before
dnsPolicy: Default

# After
dnsPolicy: ClusterFirst

2. Provide explicit search domains for short names

If short hostnames are used (e.g., pushgw), add a dnsConfig block.


# Before (no dnsConfig)
...

# After
dnsConfig:
  searches:
    - default.svc.cluster.local
    - svc.cluster.local
    - cluster.local

3. Fix node‑level /etc/resolv.conf to point at the cluster DNS

On each compute node, replace the upstream DNS entries with the cluster DNS IP (usually 10.96.0.10).


# Before (host DNS)
nameserver 192.168.1.1
search corp.example.com

# After (cluster DNS)
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Restart the kubelet service to pick up the new resolv.conf.

4. Ensure network policies allow UDP/TCP port 53 to the DNS service


# Example NetworkPolicy allowing DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: benchmark
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.96.0.0/12   # CoreDNS service CIDR
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

5. Add fallback IP addresses for critical services

For immediate mitigation, add static entries to /etc/hosts inside the container image (or via an initContainer).


# Before (no hosts entry)
...

# After (initContainer adds)
10.200.30.45   datasets.example.com
10.200.30.46   pushgw

6. Enable DNS caching (optional)

Deploy nscd or dnsmasq as a sidecar to cache successful lookups, reducing latency for repeated dataset accesses.

Verify

After applying the fixes, perform the following checks:

  1. Confirm /etc/resolv.conf inside a benchmark pod reflects the cluster DNS and search domains.
  2. Run a direct DNS query from the pod:

nslookup datasets.example.com


Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   datasets.example.com
Address: 34.215.12.78
  1. Execute a lightweight data‑load test:

python - <<'PY'
import torch
from torchvision import datasets, transforms
transform = transforms.Compose([transforms.ToTensor()])
ds = datasets.ImageFolder('s3://datasets.example.com/imagenet', transform=transform)
loader = torch.utils.data.DataLoader(ds, batch_size=32, num_workers=4)
for _ in loader:
    break
print("Data load succeeded")
PY


Data load succeeded
  1. Run the full benchmark and verify that no DNS‑related errors appear in the logs.
  2. Check Prometheus metrics endpoint:

curl -s http://pushgw:9091/metrics | grep benchmark_duration_seconds


benchmark_duration_seconds{job="pytorch-bench"} 12.34

Prevent

Implement the following guardrails to avoid recurrence:

  • Cluster‑wide DNS health checks: Deploy a DaemonSet that periodically runs dig against known external domains and reports to Prometheus.
  • Admission controller policy: Enforce that all benchmark workloads set dnsPolicy: ClusterFirst unless explicitly overridden.
  • NetworkPolicy templates: Include a reusable “allow‑dns” policy in every namespace that hosts compute workloads.
  • Configuration drift detection: Use kube-bench or OPA Gatekeeper to flag nodes whose /etc/resolv.conf deviates from the expected cluster DNS IP.
  • Retry logic in data loaders: Wrap dataset initialization in a retry loop with exponential back‑off to tolerate transient DNS hiccups.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does DNS resolution work on the login node but fail inside the benchmark pod?

    The login node uses the host’s /etc/resolv.conf, which points at a functional corporate DNS. Pods inherit the host’s resolver only when dnsPolicy: Default is set; otherwise they use the cluster DNS. The mis‑configured policy caused the pod to use the broken host resolver.

  2. Can I use a public DNS server (e.g., 8.8.8.8) inside the cluster?

    Yes, but it bypasses the internal CoreDNS service, breaking service‑discovery for Kubernetes‑internal names. The recommended approach is to keep nameserver 10.96.0.10 and configure forward plugins in CoreDNS to forward external queries to public resolvers.

  3. What timeout should I set for the object‑storage SDK to avoid premature failures?

    Increase the SDK’s connect_timeout and read_timeout to at least 15 seconds when DNS latency is suspected. For boto3 this can be done via botocore.config.Config(connect_timeout=15, read_timeout=30).

  4. How do I debug DNS issues when the cluster uses a custom DNS service (e.g., external CoreDNS deployment)?

    Use kubectl exec -it <pod> -- dig @dns‑svc-ip +trace <hostname> to see the full resolution chain. Compare the trace with a known‑good node to spot missing forwarders or ACL blocks.

  5. Is there a way to automatically reload /etc/resolv.conf after I change the node’s DNS configuration?

    Restarting the kubelet service (or the node) forces the kubelet to re‑read the host’s resolver file and propagate it to newly created pods. For a rolling update, drain the node, update /etc/resolv.conf, restart kubelet, and uncordon.