TGI Service Unreachable During Benchmark Due To DNS Resolution Failure

Q: Debug – Step‑by‑Step Investigation 1. Verify the symptom from inside the benchmark pod

$ kubectl exec -it benchmark-pod -- nslookup tgi-model-prod.example.com Server: 10.96.0.10 Address: 10.96.0.10#53 ** server can't find tgi-model-prod.example.com: SERVFAIL 2. Check CoreDNS logs for rate‑limiting messages $ kubectl logs -n kube-system -l k8s-app=kube-dns -c coredns ... time="2026-10-18T14:31:58Z" level=error msg="too many concurrent queries, dropping request" 3. Inspect CoreDNS ConfigMap apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf { max_concurrent 100 } cache 30 loop reload loadbalance } 4. Examine pod /etc/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5 timeout:2 attempts:3 The default ndots is 5, which forces the resolver to query the external DNS for short names, adding latency.

Q: 5. Reproduce the failure with a load generator

$ kubectl run dns-load --image=busybox --restart=Never --rm -i -- /bin/sh / # while true; do nslookup tgi-model-prod.example.com > /dev/null || echo "fail"; done fail fail ... Solution – Scale DNS Capacity and Adjust Resolver Settings 1. Increase CoreDNS max_concurrent to handle benchmark load # Before (ConfigMap snippet) forward . /etc/resolv.conf { max_concurrent 100 } # After forward . /etc/resolv.conf { max_concurrent 500 } Apply the updated ConfigMap and roll the CoreDNS deployment:

Q: 2. Enable DNS caching with a larger TTL to reduce upstream queries

# Before cache 30 # After cache 300 3. Add ndots:1 to the pod’s DNS options to avoid unnecessary external lookups Patch the benchmark Deployment’s pod template:

Q: 4. (Optional) Deploy a dedicated NodeLocal DNSCache for high‑throughput workloads

kubectl apply -f https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml This moves DNS caching to each node, eliminating the CoreDNS bottleneck for massive parallel lookups.

Q: Verify – Confirm the Fix Works 1. Re‑run the benchmark script

2026-10-18 15:02:13,021 INFO benchmark.py: Resolved host "tgi-model-prod.example.com" to 172.20.4.12 2026-10-18 15:02:13,045 INFO benchmark.py: Starting load test... ... Benchmark completed: 95k req/s, 99th‑pct latency 12ms 2. Check CoreDNS metrics for concurrency $ kubectl -n kube-system exec -it $(kubectl get pod -l k8s-app=kube-dns -n kube-system -o jsonpath="{.items[0].metadata.name}") -- curl -s http://localhost:9153/metrics | grep coredns_forward_requests_concurrent coredns_forward_requests_concurrent 0 3. Verify DNS resolution from the benchmark pod under load $ kubectl exec -it benchmark-pod -- sh -c "for i in \$(seq 1 200); do nslookup tgi-model-prod.example.com > /dev/null && echo ok || echo fail; done" ok ok ... Prevent – Operational Guardrails Monitor CoreDNS concurrency – set an alert on coredns_forward_requests_concurrent exceeding 80% of max_concurrent. Capacity planning – benchmark DNS load before scaling a new service; allocate max_concurrent based on projected parallel lookups. Enforce pod DNS options via a PodSecurityPolicy or OPA Gatekeeper rule that requires ndots ≤ 2 for production workloads. Deploy NodeLocal DNSCache in clusters that run high‑frequency micro‑benchmarks or service mesh sidecars. Cache TTL tuning – set cache TTL to at least 5 minutes for stable service endpoints to reduce upstream traffic. Related Topic Hub: Model Serving Troubleshooting Hub

In this article

Table of Contents

Problem – TGI Service Unreachable During Benchmark

During automated performance benchmarking of a Text Generation Inference (TGI) deployment, the benchmark script repeatedly fails with a DNS resolution error:


2026-10-18 14:32:07,123 ERROR benchmark.py: Failed to resolve host "tgi-model-prod.example.com": Name or service not known
Traceback (most recent call last):
  File "/opt/benchmark/benchmark.py", line 87, in run
    ip = socket.gethostbyname(endpoint)
  File "/usr/lib/python3.11/socket.py", line 955, in gethostbyname
    return getaddrinfo(host, None, 0, SOCK_STREAM)[0][4][0]
socket.gaierror: [Errno -2] Name or service not known

The failure occurs only under the high‑load benchmark run; manual curl calls from the same node succeed when the load is low.

Root Cause – Intermittent DNS Resolution Failure Under Load

The underlying issue is a combination of:

DNS cache exhaustion in the kube-dns CoreDNS deployment caused by a surge of parallel lookups from the benchmark client.
Insufficient max_concurrent setting in the CoreDNS forward plugin, which defaults to 100 concurrent upstream queries.
Missing ndots configuration in the pod’s /etc/resolv.conf, causing the resolver to fall back to external DNS after a timeout, which is throttled by the upstream provider.

When the benchmark spawns hundreds of threads, each thread performs a DNS lookup for the model endpoint. CoreDNS reaches its concurrency limit, drops new queries, and returns SERVFAIL. The client interprets this as “Name or service not known”.

Debug – Step‑by‑Step Investigation

1. Verify the symptom from inside the benchmark pod


$ kubectl exec -it benchmark-pod -- nslookup tgi-model-prod.example.com
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find tgi-model-prod.example.com: SERVFAIL

2. Check CoreDNS logs for rate‑limiting messages


$ kubectl logs -n kube-system -l k8s-app=kube-dns -c coredns
... 
time="2026-10-18T14:31:58Z" level=error msg="too many concurrent queries, dropping request"

3. Inspect CoreDNS ConfigMap


apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf {
            max_concurrent 100
        }
        cache 30
        loop
        reload
        loadbalance
    }

4. Examine pod `/etc/resolv.conf`


search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5 timeout:2 attempts:3

The default ndots is 5, which forces the resolver to query the external DNS for short names, adding latency.

5. Reproduce the failure with a load generator


$ kubectl run dns-load --image=busybox --restart=Never --rm -i -- /bin/sh
/ # while true; do nslookup tgi-model-prod.example.com > /dev/null || echo "fail"; done
fail
fail
...

Solution – Scale DNS Capacity and Adjust Resolver Settings

1. Increase CoreDNS `max_concurrent` to handle benchmark load


# Before (ConfigMap snippet)
forward . /etc/resolv.conf {
    max_concurrent 100
}

# After
forward . /etc/resolv.conf {
    max_concurrent 500
}

Apply the updated ConfigMap and roll the CoreDNS deployment:


kubectl apply -f coredns-configmap.yaml
kubectl rollout restart deployment coredns -n kube-system

2. Enable DNS caching with a larger TTL to reduce upstream queries


# Before
cache 30

# After
cache 300

3. Add `ndots:1` to the pod’s DNS options to avoid unnecessary external lookups

Patch the benchmark Deployment’s pod template:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: benchmark
spec:
  template:
    spec:
      dnsPolicy: ClusterFirst
      dnsConfig:
        options:
        - name: ndots
          value: "1"

4. (Optional) Deploy a dedicated `NodeLocal DNSCache` for high‑throughput workloads


kubectl apply -f https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

This moves DNS caching to each node, eliminating the CoreDNS bottleneck for massive parallel lookups.

Verify – Confirm the Fix Works

1. Re‑run the benchmark script


2026-10-18 15:02:13,021 INFO benchmark.py: Resolved host "tgi-model-prod.example.com" to 172.20.4.12
2026-10-18 15:02:13,045 INFO benchmark.py: Starting load test...
...
Benchmark completed: 95k req/s, 99th‑pct latency 12ms

2. Check CoreDNS metrics for concurrency


$ kubectl -n kube-system exec -it $(kubectl get pod -l k8s-app=kube-dns -n kube-system -o jsonpath="{.items[0].metadata.name}") -- curl -s http://localhost:9153/metrics | grep coredns_forward_requests_concurrent
coredns_forward_requests_concurrent 0

3. Verify DNS resolution from the benchmark pod under load


$ kubectl exec -it benchmark-pod -- sh -c "for i in \$(seq 1 200); do nslookup tgi-model-prod.example.com > /dev/null && echo ok || echo fail; done"
ok
ok
...

Prevent – Operational Guardrails

Monitor CoreDNS concurrency – set an alert on coredns_forward_requests_concurrent exceeding 80% of max_concurrent.
Capacity planning – benchmark DNS load before scaling a new service; allocate max_concurrent based on projected parallel lookups.
Enforce pod DNS options via a PodSecurityPolicy or OPA Gatekeeper rule that requires ndots ≤ 2 for production workloads.
Deploy NodeLocal DNSCache in clusters that run high‑frequency micro‑benchmarks or service mesh sidecars.
Cache TTL tuning – set cache TTL to at least 5 minutes for stable service endpoints to reduce upstream traffic.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does DNS work when I run a single curl but fail under benchmark?
The benchmark spawns hundreds of concurrent lookups, exhausting CoreDNS’s default max_concurrent. A single request stays well below the limit.
Can I increase the DNS timeout instead of max_concurrent?
Increasing the timeout only delays failures; the underlying concurrency bottleneck remains. Scaling max_concurrent or adding a cache is the correct approach.
Is NodeLocal DNSCache required for all clusters?
It is recommended for clusters that host workloads with high DNS query rates (e.g., large‑scale benchmarks, service mesh sidecars, or API gateways). For low‑traffic clusters, tuning CoreDNS may suffice.
How do I know what ndots value to use?
Set ndots to 1 or 2 for fully qualified domain names (FQDNs) that already contain the cluster domain. This prevents the resolver from unnecessarily contacting external DNS servers.
After the fix, I still see occasional SERVFAIL entries in logs. What should I check?
Verify that upstream DNS servers (e.g., the corporate resolver) are not rate‑limiting queries. Also ensure that the CoreDNS cache TTL is long enough to keep frequently queried records in memory.