LangChain DNS Resolution Failure During Batch API Calls

Q: Root Cause – Why DNS resolution breaks under load

LangChain relies on the underlying HTTP client (requests for sync calls, aiohttp for async calls) which, by default, uses the operating system resolver. In a high‑concurrency batch:

Q: Investigation – Debugging steps 1. Capture failing logs

2024-06-15 12:03:41,112 ERROR langchain.utils.network - DNS resolution failed for endpoint https://api.openai.com/v1/chat/completions Traceback (most recent call last): File ".../langchain/utils/network.py", line 78, in _resolve result = socket.getaddrinfo(host, port, family=socket.AF_UNSPEC, type=socket.SOCK_STREAM) socket.gaierror: [Errno -3] Temporary failure in name resolution 2. Verify resolver limits on a worker node # Check systemd-resolved query rate limit (if applicable) $ sudo resolvectl statistics Current DNS Server: 1.1.1.1 Queries sent: 124578 Queries dropped (rate limit): 342 3. Inspect /etc/resolv.conf and /etc/nsswitch.conf # cat /etc/resolv.conf nameserver 10.0.0.2 options attempts:3 rotate timeout:2 # cat /etc/nsswitch.conf hosts: files dns 4. Reproduce the failure locally with a high concurrency test import asyncio, aiohttp, socket async def fetch(session, url): async with session.get(url) as resp: return await resp.text() async def main(): connector = aiohttp.TCPConnector(limit=0) # no limit async with aiohttp.ClientSession(connector=connector) as session: tasks = [fetch(session, "https://api.example.com") for _ in range(500)] await asyncio.gather(*tasks, return_exceptions=True) asyncio.run(main()) Running the script on a node with default resolver often reproduces ClientConnectorError after ~200 concurrent requests, matching the Ray cluster incident.

Q: 5. Check for proxy‑intercepted DNS

# tcpdump -i eth0 -n udp port 53 12:34:56.789012 IP 10.1.2.3.53000 > 10.0.0.2.53: 12345+ A? api.example.com. (28) 12:34:56.789045 IP 10.0.0.2.53 > 10.1.2.3.53000: 12345 NXDomain 0/0/0 (108) The “NXDomain” response from a corporate DNS interceptor explains the “SERVFAIL” observed in the corporate proxy scenario.

Q: Solution – Making LangChain DNS‑robust for batch jobs 1. Use a custom aiohttp resolver with a shared DNS cache

LangChain allows injecting a custom aiohttp.TCPConnector via the http_client argument of agents/tools. The following patch creates a thread‑safe aiodns resolver that respects a higher concurrency limit.

In this article

Table of Contents

Problem – DNS resolution failures during LangChain batch API calls

When running a LangChain batch pipeline on a distributed compute cluster (Kubernetes, Ray, Azure Batch, or on‑premise HPC), the jobs intermittently abort with DNS‑related exceptions. Typical log excerpts look like:

socket.gaierror: [Errno -3] Temporary failure in name resolution
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.example.com:443 ssl:default [Name or service not known]
requests.exceptions.ConnectionError: DNS lookup failed for https://api.example.com
ERROR langchain.utils.network - DNS resolution failed for endpoint https://api.example.com

These errors surface during the batch execution phase when LangChain orchestrates dozens to hundreds of concurrent HTTP calls to external services (LLM providers, vector stores, third‑party APIs). The failure is not isolated to a single worker; it propagates across the whole job, causing the batch to time out or produce incomplete results.

Root Cause – Why DNS resolution breaks under load

LangChain relies on the underlying HTTP client (requests for sync calls, aiohttp for async calls) which, by default, uses the operating system resolver. In a high‑concurrency batch:

Resolver limits: The OS resolver (glibc’s nss or systemd-resolved) caps the number of simultaneous DNS queries. Scaling a Ray cluster beyond ~200 workers exhausts the DNS cache entries, as documented in the LangChain “Network Configuration & HTTP Clients” section.
Throttling by DNS infrastructure: Cloud providers (Azure DNS, corporate DNS forwarders) enforce query‑per‑second limits. Auto‑scaled VMs can collectively exceed these limits, leading to Temporary failure in name resolution (see the Azure Batch incident).
Mis‑ordered /etc/nsswitch.conf: On some HPC clusters the IPv6 lookup precedes IPv4, and external APIs lack AAAA records. The resolver falls back to IPv6, fails, and does not retry IPv4, producing “Name or service not known”.
Shared resolver cache contention: When many processes share the same resolver cache (e.g., Docker containers on the same node), cache eviction can cause intermittent misses.

LangChain’s own retry and timeout settings (LANGCHAIN_HTTP_TIMEOUT, LANGCHAIN_RETRY_STRATEGY) do not address DNS lookup failures because the error occurs before a TCP connection is attempted.

Investigation – Debugging steps

1. Capture failing logs

2024-06-15 12:03:41,112 ERROR langchain.utils.network - DNS resolution failed for endpoint https://api.openai.com/v1/chat/completions
Traceback (most recent call last):
  File ".../langchain/utils/network.py", line 78, in _resolve
    result = socket.getaddrinfo(host, port, family=socket.AF_UNSPEC, type=socket.SOCK_STREAM)
socket.gaierror: [Errno -3] Temporary failure in name resolution

2. Verify resolver limits on a worker node

# Check systemd-resolved query rate limit (if applicable)
$ sudo resolvectl statistics
Current DNS Server: 1.1.1.1
Queries sent: 124578
Queries dropped (rate limit): 342

3. Inspect `/etc/resolv.conf` and `/etc/nsswitch.conf`

# cat /etc/resolv.conf
nameserver 10.0.0.2
options attempts:3 rotate timeout:2

# cat /etc/nsswitch.conf
hosts:          files dns

4. Reproduce the failure locally with a high concurrency test

import asyncio, aiohttp, socket

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main():
    connector = aiohttp.TCPConnector(limit=0)  # no limit
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch(session, "https://api.example.com") for _ in range(500)]
        await asyncio.gather(*tasks, return_exceptions=True)

asyncio.run(main())

Running the script on a node with default resolver often reproduces ClientConnectorError after ~200 concurrent requests, matching the Ray cluster incident.

5. Check for proxy‑intercepted DNS

# tcpdump -i eth0 -n udp port 53
12:34:56.789012 IP 10.1.2.3.53000 > 10.0.0.2.53: 12345+ A? api.example.com. (28)
12:34:56.789045 IP 10.0.0.2.53 > 10.1.2.3.53000: 12345 NXDomain 0/0/0 (108)

The “NXDomain” response from a corporate DNS interceptor explains the “SERVFAIL” observed in the corporate proxy scenario.

Solution – Making LangChain DNS‑robust for batch jobs

1. Use a custom aiohttp resolver with a shared DNS cache

LangChain allows injecting a custom aiohttp.TCPConnector via the http_client argument of agents/tools. The following patch creates a thread‑safe aiodns resolver that respects a higher concurrency limit.

Before (default behavior):

from langchain.tools import RequestsGetTool

tool = RequestsGetTool()
# Internally uses requests.get() which relies on system resolver

After (custom resolver):

import aiohttp
import aiodns
from langchain.utilities import RequestsWrapper

class AiohttpResolver(aiohttp.abc.AbstractResolver):
    def __init__(self, loop):
        self._resolver = aiodns.DNSResolver(loop=loop, timeout=5)

    async def resolve(self, host, port=0, family=socket.AF_INET):
        result = await self._resolver.gethostbyname(host, socket.AF_UNSPEC)
        return [{
            'hostname': host,
            'host': ip,
            'port': port,
            'family': family,
            'proto': 0,
            'flags': socket.AI_NUMERICHOST,
        } for ip in result.addresses]

    async def close(self):
        pass

# Create a shared connector for the whole batch
connector = aiohttp.TCPConnector(
    limit=0,               # unlimited concurrent connections
    resolver=AiohttpResolver(asyncio.get_event_loop())
)

# Wrap LangChain HTTP calls to use the custom connector
http_client = RequestsWrapper(
    client=aiohttp.ClientSession(connector=connector)
)

# Example tool that uses the custom client
from langchain.tools import BaseTool

class GetJsonTool(BaseTool):
    name = "get_json"
    description = "Fetch JSON from a URL"

    def __init__(self, client):
        self.client = client

    async def _run(self, url: str):
        async with self.client.get(url) as resp:
            return await resp.json()

# Instantiate with shared client
tool = GetJsonTool(client=http_client)

2. Tune environment variables for timeout and retries

Set LangChain‑specific variables to give the resolver enough time and to retry transient failures.

# .env or cluster startup script
export LANGCHAIN_HTTP_TIMEOUT=30          # seconds, longer than default 10
export LANGCHAIN_RETRY_STRATEGY=exponential_backoff
export LANGCHAIN_MAX_RETRIES=5
export LANGCHAIN_DNS_RESOLVER=aiodns      # documented in “Network Configuration & HTTP Clients”

3. Adjust OS resolver configuration

Increase options attempts and timeout in /etc/resolv.conf (e.g., options attempts:5 timeout:5 rotate).
On systems using systemd-resolved, raise DNSStubListener query rate limits via /etc/systemd/resolved.conf (Cache=yes, CacheSize=0 for unlimited).
Ensure hosts: files dns order in /etc/nsswitch.conf to prefer /etc/hosts entries before DNS.

4. Deploy a local DNS forwarder or cache (e.g., `unbound`)

Running a lightweight DNS cache on each node reduces external query volume and mitigates provider throttling. Example unbound.conf snippet:

server:
    interface: 0.0.0.0
    access-control: 0.0.0.0/0 allow
    cache-max-ttl: 86400
    cache-min-ttl: 300
    num-threads: 4
    outgoing-num-tcp: 100
    outgoing-num-udp: 200

Configure the node’s /etc/resolv.conf to point to 127.0.0.1 as the primary nameserver.

Verification – Confirming the fix

Run a high‑concurrency sanity test after deploying the custom resolver or DNS cache:

asyncio.run(main())  # the script from the investigation step
# Expect zero exceptions, all 500 requests succeed.

Check LangChain logs for absence of DNS errors:

2024-06-15 12:05:02,331 INFO langchain.utils.network - DNS resolution succeeded for https://api.example.com

Monitor DNS query rate on a node:

# sudo resolvectl statistics
Queries sent: 124578
Queries dropped (rate limit): 0

Validate end‑to‑end batch output against a known checksum or result set to ensure functional correctness.

Prevention – Operational guardrails

Capacity planning: Align the maximum parallel requests (max_parallel_requests in LangChain’s async batch guide) with the DNS infrastructure’s query‑per‑second limits.
Health checks: Add a lightweight DNS probe (e.g., dig +short api.example.com) to the cluster’s readiness/liveness checks.
Alerting: Create alerts on socket.gaierror or ClientConnectorError rate spikes using Prometheus metrics from aiohttp or LangChain’s internal logger.
Configuration as code: Store resolver settings, environment variables, and unbound config in version‑controlled deployment manifests (Helm charts, Terraform).
Periodic DNS cache warm‑up: Run a short “warm‑up” job that resolves all external endpoints before the main batch starts, reducing burst queries.

FAQ – Common follow‑up questions

Q1: Why does DNS resolution fail only when the batch scales beyond a certain number of workers?

A: The OS resolver has a hard limit on concurrent DNS queries (often ~200). When the batch spawns more workers, the limit is exceeded, causing socket.gaierror. Using a custom aiohttp resolver or a local DNS cache removes this bottleneck.

Q2: Can I rely on LangChain’s built‑in retry strategy to fix DNS errors?

A: No. Retries are applied after a successful DNS lookup. DNS failures happen before the HTTP request is created, so you must address the resolver itself (custom resolver, increased timeout, or DNS cache).

Q3: How do I configure LangChain to use a specific DNS server?

A: Set the LANGCHAIN_DNS_RESOLVER env var to aiodns and ensure the underlying system /etc/resolv.conf lists the desired nameserver. Alternatively, instantiate an aiohttp.TCPConnector with a custom resolver that points to the target DNS IP.

Q4: My cluster uses a corporate DNS proxy that returns SERVFAIL for some domains. Do I need to bypass the proxy?

A: Yes. Either add the external domains to the proxy’s allow‑list or configure a separate forwarder (e.g., unbound) that queries public DNS directly. Updating the proxy’s DNS rules resolved the issue in the corporate network case.

Q5: Is there a performance impact when using a shared aiodns resolver?

A: The impact is minimal. aiodns runs DNS queries asynchronously in a thread pool, allowing thousands of concurrent lookups without blocking the event loop. In practice, latency improves because cached entries are reused across workers.

Related Topic Hub: RAG Systems Troubleshooting Hub