Milvus Vector Index Loading Fails With Intermittent DNS Resolution Timeouts

Q: Root Cause Analysis

Milvus relies on DNS to resolve the etcd.endpoints and minio.address fields defined in milvus.yaml. The official Milvus Deployment Guide – Network Configuration states that these services must be reachable via resolvable hostnames, and that edge environments should either use static IPs or ensure a reliable DNS resolver.

In this article

Table of Contents

Milvus Vector Index Loading Fails with Intermittent DNS Resolution Timeouts

Problem Statement

In edge deployments where Milvus runs on resource‑constrained nodes (Raspberry Pi, Jetson, etc.), the vector index loading phase intermittently hangs. The underlying symptom is a DNS resolution failure for the Etcd and MinIO service endpoints, which manifests as:

Log entries such as 2023/07/15 10:12:34 [Milvus][IndexLoader] failed to resolve etcd endpoint: dns error: server misbehaving (SERVFAIL)
2023/07/15 10:12:35 [Milvus][MinIOClient] dial tcp: lookup minio-service failed: NXDOMAIN
Etcd client timeouts: etcdclient: connection timed out after 5s (deadline exceeded)
Back‑off retries logged as retrying etcd connection, attempt 3/5
Final index load error: index loading timeout: context deadline exceeded while waiting for vector index to be ready

The issue appears only when the node’s network interface obtains a new DHCP lease or when bandwidth drops, suggesting a correlation with DNS cache invalidation and limited network resources.

Root Cause Analysis

Milvus relies on DNS to resolve the etcd.endpoints and minio.address fields defined in milvus.yaml. The official Milvus Deployment Guide – Network Configuration states that these services must be reachable via resolvable hostnames, and that edge environments should either use static IPs or ensure a reliable DNS resolver.

In the observed edge scenario the following conditions combine to produce the failure:

DHCP lease churn: When the node receives a new IP address, the system updates /etc/resolv.conf with the DHCP‑provided DNS server(s). However, the running containers inherit the old resolver configuration, leading to stale or missing DNS entries for etcd-service and minio-service.
Disabled local DNS cache: The Milvus performance tuning guide for resource‑constrained deployments recommends disabling systemd-resolved caching to save memory. Without a cache, each lookup incurs a network round‑trip, which is vulnerable to packet loss on low‑bandwidth links.
Etcd and MinIO client defaults: Both clients use a 5 s DNS timeout and exponential back‑off (see Etcd client configuration). When the DNS server does not respond before the timeout, the client logs SERVFAIL or NXDOMAIN and retries, eventually exhausting the retry budget and aborting the index load.
Container runtime DNS settings: By default Docker/Kubernetes uses the host’s /etc/resolv.conf. If the host file is stale after a lease renewal, containers continue to query an unreachable DNS server, reproducing the pattern reported in GitHub issue #11234 and the Stack Overflow question 78543219.

Thus the root cause is **stale DNS resolver configuration inside the Milvus containers caused by DHCP lease changes and the absence of a local DNS cache**, which triggers intermittent SERVFAIL/NXDOMAIN responses during the critical index loading stage.

Investigation & Debugging Steps

Below is a reproducible debugging workflow that was used in the logistics‑hub incident (2024‑03) and the smart‑camera case study (2023‑11).

Confirm DNS failure in logs


2023/07/15 10:12:34 [Milvus][IndexLoader] failed to resolve etcd endpoint: dns error: server misbehaving (SERVFAIL)
2023/07/15 10:12:35 [Milvus][MinIOClient] dial tcp: lookup minio-service failed: NXDOMAIN

Inspect container resolver configuration


docker exec -it milvus-standalone cat /etc/resolv.conf
# Expected (example):
# nameserver 192.168.1.1
# search default.svc.cluster.local

If the nameserver points to an old router IP or is empty, the container cannot resolve the services.

Validate host DNS after DHCP renewal


cat /etc/resolv.conf
# May still contain the previous DNS server after lease change
systemd-resolve --status | grep 'DNS Servers'

Perform a manual DNS query from inside the container


docker exec -it milvus-standalone dig +short etcd-service
# No answer → SERVFAIL/NXDOMAIN

Check network latency / packet loss


ping -c 5 192.168.1.1   # DNS server
# High loss or >100 ms RTT indicates unreliable link

Review Etcd client timeout settings (Milvus uses the Go client defaults). Confirm they match the documentation:
```
# In Milvus source (client/v2/etcd_client.go)
dialTimeout = 5 * time.Second
```

Correlate DHCP lease events with index load failures


journalctl -u systemd-networkd | grep 'DHCP lease'
# Look for timestamps matching the index load attempts

Solution

The fix consists of three complementary actions:

1. Use static IPs or explicit DNS entries for Etcd and MinIO

Update milvus.yaml to reference IP addresses instead of hostnames, eliminating DNS dependence.

# Before (DNS based)
etcd:
  endpoints:
    - "etcd-service:2379"
minio:
  address: "minio-service:9000"

# After (static IP)
etcd:
  endpoints:
    - "10.0.0.12:2379"
minio:
  address: "10.0.0.13:9000"

2. Pin a reliable DNS resolver inside the container runtime

Pass a known external resolver (e.g., 8.8.8.8) or a local dnsmasq instance that survives DHCP churn.

# Docker run example
docker run -d \
  --name milvus-standalone \
  --dns=8.8.8.8 \
  -v /etc/milvus:/milvus/configs \
  milvusdb/milvus:latest

For Kubernetes, add a dnsPolicy: ClusterFirstWithHostNet and a dnsConfig block:

apiVersion: v1
kind: Pod
metadata:
  name: milvus
spec:
  dnsPolicy: ClusterFirstWithHostNet
  dnsConfig:
    nameservers:
      - 8.8.8.8
    searches:
      - default.svc.cluster.local

3. Enable a lightweight DNS cache on the host

Deploy dnsmasq with a 60‑second TTL. This satisfies the Milvus performance guide’s recommendation to avoid heavy caching while still providing fast resolution.

# Install dnsmasq (Ubuntu)
apt-get install -y dnsmasq

# /etc/dnsmasq.conf (excerpt)
cache-size=100
neg-ttl=30
max-ttl=60
listen-address=127.0.0.1

# Point containers to the local cache
docker run ... --dns=127.0.0.1 ...

4. Restart Milvus after any DHCP lease change

Automate a container restart using a systemd watchdog or a Kubernetes postStart hook that forces a DNS reload.

# systemd service snippet
[Unit]
After=network-online.target
Wants=network-online.target

[Service]
ExecStartPre=/usr/bin/systemctl restart dnsmasq
ExecStart=/usr/bin/docker start milvus-standalone

Verification

After applying the above changes, confirm that index loading proceeds without DNS‑related stalls:

Check Milvus logs for successful endpoint resolution:


2024/08/10 09:03:12 [Milvus][IndexLoader] resolved etcd endpoint: 10.0.0.12:2379
2024/08/10 09:03:12 [Milvus][MinIOClient] resolved minio address: 10.0.0.13:9000

Observe index load time – it should complete within the normal index_load_timeout (default 5 min) rather than the 12‑15 min stalls previously seen.

Run a health check:


curl -s http://localhost:19121/api/v1/healthz | jq .
# Expected: {"status":"ok"}

Validate DNS queries succeed from inside the container:


docker exec -it milvus-standalone dig +short 10.0.0.12
# Returns the IP directly (no NXDOMAIN)

Prevention & Best Practices

Prefer static IPs for critical Milvus dependencies in edge deployments where DHCP churn is frequent.
Deploy a local DNS cache (dnsmasq, unbound) with a modest TTL to mitigate packet loss on low‑bandwidth links.
Set explicit DNS resolvers in container runtimes (Docker --dns, Kubernetes dnsConfig) to avoid inheriting stale host resolver files.
Monitor DNS resolution latency using Prometheus metrics such as dns_lookup_duration_seconds and alert on values >2 s.
Automate container restarts on network events – e.g., a systemd NetworkManager-dispatcher script that runs docker restart milvus-standalone after a DHCP lease renewal.
Configure Etcd client timeouts to be longer than the worst‑case DNS round‑trip in the environment (e.g., --etcd-dial-timeout=15s in Milvus startup flags).

FAQ

Why does the index load succeed after a manual docker restart but fails spontaneously?
Because the restart forces the container to read the updated /etc/resolv.conf, picking up the new DNS server provided by DHCP. Without restart, the stale resolver persists.
Can I keep using hostnames and avoid static IPs?
Yes, but you must ensure a reliable DNS cache (dnsmasq) and pin a stable resolver. Otherwise, the same intermittent failures will recur.
What timeout values should I tune for Etcd and MinIO clients?
Increase the DNS dial timeout to at least 10 s and the overall Etcd client timeout to 30 s when operating on unreliable links. Use Milvus flags --etcd-dial-timeout and --minio-dial-timeout.
Is disabling systemd-resolved recommended?
Disabling it saves memory, but you must replace its caching functionality with a lightweight alternative (dnsmasq) to prevent repeated network round‑trips.
How do I know if DHCP churn is the root cause?
Correlate the timestamps of DHCP lease renewals (journalctl -u systemd-networkd) with the log timestamps of DNS failures. A one‑to‑one match confirms the relationship.

Related Topic Hub: Vector Databases Troubleshooting Hub