MLflow tracking server run registration fails due to DNS lookup error

Problem: MLflow Tracking Server Fails to Register Runs Due to DNS Lookup Errors

When a client attempts to start a new run, the tracking server raises an exception similar to:

socket.gaierror: [Errno -2] Name or service not known
mlflow.exceptions.MlflowException: Unable to register run
Traceback (most recent call last):
  ...
  urllib3.exceptions.NewConnectionError: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution.

The failure occurs before any artifact is written; the server cannot resolve the hostname of the configured artifact_uri (or, less commonly, the backend_store_uri). This symptom is reported across cloud managed services (AWS SageMaker, GCP AI Platform, Azure ML) and on-prem Kubernetes clusters.

Root Cause Analysis

MLflow creates a RunInfo record during registration and immediately validates that the artifact store endpoint is reachable. The validation logic performs a DNS lookup using the standard resolver of the host process. If the resolver cannot translate the hostname to an IP address, the registration aborts with the socket.gaierror shown above.

Key points from the official documentation:

• Both backend_store_uri and artifact_uri must be reachable from the tracking server process (MLflow Tracking Server docs).
• When a hostname is used, MLflow assumes standard DNS resolution; private DNS zones, VPC‑only resolvers, or load‑balancer hostnames require explicit network configuration (Remote Tracking Server guide).
• The troubleshooting page lists “Network errors and DNS failures” as a primary cause of run registration failures (MLflow troubleshooting).

In real incidents, the underlying cause was one of the following:

• CoreDNS pods evicted in an EKS cluster, leaving the node without a DNS resolver.
• Private DNS zone mis‑configuration for a VPC‑only S3 endpoint (GitHub issue #4187).
• Network Security Group rule that blocked UDP/TCP 53 traffic to the internal DNS server.
• Load balancer DNS name not propagated to the VPC’s resolver cache after a recent change.

Investigation and Debugging Steps

1. Verify the exact error and its source

journalctl -u mlflow.service -n 50 | grep -i "gaierror"

Typical log line:

2024-09-12 14:32:07,123 ERROR mlflow.tracking.fluent._run: Unable to register run: socket.gaierror: [Errno -2] Name or service not known (artifact_uri= s3://mlflow-artifacts.private.example.com/)

2. Test DNS resolution from the server host

# Resolve the artifact store hostname
nslookup mlflow-artifacts.private.example.com
# Or using dig for more detail
dig +short mlflow-artifacts.private.example.com
# Verify that the resolver is the expected VPC DNS
cat /etc/resolv.conf

Expected output (successful):

10.0.0.2

If the command returns ** server can't find mlflow-artifacts.private.example.com: NXDOMAIN or times out, DNS is the blocker.

3. Check network path to the DNS server

# Verify UDP/TCP 53 connectivity
nc -vz 10.0.0.2 53
# Or using ss
ss -u -a | grep ':53'

4. Confirm that the artifact store endpoint is reachable once the hostname resolves

# Using AWS CLI for an S3 private endpoint
aws s3 ls s3://mlflow-artifacts.private.example.com/ --endpoint-url https://mlflow-artifacts.private.example.com

5. Review Kubernetes DNS health (if applicable)

# List CoreDNS pods
kubectl -n kube-system get pods -l k8s-app=kube-dns
# Describe a pod for events
kubectl -n kube-system describe pod 

6. Cross‑reference community reports

• GitHub issue #3621 describes the same stack trace when the artifact store DNS is missing.
• Stack Overflow question 68214512 shows a “Name or service not known” error caused by a missing /etc/hosts entry for a private endpoint.

Solution: Restoring Reliable DNS Resolution for the Artifact Store

Option A – Add a static host entry (quick fix)

If the hostname is static and the IP is known, add it to /etc/hosts on every tracking server node.

# Before
cat /etc/hosts
# After (add line)
10.12.34.56 mlflow-artifacts.private.example.com

This bypasses the resolver but should be used only as a temporary measure.

Option B – Configure the correct DNS resolver (recommended)

Ensure the server’s /etc/resolv.conf points to a DNS server that can resolve the private zone.

# Before
cat /etc/resolv.conf
# Expected content (e.g., VPC DNS)
nameserver 169.254.169.253
search myvpc.internal

# After (if missing)
sudo bash -c 'cat > /etc/resolv.conf <

# Deployment snippet
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow-tracking
spec:
  template:
    spec:
      dnsPolicy: ClusterFirst
      dnsConfig:
        nameservers:
          - 10.0.0.2   # VPC DNS
        searches:
          - myvpc.internal

# Before (private DNS)
artifact_uri = "s3://mlflow-artifacts.private.example.com/"

# After (regional endpoint)
artifact_uri = "s3://mlflow-artifacts/"

# In mlflow server start command
mlflow server \
  --backend-store-uri postgresql://mlflow:pwd@db.internal:5432/mlflow \
  --default-artifact-root s3://mlflow-artifacts/ \
  --host 0.0.0.0 --port 5000

nslookup mlflow-artifacts.private.example.com
# Should return an IP address

python - <<'PY'
import mlflow
mlflow.set_tracking_uri("http://mlflow-tracking:5000")
with mlflow.start_run():
    mlflow.log_param("p", 1)
    mlflow.log_metric("m", 0.5)
print("Run registered:", mlflow.active_run().info.run_id)
PY

Run registered: 1234567890abcdef1234567890abcdef

journalctl -u mlflow.service -n 20 | grep -i "gaierror"