MLflow tracking API unreachable during batch job execution

Problem – MLflow Tracking API Unreachable During Batch Job Execution

Batch jobs that run model training or data preprocessing attempt to log parameters, metrics, and artifacts to an MLflow tracking server. In production the jobs fail with errors such as:

2026-10-30 14:12:03,721 ERROR mlflow.tracking.client: Failed to log metric: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/mlflow/tracking/client.py", line 1123, in log_metric
    resp = self._tracking_client.log_metric(run_id, key, value, timestamp, step)
  File "/usr/local/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 215, in log_metric
    raise MlflowException("Failed to log metric: %s" % e)
mlflow.exceptions.MlflowException: Failed to log metric: 
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mlflow-tracking.mycorp.com', port=443): Max retries exceeded with url: /api/2.0/mlflow/runs/log-metric (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))

The failure halts the batch pipeline, causing downstream steps to be skipped and SLA violations.

Root Cause – Network Path Disruption and Inconsistent TLS Configuration

The underlying issue is a combination of:

• Network segmentation: The batch job runs in a Kubernetes namespace that is isolated by NetworkPolicy. Outbound traffic to the MLflow tracking server’s IP (10.2.45.12) on port 443 is blocked for pods that do not have the mlflow-client label.
• TLS certificate mismatch: The tracking server presents a certificate signed by an internal CA that is not present in the ca-bundle.crt of the batch job container image. The OpenSSL handshake succeeds only when curl is invoked with --insecure, but the MLflow Python client aborts on verification failure.
• Service discovery drift: The DNS entry mlflow-tracking.mycorp.com was recently updated to point to a new load balancer IP. The batch job’s DNS cache (glibc) retained the stale IP for 5 minutes, causing initial connection attempts to hit a non‑listening endpoint.

Debug – Systematic Investigation Steps

1. Verify DNS Resolution

$ nslookup mlflow-tracking.mycorp.com
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
Name:   mlflow-tracking.mycorp.com
Address: 10.2.45.12

Compare with the expected load balancer IP from the service manifest:

apiVersion: v1
kind: Service
metadata:
  name: mlflow-tracking
spec:
  type: LoadBalancer
  loadBalancerIP: 10.2.45.12

2. Test TCP Connectivity from a Job Pod

$ kubectl exec -it batch-job-7f9c9c9c9c-abcde -- /bin/sh
/ # ss -tn src :0 dst 10.2.45.12:443
State      Recv-Q Send-Q   Local Address:Port               Peer Address:Port
ESTAB      0      0        10.244.2.15:54231                10.2.45.12:443

If the state is SYN‑SENT or no entry appears, the network path is blocked.

3. Capture TLS Handshake with OpenSSL

$ openssl s_client -connect mlflow-tracking.mycorp.com:443 -servername mlflow-tracking.mycorp.com 

CONNECTED(00000003)
depth=0 OU = MyCorp, CN = mlflow-tracking.mycorp.com
verify error:num=20:unable to get local issuer certificate
verify return:1
...

The error confirms the client cannot verify the server’s certificate chain.

4. Inspect NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-mlflow
spec:
  podSelector:
    matchLabels:
      app: batch-job
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.2.45.12/32
    ports:
    - protocol: TCP
      port: 443

The policy is missing the required mlflow-client label, causing egress denial.

5. Review MLflow Client Configuration

import mlflow
print(mlflow.get_tracking_uri())
# Output: https://mlflow-tracking.mycorp.com

The URI is correct, but the client environment variable MLFLOW_TRACKING_INSECURE_TLS is not set, so verification is enforced.

Solution – Restoring Reliable Connectivity

1. Update NetworkPolicy to Allow Egress

Add the mlflow-client label to batch job pods and adjust the policy to match that label.

# Before
spec:
  podSelector:
    matchLabels:
      app: batch-job
# After
spec:
  podSelector:
    matchLabels:
      app: batch-job
      role: mlflow-client

Apply the revised policy:

kubectl apply -f networkpolicy-mlflow.yaml

2. Propagate Internal CA Certificate to Job Image

Copy the corporate CA bundle into the container and update the trust store.

# Dockerfile snippet (before)
FROM python:3.11-slim
# ... other layers ...

# Dockerfile snippet (after)
FROM python:3.11-slim
COPY ca-bundle.crt /usr/local/share/ca-certificates/internal-ca.crt
RUN update-ca-certificates
# ... other layers ...

3. Enable TLS Verification Bypass as a Temporary Measure (not recommended for long term)

export MLFLOW_TRACKING_INSECURE_TLS=true

Only use this while rolling out the CA update to avoid downtime.

4. Flush DNS Cache in Long‑Running Pods

Set ndots and options timeout:1 attempts:3 in the kube-dns ConfigMap, then restart pods, or use kubectl exec to clear the glibc cache:

# Inside pod
$ echo 2 > /proc/sys/vm/drop_caches

5. Redeploy Batch Jobs with Updated Image

kubectl set image deployment/batch-job batch-job=myrepo/batch-job:2.1.0

Verify – Confirming the Fix

Successful Log Write

import mlflow
mlflow.set_tracking_uri("https://mlflow-tracking.mycorp.com")
with mlflow.start_run() as run:
    mlflow.log_param("alpha", 0.01)
    mlflow.log_metric("accuracy", 0.93)
print("Run ID:", run.info.run_id)

Expected console output:

Run ID: 3f2c9d8e5a1e4b7b9c6d2a1e7f4b8c9d

Check the tracking server UI for the new run.

NetworkPolicy Effectiveness

$ kubectl exec -it batch-job-7f9c9c9c9c-abcde -- ss -tn dst :443
State      Recv-Q Send-Q   Local Address:Port               Peer Address:Port
ESTAB      0      0        10.244.2.15:54231                10.2.45.12:443

TLS Verification Pass

$ openssl s_client -connect mlflow-tracking.mycorp.com:443 -servername mlflow-tracking.mycorp.com 

CONNECTED(00000003)
depth=0 OU = MyCorp, CN = mlflow-tracking.mycorp.com
Verify return code: 0 (ok)

Prevent – Operational Guardrails

• Automated Certificate Distribution: Use a sidecar init container that injects the latest ca-bundle.crt from a ConfigMap mounted as a secret.
• NetworkPolicy Auditing: Run kubectl netpol lint in CI to ensure egress rules for external services are declared.
• Health Checks: Add a liveness probe that performs a lightweight curl -f https://mlflow-tracking.mycorp.com/api/2.0/mlflow/experiments/list call.
• DNS TTL Management: Set a low TTL (30 s) for the MLflow service DNS record to reduce stale cache impact during IP changes.
• Observability: Emit a custom metric mlflow_tracking_api_success_total from the batch job; alert on a drop below a threshold.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the batch job see “Connection refused” while an interactive pod can reach the MLflow server?

   The interactive pod likely has the mlflow-client label that satisfies the NetworkPolicy, whereas the batch job pod does not. Adjust the pod selector or add the label.

2. Can I use HTTP instead of HTTPS to avoid TLS issues?

   MLflow tracking server can be configured for plain HTTP, but this disables encryption and is not recommended for production. Proper CA distribution is the safer fix.

3. How do I know if the DNS cache is stale inside a long‑running pod?

   Run dig +short mlflow-tracking.mycorp.com inside the pod and compare the result with the current Service IP. If they differ, restart the pod or clear the glibc cache.

4. Is setting MLFLOW_TRACKING_INSECURE_TLS=true a permanent solution?

   No. It disables certificate verification and exposes traffic to man‑in‑the‑middle attacks. Use it only as a temporary bridge while fixing the trust store.

5. What monitoring alerts should I create for this failure mode?

   Alert on any of the following:

   • Increase in mlflow_tracking_api_error_total metric.
   • Pod restarts of batch jobs due to uncaught MlflowException.
   • NetworkPolicy audit failures for egress to mlflow-tracking.mycorp.com:443.