2026-08-13 12:01:45,123 INFO mlflow.tracking.client: Downloading artifact epoch_12.ckpt (size=642 MB) 2026-08-13 12:06:12,987 INFO mlflow.tracking.client: Artifact download completed in 267.4 seconds 2026-08-13 12:06:13,001 INFO training_loop: Warm‑started model from epoch 12 Metric validation Monitor mlflow.artifact_download_time_seconds (custom metric) to ensure it stays below the configured timeout. Check that no ReadTimeout exceptions appear in stderr or kubectl logs output. Prevention – Operational Guardrails Set a baseline timeout in the deployment manifest (e.g., Helm values): env: - name: MLFLOW_HTTP_REQUEST_TIMEOUT value: "300" Enable alerting on the mlflow.artifact_download_time_seconds metric when it exceeds 80 % of the configured timeout. Automate artifact size checks in CI/CD pipelines; reject checkpoints larger than a predefined threshold unless explicitly approved. Provision storage with predictable latency (e.g., S3 Transfer Acceleration, GCS Nearline with appropriate network routes) for training clusters. Document proxy timeout alignment in the runbook to avoid mismatched client/server settings. FAQ – Common Follow‑Up Questions Why does the timeout only happen on some epochs? Because object size and network tail latency vary. Larger checkpoints or transient storage throttling cause occasional downloads to exceed the 30 s default. Can I set a per‑model timeout instead of a global environment variable? Yes. Use MlflowClient(request_timeout=…) to override the timeout for a specific client instance, which can be scoped to a particular training script. Does increasing the timeout hide underlying storage performance problems? It mitigates crashes but does not solve root latency spikes. Consider storage tiering, checkpoint compression, or increasing provisioned IOPS if timeouts are frequent. How do I know the exact timeout value the MLflow client is using? Inspect client._timeout after construction or print the environment variable MLFLOW_HTTP_REQUEST_TIMEOUT. The client logs the timeout at DEBUG level when a request is sent. Will changing MLFLOW_HTTP_REQUEST_TIMEOUT affect other MLflow operations (e.g., metric logging)? No. The variable only controls HTTP request timeouts for REST calls, which include artifact downloads, metric logging, and run CRUD operations. Non‑network operations remain unaffected. Related Topic Hub: Model Serving Troubleshooting Hub

MLflow Model Loading Timeout During Iterative Training

Q: Root Cause – Default HTTP Request Timeout vs. Remote Artifact Latency

MLflow’s artifact download path uses the Python requests library with a default timeout of 30 seconds (see the MLflow Tracking Server REST API documentation). When the artifact store is an object storage service (S3, GCS, Azure Blob, Ceph), the actual download time depends on:

Q: 1. Reproduce the failure locally

# Simulate a slow download with curl curl -o /dev/null -w "%{time_total}\n" -m 35 \ "https://mlflow-tracking.mycompany.com/api/2.0/mlflow/artifacts/download?run_id=12345&path=epoch_12.ckpt" # Expected output > 30 indicates timeout will be hit by MLflow client 2. Inspect MLflow client configuration import os, mlflow print("MLFLOW_HTTP_REQUEST_TIMEOUT:", os.getenv("MLFLOW_HTTP_REQUEST_TIMEOUT")) print("mlflow.tracking.MlflowClient timeout:", mlflow.tracking.MlflowClient()._tracking_uri) If the environment variable is unset, the client falls back to the library default (30 s).

Q: 3. Capture network traces

# Capture the HTTP GET for the artifact sudo tcpdump -i eth0 -w artifact.pcap host mlflow-tracking.mycompany.com and port 443 # After reproducing the timeout, stop capture and inspect with Wireshark: # Look for long gaps between TCP ACKs or HTTP 504 responses. 4. Verify storage backend latency # Direct S3 download (bypass MLflow) to see raw latency aws s3 cp s3://my-bucket/models/epoch_12.ckpt /tmp/epoch_12.ckpt --no-progress # Observe the time taken; if >30 s, the storage layer is the bottleneck. 5. Check proxy timeout settings (if applicable) # Example Nginx config snippet http { proxy_read_timeout 30s; # default may be too low } If the tracking server sits behind such a proxy, increase proxy_read_timeout to match the expected download window.

Q: 2. Pass the timeout directly when constructing the client (Python‑level override).

import mlflow from mlflow.tracking import MlflowClient # Before – uses default timeout client = MlflowClient() # After – explicit timeout (seconds) client = MlflowClient(request_timeout=300) mlflow.set_tracking_uri("https://mlflow-tracking.mycompany.com") 3. Adjust server‑side or proxy timeout settings. For Nginx reverse proxy:

In this article

Table of Contents

Problem – Intermittent Model Loading Timeouts During Iterative Training

When a training loop repeatedly saves checkpoints to a remote MLflow tracking server and then calls mlflow.pyfunc.load_model at the start of each epoch, the script occasionally hangs or crashes with a requests.exceptions.ReadTimeout. Typical error messages observed in the logs include:

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mlflow-tracking.mycompany.com', port=443): Read timed out. (read timeout=30)
mlflow.exceptions.RestException: Error retrieving artifact: 504 Gateway Timeout
mlflow.exceptions.MlflowException: Unable to download artifact from URI s3://my-bucket/models/epoch_12.ckpt: Read timed out
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out

The failure is intermittent: some epochs load the checkpoint within seconds, while others exceed the default 30‑second HTTP request timeout, causing the training job to abort or be killed by the orchestrator.

Root Cause – Default HTTP Request Timeout vs. Remote Artifact Latency

MLflow’s artifact download path uses the Python requests library with a default timeout of 30 seconds (see the MLflow Tracking Server REST API documentation). When the artifact store is an object storage service (S3, GCS, Azure Blob, Ceph), the actual download time depends on:

Object size – checkpoints can be > 500 MB (GitHub issue #4567).
Network tail latency – spikes > 30 s have been recorded on EC2 with S3 and on‑prem Ceph during peak I/O (real incident logs).
Proxy or load‑balancer timeouts – Nginx reverse proxies default to 30 s, causing 504 responses (Kubernetes pod incident).

The combination of a large checkpoint and a transient latency spike means the client aborts the request before the server finishes sending the artifact, resulting in the observed ReadTimeout exceptions.

Investigation – Debugging the Timeout

Below is a reproducible debugging workflow that isolates the failure to the HTTP request timeout.

1. Reproduce the failure locally

# Simulate a slow download with curl
curl -o /dev/null -w "%{time_total}\n" -m 35 \
  "https://mlflow-tracking.mycompany.com/api/2.0/mlflow/artifacts/download?run_id=12345&path=epoch_12.ckpt"
# Expected output > 30 indicates timeout will be hit by MLflow client

2. Inspect MLflow client configuration

import os, mlflow
print("MLFLOW_HTTP_REQUEST_TIMEOUT:", os.getenv("MLFLOW_HTTP_REQUEST_TIMEOUT"))
print("mlflow.tracking.MlflowClient timeout:", mlflow.tracking.MlflowClient()._tracking_uri)

If the environment variable is unset, the client falls back to the library default (30 s).

3. Capture network traces

# Capture the HTTP GET for the artifact
sudo tcpdump -i eth0 -w artifact.pcap host mlflow-tracking.mycompany.com and port 443
# After reproducing the timeout, stop capture and inspect with Wireshark:
# Look for long gaps between TCP ACKs or HTTP 504 responses.

4. Verify storage backend latency

# Direct S3 download (bypass MLflow) to see raw latency
aws s3 cp s3://my-bucket/models/epoch_12.ckpt /tmp/epoch_12.ckpt --no-progress
# Observe the time taken; if >30 s, the storage layer is the bottleneck.

5. Check proxy timeout settings (if applicable)

# Example Nginx config snippet
http {
    proxy_read_timeout 30s;   # default may be too low
}

If the tracking server sits behind such a proxy, increase proxy_read_timeout to match the expected download window.

Solution – Extending the HTTP Request Timeout and Optimizing Artifact Access

The fix consists of two parts: (1) raise the client‑side timeout, and (2) optionally reduce artifact size or improve storage performance.

1. Set `MLFLOW_HTTP_REQUEST_TIMEOUT` to a value larger than the worst‑case download time.

Example: increase to 300 seconds (5 minutes).

# Before (default)
export MLFLOW_HTTP_REQUEST_TIMEOUT=30   # implicit, not set

# After
export MLFLOW_HTTP_REQUEST_TIMEOUT=300

MLflow reads this environment variable at import time (see MLflow configuration guide).

2. Pass the timeout directly when constructing the client (Python‑level override).

import mlflow
from mlflow.tracking import MlflowClient

# Before – uses default timeout
client = MlflowClient()

# After – explicit timeout (seconds)
client = MlflowClient(request_timeout=300)
mlflow.set_tracking_uri("https://mlflow-tracking.mycompany.com")

3. Adjust server‑side or proxy timeout settings.

For Nginx reverse proxy:

# /etc/nginx/conf.d/mlflow.conf
server {
    listen 443 ssl;
    location / {
        proxy_pass http://mlflow-backend:5000;
        proxy_read_timeout 300s;   # match client timeout
        proxy_connect_timeout 300s;
    }
}

4. Optional – Reduce checkpoint payload.

Compress checkpoints (e.g., torch.save(..., _use_new_zipfile_serialization=False) then gzip).
Store only model weights (exclude optimizer state) when warm‑starting.

Verification – Confirming the Fix

Run the training loop with the new timeout and observe that each epoch successfully loads the checkpoint.

Log verification

2026-08-13 12:01:45,123 INFO mlflow.tracking.client: Downloading artifact epoch_12.ckpt (size=642 MB)
2026-08-13 12:06:12,987 INFO mlflow.tracking.client: Artifact download completed in 267.4 seconds
2026-08-13 12:06:13,001 INFO training_loop: Warm‑started model from epoch 12

Metric validation

Monitor mlflow.artifact_download_time_seconds (custom metric) to ensure it stays below the configured timeout.
Check that no ReadTimeout exceptions appear in stderr or kubectl logs output.

Prevention – Operational Guardrails

Set a baseline timeout in the deployment manifest (e.g., Helm values):
```
env:
  - name: MLFLOW_HTTP_REQUEST_TIMEOUT
    value: "300"
```
Enable alerting on the mlflow.artifact_download_time_seconds metric when it exceeds 80 % of the configured timeout.
Automate artifact size checks in CI/CD pipelines; reject checkpoints larger than a predefined threshold unless explicitly approved.
Provision storage with predictable latency (e.g., S3 Transfer Acceleration, GCS Nearline with appropriate network routes) for training clusters.
Document proxy timeout alignment in the runbook to avoid mismatched client/server settings.

FAQ – Common Follow‑Up Questions

Why does the timeout only happen on some epochs?
Because object size and network tail latency vary. Larger checkpoints or transient storage throttling cause occasional downloads to exceed the 30 s default.
Can I set a per‑model timeout instead of a global environment variable?
Yes. Use MlflowClient(request_timeout=…) to override the timeout for a specific client instance, which can be scoped to a particular training script.
Does increasing the timeout hide underlying storage performance problems?
It mitigates crashes but does not solve root latency spikes. Consider storage tiering, checkpoint compression, or increasing provisioned IOPS if timeouts are frequent.
How do I know the exact timeout value the MLflow client is using?
Inspect client._timeout after construction or print the environment variable MLFLOW_HTTP_REQUEST_TIMEOUT. The client logs the timeout at DEBUG level when a request is sent.
Will changing MLFLOW_HTTP_REQUEST_TIMEOUT affect other MLflow operations (e.g., metric logging)?
No. The variable only controls HTTP request timeouts for REST calls, which include artifact downloads, metric logging, and run CRUD operations. Non‑network operations remain unaffected.

Related Topic Hub: Model Serving Troubleshooting Hub