MLflow Tracking Server Not Found In Local Docker Setup

Q: Binding address of the server process

By default mlflow server binds to 127.0.0.1 inside the container. If the --host flag is omitted, the server only accepts connections from the container’s loopback interface. Even if the host forwards port 5000, external containers cannot reach the service because the server is not listening on 0.0.0.0.

Q: 5. Verify connectivity from the client container

Run a simple curl against the tracking endpoint.

In this article

Table of Contents

Problem: MLflow tracking server not found in a local Docker development setup

During local development an engineer runs an MLflow tracking server inside a Docker container (or via docker‑compose) and then starts a separate client container (or a host‑side script) that attempts to log runs. The client immediately fails with errors such as:

mlflow.exceptions.RestException: Unable to resolve tracking URI http://localhost:5000 – Connection refused
requests.exceptions.ConnectionError: HTTPConnectionPool(host='mlflow', port=5000): Max retries exceeded with url: /api/2.0/mlflow/runs/create (Caused by NewConnectionError)
mlflow.exceptions.MlflowException: Failed to connect to tracking server at http://127.0.0.1:5000 – timeout

From the developer’s perspective the server container appears to be running, the port is published, and the mlflow ui is reachable from a browser on the host. Yet the client cannot locate the tracking endpoint.

Root Cause Analysis

Network namespace isolation

Docker isolates the network stack of each container. The hostname localhost (or 127.0.0.1) always resolves to the loopback interface of the *current* container, not the host or another container. When the client uses http://localhost:5000 it attempts to connect to its own loopback interface, where no MLflow process is listening, leading to Connection refused.

Binding address of the server process

By default mlflow server binds to 127.0.0.1 inside the container. If the --host flag is omitted, the server only accepts connections from the container’s loopback interface. Even if the host forwards port 5000, external containers cannot reach the service because the server is not listening on 0.0.0.0.

Docker‑Compose network segmentation

When services are placed on distinct custom networks, Docker DNS does not resolve the service name across networks. A client container trying to reach http://mlflow:5000 will receive “Failed to resolve tracking URI” unless both services share the same network or an explicit network_alias is defined.

Persisted backend storage loss

If the server runs with the default SQLite backend stored in a container‑local path (e.g., /mlflow/mlruns) and the container is recreated without a volume, the database disappears. Subsequent client calls receive 404 Not Found because the server cannot locate the requested experiment.

Investigation and Debugging Steps

Confirm server container is listening on the expected interface.

docker exec -it mlflow_server ss -ltnp | grep 5000
# Expected output:
LISTEN 0      128          0.0.0.0:5000          0.0.0.0:*    users:(("python",pid=12,fd=5))

If the output shows 127.0.0.1:5000 instead of 0.0.0.0:5000, the server is bound only to loopback.

Check port exposure and host mapping.

docker ps --filter "name=mlflow_server"
# Example line:
mlflow_server   mlflow:2.8.0   "mlflow server …"   0.0.0.0:5000->5000/tcp

If the PORTS column is empty, the container is not publishing the port.

Validate DNS resolution inside the client container.

docker exec -it mlflow_client getent hosts mlflow
# Expected output (if on same network):
mlflow 172.20.0.5

Failure to resolve indicates network isolation.

Capture a network trace from the client.

# Inside client container
apt-get update && apt-get install -y tcpdump
tcpdump -i any -nn host 172.20.0.5 and port 5000 -c 5

No packets means the client never reached the server IP.

Inspect MLflow server logs for incoming request errors.

docker logs mlflow_server
# Look for lines like:
INFO:uvicorn.error:Started server process [12]
ERROR:uvicorn.error:Cannot connect to database file /mlflow/mlruns/… (No such file or directory)

Resolution

1. Bind the server to `0.0.0.0`

Update the server start command to explicitly listen on all interfaces.

Before:

docker run -d --name mlflow_server -p 5000:5000 mlflow:2.8.0 mlflow server \
    --backend-store-uri sqlite:///mlflow.db \
    --default-artifact-root /mlflow/artifacts

After:

docker run -d --name mlflow_server -p 5000:5000 mlflow:2.8.0 mlflow server \
    --host 0.0.0.0 \
    --backend-store-uri sqlite:///mlflow.db \
    --default-artifact-root /mlflow/artifacts

2. Use a stable hostname or Docker‑Compose service name

When both server and client run in Docker Compose, place them on the same network and refer to the server by its service name.

# docker-compose.yml
version: "3.9"
services:
  mlflow:
    image: mlflow:2.8.0
    command: >
      mlflow server
      --host 0.0.0.0
      --backend-store-uri sqlite:///mlflow.db
      --default-artifact-root /mlflow/artifacts
    ports:
      - "5000:5000"
    volumes:
      - mlflow-data:/mlflow
    networks:
      - mlnet

  client:
    image: python:3.11
    depends_on:
      - mlflow
    environment:
      - MLFLOW_TRACKING_URI=http://mlflow:5000
    networks:
      - mlnet
    command: ["python", "-c", "import mlflow; mlflow.set_tracking_uri('http://mlflow:5000'); print('ready')"]

volumes:
  mlflow-data:

networks:
  mlnet:

3. Persist the backend store

Mount a Docker volume for the SQLite file (or use a remote DB) so that container restarts do not erase experiment data.

# In the compose snippet above, the volume `mlflow-data` ensures /mlflow persists.
# If using MySQL/PostgreSQL, replace the URI:
--backend-store-uri postgresql://mlflow_user:pwd@db:5432/mlflow

4. For host‑side clients, use `host.docker.internal`

If the client runs on the host (not inside Docker) but the server is containerised, the correct URI is:

export MLFLOW_TRACKING_URI=http://host.docker.internal:5000

5. Verify connectivity from the client container

Run a simple curl against the tracking endpoint.

docker exec -it client_container curl -s -o /dev/null -w "%{http_code}" http://mlflow:5000/api/2.0/mlflow/runs/create
# Expected output: 200

Validation

Health check the tracking server.

curl -s http://localhost:5000/health
# Expected JSON:
{"status":"OK"}

Run a quick MLflow experiment from the client.

python - <<'PY'
import mlflow
mlflow.set_tracking_uri("http://mlflow:5000")
with mlflow.start_run():
    mlflow.log_param("alpha", 0.5)
    mlflow.log_metric("rmse", 0.42)
print("Run logged:", mlflow.active_run().info.run_id)
PY

Successful execution should print a run ID and no traceback.

Inspect the server logs for the new run entry.

docker logs mlflow_server | grep "Run created"
# Example line:
INFO:mlflow.tracking:Run created with ID 1234567890abcdef

Confirm persistence across container restarts.

# Restart server
docker compose restart mlflow
# Re‑run the client script; the same experiment ID should be reusable.

Operational Experience & Prevention

Misleading symptom: The UI is reachable from the host browser, leading engineers to assume the tracking endpoint is also reachable from any container. The UI uses the same port but binds to 0.0.0.0 only when --host is set; otherwise the UI works locally but not from other containers.
Common incorrect assumption: “If I expose port 5000, any container can use localhost:5000.” In Docker each container has its own loopback; the correct address is either the service name, the container IP, or host.docker.internal for host‑side clients.
Production‑like edge case: When multiple developer machines share the same Docker network, static IPs can clash. Using Docker DNS (service name) avoids hard‑coding IPs.
Lesson learned: Always start the MLflow server with --host 0.0.0.0 in any containerised environment and persist the backend store via a volume or external database.

Best Practices and Prevention

Declare MLFLOW_TRACKING_URI in a .env file and reference it in both Docker Compose and local scripts.
Expose the tracking port only on the internal Docker network; use ports mapping only when host access is required.
Mount a persistent volume for mlflow.db or use a managed DB (PostgreSQL/MySQL) for multi‑container stability.

Add a health‑check to the Compose service:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
  interval: 30s
  timeout: 5s
  retries: 3

Enable structured logging (e.g., --workers 2 --log-level INFO) to capture request IDs for faster debugging.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does mlflow.set_tracking_uri('http://localhost:5000') work on my laptop but fail inside Docker?
Because localhost inside a container resolves to the container itself. Use the service name, container IP, or host.docker.internal instead.
Can I run the MLflow server without exposing a host port?
Yes. When both client and server share the same Docker network, you only need the internal port (e.g., 5000) and can omit the ports section. The client uses the service DNS name.
My client still receives “404 Not Found” after fixing the network. What could be wrong?
Most often the SQLite backend was stored inside the container and got lost on restart. Mount a volume or switch to an external DB to retain experiment metadata.
Is there a way to test the tracking URI without writing Python code?
Yes. The MLflow REST API is publicly documented. A simple curl to /api/2.0/mlflow/runs/create with a JSON payload will return 200 if the server is reachable.
Do I need to set MLFLOW_TRACKING_INSECURE_TLS for local Docker?
Only if you enable HTTPS in the server. For plain HTTP in a trusted local environment, no extra env vars are required.