photo of computer cables

MLflow tracking server write conflicts after etcd quorum loss

by

Problem Description

During a high‑throughput inference deployment (≈10 k requests / s), the MLflow tracking server began returning errors when logging run parameters, metrics, and feature versions. The observed symptoms included:

  • etcdserver: request timed out in the MLflow server logs.
  • etcdserver: leader changed followed by retries.
  • Duplicate run UUIDs and missing metric rows in the PostgreSQL backend.
  • Intermittent compare failed errors from the etcd concurrency API.

These failures coincided with a transient network partition that caused the etcd cluster to lose quorum.

Root Cause Analysis

MLflow’s tracking server uses a relational store (e.g., PostgreSQL) for persistent metadata and an optional etcd cluster for distributed locks and lease handling (MLflow Backend Store implementation details). When the etcd cluster cannot form a quorum, the following chain of events occurs:

  1. Leader Election Failure: etcd requires a majority of members to elect a leader (etcd Cluster Management). A network partition leaves the cluster in a minority state, causing the current leader to step down and new elections to fail.
  2. Lock/Lease Unavailability: MLflow acquires a lock via the etcd Lock primitive before creating a run ID. Without a leader, lock acquisition blocks and eventually times out, surfacing as etcdserver: request timed out.
  3. Optimistic Concurrency Conflicts: Concurrent inference workers attempt to create the same run UUID or metric key. In a healthy etcd cluster, the compare‑and‑swap operation succeeds for one worker and the others retry. During quorum loss, the compare operation cannot be committed, leading to compare failed errors (etcd Concurrency API).
  4. Split‑Brain Writes: If the MLflow server is restarted while etcd is in a minority partition, it may cache stale lease information. Subsequent writes are directed to the wrong etcd endpoint, producing out‑of‑order timestamps and duplicate run IDs (GitHub issue).
  5. Metadata Inconsistency: The relational store receives partial writes (run row persisted, metric rows rejected), resulting in gaps in time‑series logs and “missing metrics” reports (mlflow‑users group).

Investigation and Debugging

The following step‑by‑step process isolates the failure mode:

1. Inspect MLflow server logs


2026-07-28 14:12:03,721 ERROR mlflow.tracking.store.sqlalchemy_store - Unable to persist run: etcdserver: request timed out
2026-07-28 14:12:04,012 WARN  mlflow.tracking.store.sqlalchemy_store - etcdserver: leader changed, retrying
2026-07-28 14:12:04,317 ERROR mlflow.tracking.store.sqlalchemy_store - Failed to log metric: compare failed

2. Verify etcd cluster health


# Check member health
etcdctl endpoint health --cluster
# Expected output (healthy):
localhost:2379 is healthy: successfully committed proposal
localhost:2380 is healthy: successfully committed proposal
localhost:2381 is healthy: successfully committed proposal

# During partition you will see:
localhost:2379 is unhealthy: context deadline exceeded

3. Examine leader election status


etcdctl endpoint status --write-out=table
+----------------+------------------+----------+----------+------------+-----------+------------+
|   ENDPOINT     |   VERSION        | DB SIZE  | IS LEADER|  LEADER    | RAFT TERM | RAFT INDEX |
+----------------+------------------+----------+----------+------------+-----------+------------+
| localhost:2379| 3.5.9            |  12 MB   | false    | 2380       |  12       |  1023456   |
| localhost:2380| 3.5.9            |  12 MB   | true     | 2380       |  12       |  1023457   |
| localhost:2381| 3.5.9            |  12 MB   | false    | 2380       |  12       |  1023455   |
+----------------+------------------+----------+----------+------------+-----------+------------+

4. Capture a packet trace (optional)


tcpdump -i eth0 port 2379 -w etcd-partition.pcap

5. Check PostgreSQL run table for duplicate IDs


SELECT run_uuid, COUNT(*) FROM runs GROUP BY run_uuid HAVING COUNT(*) > 1;

6. Review MLflow configuration


# Current launch command (problematic)
mlflow server \
  --backend-store-uri postgresql://mlflow:pwd@db:5432/mlflow \
  --default-artifact-root s3://mlflow-artifacts \
  --host 0.0.0.0 \
  --port 5000 \
  --workers 8 \
  --etcd-hosts http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379

Resolution

The fix combines immediate remediation (restore quorum) with configuration changes that make the tracking server tolerant to transient etcd outages.

1. Restore etcd quorum

  • Identify the minority partition and bring the missing nodes back online.
  • If a node is permanently lost, remove it from the cluster and re‑add a fresh member (etcdctl member remove, etcdctl member add).

2. Enable retry with exponential backoff in MLflow

MLflow 2.7 introduced a --etcd-retry-max flag (see PR Add etcd write‑conflict handling). Update the launch command:


# Updated launch command
mlflow server \
  --backend-store-uri postgresql://mlflow:pwd@db:5432/mlflow \
  --default-artifact-root s3://mlflow-artifacts \
  --host 0.0.0.0 \
  --port 5000 \
  --workers 8 \
  --etcd-hosts http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379 \
  --etcd-retry-max 5 \
  --etcd-retry-backoff 200ms

3. Tighten etcd timeouts

Adjust the client timeout to avoid premature failures:


export ETCDCTL_API=3
export ETCDCTL_DIAL_TIMEOUT=5s
export ETCDCTL_COMMAND_TIMEOUT=10s

4. Separate lock store from main etcd cluster

Deploy a dedicated three‑node etcd cluster solely for MLflow lock/lease handling. This isolates lock traffic from other workloads that may saturate the cluster’s quota (GitHub issue).

5. Apply optimistic concurrency guard

Patch the MLflow source to catch compare failed errors and retry the entire run creation transaction. Example diff:


--- a/mlflow/store/tracking/sqlalchemy_store.py
+++ b/mlflow/store/tracking/sqlalchemy_store.py
@@ -342,6 +342,12 @@ def create_run(self, experiment_id, user_id, run_name, start_time,
         try:
             self._etcd_lock.acquire()
             # existing logic …
         except EtcdCompareFailedError:
-            raise MlflowException("Failed to create run due to etcd compare failure")
+            # Retry once with backoff
+            time.sleep(0.2)
+            self._etcd_lock.acquire()
+            # repeat the create logic
+            # if it fails again, raise the original exception
+            raise
         finally:
             self._etcd_lock.release()

Verification

After applying the changes, perform the following checks:

1. Health endpoint


curl -s http://mlflow:5000/health | jq .
{
  "status": "OK",
  "components": {
    "postgres": "UP",
    "etcd": "UP"
  }
}

2. Simulate a brief partition

  • Block network traffic to one etcd node for 10 seconds.
  • Issue a batch of mlflow.log_metric calls from a test client.
  • Confirm that the client receives no exceptions and that all metrics appear in the PostgreSQL metrics table.

3. Confirm no duplicate runs


SELECT COUNT(*) FROM runs;
SELECT COUNT(DISTINCT run_uuid) FROM runs;
-- Both counts should be equal

4. Monitor etcd leader stability


watch -n 5 "etcdctl endpoint status --write-out=table"

Prevention and Best Practices

  • Quorum‑aware deployment: Deploy etcd nodes across at least three distinct failure domains (AZs or zones) with anti‑affinity rules to avoid simultaneous loss.
  • Alert on etcd health: Set up Prometheus alerts for etcd_server_has_leader == 0 and etcd_server_is_healthy == 0.
  • Rate‑limit logging traffic: Introduce client‑side backpressure or token bucket throttling to keep etcd write QPS below the quota‑exceeded threshold.
  • Separate lock store: Use a lightweight three‑node etcd cluster dedicated to MLflow lock/lease handling; keep the main etcd cluster for other stateful services.
  • Enable retry with jitter: Configure --etcd-retry-max and --etcd-retry-backoff to avoid thundering‑herd retries during leader elections.
  • Periodic consistency checks: Run a nightly job that scans the runs and metrics tables for gaps or duplicate UUIDs and reports anomalies.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the error appear only under high‑throughput inference?
    High request rates increase the number of concurrent lock acquisitions. During a quorum loss, the probability of multiple workers colliding on the same lock spikes, exposing the compare failed path.
  2. Can I run MLflow without etcd?
    Yes. MLflow’s core metadata is stored in the relational backend. Etcd is only required for distributed locking. In single‑node deployments you can omit the --etcd-hosts flag.
  3. What is the recommended etcd timeout for MLflow?
    The official docs suggest a client timeout of 5 seconds and a request timeout of 10 seconds. Adjust higher only if your network latency is unusually large.
  4. How do I know if a run ID was duplicated?
    Query the runs table for duplicate run_uuid values. Duplicate IDs indicate that two workers succeeded in creating the same lock during a split‑brain scenario.
  5. Does increasing the number of etcd members solve the problem?
    Adding members improves fault tolerance but also raises the quorum size. Ensure the deployment can tolerate the loss of a majority of nodes (e.g., 5 members require 3 up).