Shard Rebalancing Fails in Docker Container on Azure VM

Problem Description

A distributed AI training workload runs inside a Docker container on an Azure Virtual Machine. The storage layer is sharded (e.g., Milvus, Redis Cluster, or a custom file‑system). During normal operation the coordinator attempts to relocate shards to balance load, but the operation aborts with errors such as:

ShardRebalanceFailedException: metadata version mismatch (expected 42, got 37)
TimeoutError: failed to contact peer 9f8c7e2a3b1d within 30s during shard relocation
Error acquiring shard lock: lock held by another node (possible stale metadata)
Docker container network timeout – failed to reach shard coordinator at 10.0.0.5:2379
Failed to update shard metadata in etcd: request timed out (etcdserver: request timed out)

The symptoms manifest as:

• Stalled or failed training epochs due to missing data partitions.
• Repeated “heartbeat missed” warnings in container logs.
• Azure Monitor metrics showing increased network latency spikes on the VM NIC.
• No automatic recovery; manual pod/container restarts temporarily unblock the rebalance.

Root Cause Analysis

The failure is a combination of two Azure‑specific factors:

1. Inconsistent metadata propagation – Docker containers on a single Azure VM share the host’s network stack. When the VM is rebooted or the NIC is re‑initialized, the internal etcd or coordination service loses its leader election state. Containers that retain an older metadata version continue to advertise stale shard maps, leading to the metadata version mismatch error. This matches the GitHub Milvus issue where “metadata version conflict” appeared after a VM restart.
2. Network timeout caused by NSG or MTU misconfiguration – Azure Virtual Network security groups (NSGs) default to allowing intra‑VM traffic, but custom rules often block the high‑port range used by the coordination service (e.g., etcd on 2379‑2380). Additionally, the default MTU of 1500 can conflict with the overlay network used by Docker’s bridge driver, causing packet fragmentation and timeout. The Stack Overflow Elasticsearch case identified an MTU mismatch as the source of “connection timed out” errors.

Both issues prevent containers from reliably exchanging heartbeat and lock acquisition messages, causing the coordinator to abort the rebalance.

Investigation and Debugging Steps

1. Capture Container and Host Logs

# View the latest logs from the sharding service container
docker logs -f sharding-service
# Example excerpt
2026-06-24T12:34:56Z ERROR ShardRebalanceFailedException: metadata version mismatch (expected 42, got 37)
2026-06-24T12:34:58Z WARN Heartbeat missed from peer 9f8c7e2a3b1d
2026-06-24T12:35:01Z ERROR TimeoutError: failed to contact peer 9f8c7e2a3b1d within 30s during shard relocation

2. Verify Network Connectivity Between Containers

# List container IPs on the bridge network
docker network inspect bridge --format='{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'

# Test TCP connectivity to the etcd coordinator (replace IP with actual)
nc -zv 10.0.0.5 2379
# Expected output
Connection to 10.0.0.5 2379 port [tcp/*] succeeded!

If the nc command times out, the issue is network‑level.

3. Inspect Azure NSG Rules

# List NSG rules via Azure CLI (replace  and )
az network nsg rule list --resource-group  --nsg-name  \
    --output table

Ensure inbound and outbound rules allow traffic on ports 2379‑2380 and the dynamic range used by Docker (typically 30000‑32767).

4. Check MTU Settings

# Host MTU
ip link show eth0 | grep mtu
# Docker bridge MTU (default 1500)
docker network inspect bridge --format='{{.Options}}'

# Compare with Azure VNet MTU (usually 1500, but can be lower on VPN/ExpressRoute)

Mismatch leads to fragmented packets and retransmissions, observable in tcpdump captures:

# Capture traffic to the coordinator for 10 seconds
tcpdump -i eth0 host 10.0.0.5 and port 2379 -vv -c 100

5. Validate Etcd Cluster Health

# Inside the container running etcd
etcdctl endpoint health --cluster
# Expected output
127.0.0.1:2379 is healthy

If health checks fail, the cluster likely lost quorum after the VM reboot.

Resolution

1. Align MTU Across Host, Docker Bridge, and Azure VNet

Set the Docker daemon to use the host’s MTU (e.g., 1460 for Azure VNet with VPN overhead):

# /etc/docker/daemon.json (before)
{
    "default-address-pools": [
        { "base":"10.10.0.0/16","size":24 }
    ]
}

# /etc/docker/daemon.json (after)
{
    "default-address-pools": [
        { "base":"10.10.0.0/16","size":24 }
    ],
    "mtu": 1460
}

Restart Docker to apply:

systemctl restart docker

2. Open Required Ports in the NSG

# Add inbound rule for etcd ports
az network nsg rule create \
  --resource-group MyRG \
  --nsg-name MyVM-NSG \
  --name AllowEtcd \
  --priority 1000 \
  --protocol Tcp \
  --direction Inbound \
  --source-address-prefixes VirtualNetwork \
  --source-port-ranges '*' \
  --destination-address-prefixes VirtualNetwork \
  --destination-port-ranges 2379-2380 \
  --access Allow

3. Force Metadata Synchronization on Container Startup

Modify the container entrypoint to purge stale metadata and force a fresh join to the coordination service:

# entrypoint.sh (before)
exec ./sharding-service --config /etc/sharding/config.yaml

# entrypoint.sh (after)
#!/bin/bash
# Remove any persisted metadata version file
rm -f /var/lib/sharding/metadata.version
# Wait for etcd health
until etcdctl endpoint health --cluster; do
  echo "Waiting for etcd..."
  sleep 2
done
exec ./sharding-service --config /etc/sharding/config.yaml

4. Enable Automatic Leader Re‑election in Etcd

Set a lower election timeout to recover faster after a NIC reset:

# etcd.conf (before)
ETCD_ELECTION_TIMEOUT=5000

# etcd.conf (after)
ETCD_ELECTION_TIMEOUT=2000

Verification

1. Confirm that the Docker bridge now uses the corrected MTU:

docker network inspect bridge --format='{{.Options}}'
# Output should contain "com.docker.network.driver.mtu=1460"

2. Validate NSG rule propagation (takes up to 30 seconds):

az network nsg rule list --resource-group MyRG --nsg-name MyVM-NSG \
  --query "[?name=='AllowEtcd']"

3. Check etcd health again from inside the container:

etcdctl endpoint health --cluster
# Expected: all members report "is healthy"

4. Trigger a manual rebalance (most sharding frameworks expose an API endpoint):

curl -X POST http://localhost:8080/api/rebalance
# Expected response
{"status":"success","details":"Rebalance initiated"}

5. Observe logs for successful completion:

docker logs -f sharding-service | grep -i "rebalance completed"
# Example line
2026-06-24T13:02:14Z INFO Shard rebalance completed successfully for 12 shards

Prevention and Best Practices

• Network hygiene: Keep NSG rules minimal and explicitly allow intra‑VM traffic on coordination ports. Use Azure VNet Service Tags (e.g., VirtualNetwork) to simplify management.
• Consistent MTU: Align host, Docker, and any VPN/ExpressRoute MTU values. Document the chosen MTU in the infrastructure repo.
• Stateless container start‑up: Ensure containers do not persist coordination metadata across restarts. Use volume mounts with proper cleanup scripts.
• Health‑check driven restarts: Configure Docker health checks that query etcd health; let Docker restart the container automatically on failure.
• Monitoring: Enable Azure Monitor Container Insights to track network latency, packet drops, and etcd leader changes. Set alerts on etcd_server_leader_changes_seen_total and on container restarts.
• Graceful VM reboot handling: Use Azure VM extensions or a startup script to flush stale metadata and wait for etcd quorum before launching the sharding service.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

1. Why does the rebalance succeed after a container restart but fail after a VM reboot?
   A VM reboot resets the host NIC and can temporarily drop intra‑VM traffic. Containers retain stale metadata, causing version mismatches. Restarting the container forces metadata cleanup and re‑join, while the host network may still be unstable after a reboot.
2. Can I keep the default Docker MTU of 1500?
   Only if the Azure VNet (including VPN/ExpressRoute) also uses 1500. When the underlying path reduces the MTU, packets get fragmented and time out, leading to the observed errors. Aligning the MTU eliminates this class of failures.
3. Is it safe to open the etcd ports to the entire VNet?
   Yes, because the traffic stays within the same VNet and is isolated from the public internet. Using the VirtualNetwork service tag restricts exposure to only resources in the same VNet.
4. How do I know if the metadata version mismatch is caused by a leader election loss?
   Check etcd logs for “election timeout” or “new leader elected” messages. Correlate with container logs that report “metadata version X expected, got Y”. A recent leader change often precedes the mismatch.
5. What metric should I alert on to catch future rebalance failures early?
   Alert on container_logs{message=~"ShardRebalanceFailedException|TimeoutError"} in Azure Monitor, and on etcd_server_leader_changes_seen_total spikes greater than 1 within a 5‑minute window.