Ingress Routing Misconfiguration In Docker GPU Cluster

Q: 1. Verify service reachability

# Attempt to reach the service via the ingress address curl -s -o /dev/null -w "%{http_code}\n" http://my-swarm-ingress:8080 # Expected: 200 # Observed: 502 or 504 2. Inspect service definition docker service inspect --pretty gpu-api Key fields to note:

Q: 3. Check overlay network MTU

# Show the overlay network configuration docker network inspect ingress -f '{{json .Options}}' | jq . Typical output:

Q: 4. Compare NIC MTU on GPU nodes

# On each GPU node ip link show eth0 | grep mtu If the NIC reports mtu 9000 (jumbo frames) while the overlay is 1500, fragmentation will occur.

Q: 5. Verify firewall rules

# List iptables rules that may block Swarm traffic iptables -S | grep -E '7946|4789' # Or with nftables nft list ruleset | grep -E '7946|4789' Missing ACCEPT rules for UDP 7946/4789 indicate a blockage.

Q: 6. Examine Docker daemon logs for MTU errors

journalctl -u docker -f | grep -i mtu Typical log line:

Q: 7. Check manager CPU load

top -b -n1 | grep dockerd # Or via Docker metrics docker stats $(docker ps -q --filter "name=ingress") CPU > 80 % on the manager correlates with connection resets.

In this article

Table of Contents

Problem Description

In a Docker Swarm GPU cluster (A100/H100), services that request --gpus become unreachable through the Swarm ingress routing mesh. Clients receive HTTP 502/504 errors, and the Docker daemon logs report failures such as:

Failed to create endpoint: network ingress: driver failed programming the network: MTU mismatch
Ingress routing mesh: connection timed out (504) when accessing service on a GPU node
Error response from daemon: cannot allocate resources: insufficient GPU devices
docker-proxy: error while accepting connections: bind: address already in use

The issue appears after a driver upgrade (e.g., NVIDIA driver 525) or after adding a new GPU node to the Swarm. The problem is isolated to the ingress network; direct host mode publishing works, and GPU‑enabled containers run correctly when accessed via the node’s IP.

Root Cause Analysis

Multiple interacting factors cause the ingress routing failure:

Overlay MTU mismatch – The default overlay MTU (1500) does not match the physical NIC MTU on A100/H100 nodes after the driver update, leading to fragmented VXLAN packets that the ingress router discards. This matches the error “driver failed programming the network: MTU mismatch” observed in the Docker daemon logs (GitHub issue #4532).
Firewall rules blocking Swarm control traffic – UDP ports 7946 (cluster communication) and 4789 (VXLAN) are often blocked by host‑level firewalls on GPU nodes. Without VXLAN tunneling, the routing mesh cannot reach the service endpoints (NVIDIA issue #1125).
DNS propagation delay for the ingress network – When a new GPU node joins, the manager’s internal DNS does not immediately advertise the node’s IP on the ingress overlay, causing intermittent 504 timeouts (Stack Overflow 73584219).
CPU saturation on the ingress router – High‑throughput GPU workloads generate a large number of connection handshakes. The default ingress router (built into the manager) can become a bottleneck, resetting connections (Stack Overflow 77912345).

All of these stem from the same misconfiguration: the overlay network is not tuned for the hardware and security posture of a GPU‑heavy Swarm.

Investigation and Debugging

The following step‑by‑step investigation reproduces the typical debugging workflow.

1. Verify service reachability

# Attempt to reach the service via the ingress address
curl -s -o /dev/null -w "%{http_code}\n" http://my-swarm-ingress:8080
# Expected: 200
# Observed: 502 or 504

2. Inspect service definition

docker service inspect --pretty gpu-api

Key fields to note:

EndpointSpec.Mode=vip (default routing mesh)
Resources.Reservations.GenericResources contains GPU=1
EndpointSpec.Ports shows TargetPort=8080 PublishedPort=80 Mode=ingress

3. Check overlay network MTU

# Show the overlay network configuration
docker network inspect ingress -f '{{json .Options}}' | jq .

Typical output:

{
  "com.docker.network.driver.overlay.vxlanid_list": "4096",
  "com.docker.network.driver.mtu": "1500"
}

4. Compare NIC MTU on GPU nodes

# On each GPU node
ip link show eth0 | grep mtu

If the NIC reports mtu 9000 (jumbo frames) while the overlay is 1500, fragmentation will occur.

5. Verify firewall rules

# List iptables rules that may block Swarm traffic
iptables -S | grep -E '7946|4789'
# Or with nftables
nft list ruleset | grep -E '7946|4789'

Missing ACCEPT rules for UDP 7946/4789 indicate a blockage.

6. Examine Docker daemon logs for MTU errors

journalctl -u docker -f | grep -i mtu

Typical log line:

time="2024-07-12T10:45:23.123456789Z" level=error msg="Failed to create endpoint: network ingress: driver failed programming the network: MTU mismatch"

7. Check manager CPU load

top -b -n1 | grep dockerd
# Or via Docker metrics
docker stats $(docker ps -q --filter "name=ingress")

CPU > 80 % on the manager correlates with connection resets.

Resolution

Apply the following changes in the order presented. Each step resolves a specific root cause.

1. Align overlay MTU with NIC MTU

Recreate the ingress overlay with an explicit MTU that matches the underlying NIC (e.g., 9000 for jumbo frames).

# Remove the default ingress network (requires Swarm mode to be temporarily disabled)
docker network rm ingress
# Re‑create with proper MTU
docker network create \
  --driver overlay \
  --opt com.docker.network.driver.mtu=9000 \
  --opt encrypted \
  ingress

After recreation, verify:

docker network inspect ingress -f '{{json .Options}}' | jq .

Output should show "com.docker.network.driver.mtu":"9000".

2. Open required UDP ports on all GPU nodes

# Example using firewalld
firewall-cmd --add-port=7946/udp --permanent
firewall-cmd --add-port=4789/udp --permanent
firewall-cmd --reload

# Example using iptables
iptables -I INPUT -p udp -m udp --dport 7946 -j ACCEPT
iptables -I INPUT -p udp -m udp --dport 4789 -j ACCEPT
iptables-save > /etc/iptables/rules.v4

3. Force DNS propagation after node join

Trigger a DNS refresh on the manager:

# On the manager node
docker node update --label-add dns.refresh=true $(docker node ls -q --filter "role=manager")
# Or restart the manager daemon
systemctl restart docker

4. Scale the ingress router to avoid CPU saturation

Deploy an external load balancer (e.g., HAProxy) in front of the Swarm manager and publish ports in host mode. Update the service to use mode=host for the critical GPU workloads.

# Original service (ingress mode)
docker service create \
  --name gpu-api \
  --publish published=80,target=8080,mode=ingress \
  --constraint 'node.labels.gpu==true' \
  --gpus 1 \
  myorg/gpu-api:latest

# Updated service (host mode)
docker service rm gpu-api
docker service create \
  --name gpu-api \
  --publish published=80,target=8080,mode=host \
  --constraint 'node.labels.gpu==true' \
  --gpus 1 \
  myorg/gpu-api:latest

Using host mode bypasses the routing mesh, eliminating the ingress bottleneck for high‑throughput GPU services.

Validation

After applying the fixes, perform the following checks:

Ingress connectivity

curl -s -o /dev/null -w "%{http_code}\n" http://my-swarm-ingress:8080
# Expected output: 200

Overlay health

docker network inspect ingress -f '{{json .Options}}' | jq .
# Verify MTU matches NIC (e.g., 9000)

Firewall status

iptables -L -n | grep -E '7946|4789'
# Should show ACCEPT rules for UDP ports

Manager CPU load

top -b -n1 | grep dockerd
# CPU should be < 30 % under normal load

GPU allocation

docker service ps gpu-api --no-trunc --format "{{.Node}} {{.DesiredState}} {{.CurrentState}} {{.Error}}"
# No “cannot allocate resources” errors

Prevention and Best Practices

Standardize NIC MTU across all Swarm nodes and enforce the same value on overlay networks via --opt com.docker.network.driver.mtu.
Maintain a baseline firewall policy that explicitly allows UDP 7946 and 4789 on every node, including GPU workers.
Prefer host mode publishing for GPU‑intensive services that require low latency and high throughput; reserve the ingress mesh for control‑plane or low‑traffic services.
Monitor docker_swarm_ingress_router_cpu_seconds_total (or equivalent) and set alerts when CPU usage exceeds 70 % for more than 5 minutes.
Automate DNS verification after node join/leave events using a health‑check script that queries the Swarm internal DNS (e.g., dig +short tasks.gpu-api).
Pin NVIDIA driver versions that are known to be compatible with your Docker Engine release; test driver upgrades in a staging Swarm before production rollout.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the ingress network work on CPU‑only nodes but fail on GPU nodes?
GPU nodes often run a different NIC configuration (jumbo frames) and may have stricter firewall rules. The mismatch in MTU and blocked VXLAN traffic prevents the routing mesh from establishing overlay tunnels.
Can I keep using mode=ingress with GPU services?
Yes, if you align the overlay MTU with the NIC, open the required UDP ports, and ensure the manager has sufficient CPU capacity. However, for consistently high‑throughput workloads, host mode is more reliable.

What command shows the exact MTU used by the overlay network?

docker network inspect ingress -f '{{json .Options}}' | jq '."com.docker.network.driver.mtu"'

After a driver upgrade, why do I see “Failed to create endpoint: network ingress: driver failed programming the network: MTU mismatch”?
The driver upgrade may have changed the NIC’s default MTU (e.g., enabling jumbo frames). The overlay network still uses the old 1500‑byte MTU, causing the daemon to reject endpoint creation.
How do I verify that UDP ports 7946 and 4789 are reachable between nodes?
Use nc -zu <node_ip> 7946 and nc -zu <node_ip> 4789 from each node. A successful exit status (0) indicates the ports are open.