Redis node startup error: port conflict during distributed ML training
Problem
During the launch of a distributed machine‑learning training job, each worker process starts a local Redis instance for coordination (parameter server, barrier sync, etc.). The training job aborts with errors similar to:
2026-11-17 09:42:13.721 # Fatal error, can't open server TCP socket *:6379: bind: Address already in use
2026-11-17 09:42:13.721 # Unable to start Redis server
2026-11-17 09:42:13.721 # Exiting...
Subsequent logs from the training framework show connection timeouts:
[Worker-3] ERROR: Failed to connect to Redis at 127.0.0.1:6379 - Connection refused
[Orchestrator] WARN: One or more workers could not join the barrier; aborting training.
The symptom is reproducible only when multiple workers are launched on the same host, indicating a port conflict between Redis nodes.
Root Cause
The training orchestration script statically configures each Redis node to listen on the default port 6379. When the job scales to >1 worker per host, each worker attempts to bind the same TCP port, causing the operating system to reject the second bind request (bind: Address already in use). The conflict is hidden in environments where a single worker per host is the norm, so the issue surfaces only under multi‑worker per‑node deployments.
Additional contributing factors:
- Container‑orchestrated environments (Docker, Kubernetes) often expose the host network namespace to containers, making the port namespace shared across all containers on the node.
- Some launch scripts reuse the same Redis configuration file without templating the
portdirective. - Systemd‑managed Redis instances may already be listening on 6379, colliding with the training‑time instances.
Debug
Follow these steps to confirm the conflict and locate the offending processes.
1. Identify which process holds the port
sudo lsof -iTCP:6379 -sTCP:LISTEN
# or
sudo netstat -tulpn | grep :6379
Typical output when a conflict exists:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 1123 redis 6u IPv4 12345 0t0 TCP *:6379 (LISTEN)
python 2456 root 3u IPv4 67890 0t0 TCP 127.0.0.1:6379 (LISTEN)
2. Verify the training script’s Redis configuration
# redis.conf used by the training job (before fix)
port 6379
bind 127.0.0.1
3. Check the orchestration logs for duplicate launches
2026-11-17 09:41:58,123 [launcher] INFO Starting Redis node for worker 0 on host node-01
2026-11-17 09:41:58,124 [launcher] INFO Starting Redis node for worker 1 on host node-01
4. Confirm that the host network namespace is shared
docker inspect --format='{{.HostConfig.NetworkMode}}' worker_container_0
# Output: "host"
Solution
Allocate a unique, free port per Redis instance at runtime and inject it into the worker’s configuration.
1. Modify the launch script to compute an available port
#!/usr/bin/env bash
# discover a free port in the 6000‑7000 range
BASE=6000
MAX=7000
for ((port=BASE; port<=MAX; port++)); do
if ! ss -ltn "sport = :$port" > /dev/null; then
export REDIS_PORT=$port
break
fi
done
if [[ -z $REDIS_PORT ]]; then
echo "No free port found in $BASE-$MAX"
exit 1
fi
# generate a per‑worker redis.conf
cat > redis_${WORKER_ID}.conf <<EOF
port $REDIS_PORT
bind 127.0.0.1
protected-mode yes
daemonize no
logfile ""
EOF
# start redis with the generated config
redis-server redis_${WORKER_ID}.conf &
2. Pass the selected port to the training framework
# Python snippet (example)
import os
redis_port = os.getenv('REDIS_PORT')
client = redis.StrictRedis(host='127.0.0.1', port=int(redis_port))
3. Update any static configuration files (if used)
Before:
# static_config.yaml
redis:
host: 127.0.0.1
port: 6379
After (templated with environment variable):
# static_config.yaml (templated)
redis:
host: 127.0.0.1
port: ${REDIS_PORT}
4. Ensure systemd‑managed Redis does not interfere
If a system Redis service runs on the same node, either stop it for the duration of the training job or configure it on a non‑conflicting port.
sudo systemctl stop redis
# or edit /etc/redis/redis.conf
port 6380
sudo systemctl restart redis
Verify
After applying the changes, confirm that each worker’s Redis instance starts without errors and that the training job proceeds.
1. Check Redis process list
ps -ef | grep redis-server
# Expected: multiple entries, each with a distinct --port argument
2. Verify port bindings
ss -ltnp | grep redis
# Sample output
LISTEN 0 128 127.0.0.1:6001 *:* users:(("redis-server",pid=3124,fd=6))
LISTEN 0 128 127.0.0.1:6002 *:* users:(("redis-server",pid=3125,fd=6))
3. Run a quick connectivity test from each worker
redis-cli -p 6001 ping
# Expected output: PONG
4. Observe training logs for successful barrier synchronization
[Orchestrator] INFO All workers joined barrier after 12.3s
[Worker-2] INFO Training step 1 completed
Prevent
- Dynamic port allocation: Always compute a free port at container start rather than hard‑coding 6379.
- Namespace isolation: In Docker/Kubernetes, avoid
networkMode: hostunless required. Use bridge or overlay networks so each container gets its own network namespace. - Port reservation service: For large clusters, consider a central service (e.g., etcd) that hands out unique ports to workers.
- Health‑check alerts: Add a Prometheus metric that tracks
redis_up{port="*"} == 0and fire an alert if any expected port is not listening. - Configuration linting: CI step that parses Redis config templates and ensures
portis not a constant value.
Related Topic Hub: Data Infrastructure Troubleshooting Hub
FAQ
- Why does the error only appear when scaling to multiple workers on the same node?
Because each worker tries to bind the same static port (6379). With a single worker per node the port is free, but additional workers collide.
- Can I keep using the default port 6379?
Only if you guarantee a single Redis instance per host (e.g., by using separate network namespaces). Otherwise allocate a unique port per instance.
- What if my environment already runs a production Redis on 6379?
Do not start a second Redis on the same host. Either stop the production service during training or configure the training Redis to use a non‑conflicting port range (e.g., 6000‑7000).
- How do I debug a situation where
ssshows the port is free but Redis still fails to bind?Check SELinux/AppArmor policies, container capabilities (e.g.,
CAP_NET_BIND_SERVICE), and ensure the process is not running as a non‑privileged user that is restricted from binding low‑numbered ports. - Is there a risk of exhausting the port range when many workers start simultaneously?
Yes. Reserve a sufficiently large range (e.g., 6000‑65000) and implement a retry loop that picks the next free port if the first choice is taken.