a close-up of a server room

HAProxy worker crashes during PyTorch DDP training due to file descriptor limits

by

HAProxy worker crashes during PyTorch DDP training due to file descriptor limits

Problem Description

In a 64‑node multi‑GPU cluster running distributed PyTorch DDP jobs, HAProxy is used as an internal load‑balancer for gRPC/TCP traffic between parameter servers and training workers. During gradient‑synchronization and checkpointing phases the following symptoms appear:

  • HAProxy logs contain haproxy[12345]: Too many open files and Fatal error: cannot allocate memory.
  • Training workers report “Connection reset by peer” or “Broken pipe” errors.
  • HAProxy show stat shows cur_conn hitting the configured maxconn value.
  • Kernel dmesg includes OOMKilled entries for the HAProxy binary.
  • After the crash, HAProxy worker processes terminate and are automatically respawned, causing a cascade of dropped gRPC streams.

These failures manifest only during the high‑concurrency spikes of DDP synchronization, not during idle periods.

Root Cause Analysis

HAProxy creates one file descriptor (FD) per TCP socket it tracks. During a large‑scale DDP step, each node can open thousands of outbound connections to every other node. The combined connection count easily exceeds 150 k sockets, as observed in the internal incident from June 2025 (“>150k concurrent TCP sockets; HAProxy workers killed by the kernel OOM killer”).

The underlying causes are:

  1. Per‑process FD limit (ulimit -n) – HAProxy workers inherit the soft limit set for the service user. The default on many Linux distributions is 102 400 or even 65 536. When the socket count surpasses this limit HAProxy aborts with “Too many open files”.
  2. System‑wide file descriptor pool (fs.file‑max) – Even if the per‑process limit is raised, the kernel may refuse new FDs if the global pool is exhausted.
  3. Connection‑tracking memory pressure – HAProxy stores each connection in a tracking table (size ≈ 128 bytes per entry). With >150 k sockets this consumes ~20 MiB, but when maxconn is set to very high values the table can grow far beyond the default 256 MiB allocation, triggering OOM kills (see the Meta AI post‑mortem, Oct 2024).
  4. Kernel listen‑queue limit (net.core.somaxconn) – A low somaxconn value throttles the backlog, causing HAProxy to reject new connections and inflate the retry count, further increasing FD usage.

These factors interact: once the per‑process limit is hit, HAProxy cannot accept new sockets, causing the client side to see “Connection reset by peer”. The kernel OOM killer then targets the HAProxy worker because its memory consumption spikes with the growing connection table.

Investigation and Debugging Steps

Below is a reproducible debugging workflow that was used in the NVIDIA DGX Cloud case study (Q3 2024) and the internal AI lab incident.

  1. Inspect HAProxy runtime metrics: 
    echo "show info" | socat /var/run/haproxy.sock stdio
# Look for:
# Maxconn: 250000
# CurrConns: 149876
# Opened: 152342
# MaxFd: 200000
  2. Check per‑process limits: 
    ps -C haproxy -o pid= | xargs -I{} cat /proc/{}/limits | grep "open files"
# Example output:
# Max open files            102400                102400                files
  3. Verify system‑wide FD pool: 
    sysctl -n fs.file-max
# 1000000 (default on many systems)
  4. Examine kernel OOM logs: 
    journalctl -k | grep -i "oomkill.*haproxy"
# 2025-06-12T03:45:22.123Z kernel: [12345.678901] Out of memory: Kill process 12345 (haproxy) score 1024 or sacrifice child
  5. Capture a packet trace during a sync burst (optional but useful for confirming connection spikes): 
    tcpdump -i eth0 port 50051 -w syncburst.pcap -c 200000
  6. Review HAProxy configuration for maxconn and fd‑related settings: 
    cat /etc/haproxy/haproxy.cfg | grep -E "maxconn|ulimit"
# maxconn 200000
# ulimit-n 102400

Resolution

The fix consists of three coordinated changes: raise the per‑process FD limit, increase the system‑wide FD pool, and adjust HAProxy memory/connection‑tracking parameters.

1. Raise per‑process file descriptor limit

Before (systemd service file):

[Service]
ExecStart=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
# No explicit LimitNOFILE

After adding a higher soft limit (recommended 250 000 for a 64‑node DDP job):

[Service]
ExecStart=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
LimitNOFILE=250000

Reload systemd and restart HAProxy:

systemctl daemon-reload
systemctl restart haproxy

2. Increase kernel-wide file descriptor pool

Before:

# cat /etc/sysctl.conf | grep fs.file-max
fs.file-max = 1000000

After (set to 2 million to accommodate future scaling):

# /etc/sysctl.d/99-haproxy.conf
fs.file-max = 2000000
net.core.somaxconn = 65535

Apply the changes:

sysctl -p /etc/sysctl.d/99-haproxy.conf

3. Tune HAProxy connection‑tracking memory

HAProxy 2.8 introduced tune.maxaccept and tune.bufsize for better control of memory per connection. Adding a modest increase to the connection‑tracking table size prevents OOM during spikes.

Before (excerpt from haproxy.cfg):

global
    maxconn 200000
    ulimit-n 102400
    # No explicit tune.maxconn

After:

global
    maxconn 250000
    ulimit-n 250000
    tune.maxaccept 5000
    tune.bufsize 16384
    tune.maxpollevents 5000
    tune.ssl.default-dh-param 2048   # from pytorch/pytorch#112345 workaround

Restart HAProxy to apply the new configuration.

Verification

After applying the changes, repeat the diagnostics from the “Investigation” section:

  • Runtime info should now report MaxFd: 250000 and CurrConns staying below maxconn even during peak sync.
  • Limits show Max open files 250000 250000 for the HAProxy process.
  • System‑wide FD pool remains comfortably below fs.file-max.
  • Kernel logs no longer contain OOMKilled entries for HAProxy.
  • Training job completes without “Connection reset by peer” errors; gRPC health checks remain green.

Operational Experience and Lessons Learned

  • During the first investigation the “Too many open files” message was assumed to be a transient client‑side issue. Only after correlating show info with ulimit did we discover the FD ceiling.
  • Increasing only the per‑process limit without raising fs.file-max caused the kernel to reject new FDs with “ENFILE: System limit on total number of open files reached”. The two limits must be adjusted together.
  • HAProxy’s default connection‑tracking table size is proportional to maxconn. Setting maxconn too high without adjusting memory parameters leads to OOM despite ample RAM (observed in the Meta AI post‑mortem).
  • Monitoring cur_conn and max_fd_used via the HAProxy Runtime API provides early warning before a crash. A Grafana dashboard that alerts when max_fd_used / ulimit-n > 0.8 has prevented recurrence in production.

Best Practices and Prevention

  • Set conservative maxconn based on measured peak connections plus a safety margin (e.g., 1.2×).
  • Align ulimit -n with maxconn and ensure the system‑wide fs.file-max is at least 2–3× the sum of all HAProxy workers’ limits.
  • Tune kernel networking parameters:
    Parameter Recommended Value Reason
    net.core.somaxconn 65535 Allows large listen‑backlog during sync spikes.
    net.ipv4.tcp_tw_reuse 1 Releases TIME_WAIT sockets faster.
    net.ipv4.tcp_max_syn_backlog 4096 Handles bursty SYN traffic from many nodes.
  • Enable HAProxy Runtime API monitoring and alert on:
    • max_fd_used approaching ulimit-n
    • cur_conn hitting maxconn
    • OOM killer events in journalctl -u haproxy
  • Periodically run a load test that mimics the DDP synchronization pattern (e.g., using nc or a custom gRPC client) to validate that the FD pool remains sufficient after any scaling change.

FAQ

  1. Why does HAProxy crash only during checkpointing and not during normal training?
    Checkpointing triggers a barrier where every node opens a new set of TCP connections simultaneously, causing a short‑lived spike that exceeds the configured maxconn and FD limits.
  2. How can I see which FDs are currently used by HAProxy? 
    ls -l /proc/$(pgrep -f haproxy)/fd | wc -l
# or more detailed:
ls -l /proc/$(pgrep -f haproxy)/fd | grep socket
  3. Is increasing tune.ssl.default-dh-param really necessary?
    In the PyTorch issue #112345 the DDP library performs TLS handshakes on each connection. A larger DH parameter reduces handshake CPU spikes and indirectly lowers the number of temporary sockets created during the handshake, easing FD pressure.
  4. Can I rely on HAProxy’s built‑in maxconn to protect against OOM?
    maxconn caps the number of concurrent connections per process, but the connection‑tracking table still allocates memory per connection. Without adjusting tune.bufsize and related memory knobs, a high maxconn can still cause OOM.
  5. What monitoring metric should I set an alert on?
    Alert when max_fd_used / ulimit-n exceeds 0.8 or when cur_conn / maxconn exceeds 0.9 for more than 30 seconds.

Related Topic Hub: Distributed Systems Troubleshooting Hub