PyTorch Training Webhook Timeout On A100 Cluster After Job Completion

Q: 1. Verify that the endpoint is reachable from a running training process

# Inside the training container, before the job finishes curl -v -X POST https://monitor.example.com/notify \ -H "Content-Type: application/json" \ -d '{"status":"alive"}' --max-time 5 Expected output (successful):

Q: 2. Capture the exact moment of process exit

# Run the job with strace to see when the socket is closed torchrun --nnodes=4 --nproc_per_node=8 \ --rdzv_id=job123 \ --rdzv_backend=c10d \ my_train.py \ --on_finish=my_callbacks.on_finish \ 2>&1 | tee run.log # In another terminal, monitor the socket state ss -tnp | grep $(pgrep -f my_train.py) Look for a FIN_WAIT or CLOSE_WAIT transition occurring before the webhook request is issued.

Q: 3. Simulate delayed webhook execution

# Modify the callback to sleep for 20 s before sending def on_finish(state): import time, requests time.sleep(20) # artificial delay requests.post( "https://monitor.example.com/notify", json={"job_id": state.job_id, "status": "completed"}, timeout=30, ) Run the job and observe that the timeout error appears earlier (torchrun kills the process after ~30 s).

Q: 4. Check firewall rule propagation timestamps

# On the GPU node, after job exit journalctl -u firewalld -n 50 | grep monitor.example.com If a rule disabling outbound 443 appears within a few seconds of the job exit, it confirms the firewall race.

Q: Option A: Run the webhook in a separate sidecar container

Deploy a lightweight sidecar (e.g., alpine:latest) that listens on a Unix domain socket or a local TCP port. The training script writes a completion file or sends a local HTTP request to the sidecar, which then performs the external POST.

Q: Option C: Extend the TorchElastic watchdog timeout

If decoupling is not feasible, increase the internal timeout using the --max_restarts and --monitor_interval flags, or set the environment variable TORCHELASTIC_MAX_SHUTDOWN_WAIT_S=120. This gives the on_finish hook more time before the process is killed.

In this article

Table of Contents

Problem – Webhook Timeout After PyTorch Distributed Job Completes on A100/H100 Nodes

Observed behavior

Training jobs launched with torchrun (TorchElastic) finish successfully on a multi‑node A100 cluster.
Immediately after the last epoch, the user‑defined on_finish callback attempts to POST a JSON payload to https://monitor.example.com/notify.
The job process exits with a non‑zero status and logs contain:

2024-09-15 12:34:56,789 ERROR torch.distributed.elastic.agent.server: 
Error in on_finish callback: urllib3.exceptions.MaxRetryError: 
Failed to reach host monitor.example.com after 2 retries: timed out
Traceback (most recent call last):
  File ".../torchrun.py", line 312, in main
    user_callback()
  File ".../my_callbacks.py", line 42, in on_finish
    send_completion_webhook()
  File ".../my_callbacks.py", line 28, in send_completion_webhook
    response = requests.post(url, json=payload, timeout=30)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='monitor.example.com', port=443): Read timed out. (timeout=30)
torchrun: error: Job terminated but user‑defined on_finish hook raised TimeoutError: webhook did not complete within allotted time

Operational impact

External monitoring dashboards never receive the “job completed” event, causing false‑negative alerts.
Automated downstream pipelines that depend on the webhook (e.g., model registration) stall.
Repeated timeout errors fill the job logs, making post‑mortem analysis harder.

Root Cause Analysis

The timeout is not caused by the remote endpoint itself; it is a race between the on_finish callback and the termination of the training process on the GPU node. Several evidence items converge on the same mechanism:

Network interface teardown – In a multi‑node A100 cluster, the node’s network interface is torn down as soon as the training process exits, aborting any in‑flight TCP connections (real incident).
Cgroup kill ordering – H100 containers have been observed to kill the cgroup immediately after the script exits, truncating pending outbound requests (real incident).
TorchElastic shutdown sequence – According to the TorchElastic API reference, the on_finish hook runs in the same process that will soon call sys.exit(). If the hook blocks longer than the internal watchdog (default 30 s), torchrun aborts the process and propagates a TimeoutError (official documentation).
Firewall rule propagation delay – GPU‑node firewall rules are refreshed after job termination, temporarily blocking outbound HTTPS traffic (real incident).

Combined, these factors mean that the HTTP POST often starts after the network stack has been partially disabled, leading to the ReadTimeout errors reported in the logs.

Investigation and Debugging Steps

Below is a reproducible debugging workflow that isolates the failure point.

1. Verify that the endpoint is reachable from a running training process

# Inside the training container, before the job finishes
curl -v -X POST https://monitor.example.com/notify \
     -H "Content-Type: application/json" \
     -d '{"status":"alive"}' --max-time 5

Expected output (successful):

*   Trying 203.0.113.10:443...
* Connected to monitor.example.com (203.0.113.10) port 443 (#0)
> POST /notify HTTP/1.1
> Host: monitor.example.com
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 20
>
* upload completely sent off: 20 out of 20 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 15
...

If this succeeds, the network path is healthy while the process is alive.

2. Capture the exact moment of process exit

# Run the job with strace to see when the socket is closed
torchrun --nnodes=4 --nproc_per_node=8 \
    --rdzv_id=job123 \
    --rdzv_backend=c10d \
    my_train.py \
    --on_finish=my_callbacks.on_finish \
    2>&1 | tee run.log

# In another terminal, monitor the socket state
ss -tnp | grep $(pgrep -f my_train.py)

Look for a FIN_WAIT or CLOSE_WAIT transition occurring before the webhook request is issued.

3. Simulate delayed webhook execution

# Modify the callback to sleep for 20 s before sending
def on_finish(state):
    import time, requests
    time.sleep(20)  # artificial delay
    requests.post(
        "https://monitor.example.com/notify",
        json={"job_id": state.job_id, "status": "completed"},
        timeout=30,
    )

Run the job and observe that the timeout error appears earlier (torchrun kills the process after ~30 s).

4. Check firewall rule propagation timestamps

# On the GPU node, after job exit
journalctl -u firewalld -n 50 | grep monitor.example.com

If a rule disabling outbound 443 appears within a few seconds of the job exit, it confirms the firewall race.

Resolution – Making the Webhook Reliable

The fix consists of decoupling the webhook from the training process lifecycle and ensuring the network stack remains available long enough for the request to complete.

Option A: Run the webhook in a separate sidecar container

Deploy a lightweight sidecar (e.g., alpine:latest) that listens on a Unix domain socket or a local TCP port. The training script writes a completion file or sends a local HTTP request to the sidecar, which then performs the external POST.

# docker-compose snippet
services:
  trainer:
    image: my/pytorch:latest
    command: ["torchrun", "--nnodes=4", "--nproc_per_node=8", "my_train.py"]
    depends_on:
      - notifier
  notifier:
    image: alpine:latest
    command: ["sh", "-c", "while true; do nc -l -U /tmp/notify.sock | xargs -I{} curl -X POST https://monitor.example.com/notify -d '{}' -H 'Content-Type: application/json'; done"]
    volumes:
      - /tmp:/tmp

Because the notifier runs in its own cgroup, it is not killed when the trainer exits, eliminating the race.

Option B: Use TorchElastic's `post_finish` hook with a detached thread

Modify the callback to spawn a daemon thread that performs the POST and then returns immediately.

# my_callbacks.py
import threading, requests, json, os

def _post_webhook(payload):
    try:
        resp = requests.post(
            "https://monitor.example.com/notify",
            json=payload,
            timeout=30,
        )
        resp.raise_for_status()
    except Exception as e:
        # Log to a file that survives process exit
        with open("/tmp/webhook_error.log", "a") as f:
            f.write(f"{os.getpid()}: {e}\\n")

def on_finish(state):
    payload = {"job_id": state.job_id, "status": "completed"}
    t = threading.Thread(target=_post_webhook, args=(payload,))
    t.daemon = True
    t.start()
    # Return immediately so torchrun can finish cleanly

Because the thread is daemonized, the Python interpreter does not wait for it on exit, but the OS keeps the network stack alive until the thread finishes (typically < 5 s).

Option C: Extend the TorchElastic watchdog timeout

If decoupling is not feasible, increase the internal timeout using the --max_restarts and --monitor_interval flags, or set the environment variable TORCHELASTIC_MAX_SHUTDOWN_WAIT_S=120. This gives the on_finish hook more time before the process is killed.

# Example launch
export TORCHELASTIC_MAX_SHUTDOWN_WAIT_S=120
torchrun --nnodes=4 --nproc_per_node=8 \
    --rdzv_id=job123 \
    --rdzv_backend=c10d \
    my_train.py \
    --on_finish=my_callbacks.on_finish

Before / After Comparison

Aspect	Before (inline webhook)	After (sidecar or detached thread)
Process exit order	Trainer exits → network stack torn down → webhook fails	Trainer exits → sidecar/thread continues → network stack remains
Failure mode	ReadTimeout, job marked failed	Successful POST, job marked succeeded
Code complexity	Single callback, but fragile	Additional sidecar or threading logic
Observability	Only trainer logs	Separate notifier logs (/tmp/webhook_error.log)

Verification – Confirming the Fix Works

Re‑run the training job with the new callback implementation.
Observe that the trainer exits cleanly without a TimeoutError in run.log.
Check the external monitoring service for the “job completed” event within 5 seconds of job termination.
Inspect the sidecar or thread log file for any residual errors:

# Sidecar log tail
tail -f /var/log/notifier.log
# Or thread error file
cat /tmp/webhook_error.log

Successful verification looks like:

2024-09-15 12:35:01,123 INFO monitor.example.com: Received job_id=job123 status=completed

Prevention – Operational Guardrails

Health‑check the webhook endpoint before job launch; abort early if unreachable.
Configure a dedicated outbound allowlist for the GPU nodes that does not get revoked on job termination (adjust firewall automation scripts).
Instrument a watchdog metric (e.g., Prometheus counter training_webhook_success_total) and set alerts on sudden drops.
Document the shutdown order in the cluster SOP: sidecar containers must be defined as depends_on the trainer to ensure they stay alive.
Test the callback in a CI pipeline that mimics node teardown (use docker stop on the trainer container while a mock server is listening).

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the webhook succeed when I run the script locally but fails on the GPU node?
Because the local environment does not terminate the network interface or cgroup immediately after the script exits, whereas the GPU node’s job manager aggressively tears down resources.
Can I increase the requests.post timeout to avoid the error?
Increasing the client timeout alone does not help; the underlying socket is closed by the node before the request can be sent.
Is the TORCHELASTIC_MAX_SHUTDOWN_WAIT_S variable safe to set globally?
It is safe but may delay resource reclamation on a shared cluster. Prefer per‑job overrides or sidecar solutions.
Do I need to modify the Slurm job script?
If you use Slurm, add a --kill-on-bad-exit=0 flag and ensure the notifier runs as a separate srun step that is not killed with the main task.
What if the external monitoring service uses a self‑signed certificate?
Configure requests.post(..., verify="/path/to/ca.pem") in the callback, and ensure the sidecar container has the CA bundle installed.