TensorRT training job completion timeout with mixed precision

Problem – TensorRT Mixed‑Precision Training Jobs Time Out

In a distributed training pipeline that leverages TensorRT engine building with FP16 (or INT8‑fallback) on NVIDIA A100 GPUs, the job aborts before completing the expected number of epochs. Typical symptoms include:

• Training script exits with TensorRTBuilderError: Engine building timed out after 1800 seconds.
• CUDA watchdog messages in syslog or dmesg: Watchdog timer: kernel execution timed out (code 719).
• Intermittent NCCL errors such as NCCL error: Internal error (code 2) during peer‑to‑peer initialization.
• GPU utilization drops to near‑zero after the 5th epoch on a 8‑GPU node.

The failure manifests during the engine build phase or the first few training iterations when mixed‑precision kernels are launched, leading to orchestration‑level timeouts (e.g., Kubernetes pod termination).

Root Cause Analysis

Interaction of TensorRT Builder, Mixed Precision, and Distributed Runtime

TensorRT builds an execution engine by compiling the ONNX graph into a series of CUDA kernels. When FP16 is enabled, the builder must:

1. Validate that each layer supports FP16 (per the Mixed Precision Guide).
2. Allocate a workspace whose size is bounded by maxWorkspaceSize. In a multi‑GPU scenario, each process replicates this allocation.
3. Synchronize NCCL communication channels for gradient reduction (Distributed Training Documentation).

Two conditions commonly trigger the observed timeout:

• Kernel watchdog expiration: Large dynamic‑shape tensors or excessive workspace cause a kernel launch that exceeds the OS watchdog limit (default 2 seconds on Linux). This produces CUDA error: unknown error (code 719) and aborts the builder.
• NCCL peer‑to‑peer timeout: When the builder attempts to allocate a workspace larger than the per‑GPU memory budget, NCCL cannot establish all peer connections within its default 30‑second timeout, resulting in NCCL error: Internal error (code 2).

Both symptoms were reported in the community (GitHub #2549, Stack Overflow) and match the real incident where a large‑language‑model fine‑tuning job hit a CUDA watchdog after the 5th epoch.

Debug – Investigation Steps

1. Capture Builder Logs

export TRT_LOGGING=1
python train.py 2>&1 | tee build.log

Typical excerpt:

[2026-06-20 12:34:56] [I] TensorRT: Builder timed out after 1800 seconds.
[2026-06-20 12:34:56] [E] CUDA error: unknown error (code 719) – kernel watchdog timeout.

2. Inspect GPU Memory and Workspace Allocation

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv

Look for spikes that approach the total memory (e.g., 80 GB used on a 80 GB A100).

3. Profile Kernel Execution Times

nsight-cu-cli --kernel-name-filter=*fp16* --launch-skip 0 -o profile.ncu

Identify kernels that exceed the watchdog limit.

4. Verify NCCL Communication

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,ENV
python -m torch.distributed.launch --nproc_per_node=8 train.py

Search the log for lines such as:

[NCCL] Init failed: Internal error (code 2)

5. Check TensorRT Builder Configuration

Review the code that creates the builder and network:

import tensorrt as trt

builder = trt.Builder(TRT_LOGGER)
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16)
config.max_workspace_size = 1 << 30  # 1 GiB (default)

Solution – Fixes and Workarounds

1. Increase Builder Workspace and Disable Watchdog for Long Kernels

Set a larger workspace (e.g., 8 GiB) and disable the CUDA watchdog by launching the process with CUDA_LAUNCH_BLOCKING=1 and nvidia-smi -i 0 -c EXCLUSIVE_PROCESS or by increasing the OS watchdog limit via sysctl (requires root).

# Before
config.max_workspace_size = 1 << 30  # 1 GiB

# After
config.max_workspace_size = 8 << 30  # 8 GiB

For the watchdog, add:

export CUDA_WATCHDOG_TIMEOUT=10   # seconds, increase from default 2 s

2. Reduce Dynamic‑Shape Over‑Allocation

Explicitly set shape profiles that bound the maximum tensor dimensions. This prevents the builder from allocating oversized temporary buffers.

profile = builder.create_optimization_profile()
profile.set_shape("input", (1, 3, 224, 224), (4, 3, 224, 224), (8, 3, 224, 224))
config.add_optimization_profile(profile)

3. Tune NCCL Timeouts and Enable Peer Access

Increase NCCL's internal timeout and force peer‑to‑peer communication.

export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_TIMEOUT=30   # seconds
export NCCL_P2P_LEVEL=NVL   # enforce NVLink if available

4. Stagger Engine Builds Across GPUs

When using torch.distributed, build the TensorRT engine only on rank 0 and broadcast the serialized engine to other ranks. This eliminates simultaneous large memory allocations.

# Rank 0
engine_bytes = builder.build_serialized_network(network, config)
torch.distributed.broadcast(torch.tensor([len(engine_bytes)], dtype=torch.long), src=0)
torch.distributed.broadcast(torch.from_numpy(np.frombuffer(engine_bytes, dtype=np.uint8)), src=0)

# Ranks 1‑N
engine_len = torch.empty(1, dtype=torch.long)
torch.distributed.broadcast(engine_len, src=0)
engine_buf = torch.empty(engine_len.item(), dtype=torch.uint8)
torch.distributed.broadcast(engine_buf, src=0)
engine = trt.Runtime(TRT_LOGGER).deserialize_cuda_engine(engine_buf.numpy())

5. Upgrade Drivers and TensorRT

Versions prior to TensorRT 8.5 have a known bug where FP16 kernels on A100 can exceed the watchdog limit under multi‑process launch (NVIDIA Forum thread). Updating to TensorRT 8.6+ and CUDA 12.1 resolves the kernel‑launch latency issue.

Verification – Confirming the Fix

Functional Test

python train.py --epochs 10 --batch-size 64

Expected outcome:

• No TensorRTBuilderError or watchdog messages in build.log.
• GPU memory usage stabilizes below 70 % throughout training.
• NCCL logs show successful peer initialization without timeout.

Metrics Validation

Collect the following metrics before and after the change:

Metric	Before	After
Engine build time (s)	≈ 1800 ( timeout )	≈ 350
Kernel max runtime (s)	2.8 ( watchdog )	0.9
NCCL init latency (ms)	31000 ( timeout )	1200
Peak GPU memory (GiB)	79 / 80	62 / 80

Prevention – Best Practices

• Explicit Shape Profiles: Always bound dynamic dimensions with set_shape to keep workspace predictable.
• Workspace Sizing: Allocate at least 4 GiB for FP16 workloads on A100; adjust based on model size.
• Engine Build Serialization: Centralize engine creation to a single process in distributed runs.
• Watchdog Configuration: For long kernels, increase CUDA_WATCHDOG_TIMEOUT or run under a non‑X server environment where the watchdog is disabled.
• Version Pinning: Use TensorRT 8.6+, CUDA 12.1+, and driver ≥ 525.89.07 to benefit from mixed‑precision stability fixes.
• Monitoring: Alert on TensorRTBuilderError and NCCL timeout strings; surface GPU memory pressure via Prometheus node exporter.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the timeout only appear after a few epochs?
   Because the first few epochs allocate a smaller temporary buffer; as the model sees larger input shapes (e.g., longer sequences), the workspace grows and eventually exceeds the watchdog limit.
2. Can I disable the TensorRT builder timeout entirely?
   The builder exposes set_builder_termination_timeout (default 1800 s). Setting it to a larger value can avoid premature aborts, but underlying kernel or NCCL timeouts must still be addressed.
3. Is INT8 calibration safe with FP16 fallback in distributed training?
   Yes, but ensure the calibration dataset fits within the per‑GPU memory budget and that the builder’s INT8 flag is paired with FP16 fallback. Mismatched calibration tensors often cause the “CUDA error: unknown error (code 719)”.
4. How do I know if the watchdog is the culprit?
   Check dmesg or /var/log/kern.log for messages containing “Watchdog timer”. A matching timestamp with the engine‑build log confirms the correlation.
5. What NCCL settings are recommended for TensorRT‑accelerated training?
   Set NCCL_IB_TIMEOUT ≥ 30 s, enable NCCL_P2P_LEVEL=NVL on DGX‑A100, and export NCCL_DEBUG=INFO to capture detailed peer‑setup logs.