AMD GPU Controller Manager Crash Inside Docker During AI Training

Q: Step 2 – Bind‑mount required devices

Run containers with explicit device flags or use --privileged only as a temporary test.

In this article

Table of Contents

Problem Description

During long‑running AI training jobs inside a Docker container, the AMD GPU controller manager (amdgpu) crashes repeatedly. The symptom manifests as the container exiting with code 137 (SIGKILL) and loss of GPU access. Typical log excerpts are:

dmesg: amdgpu: controller manager crashed, resetting GPU
Docker log: error while creating device /dev/kfd: No such file or directory
Kernel log: amdgpu: GPU reset due to X (GPU hang) – controller manager terminated
Container stdout: RuntimeError: ROCm driver failed to initialize – controller manager not responding
System journal: amdgpu: failed to start controller manager: segmentation fault (core dumped)

These crashes occur after several hours of training (≈4 h in the documented incident) or when multiple containers share the same GPU on a single host.

Root Cause Analysis

The controller manager is a user‑space daemon launched by the amdgpu kernel driver to handle GPU command submission, memory management, and watchdog timers. The following factors combine to trigger the crash inside Docker:

Driver‑runtime version mismatch: Host runs amdgpu from ROCm 6.0, while the container image bundles an older ROCm library set (e.g., 5.6). The mismatched ABI leads to segmentation faults when the container‑side libdrm attempts to communicate with the host driver.
Missing /dev/kfd and /dev/dri devices: The ROCm Docker integration requires these character devices to be bind‑mounted. If they are absent (common when the container is started without --device /dev/kfd --device /dev/dri or without --privileged), the controller manager cannot initialize and later crashes when the watchdog fires.
Watchdog timeout under heavy load: The controller manager enforces a 5‑second watchdog on the GPU command processor. Long‑running kernels that exceed this window (e.g., large TensorFlow kernels) cause the watchdog to trigger a reset, which the mismatched runtime cannot recover from, resulting in a segfault.
Kubernetes device plugin version: Older rocm-device-plugin binaries do not propagate the ROCM_DISABLE_GPU_RESET flag, so the plugin allows resets that the container cannot survive.

These root causes are documented in the official ROCm Installation Guide (kernel driver requirements), the ROCm Docker Integration Guide (device binding), and ROCm 6.0 release notes (known containerized workload issues).

Investigation and Debugging Steps

Confirm host driver version and kernel module parameters:

# host
$ cat /sys/module/amdgpu/version
6.0.0
$ dmesg | grep amdgpu | tail -n 5
[ 12345.678901] amdgpu: controller manager watchdog timeout
[ 12345.679012] amdgpu: GPU reset due to X (GPU hang) – controller manager terminated

Inspect container ROCm version:
```
# inside container
$ rocm-smi --showversion
ROCm Version: 5.6.0
```
If the versions differ, note the mismatch.

Verify device visibility:

# host
$ ls -l /dev/kfd /dev/dri
crw-rw---- 1 root video 250, 0 Aug 27 12:00 /dev/kfd
crw-rw---- 1 root video 226, 0 Aug 27 12:00 /dev/dri/card0

# container
$ ls -l /dev/kfd /dev/dri
ls: cannot access '/dev/kfd': No such file or directory
ls: cannot access '/dev/dri': No such file or directory

Check Docker run configuration:

# problematic run
docker run -it --rm my-rocm-image python train.py

Notice the missing --device flags.

Capture a short packet trace (optional) to see if the kernel driver receives any ioctl after the crash:
```
# host
sudo tcpdump -i any -nn -s 0 -w /tmp/rocm.pcap 'port 1024'
```

Review ROCm device plugin logs (Kubernetes):

# host
journalctl -u rocm-device-plugin -f
2024-08-27T12:34:56.789Z plugin: GPU 0: controller manager crashed, resetting GPU

Resolution

The fix consists of aligning driver/runtime versions, ensuring proper device exposure, and optionally disabling the watchdog for workloads that exceed its limits.

Step 1 – Use a matching ROCm base image

Replace the older ROCm libraries with the host‑compatible version (e.g., ROCm 6.0). Example Dockerfile change:

# Before
FROM rocm/pytorch:5.6.0

# After
FROM rocm/pytorch:6.0.0
RUN apt-get update && apt-get install -y rocm-dkms rocm-dev

Step 2 – Bind‑mount required devices

Run containers with explicit device flags or use --privileged only as a temporary test.

# Recommended
docker run -it --rm \
  --device=/dev/kfd \
  --device=/dev/dri \
  -e ROCM_DISABLE_GPU_RESET=1 \
  my-rocm-image python train.py

For Kubernetes, ensure the rocm-device-plugin version ≥ 1.12 is deployed and the ROCM_DISABLE_GPU_RESET environment variable is set in the pod spec.

Step 3 – Adjust watchdog timeout (if needed)

When workloads legitimately exceed the 5‑second watchdog, increase the timeout via a kernel module parameter. Add the following line to /etc/modprobe.d/amdgpu.conf and reload the module:

# /etc/modprobe.d/amdgpu.conf
options amdgpu watchdog_timeout=30

# host
sudo modprobe -r amdgpu
sudo modprobe amdgpu

Note: Raising the timeout may mask genuine hangs; use only after confirming kernel stability.

Step 4 – Verify driver health after changes

Restart the Docker service and run a short training loop to ensure the controller manager stays alive.

Validation

Check dmesg for absence of crash messages:

# host
$ dmesg | grep -i "controller manager"
(no output)

Confirm device visibility inside the container:

# container
$ ls -l /dev/kfd /dev/dri
crw-rw---- 1 root video 250, 0 Aug 27 12:00 /dev/kfd
crw-rw---- 1 root video 226, 0 Aug 27 12:00 /dev/dri/card0

Run a long training epoch (≥6 h) and monitor:
```
# host
$ watch -n 60 "rocm-smi --showtemp --showpower"
```
No GPU resets should appear.
Inspect container exit code:
```
# host
$ echo $?
0
```

Operational Experience and Lessons Learned

Initial suspicion fell on the container’s --gpus flag, but the real blocker was the missing /dev/kfd bind‑mount, a detail often omitted in quick‑start guides.
Version drift between host ROCm (6.0) and the image (5.6) caused subtle ABI mismatches that only manifested under heavy kernel execution, making the crash appear random.
Disabling the watchdog resolved most segfaults, yet it also concealed genuine hardware hangs; after the fix we added a Prometheus alert on amdgpu_gpu_reset_total to catch real hangs.
In multi‑container scenarios, the kernel’s per‑GPU watchdog is shared; scaling beyond two containers without adjusting watchdog_timeout repeatedly triggered resets.

Best Practices and Prevention

Pin ROCm versions on both host and container images; use the same major/minor release.
Leverage the official ROCm Docker Integration Guide to set up --device=/dev/kfd and /dev/dri automatically via the rocm-docker wrapper script.
Monitor kernel logs for amdgpu: controller manager messages and set up alerts on dmesg patterns.
Enable the ROCm device plugin in Kubernetes with the latest version and propagate ROCM_DISABLE_GPU_RESET when workloads are known to be safe.
Test long‑duration workloads in a staging environment before production deployment to surface watchdog timeouts early.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does the container work locally but crash when orchestrated by Kubernetes?
The Kubernetes rocm-device-plugin may run an older binary that does not forward the ROCM_DISABLE_GPU_RESET flag, causing the plugin to allow resets that the container cannot survive.
Can I run ROCm containers without mounting /dev/kfd?
No. /dev/kfd is the kernel‑mode driver interface required for HSA queue submission. Without it the controller manager cannot be created, leading to immediate failures.
Is disabling the watchdog safe for production?
Disabling it prevents the driver from killing a hung GPU, which can hide real hardware faults. Use it only after confirming that your kernels are well‑behaved, and keep external monitoring for GPU hangs.
How do I know which ROCm version the host is running?
Run cat /sys/module/amdgpu/version on the host or rocm-smi --showversion. Ensure the container image’s rocm packages match this output.
What kernel module parameter controls the watchdog timeout?
The parameter is watchdog_timeout in the amdgpu module. Set it via /etc/modprobe.d/amdgpu.conf and reload the module.