Problem
During automated CI/CD pipeline runs that invoke GPU‑accelerated workloads on AMD GPUs, the pipeline hangs after the test suite completes. The hanging manifests as lingering amdgpu driver processes (amdgpu_top, rocminfo, user‑level compute kernels) that never receive the expected stop sequence. Consequently, the pipeline step exceeds its timeout and fails.
Typical log excerpt from the CI runner:
[2026-11-10T14:32:07Z] INFO Starting GPU workload on node gpu‑runner-01
[2026-11-10T14:32:08Z] INFO ROCm version: 6.1.0
[2026-11-10T14:32:09Z] INFO Launching container image: myorg/ml-test:latest
[2026-11-10T14:32:15Z] ERROR Kernel execution failed: HSA_STATUS_ERROR_INVALID_ARGUMENT
[2026-11-10T14:32:15Z] INFO Initiating GPU stop sequence...
[2026-11-10T14:32:15Z] WARN amdgpu driver: stop sequence timed out after 30s
[2026-11-10T14:32:45Z] INFO Pipeline step timed out (1800s)
[2026-11-10T14:32:45Z] INFO Cleaning up container...
[2026-11-10T14:32:46Z] ERROR Process still running: /usr/bin/rocminfo (pid 3421)
The failure is reproducible only when the CI job runs inside a Docker container orchestrated by kubectl/helm on a shared GPU node. Manual runs on the same node (outside CI) complete successfully and the GPU stop sequence works as expected.
Root Cause
The AMD GPU driver expects a privileged cgroup hierarchy that propagates the SIGTERM signal to all processes belonging to the GPU device file (/dev/kfd and /dev/dri/cardX). In the CI environment the container runtime is configured with --cgroup-driver=systemd while the host uses the legacy cgroupfs driver. This mismatch prevents the kernel from delivering the stop sequence to the processes that were started inside the container’s cgroup namespace. As a result, the driver’s internal watchdog never receives the termination acknowledgement and reports a timeout.
Additional contributing factors:
- CI pipelines use
docker run --runtime=nvidia(a leftover from NVIDIA setups) which does not map AMD device files correctly. - The
amd-gpu-reset.serviceis disabled on the node to avoid accidental resets, so the driver cannot fall back to a hard reset when the graceful stop fails.
Debug
Below is a step‑by‑step debugging workflow that isolates the cgroup mismatch and confirms the missing signal propagation.
- Inspect the cgroup driver on the host:
cat /sys/fs/cgroup/cgroup.controllers
# Expected output includes: cpu memory pids rdma
- Check the Docker daemon configuration:
cat /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
- Verify the container’s cgroup path:
docker run --rm -d --name cgtest alpine sleep 3600
cat /proc/$(docker inspect -f '{{.State.Pid}}' cgtest)/cgroup
# Output shows: 0::/docker/abcdef...
- Confirm that
amdgpudevice files are bound correctly:
docker run --rm -it --device=/dev/kfd --device=/dev/dri/card0 alpine ls -l /dev/kfd /dev/dri/card0
# If missing, the container cannot interact with the GPU.
- Capture the stop sequence signal flow with
straceon the host:
strace -f -e trace=signal -p $(pgrep -f 'rocminfo')
# Look for missing SIGTERM/SIGKILL delivery to child processes.
- Check the status of
amd-gpu-reset.service:
systemctl status amd-gpu-reset.service
# Output: disabled; vendor preset: disabled
Solution
The fix consists of aligning the cgroup driver between host and container runtime, ensuring proper device binding, and enabling the fallback reset service.
1. Align cgroup drivers
Configure Docker (or containerd) to use cgroupfs, matching the host’s default.
Before:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
After:
{
"exec-opts": ["native.cgroupdriver=cgroupfs"]
}
Restart the Docker daemon:
systemctl restart docker
2. Use the AMD‑specific runtime
Replace the erroneous NVIDIA runtime flag with the official AMD ROCm runtime (--runtime=rocm) and explicitly expose the required devices.
Before (CI job definition):
docker run --runtime=nvidia -v /dev/kfd:/dev/kfd -v /dev/dri:/dev/dri ...
After:
docker run --runtime=rocm \
--device=/dev/kfd \
--device=/dev/dri/card0 \
--device=/dev/dri/renderD128 \
...
3. Enable the GPU reset service
Allow the driver to perform a hard reset if the graceful stop fails.
systemctl enable --now amd-gpu-reset.service
4. Update the CI pipeline script to explicitly stop the GPU
Add a cleanup step that sends SIGTERM to all processes that have opened /dev/kfd before the container exits.
Before (pipeline snippet):
# No explicit GPU cleanup
docker run ... myorg/ml-test:latest
After:
# Graceful GPU stop
GPU_PIDS=$(lsof /dev/kfd | awk 'NR>1 {print $2}' | sort -u)
if [ -n "$GPU_PIDS" ]; then
echo "Sending SIGTERM to GPU processes: $GPU_PIDS"
kill -TERM $GPU_PIDS
timeout 30s bash -c "while kill -0 $GPU_PIDS 2>/dev/null; do sleep 1; done"
fi
docker run ... myorg/ml-test:latest
Verify
After applying the changes, run the pipeline and confirm that the stop sequence completes without timeout.
- Trigger a CI job and watch the logs for the stop sequence:
[2026-11-10T15:02:10Z] INFO Initiating GPU stop sequence...
[2026-11-10T15:02:12Z] INFO GPU stop sequence completed successfully (elapsed 2s)
[2026-11-10T15:02:13Z] INFO Pipeline step finished in 1723s
pgrep -f 'rocminfo|amdgpu_top' || echo "No lingering GPU processes"
No lingering GPU processes
systemctl is-active amd-gpu-reset.service
# Expected output: active
Prevent
- Standardize cgroup drivers: Enforce a cluster‑wide policy (e.g., via
KubeletConfiguration) that setscgroupDriver: cgroupfsfor all nodes. - Use the ROCm runtime in all CI/CD definitions; remove legacy NVIDIA flags.
- Enable amd‑gpu‑reset.service by default on GPU nodes to provide a safety net.
- Automated health check: Add a post‑job step that runs
lsof /dev/kfdand fails the job if any process is still attached. - Version pinning: Keep the ROCm driver and container runtime versions aligned (e.g., ROCm 6.1.x with Docker 24.x).
Related Topic Hub: GPU Infrastructure Troubleshooting Hub
FAQ
- Why does the stop sequence work on manual runs but not in CI?
Because manual runs use the host’s default cgroup driver (cgroupfs), while the CI Docker daemon was configured forsystemd, causing signal propagation failures. - Can I keep the
systemdcgroup driver and still make the GPU stop work?
Yes, but you must also configure the kubelet and container runtime to usesystemdon the host, and ensure the container’s cgroup namespace mirrors the host’s hierarchy. This adds complexity and is generally not recommended for mixed‑driver clusters. - Do I need to rebuild my container images after fixing the runtime flag?
No. The change is purely in thedocker run(or Kubernetes pod spec) command line. Existing images can be reused. - What if enabling
amd-gpu-reset.servicecauses unexpected GPU resets?
The service only triggers when the driver reports a graceful stop timeout. In a healthy pipeline it remains idle. Monitor thejournalctl -u amd-gpu-reset.servicelogs for any reset events. - How can I detect cgroup driver mismatches before they affect pipelines?
Add a pre‑flight script to the CI agent that compares/etc/docker/daemon.jsonagainst the host’s/proc/1/cgroupoutput and aborts the job if they differ.