AMD GPU stop sequence fails during CI/CD pipeline execution

Problem

During automated CI/CD pipeline runs that invoke GPU‑accelerated workloads on AMD GPUs, the pipeline hangs after the test suite completes. The hanging manifests as lingering amdgpu driver processes (amdgpu_top, rocminfo, user‑level compute kernels) that never receive the expected stop sequence. Consequently, the pipeline step exceeds its timeout and fails.

Typical log excerpt from the CI runner:


[2026-11-10T14:32:07Z] INFO  Starting GPU workload on node gpu‑runner-01
[2026-11-10T14:32:08Z] INFO  ROCm version: 6.1.0
[2026-11-10T14:32:09Z] INFO  Launching container image: myorg/ml-test:latest
[2026-11-10T14:32:15Z] ERROR Kernel execution failed: HSA_STATUS_ERROR_INVALID_ARGUMENT
[2026-11-10T14:32:15Z] INFO  Initiating GPU stop sequence...
[2026-11-10T14:32:15Z] WARN  amdgpu driver: stop sequence timed out after 30s
[2026-11-10T14:32:45Z] INFO  Pipeline step timed out (1800s)
[2026-11-10T14:32:45Z] INFO  Cleaning up container...
[2026-11-10T14:32:46Z] ERROR  Process still running: /usr/bin/rocminfo (pid 3421)

The failure is reproducible only when the CI job runs inside a Docker container orchestrated by kubectl/helm on a shared GPU node. Manual runs on the same node (outside CI) complete successfully and the GPU stop sequence works as expected.

Root Cause

The AMD GPU driver expects a privileged cgroup hierarchy that propagates the SIGTERM signal to all processes belonging to the GPU device file (/dev/kfd and /dev/dri/cardX). In the CI environment the container runtime is configured with --cgroup-driver=systemd while the host uses the legacy cgroupfs driver. This mismatch prevents the kernel from delivering the stop sequence to the processes that were started inside the container’s cgroup namespace. As a result, the driver’s internal watchdog never receives the termination acknowledgement and reports a timeout.

Additional contributing factors:

  • CI pipelines use docker run --runtime=nvidia (a leftover from NVIDIA setups) which does not map AMD device files correctly.
  • The amd-gpu-reset.service is disabled on the node to avoid accidental resets, so the driver cannot fall back to a hard reset when the graceful stop fails.

Debug

Below is a step‑by‑step debugging workflow that isolates the cgroup mismatch and confirms the missing signal propagation.

  1. Inspect the cgroup driver on the host:
cat /sys/fs/cgroup/cgroup.controllers
# Expected output includes: cpu memory pids rdma
  1. Check the Docker daemon configuration:
cat /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
  1. Verify the container’s cgroup path:
docker run --rm -d --name cgtest alpine sleep 3600
cat /proc/$(docker inspect -f '{{.State.Pid}}' cgtest)/cgroup
# Output shows: 0::/docker/abcdef...
  1. Confirm that amdgpu device files are bound correctly:
docker run --rm -it --device=/dev/kfd --device=/dev/dri/card0 alpine ls -l /dev/kfd /dev/dri/card0
# If missing, the container cannot interact with the GPU.
  1. Capture the stop sequence signal flow with strace on the host:
strace -f -e trace=signal -p $(pgrep -f 'rocminfo')
# Look for missing SIGTERM/SIGKILL delivery to child processes.
  1. Check the status of amd-gpu-reset.service:
systemctl status amd-gpu-reset.service
# Output: disabled; vendor preset: disabled

Solution

The fix consists of aligning the cgroup driver between host and container runtime, ensuring proper device binding, and enabling the fallback reset service.

1. Align cgroup drivers

Configure Docker (or containerd) to use cgroupfs, matching the host’s default.

Before:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

After:

{
  "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

Restart the Docker daemon:

systemctl restart docker

2. Use the AMD‑specific runtime

Replace the erroneous NVIDIA runtime flag with the official AMD ROCm runtime (--runtime=rocm) and explicitly expose the required devices.

Before (CI job definition):

docker run --runtime=nvidia -v /dev/kfd:/dev/kfd -v /dev/dri:/dev/dri ...

After:

docker run --runtime=rocm \
  --device=/dev/kfd \
  --device=/dev/dri/card0 \
  --device=/dev/dri/renderD128 \
  ...

3. Enable the GPU reset service

Allow the driver to perform a hard reset if the graceful stop fails.

systemctl enable --now amd-gpu-reset.service

4. Update the CI pipeline script to explicitly stop the GPU

Add a cleanup step that sends SIGTERM to all processes that have opened /dev/kfd before the container exits.

Before (pipeline snippet):

# No explicit GPU cleanup
docker run ... myorg/ml-test:latest

After:

# Graceful GPU stop
GPU_PIDS=$(lsof /dev/kfd | awk 'NR>1 {print $2}' | sort -u)
if [ -n "$GPU_PIDS" ]; then
  echo "Sending SIGTERM to GPU processes: $GPU_PIDS"
  kill -TERM $GPU_PIDS
  timeout 30s bash -c "while kill -0 $GPU_PIDS 2>/dev/null; do sleep 1; done"
fi

docker run ... myorg/ml-test:latest

Verify

After applying the changes, run the pipeline and confirm that the stop sequence completes without timeout.

  1. Trigger a CI job and watch the logs for the stop sequence:

[2026-11-10T15:02:10Z] INFO  Initiating GPU stop sequence...
[2026-11-10T15:02:12Z] INFO  GPU stop sequence completed successfully (elapsed 2s)
[2026-11-10T15:02:13Z] INFO  Pipeline step finished in 1723s
  • Verify that no GPU processes remain on the host:
  • pgrep -f 'rocminfo|amdgpu_top' || echo "No lingering GPU processes"
    No lingering GPU processes
  • Check the systemd service status:
  • systemctl is-active amd-gpu-reset.service
    # Expected output: active

    Prevent

    • Standardize cgroup drivers: Enforce a cluster‑wide policy (e.g., via KubeletConfiguration) that sets cgroupDriver: cgroupfs for all nodes.
    • Use the ROCm runtime in all CI/CD definitions; remove legacy NVIDIA flags.
    • Enable amd‑gpu‑reset.service by default on GPU nodes to provide a safety net.
    • Automated health check: Add a post‑job step that runs lsof /dev/kfd and fails the job if any process is still attached.
    • Version pinning: Keep the ROCm driver and container runtime versions aligned (e.g., ROCm 6.1.x with Docker 24.x).

    Related Topic Hub: GPU Infrastructure Troubleshooting Hub

    FAQ

    1. Why does the stop sequence work on manual runs but not in CI?
      Because manual runs use the host’s default cgroup driver (cgroupfs), while the CI Docker daemon was configured for systemd, causing signal propagation failures.
    2. Can I keep the systemd cgroup driver and still make the GPU stop work?
      Yes, but you must also configure the kubelet and container runtime to use systemd on the host, and ensure the container’s cgroup namespace mirrors the host’s hierarchy. This adds complexity and is generally not recommended for mixed‑driver clusters.
    3. Do I need to rebuild my container images after fixing the runtime flag?
      No. The change is purely in the docker run (or Kubernetes pod spec) command line. Existing images can be reused.
    4. What if enabling amd-gpu-reset.service causes unexpected GPU resets?
      The service only triggers when the driver reports a graceful stop timeout. In a healthy pipeline it remains idle. Monitor the journalctl -u amd-gpu-reset.service logs for any reset events.
    5. How can I detect cgroup driver mismatches before they affect pipelines?
      Add a pre‑flight script to the CI agent that compares /etc/docker/daemon.json against the host’s /proc/1/cgroup output and aborts the job if they differ.