Container crash loop on AMD GPU during preprocessing stage

Problem

A container that runs the data‑pipeline preprocessing and model‑inference stages crashes repeatedly when the node is equipped with an AMD GPU. The pod enters a CrashLoopBackOff after the preprocessing step completes, preventing the downstream inference from ever starting.

Symptoms and Impact

• Pod status: CrashLoopBackOff after 0/1 containers ready.
• Container logs end with a stack trace from the AMD ROCm runtime:

2026-10-30T12:45:01.123Z ERROR [preprocess] ROCm error: HSA_STATUS_ERROR_INVALID_ARGUMENT (0x5)
2026-10-30T12:45:01.124Z TRACE [preprocess] at /app/preprocess.py:112
2026-10-30T12:45:01.124Z   torch.cuda.is_available() -> False
2026-10-30T12:45:01.124Z   torch.backends.mps.is_available() -> False
2026-10-30T12:45:01.124Z   torch.backends.amd.is_available() -> False
2026-10-30T12:45:01.124Z FATAL: Unhandled exception in preprocessing worker
2026-10-30T12:45:01.124Z   File "/app/preprocess.py", line 115, in main
2026-10-30T12:45:01.124Z     device = torch.device("amd")
2026-10-30T12:45:01.124Z RuntimeError: AMD GPU not found
2026-10-30T12:45:01.124Z Container terminated with exit code 1

Metrics: GPU utilization stays at 0 % while CPU spikes during the preprocessing step.

Impact: Entire batch fails; downstream training jobs are blocked, causing SLA breach.

Root Cause

The failure originates from a mismatch between the container’s ROCm driver version and the host’s kernel‑level AMDGPU driver. The host runs ROCm 5.4, but the container image bundles ROCm 5.2 libraries. When the preprocessing code calls torch.device("amd"), the ROCm runtime attempts to open /dev/kfd and /dev/dri/renderD*. Because the user‑space libraries expect a newer kernel ABI, the kernel returns HSA_STATUS_ERROR_INVALID_ARGUMENT, which propagates as “AMD GPU not found”. The crash loop is triggered because the container exits with a non‑zero status and the orchestrator restarts it.

Additional contributing factors:

• The Docker runtime was launched with --gpus all, which is a NVIDIA‑specific flag; AMD GPUs require the --device=/dev/kfd --device=/dev/dri flags or the rocm runtime class.
• The base image (python:3.11-slim) does not contain the libdrm-amdgpu1 package, causing missing kernel‑user space symbols at load time.

Debug

1. Verify host driver version

$ /opt/rocm/bin/rocminfo | grep "ROCm Version"
ROCm Version: 5.4.0

2. Inspect container runtime configuration

$ docker inspect my-pipeline-pod | jq '.[0].HostConfig.DeviceRequests'
[
  {
    "Count": -1,
    "DeviceIDs": null,
    "Capabilities": [
      [
        "gpu"
      ]
    ],
    "Driver": "",
    "Options": {}
  }
]

The request uses the generic gpu capability, which maps to the NVIDIA runtime.

3. Check library versions inside the container

$ docker run --rm my-pipeline-image dpkg -l | grep rocm
ii  rocm-dev          5.2.0-1   AMD ROCm Development Packages
ii  rocm-libs         5.2.0-1   AMD ROCm Runtime Libraries

4. Capture a low‑level ROCm error

$ sudo dmesg | grep -i hsa
[ 12345.678901] hsaKmt: Invalid argument passed to hsaKmtOpenKFD()

5. Reproduce the failure locally

# Inside the container
$ python -c "import torch; print(torch.backends.amd.is_available())"
False
$ python -c "import torch; torch.device('amd')"
Traceback (most recent call last):
  File "", line 1, in <module>
RuntimeError: AMD GPU not found

Solution

1. Align ROCm versions

Rebuild the container image using the same ROCm version as the host (5.4). The official rocm/pytorch base image guarantees ABI compatibility.

# Dockerfile (before)
FROM python:3.11-slim
RUN apt-get update && apt-get install -y \
    rocm-dev=5.2.0-1 rocm-libs=5.2.0-1 \
    && rm -rf /var/lib/apt/lists/*

# Dockerfile (after)
FROM rocm/pytorch:5.4.0-py3.11
# No need to install rocm-dev manually; the base image provides matching libs
WORKDIR /app
COPY . /app
RUN pip install -r requirements.txt

2. Use the correct device request for AMD GPUs

In the pod spec, replace the NVIDIA‑style gpu request with the rocm runtime class or explicit device mounts.

# pod.yaml (before)
resources:
  limits:
    nvidia.com/gpu: 1

# pod.yaml (after)
runtimeClassName: rocm
containers:
- name: preprocess
  image: my-pipeline-image:latest
  resources:
    limits:
      amd.com/gpu: 1
  volumeMounts:
  - name: kfd
    mountPath: /dev/kfd
  - name: dri
    mountPath: /dev/dri
volumes:
- name: kfd
  hostPath:
    path: /dev/kfd
- name: dri
  hostPath:
    path: /dev/dri

3. Install missing kernel‑user space dependencies

Add libdrm-amdgpu1 and rocm-dkms to the image if a custom base is required.

# Dockerfile snippet (if not using rocm/pytorch)
RUN apt-get update && apt-get install -y \
    libdrm-amdgpu1 rocm-dkms \
    && rm -rf /var/lib/apt/lists/*

4. Re‑deploy

$ kubectl apply -f pod.yaml
$ kubectl rollout status deployment/pipeline-preprocess
deployment "pipeline-preprocess" successfully rolled out

Verify

• Pod status should be Running with READY 1/1.
• Container logs show successful GPU detection:

2026-10-30T12:50:12.001Z INFO [preprocess] AMD GPU detected: gfx1100
2026-10-30T12:50:12.002Z INFO [preprocess] torch.backends.amd.is_available() -> True
2026-10-30T12:50:12.003Z INFO [preprocess] Starting preprocessing on device: amd

• GPU utilization rises above 0 % during preprocessing (check with rocm-smi or Prometheus metrics).
• Subsequent inference pods start without entering CrashLoopBackOff.

Prevent

Preventive Action	Why it helps
Pin container ROCm version to host version	Ensures ABI compatibility between user‑space libraries and kernel driver.
Use the rocm runtime class or explicit device mounts	Prevents the orchestrator from applying NVIDIA‑specific device plugins.
Validate GPU availability in CI	Catches version mismatches before production rollout.
Monitor rocm-smi health metrics	Early detection of driver or firmware regressions.
Document required host packages (rocm-dkms, libdrm-amdgpu1)	Guarantees that all kernel‑user space dependencies are present on every node.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

1. Why does the container work on a test node with a newer ROCm version but fail in production? The test node had ROCm 5.5, matching the container’s libraries. Production nodes run ROCm 5.4, causing an ABI mismatch that manifests as “AMD GPU not found”.
2. Can I keep the --gpus all flag and still use AMD GPUs? No. That flag is interpreted by the NVIDIA container runtime. For AMD you must either use the rocm runtime class or manually mount /dev/kfd and /dev/dri.
3. What if I need both NVIDIA and AMD GPUs on the same node? Deploy separate pod specs with the appropriate runtimeClassName (e.g., nvidia and rocm) and ensure the node runs both drivers side‑by‑side, which is supported by recent ROCm releases.
4. How can I programmatically verify GPU availability before starting preprocessing? Use the PyTorch check:

import torch
if not torch.backends.amd.is_available():
    raise RuntimeError("AMD GPU not detected – aborting pipeline")

Is there a way to automatically reconcile ROCm versions across nodes? Yes. Deploy a DaemonSet that runs rocm-smi --showversion and annotates the node; a mutating webhook can then inject the matching image tag into pods.