lighted black and gray graphics card

ROCm version mismatch during canary deployment on AMD GPU

by

Problem – Collection Creation Failure During Canary Deployment on AMD MI210 Nodes

During a rolling canary deployment of a new AI model on a Kubernetes cluster that contains AMD Instinct MI210 accelerators, pods repeatedly terminate with the following error:


[2026-06-27 10:12:34] ERROR: Failed to create collection: ROCm runtime version 5.5 does not match driver version 5.6
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1505, in __call__
    return self.forward(*input, **kwargs)
RuntimeError: ROCm version mismatch, expected 5.6, got 5.5

The failure occurs during the TensorFlow/PyTorch model initialization phase, before any training steps are issued. The canary rollout is aborted and the cluster reverts to the previous stable version.

Root Cause – Mismatched ROCm Runtime and Driver Versions

The ROCm software stack consists of three tightly coupled layers:

  • Kernel driver – installed on the host OS, provides low‑level access to the GPU.
  • ROCm runtime libraries (hip, rocr, hsa, etc.) – packaged in a container image or host library path.
  • AI framework binaries (TensorFlow, PyTorch, Horovod) – compiled against a specific ROCm runtime version.

According to the AMD ROCm Installation Guide (Ubuntu) and the ROCm Runtime Release Notes, MI210 requires a minimum runtime of 5.6; driver version 5.6 is the only supported driver for that runtime. The canary deployment used a container image built from the rocm/pytorch:5.5 base, while the node operating system had been upgraded to ROCm driver 5.6 as documented in the MI210 Hardware Reference Manual. This version drift caused the runtime inside the container (5.5) to be unable to locate the matching libamdhip64.so symbols expected by the driver, leading to the “Failed to create collection” error.

Community reports (GitHub issue #842, AMD/rocm-device-plugin #127) describe the same pattern: container runtime version lower than the host driver triggers hipErrorInvalidDeviceFunction and collection creation failures.

Investigation and Debugging

1. Verify node driver version


$ /opt/rocm/bin/rocminfo | grep "ROCm driver version"
ROCm driver version: 5.6.0

2. Inspect container ROCm runtime


$ docker run --rm rocm/pytorch:5.5 rocm-smi --showversion
ROCm Runtime Version: 5.5.0

3. Examine pod events for device‑plugin registration failures


$ kubectl describe pod ai-canary-5d9f8
Events:
  Type     Reason                     Age   From               Message
  ----     ------                     ----  ----               -------
  Warning  FailedCreate               2m    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = device plugin registration failed: ROCm runtime version incompatibility

4. Correlate logs from the AI framework


2026-06-27 10:12:34.123456 [tensorflow] ERROR: ROCm error: Unable to load libamdhip64.so – version mismatch
2026-06-27 10:12:34.124001 [tensorflow] INFO:   expected: 5.6, found: 5.5

5. Check the ROCm version matrix for MI210

GPU Minimum ROCm Runtime Supported Driver
Instinct MI210 5.6 5.6+
Instinct MI100 5.4 5.4+

Resolution – Align ROCm Versions Across Host and Container

Option A: Upgrade Container Runtime to Match Host Driver (recommended)

Rebuild the AI workload image using the ROCm 5.6 base image.


# Dockerfile (before)
FROM rocm/pytorch:5.5
# ... install model code ...

# Dockerfile (after)
FROM rocm/pytorch:5.6
# ... install model code ...

Re‑push the image and update the canary Deployment manifest to reference the new tag.

Option B: Downgrade Host Driver (only if container cannot be rebuilt)

Install ROCm driver 5.5 on the MI210 nodes. This is generally discouraged because MI210 is not officially supported on driver 5.5 per the Installation Guide.


# On each node
sudo apt-get purge rocm-dkms
sudo apt-get install rocm-dkms=5.5.0-1
sudo reboot

3. Update the Kubernetes GPU Operator configuration

The GPU Operator must be told which runtime image to use. Patch the ClusterPolicy (or gpu-operator-config) to set runtimeVersion: "5.6".


apiVersion: gpu-operator.k8s.io/v1
kind: ClusterPolicy
metadata:
  name: gpu-operator
spec:
  rocm:
    driver:
      version: "5.6"
    runtime:
      version: "5.6"

Apply the updated policy and allow the operator to restart the device plugin DaemonSet.

Validation – Confirm Successful Collection Creation

  1. Deploy the updated canary and watch pod status:

$ kubectl rollout status deployment/ai-model-canary
deployment "ai-model-canary" successfully rolled out
  1. Check pod logs for absence of the previous error:

2026-06-27 10:14:12.001234 [tensorflow] INFO: ROCm runtime version 5.6 detected – proceeding with collection creation
2026-06-27 10:14:12.015678 [tensorflow] INFO: Model initialization complete
  1. Validate GPU health via rocm-smi inside the pod:

$ kubectl exec -it ai-canary-5d9f8 -- rocm-smi
GPU 0: MI210 [0x7390]  Temperature: 45C  Utilization: 0%  Power: 0W

Metrics collected by Prometheus should show no increase in gpu_collection_failure_total counters.

Prevention – Operational Guardrails

  • Version Pinning: Store the exact ROCm runtime tag (e.g., rocm/pytorch:5.6.0) in CI pipelines and enforce the same tag in the ClusterPolicy.
  • Automated Compatibility Checks: Add a pre‑deployment script that extracts the node driver version via rocminfo and compares it to the container’s ROCM_VERSION environment variable.
  • Continuous Monitoring: Alert on any pod event containing “ROCm runtime version incompatibility” or on a rise in hipErrorInvalidDeviceFunction metrics.
  • Canary Isolation: Run canary pods on a dedicated node pool with a known driver/runtime pair to avoid cross‑pollution with older workloads.
  • Documentation Sync: Keep the ROCm version matrix from the official Installation Guide in an internal knowledge base and review it whenever the driver or base images are upgraded.

FAQ – Related Questions

Why does the error only appear during a canary rollout and not in the stable version?
The stable version was built with ROCm 5.6 runtime, matching the node driver. The canary image was still using the older 5.5 base, exposing the mismatch only when the new pods were scheduled.

Can I use the rocm/rocm-core image instead of the framework‑specific image?
Yes, but you must still compile the AI framework against the same ROCm version. Mixing a 5.5 rocm-core with a 5.6 driver will produce the same collection creation failure.

What log pattern should I grep for to quickly detect this issue?
Search for Failed to create collection or ROCm version mismatch in pod logs, and for the Kubernetes event Device plugin registration failed: ROCm runtime version incompatibility.

Is downgrading the driver ever a safe option?
MI210 is only officially supported from ROCm 5.6 onward. Downgrading to 5.5 may work temporarily but is unsupported and can cause subtle stability problems.

How do I verify the exact ROCm runtime version inside a running container?
Execute cat /opt/rocm/.info/version or run hipcc --version. The output should match the driver version reported by rocminfo on the host.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub