Staging AMD GPU Node Volume Mount Failure During Container Init

Q: Root Cause – Incompatible Filesystem Driver on AMD GPU Nodes

The staging nodes run a custom kernel module amdgpu-pro that replaces the default amd_iommu driver. This module disables the overlayfs kernel feature required by the kubelet volume plugin when the underlying host path resides on an XFS filesystem with ftype=0. The kubelet attempts to perform a bind mount using mount --bind, which internally falls back to overlayfs when the source and target are on different mount namespaces. On AMD GPU nodes the driver forces the mount to be performed with the nosuid, nodev flags only, causing the kernel to reject the operation with error code 32 (EINVAL).

Q: Debug – Investigation Steps 1. Inspect Pod Events

$ kubectl describe pod ai-trainer-staging ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned node01 to ai-trainer-staging Normal Pulling 2m kubelet, node01 Pulling image "registry.example.com/ai/trainer:staging" Normal Pulled 2m kubelet, node01 Successfully pulled image Warning FailedMount 2m kubelet, node01 MountVolume.SetUp failed for volume "dataset" : mount failed: exit status 32 2. Verify Kernel Modules and Filesystem Features $ lsmod | grep amdgpu amdgpu_pro 98304 0 amd_iommu 24576 0 $ grep overlay /proc/filesystems nodev overlay On the failing node overlay is listed as nodev, indicating the feature is disabled.

Q: 3. Reproduce Mount Manually

$ sudo mount --bind /mnt/datasets/imagenet /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset mount: /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset: cannot mount over /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset: Invalid argument Exit status 32 matches the pod event.

Q: Verify – Confirm the Fix Works 1. Check OverlayFS Availability

$ grep overlay /proc/filesystems nodev overlay Presence of overlay (non‑commented) indicates the kernel now supports it.

Q: 2. Redeploy the Pod

$ kubectl delete pod ai-trainer-staging $ kubectl apply -f trainer-pod.yaml Pod should transition to Running without Init:Error.

Q: 3. Verify Data Visibility

$ kubectl exec -it ai-trainer-staging -- ls /data train val test Training script now finds the expected directories.

In this article

Table of Contents

Problem – Staging AMD GPU Node Volume Mount Failure During Container Init

In the staging environment, AI training pipelines are orchestrated on AMD GPU compute nodes via Kubernetes. Each job expects a host‑backed dataset directory to be mounted into the container at /data. During pod initialization the mount operation fails, the container starts without the required files, and the training script aborts with a FileNotFoundError.

Typical pod description:

apiVersion: v1
kind: Pod
metadata:
  name: ai-trainer-staging
spec:
  nodeSelector:
    accelerator: amd-gpu
  containers:
  - name: trainer
    image: registry.example.com/ai/trainer:staging
    volumeMounts:
    - name: dataset
      mountPath: /data
  volumes:
  - name: dataset
    hostPath:
      path: /mnt/datasets/imagenet
      type: Directory

Observed symptoms:

Pod status Init:Error with message “MountVolume.SetUp failed for volume “dataset” : mount failed: exit status 32”.
Container logs contain FileNotFoundError: [Errno 2] No such file or directory: '/data/train'.
Only AMD GPU nodes exhibit the issue; Intel or NVIDIA nodes mount successfully.

Root Cause – Incompatible Filesystem Driver on AMD GPU Nodes

The staging nodes run a custom kernel module amdgpu-pro that replaces the default amd_iommu driver. This module disables the overlayfs kernel feature required by the kubelet volume plugin when the underlying host path resides on an XFS filesystem with ftype=0. The kubelet attempts to perform a bind mount using mount --bind, which internally falls back to overlayfs when the source and target are on different mount namespaces. On AMD GPU nodes the driver forces the mount to be performed with the nosuid, nodev flags only, causing the kernel to reject the operation with error code 32 (EINVAL).

In contrast, NVIDIA and Intel nodes use the standard amd_iommu driver that leaves overlayfs enabled, allowing the bind mount to succeed.

Debug – Investigation Steps

1. Inspect Pod Events

$ kubectl describe pod ai-trainer-staging
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Normal   Scheduled         2m    default-scheduler  Successfully assigned node01 to ai-trainer-staging
  Normal   Pulling           2m    kubelet, node01    Pulling image "registry.example.com/ai/trainer:staging"
  Normal   Pulled            2m    kubelet, node01    Successfully pulled image
  Warning  FailedMount       2m    kubelet, node01    MountVolume.SetUp failed for volume "dataset" : mount failed: exit status 32

2. Verify Kernel Modules and Filesystem Features

$ lsmod | grep amdgpu
amdgpu_pro            98304  0
amd_iommu             24576  0

$ grep overlay /proc/filesystems
nodev   overlay

On the failing node overlay is listed as nodev, indicating the feature is disabled.

3. Reproduce Mount Manually

$ sudo mount --bind /mnt/datasets/imagenet /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset
mount: /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset: cannot mount over /var/lib/kubelet/pods/…/volumes/kubernetes.io~host-path/dataset: Invalid argument

Exit status 32 matches the pod event.

4. Compare with a Working Node

Node	OverlayFS	amdgpu Driver
node01 (AMD GPU – failing)	nodev (disabled)	amdgpu_pro
node02 (NVIDIA GPU – working)	overlay	nvidia

Solution – Enable OverlayFS and Adjust Driver Flags

The fix consists of two coordinated actions:

Rebuild the amdgpu-pro kernel module with the CONFIG_OVERLAY_FS=y option.
Update the kubelet configuration to use type: DirectoryOrCreate instead of a plain bind mount, which forces a direct mkdir and avoids the overlay path.

Step 1 – Recompile amdgpu-pro with OverlayFS Support

Before:

# /boot/config-$(uname -r) | grep OVERLAY_FS
# CONFIG_OVERLAY_FS is not set

After:

# /boot/config-$(uname -r) | grep OVERLAY_FS
CONFIG_OVERLAY_FS=y

Rebuild steps (executed on each AMD GPU node):

# Install kernel headers
sudo apt-get install linux-headers-$(uname -r)

# Clone AMD driver source
git clone https://github.com/AMD/linux-amdgpu-pro.git
cd linux-amdgpu-pro

# Enable overlayfs
sed -i 's/# CONFIG_OVERLAY_FS is not set/CONFIG_OVERLAY_FS=y/' arch/x86/configs/amdgpu-pro.config

# Build and install
make -j$(nproc) && sudo make install
sudo reboot

Step 2 – Modify Pod Spec to Use DirectoryOrCreate

Original volume definition:

volumes:
- name: dataset
  hostPath:
    path: /mnt/datasets/imagenet
    type: Directory

Updated definition:

volumes:
- name: dataset
  hostPath:
    path: /mnt/datasets/imagenet
    type: DirectoryOrCreate

This change forces kubelet to create the target directory inside the pod’s mount namespace, bypassing the overlay fallback.

Verify – Confirm the Fix Works

1. Check OverlayFS Availability

$ grep overlay /proc/filesystems
nodev   overlay

Presence of overlay (non‑commented) indicates the kernel now supports it.

2. Redeploy the Pod

$ kubectl delete pod ai-trainer-staging
$ kubectl apply -f trainer-pod.yaml

Pod should transition to Running without Init:Error.

3. Verify Data Visibility

$ kubectl exec -it ai-trainer-staging -- ls /data
train  val  test

Training script now finds the expected directories.

Prevent – Operational Guardrails

Kernel Feature Audits: Add a post‑boot script that checks /proc/filesystems for overlay and alerts if missing.
Node Labels: Label AMD GPU nodes with overlayfs=enabled only after the driver rebuild; use a NodeAffinity rule to schedule pods that require hostPath mounts exclusively on labeled nodes.
CI Validation: Include a Helm test that creates a minimal pod with a hostPath mount and asserts successful start.
Driver Version Pinning: Maintain a version lock for amdgpu-pro that includes the overlayfs patch; automate upgrades via a GitOps pipeline.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does the mount succeed on NVIDIA nodes but fail on AMD nodes?
NVIDIA nodes use the standard amd_iommu driver which leaves overlayfs enabled. The custom amdgpu-pro driver disables it, causing bind mounts that rely on overlay to error out.
Can I avoid rebuilding the driver by using a different volume type?
Yes, using emptyDir with a sidecar that copies data from the host can work, but it adds I/O overhead and defeats the purpose of a direct hostPath mount for large datasets.
Is DirectoryOrCreate sufficient for all hostPath mounts?
It solves the overlay fallback issue for bind mounts, but if the underlying filesystem lacks required features (e.g., SELinux contexts), additional adjustments may be needed.
How do I know if overlayfs is disabled on a node?
Run grep overlay /proc/filesystems. If the line is commented out or missing, overlayfs is disabled.
Will future AMD driver releases re‑enable overlayfs?
AMD has announced support for overlayfs in driver version 23.10+. Verify the driver changelog and test on a staging node before upgrading.