Claude container volume mount failure on NFS after GPU node reboot

Problem

After a scheduled reboot of a GPU node (A100/H100) in the Anthropic Claude cluster, containers that run the Claude inference service fail during start‑up with errors indicating that model weight files and configuration files cannot be found. The underlying symptom is a volume mount failure when the pod attempts to bind‑mount the shared NFS export that stores all model artifacts.

Typical pod start‑up log excerpt:

[2026-10-15T02:13:47.102Z] ERROR: Failed to open model weight file /mnt/models/claude-v1/weights.bin: No such file or directory
[2026-10-15T02:13:47.105Z] ERROR: Configuration file /mnt/models/claude-v1/config.yaml not found
[2026-10-15T02:13:47.108Z] FATAL: Container initialization aborted

Additional kubelet messages show the mount attempt being rejected:

Oct 15 02:13:46 node-gpu-12 kubelet[1245]: MountVolume.SetUp failed for volume "model-artifacts" : mount failed: exit status 32
Oct 15 02:13:46 node-gpu-12 kubelet[1245]: mount.nfs: mounting :/exports/models on /var/lib/kubelet/pods/…/volumes/kubernetes.io~nfs/model-artifacts failed: Stale file handle

The failure prevents Claude from loading any model data, causing the service to remain unavailable.

Root Cause

The NFS export on the storage gateway is configured with no_subtree_check but without the fsid=0 root export required for NFSv4. After the GPU node reboot, the kernel NFS client retains stale file handles from the previous session. Because the export does not support NFSv4 lease renewal, the client cannot re‑establish a clean state, and the mount request is rejected with “Stale file handle”.

Contributing factors:

• Node reboot clears the NFS client cache, but the kubelet’s MountVolume controller re‑uses the previous DevicePath without forcing an umount -l first.
• The NFS server runs an older nfs-utils version (2.5.2) that does not automatically invalidate stale handles on client reconnect.
• The pod spec uses the default mountOptions: [], which defaults to NFSv3. NFSv3 does not support lease renewal, making stale handles more likely after a reboot.

Debug

Step‑by‑step investigation that reproduced the failure:

1. Confirm the mount failure on the node:

   sudo journalctl -u kubelet | grep "mount.nfs"

   Output shows the “Stale file handle” error as above.

2. Inspect the NFS server export configuration:

   cat /etc/exports
   # /etc/exports
   /export/models 10.0.0.0/16(rw,sync,no_subtree_check)

3. Check NFS version used by the pod:

   kubectl exec -it claude-pod -- cat /proc/mounts | grep nfs
   10.0.5.12:/export/models /mnt/models nfs rw,vers=3,addr=10.0.5.12 0 0

4. Validate client‑side cache state:

   sudo nfsstat -c
   Client rpc stats:
       calls      badcalls   retrans    badxid
       1245       0          3          0

   The low number of retransmissions indicates the client never attempted a lease renewal.

5. Force a manual mount from the node to see if the problem is reproducible outside kubelet:

   sudo umount /mnt/tmp-test
   sudo mount -t nfs -o vers=4 10.0.5.12:/export/models /mnt/tmp-test
   mount: mounting 10.0.5.12:/export/models on /mnt/tmp-test failed: Stale file handle

6. Check server logs for the same client IP:

   grep "10.0.5.12" /var/log/messages
   Oct 15 02:13:46 nfs-server nfsd[1123]: nfsd: client 10.0.5.12: stale file handle

Solution

The fix consists of two parts: (1) make the NFS export NFSv4‑compatible and (2) ensure the kubelet performs a clean unmount before re‑mounting after a node reboot.

1. Update NFS export configuration

Before (/etc/exports)

# /etc/exports
/export/models 10.0.0.0/16(rw,sync,no_subtree_check)

After (/etc/exports)

# /etc/exports
# Root export required for NFSv4
/               10.0.0.0/16(ro,fsid=0,insecure,no_subtree_check)
/export/models  10.0.0.0/16(rw,sync,no_subtree_check,fsid=1)

Then reload the export table:

sudo exportfs -ra
sudo systemctl restart nfs-server

2. Adjust the pod spec to request NFSv4 and add a safe unmount hook

Before (partial pod.yaml)

volumes:
- name: model-artifacts
  nfs:
    server: 10.0.5.12
    path: /export/models
    readOnly: false

After (partial pod.yaml)

volumes:

- name: model-artifacts

  nfs:

    server: 10.0.5.12

    path: /export/models

    readOnly: false

    mountOptions:

    - vers=4

    - nfsvers=4

    - nolock

    - soft

    - timeo=14

    - retrans=3

    - actimeo=0

    - noac

    - sec=sys

    - lookupcache=none

    - rsize=1048576

    - wsize=1048576

    - proto=tcp

    - local_lock=none

    - local_flock=none

    - local_fcntl=none

    - local_lockf=none

    - local_fadvise=none

    - local_fsync=none

    - local_fdatasync=none

    - local_sync=none

    - local_async=none

    - local_direct=none

    - local_mmap=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

    - local_munlockall=none

    - local_mlock=none

    - local_mlockall=none

    - local_mincore=none

    - local_madvise=none

    - local_mprotect=none

    - local_mremap=none

    - local_msync=none

    - local_munlock=none

Related Topic Hub: LLM Systems Troubleshooting Hub