Hugging Face Trainer FileNotFoundError on shared dataset directory

Problem Description

The Hugging Face Trainer crashes during the initialization phase of a distributed training run on a Kubernetes cluster. The failure manifests as a FileNotFoundError or PermissionError when the trainer tries to read the shared dataset directory or create the checkpoint folder.

Typical log excerpts from the failing pod (rank 2) are:

2026-07-31 10:12:45,123 | ERROR | transformers.trainer | FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data/train'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/transformers/trainer.py", line 2154, in _setup_datasets
    self.train_dataset = self._get_train_dataset()
  File "/usr/local/lib/python3.10/site-packages/transformers/trainer.py", line 2121, in _get_train_dataset
    dataset = load_dataset(self.args.dataset_name, split=self.args.train_split)
...
2026-07-31 10:12:46,078 | ERROR | transformers.trainer | PermissionError: [Errno 13] Permission denied: '/mnt/checkpoints/epoch_0'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/transformers/trainer.py", line 3061, in _save
    os.makedirs(output_dir, exist_ok=True)
  File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/mnt/checkpoints/epoch_0'

Impact:

• Training never starts; the distributed process group is never initialized, leading to cascading RuntimeError: Distributed process group is not initialized.
• All GPU resources remain idle, causing cost waste.
• In multi‑node setups, only a subset of ranks may see the error, making the failure appear intermittent.

Root Cause Analysis

Three inter‑related factors cause the observed filesystem errors:

1. RWX volume mount propagation – The ReadWriteMany PersistentVolumeClaim (PVC) is mounted correctly on the primary pod, but Kubernetes does not propagate the mount to the worker containers spawned by torchrun / FSDP unless shareProcessNamespace and proper securityContext are configured. This is documented in the Kubernetes PersistentVolume documentation.
2. UID/GID mismatch – The PVC export (e.g., NFS) applies root_squash and enforces file ownership based on the client’s UID/GID. The container runs as a non‑root user (UID 1000) while the NFS export expects a different group. As reported in the GitHub issue Trainer fails with FileNotFoundError on shared dataset dir in k8s, this mismatch leads to read‑only visibility for some ranks.
3. Concurrent directory creation race – FSDP ranks start simultaneously and each attempts to create the same checkpoint directory (/mnt/checkpoints/epoch_0). On a shared filesystem without atomic directory creation handling, one rank succeeds while others encounter OSError: [Errno 17] File exists, which can cascade into a FileNotFoundError on subsequent accesses. This behavior matches the incident described in the evidence package where “multiple ranks tried to create the same checkpoint directory simultaneously”.

Combined, these issues prevent the Trainer from locating the dataset and writing checkpoints, causing the early crash.

Investigation and Debugging

Below is a step‑by‑step debugging workflow that reproduces the findings from the community sources.

1. Verify volume mount on each rank

# Inside each rank container
kubectl exec -it trainer-0-0 -- bash -c "ls -l /mnt/data"

Expected output:

total 0
-rw-r--r-- 1 1000 1000 0 Jul 31 10:10 train.csv

If a rank reports “No such file or directory”, the mount did not propagate.

2. Inspect UID/GID mapping

kubectl exec -it trainer-0-0 -- bash -c "id"

Typical output:

uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)

Compare with the NFS export’s fsid permissions (on the NFS server):

showmount -e nfs-server
/export/data  *(rw,sync,no_root_squash)

If no_root_squash is missing, non‑root users are mapped to nobody, causing Permission denied.

3. Capture race condition logs

Enable verbose FSDP logging:

export TORCH_SHOW_CPP_STACKTRACES=1
export FSDP_LOG_LEVEL=debug

Look for interleaved messages such as:

[rank:2] OSError: [Errno 17] File exists: '/mnt/checkpoints/epoch_0'
[rank:3] FileNotFoundError: [Errno 2] No such file or directory: '/mnt/checkpoints/epoch_0'

4. Review pod spec for readOnly flag

apiVersion: v1
kind: Pod
metadata:
  name: trainer
spec:
  containers:
  - name: trainer
    image: huggingface/transformers:latest
    volumeMounts:
    - name: shared-data
      mountPath: /mnt/data
      readOnly: true   # <-- common mistake
  volumes:
  - name: shared-data
    persistentVolumeClaim:
      claimName: dataset-pvc

The readOnly: true flag forces a read‑only mount, reproducing the OSError: [Errno 30] Read-only file system observed in the evidence.

Resolution

Apply the following changes to the deployment manifest and container entrypoint. The before/after comparison highlights the critical fixes.

1. Ensure mount propagation and shared process namespace

# Before
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trainer
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: trainer
        image: huggingface/transformers:latest
        command: ["torchrun", "--nproc_per_node=4", "run_training.py"]
        volumeMounts:
        - name: shared-data
          mountPath: /mnt/data
      volumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: dataset-pvc

# After
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trainer
spec:
  replicas: 1
  template:
    spec:
      shareProcessNamespace: true               # Enables child processes to see the same mounts
      securityContext:
        fsGroup: 1000                           # Aligns NFS group ownership
      containers:
      - name: trainer
        image: huggingface/transformers:latest
        command: ["torchrun", "--nproc_per_node=4", "--rdzv_id=trainer", "--rdzv_backend=c10d", "run_training.py"]
        env:
        - name: TORCH_DISTRIBUTED_DEBUG
          value: "DETAIL"
        volumeMounts:
        - name: shared-data
          mountPath: /mnt/data
          readOnly: false                       # Ensure write access for checkpoints
        - name: checkpoint-dir
          mountPath: /mnt/checkpoints
      initContainers:
      - name: permission-fix
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /mnt/data /mnt/checkpoints"]
        volumeMounts:
        - name: shared-data
          mountPath: /mnt/data
        - name: checkpoint-dir
          mountPath: /mnt/checkpoints
      volumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: dataset-pvc
      - name: checkpoint-dir
        persistentVolumeClaim:
          claimName: checkpoint-pvc

2. Use a pre‑creation barrier for checkpoint directory

Modify the training script to create the checkpoint directory only on rank 0 and then synchronize.

import os
import torch
from torch.distributed import barrier

def prepare_checkpoint_dir(output_dir):
    if torch.distributed.get_rank() == 0:
        os.makedirs(output_dir, exist_ok=True)
    barrier()
    # All ranks now see the directory
    return output_dir

# In the Trainer arguments
training_args = TrainingArguments(
    output_dir="/mnt/checkpoints",
    ...
)

training_args.output_dir = prepare_checkpoint_dir(training_args.output_dir)
trainer = Trainer(..., args=training_args)

3. Align NFS export permissions

On the NFS server, export the volume with no_root_squash and set the appropriate fsid group:

/export/data *(rw,sync,no_root_squash,fsid=1000)

Validation

After applying the fixes, perform the following checks:

1. Dataset visibility

   
   kubectl exec -it trainer-0-0 -- bash -c "ls -l /mnt/data"
   

   Expected output shows the train.csv file owned by UID 1000.

2. Checkpoint directory creation

   
   kubectl exec -it trainer-0-0 -- bash -c "ls -ld /mnt/checkpoints"
   

   Should return something like:

   
   drwxr-xr-x 2 1000 1000 4096 Jul 31 10:20 /mnt/checkpoints
   

3. Trainer start‑up logs – No FileNotFoundError or PermissionError lines. The first log line after initialization should be:

   
   2026-07-31 10:15:02,001 | INFO | transformers.trainer | Training/evaluation started
   

4. Distributed barrier – Verify that all ranks pass the barrier by checking that each rank logs “Barrier passed”.
5. Checkpoint persistence – After the first epoch, confirm that a checkpoint file exists:

   
   kubectl exec -it trainer-0-0 -- bash -c "ls /mnt/checkpoints"
   

   Should list epoch_0 and associated model files.

Operational Experience

During the investigation, several misleading symptoms were observed:

• Only rank 2 reported FileNotFoundError while rank 0 succeeded, leading to the mistaken belief that the dataset path was dynamically generated per rank.
• The initial error “RuntimeError: Distributed process group is not initialized” masked the underlying filesystem problem; fixing the mount restored the process group.
• In a previous deployment, the fsGroup setting was omitted, causing NFS’s root_squash to map the trainer’s UID to nobody. The trainer could read the dataset (NFS allowed read for everyone) but could not write checkpoints, reproducing the exact PermissionError seen in the logs.

Best Practices and Prevention

Practice	Why it matters	Implementation
Use fsGroup in pod securityContext	Ensures NFS file ownership matches container UID/GID.	securityContext: { fsGroup: 1000 }
Set shareProcessNamespace: true	Allows child processes spawned by torchrun to inherit the same volume mounts.	Pod spec flag as shown in the “After” manifest.
Initialize shared directories on rank 0 only	Prevents race conditions that cause OSError: File exists.	Use a barrier after os.makedirs (see code snippet).
Validate PVC mount mode	Accidental readOnly: true leads to checkpoint write failures.	Inspect the pod spec or run mount | grep /mnt/checkpoints inside the container.
Monitor filesystem health	Detect NFS outages or permission changes before they affect training.	Export Prometheus metrics from node_exporter for mountpoint availability.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the dataset appear on the driver pod but not on the worker ranks?
   Because the volume mount is not propagated to the child processes created by torchrun. Enabling shareProcessNamespace and ensuring the mount is not marked readOnly resolves this.
2. Can I keep the existing PVC without creating a new one for checkpoints?
   Yes, but you must mount the same PVC at two distinct paths (e.g., /mnt/data and /mnt/checkpoints) and apply the same fsGroup so that both read and write operations succeed.
3. What if the NFS server enforces root_squash and I cannot change the export?
   Run the container as the UID/GID that matches the NFS export’s allowed group, or use an initContainer to chown the mount point to the container’s user.
4. How do I avoid the checkpoint directory race condition without adding code?
   Set the environment variable FSDP_CHECKPOINT_DIR to a unique per‑rank subdirectory (e.g., /mnt/checkpoints/rank_${RANK}) or use the --output_dir flag with a rank‑aware suffix.
5. Is the issue specific to FSDP or does it affect other distributed strategies?
   Any strategy that spawns multiple processes on the same node (DDP, FSDP, DeepSpeed) will encounter the same mount‑propagation and permission problems if the pod is not configured correctly. The root cause is Kubernetes, not the specific PyTorch backend.