Permission denied on PVC mount during blue‑green rollout of Hugging Face Transformers

Problem Description

A blue‑green rollout of a Hugging Face transformers service on Kubernetes fails during container start‑up. The new pods report:

MountVolume.SetUp failed for volume "model-pvc": permission denied, are you root?
...
OSError: [Errno 13] Permission denied: '/model/pytorch_model.bin'
Traceback (most recent call last):
  File "/app/serve.py", line 42, in <module>
    model = AutoModel.from_pretrained("/model")
  File ".../transformers/modeling_utils.py", line 1234, in from_pretrained
    state_dict = torch.load(weights_path, map_location="cpu")
FileNotFoundError: [Errno 2] No such file or directory: '/model/config.json'

The PVC model-pvc is correctly bound and the old (green) deployment continues to serve requests, but the new (blue) pods cannot read the model weight files stored on the shared volume.

Root Cause Analysis

The failure originates from a mismatch between the file ownership on the Persistent Volume and the user identity of the new containers:

• The PVC was initially populated by a pod running as root. The CSI driver created the directory structure with owner=root:root and mode 0700 (see the incident where a CSI driver “creates volumes with mode 0700”).
• The blue deployment was configured with securityContext.runAsUser: 1001 (or omitted, causing the default non‑root user from the container image). Because the volume files are owned by root, the non‑root process cannot read them, leading to the “mount error: permission denied” and subsequent FileNotFoundError from transformers.
• Kubernetes documentation on “Configure a Pod to Use a Persistent Volume with the Correct Permissions” explains that fsGroup or an init‑container that chowns the mount point is required when a pod runs as a non‑root user.
• During the blue‑green swap the two Deployments shared the same PVC but had different securityContext settings, violating the assumption that file permissions remain compatible across revisions (as highlighted in the GitHub issue “Permission denied when loading model from PVC #23987”).

  Investigation and Debugging Steps

  1. Inspect pod events and kubelet logs:

     
     kubectl describe pod -l app=transformers,track=blue
     # Look for:
     #   Events:
     #     Type    Reason       Age   From               Message
     #     ----    ------       ----  ----               -------
     #     Warning FailedMount   2m    kubelet,worker-1  MountVolume.SetUp failed for volume "model-pvc": permission denied, are you root?
     

  2. Check the ownership of the mounted directory from inside the pod (using an initContainer that sleeps or a temporary debug pod):

     
     kubectl run debug-pod --rm -i --tty --image=busybox \
       --overrides='{
         "spec": {
           "containers": [{
             "name":"debug",
             "image":"busybox",
             "command":["sh"],
             "volumeMounts":[{"name":"model","mountPath":"/model"}]
           }],
           "volumes":[{"name":"model","persistentVolumeClaim":{"claimName":"model-pvc"}}]
         }
       }' -- sh
     # Inside the container:
     ls -l /model
     # Expected output (problem case):
     drwx------ 2 root root 4096 Jan 10 12:00 .
     

  3. Verify the UID/GID the application runs as:

     
     kubectl exec -it $(kubectl get pod -l app=transformers,track=blue -o jsonpath="{.items[0].metadata.name}") -- cat /proc/self/status | grep ^Uid
     # Output:
     Uid:    1001    1001    1001    1001    1001
     

  4. Confirm PVC access mode (should be ReadWriteMany for shared use):

     
     kubectl get pvc model-pvc -o yaml | grep accessModes
     # Output:
     accessModes:
     - ReadWriteMany
     

  5. Review the Deployment manifest for securityContext differences between green and blue:

     
     kubectl get deployment transformers-green -o yaml | grep -A3 securityContext
     kubectl get deployment transformers-blue -o yaml | grep -A3 securityContext
     

  Resolution

  Two complementary approaches resolve the permission mismatch:

  Approach A – Use fsGroup in the pod securityContext

  Setting fsGroup causes the kubelet to chmod the volume to group‑read/write (mode 770) and to change the group ownership to the specified GID. This is the recommended solution in the Kubernetes docs.

  
  # Before (problematic manifest)
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: transformers-blue
  spec:
    template:
      spec:
        securityContext:
          runAsUser: 1001
        containers:
        - name: app
          image: myrepo/transformers:latest
          volumeMounts:
          - name: model
            mountPath: /model
        volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
  

  
  # After – add fsGroup (e.g., 2000) and optionally align runAsGroup
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: transformers-blue
  spec:
    template:
      spec:
        securityContext:
          runAsUser: 1001
          runAsGroup: 2000
          fsGroup: 2000          # <-- added
        containers:
        - name: app
          image: myrepo/transformers:latest
          env:
          - name: HF_HOME
            value: /model          # optional, per Transformers docs
          volumeMounts:
          - name: model
            mountPath: /model
        volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
  

  Why it works: The kubelet recursively changes the group ownership of /model to 2000 and sets the group read/write bits, allowing the non‑root process (UID 1001, GID 2000) to access the files.

  Approach B – Init container that chown the mount point

  If the CSI driver enforces 0700 and fsGroup cannot be used (e.g., due to restricted PSP), an init container can perform the ownership change before the main container starts.

  
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: transformers-blue
  spec:
    template:
      spec:
        initContainers:
        - name: permission-fix
          image: busybox
          command: ["sh", "-c", "chown -R 1001:1001 /model"]
          securityContext:
            runAsUser: 0          # run as root to change ownership
          volumeMounts:
          - name: model
            mountPath: /model
        containers:
        - name: app
          image: myrepo/transformers:latest
          securityContext:
            runAsUser: 1001
          env:
          - name: HF_HOME
            value: /model
          volumeMounts:
          - name: model
            mountPath: /model
        volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
  

  Why it works: The init container runs as root, recursively changes ownership to the UID/GID used by the application, and exits. The main container then sees the files with proper permissions.

  Verification

  1. Redeploy the corrected manifest:

     
     kubectl apply -f deployment-blue.yaml
     

  2. Confirm the pod reaches Running without FailedMount events:

     
     kubectl get pod -l app=transformers,track=blue
     kubectl describe pod -l app=transformers,track=blue | grep -i mount
     

  3. Check file accessibility from inside the pod:

     
     kubectl exec -it $(kubectl get pod -l app=transformers,track=blue -o jsonpath="{.items[0].metadata.name}") -- ls -l /model
     # Expected output:
     drwxrwxr-x 2 1001 1001 4096 Jan 10 12:00 .
     

  4. Validate that transformers can load the model:

     
     kubectl logs $(kubectl get pod -l app=transformers,track=blue -o jsonpath="{.items[0].metadata.name}") | grep "Model loaded"
     # Should contain something like:
     [INFO] Loading model from /model
     [INFO] Model loaded successfully
     

  5. Run a health‑check endpoint (if exposed) or issue a curl request to the inference API to ensure end‑to‑end functionality.

  Prevention and Best Practices

  • Standardize securityContext across all revisions of a blue‑green pair. Keep runAsUser, runAsGroup, and fsGroup identical.
  • Prefer fsGroup over init‑containers when the CSI driver supports it; it is declarative and less error‑prone.
  • Document the expected UID/GID in the deployment repository and enforce via CI linting (e.g., kubeval or kube-score checks).
  • Monitor PVC mount events with alerts on FailedMount or PermissionDenied messages in kubelet logs.
  • Set TRANSFORMERS_CACHE or HF_HOME to the mounted path as recommended by the Hugging Face docs, ensuring the library looks in the correct directory.
  • Use ReadWriteMany access mode for shared model volumes and verify the underlying storage class respects the mode.

  Related Topic Hub: Model Serving Troubleshooting Hub

  FAQ

  1. Why does the green deployment continue to work while the blue one fails?
     The green pods were created when the PVC files were owned by root and the containers also ran as root. The blue pods run as a non‑root user, so the same file ownership now blocks access.
  2. Can I keep the container image as non‑root and avoid using fsGroup?
     Yes, but you must ensure the volume is provisioned with the correct permissions. An init container that chowns the mount point is a viable alternative when fsGroup is unavailable.
  3. What if the CSI driver forces mode 0700 and I cannot change it?
     Deploy an init container that runs as root to recursively chmod 755 or chown the directory before the application starts.
  4. Do I need to set HF_HOME or TRANSFORMERS_CACHE for the model to be found?
     When loading from a custom path, the Transformers library respects the explicit path passed to from_pretrained. Setting HF_HOME or TRANSFORMERS_CACHE to the PVC mount point is recommended for any auxiliary files (tokenizers, config) that the library may download.
  5. How can I detect this issue before a rollout?
     Add a pre‑deployment health check that runs ls -l /model and verifies read access for the configured UID/GID. Automate the check in your CI/CD pipeline.