Meta LLaMA container fails to mount volume on startup due to permission denied

Problem

A Meta LLaMA inference container is launched in a managed cloud service (Kubernetes or ECS‑style) with a persistent volume that stores model checkpoints. The container crashes during its entrypoint.sh initialization with a Permission denied error when attempting to stat or ls the mount point. As a result the model cannot be loaded and the service never becomes ready.

Typical symptom in pod logs:

2026-09-30T14:12:07.321Z ERROR: Failed to access checkpoint directory /mnt/model/checkpoint: Permission denied
Traceback (most recent call last):
  File "/app/entrypoint.py", line 42, in load_checkpoint
    os.listdir(CHECKPOINT_PATH)
PermissionError: [Errno 13] Permission denied: '/mnt/model/checkpoint'
2026-09-30T14:12:07.322Z INFO: Container exiting with code 1

In the orchestrator events the pod shows:

Warning  FailedMount  2m45s (x3 over 5m)  kubelet, node-1  MountVolume.SetUp failed for volume "model-pvc": permission denied while mounting volume

Root Cause

The failure is caused by a mismatch between the security context of the container and the file system permissions of the underlying persistent volume:

• The PVC is provisioned with a fsGroup of 1000 (the default for the storage class), which results in the volume being owned by uid:1000, gid:1000.
• The LLaMA container image runs as uid:2000 (non‑root) and does not belong to the fsGroup. Consequently the process has only read‑only access to the mount point.
• Additionally, the mountPath in the pod spec was set to /model/checkpoint while the application expects /mnt/model/checkpoint. The mismatch forces the container to create a bind‑mount that inherits the original permissions, compounding the denial.

Both issues combine to produce the Permission denied error during startup.

Debug

1. Inspect pod description

kubectl describe pod llama-inference-7c9f9d6b9c-xyz

Key fields to note:

Field	Observed Value
Security Context	runAsUser: 2000, runAsGroup: 2000
fsGroup	1000
Volumes	model-pvc (PersistentVolumeClaim)
Volume Mounts	mountPath: /model/checkpoint

2. Examine volume permissions on the node

ssh node-1
sudo ls -ld /var/lib/kubelet/pods/7c9f9d6b9c-xyz/volumes/kubernetes.io~csi/model-pvc
# Example output
drwxr-x--- 2 root 1000 4096 Sep 30 13:58 .

The directory is owned by root:1000 with group read/write only.

3. Verify container runtime user

kubectl exec -it llama-inference-7c9f9d6b9c-xyz -- id
uid=2000(gpuuser) gid=2000(gpuuser) groups=2000

4. Check application expected path

grep CHECKPOINT_PATH /app/entrypoint.py
# Returns: CHECKPOINT_PATH = "/mnt/model/checkpoint"

Solution

Option A – Align security context with volume group

Update the pod spec to use the same fsGroup as the storage class, allowing the container user to read/write the mounted directory.

# Before (snippet from deployment.yaml)
spec:
  securityContext:
    runAsUser: 2000
    runAsGroup: 2000
  # fsGroup omitted, defaults to storage class value (1000)

# After
spec:
  securityContext:
    runAsUser: 2000
    runAsGroup: 2000
    fsGroup: 2000        # <-- added

Setting fsGroup: 2000 causes the kubelet to chown the volume to gid=2000 before the container starts, granting the required permissions.

Option B – Run container as root (less preferred)

If changing fsGroup is not possible, modify the security context to run as root:

spec:
  securityContext:
    runAsUser: 0
    runAsGroup: 0

Root has unrestricted access, but this bypasses the principle of least privilege.

Option C – Correct mountPath to match application expectation

Update the volumeMounts entry so the path aligns with CHECKPOINT_PATH used by the code.

# Before
volumeMounts:
  - name: model-pvc
    mountPath: /model/checkpoint

# After
volumeMounts:
  - name: model-pvc
    mountPath: /mnt/model/checkpoint

Combined recommended fix

Apply both the corrected fsGroup and the proper mountPath to eliminate permission errors and path mismatches.

Verify

1. Redeploy the updated manifest:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/llama-inference

Confirm the pod reaches Ready state:

kubectl get pod -l app=llama-inference -w

Inspect the container logs for successful checkpoint load:

kubectl logs -f llama-inference-7c9f9d6b9c-xyz
2026-09-30T14:15:12.004Z INFO: Loading checkpoint from /mnt/model/checkpoint
2026-09-30T14:15:13.721Z INFO: Model loaded successfully, ready to serve requests

Validate file system ownership on the node after pod start:

sudo ls -ld /var/lib/kubelet/pods/7c9f9d6b9c-xyz/volumes/kubernetes.io~csi/model-pvc
drwxr-xr-x 2 root 2000 4096 Sep 30 14:15 .

Prevent

• Standardize mount paths: Store the expected checkpoint directory in a Helm value (e.g., checkpointPath) and reference it both in the container code and the pod spec.
• Enforce securityContext policies: Use a PodSecurityPolicy or OPA Gatekeeper rule that requires fsGroup to match runAsGroup for pods mounting PVCs.
• Automated permission checks: Add an init container that runs stat on the mount point and exits with a clear error if the UID/GID does not have read/write access.
• Monitoring alerts: Create a Prometheus alert on kube_pod_container_status_waiting_reason{reason="CreateContainerConfigError"} or on container logs containing Permission denied within the first 30 seconds of startup.
• Documentation sync: Keep the repository README up‑to‑date with the required fsGroup and mountPath values for each environment (dev, staging, prod).

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

• Why does the container work locally but fail in the managed service?
  Locally you likely run the image as root or with the host’s user ID, which bypasses the fsGroup mismatch. In the managed service the orchestrator enforces the security context defined in the pod spec.
• Can I keep the original mountPath and change the application code instead?
  Yes, updating CHECKPOINT_PATH in the entrypoint or using an environment variable to override the path is a viable alternative, but aligning the mountPath avoids code changes and keeps the container portable.
• What if the storage class does not support fsGroup changes?
  Some CSI drivers (e.g., certain block storage providers) ignore fsGroup. In that case you must either run the container as root or use an init container to chmod the volume after it is mounted.
• Is it safe to set runAsUser: 0 in production?
  Running as root removes the isolation benefits of non‑privileged containers and can expose the host to privilege escalation attacks. It should be a last resort and only used behind additional hardening (e.g., seccomp, AppArmor).
• How can I debug similar permission issues for other volumes?
  Use kubectl exec to run id and stat inside the container, and compare against the host’s directory ownership via ls -l. The kubectl describe pod output will also reveal the effective fsGroup.