Problem
A containerized GPT‑4 inference service is deployed on an air‑gapped server. The model checkpoint files are stored on the host filesystem at /opt/gpt4/models and are expected to be bind‑mounted into the container at /app/models. The container fails to start with the following error:
docker: Error response from daemon: mount /opt/gpt4/models:/app/models, type=bind, source=/opt/gpt4/models,target=/app/models,readonly: permission denied
Additional log excerpts from the container runtime:
2026-11-19T12:03:41.112Z time="2026-11-19T12:03:41Z" level=error msg="failed to create mount point: permission denied"
2026-11-19T12:03:41.113Z time="2026-11-19T12:03:41Z" level=error msg="container start failed: OCI runtime create failed: runc create failed: unable to start container process: exec: \"python\": permission denied"
The failure prevents the inference service from loading the required model weights, causing the entire deployment to be non‑functional.
Root Cause
The permission denial originates from a combination of Linux security mechanisms that are stricter on air‑gapped, hardened hosts:
- Filesystem ACLs and SELinux/AppArmor profiles – The host directory
/opt/gpt4/modelsis owned byroot:rootwith mode0750. The container runtime runs as an unprivileged user (UID 1000) and is confined by anapparmorprofile that deniesmountof directories not explicitly allowed. - Docker daemon’s
userns-remapconfiguration – The daemon is configured to map the host root to UID 165536 inside containers. Consequently, the container sees the host directory as owned by UID 165536, which does not have read permissions. - Read‑only root filesystem in the container image – The image is built with
VOLUME /app/modelsandRUN chmod 0755 /app/models, but the bind mount overrides these permissions, and the runtime enforces the host’s ACLs.
Because the host directory is not explicitly labeled for the container runtime and the UID mapping does not grant access, the kernel returns EACCES during the bind‑mount operation, resulting in the observed “permission denied” error.
Debugging Steps
- Confirm host directory permissions and ownership
ls -ld /opt/gpt4/models drwxr-x--- 2 root root 4096 Nov 19 11:58 /opt/gpt4/models - Check Docker daemon user namespace remapping
docker info | grep -i userns UsernsRoot: 165536 UsernsRemap: default - Inspect AppArmor or SELinux denials
journalctl -k | grep -i apparmor type=1400 audit(2026-11-19 12:04:02.123:456): apparmor="DENIED" operation="mount" profile="/usr/bin/docker-runc" name="/opt/gpt4/models" pid=12345 comm="runc" - Attempt a manual bind mount as the container runtime user
sudo -u 165536 mount --bind /opt/gpt4/models /tmp/testmount mount: permission denied (are you root?) - Verify the effective UID/GID inside a test container
docker run --rm -it --entrypoint /bin/sh \ -v /opt/gpt4/models:/app/models \ gpt4-inference:latest -c 'id && ls -l /app/models' uid=165536(gpt4) gid=165536(gpt4) groups=165536(gpt4) ls: cannot access '/app/models': Permission denied
Solution
The fix consists of aligning host directory permissions with the container’s user namespace and relaxing the security profile for the specific bind mount.
Step 1 – Adjust host directory ownership for the remapped UID
# Before
drwxr-x--- 2 root root 4096 Nov 19 11:58 /opt/gpt4/models
# Change ownership to the remapped UID/GID (165536)
sudo chown -R 165536:165536 /opt/gpt4/models
sudo chmod -R 0755 /opt/gpt4/models
Step 2 – Update the Docker daemon to disable user‑namespace remapping for this service (optional)
If the environment policy permits, create a daemon override that disables userns-remap for the GPT‑4 service only:
# /etc/docker/daemon.json
{
"default-runtime": "runc",
"runtimes": {
"gpt4": {
"path": "runc",
"runtimeArgs": ["--userns", "host"]
}
}
}
sudo systemctl restart docker
Then launch the container with the custom runtime:
docker run -d --runtime=gpt4 \
-v /opt/gpt4/models:/app/models:ro \
--name gpt4-inference \
gpt4-inference:latest
Step 3 – Relax the AppArmor profile for the mount point
Create a local profile that allows bind mounts from /opt/gpt4/models:
# /etc/apparmor.d/local/docker
profile docker-default flags=(attach_disconnected) {
# Existing rules …
mount options=(bind) /opt/gpt4/models/** rw,
}
sudo apparmor_parser -r /etc/apparmor.d/docker
Step 4 – Verify the container starts successfully
docker logs gpt4-inference
2026-11-19T12:15:02.001Z INFO Loading model checkpoint from /app/models/checkpoint.pt
2026-11-19T12:15:04.123Z INFO Model loaded successfully, ready to serve requests
Verification
- Container health check
curl -s http://localhost:8080/healthz {"status":"ok"} - Filesystem access inside the container
docker exec -it gpt4-inference ls -l /app/models -rw-r--r-- 1 gpt4 gpt4 2.3G Nov 19 12:14 checkpoint.pt - Metrics – Confirm that the
model_load_success_totalPrometheus counter increments:# curl -s http://localhost:9090/api/v1/query?query=model_load_success_total {"status":"success","data":{"result":[{"metric":{"service":"gpt4-inference"},"value":[1700417704,"1"]}]}}
Prevention
| Practice | Why it helps |
|---|---|
Standardize host directory ownership to match Docker’s userns-remap UID range |
Eliminates mismatched UID/GID that cause bind‑mount denials. |
Include explicit :ro or :rw mount flags in deployment manifests |
Clarifies intent and prevents accidental read‑only defaults. |
| Maintain a minimal AppArmor/SELinux profile per service | Reduces the surface area for accidental denials while preserving isolation. |
| Automate permission validation in CI/CD pipelines | Detects permission mismatches before code reaches production. |
| Document required host directory layout and ACLs in runbooks | Ensures on‑boarding engineers apply the correct ownership without guesswork. |
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why does the error appear only on air‑gapped servers?
Air‑gapped hosts are often hardened with stricter SELinux/AppArmor policies and user‑namespace remapping enabled by default. Non‑air‑gapped environments may run Docker without these constraints, masking the permission issue.
- Can I keep
userns-remapenabled and still mount the directory?Yes. Change the directory ownership to the remapped UID/GID (e.g.,
chown -R 165536:165536 /opt/gpt4/models) and ensure the mode grants at least read access (0755). - Is it safe to disable AppArmor for the GPT‑4 service?
Disabling the entire profile reduces isolation. Instead, extend the existing profile to allow the specific bind mount, as shown in Step 3.
- Do I need to rebuild the container image after fixing host permissions?
No. The issue is purely at the host‑to‑container mount layer. Rebuilding the image is unnecessary unless you change the internal
VOLUMEdeclaration. - How can I automate the permission fix for new servers?
Include a provisioning script (e.g., Ansible, Chef) that creates the model directory, sets ownership to the Docker remapped UID, and applies the required AppArmor snippet. Run the script as part of the server hardening playbook.