GPT-4 inference container volume mount permission denied on air-gapped server

Problem

A containerized GPT‑4 inference service is deployed on an air‑gapped server. The model checkpoint files are stored on the host filesystem at /opt/gpt4/models and are expected to be bind‑mounted into the container at /app/models. The container fails to start with the following error:

docker: Error response from daemon: mount /opt/gpt4/models:/app/models, type=bind, source=/opt/gpt4/models,target=/app/models,readonly: permission denied

Additional log excerpts from the container runtime:

2026-11-19T12:03:41.112Z time="2026-11-19T12:03:41Z" level=error msg="failed to create mount point: permission denied"
2026-11-19T12:03:41.113Z time="2026-11-19T12:03:41Z" level=error msg="container start failed: OCI runtime create failed: runc create failed: unable to start container process: exec: \"python\": permission denied"

The failure prevents the inference service from loading the required model weights, causing the entire deployment to be non‑functional.

Root Cause

The permission denial originates from a combination of Linux security mechanisms that are stricter on air‑gapped, hardened hosts:

  • Filesystem ACLs and SELinux/AppArmor profiles – The host directory /opt/gpt4/models is owned by root:root with mode 0750. The container runtime runs as an unprivileged user (UID 1000) and is confined by an apparmor profile that denies mount of directories not explicitly allowed.
  • Docker daemon’s userns-remap configuration – The daemon is configured to map the host root to UID 165536 inside containers. Consequently, the container sees the host directory as owned by UID 165536, which does not have read permissions.
  • Read‑only root filesystem in the container image – The image is built with VOLUME /app/models and RUN chmod 0755 /app/models, but the bind mount overrides these permissions, and the runtime enforces the host’s ACLs.

Because the host directory is not explicitly labeled for the container runtime and the UID mapping does not grant access, the kernel returns EACCES during the bind‑mount operation, resulting in the observed “permission denied” error.

Debugging Steps

  1. Confirm host directory permissions and ownership
    ls -ld /opt/gpt4/models
    drwxr-x--- 2 root root 4096 Nov 19 11:58 /opt/gpt4/models
    
  2. Check Docker daemon user namespace remapping
    docker info | grep -i userns
    UsernsRoot: 165536
    UsernsRemap: default
    
  3. Inspect AppArmor or SELinux denials
    journalctl -k | grep -i apparmor
    type=1400 audit(2026-11-19 12:04:02.123:456): apparmor="DENIED" operation="mount" profile="/usr/bin/docker-runc" name="/opt/gpt4/models" pid=12345 comm="runc"
    
  4. Attempt a manual bind mount as the container runtime user
    sudo -u 165536 mount --bind /opt/gpt4/models /tmp/testmount
    mount: permission denied (are you root?)
    
  5. Verify the effective UID/GID inside a test container
    docker run --rm -it --entrypoint /bin/sh \
      -v /opt/gpt4/models:/app/models \
      gpt4-inference:latest -c 'id && ls -l /app/models'
    uid=165536(gpt4) gid=165536(gpt4) groups=165536(gpt4)
    ls: cannot access '/app/models': Permission denied
    

Solution

The fix consists of aligning host directory permissions with the container’s user namespace and relaxing the security profile for the specific bind mount.

Step 1 – Adjust host directory ownership for the remapped UID

# Before
drwxr-x--- 2 root root 4096 Nov 19 11:58 /opt/gpt4/models

# Change ownership to the remapped UID/GID (165536)
sudo chown -R 165536:165536 /opt/gpt4/models
sudo chmod -R 0755 /opt/gpt4/models

Step 2 – Update the Docker daemon to disable user‑namespace remapping for this service (optional)

If the environment policy permits, create a daemon override that disables userns-remap for the GPT‑4 service only:

# /etc/docker/daemon.json
{
  "default-runtime": "runc",
  "runtimes": {
    "gpt4": {
      "path": "runc",
      "runtimeArgs": ["--userns", "host"]
    }
  }
}
sudo systemctl restart docker

Then launch the container with the custom runtime:

docker run -d --runtime=gpt4 \
  -v /opt/gpt4/models:/app/models:ro \
  --name gpt4-inference \
  gpt4-inference:latest

Step 3 – Relax the AppArmor profile for the mount point

Create a local profile that allows bind mounts from /opt/gpt4/models:

# /etc/apparmor.d/local/docker
profile docker-default flags=(attach_disconnected) {
    # Existing rules …
    mount options=(bind) /opt/gpt4/models/** rw,
}
sudo apparmor_parser -r /etc/apparmor.d/docker

Step 4 – Verify the container starts successfully

docker logs gpt4-inference
2026-11-19T12:15:02.001Z INFO Loading model checkpoint from /app/models/checkpoint.pt
2026-11-19T12:15:04.123Z INFO Model loaded successfully, ready to serve requests

Verification

  • Container health check
    curl -s http://localhost:8080/healthz
    {"status":"ok"}
    
  • Filesystem access inside the container
    docker exec -it gpt4-inference ls -l /app/models
    -rw-r--r-- 1 gpt4 gpt4 2.3G Nov 19 12:14 checkpoint.pt
    
  • Metrics – Confirm that the model_load_success_total Prometheus counter increments:
    # curl -s http://localhost:9090/api/v1/query?query=model_load_success_total
    {"status":"success","data":{"result":[{"metric":{"service":"gpt4-inference"},"value":[1700417704,"1"]}]}}
    

Prevention

Practice Why it helps
Standardize host directory ownership to match Docker’s userns-remap UID range Eliminates mismatched UID/GID that cause bind‑mount denials.
Include explicit :ro or :rw mount flags in deployment manifests Clarifies intent and prevents accidental read‑only defaults.
Maintain a minimal AppArmor/SELinux profile per service Reduces the surface area for accidental denials while preserving isolation.
Automate permission validation in CI/CD pipelines Detects permission mismatches before code reaches production.
Document required host directory layout and ACLs in runbooks Ensures on‑boarding engineers apply the correct ownership without guesswork.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the error appear only on air‑gapped servers?

    Air‑gapped hosts are often hardened with stricter SELinux/AppArmor policies and user‑namespace remapping enabled by default. Non‑air‑gapped environments may run Docker without these constraints, masking the permission issue.

  2. Can I keep userns-remap enabled and still mount the directory?

    Yes. Change the directory ownership to the remapped UID/GID (e.g., chown -R 165536:165536 /opt/gpt4/models) and ensure the mode grants at least read access (0755).

  3. Is it safe to disable AppArmor for the GPT‑4 service?

    Disabling the entire profile reduces isolation. Instead, extend the existing profile to allow the specific bind mount, as shown in Step 3.

  4. Do I need to rebuild the container image after fixing host permissions?

    No. The issue is purely at the host‑to‑container mount layer. Rebuilding the image is unnecessary unless you change the internal VOLUME declaration.

  5. How can I automate the permission fix for new servers?

    Include a provisioning script (e.g., Ansible, Chef) that creates the model directory, sets ownership to the Docker remapped UID, and applies the required AppArmor snippet. Run the script as part of the server hardening playbook.