PostgreSQL init fails with empty /etc/postgresql/conf.d in Kubernetes

Problem Description

During development of an AI training pipeline on Docker Desktop’s integrated Kubernetes cluster, the PostgreSQL pod that stores experiment metadata and feature‑store data fails to start. The pod repeatedly enters CrashLoopBackOff and the container logs contain errors such as:

2024-07-15 10:12:34.567 UTC [1] LOG:  could not open configuration file "/etc/postgresql/conf.d/custom.conf": Permission denied
2024-07-15 10:12:34.568 UTC [1] FATAL:  could not open configuration file "/run/secrets/pg_hba.conf": No such file or directory

Inspection of the filesystem inside the container shows that both /etc/postgresql/conf.d and /run/secrets are either empty or contain files with mode 0000. The failure blocks the downstream MLflow/feature‑store services, preventing any experiment metadata from being persisted.

Root Cause Analysis

The PostgreSQL Docker image expects configuration fragments in /etc/postgresql/conf.d and secret‑based files (e.g., pg_hba.conf) in /run/secrets. Kubernetes mounts ConfigMaps and Secrets as files, but several interactions cause the observed failures:

• Default file mode: When a ConfigMap is mounted without an explicit defaultMode, the API server creates files with mode 0644. However, on Docker Desktop (macOS/Windows) the underlying file‑sharing layer can reset permissions to 0000, leading to “Permission denied”.
• Missing fsGroup: The PostgreSQL container runs as the postgres user (UID 999). Files created by the kubelet belong to root. Without a securityContext.fsGroup, the group ownership is not changed, so the postgres user cannot read the mounted files.
• Docker Desktop hostPath limitation: The /run/secrets directory is often backed by a hostPath volume for local debugging. Docker Desktop does not support arbitrary hostPath mounts on macOS/Windows, resulting in an empty directory at runtime.
• SELinux/AppArmor context: On Linux hosts, the default security context may label the volume with root:root and a restrictive SELinux type, preventing the container from accessing the files.

These factors combine to produce empty or unreadable configuration directories, which PostgreSQL treats as missing configuration files, causing the startup failure documented in the official PostgreSQL runtime‑config file locations (PostgreSQL Documentation).

Investigation and Debugging

Below is a reproducible debugging workflow that isolates the problem.

1. Inspect pod description to verify volume mounts and security context.

kubectl describe pod pg-primary-0 -n ml-pipeline

Key fields to look for:

• Volumes: ensure a configMap entry for conf.d and a secret entry for pg_hba.conf.
• Security Context: check for runAsUser, runAsGroup, and fsGroup.

2. Exec into the container before PostgreSQL starts (use an initContainer or sleep entrypoint).

kubectl exec -it pg-primary-0 -n ml-pipeline -- /bin/bash

Run:

ls -l /etc/postgresql/conf.d
ls -l /run/secrets
stat -c "%a %U %G" /etc/postgresql/conf.d/custom.conf

Typical output from a failing pod:

-rw------- 0 root root 0 Jan 01 00:00 custom.conf
total 0

3. Check pod events for mount errors.

kubectl get events -n ml-pipeline --field-selector involvedObject.name=pg-primary-0

Look for messages such as “Error mounting volume: permission denied”.

4. Validate Docker Desktop file‑sharing settings.

On macOS/Windows, open Docker Desktop → Settings → Resources → File Sharing and confirm that the directory containing the secret files is shared. Missing entries cause the secret volume to be empty, as reported in the Docker Desktop documentation (Docker Desktop Kubernetes).

Resolution

The fix consists of three orthogonal adjustments:

1. Explicitly set file mode and group ownership

Add defaultMode: 0644 to the ConfigMap volume and define fsGroup: 999 (PostgreSQL’s primary group) in the pod’s securityContext. This ensures the postgres user can read the files.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pg-primary
spec:
  serviceName: pg
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999            # <-- added
      containers:
      - name: postgres
        image: postgres:15
        volumeMounts:
        - name: conf-d
          mountPath: /etc/postgresql/conf.d
        - name: pg-secret
          mountPath: /run/secrets
      volumes:
      - name: conf-d
        configMap:
          name: pg-conf-d
          defaultMode: 0644   # <-- added
      - name: pg-secret
        secret:
          secretName: pg-secret
          defaultMode: 0644   # <-- added

2. Use an initContainer to copy files with proper ownership

If the platform still enforces restrictive permissions (e.g., Docker Desktop on macOS), copy the files into an emptyDir volume owned by postgres before the main container starts.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pg-primary
spec:
  template:
    spec:
      initContainers:
      - name: init-conf
        image: busybox
        command: ["/bin/sh", "-c"]
        args:
        - |
          cp /src/* /dest/ && \
          chown -R 999:999 /dest
        volumeMounts:
        - name: src-conf
          mountPath: /src
        - name: dest-conf
          mountPath: /dest
      containers:
      - name: postgres
        volumeMounts:
        - name: dest-conf
          mountPath: /etc/postgresql/conf.d
      volumes:
      - name: src-conf
        configMap:
          name: pg-conf-d
          defaultMode: 0644
      - name: dest-conf
        emptyDir: {}

3. Avoid hostPath for secrets on Docker Desktop

Replace any hostPath volume used for /run/secrets with a native Secret volume. Docker Desktop does not reliably mount hostPath on macOS/Windows, leading to the empty directory observed in the “real incidents” evidence.

# Bad (hostPath)
volumes:
- name: pg-secret
  hostPath:
    path: /Users/me/k8s/secrets/pg_hba.conf
    type: File

# Good (Secret)
volumes:
- name: pg-secret
  secret:
    secretName: pg-secret
    defaultMode: 0644

Validation

After applying the changes, verify the pod starts cleanly:

kubectl rollout restart statefulset/pg-primary -n ml-pipeline
kubectl get pod -n ml-pipeline -w

Expected log snippet:

2024-07-15 10:15:02.123 UTC [1] LOG:  database system was shut down at 2024-07-15 10:15:01 UTC
2024-07-15 10:15:02.124 UTC [1] LOG:  autovacuum launcher started
2024-07-15 10:15:02.125 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432

Additional checks:

• kubectl exec -it pg-primary-0 -n ml-pipeline -- ls -l /etc/postgresql/conf.d should list files with mode -rw-r--r-- and owner postgres:postgres.
• kubectl exec -it pg-primary-0 -n ml-pipeline -- cat /run/secrets/pg_hba.conf should output the expected HBA rules.
• Run a simple client connection:

psql -h localhost -U postgres -c "SELECT 1"

It should return 1 without errors.

Operational Experience

During the investigation we observed a few misleading cues:

• The pod status showed CrashLoopBackOff with no explicit “permission denied” in the first few log lines, leading some teams to suspect database corruption.
• Running kubectl logs on the initContainer revealed “copying files” succeeded, yet the main container still failed because the fsGroup was missing, a nuance highlighted in the GitHub issue postgres#1245.
• On macOS, the Docker Desktop file‑sharing UI silently dropped the secret directory from the mount, which manifested as an empty /run/secrets directory rather than a permission error.

Best Practices and Prevention

Practice	Why it matters
Declare defaultMode: 0644 for all ConfigMap/Secret volumes	Prevents files from being created with 0000 permissions on Docker Desktop.
Set securityContext.fsGroup to the PostgreSQL group (999)	Ensures the postgres user can read mounted files regardless of owner.
Avoid hostPath for secrets on local clusters	Docker Desktop does not reliably expose hostPath on macOS/Windows.
Use an initContainer to enforce ownership when needed	Works around platform‑specific permission quirks without altering the main image.
Monitor volume mount events	Pod events surface mount failures early, reducing MTTR.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the pod work on a Linux‑based cluster but fail on Docker Desktop?
   Docker Desktop’s file‑sharing layer can reset file permissions to 0000 and does not support arbitrary hostPath mounts, causing empty or unreadable ConfigMap/Secret files.
2. Do I need to set runAsUser and fsGroup together?
   Yes. runAsUser defines the UID the process runs as; fsGroup changes the group ownership of all volume files, allowing the process to read them.
3. Can I keep the default defaultMode and still avoid permission errors?
   Only if the underlying platform preserves the mode. On Docker Desktop you must explicitly set defaultMode: 0644 or copy files via an initContainer.
4. What if I need to mount a custom postgresql.conf?
   Mount the entire /etc/postgresql directory from a ConfigMap, set defaultMode: 0644, and ensure fsGroup matches the PostgreSQL group.
5. Is SELinux the cause of the “Permission denied” errors?
   On Linux hosts with SELinux enforcing, you may need to add securityContext.seLinuxOptions.type: spc_t or use the runAsNonRoot flag. However, the primary cause in Docker Desktop environments is the permission mode, not SELinux.