Server rack with blinking green lights

HAProxy deployment fails: ConfigMap mount error after Helm upgrade

by

Problem Description

After a Helm upgrade of the haproxy-ingress chart, the HAProxy pods enter CrashLoopBackOff. The container logs show errors such as:


Error opening configuration file /etc/haproxy/haproxy.cfg: No such file or directory
MountVolume.SetUp failed for volume "haproxy-config": configmap "haproxy-config" not found
cannot open /etc/haproxy/haproxy.cfg: permission denied
failed to reload haproxy: configuration file is not readable

Consequently, traffic to all AI model inference services returns HTTP 503, breaking the inference pipeline.

Root Cause Analysis

The failure stems from the way Helm handles ConfigMap and Secret resources during an upgrade:

  • Helm creates a new ConfigMap (or Secret) with a different metadata.resourceVersion but does not delete the previous object if the name is unchanged (Helm chart hooks).
  • Kubernetes mounts the volume based on the uid of the original object. When the pod is not restarted, the mount continues to reference the stale object, which may have been garbage‑collected or left empty (ConfigMaps as Volumes).
  • If the new ConfigMap contains a different file name or uses subPath incorrectly, the mount can overwrite /etc/haproxy/haproxy.cfg with an empty directory, leading to “permission denied” or “file not found” errors (Secrets as Volumes).
  • In multi‑namespace deployments, concurrent upgrades can race, causing a pod to see a partially created ConfigMap. HAProxy then attempts to parse an incomplete haproxy.cfg and exits (GitHub issue kubernetes/kubernetes#101123).

In short, the upgrade leaves HAProxy with a volume mount that either points to a non‑existent ConfigMap/Secret or to a file with incorrect permissions, preventing HAProxy from loading its configuration.

Investigation and Debugging

1. Inspect pod events and volume status


kubectl describe pod haproxy-xxxx -n ingress

Typical output:


Events:
  Type     Reason                Age   From               Message
  ----     ------                ----  ----               -------
  Warning  FailedMount           2m    kubelet, node-1    MountVolume.SetUp failed for volume "haproxy-config": configmap "haproxy-config" not found

2. Verify the existence and content of the ConfigMap/Secret


kubectl get configmap haproxy-config -n ingress -o yaml
kubectl get secret haproxy-tls -n ingress -o yaml

If the ConfigMap is missing or its data section is empty, the mount will fail.

3. Check the pod’s volume mount paths


kubectl exec -it haproxy-xxxx -n ingress -- ls -l /etc/haproxy

Expected output (when mount succeeds):


-rw-r--r-- 1 root root  1234 Sep 5 12:00 haproxy.cfg
drwxr-xr-x 2 root root   4096 Sep 5 12:00 ssl

Observed output in failing pods often shows an empty directory or missing file.

4. Review Helm release history


helm history haproxy-ingress -n ingress
helm get manifest haproxy-ingress -n ingress --revision

Look for changes to the ConfigMap name, subPath usage, or added hooks that might affect rollout order.

5. Capture a short pod lifecycle trace


kubectl logs haproxy-xxxx -n ingress --previous

Logs typically contain the exact error strings listed in the Common errors evidence.

Resolution

1. Ensure immutable ConfigMap naming

Append a hash to the ConfigMap name so Helm creates a new object on every change, forcing a pod restart.


# values.yaml
configMap:
  name: haproxy-config-{{ .Release.Revision }}
  data:
    haproxy.cfg: |
      {{ .Files.Get "files/haproxy.cfg" }}

2. Add a pre-upgrade hook to delete the old ConfigMap


apiVersion: batch/v1
kind: Job
metadata:
  name: haproxy-config-cleanup
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: cleanup
        image: bitnami/kubectl:latest
        command: ["kubectl", "delete", "configmap", "haproxy-config", "-n", "{{ .Release.Namespace }}"]

3. Use subPath correctly to avoid overwriting the entire directory

Incorrect:


volumeMounts:
- name: haproxy-config
  mountPath: /etc/haproxy

Correct (mount only the file):


volumeMounts:
- name: haproxy-config
  mountPath: /etc/haproxy/haproxy.cfg
  subPath: haproxy.cfg

4. Enforce proper file permissions on Secrets

Kubernetes mounts Secrets with mode 0400 by default, which HAProxy can read as root. If the container runs as a non‑root user, set defaultMode to 0444 or run HAProxy as root.


apiVersion: v1
kind: Secret
metadata:
  name: haproxy-tls
type: Opaque
data:
  tls.crt: 
  tls.key: 
---
volumeMounts:
- name: haproxy-tls
  mountPath: /etc/haproxy/ssl
  readOnly: true
volumes:
- name: haproxy-tls
  secret:
    secretName: haproxy-tls
    defaultMode: 0444

5. Force a rolling restart after upgrade


kubectl rollout restart deployment haproxy -n ingress

This guarantees that pods pick up the newly created ConfigMap/Secret.

Validation

1. Verify pod health


kubectl get pods -n ingress -l app=haproxy
kubectl describe pod haproxy-xxxx -n ingress | grep -i "MountVolume"

All pods should be in Running state with no FailedMount events.

2. Check HAProxy logs for successful config load


kubectl logs haproxy-xxxx -n ingress | grep "HAProxy version"
kubectl logs haproxy-xxxx -n ingress | grep "configuration file"

Expected lines:


[INFO] HAProxy version 2.6.0-1ubuntu1 compiled on ...
[INFO] Configuration file /etc/haproxy/haproxy.cfg loaded successfully

3. Perform an end‑to‑end request to a model endpoint


curl -s -o /dev/null -w "%{http_code}" http://model-ingress.example.com/v1/predict

HTTP 200 indicates traffic is correctly routed through HAProxy to the inference pod.

Prevention and Best Practices

  • Immutable ConfigMap/Secret names: Include the Helm revision or a hash in the resource name to avoid stale volume mounts.
  • Chart hooks for cleanup: Use pre-upgrade or post-upgrade hooks to delete or recreate dependent resources.
  • Explicit subPath mounts: Prevent whole‑directory overwrites that can erase existing files.
  • Permission alignment: Ensure the HAProxy container user can read mounted Secrets, or configure defaultMode accordingly.
  • Readiness probe on config load: Define a probe that checks haproxy -c -f /etc/haproxy/haproxy.cfg so Kubernetes will not mark the pod ready until the config is valid.
  • Monitoring: Alert on CrashLoopBackOff for the HAProxy deployment and on events with reason FailedMount.
  • Rollback safety: Keep previous ConfigMap versions in a separate namespace or as a backup ConfigMap to avoid loss during rollbacks.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the HAProxy pod work after a fresh install but fail after a Helm upgrade?

    The initial install creates the ConfigMap and mounts it correctly. During upgrade Helm may reuse the same ConfigMap name, leaving the pod attached to a stale volume that gets deleted or overwritten, causing mount failures.

  2. Can I use helm upgrade --reuse-values to avoid this issue?

    Reusing values does not affect ConfigMap naming. The problem is the resource lifecycle, not the values. Use immutable names or cleanup hooks instead.

  3. How do I know if a secret permission issue is the cause?

    Check pod logs for “permission denied while trying to access /etc/haproxy/ssl/tls.crt”. Also inspect the secret volume mode with kubectl get pod -o yaml and verify defaultMode is set to a readable value for the HAProxy user.

  4. Is it safe to run HAProxy as root to bypass secret permission errors?

    Running as root removes the permission barrier but violates the principle of least privilege. Prefer adjusting defaultMode or running HAProxy with the appropriate user ID.

  5. What metric should I monitor to detect a ConfigMap mount failure early?

    Watch the kube_pod_status_phase{phase="CrashLoopBackOff"} metric and the kubelet_volume_mount_errors_total counter for the haproxy-config volume.