Haystack ConfigMap Mount Error In Multi-region Kubernetes Deployment

Q: 2. Inconsistent mountPath Between Helm Chart and Container Image

The Helm chart used a relative mountPath (/app/config) while the Haystack Docker image expects an absolute path (/etc/haystack/config). Regions that built the image from a different base (e.g., Alpine vs. Debian) have different default working directories, causing the file‑not‑found error only in those regions. This mirrors the real incident:

Q: 3. Secret Rotation with Stale Key References

A secret rotation introduced a new key name (HAYSTACK_API_KEY) while the pod spec still referenced the old key (API_KEY). The kubelet reported “key not found in secret” and the init container exited with code 1. This is documented in the community issue about projected volumes across clusters.

Q: Investigation and Debugging Steps Step 1 – Verify ConfigMap Presence in Each Cluster

# List ConfigMaps in the expected namespace kubectl get configmap -n haystack-config # Example output (us-east-1) NAME DATA AGE haystack-config 2 12d # Example output (eu-west-1) No resources found in haystack-config namespace. If the ConfigMap is missing, the mount will always fail.

Q: Step 2 – Inspect the Pod Spec for Volume Definitions

kubectl get pod haystack-api-xxxx -o yaml | grep -A5 volumeMounts ... volumeMounts: - name: config mountPath: /app/config #

Q: Step 4 – Examine Secret Keys

kubectl get secret haystack-secret -n haystack-config -o yaml # Look for data keys data: HAYSTACK_API_KEY: # API_KEY is missing after rotation Step 5 – Capture Kubelet Events kubectl describe pod haystack-api-xxxx | grep -i mount Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 2m kubelet, node-1 MountVolume.SetUp failed for volume "config" : hostPath type check failed: path /etc/haystack does not exist Resolution – Align ConfigMap, Secret, and Mount Paths Across Regions 1. Replicate ConfigMap to All Target Clusters Use a CI/CD pipeline or kubectl apply --context to push the same ConfigMap to every region.

Q: 2. Standardize mountPath to the Image‑Expected Location

Update the Helm values file (values.yaml) to use the absolute path.

Q: 4. Verify Volume Type Compatibility

If the pod uses a hostPath volume (as hinted by the “hostPath type check failed” message), replace it with a proper ConfigMap volume. The hostPath approach only works when the node already contains the directory, which is not guaranteed across regions.

In this article

Table of Contents

Problem – ConfigMap/Secret Mount Errors in a Multi‑Region Haystack Deployment

In a production Haystack deployment spanning multiple AWS regions, pods repeatedly fail during initialization with messages such as:


Error: "failed to sync config: open /etc/haystack/config.yaml: no such file or directory"
MountVolume.SetUp failed for volume "config" : hostPath type check failed: path /etc/haystack does not exist
Kubelet warning: "FailedMount for pod \"haystack-api-xxxx\" volume \"config\" : mount failed: not a directory"
Init container exit code 1 with message "cannot read config from /app/config/config.yaml: permission denied"

The failure is isolated to certain regions (e.g., eu‑west‑1) while the same Helm chart works in us‑east‑1. The root symptom is that the Haystack components cannot locate or read their configuration files, leading to startup crashes and inconsistent behavior across regions.

Root Cause Analysis

1. Namespace and Cluster Scope Mismatch

Haystack’s central configuration lives in a dedicated Kubernetes cluster (the “config hub”). In the incident, the ConfigMap was created only in the us-east-1 cluster under namespace haystack-config. Pods in the eu-west-1 cluster referenced the same namespace, but because ConfigMaps are namespaced and cluster‑scoped, the objects simply did not exist there. This aligns with the community report:

“ConfigMap mount path mismatch causes pod init failure in multi‑region setup” (GitHub #1245).

2. Inconsistent `mountPath` Between Helm Chart and Container Image

The Helm chart used a relative mountPath (/app/config) while the Haystack Docker image expects an absolute path (/etc/haystack/config). Regions that built the image from a different base (e.g., Alpine vs. Debian) have different default working directories, causing the file‑not‑found error only in those regions. This mirrors the real incident:

“Helm chart used a relative mountPath (e.g., /app/config) but the container image expected an absolute path (/etc/haystack/config), causing FileNotFound errors only in regions with different base images.”

3. Secret Rotation with Stale Key References

A secret rotation introduced a new key name (HAYSTACK_API_KEY) while the pod spec still referenced the old key (API_KEY). The kubelet reported “key not found in secret” and the init container exited with code 1. This is documented in the community issue about projected volumes across clusters.

Investigation and Debugging Steps

Step 1 – Verify ConfigMap Presence in Each Cluster


# List ConfigMaps in the expected namespace
kubectl get configmap -n haystack-config

# Example output (us-east-1)
NAME               DATA   AGE
haystack-config    2      12d

# Example output (eu-west-1)
No resources found in haystack-config namespace.

If the ConfigMap is missing, the mount will always fail.

Step 2 – Inspect the Pod Spec for Volume Definitions


kubectl get pod haystack-api-xxxx -o yaml | grep -A5 volumeMounts
...
volumeMounts:
- name: config
  mountPath: /app/config   # <-- chart value
  readOnly: true
...
volumes:
- name: config
  configMap:
    name: haystack-config
    items:
    - key: config.yaml
      path: config.yaml

Step 3 – Correlate Container Image Expectations

Check the image’s entrypoint or documentation for the expected config location:


docker run --rm deepset/haystack:latest cat /etc/haystack/README.md
# The application looks for /etc/haystack/config.yaml

Step 4 – Examine Secret Keys


kubectl get secret haystack-secret -n haystack-config -o yaml
# Look for data keys
data:
  HAYSTACK_API_KEY: 
  # API_KEY is missing after rotation

Step 5 – Capture Kubelet Events


kubectl describe pod haystack-api-xxxx | grep -i mount
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Warning FailedMount      2m    kubelet, node-1    MountVolume.SetUp failed for volume "config" : hostPath type check failed: path /etc/haystack does not exist

Resolution – Align ConfigMap, Secret, and Mount Paths Across Regions

1. Replicate ConfigMap to All Target Clusters

Use a CI/CD pipeline or kubectl apply --context to push the same ConfigMap to every region.


# Export from source cluster
kubectl --context us-east-1 get configmap haystack-config -n haystack-config -o yaml > cm.yaml

# Apply to destination clusters
kubectl --context eu-west-1 apply -f cm.yaml
kubectl --context ap-southeast-2 apply -f cm.yaml

2. Standardize `mountPath` to the Image‑Expected Location

Update the Helm values file (values.yaml) to use the absolute path.


# Before (values.yaml)
config:
  mountPath: /app/config

# After
config:
  mountPath: /etc/haystack

Regenerate the chart and redeploy:


helm upgrade --install haystack ./haystack-chart -f values.yaml --namespace haystack

3. Align Secret Key Names with Application Expectations

Either rename the key in the secret or adjust the environment variable mapping.


# Option A – Rename secret key
kubectl -n haystack-config create secret generic haystack-secret \
  --from-literal=API_KEY=$(kubectl -n haystack-config get secret haystack-secret -o jsonpath="{.data.HAYSTACK_API_KEY}" | base64 --decode) \
  --dry-run=client -o yaml | kubectl apply -f -

# Option B – Update envFrom in pod spec
envFrom:
- secretRef:
    name: haystack-secret
    optional: false
# and reference HAYSTACK_API_KEY directly in the container

4. Verify Volume Type Compatibility

If the pod uses a hostPath volume (as hinted by the “hostPath type check failed” message), replace it with a proper ConfigMap volume. The hostPath approach only works when the node already contains the directory, which is not guaranteed across regions.


# Bad (hostPath)
volumes:
- name: config
  hostPath:
    path: /etc/haystack
    type: Directory

# Good (ConfigMap)
volumes:
- name: config
  configMap:
    name: haystack-config
    items:
    - key: config.yaml
      path: config.yaml

Verification – Confirm That Pods Start Successfully and Read Configuration

1. Pod Status


kubectl get pods -n haystack -l app=haystack-api
NAME                READY   STATUS    RESTARTS   AGE
haystack-api-1      1/1     Running   0          2m

2. Log Inspection


kubectl logs -n haystack haystack-api-1
2026-06-06 10:12:34 INFO  haystack.core.config Loading configuration from /etc/haystack/config.yaml
2026-06-06 10:12:34 INFO  haystack.core.startup Haystack API ready on port 8000

3. File Presence Inside the Container


kubectl exec -n haystack haystack-api-1 -- ls -l /etc/haystack
total 4
-rw-r--r-- 1 root root 1234 Jun  6 10:12 config.yaml

4. Cross‑Region Consistency Check

Run the same commands in each region’s namespace; all should report Running and a valid config file.

Prevention – Guardrails for Multi‑Region Haystack Deployments

ConfigMap Synchronization Pipeline: Automate propagation of ConfigMaps and Secrets using GitOps (ArgoCD, Flux) with syncWave to ensure every cluster receives identical objects.
Helm Value Validation: Add a pre‑install hook that asserts the mountPath matches the image’s expected path (e.g., using helm test with a temporary pod that reads /etc/haystack/README.md).
Secret Versioning: Store secret keys in a versioned map (e.g., HAYSTACK_API_KEY_V1, HAYSTACK_API_KEY_V2) and reference the version via an environment variable, reducing breakage on rotation.
Namespace Consistency Checks: Include a CI step that runs kubectl get configmap -n haystack-config against each target context and fails if any cluster reports missing objects.
Monitoring Alerts: Create Prometheus alerts on KubePodNotReady and on the metric container_fs_errors_total for the Haystack namespace, with a notification threshold of 1 minute.

FAQ – Common Follow‑Up Questions

Why does the ConfigMap mount succeed in one region but not another? ConfigMaps are namespaced and cluster‑scoped. If the object exists only in the source cluster, pods in other clusters will see a “mount path not found” error.
Can I use a single ConfigMap across clusters with a federation? Federation can expose ConfigMaps, but the projected volume implementation still requires the object to be present in the target cluster’s namespace. A sync pipeline is usually simpler.
What is the difference between mountPath and the file path inside the container? mountPath is the directory where Kubernetes places the volume. The application reads files relative to that directory. If the container expects /etc/haystack/config.yaml but the volume is mounted at /app/config, the file will not be found.
How do I troubleshoot “permission denied” errors on ConfigMap files? Verify the securityContext of the pod (runAsUser, fsGroup) and the file mode of the ConfigMap volume (default 0644). Adjust fsGroup or use an initContainer to chmod the files if the image runs as a non‑root user.
After rotating a secret, why do some pods still reference the old key? Pods that were created before the rotation keep the old environment variable mapping. A rolling restart (or kubectl rollout restart deployment/haystack) forces the pods to read the updated secret.

Related Topic Hub: RAG Systems Troubleshooting Hub