LlamaIndex init fails with ConfigMap mount error in Kubernetes

Problem

LlamaIndex initialization fails inside a Kubernetes pod that is part of a data‑pipeline job. The failure manifests as an exception when the library attempts to read credential files or configuration values that are expected to be provided via a ConfigMap or Secret volume mount.

Typical error output:

Traceback (most recent call last):
  File "/app/pipeline.py", line 42, in <module>
    index = LlamaIndex.from_documents(docs, config_path="/etc/llama/config.yaml")
  File "/usr/local/lib/python3.11/site-packages/llama_index/__init__.py", line 128, in from_documents
    with open(config_path, "r") as cfg:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/llama/config.yaml'

In addition, the pod logs contain Kubernetes volume mount warnings:

Warning  FailedMount  kubelet, node-1  MountVolume.SetUp failed for volume "llama-config" : 
    configmap "llama-config" not found

Root Cause

The failure is caused by a mismatch between the ConfigMap/Secret definitions and the pod’s volume specification:

• Name mismatch: The manifest references llama-config but the actual ConfigMap is named llama-config-v2.
• Namespace drift: The ConfigMap lives in data-pipeline namespace while the pod runs in default. Kubernetes does not cross‑namespace mount ConfigMaps unless explicitly referenced with namespace field (available only in CSI drivers, not in native volume mounts).
• Incorrect mount path permissions: The pod runs as a non‑root user (UID 1001) and the mount point /etc/llama is owned by root with 0755 permissions, preventing the user from reading the files.

These three issues combine to make the file unavailable at runtime, leading LlamaIndex to raise FileNotFoundError during index initialization.

Debug

Step‑by‑step investigation that reproduced the problem:

1. Inspect pod description

kubectl describe pod llama-index-init-abc123 -n default

Key excerpt:

Volumes:
  llama-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      llama-config
    Optional:  false
Mounts:
  /etc/llama from llama-config (ro)

2. List ConfigMaps in the namespace

kubectl get configmap -n default

Output shows no object named llama-config:

No resources found in default namespace.

3. Check the intended ConfigMap

kubectl get configmap llama-config-v2 -n data-pipeline -o yaml

Configuration is present and contains the required config.yaml key.

4. Verify file permissions inside the container

kubectl exec -it llama-index-init-abc123 -n default -- sh -c "ls -l /etc/llama"

Result:

total 0
drwxr-xr-x    2 root     root          4096 Jan  1 00:00 .
drwxr-xr-x    1 root     root          4096 Jan  1 00:00 ..

The mount point exists but contains no files, confirming the mount failure.

Solution

Apply three corrective actions: correct the name, align namespaces, and set appropriate security context.

1. Align ConfigMap name and namespace

Update the pod (or Deployment) manifest to reference the existing ConfigMap and specify the correct namespace using a ConfigMap volume from the same namespace. If cross‑namespace sharing is required, create a duplicate ConfigMap in the pod’s namespace.

Before (faulty manifest snippet):

<!-- pod.yaml -->
apiVersion: v1
kind: Pod
metadata:
  name: llama-index-init
spec:
  containers:
  - name: init
    image: myrepo/llama-index:latest
    volumeMounts:
    - name: llama-config
      mountPath: /etc/llama
      readOnly: true
  volumes:
  - name: llama-config
    configMap:
      name: llama-config   # <-- wrong name, wrong namespace

After (corrected manifest):

<!-- pod.yaml -->
apiVersion: v1
kind: Pod
metadata:
  name: llama-index-init
  namespace: data-pipeline          # ensure pod runs in same namespace
spec:
  containers:
  - name: init
    image: myrepo/llama-index:latest
    securityContext:
      runAsUser: 1001               # non‑root user
      runAsGroup: 1001
    volumeMounts:
    - name: llama-config
      mountPath: /etc/llama
      readOnly: true
  volumes:
  - name: llama-config
    configMap:
      name: llama-config-v2        # correct ConfigMap name

2. Adjust file permissions via securityContext

Set fsGroup so that the mounted files are readable by the non‑root user.

<!-- add to pod spec -->
securityContext:
  runAsUser: 1001
  runAsGroup: 1001
  fsGroup: 1001   # Kubernetes will chown volume files to this GID

3. Redeploy and confirm

kubectl apply -f pod.yaml

Wait for the pod to become ready, then re‑run the diagnostic commands from the Debug section.

Verify

Confirm that the ConfigMap is correctly mounted and readable before LlamaIndex starts.

1. Check the mount contents:

kubectl exec -it llama-index-init-abc123 -n data-pipeline -- sh -c "cat /etc/llama/config.yaml"

Expected output (truncated):

api_key: "sk-********"
model: "meta-llama/7B"
index_path: "/data/index"

Run the pipeline with a dry‑run flag (if available) to ensure LlamaIndex can load the config without creating an index.

python pipeline.py --dry-run

Successful run ends with:

INFO:root:Configuration loaded from /etc/llama/config.yaml
INFO:root:LlamaIndex initialized successfully (dry run)

Inspect pod events for mount errors:

kubectl get events -n data-pipeline --field-selector involvedObject.name=llama-index-init-abc123

No FailedMount events should appear.

Prevent

Implement guardrails to avoid recurrence:

• Namespace‑aware CI lint: Add a kubectl kustomize or kubeval step that verifies every ConfigMap volume reference resolves to an existing object in the same namespace.
• Automated ConfigMap diff: Store the expected ConfigMap name in a Helm/ Kustomize value file and enforce consistency via a Helm test hook.
• SecurityContext defaults: Define a PodSecurityPolicy or OPA Gatekeeper policy that requires fsGroup on all volume mounts that are read by non‑root containers.
• Health probe for config availability: Add an init container that checks /etc/llama/config.yaml existence and exits with non‑zero status if missing, preventing the main container from starting.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

• Why does the pod start but LlamaIndex still cannot read the file?
  Because the pod’s init container may succeed before the volume is populated; the mount is performed after the container starts if the volume definition is incorrect, leading to an empty directory at runtime.
• Can I mount a ConfigMap from a different namespace without duplicating it?
  Native configMap volume sources are limited to the pod’s namespace. To share across namespaces you must either copy the ConfigMap into the target namespace or use a CSI driver that supports cross‑namespace mounts.
• What if I need the ConfigMap to be updated without restarting the pod?
  ConfigMap volumes are refreshed automatically every kubelet sync period (default 1 minute). Ensure the metadata.resourceVersion changes; otherwise, the pod will continue using the stale cache.
• My pod runs as a non‑root user but the mount is still unreadable even after setting fsGroup. What else should I check?
  Verify that the underlying SecurityContext does not have runAsNonRoot: true combined with a custom readOnlyRootFilesystem that may prevent the kubelet from chowning the volume. Also check that the node’s --allow-privileged flag is not interfering with volume permissions.
• Is there a way to see which exact file caused the FileNotFoundError in LlamaIndex?
  Enable LlamaIndex debug logging:

  export LOG_LEVEL=DEBUG
  python pipeline.py

  The debug output prints the resolved absolute path before opening the file.