TGI canary deployment fails with ConfigMap mount error after rollout

Problem – Canary Deployment of TGI Fails with ConfigMap/Secret Mount Errors

During a canary rollout of the Text Generation Inference (TGI) service, the new replica set never becomes ready. The pod logs show missing environment variables and volume mount failures related to a ConfigMap and a Secret that provide model configuration and API keys. The stable version continues serving traffic, but the canary pods are stuck in Init:CreateMount or CrashLoopBackOff.

Observed symptoms:

  • Pod status: Init:CreateMount → CrashLoopBackOff
  • Container logs contain:
Error: failed to load configuration: configmap "tgi-model-config" not found
Error: environment variable MODEL_PATH is not set
MountVolume.SetUp failed for volume "model-config": configmap "tgi-model-config" not found

These errors prevent the TGI binary from starting, causing the canary deployment to be rejected by the Deployment controller.

Root Cause – Mismatch Between Canary Deployment Manifest and Namespace Resources

The canary deployment uses a separate namespace (tgi-canary) for traffic shadowing. The original stable deployment resides in tgi-prod. The canary Deployment references the same ConfigMap and Secret names without specifying a namespace selector. Because Kubernetes treats ConfigMap and Secret as namespace‑scoped resources, the controller looks for them in tgi-canary and fails to locate the objects that exist only in tgi-prod.

Additional contributing factors:

  • The canary Deployment omitted the metadata.namespace field for the volume sources, relying on the default namespace of the pod spec.
  • RBAC policies allowed the canary service account to read ConfigMaps in its own namespace but not cross‑namespace resources.
  • A recent Helm chart change introduced a --set canary.enabled=true flag that duplicated the deployment without updating the resource references.

Debug – Step‑by‑Step Investigation

1. Verify pod status and events

kubectl get pods -n tgi-canary -l app=tgi-canary -o wide
kubectl describe pod <pod-name> -n tgi-canary

Typical output from kubectl describe:

Events:
  Type    Reason            Age   From               Message
  ----    ------            ----  ----               -------
  Normal  Scheduled         2m    default-scheduler  Successfully assigned tgi-canary/...
  Warning FailedMount       1m    kubelet            MountVolume.SetUp failed for volume "model-config": configmap "tgi-model-config" not found
  Warning FailedMount       1m    kubelet            MountVolume.SetUp failed for volume "api-keys": secret "tgi-api-keys" not found

2. Check existence of the referenced resources

# In the production namespace
kubectl get configmap tgi-model-config -n tgi-prod
kubectl get secret tgi-api-keys -n tgi-prod

# In the canary namespace (expected to be missing)
kubectl get configmap tgi-model-config -n tgi-canary
kubectl get secret tgi-api-keys -n tgi-canary

Output shows the objects exist only in tgi-prod and not in tgi-canary.

3. Inspect the canary Deployment manifest

kubectl get deployment tgi-canary -n tgi-canary -o yaml

Relevant excerpt (before fix):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-canary
  namespace: tgi-canary
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tgi-canary
  template:
    metadata:
      labels:
        app: tgi-canary
    spec:
      containers:
      - name: tgi
        image: ghcr.io/huggingface/text-generation-inference:latest
        env:
        - name: MODEL_PATH
          valueFrom:
            configMapKeyRef:
              name: tgi-model-config   # <-- missing namespace
              key: model_path
        volumeMounts:
        - name: model-config
          mountPath: /etc/tgi/config
        - name: api-keys
          mountPath: /etc/tgi/keys
      volumes:
      - name: model-config
        configMap:
          name: tgi-model-config   # <-- missing namespace
      - name: api-keys
        secret:
          secretName: tgi-api-keys   # <-- missing namespace

4. Confirm RBAC does not allow cross‑namespace reads

kubectl get rolebinding -n tgi-canary -o yaml | grep -i tgi

The role binding only grants get on configmaps and secrets within tgi-canary.

Solution – Align Canary Deployment with Production Resources

Option A: Duplicate ConfigMap and Secret into the Canary Namespace

If isolation is required, copy the objects:

# Export from prod
kubectl get configmap tgi-model-config -n tgi-prod -o yaml | \
  sed 's/namespace: tgi-prod/namespace: tgi-canary/' | \
  kubectl apply -f -

kubectl get secret tgi-api-keys -n tgi-prod -o yaml | \
  sed 's/namespace: tgi-prod/namespace: tgi-canary/' | \
  kubectl apply -f -

Option B: Reference Production Namespace Directly (Preferred for Canary)

Update the canary Deployment to include the namespace field in configMapKeyRef, configMap, and secret sections.

Before (faulty):

env:
- name: MODEL_PATH
  valueFrom:
    configMapKeyRef:
      name: tgi-model-config
      key: model_path
volumes:
- name: model-config
  configMap:
    name: tgi-model-config
- name: api-keys
  secret:
    secretName: tgi-api-keys

After (fixed):

env:
- name: MODEL_PATH
  valueFrom:
    configMapKeyRef:
      name: tgi-model-config
      key: model_path
      namespace: tgi-prod   # <-- added
volumes:
- name: model-config
  configMap:
    name: tgi-model-config
    namespace: tgi-prod   # <-- added
- name: api-keys
  secret:
    secretName: tgi-api-keys
    optional: false
    namespace: tgi-prod   # <-- added

Apply the corrected manifest:

kubectl apply -f tgi-canary-deployment-fixed.yaml

Option C: Use Helm Values to Propagate Namespace

If the deployment is managed by Helm, add a values entry:

canary:
  enabled: true
  resourceNamespace: tgi-prod   # used by the chart to set .Values.canary.resourceNamespace

Then run helm upgrade --install with the updated values.

Verify – Confirm the Canary Pods Reach Ready State

# Watch rollout status
kubectl rollout status deployment/tgi-canary -n tgi-canary --watch

# Expect:
deployment "tgi-canary" successfully rolled out

Check pod logs for successful configuration load:

kubectl logs -n tgi-canary -l app=tgi-canary -c tgi --tail=20

Sample successful log snippet:

2026-11-09T12:34:56.789Z INFO  Loaded model configuration from /etc/tgi/config/model.yaml
2026-11-09T12:34:56.790Z INFO  API keys loaded, total 3 entries
2026-11-09T12:34:56.791Z INFO  TGI service started, listening on :8080

Validate that the service is receiving traffic:

curl -s http://tgi-canary.tgi.svc.cluster.local:8080/healthz
# Expected output: {"status":"ok"}

Prevent – Guardrails for Future Canary Deployments

  • Namespace‑aware manifests: Always specify namespace for ConfigMapKeyRef, configMap, and secret when the consuming pod lives in a different namespace.
  • Helm chart validation: Add a lint test that renders the chart with canary.enabled=true and asserts that no namespace field is omitted.
  • RBAC review: Grant the canary service account get permission on the target namespace’s configmaps and secrets if cross‑namespace access is intended.
  • CI/CD gate: Include a step that runs kubectl diff between the rendered manifests and a baseline that checks for missing namespace fields.
  • Monitoring alert: Create an alert on kube_pod_status_phase{phase="FailedMount"} or on container logs containing “configmap .* not found”.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the stable deployment work while the canary fails?
    The stable deployment runs in the same namespace as its ConfigMap and Secret. The canary was created in a separate namespace, so the implicit namespace lookup fails.
  2. Can I reference a ConfigMap from another namespace without copying it?
    Yes. Starting with Kubernetes 1.22, configMapKeyRef and secretKeyRef support a namespace field. The same applies to volume sources (configMap and secret sections).
  3. What if I need the canary to have read‑only access to production secrets?
    Create a Role in the production namespace that permits get on the required secrets, then bind it to the canary service account via a RoleBinding that references the production namespace.
  4. How do I debug a similar “FailedMount” error for a PersistentVolumeClaim?
    Inspect the PVC status with kubectl describe pvc <name> -n <namespace>, verify the storage class, and ensure the pod’s volumeMounts reference the correct PVC name and namespace.
  5. Is there a way to automatically copy ConfigMaps/Secrets during a canary rollout?
    You can use a Job or an initContainer that runs kubectl get ... -o yaml | kubectl apply -n <canary-namespace> -f -, but be cautious about secret leakage and ensure RBAC permits the operation.