Problem – Canary Deployment of TGI Fails with ConfigMap/Secret Mount Errors
During a canary rollout of the Text Generation Inference (TGI) service, the new replica set never becomes ready. The pod logs show missing environment variables and volume mount failures related to a ConfigMap and a Secret that provide model configuration and API keys. The stable version continues serving traffic, but the canary pods are stuck in Init:CreateMount or CrashLoopBackOff.
Observed symptoms:
- Pod status:
Init:CreateMount→CrashLoopBackOff - Container logs contain:
Error: failed to load configuration: configmap "tgi-model-config" not found
Error: environment variable MODEL_PATH is not set
MountVolume.SetUp failed for volume "model-config": configmap "tgi-model-config" not found
These errors prevent the TGI binary from starting, causing the canary deployment to be rejected by the Deployment controller.
Root Cause – Mismatch Between Canary Deployment Manifest and Namespace Resources
The canary deployment uses a separate namespace (tgi-canary) for traffic shadowing. The original stable deployment resides in tgi-prod. The canary Deployment references the same ConfigMap and Secret names without specifying a namespace selector. Because Kubernetes treats ConfigMap and Secret as namespace‑scoped resources, the controller looks for them in tgi-canary and fails to locate the objects that exist only in tgi-prod.
Additional contributing factors:
- The canary
Deploymentomitted themetadata.namespacefield for the volume sources, relying on the default namespace of the pod spec. - RBAC policies allowed the canary service account to read
ConfigMaps in its own namespace but not cross‑namespace resources. - A recent Helm chart change introduced a
--set canary.enabled=trueflag that duplicated the deployment without updating the resource references.
Debug – Step‑by‑Step Investigation
1. Verify pod status and events
kubectl get pods -n tgi-canary -l app=tgi-canary -o wide
kubectl describe pod <pod-name> -n tgi-canary
Typical output from kubectl describe:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned tgi-canary/...
Warning FailedMount 1m kubelet MountVolume.SetUp failed for volume "model-config": configmap "tgi-model-config" not found
Warning FailedMount 1m kubelet MountVolume.SetUp failed for volume "api-keys": secret "tgi-api-keys" not found
2. Check existence of the referenced resources
# In the production namespace
kubectl get configmap tgi-model-config -n tgi-prod
kubectl get secret tgi-api-keys -n tgi-prod
# In the canary namespace (expected to be missing)
kubectl get configmap tgi-model-config -n tgi-canary
kubectl get secret tgi-api-keys -n tgi-canary
Output shows the objects exist only in tgi-prod and not in tgi-canary.
3. Inspect the canary Deployment manifest
kubectl get deployment tgi-canary -n tgi-canary -o yaml
Relevant excerpt (before fix):
apiVersion: apps/v1
kind: Deployment
metadata:
name: tgi-canary
namespace: tgi-canary
spec:
replicas: 2
selector:
matchLabels:
app: tgi-canary
template:
metadata:
labels:
app: tgi-canary
spec:
containers:
- name: tgi
image: ghcr.io/huggingface/text-generation-inference:latest
env:
- name: MODEL_PATH
valueFrom:
configMapKeyRef:
name: tgi-model-config # <-- missing namespace
key: model_path
volumeMounts:
- name: model-config
mountPath: /etc/tgi/config
- name: api-keys
mountPath: /etc/tgi/keys
volumes:
- name: model-config
configMap:
name: tgi-model-config # <-- missing namespace
- name: api-keys
secret:
secretName: tgi-api-keys # <-- missing namespace
4. Confirm RBAC does not allow cross‑namespace reads
kubectl get rolebinding -n tgi-canary -o yaml | grep -i tgi
The role binding only grants get on configmaps and secrets within tgi-canary.
Solution – Align Canary Deployment with Production Resources
Option A: Duplicate ConfigMap and Secret into the Canary Namespace
If isolation is required, copy the objects:
# Export from prod
kubectl get configmap tgi-model-config -n tgi-prod -o yaml | \
sed 's/namespace: tgi-prod/namespace: tgi-canary/' | \
kubectl apply -f -
kubectl get secret tgi-api-keys -n tgi-prod -o yaml | \
sed 's/namespace: tgi-prod/namespace: tgi-canary/' | \
kubectl apply -f -
Option B: Reference Production Namespace Directly (Preferred for Canary)
Update the canary Deployment to include the namespace field in configMapKeyRef, configMap, and secret sections.
Before (faulty):
env:
- name: MODEL_PATH
valueFrom:
configMapKeyRef:
name: tgi-model-config
key: model_path
volumes:
- name: model-config
configMap:
name: tgi-model-config
- name: api-keys
secret:
secretName: tgi-api-keys
After (fixed):
env:
- name: MODEL_PATH
valueFrom:
configMapKeyRef:
name: tgi-model-config
key: model_path
namespace: tgi-prod # <-- added
volumes:
- name: model-config
configMap:
name: tgi-model-config
namespace: tgi-prod # <-- added
- name: api-keys
secret:
secretName: tgi-api-keys
optional: false
namespace: tgi-prod # <-- added
Apply the corrected manifest:
kubectl apply -f tgi-canary-deployment-fixed.yaml
Option C: Use Helm Values to Propagate Namespace
If the deployment is managed by Helm, add a values entry:
canary:
enabled: true
resourceNamespace: tgi-prod # used by the chart to set .Values.canary.resourceNamespace
Then run helm upgrade --install with the updated values.
Verify – Confirm the Canary Pods Reach Ready State
# Watch rollout status
kubectl rollout status deployment/tgi-canary -n tgi-canary --watch
# Expect:
deployment "tgi-canary" successfully rolled out
Check pod logs for successful configuration load:
kubectl logs -n tgi-canary -l app=tgi-canary -c tgi --tail=20
Sample successful log snippet:
2026-11-09T12:34:56.789Z INFO Loaded model configuration from /etc/tgi/config/model.yaml
2026-11-09T12:34:56.790Z INFO API keys loaded, total 3 entries
2026-11-09T12:34:56.791Z INFO TGI service started, listening on :8080
Validate that the service is receiving traffic:
curl -s http://tgi-canary.tgi.svc.cluster.local:8080/healthz
# Expected output: {"status":"ok"}
Prevent – Guardrails for Future Canary Deployments
- Namespace‑aware manifests: Always specify
namespaceforConfigMapKeyRef,configMap, andsecretwhen the consuming pod lives in a different namespace. - Helm chart validation: Add a lint test that renders the chart with
canary.enabled=trueand asserts that nonamespacefield is omitted. - RBAC review: Grant the canary service account
getpermission on the target namespace’sconfigmapsandsecretsif cross‑namespace access is intended. - CI/CD gate: Include a step that runs
kubectl diffbetween the rendered manifests and a baseline that checks for missingnamespacefields. - Monitoring alert: Create an alert on
kube_pod_status_phase{phase="FailedMount"}or on container logs containing “configmap .* not found”.
Related Topic Hub: Model Serving Troubleshooting Hub
FAQ
- Why does the stable deployment work while the canary fails?
The stable deployment runs in the same namespace as itsConfigMapandSecret. The canary was created in a separate namespace, so the implicit namespace lookup fails. - Can I reference a ConfigMap from another namespace without copying it?
Yes. Starting with Kubernetes 1.22,configMapKeyRefandsecretKeyRefsupport anamespacefield. The same applies to volume sources (configMapandsecretsections). - What if I need the canary to have read‑only access to production secrets?
Create aRolein the production namespace that permitsgeton the required secrets, then bind it to the canary service account via aRoleBindingthat references the production namespace. - How do I debug a similar “FailedMount” error for a PersistentVolumeClaim?
Inspect the PVC status withkubectl describe pvc <name> -n <namespace>, verify the storage class, and ensure the pod’svolumeMountsreference the correct PVC name and namespace. - Is there a way to automatically copy ConfigMaps/Secrets during a canary rollout?
You can use aJobor aninitContainerthat runskubectl get ... -o yaml | kubectl apply -n <canary-namespace> -f -, but be cautious about secret leakage and ensure RBAC permits the operation.