Pinecone pods ErrImagePull after Helm upgrade on GKE

Problem: Pinecone pods stuck in ErrImagePull after a Helm upgrade on GKE

After upgrading the pinecone Helm release on a Google Kubernetes Engine (GKE) cluster, the newly‑created pods enter the ErrImagePull / ImagePullBackOff state. Typical pod description output looks like:

$ kubectl describe pod pinecone-0
...
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       2m    default-scheduler  Successfully assigned default/pinecone-0 to gke-cluster-node-1
  Normal   Pulling         2m    kubelet            Pulling image "registry.pinecone.io/pinecone:latest"
  Warning  Failed          2m    kubelet            Failed to pull image "registry.pinecone.io/pinecone:latest": rpc error: code = Unknown desc = authentication required
  Warning  Failed          2m    kubelet            Error: ErrImagePull
  Normal   BackOff         1m    kubelet            Back-off pulling image "registry.pinecone.io/pinecone:latest"

Impact:

• Search and vector‑similarity services provided by Pinecone become unavailable.
• Dependent micro‑services experience cascading timeouts.
• Cluster resource usage spikes as pods repeatedly retry image pulls.

Root Cause Analysis

The failure originates from the pod’s inability to authenticate against the private Pinecone container registry (registry.pinecone.io). Two closely related mechanisms can break this authentication during a Helm upgrade:

1. ImagePullSecret name mismatch – The Helm chart creates a dockerconfigjson secret (default name pinecone-registry) and references it via imagePullSecrets. During an upgrade the secret may be recreated with a new autogenerated name (e.g., pinecone-registry-20231108) while the pod spec still points to the old name. This is documented in the Pinecone Helm chart required imagePullSecrets section and reproduced in the GitHub issue “ErrImagePull after helm upgrade – secret not applied”.
2. Workload Identity / IAM permission drift – When using GKE Workload Identity, the Kubernetes service account must have roles/artifactregistry.reader on the Artifact Registry repository that mirrors registry.pinecone.io. An upgrade that changes the service‑account annotation or a recent IAM change can leave the pod without the required permission, leading to the same “authentication required” error, as seen in the “service account lacked roles/artifactregistry.reader” incident.

Both conditions result in the same low‑level error message:

Failed to pull image "registry.pinecone.io/pinecone:latest": rpc error: code = Unknown desc = authentication required

Investigation and Debugging

1. Verify the secret referenced by the pod

$ kubectl get pod pinecone-0 -o jsonpath='{.spec.imagePullSecrets[*].name}'
pinecone-registry

Check that the secret actually exists:

$ kubectl get secret pinecone-registry -o yaml
apiVersion: v1
data:
  .dockerconfigjson: 
kind: Secret
type: kubernetes.io/dockerconfigjson

If the secret is missing, kubectl describe pod will show secret "pinecone-registry" not found (see common errors list).

2. Inspect the secret’s content

$ kubectl get secret pinecone-registry -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .
{
  "auths": {
    "registry.pinecone.io": {
      "username": "my-service-account",
      "password": "",
      "email": "none",
      "auth": "bXktc2VydmljZS1hY2NvdW50OnBhc3N3b3Jk"
    }
  }
}

If the auth field is empty or the credentials are stale (e.g., after a rotation script), the registry will reject the pull.

3. Confirm Workload Identity bindings (if used)

# Identify the KSA used by the deployment
$ kubectl get deployment pinecone -o jsonpath='{.spec.template.spec.serviceAccountName}'
pinecone-sa

# Show the GSA annotation
$ kubectl get serviceaccount pinecone-sa -o yaml | grep annotations -A2
annotations:
  iam.gke.io/gcp-service-account: pinecone-gsa@my-project.iam.gserviceaccount.com

Verify IAM permissions:

$ gcloud projects get-iam-policy my-project \
    --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:pinecone-gsa@my-project.iam.gserviceaccount.com"
roles/artifactregistry.reader

Missing roles/artifactregistry.reader explains the authentication failure.

4. Examine Helm release history for secret recreation

$ helm history pinecone
REVISION    UPDATED                     STATUS      CHART               APP VERSION DESCRIPTION
1           2023-11-01 10:12:33         superseded  pinecone-1.2.0      1.0          Install complete
2           2023-11-08 14:05:12         superseded  pinecone-1.3.0      1.1          Upgrade "pinecone"
3           2023-11-15 09:27:45         deployed    pinecone-1.3.1      1.2          Upgrade "pinecone"

Inspect the manifest of revision 2 to see if the secret name changed:

$ helm get manifest pinecone --revision 2 | grep -A2 "kind: Secret"
apiVersion: v1
kind: Secret
metadata:
  name: pinecone-registry-20231108
type: kubernetes.io/dockerconfigjson

If the secret name differs from the one referenced in the deployment, the pod will continue to look for the old secret.

Resolution

Option 1 – Preserve the secret name across upgrades

Modify the Helm values to set a static secret name and prevent Helm from recreating it with a timestamped suffix.

Before (default values.yaml snippet):

imagePullSecrets:
  create: true
  name: ""

After (override via --set or custom values.yaml):

imagePullSecrets:
  create: true
  name: pinecone-registry   # static name

Run the upgrade:

helm upgrade pinecone pinecone-io/pinecone \
  --namespace default \
  --set imagePullSecrets.name=pinecone-registry \
  --reuse-values

This forces Helm to reuse the existing secret instead of generating a new one, aligning the pod spec with the actual secret.

Option 2 – Add helm.sh/resource-policy: keep to the secret

Annotating the secret tells Helm not to delete it on upgrade, avoiding name changes.

apiVersion: v1
kind: Secret
metadata:
  name: pinecone-registry
  annotations:
    "helm.sh/resource-policy": keep
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: 

Apply the secret before the next upgrade, then re‑run helm upgrade. The secret will survive and retain its name.

Option 3 – Fix Workload Identity permissions

If using GKE Workload Identity, grant the required role to the Google service account:

gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:pinecone-gsa@my-project.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.reader"

After the IAM change propagates (usually < 5 minutes), delete the failing pods so they are recreated with the refreshed token:

kubectl delete pod -l app=pinecone

Option 4 – Refresh stale credentials

If a rotation script updated the Docker credentials but the secret was not refreshed, recreate the secret manually:

kubectl delete secret pinecone-registry
kubectl create secret docker-registry pinecone-registry \
  --docker-server=registry.pinecone.io \
  --docker-username=my-service-account \
  --docker-password=$(gcloud auth print-access-token) \
  --docker-email=none

Then restart the deployment:

kubectl rollout restart deployment pinecone

Verification

After applying one of the fixes, confirm that pods transition to Running:

$ kubectl get pods -l app=pinecone
NAME          READY   STATUS    RESTARTS   AGE
pinecone-0    1/1     Running   0          2m
pinecone-1    1/1     Running   0          2m

Check the events for a successful image pull:

$ kubectl describe pod pinecone-0 | grep -i "pull"
    Pulling image "registry.pinecone.io/pinecone:latest"
    Successfully pulled image "registry.pinecone.io/pinecone:latest"

Optionally, query the Pinecone health endpoint to ensure the service is functional:

curl -s http://pinecone.default.svc.cluster.local:8080/health | jq .
{
  "status": "ok",
  "version": "1.2.0"
}

Prevention and Best Practices

• Static secret names: Always set imagePullSecrets.name explicitly in the Helm values file to avoid Helm‑generated suffixes.
• Resource‑policy annotation: Add helm.sh/resource-policy: keep to any secret that stores registry credentials.
• Workload Identity audit: Include a periodic IAM‑policy check in your CI/CD pipeline to verify that the GSA still has roles/artifactregistry.reader.
• Credential rotation workflow: When rotating Docker credentials, script a kubectl rollout restart of dependent deployments, or use a post‑upgrade Helm hook that patches the deployment with the new secret name.
• Monitoring: Add alerts on pod phases ErrImagePull and ImagePullBackOff (e.g., Prometheus rule kube_pod_container_status_waiting_reason{reason="ErrImagePull"}) to catch regressions early.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

1. Why does the error appear only after a Helm upgrade and not on the initial install?
   During the initial install the secret is created once and the deployment references it correctly. An upgrade may recreate the secret with a new name (Helm’s default behavior) while the deployment spec still points to the old name, causing the mismatch.
2. Can I use a single secret for multiple Pinecone releases?
   Yes. Define a static secret name and reference it in each release’s imagePullSecrets. Ensure the secret is created before any release that needs it.
3. Do I need to enable Workload Identity for private registry access?
   Workload Identity is optional. You can also use a traditional docker-registry secret. If you choose Workload Identity, the GSA must have roles/artifactregistry.reader on the target repository.
4. What if the private registry is behind a VPC‑native network policy?
   Make sure the pod’s egress rules allow traffic to registry.pinecone.io:443. Verify with kubectl exec -it <pod> -- curl -v https://registry.pinecone.io/v2/. Adjust the NetworkPolicy accordingly.
5. How do I know which secret a pod is actually using?
   Run kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets[*].name}'. Compare that name with kubectl get secret output to ensure the secret exists and contains valid .dockerconfigjson data.