A classical painting of a reclining nude woman

Prometheus Docker image pull failure after registry credential rotation

by

Problem

During a routine rollout of the Prometheus monitoring service for AI workloads, the container failed to start with an image pull error. The Kubernetes pod entered ImagePullBackOff and the following log snippet was observed in the node’s kubelet output:


Failed to pull image "registry.example.com/monitoring/prometheus:2.45.0": rpc error: code = Unknown desc = failed to resolve reference "registry.example.com/monitoring/prometheus:2.45.0": unauthorized: authentication required

Simultaneously, the docker daemon on the host reported:


time="2026-11-02T08:12:34Z" level=error msg="Error response from daemon: pull access denied for registry.example.com/monitoring/prometheus, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"

The failure manifested only after the organization rotated the credentials used for the private Docker registry.

Root Cause

The image pull failure was caused by a combination of three factors that surfaced after the credential rotation:

  1. Stale Kubernetes secret: The imagePullSecret referenced by the Prometheus deployment still contained the previous .dockerconfigjson payload. Kubernetes does not automatically refresh secrets when the underlying registry credentials change.
  2. Missing image tag in the manifest: The deployment manifest used the latest tag implicitly (registry.example.com/monitoring/prometheus) while the registry now requires an explicit tag for authentication scoping. The registry rejected the request because the token generated for the old tag did not match the new policy.
  3. Network ACL change: As part of the credential rotation, a new service‑mesh policy was applied that limited outbound traffic from the monitoring namespace to the registry’s IP range. The pod’s egress attempt was blocked, causing a timeout that was later reported as an authentication error.

These issues together produced the observed “unauthorized” messages even though the credentials themselves were valid.

Debug

1. Verify the secret content


# Inspect the secret referenced by the deployment
kubectl get secret prometheus-registry-secret -n monitoring -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

Expected output (redacted):


{
  "auths": {
    "registry.example.com": {
      "username": "ci-bot",
      "password": "********",
      "auth": "Y2ktYm90OnBhc3N3b3Jk"
    }
  }
}

If the password field still contains the old credential, the secret must be refreshed.

2. Check the image reference in the deployment


kubectl get deployment prometheus -n monitoring -o yaml | grep image

Typical output after rotation:


      image: registry.example.com/monitoring/prometheus

Notice the missing tag. The registry now enforces tag‑specific tokens.

3. Test connectivity from a pod in the same namespace


kubectl run -i --tty --rm debug-pod --image=alpine --restart=Never -n monitoring -- sh
apk add --no-cache curl
curl -v https://registry.example.com/v2/_catalog

Expected HTTP 200 with a JSON catalog. A timeout or 403 Forbidden indicates the ACL is blocking egress.

4. Review the service‑mesh policy


istioctl proxy-config listeners $(kubectl get pod -l app=prometheus -n monitoring -o name) -n monitoring

Look for egress rules that restrict traffic to registry.example.com. If missing, the request will be dropped.

Solution

1. Regenerate the image pull secret

Obtain the new registry password from the secret manager and recreate the secret:


# Create a new .dockerconfigjson file
cat > /tmp/dockerconfig.json <<EOF
{
  "auths": {
    "registry.example.com": {
      "username": "ci-bot",
      "password": "NEW_PASSWORD"
    }
  }
}
EOF

# Encode and apply
kubectl create secret generic prometheus-registry-secret \
  --from-file=.dockerconfigjson=/tmp/dockerconfig.json \
  --type=kubernetes.io/dockerconfigjson \
  -n monitoring --dry-run=client -o yaml | kubectl apply -f -

After applying, the secret contains the fresh credentials.

2. Pin the image tag explicitly

Update the deployment manifest to reference the exact version that exists in the registry:


# Before
image: registry.example.com/monitoring/prometheus

# After
image: registry.example.com/monitoring/prometheus:2.45.0

Apply the change:


kubectl apply -f prometheus-deployment.yaml

3. Adjust the egress policy

If using Istio, add an egress rule that permits traffic to the registry’s IP range (example CIDR 203.0.113.0/24).


apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: registry-egress
  namespace: monitoring
spec:
  hosts:
  - "registry.example.com"
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL
  endpoints:
  - address: 203.0.113.45

Apply the ServiceEntry:


kubectl apply -f registry-egress.yaml

Verify

After the three corrective actions, confirm that the pod pulls the image successfully:


kubectl rollout restart deployment/prometheus -n monitoring
kubectl get pod -l app=prometheus -n monitoring -w

Expected pod status transition:


Pending → ContainerCreating → Running

Check the node logs for a successful pull:


journalctl -u kubelet | grep "Successfully pulled image"

Sample successful log entry:


time="2026-11-02T08:15:12Z" level=info msg="Successfully pulled image \"registry.example.com/monitoring/prometheus:2.45.0\""

Run a quick Prometheus query to ensure the service is operational:


curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=up | jq .

The response should contain a non‑zero value for the prometheus target.

Prevent

  • Automate secret rotation: Use a controller (e.g., external-secrets or sealed-secrets) that watches the credential store and updates the imagePullSecret automatically.
  • Pin image tags in manifests: Enforce a CI policy that rejects manifests lacking an explicit tag. Tag immutability reduces token‑scope mismatches.
  • Version‑controlled egress policies: Store ServiceEntry or NetworkPolicy definitions in Git and review them during any registry credential change.
  • Health‑check the registry: Deploy a sidecar that periodically performs a curl -f https://registry.example.com/v2/ and emits a Prometheus metric; alert on failures.
  • Document rotation procedures: Include steps to refresh secrets, update tags, and verify egress in the runbook.

FAQ

  1. Why does the pod show ImagePullBackOff instead of a clear “authentication failed” message?
    Kubernetes normalizes registry errors into a generic back‑off state. The underlying kubelet logs contain the precise unauthorized message, which must be inspected.
  2. Can I keep using the latest tag if I rotate credentials?
    No. The registry token is scoped to the specific image reference, including the tag. Using latest forces a new token request on each pull, which fails if the secret is stale.
  3. Do I need to restart the kubelet after updating the secret?
    No. Updating the secret and redeploying the workload forces the kubelet to fetch the new credentials on the next pull attempt.
  4. How do I know if a network policy, not a secret, is the root cause?
    Perform a manual curl from a debug pod in the same namespace. A timeout or 403 indicates egress blockage, whereas a 401/403 after a successful TCP handshake points to authentication.
  5. Is there a way to test the token generation before applying the secret?
    Yes. Use docker login registry.example.com with the new credentials on a workstation; a successful login confirms that the token can be generated.

Related Topic Hub: Observability Troubleshooting Hub

Related Articles