aerial view of city during daytime

Prometheus image pull failure on edge node after registry auth error

by

Problem – Prometheus Fails to Pull Its Image on an Edge Node

An edge‑computing node that runs a lightweight Kubernetes distribution (e.g., K3s) attempts to start the prometheus pod. The pod remains in ImagePullBackOff and the node reports a registry authentication error combined with intermittent network timeouts.


$ kubectl get pod -n monitoring prometheus-0 -o wide
NAME          READY   STATUS             RESTARTS   AGE   IP           NODE
prometheus-0  0/2     ImagePullBackOff   0          3m    10.42.0.12   edge-node-01


$ kubectl describe pod -n monitoring prometheus-0
...
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Pulling         2m    kubelet, edge-node-01  Pulling image "registry.example.com/prometheus:2.45.0"
  Warning  Failed          2m    kubelet, edge-node-01  Failed to pull image "registry.example.com/prometheus:2.45.0": rpc error: code = Unknown desc = failed to resolve auth config for registry.example.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff         1m    kubelet, edge-node-01  Back-off pulling image "registry.example.com/prometheus:2.45.0"

Root Cause – Combined Auth Token Expiry and Bandwidth Saturation

The failure is not a simple typo in the image name. Two independent conditions interact:

  • Registry authentication token expiration: The edge node uses a short‑lived docker/config.json token (default 1 hour) generated by an external CI pipeline. When the token expires, the registry returns 401 Unauthorized. The kubelet logs only surface the generic “failed to resolve auth config”.
  • Intermittent network bandwidth: The edge node connects to the central registry over a satellite link with < 2 Mbps average throughput and frequent packet loss. When the token is refreshed, the pull request often times out before the registry can respond, causing the Client.Timeout error shown above.

Because the kubelet retries with exponential back‑off, the pod stays in ImagePullBackOff even after the token is refreshed, until the back‑off timer expires.

Debug – Step‑by‑Step Investigation

1. Verify the kubelet’s credential source


$ cat /var/lib/kubelet/config.json | jq .auths
{
  "registry.example.com": {
    "auth": "c2VjcmV0dG9rZW4="
  }
}

The auth field is a base64‑encoded username:password pair that expires after one hour.

2. Test registry connectivity manually


$ curl -v -u user:$(echo c2VjcmV0dG9rZW4= | base64 -d) \
  https://registry.example.com/v2/_catalog
*   Trying 203.0.113.10:443...
* Connected to registry.example.com (203.0.113.10) port 443 (#0)
* TLS handshake ... (omitted)
* Connected to registry.example.com (203.0.113.10) port 443 (#0)
* Operation timed out after 30000 milliseconds with 0 bytes received

The request times out, confirming bandwidth constraints.

3. Check token validity directly


$ docker login registry.example.com -u user -p $(echo c2VjcmV0dG9rZW4= | base64 -d)
Error response from daemon: Get https://registry.example.com/v2/: unauthorized: authentication required

The registry rejects the token, indicating expiration.

4. Inspect node‑level network metrics


$ ss -i state established '( dport = :443 )'
Recv-Q Send-Q Local Address:Port Peer Address:Port
0      0      10.42.0.12:53024    203.0.113.10:443
    skmem:(r0,rb87380,t0,tb131072) bbr wscale:7,7 rto:200 rtt:150ms/30ms ato:40ms mss:1460 cwnd:10 ssthresh:20

Round‑trip time spikes (>150 ms) and low congestion window confirm the link is saturated.

Solution – Refresh Credentials and Add a Pull‑Through Cache

1. Deploy a local registry cache on the edge node

Running a lightweight registry:2 instance with proxy mode eliminates the need for the edge node to contact the central registry for every image.

Before (no cache)

Component Registry Access
Edge kubelet Direct HTTPS to central registry

After (cache added)

Component Registry Access
Edge kubelet Local HTTP to localhost:5000 (proxy)
Local cache HTTPS to central registry (only on miss) 

# Deploy cache (manifest saved as cache-registry.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-registry-cache
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: edge-registry-cache
  template:
    metadata:
      labels:
        app: edge-registry-cache
    spec:
      containers:
      - name: registry
        image: registry:2
        env:
        - name: REGISTRY_PROXY_REMOTEURL
          value: https://registry.example.com
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: edge-registry-cache
  namespace: kube-system
spec:
  selector:
    app: edge-registry-cache
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 5000

2. Update the Prometheus image reference to use the cache


# Before (central registry)
image: registry.example.com/prometheus:2.45.0

# After (local cache)
image: localhost:5000/prometheus:2.45.0

3. Automate token rotation with a short‑lived secret

Create a Kubernetes docker-registry secret that is refreshed by a CronJob every 45 minutes.


# cronjob-refresh-registry-secret.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: refresh-registry-secret
  namespace: kube-system
spec:
  schedule: "*/45 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: refresher
            image: alpine:3.18
            command: ["/bin/sh", "-c"]
            args:
            - |
              TOKEN=$(curl -s -X POST https://auth.example.com/token -d '{"user":"edge","pass":"$EDGE_PASS"}')
              kubectl create secret docker-registry edge-registry \
                --docker-server=localhost:5000 \
                --docker-username=edge \
                --docker-password=$TOKEN \
                --dry-run=client -o yaml | kubectl apply -f -
          restartPolicy: OnFailure

Verify – Confirm the Pod Starts and Metrics Flow


$ kubectl rollout restart deployment/prometheus -n monitoring
$ kubectl get pod -n monitoring -w
NAME          READY   STATUS    RESTARTS   AGE
prometheus-0  2/2     Running   0          30s

Check that the image was pulled from the cache:


$ kubectl logs -n kube-system -l app=edge-registry-cache -c registry
time="2026-10-09T12:34:56Z" level=info msg="GET /v2/prometheus/manifests/2.45.0 from 10.42.0.12:53024"

Validate Prometheus is scraping targets:


$ curl -s http://edge-node-01:9090/api/v1/targets | jq '.data.activeTargets[] | .scrapeUrl'
"http://node-exporter:9100/metrics"

Prevent – Operational Guardrails for Edge Registries

  • Cache first strategy: Deploy a pull‑through cache on every edge site. It reduces bandwidth usage and isolates the edge from central registry outages.
  • Token lifetime management: Use secrets with kubectl apply --server-side to ensure atomic updates and avoid stale tokens.
  • Health checks: Add a liveness probe to the cache deployment that queries /v2/_catalog every 5 minutes; alert on failures.
  • Network QoS: Prioritize registry traffic (port 443) with a traffic‑shaping rule (e.g., tc qdisc add dev eth0 root tbf rate 2mbit burst 32kbit latency 100ms) to guarantee enough headroom for image pulls.
  • Back‑off tuning: Set imagePullBackOff back‑off limits in the kubelet config to a maximum of 30 seconds on edge nodes, so a refreshed token is retried sooner.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  • Why does the pod stay in ImagePullBackOff after I manually re‑login?
    Because the kubelet’s internal back‑off timer continues to delay retries. Restarting the pod or clearing the back‑off (e.g., kubectl delete pod) forces an immediate pull.
  • Can I use a public mirror (e.g., Docker Hub) instead of a local cache?
    A public mirror works only if the image is publicly available. For private registries you must either proxy through a cache or replicate the image to a public registry.
  • How do I know if the token is the root cause versus network latency?
    Inspect the registry response code. A 401 Unauthorized indicates token issues; a timeout or 504 Gateway Timeout points to network problems. Correlate with kubectl logs from the cache or registry.
  • Is it safe to store the registry password in a plain‑text secret?
    Kubernetes secrets are base64‑encoded, not encrypted. On edge nodes enable EncryptionConfiguration for secret data at rest, and restrict secret access via RBAC.
  • What if the edge node loses connectivity for an extended period?
    The local cache will serve any previously pulled layers. Ensure the cache’s storage (e.g., emptyDir on SSD) is sized to hold all required images for the expected outage duration.