Prometheus Scrape Timeout From Inference Pods After Network Policy Applied

Q: Problem

After applying a restrictive NetworkPolicy to the AI inference serving namespace, Prometheus started reporting scrape timeouts for all /metrics endpoints exposed by the inference pods. The failure manifests as:

Q: Root Cause

The newly applied NetworkPolicy only allowed inbound traffic from pods within the inference namespace and from a specific set of service accounts. Prometheus runs in the monitoring namespace with a default service account, which was not whitelisted. Consequently, the TCP connection from Prometheus to inference-*.svc.cluster.local:8080 was dropped by the kernel’s network policy enforcement, leading to the observed scrape timeout.

Q: Debug 1. Verify the scrape failure

kubectl -n monitoring logs deploy/prometheus-server -c prometheus \ | grep "Scrape failed" | tail -n 5 Sample output:

Q: 2. Test connectivity from a Prometheus pod

kubectl -n monitoring exec -it $(kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}") -- \ curl -v http://inference-0.inference.svc.cluster.local:8080/metrics Result:

Q: 3. Inspect the applied NetworkPolicy

kubectl -n inference get networkpolicy inference-restrict -o yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: inference-restrict namespace: inference spec: podSelector: {} policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: inference ports: - protocol: TCP port: 8080 The policy permits inbound traffic only from pods labeled app=inference within the same namespace.

Q: 4. Confirm Prometheus namespace is not covered

kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}" Result: prometheus-server-7d9f5c9b9-abcde

Q: Solution

Update the NetworkPolicy to explicitly allow Prometheus (or any monitoring component) to reach the /metrics endpoint.

Q: Before (restrictive policy)

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: inference-restrict namespace: inference spec: podSelector: {} policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: inference ports: - protocol: TCP port: 8080 After (allow monitoring) apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: inference-allow-monitoring namespace: inference spec: podSelector: {} policyTypes: - Ingress ingress: # Existing rule for intra‑namespace traffic - from: - podSelector: matchLabels: app: inference ports: - protocol: TCP port: 8080 # New rule for Prometheus in monitoring namespace - from: - namespaceSelector: matchLabels: name: monitoring podSelector: matchLabels: app: prometheus ports: - protocol: TCP port: 8080 Key changes:

Q: Verify 1. Re‑run the connectivity test from a Prometheus pod

kubectl -n monitoring exec -it $(kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}") -- \ curl -s -o /dev/null -w "%{http_code}" http://inference-0.inference.svc.cluster.local:8080/metrics Expected output: 200

Q: 2. Check Prometheus scrape status

kubectl -n monitoring logs deploy/prometheus-server -c prometheus \ | grep "Scrape succeeded" | tail -n 5 Sample output:

In this article

Table of Contents

Problem

After applying a restrictive NetworkPolicy to the AI inference serving namespace, Prometheus started reporting scrape timeouts for all /metrics endpoints exposed by the inference pods. The failure manifests as:


level=error ts=2026-11-03T12:45:27.123Z caller=scrape.go:842 component=prometheus msg="Scrape failed" target="inference-0.inference.svc.cluster.local:8080" err="context deadline exceeded (client timeout exceeded while awaiting headers)"

Metrics collection stops, causing dashboards to show stale data and alerting rules to fire.

Root Cause

The newly applied NetworkPolicy only allowed inbound traffic from pods within the inference namespace and from a specific set of service accounts. Prometheus runs in the monitoring namespace with a default service account, which was not whitelisted. Consequently, the TCP connection from Prometheus to inference-*.svc.cluster.local:8080 was dropped by the kernel’s network policy enforcement, leading to the observed scrape timeout.

Debug

1. Verify the scrape failure


kubectl -n monitoring logs deploy/prometheus-server -c prometheus \
  | grep "Scrape failed" | tail -n 5

Sample output:


level=error ts=2026-11-03T12:45:27.123Z caller=scrape.go:842 component=prometheus msg="Scrape failed" target="inference-0.inference.svc.cluster.local:8080" err="context deadline exceeded (client timeout exceeded while awaiting headers)"
level=error ts=2026-11-03T12:45:27.124Z caller=scrape.go:842 component=prometheus msg="Scrape failed" target="inference-1.inference.svc.cluster.local:8080" err="context deadline exceeded (client timeout exceeded while awaiting headers)"

2. Test connectivity from a Prometheus pod


kubectl -n monitoring exec -it $(kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}") -- \
  curl -v http://inference-0.inference.svc.cluster.local:8080/metrics

Result:


*   Trying 10.96.12.34:8080...
* connect to 10.96.12.34 port 8080 failed: Connection timed out
* Failed to connect to inference-0.inference.svc.cluster.local port 8080: Connection timed out

3. Inspect the applied NetworkPolicy


kubectl -n inference get networkpolicy inference-restrict -o yaml


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-restrict
  namespace: inference
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: inference
    ports:
    - protocol: TCP
      port: 8080

The policy permits inbound traffic only from pods labeled app=inference within the same namespace.

4. Confirm Prometheus namespace is not covered


kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}"

Result: prometheus-server-7d9f5c9b9-abcde

This pod has labels app=prometheus and resides in monitoring, so it does not satisfy the from clause.

Solution

Update the NetworkPolicy to explicitly allow Prometheus (or any monitoring component) to reach the /metrics endpoint.

Before (restrictive policy)


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-restrict
  namespace: inference
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: inference
    ports:
    - protocol: TCP
      port: 8080

After (allow monitoring)


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-allow-monitoring
  namespace: inference
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  # Existing rule for intra‑namespace traffic
  - from:
    - podSelector:
        matchLabels:
          app: inference
    ports:
    - protocol: TCP
      port: 8080
  # New rule for Prometheus in monitoring namespace
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
      podSelector:
        matchLabels:
          app: prometheus
    ports:
    - protocol: TCP
      port: 8080

Key changes:

Add a namespaceSelector that matches the monitoring namespace.
Match pods with label app=prometheus (adjust if your Prometheus deployment uses a different label).
Retain the original intra‑namespace rule to preserve existing traffic flow.

Apply the updated policy:


kubectl apply -f inference-allow-monitoring.yaml

Verify

1. Re‑run the connectivity test from a Prometheus pod


kubectl -n monitoring exec -it $(kubectl -n monitoring get pods -l app=prometheus -o jsonpath="{.items[0].metadata.name}") -- \
  curl -s -o /dev/null -w "%{http_code}" http://inference-0.inference.svc.cluster.local:8080/metrics

Expected output: 200

2. Check Prometheus scrape status


kubectl -n monitoring logs deploy/prometheus-server -c prometheus \
  | grep "Scrape succeeded" | tail -n 5

Sample output:


level=info ts=2026-11-03T12:48:12.456Z caller=scrape.go:1150 component=prometheus msg="Scrape succeeded" target="inference-0.inference.svc.cluster.local:8080"
level=info ts=2026-11-03T12:48:12.457Z caller=scrape.go:1150 component=prometheus msg="Scrape succeeded" target="inference-1.inference.svc.cluster.local:8080"

3. Verify metrics appear in Grafana or Prometheus UI

Navigate to http://prometheus-server.monitoring.svc:9090/graph and query a known inference metric, e.g., inference_requests_total. The series should now return recent samples.

Prevent

Policy Review Process: Include a checklist item to verify that monitoring namespaces are whitelisted before merging any new NetworkPolicy.
Automated Tests: Add an integration test that attempts a curl to /metrics from a monitoring pod after applying policies.
Policy Templates: Maintain a base NetworkPolicy that contains a “monitoring allowlist” block and extend it for service‑specific rules.
Observability Guardrails: Configure an alert on up{job="inference"} == 0 with a short for‑duration to catch future scrape failures early.
Documentation: Record the required label and namespace selectors for any future monitoring components (e.g., Loki, Tempo) that need access.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the scrape work when I run curl from inside the inference pod but not from Prometheus?
The inference pod is allowed to accept traffic from itself (the default podSelector matches its own labels). The restrictive policy does not include the monitoring namespace, so external pods are blocked.
Can I use a NetworkPolicy that allows all traffic from the monitoring namespace without specifying pod labels?
Yes. Replace the podSelector block with an empty selector ({}) under the from clause. This grants any pod in monitoring access, which may be acceptable in low‑risk environments.
My cluster uses Calico with GlobalNetworkPolicy. Do the same rules apply?
Calico’s GlobalNetworkPolicy follows the same Ingress semantics. Ensure the selector includes both the namespace and pod labels, or create a separate NetworkPolicy in the target namespace that references the global policy.
After fixing the policy, Prometheus still shows “scrape timeout”. What else should I check?
Confirm that the metrics_path and port in the Prometheus scrape_config match the service definition. Also verify that the service’s ClusterIP resolves correctly and that there are no sidecar containers intercepting traffic.
Is there a way to see which pods are blocked by a NetworkPolicy without manual testing?
Use kubectl exec with nc -zv or the networkpolicy-pod-selector plugin, or enable Calico’s policy audit feature which logs denied connections to kube-apiserver audit logs.