Prometheus scrape target creation failure for TensorFlow Serving after Helm upgrade

Problem Description

After upgrading the tf-serving Helm chart, Prometheus stopped collecting latency and throughput metrics from TensorFlow Serving pods. The Prometheus server logs contain entries such as:

scrape target creation failed: connection refused (target: http://10.1.2.3:8501/metrics)
no endpoints found for service "tf-serving" in namespace "ml" – check ServiceMonitor selector
error reading body: EOF while scraping TensorFlow Serving metrics

Consequences observed in the monitoring dashboards:

• Missing tf_serving_request_latency_seconds and tf_serving_inference_requests_total series.
• Alert rules that depend on these metrics fire continuously.
• Prometheus UI shows “0 active targets” for the tf-serving job.

Root Cause Analysis

The failure originates from the service discovery layer of the Prometheus Operator. Two intertwined mis‑configurations are typical after a Helm upgrade:

1. ServiceMonitor selector mismatch – The ServiceMonitor CRD created by the chart uses a matchLabels selector (e.g., app: tf-serving). During the upgrade the chart changed the label key to app.kubernetes.io/name: tf-serving (see GitHub issue #2314). Prometheus therefore cannot associate any pods with the existing ServiceMonitor, resulting in “no active targets”.
2. Metrics port name change – The chart version bump renamed the container port from metrics to http-metrics. The default relabel rule in the Operator’s scrape_config filters endpoints by __meta_kubernetes_pod_container_port_name=metrics (Prometheus docs – Kubernetes Service Discovery). Because the new name does not match, the endpoint is dropped, leading to “connection refused” or “EOF” errors when Prometheus attempts to scrape the old address.

Both issues are documented in the official Prometheus Operator CRD reference (ServiceMonitor CRD) and the TensorFlow Serving metrics guide (TensorFlow Serving Monitoring).

Investigation and Debugging

Step‑by‑step diagnostics that reproduced the failure:

1. Inspect Prometheus logs for target errors:

   
   kubectl -n monitoring logs prometheus-0 | grep "scrape target creation failed"
   

2. Verify ServiceMonitor selector against pod labels:

   
   kubectl -n ml get servicemonitor tf-serving -o yaml
   kubectl -n ml get pods -l app=tf-serving -o jsonpath="{.items[*].metadata.labels}"
   

   Expected output shows a mismatch:

   
   serviceMonitor.spec.selector.matchLabels:
     app: tf-serving
   pod.labels:
     app.kubernetes.io/name: tf-serving
   

3. Check the metrics port name on the pods:

   
   kubectl -n ml get pod tf-serving-abc123 -o jsonpath="{.spec.containers[0].ports}"
   

   Output after upgrade:

   
   [
     {
       "containerPort": 8501,
       "name": "http-metrics"
     }
   ]
   

4. Confirm ServiceMonitor endpoint definition:

   
   kubectl -n ml get servicemonitor tf-serving -o yaml | grep port
   

   Shows port: metrics, which no longer exists.

5. Run a manual curl against a pod IP to see if the endpoint is reachable:

   
   POD_IP=$(kubectl -n ml get pod tf-serving-abc123 -o jsonpath="{.status.podIP}")
   curl -s http://$POD_IP:8501/metrics | head -n 5
   

   Result: curl: (7) Failed to connect – confirming the port name mismatch.

Resolution

Two corrective actions are required: align ServiceMonitor selectors with pod labels and ensure the metrics port name matches the relabel rule.

1. Update ServiceMonitor selector

Before:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tf-serving
spec:
  selector:
    matchLabels:
      app: tf-serving
  namespaceSelector:
    matchNames:
    - ml
  endpoints:
  - port: metrics
    path: /metrics

After (matching the chart’s label schema):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tf-serving
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: tf-serving
  namespaceSelector:
    matchNames:
    - ml
  endpoints:
  - port: http-metrics
    path: /metrics

2. Adjust the port name in the Helm values (or chart)

If you prefer to keep the original ServiceMonitor, rename the container port back to metrics via Helm values:

# values.yaml
serving:
  containerPort: 8501
  metricsPortName: metrics   # ← add or override

Or patch the existing deployment:

kubectl -n ml patch deployment tf-serving \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"tf-serving","ports":[{"containerPort":8501,"name":"metrics"}]}]}}}}'

3. Re‑apply the ServiceMonitor

Because Helm upgrade hooks may delete and recreate CRDs, ensure the ServiceMonitor is recreated after the deployment:

helm upgrade tf-serving ./tf-serving-chart \
  --namespace ml \
  --set prometheusOperator.enabled=true \
  --set prometheusOperator.serviceMonitor.enabled=true \
  --reuse-values

Adding the annotation helm.sh/hook: post-install,post-upgrade to the ServiceMonitor manifest (see Helm best practices) guarantees it persists across upgrades.

Verification

After applying the fixes, perform the following checks:

1. Prometheus target status:

   
   kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 &
   curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="tf-serving")'
   

   Output should show health":"up" and a valid scrapeUrl.

2. Metric presence:

   
   curl -s http://localhost:9090/api/v1/query?query=tf_serving_inference_requests_total | jq '.data.result | length'
   

   Non‑zero result indicates successful collection.

3. Manual scrape test:

   
   curl -s http://$(kubectl -n ml get svc tf-serving -o jsonpath="{.spec.clusterIP}"):8501/metrics | grep tf_serving
   

   Should return metric lines without errors.

Prevention and Best Practices

Area	Recommendation
Helm chart versioning	Pin chart versions and review values.yaml diffs for label or port name changes before upgrade.
ServiceMonitor stability	Declare ServiceMonitor as a helm.sh/hook with post-upgrade to ensure recreation after CRD upgrades.
Label consistency	Adopt a single label convention (e.g., app.kubernetes.io/name) across all workloads and ServiceMonitors.
Port naming	Keep the metrics port name constant (default metrics) or update relabel rules in scrape_config accordingly.
Observability testing	Include a CI step that runs promtool test rules and a smoke‑test curl against /metrics after each chart release.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does Prometheus show “no active targets” after a Helm upgrade?
   Because the ServiceMonitor selector no longer matches any pod labels or the endpoint port name changed, causing the discovery process to drop all endpoints.
2. Can I rely solely on pod annotations (prometheus.io/scrape) instead of ServiceMonitors?
   Annotations work with the default Prometheus configuration, but the Prometheus Operator disables them by default in favor of ServiceMonitors. Mixing both can lead to duplicate or missing targets after upgrades.
3. How do I debug a “connection refused” error when scraping TensorFlow Serving?
   Verify the pod’s IP and port with kubectl exec or curl, ensure the container exposes the correct port name, and confirm the ServiceMonitor endpoint references that name.
4. What if the chart upgrades the namespace selector?
   Update the ServiceMonitor’s namespaceSelector to include the new namespace or use any: true if you want to monitor across all namespaces.
5. Is there a way to automatically detect mismatched ServiceMonitor selectors?
   Enable the Prometheus Operator’s --log-level=debug flag; it emits warnings when a ServiceMonitor has no matching endpoints, which can be routed to an alerting rule.