Prometheus Scrape TLS Handshake Failure After Streaming Service Cert Rotation

Q: Root Cause

The streaming service switched from a RSA 2048‑bit certificate signed by an internal CA to an ECDSA P‑256 certificate signed by the same CA. The new certificate introduced two subtle incompatibilities:

Q: Solution Update Prometheus TLS configuration

Modify the scrape_config for the streaming job to:

Q: OpenSSL confirmation

openssl s_client -connect streaming-prod.example.com:443 -tls1_3 -servername streaming-prod.example.com The handshake completes, and the certificate chain validates against the new intermediate.

Prometheus scrape TLS handshake failure after streaming service cert rotation

Problem

Symptoms and impact

After a scheduled certificate rotation on the real‑time streaming platform, Prometheus began reporting scrape errors for the affected endpoint:


2026-10-19T08:15:32.123Z level=error msg="scrape failed" target="streaming-prod.example.com:443" err="Get \"https://streaming-prod.example.com/metrics\": remote error: tls: handshake failure"
2026-10-19T08:15:32.124Z level=error msg="scrape failed" target="streaming-prod.example.com:443" err="connection reset by peer"

Prometheus UI shows 0/1 up for the job.
Alertmanager fires PrometheusTargetDown alerts.
Other services that use the same endpoint (e.g., internal dashboards) continue to work, suggesting a client‑side issue.

Root Cause

The streaming service switched from a RSA 2048‑bit certificate signed by an internal CA to an ECDSA P‑256 certificate signed by the same CA. The new certificate introduced two subtle incompatibilities:

Server advertised only TLS 1.3 and the TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 cipher suite. The Prometheus binary (v2.31.0) in the production cluster was built with GOOS=linux GOARCH=amd64 and linked against an older Go runtime (1.17) that disables TLS 1.3 by default unless GODEBUG=tls13=1 is set. Consequently, the client attempted TLS 1.2 handshakes, which the server rejected.
Certificate chain validation failed. The internal CA’s intermediate certificate was renewed at the same time, but the ca_file reference in the Prometheus tls_config still pointed to the previous intermediate PEM. The new leaf certificate required the new intermediate, causing a verification error that manifested as a handshake failure.

Both factors combined to produce the observed “handshake failure” and “connection reset by peer” messages.

Debug

Log inspection

Prometheus logs only show the generic handshake error. To get more detail, increase the log level:


kubectl -n monitoring exec -it prometheus-0 -- \
  /bin/prometheus --log.level=debug \
  --config.file=/etc/prometheus/prometheus.yml

Debug output revealed:


2026-10-19T08:16:01.456Z level=debug msg="TLS handshake failed" err="tls: handshake failure"
2026-10-19T08:16:01.456Z level=debug msg="TLS version mismatch: client supports TLS1.2 only, server requires TLS1.3"

OpenSSL sanity check

Running an OpenSSL client from the same node confirms the server accepts TLS 1.3 only:


openssl s_client -connect streaming-prod.example.com:443 -tls1_2 -servername streaming-prod.example.com

Output:


CONNECTED(00000003)
140735274752512:error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number:s3_srvr.c:1604:SSL alert number 70

Using TLS 1.3 succeeds:


openssl s_client -connect streaming-prod.example.com:443 -tls1_3 -servername streaming-prod.example.com

Certificate chain verification

Inspect the server’s presented chain:


openssl s_client -connect streaming-prod.example.com:443 -showcerts

The output lists the leaf ECDSA cert followed by an old intermediate that does not match the leaf’s Authority Key Identifier. The correct intermediate is stored in /etc/prometheus/certs/streaming-intermediate-new.pem, but Prometheus still references /etc/prometheus/certs/streaming-intermediate-old.pem.

Solution

Update Prometheus TLS configuration

Modify the scrape_config for the streaming job to:


# Before
scrape_configs:
  - job_name: streaming_metrics
    scheme: https
    static_configs:
      - targets: ["streaming-prod.example.com:443"]
    tls_config:
      ca_file: /etc/prometheus/certs/streaming-intermediate-old.pem
      insecure_skip_verify: false

Replace the old intermediate and enable TLS 1.3:


# After
scrape_configs:
  - job_name: streaming_metrics
    scheme: https
    static_configs:
      - targets: ["streaming-prod.example.com:443"]
    tls_config:
      ca_file: /etc/prometheus/certs/streaming-intermediate-new.pem
      min_version: TLS13
      max_version: TLS13
      insecure_skip_verify: false

Upgrade Prometheus binary (optional but recommended)

Prometheus 2.31.0 built with Go 1.17 does not enable TLS 1.3 by default. Deploy a newer version (≥2.38.0) compiled with Go 1.20, which enables TLS 1.3 out of the box.


# Example Helm upgrade
helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.version=v2.42.0 \
  --namespace monitoring

Reload configuration

After updating the ConfigMap (or Secret) that holds the TLS files, trigger a reload:


kubectl -n monitoring exec prometheus-0 -- kill -HUP 1

Why the fix works

Pointing ca_file at the correct intermediate allows the client to build a valid trust chain for the new ECDSA leaf.
Setting min_version and max_version to TLS13 forces the Go TLS stack to negotiate TLS 1.3, matching the server’s advertised capabilities.
Upgrading Prometheus ensures future Go runtime defaults include TLS 1.3, reducing the need for explicit version constraints.

Verify

Successful scrape

Check Prometheus target status:


kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 &
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.scrapeUrl|contains("streaming-prod"))'

Expected JSON fragment:


{
  "scrapeUrl":"https://streaming-prod.example.com:443/metrics",
  "health":"up",
  "lastError":"",
  "lastScrape":"2026-10-19T08:20:12.345Z",
  "lastScrapeDuration":0.123
}

OpenSSL confirmation


openssl s_client -connect streaming-prod.example.com:443 -tls1_3 -servername streaming-prod.example.com

The handshake completes, and the certificate chain validates against the new intermediate.

Metrics availability


curl -s https://streaming-prod.example.com/metrics | head -n 5

Output should show typical streaming metrics, e.g.:


# HELP streaming_ingest_bytes_total Total bytes ingested
# TYPE streaming_ingest_bytes_total counter
streaming_ingest_bytes_total{topic="live"} 1.234e+09

Prevent

Automated certificate rotation checks: Add a post‑rotation job that runs openssl s_client against all TLS‑protected endpoints and verifies that the presented chain matches the expected CA bundle.
Prometheus TLS version policy: Set min_version: TLS12 globally and enforce max_version: TLS13 to avoid silent fallback to unsupported versions.
Configuration drift detection: Store ca_file references in a ConfigMap and use kubectl diff in CI pipelines to catch stale paths after certificate updates.
Upgrade cadence: Keep Prometheus (and the underlying Go runtime) within two minor releases of the upstream project to benefit from TLS improvements and default secure settings.
Observability: Create a Prometheus rule that alerts on scrape_duration_seconds{job="streaming_metrics"} > 5 or on any up{job="streaming_metrics"} == 0 for more than two consecutive scrapes.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why did other services (e.g., Grafana) continue to scrape the endpoint? Those services were built with newer libraries (Go 1.20, Java 11) that enable TLS 1.3 by default, so they could negotiate the server’s only supported version.
Can I keep the old RSA certificate as a fallback? Yes, configure the streaming service to present a dual‑cert chain (RSA + ECDSA) and ensure the client’s tls_config allows both TLS 1.2 and TLS 1.3. However, this adds operational complexity and should be documented.
What if I cannot upgrade Prometheus immediately? Set the environment variable GODEBUG=tls13=1 on the Prometheus container to enable TLS 1.3 support in Go 1.17, then reload the configuration.
How do I verify which intermediate certificate Prometheus is actually loading? Use the promtool test rules command with the --debug flag, or exec into the pod and run:


cat /etc/prometheus/certs/streaming-intermediate-*.pem | openssl x509 -noout -subject -issuer -fingerprint

Is the handshake failure ever caused by client certificate authentication? In this scenario the streaming service does not request client certificates. If tls_config.cert_file and key_file are set incorrectly, the handshake would fail with “tls: bad certificate” rather than “handshake failure”. Ensure those fields are omitted unless mutual TLS is required.