Anthropic Claude Certificate Expiration In Kubernetes Cluster

In this article

Table of Contents

Anthropic Claude Certificate Expiration in a Kubernetes Cluster

Problem Description (Symptoms and Impact)

Several microservices that call the Anthropic Claude API began failing with TLS‑related errors after a routine maintenance window. Typical log excerpts from affected pods include:


2024-03-12T14:22:07Z ERROR pod/claude-client-7f9c9d9c5b-ktz9l: x509: certificate has expired
2024-03-12T14:22:07Z ERROR pod/claude-client-7f9c9d9c5b-ktz9l: failed to do request to https://api.anthropic.com/v1/complete: Get "https://api.anthropic.com/v1/complete": TLS handshake timeout
2024-03-12T14:22:08Z WARN  ingress-nginx-controller-7c5d9b5c9d-8kzvf: SSL certificate for host api.anthropic.com is expired

Operational impact observed:

Upstream AI calls return HTTP 502/504, causing request timeouts in user‑facing services.
Background jobs that depend on Claude for content generation enter retry loops, increasing queue length.
Alerting thresholds for “Claude API error rate” breached, triggering incident response.

Root Cause Analysis

The failure originates from an expired TLS certificate used during the HTTPS handshake with api.anthropic.com. The Anthropic API authentication guide (Anthropic API Authentication Guide) specifies that clients must validate the server’s certificate chain against trusted roots. In this environment the cluster relied on an Ingress TLS secret that was manually created and not automatically rotated.

Key factors that led to the outage:

Static secret management: The secret anthropic-tls was populated once during the initial deployment (see Anthropic Claude Kubernetes Deployment Guide) and never refreshed.
cert-manager mis‑configuration: Although cert-manager was installed, the Certificate resource targeting api.anthropic.com was missing the issuerRef to an external CA, so renewal webhook never fired (as discussed in GitHub issue cert-manager/cert-manager #2741).
Sidecar proxy caching: A sidecar Envoy proxy cached the expired certificate in its TLS context, preventing the pod‑level OpenSSL library from re‑loading a refreshed secret.

Consequently, every TLS handshake attempted by the Claude SDK (see GitHub issue anthropic/claude-sdk #342) failed with the error x509: certificate has expired, matching the error codes listed in the Anthropic API Reference.

Investigation and Debugging Steps

The following procedure reproduces the diagnostic path taken during the incident:

Confirm the error originates from TLS validation:
```
kubectl logs -l app=claude-client -c app | grep "certificate has expired"
```
Expected output contains the snippet shown above.
Inspect the Ingress TLS secret:
```
kubectl get secret anthropic-tls -o yaml
```
If the tls.crt data decodes to a certificate whose NotAfter field is in the past, the secret is stale.

Validate the certificate chain manually:

kubectl exec -it $(kubectl get pod -l app=claude-client -o name | head -n1) -- \
  openssl s_client -connect api.anthropic.com:443 -servername api.anthropic.com

Expected output will end with verify return:1 for a valid cert; instead you will see certificate has expired.

Check cert-manager resources:
```
kubectl get certificate -n monitoring anthro-cert -o yaml
```
Look for status.conditions showing Ready=False and reason Failed.

Verify sidecar proxy configuration:

kubectl exec -it $(kubectl get pod -l app=claude-proxy -o name | head -n1) -- \
  curl -v https://api.anthropic.com/v1/complete

The curl output will contain SSL certificate problem: certificate has expired.

Resolution (Fix Implementation)

The fix consists of three coordinated actions: create a renewable Certificate resource, update the Ingress to reference the new secret, and reload the sidecar proxy.

Step 1 – Define a cert‑manager `Certificate` for the external API

Before (static secret):


apiVersion: v1
kind: Secret
metadata:
  name: anthropic-tls
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJ...
  tls.key: LS0tLS1CRUdJTiBSU0...

After (renewable cert‑manager resource):


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: anthro-cert
  namespace: monitoring
spec:
  secretName: anthropic-tls
  dnsNames:
  - api.anthropic.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  renewBefore: 720h # 30 days before expiry

This configuration tells cert-manager to obtain a certificate from Let’s Encrypt (or another trusted external CA) and store it in the anthropic-tls secret. The renewBefore field ensures renewal starts 30 days prior to expiry.

Step 2 – Patch the Ingress to use the renewed secret


kubectl patch ingress claude-ingress -p \
  '{"spec":{"tls":[{"hosts":["api.anthropic.com"],"secretName":"anthropic-tls"}]}}'

The Ingress controller (NGINX, Traefik, etc.) will automatically reload the TLS context when the secret changes, eliminating the need for a manual pod restart.

Step 3 – Reload the Envoy sidecar (if used)


kubectl rollout restart deployment claude-proxy

Restart forces the sidecar to read the updated secret from the mounted volume, clearing the cached expired certificate.

Validation (Verification Steps)

Confirm secret contains a valid cert:

kubectl get secret anthropic-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates

Output should show notAfter=... (future date).

Test TLS handshake from a pod:

kubectl exec -it $(kubectl get pod -l app=claude-client -o name | head -n1) -- \
  openssl s_client -connect api.anthropic.com:443 -servername api.anthropic.com

The final line should be Verify return code: 0 (ok).

Run a functional API call:

kubectl exec -it $(kubectl get pod -l app=claude-client -o name | head -n1) -- \
  curl -s -H "x-api-key: $ANTHROPIC_API_KEY" \
  -d '{"model":"claude-2.1","prompt":"Hello"}' \
  https://api.anthropic.com/v1/complete | jq .

Response should contain a valid JSON payload, not an error.

Check monitoring alerts: Verify that the “Claude API error rate” metric returns to baseline and that no new TLS‑related alerts fire.

Prevention (Best Practices and Guardrails)

Automate certificate lifecycle: Always use cert-manager (or an equivalent controller) for any TLS secret, even when the certificate is for an external service.
Enable alerting on secret age: Create a Prometheus rule that fires when kube_secret_created_timestamp_seconds exceeds (now() - 30d) for secrets used in TLS termination.
Validate certificates in CI/CD: Include a step that runs openssl s_client against target hosts to ensure the certificate is valid before promotion.
Sidecar reload policy: Configure the sidecar proxy with dynamic_certificates: true (Envoy) or equivalent so it watches the secret file for changes without a full pod restart.
Document renewal ownership: Assign a clear on‑call responsibility for cert-manager Certificate resources and verify that CertificateRequest events are not stuck in pending state.

FAQ (Related Questions)

Why does the Claude SDK return “x509: certificate has expired” only after a weekend?
Because the static TLS secret was created with a 90‑day certificate that expired on Saturday; the Ingress controller only reloads TLS material on secret change, so the failure persisted until the secret was manually updated.
Can I use a self‑signed certificate for api.anthropic.com?
No. The Anthropic API requires a publicly trusted certificate chain (see the Anthropic Security Best Practices). Self‑signed certs will be rejected during the TLS handshake.
How do I verify which certificate my pod is actually presenting?
Run openssl s_client -connect api.anthropic.com:443 -servername api.anthropic.com from inside the pod and inspect the Certificate chain section. The Not After field shows the expiry.
Why didn’t cert-manager automatically renew the secret?
The Certificate resource lacked a proper issuerRef to an external CA, so cert-manager never created a CertificateRequest. Adding a valid ClusterIssuer resolves this.
Is there a way to force a rolling restart of all Claude‑related deployments without downtime?
Yes. Use a kubectl rollout restart on the deployments after the secret is updated; the Ingress controller will serve the new cert immediately, and each pod will reload its sidecar on its own restart schedule.

Related Topic Hub: LLM Systems Troubleshooting Hub