Weaviate Service Unavailable After TLS Certificate Expiration

Q: Debug – Investigation Steps 1. Verify Certificate Expiry

openssl s_client -connect weaviate.example.com:443 -servername weaviate.example.com Expected output snippet when expired:

Q: 2. Inspect Ingress TLS Secret

kubectl get secret weaviate-tls -n weaviate -o yaml | grep notAfter If the notAfter date is in the past, the secret is stale.

Q: 3. Check cert‑manager Certificate Resource

kubectl describe certificate weaviate-cert -n weaviate Look for Ready: True and a recent Renewal Time. If the certificate is Ready but the secret still shows the old dates, the secret update did not propagate.

Q: 4. Review Weaviate Pod Logs

kubectl logs -l app=weaviate -n weaviate --tail=50 Typical error log:

Q: 5. Confirm Ingress Controller Status

kubectl describe ingress weaviate-ingress -n weaviate Check for events such as “TLS secret weaviate-tls was not updated” or “configuration reload failed”.

Q: 6. Verify ALB Listener (AWS only)

aws elbv2 describe-listeners --load-balancer-arn $ALB_ARN --query 'Listeners[*].Certificates' If the ARN points to an expired ACM certificate, the ALB will return 502/504.

Q: Option B: Manual Pod Restart (quick fix)

kubectl rollout restart deployment weaviate -n weaviate This forces all Weaviate pods to reload the updated secret. Use only as a temporary measure while implementing automated reload.

In this article

Table of Contents

Problem – Weaviate Service Unavailable After TLS Certificate Expiration

In a cloud‑managed Weaviate deployment (e.g., on AWS EKS or GKE) the API endpoint becomes unreachable shortly after the TLS/SSL certificate used by the Ingress controller or the ALB expires. Clients see errors such as:

x509: certificate has expired or is not yet valid
TLS handshake timeout – client failed to negotiate a secure connection
HTTP 502 Bad Gateway
connection refused / EOF while reading response body

Typical symptoms include:

HTTP 502/504 responses from the Ingress/Nginx controller.
Weaviate logs showing “error loading TLS keypair: tls: failed to find any PEM data in certificate input”.
Client libraries reporting “TLS handshake failed”.

Root Cause – Stale TLS Secret After Automatic Renewal

The underlying mechanism for TLS in a managed Weaviate service is:

The cloud provider (AWS ACM or Let’s Encrypt via cert‑manager) issues a certificate and stores it in a Kubernetes Secret (or directly on an ALB).
The Ingress controller (NGINX, ALB) mounts this secret and presents the certificate to clients.
Weaviate pods obtain the certificate either directly from the secret (when TLS termination is inside the pod) or rely on the Ingress termination.

When the certificate reaches its notAfter date, the provider automatically renews it. However, two common gaps cause the outage:

Kubernetes secret not refreshed – cert‑manager updates the secret, but the Deployment is not annotated to trigger a rolling restart. The Weaviate containers continue using the in‑memory, expired certificate.
ALB/Ingress does not pick up the new secret – the load balancer caches the TLS bundle and serves the expired cert until the target group is re‑registered or the Ingress controller is reloaded.

These gaps are documented in the Weaviate TLS configuration guide (Weaviate Documentation – TLS Configuration guide) and the Helm chart values for automatic provisioning (Weaviate Helm Chart – TLS and Ingress settings). Real incidents (e.g., “AWS ACM certificate for the ALB front‑ending a managed Weaviate cluster expired” and “Let’s Encrypt renewal failed due to rate‑limit”) confirm that the failure mode is a stale secret combined with a missing pod restart.

Debug – Investigation Steps

1. Verify Certificate Expiry

openssl s_client -connect weaviate.example.com:443 -servername weaviate.example.com

Expected output snippet when expired:

...
Verify return code: 10 (certificate has expired)
...

2. Inspect Ingress TLS Secret

kubectl get secret weaviate-tls -n weaviate -o yaml | grep notAfter

If the notAfter date is in the past, the secret is stale.

3. Check cert‑manager Certificate Resource

kubectl describe certificate weaviate-cert -n weaviate

Look for Ready: True and a recent Renewal Time. If the certificate is Ready but the secret still shows the old dates, the secret update did not propagate.

4. Review Weaviate Pod Logs

kubectl logs -l app=weaviate -n weaviate --tail=50

Typical error log:

error loading TLS keypair: tls: failed to find any PEM data in certificate input
x509: certificate has expired or is not yet valid

5. Confirm Ingress Controller Status

kubectl describe ingress weaviate-ingress -n weaviate

Check for events such as “TLS secret weaviate-tls was not updated” or “configuration reload failed”.

6. Verify ALB Listener (AWS only)

aws elbv2 describe-listeners --load-balancer-arn $ALB_ARN --query 'Listeners[*].Certificates'

If the ARN points to an expired ACM certificate, the ALB will return 502/504.

Solution – Restoring Service Availability

Option A: Rolling Restart Triggered by Secret Annotation (Kubernetes)

Annotate the Deployment so that a change in the TLS secret forces a pod restart. This is the recommended fix in the Weaviate Helm chart (GitHub Issue #987).

# Before (helm values.yaml)
deployment:
  annotations: {}

# After – add secret hash annotation
deployment:
  annotations:
    secret.reloader.stakater.com/reload: "weaviate-tls"

Apply the change:

helm upgrade weaviate ./weaviate-chart -f values.yaml --namespace weaviate

The secret-reloader sidecar (or Stakater’s reloader) watches the secret and restarts the pod when the secret changes.

Option B: Manual Pod Restart (quick fix)

kubectl rollout restart deployment weaviate -n weaviate

This forces all Weaviate pods to reload the updated secret. Use only as a temporary measure while implementing automated reload.

Option C: Force Ingress Reload (NGINX)

kubectl annotate ingress weaviate-ingress \
  kubernetes.io/ingress.class=nginx \
  -n weaviate --overwrite

Annotating forces NGINX to re‑read the secret. For AWS ALB, re‑associate the new ACM certificate:

aws elbv2 modify-listener --listener-arn $LISTENER_ARN \
  --certificates CertificateArn=$NEW_CERT_ARN

Option D: Ensure cert‑manager Updates the Secret

If cert‑manager is not updating the secret, add a renewBefore field and verify the Issuer configuration:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: weaviate-cert
spec:
  secretName: weaviate-tls
  dnsNames:
  - weaviate.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  renewBefore: 48h   # renew at least 2 days before expiry

Verify – Confirming the Fix

TLS Handshake Test

openssl s_client -connect weaviate.example.com:443 -servername weaviate.example.com

Expect Verify return code: 0 (ok).

Ingress Health Check
```
curl -k https://weaviate.example.com/v1/.well-known/ready
```
Response should be 200 OK with a JSON readiness payload.
Weaviate Log Scan
```
kubectl logs -l app=weaviate -n weaviate | grep -i tls
```
No error lines about expired certificates should appear.

Metrics Validation (if Prometheus is enabled)

curl -s http://prometheus.weaviate.svc:9090/api/v1/query?query=weaviate_http_requests_total | jq .

Observe that error rate drops to zero.

Prevent – Operational Guardrails

Guardrail	Implementation
Automated Secret Reload	Deploy `secret-reloader` or use `kubectl.kubernetes.io/restartedAt` annotation on Deployments; set `secret.reloader.stakater.com/reload` as shown above.
Certificate Expiry Monitoring	Prometheus `certmanager_certificate_expiration_timestamp_seconds` alert – fire `warning` 30d before expiry, `critical` 7d before.
Ingress Reload Hook	Enable `nginx.ingress.kubernetes.io/force-ssl-redirect` and set `nginx.ingress.kubernetes.io/rewrite-target` to trigger reload on secret change.
ALB Certificate Rotation	Use AWS ACM automatic renewal and attach a Lambda that calls `aws elbv2 modify-listener` on `CertificateRenewal` events.
Rate‑Limit Awareness	Monitor Let’s Encrypt rate limits (`cert-manager` logs) and pre‑emptively request larger quota or use a private CA for internal clusters.

FAQ – Common Follow‑Up Questions

Why does the Ingress return 502 only after the cert expires, not before?
Because the Ingress controller validates the TLS secret before establishing a backend connection. An expired cert causes the controller to abort the TLS handshake, resulting in a 502 from the load balancer.
Can I avoid pod restarts by terminating TLS inside the Ingress only?
Yes, if you terminate TLS at the Ingress and do not enable --tls-cert-file in the Weaviate container. However, you must still ensure the Ingress reloads the secret, which the secret-reloader or an annotation can automate.
How do I verify which certificate the ALB is currently serving?
Run aws elbv2 describe-listeners … --query 'Listeners[*].Certificates[*].CertificateArn' and compare the ARN to the ACM console. You can also use openssl s_client against the ALB endpoint.
What if cert‑manager fails to renew because of a rate limit?
Inspect kubectl logs -n cert-manager deploy/cert-manager for “rate limit exceeded”. Mitigate by using a staging issuer for testing, increasing the quota, or switching to a private CA for internal services.
Is there a way to test certificate rotation in a staging environment?
Create a short‑lived Certificate resource (e.g., duration: 24h) and observe the secret update flow. Verify that the Deployment reloads automatically using the same annotations as production.

Related Topic Hub: Vector Databases Troubleshooting Hub