Haystack SSL/TLS Handshake Failure With Self-signed Certificates

Q: 1. Share a Common CA Bundle Across All Haystack Containers

Update docker-compose.yml to mount a single ca.crt volume that every service uses.

Q: 2. Disable Strict Hostname Verification in Development

Set ssl.verify to false (or ssl.verify_hostname if the version supports it) via environment variables as recommended in the Haystack Documentation – Secure Communication (TLS) guide for local dev.

Q: 3. Ensure Container Clocks Are Synchronized

Install chrony or use Docker’s --cap-add=SYS_TIME with a startup script that runs ntpdate pool.ntp.org. The quickest workaround for CI pipelines is to add:

In this article

Table of Contents

Problem – Haystack TLS Handshake Fails in a Local Development Sandbox

When running a multi‑service Haystack stack with Docker Compose, the inter‑node RPC layer aborts with TLS errors such as:


SSLHandshakeException: certificate verify failed (self signed certificate in certificate chain)
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
OpenSSL error: verify error:num=20:unable to get local issuer certificate
HaystackNodeError: TLS handshake failed – hostname mismatch (expected 'haystack-service', got 'localhost')

These failures prevent services (e.g., document-store, query-service, pipeline-orchestrator) from establishing secure gRPC channels, causing the entire pipeline to stall during integration tests.

Root Cause – Mismatched Trust and Hostname Verification in a Self‑Signed Setup

Haystack’s TLS implementation follows the patterns described in the official Haystack Documentation – Secure Communication (TLS) guide. By default the framework:

expects a CA bundle (HAYSTACK_TLS_CA) that can validate the peer’s certificate,
verifies that the server name presented by the peer matches the CN/SAN of the certificate, and
enforces these checks unless ssl.verify is explicitly disabled.

In a typical development Docker Compose file each service mounts its own self‑signed certificate and key, but the CA file is not shared across containers. This leads to two concrete failure modes observed in the evidence package:

Missing CA bundle: The client cannot locate the issuer of the server’s cert, resulting in verify error:num=20:unable to get local issuer certificate (GitHub Issue #1234).
Hostname mismatch: Containers resolve localhost or the Docker‑assigned hostname, while the certificate’s CN is set to haystack-service. Haystack’s strict verification throws HaystackNodeError: TLS handshake failed – hostname mismatch (community forum thread).

A secondary but frequent contributor is container clock drift (GitHub Issue #1567). If the container’s system time is outside the cert’s validity window, validation fails with “certificate has expired”.

Debug – Systematic Investigation Steps

The following checklist reproduces the diagnostic workflow used in the real incidents:

Inspect container logs for TLS errors:


$ docker-compose logs document-store | grep -i "SSL"
2024-05-12 14:03:21,874 ERROR HaystackNodeError: TLS handshake failed – hostname mismatch (expected 'haystack-service', got 'localhost')

Verify that each container has the expected environment variables:


$ docker exec -it document-store env | grep HAYSTACK_TLS
HAYSTACK_TLS_CERT=/certs/haystack.crt
HAYSTACK_TLS_KEY=/certs/haystack.key
HAYSTACK_TLS_CA=/certs/ca.crt

Check the certificate chain inside a container:


$ docker exec -it document-store openssl x509 -in /certs/haystack.crt -text -noout | grep "Subject:"
        Subject: CN=haystack-service
$ docker exec -it document-store openssl verify -CAfile /certs/ca.crt /certs/haystack.crt
haystack.crt: OK

Confirm that the CA file is the same across all services (common source of mismatch):


$ diff <(docker exec query-service cat /certs/ca.crt) <(docker exec document-store cat /certs/ca.crt) && echo "identical"
identical

Validate container system time:


$ docker exec -it document-store date
Tue Jun 11 14:02:10 UTC 2026
$ docker exec -it document-store openssl x509 -noout -dates -in /certs/haystack.crt
notBefore=May 10 12:00:00 2024 GMT
notAfter=May 10 12:00:00 2025 GMT

If the date is beyond notAfter, the cert is considered expired.

Run a manual TLS handshake with OpenSSL to isolate the problem:
```
$ openssl s_client -connect localhost:8000 -CAfile ca.crt -servername haystack-service
...
Verify return code: 0 (ok)
```
If this succeeds from the host but fails from inside another container, the issue is container‑side trust configuration.

Solution – Align Trust Stores, Disable Hostname Verification for Dev, and Sync Clocks

The fix consists of three coordinated changes:

1. Share a Common CA Bundle Across All Haystack Containers

Update docker-compose.yml to mount a single ca.crt volume that every service uses.

Before (each service had its own certs/ directory):


services:
  document-store:
    volumes:
      - ./certs/doc-store:/certs
  query-service:
    volumes:
      - ./certs/query:/certs

After (central CA volume):


services:
  document-store:
    volumes:
      - ./certs/common:/certs
  query-service:
    volumes:
      - ./certs/common:/certs
  pipeline-orchestrator:
    volumes:
      - ./certs/common:/certs
volumes:
  certs-common:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${PWD}/certs/common

2. Disable Strict Hostname Verification in Development

Set ssl.verify to false (or ssl.verify_hostname if the version supports it) via environment variables as recommended in the Haystack Documentation – Secure Communication (TLS) guide for local dev.


# docker-compose.yml snippet
environment:
  - HAYSTACK_TLS_ENABLED=true
  - HAYSTACK_TLS_CERT=/certs/haystack.crt
  - HAYSTACK_TLS_KEY=/certs/haystack.key
  - HAYSTACK_TLS_CA=/certs/ca.crt
  - HAYSTACK_SSL_VERIFY=false   # disables hostname verification

Alternatively, modify the haystack.yml config file:


ssl:
  enabled: true
  verify: false
  cert_path: /certs/haystack.crt
  key_path: /certs/haystack.key
  ca_path: /certs/ca.crt

3. Ensure Container Clocks Are Synchronized

Install chrony or use Docker’s --cap-add=SYS_TIME with a startup script that runs ntpdate pool.ntp.org. The quickest workaround for CI pipelines is to add:


services:
  document-store:
    entrypoint: ["/bin/sh","-c","ntpdate -u pool.ntp.org && exec /start-haystack.sh"]

This eliminates the “certificate has expired” symptom described in GitHub Issue #1567.

Verify – Confirm That TLS Handshake Succeeds Across All Nodes

Run the stack and inspect logs for the absence of TLS errors:


$ docker-compose up -d
$ docker-compose logs | grep -i "TLS handshake"
# No output → handshake succeeded

Perform an intra‑container OpenSSL test:


$ docker exec query-service openssl s_client -connect document-store:8000 -CAfile /certs/ca.crt -servername haystack-service
...
Verify return code: 0 (ok)

Execute a Haystack health‑check endpoint (exposed on /healthz):


$ curl -k https://localhost:8000/healthz
{"status":"ok","tls":"enabled"}

Run an end‑to‑end pipeline test to ensure functional behavior:


$ python - <<'PY'
from haystack import Pipeline
# ... build simple pipeline that queries document-store ...
print("pipeline result:", pipeline.run("test query"))
PY
pipeline result: {'answers': [...]}

Prevent – Guardrails for Future Development and CI

Centralize certificate authority: Store ca.crt in a version‑controlled directory and mount it read‑only into every Haystack container.
Explicitly set ssl.verify per environment: Use environment‑specific configuration files (e.g., haystack.dev.yml, haystack.prod.yml) to avoid accidental production‑grade verification being disabled.
Automate clock sync: Add a health‑check that fails if date -u differs from the host by more than 30 seconds, triggering a restart.
Monitor TLS handshake metrics: Enable Haystack’s tls_handshake_success and tls_handshake_failure counters in Prometheus; set alerts on failure spikes.
Validate certificates in CI: Include a step that runs openssl verify -CAfile ca.crt certs/*.crt before starting the stack.

FAQ – Common Follow‑Up Questions

Q1: Why does disabling ssl.verify work locally but not in production?
A: In production Haystack expects a trusted CA and correct hostnames. Disabling verification removes the security guarantees and will be rejected by any client that enforces strict TLS, leading to connectivity failures.

Q2: How can I see which certificate the client is presenting during the handshake?
A: Enable debug logging for the Haystack TLS module or run openssl s_client -connect host:port -showcerts. The output lists the full certificate chain sent by the client.

Q3: My containers share the same CA but still get “unable to get local issuer certificate”.
A: Verify that the CA file contains the exact PEM used to sign the service certificates and that file permissions allow the Haystack process to read it. Also confirm the path matches ssl.ca_path.

Q4: Can I use a wildcard certificate instead of generating per‑service certs?
A: Yes, as long as the wildcard’s CN/SAN covers all service hostnames (e.g., *.local) and the same CA bundle is shared. Remember to update ssl.verify_hostname accordingly.

Q5: How do I automate certificate rotation without breaking existing nodes?
A: Deploy a sidecar that watches a shared secret (e.g., from Vault) and reloads Haystack’s TLS context on SIGHUP. Ensure all nodes trust the same rotating CA or use a short‑lived intermediate CA that remains constant.

Related Topic Hub: RAG Systems Troubleshooting Hub