Rows of black server racks with white logos in a data center

LangChain SSL handshake failure after Kubernetes service mesh upgrade

by

Problem

After upgrading the Kubernetes service mesh (Istio 1.18 → 1.20), LangChain‑based microservices that call external Large Language Model (LLM) APIs over HTTPS start failing with SSL/TLS handshake errors. The failures manifest as connection timeouts and requests.exceptions.SSLError exceptions in the Python runtime.

Typical Symptoms

  • Python traceback in LangChain worker pods:
Traceback (most recent call last):
  File "/app/chain.py", line 42, in run
    response = openai.ChatCompletion.create(...)
  File "/usr/local/lib/python3.11/site-packages/openai/api_requestor.py", line 210, in request
    raise error.SSLError("SSL handshake failed")
openai.error.SSLError: SSL handshake failed
  • Istio sidecar logs contain:
2024-09-30T12:45:12.345Z	istio-proxy	[warning]	SSL handshake error: tls: unknown certificate authority
2024-09-30T12:45:12.350Z	istio-proxy	[debug]	Outbound request to https://api.openai.com/v1/chat/completions blocked
  • Prometheus metric istio_tcp_sent_bytes_total shows zero bytes for egress traffic to api.openai.com.
  • Application health checks report “unhealthy – external LLM unreachable”.

Root Cause

The mesh upgrade enabled strict mTLS enforcement for all outbound traffic. By default, Istio now attempts to terminate TLS on the egress sidecar and re‑encrypt using its own certificates. External LLM providers expect a direct TLS handshake with a public CA‑signed certificate and specific SNI values. The sidecar’s self‑signed “istio‑egress” certificate is not trusted by the remote endpoint, causing the TLS handshake to abort with “unknown certificate authority”.

Additionally, the upgrade changed the default meshConfig.outboundTrafficPolicy.mode from ALLOW_ANY to REGISTRY_ONLY. Since the external LLM domains are not registered in the ServiceEntry, the sidecar drops the request before any TLS negotiation occurs.

Debug

1. Verify that the sidecar intercepts outbound HTTPS

# Exec into a LangChain pod
kubectl exec -it $(kubectl get pod -l app=langchain-worker -o jsonpath='{.items[0].metadata.name}') -- /bin/sh

# Use curl with -v to see TLS details
curl -v https://api.openai.com/v1/models 2>&1 | grep "SSL"

Expected output (when mesh is bypassed):

*   Trying 104.18.12.34:443...
* Connected to api.openai.com (104.18.12.34) port 443 (#0)
* TLS 1.3 connection using TLS_AES_256_GCM_SHA384
* Server certificate: CN=api.openai.com, O=OpenAI, C=US

Observed output (with mesh intercept):

*   Trying 10.96.0.1:443...
* Connected to api.openai.com (10.96.0.1) port 443 (#0)
* SSL handshake failed: unknown certificate authority

2. Inspect Istio sidecar logs for TLS errors

kubectl logs -l app=langchain-worker -c istio-proxy | grep "SSL handshake"

3. Check ServiceEntry for the external LLM domain

kubectl get serviceentry openai -o yaml

If the ServiceEntry does not exist, the egress request will be blocked under REGISTRY_ONLY mode.

4. Confirm mesh config changes

kubectl -n istio-system get configmap istio -o yaml | grep outboundTrafficPolicy

Output after upgrade:

outboundTrafficPolicy:
  mode: REGISTRY_ONLY

Solution

Option A – Configure an Egress Gateway with TLS Origination

Route external LLM traffic through an Istio Egress Gateway that performs TLS origination using a trusted public certificate.

# 1. Create ServiceEntry for the LLM endpoint
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: openai
spec:
  hosts:
  - api.openai.com
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL
---
# 2. Define a Gateway for egress
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: openai-egress-gateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: TLS
    hosts:
    - api.openai.com
    tls:
      mode: PASSTHROUGH
---
# 3. Create a VirtualService that directs traffic to the gateway
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: openai-egress
spec:
  hosts:
  - api.openai.com
  gateways:
  - openai-egress-gateway
  - mesh
  tcp:
  - match:
    - port: 443
    route:
    - destination:
        host: api.openai.com
        port:
          number: 443
        subset: external

With PASSTHROUGH mode, the egress gateway forwards the original TLS handshake unchanged, preserving the correct SNI and certificate chain.

Option B – Disable mTLS for Specific External Calls

If an egress gateway is not desirable, annotate the pod to bypass sidecar TLS for the LLM host.

# Add the following annotation to the LangChain Deployment
metadata:
  annotations:
    traffic.sidecar.istio.io/excludeOutboundIPRanges: "api.openai.com/32"

Or create a DestinationRule that disables TLS for the host:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: openai-disable-mtls
spec:
  host: api.openai.com
  trafficPolicy:
    tls:
      mode: DISABLE

Option C – Revert Mesh Policy (Temporary)

For a quick rollback, set outboundTrafficPolicy.mode back to ALLOW_ANY:

kubectl -n istio-system edit configmap istio
# Change:
#   mode: REGISTRY_ONLY
# to:
#   mode: ALLOW_ANY
kubectl rollout restart deployment -n istio-system istiod

Note: This re‑enables unrestricted outbound traffic and should only be used as a stop‑gap.

Verify

1. Successful TLS handshake from the pod

# Inside the LangChain pod
curl -v https://api.openai.com/v1/models 2>&1 | grep "TLS"

Expected output:

* TLS 1.3 connection using TLS_AES_256_GCM_SHA384
* Server certificate: CN=api.openai.com

2. Application logs show no SSL errors

kubectl logs -l app=langchain-worker | grep "SSLError" || echo "No SSL errors"

Output:

No SSL errors

3. Istio metrics indicate egress traffic

curl -s http://prometheus.istio-system.svc:9090/api/v1/query?query=istio_tcp_sent_bytes_total{destination_service=\"api.openai.com\"}

Non‑zero byte count confirms traffic is flowing.

Prevent

  • Explicit ServiceEntry for every external dependency. Keep a version‑controlled manifest of all egress ServiceEntries.
  • Use an Egress Gateway for TLS origination. This isolates external TLS handling and avoids sidecar certificate mismatches.
  • Monitor handshake failures. Create a Prometheus alert on istio_tcp_ssl_handshake_errors_total with a threshold of >0 over 5 minutes.
  • Pin mesh configuration changes. When upgrading Istio, review release notes for default policy changes (e.g., outboundTrafficPolicy).
  • Validate CA bundles. Ensure the sidecar’s trust store includes the public root CAs required by external providers, or mount a custom ca-bundle.crt into the sidecar.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

  1. Why does the handshake succeed when I run curl from a node but fail from inside the pod?
    Because the node bypasses the Istio sidecar, while the pod’s outbound traffic is intercepted and re‑encrypted with the sidecar’s certificate, which the remote server does not trust.
  2. Can I keep mTLS for internal services and still talk to external APIs?
    Yes. Use a ServiceEntry + DestinationRule to set tls.mode: DISABLE or route through an egress gateway with PASSTHROUGH TLS, leaving internal mesh traffic unchanged.
  3. Do I need to update the LangChain library after fixing the mesh?
    No code change is required in LangChain once the mesh correctly forwards the TLS handshake. However, ensure the library’s HTTP client respects the system proxy settings if an egress gateway is used.
  4. What if the external provider uses mutual TLS (mTLS) instead of standard TLS?
    Create a DestinationRule with tls.mode: MUTUAL and provide the client certificate and key via a Kubernetes secret mounted into the egress gateway.
  5. How can I test that the sidecar is no longer terminating TLS?
    Run openssl s_client -connect api.openai.com:443 -servername api.openai.com -showcerts inside the pod. The output should display the remote server’s certificate chain, not the Istio egress certificate.