Transformer robot

Hugging Face Transformers SSL/TLS handshake failure in Kubernetes

by

Hugging Face Transformers SSL/TLS Handshake Failure in a Canary Deployment on Kubernetes

Problem Description

During a staged rollout of a model‑inference service that uses Hugging Face Transformers, a subset of pods (the canary) repeatedly fail to load pretrained models from https://huggingface.co. The failure manifests as:

  • Log entry from the Transformers library: 
    urllib3.exceptions.SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /... (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate')))
  • Envoy sidecar log (Istio): 
    [2023-11-07T12:34:56.789Z] "tls: handshake failure" reason=TLS_HANDSHAKE_FAILED
  • Application error when calling transformers.AutoModel.from_pretrained(): 
    Connection aborted. Retry limit exceeded

The issue is isolated to the canary pods; the baseline version continues to download models successfully.

Root Cause Analysis

The failure originates from the interaction between:

  1. Transformers library – uses the requests library, which validates the server certificate against the process’s trusted CA bundle (official docs).
  2. Kubernetes service mesh (Istio) – injects an Envoy sidecar that terminates outbound TLS. By default, Envoy validates the remote certificate against the mesh’s root CA store (Istio docs).
  3. Hugging Face Hub certificate chain – the Hub presents a chain anchored by a public CA that is not present in the mesh’s trust bundle.

In the canary rollout a new DestinationRule was applied that forced tls.mode: MUTUAL for all outbound traffic. Envoy therefore attempted a mutual TLS handshake with the Hub, expecting a client certificate signed by the mesh’s CA. The Hub, which only supports server‑side TLS, responded with its public certificate, which Envoy could not verify because the mesh’s root CA bundle lacked the Hub’s intermediate certificates. The resulting error is the “CERTIFICATE_VERIFY_FAILED” seen by requests and the “tls: handshake failure” in Envoy logs.

Evidence from real incidents confirms this pattern:

  • Fintech canary rollout where 10 % of pods failed due to missing Hub CA in the mesh (incident report).
  • GitHub issue #12456 where users resolved the same error by adding the Hub CA to the mesh trust store (GitHub).

Investigation and Debugging Steps

  1. Confirm the failure is mesh‑related. 
    # Inside a failing pod
kubectl exec -it $POD -- curl -v https://huggingface.co

    Expected output shows SSL certificate problem: unable to get local issuer certificate. If the same command succeeds from a pod without the sidecar, the mesh is implicated.

  2. Inspect Envoy configuration. 
    # Retrieve the sidecar config dump
kubectl exec -it $POD -c istio-proxy -- curl -s localhost:15000/config_dump | grep -i tls

    Look for entries where tls_context has mode: MUTUAL for the huggingface.co host.

  3. Check the mesh’s trusted CA bundle. 
    # List CA certificates mounted in the sidecar
kubectl exec -it $POD -c istio-proxy -- ls /etc/istio/certs/

    If the Hub’s root/intermediate CA is missing, verification will fail.

  4. Validate the certificate chain presented by the Hub. 
    openssl s_client -showcerts -connect huggingface.co:443

    The output should include the full chain. Compare it against the certificates in /etc/istio/certs/.

  5. Review the applied DestinationRule. 
    kubectl get destinationrule -n $NAMESPACE -o yaml | grep -A5 'host: "huggingface.co"'

    If a generic rule with tls.mode: MUTUAL applies, it explains the forced client‑auth attempt.

Resolution

Three complementary fixes are presented. Choose the one that best fits your security posture.

Option 1 – Exempt the Hugging Face Hub from mesh mTLS

Modify or add a DestinationRule that disables mutual TLS for the Hub host.

# Before (problematic rule)
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: outbound-mtls-all
spec:
  host: "*.example.com"
  trafficPolicy:
    tls:
      mode: MUTUAL
      clientCertificate: /etc/istio/certs/cert-chain.pem
      privateKey: /etc/istio/certs/key.pem
      caCertificates: /etc/istio/certs/root-cert.pem

# After – add an explicit rule for huggingface.co
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: huggingface-external
  namespace: $NAMESPACE
spec:
  host: "huggingface.co"
  trafficPolicy:
    tls:
      mode: DISABLE   # or SIMPLE if you only need server verification

Applying this rule ensures Envoy forwards TLS unchanged to the Hub, allowing the requests library to perform its own verification.

Option 2 – Extend the Mesh Trust Store with the Hub CA

If you prefer to keep mutual TLS for all outbound traffic, import the Hub’s root and intermediate certificates into the mesh’s CA bundle.

# Retrieve Hub chain (example)
openssl s_client -showcerts -connect huggingface.co:443

Save the chain to a file huggingface-ca.pem and update the mesh config map (Istio uses istio-ca-root-cert config map).

# Append to the existing bundle
kubectl -n istio-system edit configmap istio-ca-root-cert
# Add the PEM contents under data.root-cert.pem

Restart the sidecar pods to reload the bundle:

kubectl rollout restart deployment/$DEPLOYMENT -n $NAMESPACE

Option 3 – Disable verification in the Transformers client (temporary)

For debugging or short‑lived canary runs you can bypass verification:

# Export environment variable before launching the service
export HF_HUB_DISABLE_TLS_VERIFY=1
# Or set a custom CA bundle
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt

**Warning:** This disables certificate validation and should never be used in production.

Verification

After applying the chosen fix, repeat the earlier curl test:

kubectl exec -it $POD -- curl -v https://huggingface.co

Successful output should include HTTP/2 200 and no SSL errors. Additionally, confirm that Transformers can load a model:

python - <<'PY'
from transformers import AutoModel
model = AutoModel.from_pretrained("distilbert-base-uncased")
print("model loaded:", model.__class__.__name__)
PY

Logs should no longer contain CERTIFICATE_VERIFY_FAILED or tls: handshake failure. Monitor the canary rollout until 100 % traffic is shifted and verify that no new TLS errors appear in istio-proxy logs.

Prevention and Best Practices

  • Scope mTLS rules narrowly. Avoid wildcard *.example.com rules that unintentionally affect external services.
  • Maintain a separate trust store for external public services. Add well‑known public CAs to the mesh bundle or use tls.mode: SIMPLE for outbound HTTPS.
  • Automate CA bundle updates. Use a ConfigMap watcher or CI pipeline to refresh the mesh’s root‑cert ConfigMap when public CAs rotate.
  • Include TLS health checks in CI. Run a quick curl or openssl s_client test against external endpoints as part of deployment validation.
  • Document exceptions. Keep a manifest of DestinationRule overrides for public APIs (e.g., Hugging Face Hub, S3, Docker registries) to avoid accidental policy regressions.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the error appear only in the canary pods?
    The canary deployment introduced a new DestinationRule that applied tls.mode: MUTUAL globally. Existing pods still used the older rule set, so only the updated pods hit the handshake failure.
  2. Can I use HF_ENDPOINT to point to an internal mirror of the Hub?
    Yes. Setting HF_ENDPOINT to an internal HTTPS endpoint that presents a certificate signed by a CA trusted by the mesh eliminates the external verification step. Remember to update the mesh trust store if the internal endpoint uses a private CA.
  3. What if the service mesh is Linkerd instead of Istio?
    Linkerd also validates outbound TLS against its trust anchors. The same mitigation applies: add the Hub’s intermediate certificates to Linkerd’s trust bundle or create a ServiceProfile that disables TLS verification for huggingface.co.
  4. Is disabling HF_HUB_DISABLE_TLS_VERIFY safe for production?
    No. It disables all server‑certificate verification, exposing the service to man‑in‑the‑middle attacks. Use it only for temporary debugging.
  5. How can I confirm which CA bundle Envoy is using?
    Exec into the sidecar and inspect /etc/istio/certs/root-cert.pem. You can also query Envoy’s admin API:

    curl localhost:15000/debug/ca

    which lists the loaded trust anchors.