a purple and black background with a purple and black logo

PyTorch model evaluation fails due to SSL certificate expiration

by

Problem – Model Evaluation Fails with SSL Certificate Expiration

During a distributed evaluation run, PyTorch attempts to download model checkpoints, dataset shards, or auxiliary assets from remote HTTPS endpoints (e.g., torch.hub, torch.utils.model_zoo, torchvision.datasets, or torch.distributed RPC). When the TLS certificate presented by the server has expired, the download aborts and the entire evaluation job terminates.

Typical error output observed in the job logs:

urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)
requests.exceptions.SSLError: HTTPSConnectionPool(host='models.example.com', port=443): Max retries exceeded with url: /my_model.pt (Caused by SSLError(SSLCertVerificationError("certificate verify failed: certificate has expired")))
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired
RuntimeError: Failed to download https://models.example.com/checkpoint.pt: SSL certificate problem: certificate has expired
torch.distributed.rpc.RpcError: SSL handshake failed – certificate verification failed (expired certificate)

These failures surface in environments such as:

  • Kubernetes clusters running torch.distributed training where checkpoint synchronization uses an HTTPS‑backed storage (AWS S3, Azure Blob, internal Nginx proxy).
  • CI pipelines that pull pretrained weights via torch.hub from the Hugging Face model hub.
  • DataLoader instances that download dataset shards from a private HTTPS repository.

Root Cause – Why the Expired Certificate Breaks PyTorch Downloads

PyTorch’s high‑level loading helpers delegate network I/O to Python’s standard libraries (urllib.request) or requests, which in turn rely on the OpenSSL verification chain. The verification process follows these steps (as described in the official torch.hub documentation and model_zoo documentation):

  1. Resolve the hostname and establish a TCP connection on port 443.
  2. Perform the TLS handshake; the server presents its X.509 certificate chain.
  3. OpenSSL checks each certificate’s notBefore and notAfter fields against the local clock.
  4. If any certificate in the chain is outside its validity period, OpenSSL aborts the handshake with CERTIFICATE_VERIFY_FAILED.
  5. The exception propagates up through urllib/requests and is re‑raised as a RuntimeError by PyTorch’s download helper.

When the remote endpoint’s TLS certificate expires (e.g., the Hugging Face model hub incident in March 2024), the verification step fails for every client that respects the default system CA bundle. Because PyTorch does not provide a built‑in “ignore SSL errors” flag, the failure is fatal unless the user intervenes.

Investigation – Debugging the SSL Failure

Follow these steps to confirm that the root cause is an expired certificate:

  1. Inspect the PyTorch traceback. The stack trace will end in a call to torch.utils.model_zoo.load_url or torch.hub.load_state_dict_from_url, wrapping a urllib.error.URLError or requests.exceptions.SSLError.
  2. Verify the certificate manually. Use openssl s_client to view the server’s certificate dates:
openssl s_client -connect models.example.com:443 -servername models.example.com

Typical output indicating expiration:

...
Verify return code: 10 (certificate has expired)
---
  • Check the local system clock. A skewed clock can produce false positives. Run:
    • date -u
  • Confirm the CA bundle used by Python. Print the path that certifi.where() returns and compare it with the OS bundle:
    • python - <<'PY'
import certifi, ssl, sys
print("certifi bundle:", certifi.where())
print("default SSLContext cafile:", ssl.get_default_verify_paths().cafile)
PY
  • Reproduce the failure with a minimal script. This isolates the problem from distributed runtime complexities:
    • python - <<'PY'
import torch
url = "https://models.example.com/checkpoint.pt"
try:
    torch.hub.load_state_dict_from_url(url, progress=False)
except Exception as e:
    print("Download failed:", e)
PY
  • Capture network traffic (optional). A tcpdump capture can confirm that the TLS handshake aborts before any application data is exchanged:
    • sudo tcpdump -i eth0 -w ssl_failure.pcap host models.example.com and port 443

    Resolution – Restoring Successful Model Evaluation

    The fix consists of either renewing the server certificate or adjusting the client side to trust a valid certificate chain. Below are three practical client‑side remedies that can be applied without waiting for the remote operator.

    1. Update the local CA bundle (preferred)

    If the server has already replaced its expired cert with a new one signed by a trusted CA, the client may be using an outdated CA bundle.

    Before:

    # Python process uses the OS bundle (may be missing the new root)
import ssl
print(ssl.get_default_verify_paths())

    After: Install the latest certifi package and point Python to it.

    # Upgrade certifi
pip install --upgrade certifi

# Export environment variable so urllib/requests use the updated bundle
export SSL_CERT_FILE=$(python -c "import certifi, sys; sys.stdout.write(certifi.where())")

    Rerun the evaluation; the download should succeed.

    2. Inject the renewed certificate manually

    If the new server certificate is self‑signed or signed by a private CA, add it to a custom bundle and reference it.

    Before (failure):

    RuntimeError: Failed to download https://internal-registry.company.com/model.pt: SSL certificate problem: certificate has expired

    After:

    # Save the new PEM‑encoded certificate as /etc/ssl/certs/company-ca.pem
cat > /etc/ssl/certs/company-ca.pem <<EOF
-----BEGIN CERTIFICATE-----
MIID...
-----END CERTIFICATE-----
EOF

# Combine with existing bundle (optional)
cat /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/company-ca.pem > /tmp/combined-ca.pem

# Export variable
export SSL_CERT_FILE=/tmp/combined-ca.pem

# Verify that Python sees the new bundle
python -c "import ssl, os; print(ssl.get_default_verify_paths())"

    3. Bypass verification temporarily (use with caution)

    When an immediate fix is required and security impact is acceptable (e.g., in an isolated test cluster), disable verification for the specific download call.

    Before (exception raised):

    urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

    After:

    import torch
import urllib3
import ssl

# Create an unverified SSL context
ctx = ssl._create_unverified_context()

# Monkey‑patch urllib to use the context
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
urllib3.util.ssl_.DEFAULT_SSL_CONTEXT = ctx

# Now load the checkpoint
url = "https://models.example.com/checkpoint.pt"
state_dict = torch.hub.load_state_dict_from_url(url, progress=False, trust_repo=True)

    Note: The trust_repo=True flag (available from PyTorch 1.12) tells torch.hub to skip the repository‑level integrity check, but SSL verification still occurs unless the global context is altered as shown.

    Verification – Confirming the Fix

    After applying one of the remedies, perform the following checks:

    1. Successful download. The script from the investigation step should now print a dictionary of tensors without raising an exception.
    2. Log inspection. Look for the absence of SSL: CERTIFICATE_VERIFY_FAILED messages in the job logs.
    3. Metrics. If you have a Prometheus metric such as pytorch_download_success_total, verify that the counter increments.
    4. Health endpoint. For distributed jobs, query the /healthz endpoint of each node; the status should be OK rather than SSL_ERROR.
    # Example verification script
import torch, sys
url = "https://models.example.com/checkpoint.pt"
try:
    torch.hub.load_state_dict_from_url(url, progress=False)
    print("✅ Download succeeded")
except Exception as e:
    print("❌ Still failing:", e)
    sys.exit(1)

    Prevention – Guardrails to Avoid Future Certificate‑Related Outages

    • Automated certificate monitoring. Use a tool like cert-manager (Kubernetes) or a dedicated monitoring job that runs openssl s_client -connect $HOST:443 -servername $HOST daily and alerts on Verify return code: 10.
    • Pin certificates. For critical internal registries, store the expected certificate fingerprint (SHA‑256) in a ConfigMap and validate it in a startup script before any PyTorch download.
    • Centralize CA bundle updates. Keep certifi and the OS CA store in sync with your CI/CD image builds. Include a step in Dockerfiles:
    RUN pip install --upgrade certifi && \
    cp $(python -c "import certifi, sys; sys.stdout.write(certifi.where())") /etc/ssl/certs/ca-certificates.crt
  • Graceful fallback. Wrap PyTorch download calls in a retry block that catches ssl.SSLCertVerificationError and falls back to a pre‑cached local copy.
  • Document certificate lifecycle. Record expiration dates of all external HTTPS endpoints used by your pipelines and set calendar reminders 30 days before expiry.

    • FAQ – Common Follow‑Up Questions

    • Why does the same code work locally but fail in the CI cluster?
      Because the CI runner uses a minimal base image with an outdated CA bundle. Updating certifi or copying the host’s /etc/ssl/certs/ca‑bundle.crt resolves the discrepancy.
    • Can I configure PyTorch to ignore SSL errors without monkey‑patching?
      Starting with PyTorch 1.12, torch.hub.load_state_dict_from_url accepts a trust_repo=True flag, but SSL verification is still performed by the underlying requests library. The only built‑in bypass is to set the environment variable PYTORCH_DISABLE_CERT_VERIFY=1, which disables verification globally (use only in isolated environments).
    • Is there a way to see which certificate chain PyTorch is validating?
      Enable urllib3 debug logging:
    export PYTHONVERBOSE=1
export urllib3_debug=1
python - <<'PY'
import logging, urllib3
urllib3.enable_debug_logging()
import torch
torch.hub.load_state_dict_from_url("https://models.example.com/checkpoint.pt")
PY

    This prints the full certificate chain OpenSSL receives.

    • Do distributed checkpoints use the same SSL verification path as torch.hub?
      Yes. torch.distributed RPC and collective I/O ultimately call torch.utils.model_zoo.load_url for remote files, so the same OpenSSL verification logic applies.
    • How can I test certificate renewal without redeploying the whole job?
      Run a one‑off pod that executes the verification script against the target URL. Once the new certificate is in place, the script should succeed, confirming that the production job will not encounter the same error after the next rollout.

    Related Topic Hub: Model Serving Troubleshooting Hub