Hugging Face Transformers evaluation webhook timeout during CI/CD run

Problem – Webhook Timeout During Hugging Face Transformers Evaluation in CI/CD

During the Trainer.evaluate step of an automated CI/CD pipeline, the post‑evaluation callback that ships evaluation metrics to an external monitoring service (e.g., Prometheus Pushgateway, Datadog, custom HTTP endpoint) consistently fails with a timeout error. The CI job aborts after the runner‑enforced timeout (typically 30 seconds) and marks the evaluation stage as failed.

Typical error messages observed in CI logs:

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='monitor.example.com', port=443): Read timed out. (timeout=30)
MetricLoggingCallbackError: Webhook request timed out after 20 seconds – metrics not recorded
TimeoutError: Evaluation callback exceeded the allowed execution time of 30 seconds in CI runner

These errors appear in multiple environments:

• GitHub Actions self‑hosted runner behind a corporate firewall (outbound traffic throttled).
• Azure DevOps self‑hosted agent that cannot resolve the monitoring domain.
• GitLab CI job where the monitoring service experiences a spike, causing the request to exceed a 10 second client timeout.
• Jenkins Docker container with outbound HTTPS blocked, leading to ConnectTimeout.

Root Cause – Why the Evaluation Callback Times Out

The evaluation pipeline itself is fast (seconds to a few minutes) and the Trainer.evaluate method returns a dictionary of metrics. The timeout occurs only when the custom callback attempts to POST the JSON payload to the external endpoint. The root causes can be grouped into three categories:

1. Insufficient client‑side timeout configuration – The default requests.post(..., timeout=30) used in many community examples (see GitHub issue #11234) is shorter than the actual latency of the monitoring service under load. When the service takes >30 s to acknowledge the payload, the client raises ReadTimeout.
2. Network egress restrictions on CI runners – Self‑hosted runners often sit behind firewalls that limit outbound bandwidth or enforce per‑connection throttling. The 200 KB JSON payload (as in the real incident with Prometheus Pushgateway) can be delayed enough to hit the client timeout.
3. Synchronous webhook handling in the CI process – The callback runs in the same thread as the evaluation step. CI runners enforce a hard execution limit (e.g., 30 s for a single step). Even if the HTTP client is configured with a longer timeout, the CI runner may kill the process once its step timeout expires (see the “TimeoutError: Evaluation callback exceeded the allowed execution time” message).

Official documentation for the Trainer.evaluate method (Transformers docs) mentions that users can inject Callback objects to run arbitrary code after evaluation, but it does not prescribe timeout handling for outbound HTTP calls. The Hub webhook guide (Hub docs) assumes the endpoint is reachable and responsive, which is not guaranteed in CI environments.

Debug – Investigation Steps

Below is a reproducible debugging workflow that isolates the timeout source.

1. Confirm the evaluation step completes locally. Run the same script on a developer workstation with unrestricted network access.
2. Capture the exact HTTP request/response. Add http.client.HTTPConnection.debuglevel = 1 or use tcpdump inside the CI container.
3. Inspect CI runner logs for step‑level timeouts. In GitHub Actions, look for ##[error]Process completed with exit code 1 preceded by a “timeout” line.
4. Validate DNS resolution and connectivity. Run inside the CI job:

nslookup monitor.example.com
curl -v https://monitor.example.com/healthz

Expected output on a healthy runner:

Server:  10.0.0.2
Address: 10.0.0.2#53

Non-authoritative answer:
Name:   monitor.example.com
Address: 203.0.113.42

*   Trying 203.0.113.42:443...
* Connected to monitor.example.com (203.0.113.42) port 443 (#0)
> GET /healthz HTTP/1.1
> Host: monitor.example.com
> User-Agent: curl/7.79.1
> Accept: */*
...

Measure round‑trip latency. Use time curl -X POST ... with a small payload to see if the service consistently exceeds 30 s under load.

Check CI runner timeout settings. In GitHub Actions, the default step timeout is 6 hours, but self‑hosted runners may enforce a per‑process limit (e.g., 30 s). Azure DevOps agents expose system.jobTimeoutInMinutes and similar variables.

Solution – Making the Evaluation Callback Reliable

The fix combines three changes:

1. Increase the HTTP client timeout and make it configurable.
2. Offload the POST to an asynchronous worker so the CI step does not block.
3. Ensure network egress is permitted (firewall rule or VPC endpoint).

Before – Synchronous Callback with Hard‑coded Timeout

import json, requests
from transformers import Trainer, TrainingArguments

class MetricLoggingCallback:
    def __init__(self, endpoint):
        self.endpoint = endpoint

    def on_evaluate(self, args, state, control, metrics=None, **kwargs):
        payload = json.dumps(metrics)
        # Default 30 s timeout – fails under load
        response = requests.post(self.endpoint, data=payload, timeout=30)
        response.raise_for_status()

training_args = TrainingArguments(output_dir="./model")
trainer = Trainer(
    model=model,
    args=training_args,
    callbacks=[MetricLoggingCallback("https://monitor.example.com/metrics")]
)
trainer.evaluate()

After – Async Callback with Configurable Timeout and Retry

import json, os, threading, time
import httpx  # httpx supports async and fine‑grained timeout control

class AsyncMetricLoggingCallback:
    def __init__(self, endpoint, timeout=None, max_retries=3, backoff=5):
        self.endpoint = endpoint
        # Pull timeout from env or default to 120 s
        self.timeout = timeout or int(os.getenv("METRIC_TIMEOUT", "120"))
        self.max_retries = max_retries
        self.backoff = backoff

    def _post_metrics(self, payload):
        for attempt in range(1, self.max_retries + 1):
            try:
                with httpx.Client(timeout=self.timeout) as client:
                    resp = client.post(self.endpoint, json=payload)
                    resp.raise_for_status()
                return  # success
            except (httpx.ReadTimeout, httpx.ConnectTimeout) as exc:
                if attempt == self.max_retries:
                    raise
                time.sleep(self.backoff * attempt)  # exponential back‑off

    def on_evaluate(self, args, state, control, metrics=None, **kwargs):
        payload = metrics or {}
        # Fire‑and‑forget in a daemon thread so CI step can finish
        thread = threading.Thread(target=self._post_metrics, args=(payload,), daemon=True)
        thread.start()

training_args = TrainingArguments(output_dir="./model")
trainer = Trainer(
    model=model,
    args=training_args,
    callbacks=[
        AsyncMetricLoggingCallback(
            endpoint="https://monitor.example.com/metrics",
            timeout=120,          # matches Inference API outbound timeout guidance
            max_retries=5,
            backoff=10
        )
    ]
)
trainer.evaluate()

Why this works:

• httpx respects the Inference API timeout recommendations, allowing a longer window (e.g., 120 s) without hitting the default 30 s limit.
• Running the POST in a daemon thread decouples it from the evaluation step, preventing the CI runner from aborting the entire job when the HTTP call lingers.
• Configurable environment variable (METRIC_TIMEOUT) lets ops teams tune the value per environment without code changes.
• Retry logic with exponential back‑off mitigates transient spikes in the monitoring service (as seen in the GitLab CI incident).

Network Configuration Fix

If the CI runner is self‑hosted, add an outbound rule for monitor.example.com:443 or configure a VPC service endpoint that bypasses the corporate firewall. Verify with:

curl -I https://monitor.example.com/metrics

Successful response:

HTTP/1.1 200 OK
Date: Thu, 15 Sep 2026 12:34:56 GMT
Content-Type: application/json
Content-Length: 0

Verify – Confirming the Fix Works

1. Re‑run the CI job with the updated callback. Look for the absence of ReadTimeout or ConnectionError in the logs.
2. Check the monitoring service for the newly posted metrics. For Prometheus Pushgateway, query the metric name to see the latest timestamp.
3. Validate that the CI step finishes within its allocated time (e.g., duration: 12.3s instead of timing out).
4. Optionally, add a health‑check in the pipeline:

# After evaluation
curl -sSf https://monitor.example.com/metrics/last | jq .

If the command returns the JSON payload posted by the callback, the end‑to‑end flow is functional.

Prevent – Operational Guardrails and Best Practices

• Make timeout values explicit. Never rely on library defaults; set them via environment variables.
• Prefer async or background workers for outbound HTTP in CI. This avoids step‑level timeouts imposed by runners.
• Instrument network latency. Export a metric_http_latency_seconds gauge from the callback to detect slow egress early.
• Monitor firewall logs. Alert on dropped outbound connections to the monitoring domain.
• Use a circuit‑breaker pattern. If the monitoring service is unavailable for >N attempts, skip posting and record a warning metric.

FAQ – Common Follow‑Up Questions

1. Why does the webhook succeed locally but fail in CI?
   Because CI runners often have stricter outbound network policies and lower per‑process time limits. The local machine has unrestricted egress and no enforced step timeout.
2. Can I keep the synchronous callback and just increase the CI step timeout?
   Increasing the CI step timeout may prevent the runner from killing the job, but it does not solve network throttling or firewall restrictions. Async handling is more robust.
3. Is httpx required, or can I stay with requests?
   You can stay with requests if you configure a larger timeout and run the POST in a separate thread or process. httpx simplifies async usage and provides clearer timeout semantics.
4. How do I know what timeout the monitoring service expects?
   Consult the service’s API documentation (e.g., Inference API outbound timeout guidance) and perform a quick curl -v with --max-time to measure typical response times under load.
5. What should I do if the monitoring endpoint returns HTTP 504?
   A 504 indicates the service itself timed out. Implement retry with back‑off, and consider buffering metrics locally (e.g., write to a file) for later bulk upload.

Related Topic Hub: Model Serving Troubleshooting Hub