GPT-3.5 webhook timeout after moving to air‑gapped network

GPT‑3.5 Webhook Timeout After Moving to an Air‑Gapped Network

Problem

After migrating an OpenAI GPT‑3.5 integration to an air‑gapped data center, the application began receiving webhook timeout errors when the model attempted to send its response back to the caller. The failure manifests as HTTP 504 responses from the webhook endpoint and corresponding error entries in the OpenAI SDK logs.

Root Cause

OpenAI’s hosted inference service initiates an outbound HTTPS request to the user‑provided webhook URL once the model finishes generating a response. In an air‑gapped environment, outbound traffic is blocked by a firewall that only permits traffic to a limited set of internal services. The firewall silently drops the outbound connection, causing the OpenAI service to wait until its internal request timeout (default 30 seconds) expires, after which it returns a 504 to the original API caller.

Key points:

• The webhook URL is reachable from inside the air‑gapped network but not from the public OpenAI endpoint.
• OpenAI does not provide a “pull” mode; it always pushes the result.
• Network policies were updated to block all egress except to a whitelist of internal DNS zones, inadvertently excluding the webhook host.

Debugging Steps

Below is a reproducible investigation workflow that isolates the timeout source.

1. Confirm the webhook works locally. From a pod inside the air‑gapped cluster, issue a manual POST to the webhook endpoint.

curl -v -X POST https://webhook.internal.example.com/response \
  -H "Content-Type: application/json" \
  -d '{"id":"test","object":"chat.completion","choices":[{"message":{"role":"assistant","content":"OK"}}]}'

Expected output:

*   Trying 10.2.5.12:443...
* Connected to webhook.internal.example.com (10.2.5.12) port 443 (#0)
> POST /response HTTP/1.1
> Host: webhook.internal.example.com
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 124
>
* upload completely sent off: 124 out of 124 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 15
<
{"status":"ok"}

2. Inspect OpenAI SDK logs for timeout details.

2024-09-15T12:34:56.789Z ERROR openai.sdk.webhook - Webhook request failed: timeout after 30000ms
2024-09-15T12:34:56.789Z DEBUG openai.sdk.http - Outbound request: POST https://webhook.internal.example.com/response
2024-09-15T12:34:56.789Z DEBUG openai.sdk.http - Request headers: {Authorization=Bearer ***}

3. Validate egress path from OpenAI IP ranges. Capture a packet trace on the firewall for a known OpenAI egress IP (e.g., 52.215.0.0/16).

# tcpdump -i eth0 -n host 52.215.12.34 and port 443 -w /tmp/openai.pcap

The resulting capture shows no SYN packets leaving the interface, confirming the firewall drop.

4. Check firewall rule set.

# iptables -L OUTPUT -v -n
Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   10  6000 ACCEPT     tcp  --  *      *       0.0.0.0/0            10.2.5.12 tcp dpt:443
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Only internal IPs are permitted; the OpenAI egress range is not listed.

Solution

Two practical approaches exist, depending on security policy constraints.

Option 1 – Expose a Proxy Endpoint in the DMZ

Deploy a lightweight HTTPS reverse‑proxy (e.g., nginx) in a demilitarized zone (DMZ) that has outbound internet access. The proxy forwards the request to the internal webhook.

Before (blocked configuration)

# /etc/nginx/conf.d/webhook.conf (non‑existent)

After (proxy added)

# /etc/nginx/conf.d/webhook.conf
server {
    listen 443 ssl;
    server_name webhook-proxy.example.com;

    ssl_certificate /etc/ssl/certs/proxy.crt;
    ssl_certificate_key /etc/ssl/private/proxy.key;

    location /response {
        proxy_pass https://webhook.internal.example.com/response;
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/ssl/certs/internal-ca.crt;
    }
}

Update the OpenAI webhook registration to point to https://webhook-proxy.example.com/response. Ensure the firewall allows egress from the proxy host to the internet (port 443) and inbound from the internet to the proxy (port 443).

Option 2 – Pull‑Based Retrieval (Polling)

If introducing a DMZ proxy violates compliance, switch to a pull model: store the model output in a shared storage bucket (e.g., S3 compatible in‑house object store) and let the internal service poll for completion.

Implementation sketch:

# OpenAI request with stream=false and response_format=object
response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[...],
    stream=False,
    user="service-id"
)

# Store result locally
with open("/var/tmp/gpt-output-$(date +%s).json", "w") as f:
    json.dump(response, f)

The internal consumer watches the directory or a message queue for new files, eliminating the need for outbound webhook calls.

Verification

After applying the chosen solution, repeat the validation steps.

1. Trigger a GPT‑3.5 request and observe the webhook response time.
2. Check OpenAI SDK logs for the absence of timeout entries.
3. Confirm the proxy logs (if Option 1) show a successful 200 response from the internal webhook.

Sample successful log entry (Option 1):

2024-09-15T12:40:12.123Z INFO openai.sdk.webhook - Webhook delivered successfully (200 OK) in 152ms

Prevention

• Document outbound dependencies. Maintain a version‑controlled list of required egress IP ranges (OpenAI publishes them at https://api.openai.com/v1/ip-ranges) and integrate it into firewall change‑management.
• Automated connectivity test. Deploy a cron job that performs a HEAD request to the registered webhook URL from a public IP (e.g., via a bastion host) and alerts on failure.
• Infrastructure as Code (IaC) guardrails. Use terraform or pulumi to enforce that any webhook registration resource includes a tag requires_egress=true, triggering a policy check that validates firewall rules.
• Graceful degradation. Implement a fallback path in the application that, on webhook timeout, stores the response in a retry queue for later processing.

FAQ

1. Why does the webhook work when called from inside the cluster but fail from OpenAI?
   Because OpenAI’s service resides outside the air‑gapped network and must traverse the firewall. Internal calls succeed because they stay within allowed internal routes.
2. Can I whitelist OpenAI’s IP ranges permanently?
   Yes, but the ranges change occasionally. Automate the whitelist update using OpenAI’s /v1/ip-ranges endpoint and schedule a weekly sync.
3. Is there a way to increase the webhook timeout?
   OpenAI’s timeout is fixed at 30 seconds for push‑based webhooks. Extending it is not supported; the solution must address connectivity.
4. Will using a proxy introduce latency?
   Minimal latency is added (typically < 50 ms) for TLS termination and internal forwarding. Measure end‑to‑end latency after deployment to ensure it meets SLA.
5. What security considerations apply to the proxy approach?
   Ensure the proxy terminates TLS with a certificate signed by a trusted internal CA, validates client certificates if required, and restricts inbound access to only OpenAI IP ranges.

Related Topic Hub: LLM Systems Troubleshooting Hub