LlamaIndex Training Loop Failed To Fetch Data Due To Network Policy

Q: Problem Description

During a nightly training run, the LlamaIndex training loop crashes when it attempts to load external documents or call remote APIs. The pod logs contain errors such as:

Q: Investigation and Debugging Confirm the failing connector. Enable verbose logging in LlamaIndex: import logging logging.basicConfig(level=logging.DEBUG) from llama_index import SimpleWebPageReader docs = SimpleWebPageReader().load_data(urls=["https://example.com/data.json"]) Log excerpt shows the request attempt and the immediate ConnectionError. Inspect the pod’s NetworkPolicy. Run: kubectl get networkpolicy -n ml-workloads -o yaml Identify a policy with egress: [] or missing to selectors. Test outbound connectivity from the pod. Exec into the pod and use curl and nc: kubectl exec -it training-pod-abc123 -n ml-workloads -- /bin/sh # curl -I https://example.com curl: (7) Failed to connect to example.com port 443: Connection refused # nc -zv s3.amazonaws.com 443 Connection timed out Check for existing egress allowances. Look for policies that whitelist specific CIDR blocks or DNS names. If none exist for the required endpoints (e.g., *.amazonaws.com, api.openai.com), the block is confirmed. Resolution Step 1 – Define an explicit egress allowlist

Create a dedicated NetworkPolicy that permits traffic to the domains required by LlamaIndex data loaders. The following example allows HTTP/HTTPS to arbitrary external hosts while still denying other ports:

Q: Step 2 – (Optional) Configure a forward proxy for tighter control

If the organization prefers to keep 0.0.0.0/0 open only through a proxy, add the proxy environment variables as described in the LlamaIndex Network‑Configuration wiki:

In this article

Table of Contents

Problem Description

During a nightly training run, the LlamaIndex training loop crashes when it attempts to load external documents or call remote APIs. The pod logs contain errors such as:

urllib3.exceptions.MaxRetryError: Failed to establish a new connection: [Errno 111] Connection refused
requests.exceptions.ConnectionError: HTTPConnectionPool(host='example.com', port=80): Max retries exceeded with url: /data.json (Caused by NewConnectionError)
EgressPolicyDenied: traffic to 0.0.0.0/0 blocked by NetworkPolicy "default-deny-egress"
TimeoutError: request timed out while fetching remote document via SimpleWebPageReader
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.amazonaws.com"

The training pod runs inside a Kubernetes cluster where a NetworkPolicy with policyTypes: [Egress] denies all outbound traffic by default. The failure appears only when LlamaIndex invokes its data loaders (e.g., SimpleWebPageReader, S3Reader, or the OpenAI API connector).

Root Cause Analysis

Interaction Between LlamaIndex Connectors and Kubernetes NetworkPolicy

LlamaIndex fetches external resources via HTTP(S), S3, or custom APIs as described in the official Data Loaders documentation. Each connector ultimately uses requests or urllib3 to open outbound sockets.

The cluster’s NetworkPolicy API enforces egress rules at the pod level. A default-deny-egress policy like the one below blocks any traffic to external IP ranges unless explicitly allowed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: ml-workloads
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress: []   # empty list = deny all

Because the training pod inherits this policy, every outbound request generated by LlamaIndex is dropped, leading to the connection‑refused and timeout errors observed in the logs. This matches community reports such as the GitHub issue “Egress blocked when using SimpleWebPageReader in LlamaIndex” and the Stack Overflow discussion on required egress rules.

Investigation and Debugging

Confirm the failing connector. Enable verbose logging in LlamaIndex:

import logging
logging.basicConfig(level=logging.DEBUG)
from llama_index import SimpleWebPageReader

docs = SimpleWebPageReader().load_data(urls=["https://example.com/data.json"])

Log excerpt shows the request attempt and the immediate ConnectionError.

Inspect the pod’s NetworkPolicy. Run:

kubectl get networkpolicy -n ml-workloads -o yaml

Identify a policy with egress: [] or missing to selectors.

Test outbound connectivity from the pod. Exec into the pod and use curl and nc:

kubectl exec -it training-pod-abc123 -n ml-workloads -- /bin/sh
# curl -I https://example.com
curl: (7) Failed to connect to example.com port 443: Connection refused
# nc -zv s3.amazonaws.com 443
Connection timed out

Check for existing egress allowances. Look for policies that whitelist specific CIDR blocks or DNS names. If none exist for the required endpoints (e.g., *.amazonaws.com, api.openai.com), the block is confirmed.

Resolution

Step 1 – Define an explicit egress allowlist

Create a dedicated NetworkPolicy that permits traffic to the domains required by LlamaIndex data loaders. The following example allows HTTP/HTTPS to arbitrary external hosts while still denying other ports:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: llamaindex-egress-allow
  namespace: ml-workloads
spec:
  podSelector:
    matchLabels:
      app: training-job
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0   # allow all IPs for simplicity; replace with specific CIDRs in production
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

Before (deny all):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: ml-workloads
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress: []

After (allow required egress):

# Apply the new policy
kubectl apply -f llamaindex-egress-allow.yaml

Step 2 – (Optional) Configure a forward proxy for tighter control

If the organization prefers to keep 0.0.0.0/0 open only through a proxy, add the proxy environment variables as described in the LlamaIndex Network‑Configuration wiki:

export HTTP_PROXY=http://proxy.corp:3128
export HTTPS_PROXY=http://proxy.corp:3128
export NO_PROXY=localhost,127.0.0.1

Then modify the NetworkPolicy to allow traffic only to the proxy host:

egress:
- to:
  - ipBlock:
      cidr: 10.2.5.0/24   # proxy subnet
  ports:
  - protocol: TCP
    port: 3128

Verification

Re‑run the training pod and monitor the logs for successful document loads:

INFO:root:Fetching https://example.com/data.json
INFO:root:Successfully downloaded 1 document(s)

From inside the pod, verify that curl now reaches external endpoints:

# curl -I https://example.com
HTTP/1.1 200 OK
...
# curl -I https://s3.amazonaws.com
HTTP/1.1 200 OK
...

Check that the training job completes without raising urllib3.exceptions.MaxRetryError or EndpointConnectionError.

Operational Experience & Prevention

Misleading symptom: The initial error “Connection refused” suggested the remote service was down, but the root cause was local egress denial.
Common incorrect assumption: Adding a NetworkPolicy that only whitelists a namespace does not automatically open egress; explicit to rules are required.
Edge case: When using the OpenAI connector, DNS resolution succeeds, but the subsequent TCP handshake is blocked, producing a timeout rather than a DNS error.
Lesson learned: Keep a minimal “egress‑allowlist” policy version‑controlled alongside the training job manifests. Include a health‑check pod that periodically validates outbound connectivity to the required services.

Best Practices & Preventive Controls

Maintain a central NetworkPolicy library that defines egress rules per external dependency (e.g., *.amazonaws.com, api.openai.com, *.githubusercontent.com).
Enable NetworkPolicy audit logs (via the Kubernetes audit backend) to detect accidental egress denials after policy changes.
Instrument LlamaIndex connectors with retry‑backoff and explicit timeout settings to surface egress issues quickly.
When possible, route external traffic through a managed egress gateway or proxy to centralize firewall rules.
Automate a post‑deployment test that runs a lightweight LlamaIndex fetch (e.g., SimpleWebPageReader on a known URL) and fails the CI pipeline if the test does not succeed.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does LlamaIndex work locally but fail inside the Kubernetes pod? Local environments typically have unrestricted outbound networking. Inside the cluster, a NetworkPolicy with egress: [] blocks all external connections, causing the failures.
Do I need to open all ports 80/443 for every external service? Opening ports 80 and 443 is sufficient for HTTP(S) connectors. For S3 over HTTPS you still use port 443. If you use non‑standard ports (e.g., a custom vector store on 8080), add those to the egress rule.
Can I use a ServiceEntry (Istio) instead of a NetworkPolicy? Yes, Istio’s ServiceEntry can expose external hosts to the mesh. However, you still need to ensure the underlying Kubernetes egress policy allows traffic to the Istio egress gateway.
How do I debug a “NewConnectionError” when the pod can resolve DNS? Verify that the pod can reach the destination IP with nc -zv <host> <port>. If the connection times out, the egress is blocked; check the applicable NetworkPolicy and any sidecar proxy rules.
Is configuring a proxy enough without changing NetworkPolicy? No. The proxy itself requires outbound connectivity. You must allow egress to the proxy host (or to the final destination if the proxy runs inside the cluster). Without an allow rule, the pod cannot reach the proxy, and requests will still fail.