DeepSeek Model Download Blocked By Network Policy On Edge Node

Q: Observed logs

2024-10-21 08:12:03,721 INFO loading model configuration... 2024-10-21 08:12:04,112 ERROR failed to download https://modelhub.deepseek.com/deepseek-v2/model.bin: Connection timed out 2024-10-21 08:12:04,113 TRACE retrying in 5 seconds... 2024-10-21 08:12:09,118 ERROR pip install -r requirements.txt failed: [Errno 111] Connection refused 2024-10-21 08:12:09,119 TRACE aborting startup The same container image works on a development workstation with unrestricted outbound connectivity, confirming that the failure is specific to the edge node’s network controls.

Q: Debug – Investigating the network blockage Step 1 – Verify pod network isolation

kubectl get networkpolicy -n edge‑services NAME POD-SELECTOR AGE default-deny-egress {} 45d allow-internal-egress app=deepseek 30d Step 2 – Inspect generated iptables on the node sudo iptables -S -t filter | grep -i "DEEPSEEK" -A KUBE-OUTPUT -s 10.244.0.0/16 -d 0.0.0.0/0 -p tcp -m multiport --dports 443 -j DROP The rule drops all outbound TCP traffic on port 443 from the pod CIDR.

Q: Step 3 – Test connectivity from inside the container

kubectl exec -it deepseek-0 -n edge‑services -- /bin/sh # curl -v https://modelhub.deepseek.com/deepseek-v2/model.bin * Trying 34.120.45.22:443... * connect to 34.120.45.22 port 443 failed: Connection timed out Step 4 – Confirm that the same command works on a node without the policy curl -I https://modelhub.deepseek.com/deepseek-v2/model.bin HTTP/2 200 content-length: 124567890 content-type: application/octet-stream Solution – Extend egress policy to allow model and package repositories Two complementary approaches were applied:

Q: Approach B – Use a local proxy cache (optional hardening)

Deploy a small nginx cache in the same namespace that mirrors modelhub.deepseek.com and pypi.org. Update the DeepSeek container’s environment variables to point to the proxy, reducing reliance on external egress.

Q: Verify – Confirm that model download and dependency installation succeed Log check after policy update

2024-10-21 08:15:12,004 INFO downloading model from https://modelhub.deepseek.com/deepseek-v2/model.bin 2024-10-21 08:15:13,210 INFO model.bin downloaded (124.5 MiB) in 1.2 s 2024-10-21 08:15:13,215 INFO installing Python dependencies 2024-10-21 08:15:18,543 INFO pip install completed successfully 2024-10-21 08:15:18,550 INFO DeepSeek service started, listening on :8080 Network test from inside the container curl -I https://modelhub.deepseek.com/deepseek-v2/model.bin HTTP/2 200 content-length: 124567890 content-type: application/octet-stream Readiness probe kubectl get pod deepseek-0 -n edge-services -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' True Prevent – Guardrails to avoid recurrence Policy as code review: Require any change to NetworkPolicy to be reviewed for outbound dependencies of new workloads. Automated CIDR sync: Use a CI job that resolves the current IP ranges of modelhub.deepseek.com and pypi.org and updates the policy YAML, preventing stale blocks when CDN IPs change. Internal artifact mirror: Mirror required model files and Python wheels to an internal registry (e.g., harbor or artifactory) and point DeepSeek to those locations via environment variables. Observability: Add a Prometheus metric deepseek_model_download_success_total and alert on zero increments over a 5‑minute window. Fail‑fast startup: Configure the container to abort after a configurable number of download retries, so that policy violations surface quickly in CI pipelines. Related Topic Hub: LLM Systems Troubleshooting Hub

In this article

Table of Contents

Problem – DeepSeek model download blocked by network policy on edge node

An edge‑computing node runs a containerized DeepSeek inference service. During startup the container attempts to pull model weights and Python dependencies from external repositories (e.g., https://modelhub.deepseek.com and https://pypi.org). The pod never reaches the Ready state and logs contain repeated connection‑refused errors.

Observed logs


2024-10-21 08:12:03,721 INFO  loading model configuration...
2024-10-21 08:12:04,112 ERROR failed to download https://modelhub.deepseek.com/deepseek-v2/model.bin: Connection timed out
2024-10-21 08:12:04,113 TRACE retrying in 5 seconds...
2024-10-21 08:12:09,118 ERROR pip install -r requirements.txt failed: [Errno 111] Connection refused
2024-10-21 08:12:09,119 TRACE aborting startup

The same container image works on a development workstation with unrestricted outbound connectivity, confirming that the failure is specific to the edge node’s network controls.

Root Cause – Egress restrictions enforced by a Kubernetes `NetworkPolicy`

The edge cluster enforces a default‑deny egress policy to limit data exfiltration. The policy permits outbound traffic only to internal services (e.g., service‑mesh.local) and blocks all other destinations. Because the DeepSeek container needs to reach external domains for model weights and pip packages, the egress packets are dropped by the node’s iptables rules generated from the NetworkPolicy.

Key points:

DeepSeek uses urllib3/requests to fetch .bin files from modelhub.deepseek.com over HTTPS (port 443).
Dependency installation pulls from pypi.org and files.pythonhosted.org, also over HTTPS.
The existing NetworkPolicy contains an egress rule that allows only podSelector: {app: internal‑api} and a CIDR 10.0.0.0/8. No rule matches the external IP ranges.

Debug – Investigating the network blockage

Step 1 – Verify pod network isolation


kubectl get networkpolicy -n edge‑services
NAME                     POD-SELECTOR   AGE
default-deny-egress      {}             45d
allow-internal-egress    app=deepseek   30d

Step 2 – Inspect generated iptables on the node


sudo iptables -S -t filter | grep -i "DEEPSEEK"
-A KUBE-OUTPUT -s 10.244.0.0/16 -d 0.0.0.0/0 -p tcp -m multiport --dports 443 -j DROP

The rule drops all outbound TCP traffic on port 443 from the pod CIDR.

Step 3 – Test connectivity from inside the container


kubectl exec -it deepseek-0 -n edge‑services -- /bin/sh
# curl -v https://modelhub.deepseek.com/deepseek-v2/model.bin
*   Trying 34.120.45.22:443...
* connect to 34.120.45.22 port 443 failed: Connection timed out

Step 4 – Confirm that the same command works on a node without the policy


curl -I https://modelhub.deepseek.com/deepseek-v2/model.bin
HTTP/2 200 
content-length: 124567890
content-type: application/octet-stream

Solution – Extend egress policy to allow model and package repositories

Two complementary approaches were applied:

Approach A – Whitelist external domains via `NetworkPolicy`

Added an egress rule that permits traffic to the required hostnames using IPBlock with CIDR ranges resolved from DNS. The policy is applied only to the DeepSeek pod label.

Before (restricted policy)


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-internal-egress
  namespace: edge-services
spec:
  podSelector:
    matchLabels:
      app: deepseek
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: internal-api
    ports:
    - protocol: TCP
      port: 443

After (whitelisted external egress)


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-deepseek-egress
  namespace: edge-services
spec:
  podSelector:
    matchLabels:
      app: deepseek
  policyTypes:
  - Egress
  egress:
  # Internal API
  - to:
    - podSelector:
        matchLabels:
          app: internal-api
    ports:
    - protocol: TCP
      port: 443
  # ModelHub CDN (example CIDR block)
  - to:
    - ipBlock:
        cidr: 34.120.0.0/16   # modelhub.deepseek.com
    ports:
    - protocol: TCP
      port: 443
  # PyPI CDN
  - to:
    - ipBlock:
        cidr: 151.101.0.0/16  # pypi.org & files.pythonhosted.org
    ports:
    - protocol: TCP
      port: 443

After applying the updated policy, the pod regained outbound connectivity to the required services.

Approach B – Use a local proxy cache (optional hardening)

Deploy a small nginx cache in the same namespace that mirrors modelhub.deepseek.com and pypi.org. Update the DeepSeek container’s environment variables to point to the proxy, reducing reliance on external egress.

Proxy deployment snippet


apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-proxy
  namespace: edge-services
spec:
  replicas: 1
  selector:
    matchLabels:
      app: edge-proxy
  template:
    metadata:
      labels:
        app: edge-proxy
    spec:
      containers:
      - name: nginx
        image: nginx:stable-alpine
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: cache
          mountPath: /var/cache/nginx
      volumes:
      - name: cache
        emptyDir: {}

Set environment variables in the DeepSeek pod:


env:
- name: HTTPS_PROXY
  value: http://edge-proxy.edge-services.svc.cluster.local:8080
- name: PIP_INDEX_URL
  value: http://edge-proxy.edge-services.svc.cluster.local:8080/simple

Verify – Confirm that model download and dependency installation succeed

Log check after policy update


2024-10-21 08:15:12,004 INFO  downloading model from https://modelhub.deepseek.com/deepseek-v2/model.bin
2024-10-21 08:15:13,210 INFO  model.bin downloaded (124.5 MiB) in 1.2 s
2024-10-21 08:15:13,215 INFO  installing Python dependencies
2024-10-21 08:15:18,543 INFO  pip install completed successfully
2024-10-21 08:15:18,550 INFO  DeepSeek service started, listening on :8080

Network test from inside the container


curl -I https://modelhub.deepseek.com/deepseek-v2/model.bin
HTTP/2 200 
content-length: 124567890
content-type: application/octet-stream

Readiness probe


kubectl get pod deepseek-0 -n edge-services -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
True

Prevent – Guardrails to avoid recurrence

Policy as code review: Require any change to NetworkPolicy to be reviewed for outbound dependencies of new workloads.
Automated CIDR sync: Use a CI job that resolves the current IP ranges of modelhub.deepseek.com and pypi.org and updates the policy YAML, preventing stale blocks when CDN IPs change.
Internal artifact mirror: Mirror required model files and Python wheels to an internal registry (e.g., harbor or artifactory) and point DeepSeek to those locations via environment variables.
Observability: Add a Prometheus metric deepseek_model_download_success_total and alert on zero increments over a 5‑minute window.
Fail‑fast startup: Configure the container to abort after a configurable number of download retries, so that policy violations surface quickly in CI pipelines.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the DeepSeek container work on my laptop but not on the edge node?
Because the laptop has unrestricted outbound internet access, while the edge node enforces a default‑deny egress NetworkPolicy that blocks traffic to external CDNs.
Can I use DNS names in a NetworkPolicy instead of CIDR blocks?
Kubernetes NetworkPolicy only supports IPBlock CIDRs. To use domain names you must resolve them to IP ranges externally (e.g., via a CI script) and update the policy.
Is it safe to open port 443 to the entire internet for the pod?
Opening 0.0.0.0/0 defeats the purpose of the egress restriction and may expose the node to data‑exfiltration risks. Prefer whitelisting only the required CIDRs or using a local proxy cache.
How do I troubleshoot if the policy appears correct but downloads still fail?
Check the node’s iptables chain for DROP rules, verify DNS resolution inside the pod, and capture traffic with tcpdump -i any port 443 to see whether SYN packets are being sent and ACKed.
What if the CDN IP ranges change frequently?
Automate CIDR updates: a scheduled job queries the CDN’s DNS A/AAAA records, expands them into CIDR blocks, and applies the updated NetworkPolicy via kubectl apply -f. Combine this with a short TTL on the policy to keep it fresh.