Inference API Timeout Connecting To PostgreSQL After Firewall Rule Update

Q: Problem Description

The inference API service, built with FastAPI and using psycopg2/SQLAlchemy to query a PostgreSQL instance, started returning HTTP 504 timeouts after a recent firewall rule change. The API logs contain repeated connection errors such as:

Q: Investigation and Debugging Steps 1. Verify network reachability

# From inside an inference pod nc -zv db.example.com 5432 # Expected output when allowed: # Connection to db.example.com 5432 port [tcp/postgresql] succeeded! # When blocked: # nc: connect to db.example.com port 5432 (tcp) timed out: Operation now in progress 2. Inspect firewall and security group rules # GCP example gcloud compute firewall-rules list --filter="name~'allow-inference-.*'" # AWS example aws ec2 describe-security-groups --group-ids sg-0abc1234def56789 Look for egress rules that permit tcp:5432 from the inference service’s subnet.

Q: 3. Check Kubernetes NetworkPolicy

# List policies in the namespace kubectl get networkpolicy -n model-serving -o yaml # Example policy that blocks egress apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-db-egress spec: podSelector: {} policyTypes: - Egress egress: - ports: - protocol: TCP port: 80 - protocol: TCP port: 443 4. Confirm PostgreSQL listening configuration # On the DB host cat /var/lib/postgresql/data/postgresql.conf | grep listen_addresses listen_addresses = '*' cat /var/lib/postgresql/data/postgresql.conf | grep port port = 5432 If listen_addresses is restricted to localhost, remote connections will be refused regardless of firewall state.

Q: 5. Review pg_hba.conf entries

# Sample entry that allows the original subnet host model_db inference_user 10.2.3.0/24 md5 # Missing entry for NAT‑translated IP (e.g., 192.168.100.45) 6. Capture traffic (optional) # Run tcpdump on the pod node sudo tcpdump -i any host db.example.com and port 5432 -vv Look for SYN packets being dropped or reset by the firewall.

Q: Resolution 1. Restore outbound firewall rule

# GCP – add egress rule gcloud compute firewall-rules create allow-inference-db \ --direction=EGRESS \ --priority=1000 \ --network=default \ --action=ALLOW \ --rules=tcp:5432 \ --destination-ranges=10.0.0.0/24 \ --target-tags=inference-pod # AWS – modify security group aws ec2 authorize-security-group-egress \ --group-id sg-0abc1234def56789 \ --protocol tcp \ --port 5432 \ --cidr 10.0.0.0/24 2. Adjust Kubernetes NetworkPolicy to allow DB egress # Updated policy (add port 5432) apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-db-egress spec: podSelector: {} policyTypes: - Egress egress: - ports: - protocol: TCP port: 80 - protocol: TCP port: 443 - protocol: TCP port: 5432 #

Q: 3. Update pg_hba.conf to accept the NAT‑translated source IP or CIDR

# Before (only original subnet) host model_db inference_user 10.2.3.0/24 md5 # After (include NAT range) host model_db inference_user 10.2.3.0/24,192.168.100.0/24 md5 Reload PostgreSQL configuration without restart:

Q: 4. Verify application‑level connection string

# Before – hard‑coded host without SSL SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://inference_user:pwd@db.example.com/model_db" # After – explicit port and SSL (if required) SQLALCHEMY_DATABASE_URI = ( "postgresql+psycopg2://inference_user:pwd@db.example.com:5432/model_db" "?sslmode=require" ) Validation From a running inference pod, re‑run the nc test; it should report success. Execute a simple query via psql or the API’s health endpoint: # Inside pod psql "host=db.example.com port=5432 user=inference_user dbname=model_db sslmode=require" -c "SELECT 1;" # Expected output: ?column? ---------- 1 (1 row) Or call GET /health on the FastAPI service and verify a 200 OK with { "db": "connected" }.

In this article

Table of Contents

Problem Description

The inference API service, built with FastAPI and using psycopg2/SQLAlchemy to query a PostgreSQL instance, started returning HTTP 504 timeouts after a recent firewall rule change. The API logs contain repeated connection errors such as:


2026-08-31 14:02:13,842 ERROR inference.api.handlers - could not connect to server: Connection timed out (0.00s)
Is the server running on host "db.example.com" and accepting TCP/IP connections on port 5432?
Traceback (most recent call last):
  File "/app/inference/api/db.py", line 42, in get_prediction
    conn = engine.connect()
psycopg2.OperationalError: timeout expired

In addition, occasional FATAL: no pg_hba.conf entry for host "10.2.3.4", user "inference_user", database "model_db", SSL off messages appear when the firewall permits traffic but the source IP changes.

Root Cause Analysis

Three closely related factors caused the outage:

Outbound firewall restriction – The VPC firewall rule that previously allowed egress from the inference namespace to the Cloud SQL instance on TCP port 5432 was tightened. According to PostgreSQL Documentation – Section 23.3 Network Configuration, if the client cannot reach the server’s listening port, the client library reports a timeout before any authentication occurs.
NetworkPolicy enforcement (Kubernetes) – A new NetworkPolicy was applied to the model-serving namespace, limiting egress to only ports 80 and 443. This silently blocked the PostgreSQL traffic, matching the symptom observed in the GitHub issue FastAPI inference service losing DB connectivity due to outbound firewall restrictions.
pg_hba.conf source‑IP mismatch – The corporate firewall upgrade introduced a NAT rule that rewrote the source IP of outbound packets. PostgreSQL’s pg_hba.conf (see Chapter 20. Client Authentication) still only allowed the original IP range, resulting in “no pg_hba.conf entry” errors when the NAT address was used.

Investigation and Debugging Steps

1. Verify network reachability

# From inside an inference pod
nc -zv db.example.com 5432
# Expected output when allowed:
# Connection to db.example.com 5432 port [tcp/postgresql] succeeded!
# When blocked:
# nc: connect to db.example.com port 5432 (tcp) timed out: Operation now in progress

2. Inspect firewall and security group rules

# GCP example
gcloud compute firewall-rules list --filter="name~'allow-inference-.*'"
# AWS example
aws ec2 describe-security-groups --group-ids sg-0abc1234def56789

Look for egress rules that permit tcp:5432 from the inference service’s subnet.

3. Check Kubernetes NetworkPolicy

# List policies in the namespace
kubectl get networkpolicy -n model-serving -o yaml
# Example policy that blocks egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-db-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

4. Confirm PostgreSQL listening configuration

# On the DB host
cat /var/lib/postgresql/data/postgresql.conf | grep listen_addresses
listen_addresses = '*'
cat /var/lib/postgresql/data/postgresql.conf | grep port
port = 5432

If listen_addresses is restricted to localhost, remote connections will be refused regardless of firewall state.

5. Review pg_hba.conf entries

# Sample entry that allows the original subnet
host    model_db    inference_user    10.2.3.0/24    md5
# Missing entry for NAT‑translated IP (e.g., 192.168.100.45)

6. Capture traffic (optional)

# Run tcpdump on the pod node
sudo tcpdump -i any host db.example.com and port 5432 -vv

Look for SYN packets being dropped or reset by the firewall.

Resolution

1. Restore outbound firewall rule

# GCP – add egress rule
gcloud compute firewall-rules create allow-inference-db \
    --direction=EGRESS \
    --priority=1000 \
    --network=default \
    --action=ALLOW \
    --rules=tcp:5432 \
    --destination-ranges=10.0.0.0/24 \
    --target-tags=inference-pod

# AWS – modify security group
aws ec2 authorize-security-group-egress \
    --group-id sg-0abc1234def56789 \
    --protocol tcp \
    --port 5432 \
    --cidr 10.0.0.0/24

2. Adjust Kubernetes NetworkPolicy to allow DB egress

# Updated policy (add port 5432)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 5432   # <-- added
    to:
    - ipBlock:
        cidr: 10.0.0.0/24

Apply with kubectl apply -f allow-db-egress.yaml.

3. Update `pg_hba.conf` to accept the NAT‑translated source IP or CIDR

# Before (only original subnet)
host    model_db    inference_user    10.2.3.0/24    md5

# After (include NAT range)
host    model_db    inference_user    10.2.3.0/24,192.168.100.0/24    md5

Reload PostgreSQL configuration without restart:

SELECT pg_reload_conf();

4. Verify application‑level connection string

# Before – hard‑coded host without SSL
SQLALCHEMY_DATABASE_URI = "postgresql+psycopg2://inference_user:pwd@db.example.com/model_db"

# After – explicit port and SSL (if required)
SQLALCHEMY_DATABASE_URI = (
    "postgresql+psycopg2://inference_user:pwd@db.example.com:5432/model_db"
    "?sslmode=require"
)

Validation

From a running inference pod, re‑run the nc test; it should report success.
Execute a simple query via psql or the API’s health endpoint:

# Inside pod
psql "host=db.example.com port=5432 user=inference_user dbname=model_db sslmode=require" -c "SELECT 1;"
# Expected output:
?column?
----------
1
(1 row)

Or call GET /health on the FastAPI service and verify a 200 OK with { "db": "connected" }.

Monitor the API latency chart for a return to baseline (< 100 ms per request) and confirm that no new timeout errors appear in journalctl -u inference.service.

Prevention and Best Practices

Infrastructure as Code: Keep firewall rules and NetworkPolicy objects version‑controlled (e.g., Terraform, Helm). Include a pre‑apply validation step that checks egress to required ports.
Automated Connectivity Tests: Deploy a sidecar container that periodically runs nc -zv db.example.com 5432 and emits a Prometheus metric (db_connectivity_up). Alert on metric 0 for > 2 minutes.
pg_hba.conf CIDR Management: Use broader CIDR ranges or host all all 0.0.0.0/0 md5 in controlled environments, and rotate credentials regularly.
Explicit SSL: Enforce sslmode=require in connection strings to avoid silent fallback to non‑SSL paths that may be blocked by firewalls.
Change‑Control Review: Any firewall or network‑policy change must be reviewed for downstream service dependencies, especially for stateful services like databases.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the API time out but psql from a bastion host works?
Because the bastion host is outside the VPC firewall scope and uses a different source IP that is still allowed by pg_hba.conf. The inference pods use a different IP range that was blocked.
Can I keep the NAT rule and avoid changing pg_hba.conf?
Yes, configure PostgreSQL to trust the NAT CIDR or use a hostssl entry with a client certificate, which bypasses IP‑based checks.
What metric should I monitor to catch this early?
Track db_connectivity_up (1 = reachable, 0 = unreachable) and set an alert on a 2‑minute consecutive failure.
Is it safe to open port 5432 to the entire VPC?
In a private VPC with no internet egress, it is acceptable, but you should still enforce SSL and strong passwords. For stricter security, limit egress to the specific subnet of the inference service.
How do I know if a NetworkPolicy is the culprit?
Run kubectl describe networkpolicy -n <namespace> and look for egress rules that omit port 5432. Temporarily delete the policy to see if connectivity restores.