Kafka consumer cannot reach inference API after firewall change
Problem
A Kafka consumer service running on an on‑premises host continuously pulls model‑prediction request messages from a topic and forwards them to an internal inference API (HTTP/REST). After a scheduled firewall rule update, the consumer started failing with connection‑time‑out errors. The downstream API is otherwise healthy; direct curl from an admin workstation succeeds.
Observed Symptoms
- Consumer logs show repeated
java.net.ConnectException: Connection timed outfor the inference endpoint. - Kafka consumer lag grows rapidly; the topic offset stops advancing.
- Metrics:
consumer.request.durationspikes to >30 s,consumer.request.errorsincreases. - No restart of the consumer resolves the issue.
Sample Consumer Log
2026-11-13 09:42:17,842 ERROR [consumer-1] com.example.inference.InferenceClient - Failed to POST /predict: java.net.ConnectException: Connection timed out (Connection refused)
at com.example.inference.InferenceClient.sendRequest(InferenceClient.java:78)
at com.example.consumer.PredictionProcessor.processMessage(PredictionProcessor.java:45)
...
Root Cause
The firewall change introduced a rule that blocked outbound TCP traffic from the consumer host (IP 10.1.12.34) to the inference API server (IP 10.1.20.50) on port 8080. The rule was intended to restrict access to a different service, but the CIDR range overlapped the consumer’s source address range, causing the inference API to become unreachable.
Key points:
- The inference API listens on
0.0.0.0:8080and is reachable from the same subnet. - The new firewall rule was:
iptables -A OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECT - Because the consumer’s source IP falls within
10.1.20.0/24, the rule matched and dropped the packets. - No DNS or TLS issues were involved; the failure occurs before the TCP three‑way handshake.
Debug
The investigation followed a systematic approach:
1. Verify network connectivity from the consumer host
# Attempt a TCP connection with timeout
nc -vz 10.1.20.50 8080
Connection timed out
# Compare with a host that is still allowed
nc -vz 10.1.30.60 8080
Connection to 10.1.30.60 8080 port [tcp/http-alt] succeeded!
2. Inspect firewall rules
# List OUTPUT chain
iptables -S OUTPUT
-P OUTPUT ACCEPT
-A OUTPUT -d 10.1.20.0/24 -p tcp -m tcp --dport 8080 -j REJECT
-A OUTPUT -d 10.1.30.0/24 -p tcp -m tcp --dport 8080 -j ACCEPT
3. Capture packets to confirm rejection
# Run tcpdump on the consumer host
tcpdump -i eth0 host 10.1.20.50 and port 8080 -nn -vv
14:22:31.123456 IP 10.1.12.34.54321 > 10.1.20.50.8080: Flags [S], seq 123456789, win 29200, length 0
14:22:31.124001 IP 10.1.20.50.8080 > 10.1.12.34.54321: Flags [R.], seq 0, ack 123456790, win 0, length 0
The SYN packet is immediately reset, confirming a local reject.
4. Review recent change management records
The change log indicated a firewall rule addition on 2026‑11‑12 to block port 8080 for a deprecated analytics service. The CIDR range was mistakenly set to 10.1.20.0/24 instead of the intended 10.1.25.0/24.
Solution
Two corrective actions were applied:
Option A – Adjust the firewall rule (preferred)
# Delete the overly broad rule
iptables -D OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECT
# Add a precise rule targeting only the deprecated service subnet
iptables -A OUTPUT -d 10.1.25.0/24 -p tcp --dport 8080 -j REJECT
Option B – Add an explicit allow rule for the inference API
# Insert allow rule before the reject rule
iptables -I OUTPUT 1 -d 10.1.20.50 -p tcp --dport 8080 -j ACCEPT
Option A restores the intended network segmentation while preserving the security posture. Option B is a quick fix but should be reviewed for policy compliance.
Code change (if firewall cannot be modified immediately)
Temporarily route inference traffic through a local proxy on an allowed port.
// Before – direct call
HttpResponse resp = HttpClient.newHttpClient()
.send(HttpRequest.newBuilder()
.uri(URI.create("http://10.1.20.50:8080/predict"))
.POST(BodyPublishers.ofString(payload))
.build(),
BodyHandlers.ofString());
// After – proxy via localhost:9000 (allowed)
HttpResponse resp = HttpClient.newHttpClient()
.send(HttpRequest.newBuilder()
.uri(URI.create("http://127.0.0.1:9000/predict"))
.header("X-Forwarded-Host", "10.1.20.50")
.POST(BodyPublishers.ofString(payload))
.build(),
BodyHandlers.ofString());
The proxy forwards the request to the real API, bypassing the blocked outbound path. This change should be treated as a temporary measure.
Verify
After applying the firewall correction, the following checks confirm restoration of service:
Connectivity Test
# From the consumer host
curl -s -o /dev/null -w "%{http_code}" http://10.1.20.50:8080/healthz
200
Consumer Log Confirmation
2026-11-13 10:05:12,317 INFO [consumer-1] com.example.inference.InferenceClient - POST /predict succeeded, status=200
Kafka Lag Metrics
| Metric | Before Fix | After Fix |
|---|---|---|
| consumer.lag | ≈ 150 k | ≈ 0 |
| consumer.request.errors | ≈ 12 k/min | ≈ 0/min |
Prevent
- Change Review Guardrails: Require explicit CIDR verification for firewall rule changes; use a peer‑review checklist that includes “does this rule affect existing services?”.
- Automated Diff Checks: Integrate a CI job that diffs
iptables-saveoutput against a baseline and flags newREJECT/DROPrules covering critical service subnets. - Network Monitoring: Deploy a synthetic health probe (e.g.,
curlto/healthz) from each consumer host every minute; alert on non‑200 responses. - Documentation: Maintain an inventory of service IP ranges and required outbound ports; keep it version‑controlled alongside firewall configuration.
- Rollback Procedure: Keep the previous firewall rule set in a snapshot; enable rapid rollback with a single
iptables-restorecommand.
Related Topic Hub: Data Infrastructure Troubleshooting Hub
FAQ
- Why did the consumer still see “Connection timed out” instead of “Connection refused”?
The firewall rule used
-j REJECTwith the defaulticmp-host-prohibitedresponse, which appears as a timeout to the Java socket layer. If-j DROPhad been used, the symptom would have been an even longer timeout. - Can I use
iptables -Cto test for existing rules before adding new ones?Yes.
iptables -C OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECTreturns 0 if the rule exists, non‑zero otherwise. Incorporate this in deployment scripts to avoid duplicate entries. - Is there a way to see which process is being blocked by the firewall?
Use
ss -tpnto list TCP sockets with process IDs, then correlate withiptables -L -v -ncounters. Thepktscolumn on the reject rule will increase when the consumer attempts a connection. - Will enabling
nf_conntrackhelp diagnose similar issues?Enabling connection tracking allows you to view stateful entries via
conntrack -L. A missing entry for the outbound SYN indicates the packet never left the host, confirming a local firewall block. - How should I handle firewall changes in a multi‑datacenter deployment?
Apply identical rule sets via configuration management (e.g., Ansible, Chef) and run a post‑deployment validation playbook that performs a connectivity test from each consumer node to every downstream service.