Kafka consumer cannot reach inference API after firewall change

Kafka consumer cannot reach inference API after firewall change

Problem

A Kafka consumer service running on an on‑premises host continuously pulls model‑prediction request messages from a topic and forwards them to an internal inference API (HTTP/REST). After a scheduled firewall rule update, the consumer started failing with connection‑time‑out errors. The downstream API is otherwise healthy; direct curl from an admin workstation succeeds.

Observed Symptoms

  • Consumer logs show repeated java.net.ConnectException: Connection timed out for the inference endpoint.
  • Kafka consumer lag grows rapidly; the topic offset stops advancing.
  • Metrics: consumer.request.duration spikes to >30 s, consumer.request.errors increases.
  • No restart of the consumer resolves the issue.

Sample Consumer Log


2026-11-13 09:42:17,842 ERROR [consumer-1] com.example.inference.InferenceClient - Failed to POST /predict: java.net.ConnectException: Connection timed out (Connection refused)
	at com.example.inference.InferenceClient.sendRequest(InferenceClient.java:78)
	at com.example.consumer.PredictionProcessor.processMessage(PredictionProcessor.java:45)
...

Root Cause

The firewall change introduced a rule that blocked outbound TCP traffic from the consumer host (IP 10.1.12.34) to the inference API server (IP 10.1.20.50) on port 8080. The rule was intended to restrict access to a different service, but the CIDR range overlapped the consumer’s source address range, causing the inference API to become unreachable.

Key points:

  • The inference API listens on 0.0.0.0:8080 and is reachable from the same subnet.
  • The new firewall rule was:
    iptables -A OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECT
  • Because the consumer’s source IP falls within 10.1.20.0/24, the rule matched and dropped the packets.
  • No DNS or TLS issues were involved; the failure occurs before the TCP three‑way handshake.

Debug

The investigation followed a systematic approach:

1. Verify network connectivity from the consumer host

# Attempt a TCP connection with timeout
nc -vz 10.1.20.50 8080
Connection timed out

# Compare with a host that is still allowed
nc -vz 10.1.30.60 8080
Connection to 10.1.30.60 8080 port [tcp/http-alt] succeeded!

2. Inspect firewall rules

# List OUTPUT chain
iptables -S OUTPUT
-P OUTPUT ACCEPT
-A OUTPUT -d 10.1.20.0/24 -p tcp -m tcp --dport 8080 -j REJECT
-A OUTPUT -d 10.1.30.0/24 -p tcp -m tcp --dport 8080 -j ACCEPT

3. Capture packets to confirm rejection

# Run tcpdump on the consumer host
tcpdump -i eth0 host 10.1.20.50 and port 8080 -nn -vv
14:22:31.123456 IP 10.1.12.34.54321 > 10.1.20.50.8080: Flags [S], seq 123456789, win 29200, length 0
14:22:31.124001 IP 10.1.20.50.8080 > 10.1.12.34.54321: Flags [R.], seq 0, ack 123456790, win 0, length 0

The SYN packet is immediately reset, confirming a local reject.

4. Review recent change management records

The change log indicated a firewall rule addition on 2026‑11‑12 to block port 8080 for a deprecated analytics service. The CIDR range was mistakenly set to 10.1.20.0/24 instead of the intended 10.1.25.0/24.

Solution

Two corrective actions were applied:

Option A – Adjust the firewall rule (preferred)

# Delete the overly broad rule
iptables -D OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECT

# Add a precise rule targeting only the deprecated service subnet
iptables -A OUTPUT -d 10.1.25.0/24 -p tcp --dport 8080 -j REJECT

Option B – Add an explicit allow rule for the inference API

# Insert allow rule before the reject rule
iptables -I OUTPUT 1 -d 10.1.20.50 -p tcp --dport 8080 -j ACCEPT

Option A restores the intended network segmentation while preserving the security posture. Option B is a quick fix but should be reviewed for policy compliance.

Code change (if firewall cannot be modified immediately)

Temporarily route inference traffic through a local proxy on an allowed port.


// Before – direct call
HttpResponse resp = HttpClient.newHttpClient()
    .send(HttpRequest.newBuilder()
        .uri(URI.create("http://10.1.20.50:8080/predict"))
        .POST(BodyPublishers.ofString(payload))
        .build(),
    BodyHandlers.ofString());

// After – proxy via localhost:9000 (allowed)
HttpResponse resp = HttpClient.newHttpClient()
    .send(HttpRequest.newBuilder()
        .uri(URI.create("http://127.0.0.1:9000/predict"))
        .header("X-Forwarded-Host", "10.1.20.50")
        .POST(BodyPublishers.ofString(payload))
        .build(),
    BodyHandlers.ofString());

The proxy forwards the request to the real API, bypassing the blocked outbound path. This change should be treated as a temporary measure.

Verify

After applying the firewall correction, the following checks confirm restoration of service:

Connectivity Test

# From the consumer host
curl -s -o /dev/null -w "%{http_code}" http://10.1.20.50:8080/healthz
200

Consumer Log Confirmation


2026-11-13 10:05:12,317 INFO  [consumer-1] com.example.inference.InferenceClient - POST /predict succeeded, status=200

Kafka Lag Metrics

Metric Before Fix After Fix
consumer.lag ≈ 150 k ≈ 0
consumer.request.errors ≈ 12 k/min ≈ 0/min

Prevent

  • Change Review Guardrails: Require explicit CIDR verification for firewall rule changes; use a peer‑review checklist that includes “does this rule affect existing services?”.
  • Automated Diff Checks: Integrate a CI job that diffs iptables-save output against a baseline and flags new REJECT/DROP rules covering critical service subnets.
  • Network Monitoring: Deploy a synthetic health probe (e.g., curl to /healthz) from each consumer host every minute; alert on non‑200 responses.
  • Documentation: Maintain an inventory of service IP ranges and required outbound ports; keep it version‑controlled alongside firewall configuration.
  • Rollback Procedure: Keep the previous firewall rule set in a snapshot; enable rapid rollback with a single iptables-restore command.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

  1. Why did the consumer still see “Connection timed out” instead of “Connection refused”?

    The firewall rule used -j REJECT with the default icmp-host-prohibited response, which appears as a timeout to the Java socket layer. If -j DROP had been used, the symptom would have been an even longer timeout.

  2. Can I use iptables -C to test for existing rules before adding new ones?

    Yes. iptables -C OUTPUT -d 10.1.20.0/24 -p tcp --dport 8080 -j REJECT returns 0 if the rule exists, non‑zero otherwise. Incorporate this in deployment scripts to avoid duplicate entries.

  3. Is there a way to see which process is being blocked by the firewall?

    Use ss -tpn to list TCP sockets with process IDs, then correlate with iptables -L -v -n counters. The pkts column on the reject rule will increase when the consumer attempts a connection.

  4. Will enabling nf_conntrack help diagnose similar issues?

    Enabling connection tracking allows you to view stateful entries via conntrack -L. A missing entry for the outbound SYN indicates the packet never left the host, confirming a local firewall block.

  5. How should I handle firewall changes in a multi‑datacenter deployment?

    Apply identical rule sets via configuration management (e.g., Ansible, Chef) and run a post‑deployment validation playbook that performs a connectivity test from each consumer node to every downstream service.