Female speaker presenting in front of a projector screen.

EC2 node port conflict during auto-scaling of AI inference services

by

Problem Description

During rapid auto‑scaling of a real‑time video analytics pipeline, newly launched EC2 instances intermittently crash the AI inference workers and drop video frames. The failure manifests as binding errors from both the TensorFlow Serving containers and the WebSocket ingestion daemons:


EADDRINUSE: address already in use :::8080
BindException: Address already in use (Bind failed)   // TensorFlow Serving
docker: Error response from daemon: driver failed programming external connectivity on endpoint ...: Bind for 0.0.0.0:5000 failed: port is already allocated.
WebSocket server error: listen EADDRINUSE 0.0.0.0:3000

These errors appear in docker logs, journalctl -u inference.service, and the application console. The symptoms are:

  • Worker process termination after docker run returns a non‑zero exit code.
  • Missing frames in the downstream video stream, leading to degraded model accuracy.
  • Auto Scaling events trigger the issue more frequently, indicating a per‑instance port exhaustion or collision.

Root Cause Analysis

The pipeline runs multiple containerized microservices on the same EC2 host:

  • One or more model‑serving containers (TensorFlow Serving or TorchServe) exposing gRPC/REST on a static port (e.g., 5000 or 8080).
  • A WebSocket ingestion service that opens a listener for RTSP/RTMP streams on a static port (e.g., 3000).
  • Optional side‑car data collectors that also bind to fixed ports for metrics export.

When the Auto Scaling group launches a new instance, the launch script (often a UserData or Lifecycle Hook script) starts the same Docker‑Compose file on every node. Because the compose file maps container ports directly to 0.0.0.0:PORT, each instance attempts to bind the same host ports. The EC2 network stack, as described in the AWS EC2 User Guide – Network Interfaces and ENIs, permits only one socket per IP:PORT tuple per instance. Consequently, the second container that tries to bind the same port fails with EADDRINUSE.

Two additional factors exacerbate the problem:

  1. Limited ephemeral port range: The ingestion daemon reuses a static range (3000‑3100) for each video source. Under high concurrency the host runs out of free ports, as documented in the DeepStream‑AWS‑Scale incident.
  2. Security Group overlap: The VPC security group permits inbound traffic on the static ports, but overlapping inbound rules do not resolve the binding conflict; they only allow traffic once the socket is successfully created. This is highlighted in the Amazon VPC Documentation – Security Groups and Inbound/Outbound Rules.

Investigation and Debugging

1. Identify conflicting sockets

sudo ss -tlnp | grep -E '5000|8080|3000'
# Expected output on a healthy node:
# LISTEN 0      128          0.0.0.0:5000          0.0.0.0:*     users:(("docker-proxy",pid=1234,fd=4))
# LISTEN 0      128          0.0.0.0:3000          0.0.0.0:*     users:(("docker-proxy",pid=1235,fd=4))

On a failing node you will see duplicate entries or a “Address already in use” message in the Docker daemon logs.

2. Correlate Docker compose with host ports

cat docker-compose.yml
services:
  tf_serving:
    image: tensorflow/serving
    ports:
      - "5000:8500"
  ws_ingest:
    image: custom/ws-ingest
    ports:
      - "3000:3000"

The static mapping "5000:8500" forces every instance to claim host port 5000.

3. Verify Auto Scaling lifecycle hook execution

aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name video-pipeline-asg
# Look for a hook that runs a post‑launch script (e.g., configure‑ports.sh)

4. Capture a packet trace (optional)

sudo tcpdump -i eth0 port 5000 -c 5 -w /tmp/port5000.pcap
# No traffic will be seen if the socket never bound.

5. Review EC2 ENI limits

The EC2 User Guide notes that each ENI can support a finite number of concurrent connections. When many containers bind to the same port, the ENI’s connection table fills, causing additional bind attempts to fail.

Resolution

Strategy Overview

Replace static host‑port mappings with dynamic, per‑instance offsets and ensure each container discovers a free port at runtime. The solution consists of three parts:

  1. Modify docker-compose.yml to expose container ports only internally (expose) and publish them on a random host port.
  2. Add a startup script (executed via an Auto Scaling lifecycle hook or UserData) that queries the OS for an unused port range and injects the values as environment variables.
  3. Update the inference and ingestion services to read the port from the environment variable instead of hard‑coding it.

Before – Static Port Mapping


services:
  tf_serving:
    image: tensorflow/serving
    ports:
      - "5000:8500"
  ws_ingest:
    image: custom/ws-ingest
    ports:
      - "3000:3000"

After – Dynamic Port Assignment


services:
  tf_serving:
    image: tensorflow/serving
    expose:
      - "8500"
    environment:
      - TF_SERVING_PORT=${TF_SERVING_PORT}
    command: >
      bash -c "python /serve.py --port ${TF_SERVING_PORT}"
  ws_ingest:
    image: custom/ws-ingest
    expose:
      - "3000"
    environment:
      - WS_INGEST_PORT=${WS_INGEST_PORT}
    command: >
      bash -c "node server.js --port ${WS_INGEST_PORT}"

Port Allocation Script (configure‑ports.sh)

#!/usr/bin/env bash
set -euo pipefail

# Find two free ports in the 5000‑6000 range
TF_PORT=$(comm -23 <(seq 5000 6000) <(ss -tlnp | awk '{print $5}' | cut -d: -f2 | sort -u) | head -n 1)
WS_PORT=$(comm -23 <(seq 3000 4000) <(ss -tlnp | awk '{print $5}' | cut -d: -f2 | sort -u) | head -n 1)

# Export for Docker Compose
export TF_SERVING_PORT=$TF_PORT
export WS_INGEST_PORT=$WS_PORT

# Persist for later runs (e.g., systemd service)
echo "TF_SERVING_PORT=$TF_PORT" >> /etc/environment
echo "WS_INGEST_PORT=$WS_PORT" >> /etc/environment

echo "Allocated ports: TF_SERVING=$TF_PORT, WS_INGEST=$WS_PORT"

Run this script from a lifecycle hook:

aws autoscaling complete-lifecycle-action \
  --lifecycle-hook-name configure-ports-hook \
  --auto-scaling-group-name video-pipeline-asg \
  --lifecycle-action-result CONTINUE \
  --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id)

Service Code Adjustments

TensorFlow Serving wrapper (Python example):

import os, argparse, subprocess

parser = argparse.ArgumentParser()
parser.add_argument('--port', type=int, default=int(os.getenv('TF_SERVING_PORT', '8500')))
args = parser.parse_args()

subprocess.run([
    'tensorflow_model_server',
    '--rest_api_port={}'.format(args.port),
    '--model_name=my_model',
    '--model_base_path=/models/my_model'
])

WebSocket server (Node.js example):

const http = require('http');
const ws = require('ws');
const port = process.env.WS_INGEST_PORT || 3000;

const server = http.createServer();
const wss = new ws.Server({ server });

wss.on('connection', socket => {
  // handle streaming data
});

server.listen(port, () => {
  console.log(`WebSocket ingest listening on ${port}`);
});

Verification

  1. Port allocation check – after the instance boots, run: 
    echo $TF_SERVING_PORT $WS_INGEST_PORT
# Example output: 5123 3125
  2. Socket binding verification – confirm no EADDRINUSE errors: 
    sudo ss -tlnp | grep -E '5123|3125'
# Expected output:
# LISTEN 0 128 0.0.0.0:5123 *:* users:(("docker-proxy",pid=2345,fd=4))
# LISTEN 0 128 0.0.0.0:3125 *:* users:(("docker-proxy",pid=2346,fd=4))
  3. Health endpoint – each container now exposes a /healthz that reports the bound port: 
    curl http://localhost:5123/v1/models/my_model/metadata
# 200 OK indicates TensorFlow Serving is reachable.
  4. End‑to‑end video flow – monitor the ingest pipeline (e.g., via CloudWatch metrics IncomingFrames) and confirm that frame‑drop rates return to baseline (< 0.1%).

Prevention and Best Practices

  • Never hard‑code host ports in Docker Compose for services that run on shared EC2 instances. Use expose and let the host assign a free port, or compute an offset per instance.
  • Leverage Auto Scaling lifecycle hooks to run a deterministic port‑allocation script before the service starts. The AWS Auto Scaling – Lifecycle Hooks and Instance Warm‑up guide provides the exact hook registration steps.
  • Reserve a dedicated ENI per microservice when possible. This isolates port namespaces and avoids ENI connection‑table exhaustion.
  • Monitor port binding failures with a CloudWatch metric filter on EADDRINUSE in /var/log/docker.log. Trigger an alarm to catch regressions early.
  • Document the dynamic port range in your security groups. For example, allow inbound traffic on 5000‑6000 (model serving) and 3000‑4000 (WebSocket) only from trusted sources.
  • Implement graceful shutdown in each container so that on instance termination the sockets are released, preventing “zombie” bindings on replacement instances.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

  1. Why does the error appear only after an Auto Scaling event?
    Because the static port mapping is duplicated on the newly launched instance, which shares the same host network namespace as the existing containers. The first instance already occupies the port, so the second fails.
  2. Can I keep using static ports if I assign each container to a different ENI?
    Yes, assigning each microservice to its own ENI creates separate network namespaces, but it adds operational overhead and cost. Dynamic port assignment is usually simpler and scales better.
  3. How do I expose the dynamically assigned ports to external clients?
    Publish the allocated ports via a service registry (e.g., Consul, AWS Cloud Map) or update an Application Load Balancer target group with the instance IP and allocated port using the register-targets CLI.
  4. What if the script cannot find a free port?
    Increase the searched range or reduce the number of concurrent containers per instance. You can also configure the OS net.ipv4.ip_local_port_range to enlarge the ephemeral pool.
  5. Do security groups need to be changed when using dynamic ports?
    Yes, the security group must allow the entire range from which ports are allocated (e.g., 3000‑4000 and 5000‑6000). Overly restrictive rules will cause connection attempts to be blocked even if the bind succeeds.