Milvus node startup fails with port already in use on Docker Compose

Problem: Milvus node startup fails with “port already in use” on Docker Compose

When launching the Milvus stack in a local development sandbox (Docker Compose on a Linux workstation), the milvus container exits immediately with an error similar to:

2024-11-16 10:12:34.567 INFO  Milvus[Node] - Starting Milvus node...
2024-11-16 10:12:34.568 ERROR Milvus[Node] - Failed to bind to port 19530: Address already in use
2024-11-16 10:12:34.569 FATAL Milvus[Node] - Node startup aborted

The service becomes unavailable, and dependent applications (e.g., SDK clients, vector search pipelines) cannot connect.

Root Cause

Milvus uses two primary TCP ports:

  • 19530 – gRPC service for client SDKs.
  • 19121 – HTTP REST gateway (optional).

In a Docker Compose environment, each service is isolated, but port bindings are performed on the host network namespace. The conflict arises when:

  1. Another container (or a leftover process from a previous run) is already bound to 0.0.0.0:19530 or 0.0.0.0:19121.
  2. The docker-compose.yml file maps the same host port to multiple services (e.g., a custom milvus-proxy service and the main milvus service both expose 19530).
  3. The host OS has a non‑Docker process (e.g., a local Milvus binary, a stray python -m http.server, or a previous docker run that was not cleaned up) listening on the required port.

Because Milvus aborts if it cannot bind its listening socket, the node never reaches the “ready” state, leading to the observed startup failure.

Debug: Investigation Steps

1. Identify the process holding the port

# Check which PID is bound to 19530
sudo ss -ltnp | grep ':19530'

# Example output
LISTEN 0      128          0.0.0.0:19530        0.0.0.0:*    users:(("docker-proxy",pid=12456,fd=5))

If the docker-proxy PID belongs to a container that is still running, that container is the conflict source.

2. List Docker containers exposing the conflicting ports

docker ps --filter "publish=19530"
# Example output
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS          PORTS                     NAMES
a1b2c3d4e5f6   milvusdb/milvus:2.3 "milvus run"             2 hours ago      Up 2 hours      0.0.0.0:19530->19530/tcp  milvus
d7e8f9g0h1i2   custom-proxy:latest "python app.py"          30 minutes ago   Up 30 minutes   0.0.0.0:19530->19530/tcp  milvus-proxy

Two containers publishing the same host port indicate a misconfiguration in docker-compose.yml.

3. Verify the Compose file for duplicate port mappings

# docker-compose.yml (excerpt)
services:
  milvus:
    image: milvusdb/milvus:2.3
    ports:
      - "19530:19530"
      - "19121:19121"

  milvus-proxy:
    image: custom-proxy:latest
    ports:
      - "19530:19530"   # ← conflict

4. Check for orphaned containers

# List stopped containers that might still hold network namespaces
docker ps -a --filter "status=exited"

# Remove them if they are no longer needed
docker rm $(docker ps -a -q --filter "status=exited")

5. Confirm no host‑level process is using the port

# Kill any stray process (use with caution)
sudo fuser -k 19530/tcp

Solution: Resolve the Port Conflict

Option A – Consolidate Port Exposure

If the proxy is only a thin wrapper around Milvus’s gRPC endpoint, you can remove the explicit port mapping from the proxy service and let it communicate via Docker’s internal network.

# Before (docker‑compose.yml)
services:
  milvus:
    ports:
      - "19530:19530"
  milvus-proxy:
    ports:
      - "19530:19530"

# After
services:
  milvus:
    ports:
      - "19530:19530"
  milvus-proxy:
    # No host port exposure; use service name for internal calls
    depends_on:
      - milvus

Now the proxy reaches Milvus at milvus:19530 without binding the same host port.

Option B – Change Host Port Mapping

If both services must be reachable from the host, assign distinct host ports.

# Before
ports:
  - "19530:19530"

# After (milvus-proxy)
ports:
  - "19531:19530"   # host 19531 forwards to container 19530

Clients that need to talk to the proxy must be updated to use localhost:19531.

Option C – Clean Up Orphaned Containers

When the conflict is caused by a leftover container:

# Stop and remove the offending container
docker stop milvus-proxy
docker rm milvus-proxy

Then restart the stack:

docker-compose down
docker-compose up -d

Verify: Confirm Milvus Node is Running

1. Container status

docker ps --filter "name=milvus"
# Expected output
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS          PORTS                     NAMES
a1b2c3d4e5f6   milvusdb/milvus:2.3 "milvus run"             5 seconds ago    Up 4 seconds    0.0.0.0:19530->19530/tcp  milvus

2. Log inspection

docker logs milvus | grep "Node startup"
# Expected snippet
2024-11-16 10:15:02.123 INFO  Milvus[Node] - Starting Milvus node...
2024-11-16 10:15:02.456 INFO  Milvus[Node] - Node successfully bound to 0.0.0.0:19530
2024-11-16 10:15:02.789 INFO  Milvus[Node] - Ready to accept connections

3. Connectivity test

# Using the Milvus Python SDK
from pymilvus import connections
connections.connect(alias="default", host="127.0.0.1", port="19530")
print("Connected:", connections.has_connection("default"))
# Expected output
Connected: True

Prevent: Guardrails to Avoid Future Port Collisions

  • Explicit port inventory: Maintain a single source of truth (e.g., a ports.yaml) listing host ports used by all services in the sandbox.
  • Compose validation: Add a pre‑flight script that runs docker-compose config and greps for duplicate port mappings.
  • Automated cleanup: Include docker-compose down --remove-orphans in CI/CD or developer start‑up scripts.
  • Dynamic host ports: When possible, omit the host side of the mapping (use ports: - "19530") so Docker assigns an unused host port automatically.
  • Monitoring: Add a Prometheus scrape of node_exporter to alert on process_open_fds or tcp_listen metrics that exceed a threshold for the Milvus ports.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the conflict appear only after a Docker Compose restart?
    Because Docker may leave stopped containers in the exited state with their network namespace still bound. A subsequent docker-compose up can inadvertently reuse the same host port, causing a clash.
  2. Can I run multiple Milvus instances on the same host?
    Yes, but each instance must have a unique set of host ports. Use distinct mappings (e.g., 19530:19530 and 19531:19530) or rely on Docker’s internal DNS to avoid exposing them on the host.
  3. Is the “address already in use” error ever benign?
    Only when the process holding the port is the same Milvus instance that is restarting gracefully. In that case, the old process should release the socket before the new one binds; a forced kill or abrupt stop can leave the socket in TIME_WAIT state, which is usually resolved after a few seconds.
  4. How do I differentiate between a Docker‑proxy conflict and a host‑level process?
    The ss -ltnp output shows the owning binary. docker-proxy indicates a container; any other binary (e.g., java, python) points to a host process.
  5. What should I do if I need both Milvus and a custom proxy to listen on the same external port?
    Place them behind a reverse proxy (e.g., Nginx) that terminates on a single host port and routes traffic based on path or SNI. The reverse proxy then forwards to distinct internal ports, eliminating the direct conflict.