Problem: Milvus node startup fails with “port already in use” on Docker Compose
When launching the Milvus stack in a local development sandbox (Docker Compose on a Linux workstation), the milvus container exits immediately with an error similar to:
2024-11-16 10:12:34.567 INFO Milvus[Node] - Starting Milvus node...
2024-11-16 10:12:34.568 ERROR Milvus[Node] - Failed to bind to port 19530: Address already in use
2024-11-16 10:12:34.569 FATAL Milvus[Node] - Node startup aborted
The service becomes unavailable, and dependent applications (e.g., SDK clients, vector search pipelines) cannot connect.
Root Cause
Milvus uses two primary TCP ports:
- 19530 – gRPC service for client SDKs.
- 19121 – HTTP REST gateway (optional).
In a Docker Compose environment, each service is isolated, but port bindings are performed on the host network namespace. The conflict arises when:
- Another container (or a leftover process from a previous run) is already bound to
0.0.0.0:19530or0.0.0.0:19121. - The
docker-compose.ymlfile maps the same host port to multiple services (e.g., a custommilvus-proxyservice and the mainmilvusservice both expose19530). - The host OS has a non‑Docker process (e.g., a local Milvus binary, a stray
python -m http.server, or a previousdocker runthat was not cleaned up) listening on the required port.
Because Milvus aborts if it cannot bind its listening socket, the node never reaches the “ready” state, leading to the observed startup failure.
Debug: Investigation Steps
1. Identify the process holding the port
# Check which PID is bound to 19530
sudo ss -ltnp | grep ':19530'
# Example output
LISTEN 0 128 0.0.0.0:19530 0.0.0.0:* users:(("docker-proxy",pid=12456,fd=5))
If the docker-proxy PID belongs to a container that is still running, that container is the conflict source.
2. List Docker containers exposing the conflicting ports
docker ps --filter "publish=19530"
# Example output
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a1b2c3d4e5f6 milvusdb/milvus:2.3 "milvus run" 2 hours ago Up 2 hours 0.0.0.0:19530->19530/tcp milvus
d7e8f9g0h1i2 custom-proxy:latest "python app.py" 30 minutes ago Up 30 minutes 0.0.0.0:19530->19530/tcp milvus-proxy
Two containers publishing the same host port indicate a misconfiguration in docker-compose.yml.
3. Verify the Compose file for duplicate port mappings
# docker-compose.yml (excerpt)
services:
milvus:
image: milvusdb/milvus:2.3
ports:
- "19530:19530"
- "19121:19121"
milvus-proxy:
image: custom-proxy:latest
ports:
- "19530:19530" # ← conflict
4. Check for orphaned containers
# List stopped containers that might still hold network namespaces
docker ps -a --filter "status=exited"
# Remove them if they are no longer needed
docker rm $(docker ps -a -q --filter "status=exited")
5. Confirm no host‑level process is using the port
# Kill any stray process (use with caution)
sudo fuser -k 19530/tcp
Solution: Resolve the Port Conflict
Option A – Consolidate Port Exposure
If the proxy is only a thin wrapper around Milvus’s gRPC endpoint, you can remove the explicit port mapping from the proxy service and let it communicate via Docker’s internal network.
# Before (docker‑compose.yml)
services:
milvus:
ports:
- "19530:19530"
milvus-proxy:
ports:
- "19530:19530"
# After
services:
milvus:
ports:
- "19530:19530"
milvus-proxy:
# No host port exposure; use service name for internal calls
depends_on:
- milvus
Now the proxy reaches Milvus at milvus:19530 without binding the same host port.
Option B – Change Host Port Mapping
If both services must be reachable from the host, assign distinct host ports.
# Before
ports:
- "19530:19530"
# After (milvus-proxy)
ports:
- "19531:19530" # host 19531 forwards to container 19530
Clients that need to talk to the proxy must be updated to use localhost:19531.
Option C – Clean Up Orphaned Containers
When the conflict is caused by a leftover container:
# Stop and remove the offending container
docker stop milvus-proxy
docker rm milvus-proxy
Then restart the stack:
docker-compose down
docker-compose up -d
Verify: Confirm Milvus Node is Running
1. Container status
docker ps --filter "name=milvus"
# Expected output
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a1b2c3d4e5f6 milvusdb/milvus:2.3 "milvus run" 5 seconds ago Up 4 seconds 0.0.0.0:19530->19530/tcp milvus
2. Log inspection
docker logs milvus | grep "Node startup"
# Expected snippet
2024-11-16 10:15:02.123 INFO Milvus[Node] - Starting Milvus node...
2024-11-16 10:15:02.456 INFO Milvus[Node] - Node successfully bound to 0.0.0.0:19530
2024-11-16 10:15:02.789 INFO Milvus[Node] - Ready to accept connections
3. Connectivity test
# Using the Milvus Python SDK
from pymilvus import connections
connections.connect(alias="default", host="127.0.0.1", port="19530")
print("Connected:", connections.has_connection("default"))
# Expected output
Connected: True
Prevent: Guardrails to Avoid Future Port Collisions
- Explicit port inventory: Maintain a single source of truth (e.g., a
ports.yaml) listing host ports used by all services in the sandbox. - Compose validation: Add a pre‑flight script that runs
docker-compose configand greps for duplicate port mappings. - Automated cleanup: Include
docker-compose down --remove-orphansin CI/CD or developer start‑up scripts. - Dynamic host ports: When possible, omit the host side of the mapping (use
ports: - "19530") so Docker assigns an unused host port automatically. - Monitoring: Add a Prometheus scrape of
node_exporterto alert onprocess_open_fdsortcp_listenmetrics that exceed a threshold for the Milvus ports.
Related Topic Hub: Vector Databases Troubleshooting Hub
FAQ
- Why does the conflict appear only after a Docker Compose restart?
Because Docker may leave stopped containers in theexitedstate with their network namespace still bound. A subsequentdocker-compose upcan inadvertently reuse the same host port, causing a clash. - Can I run multiple Milvus instances on the same host?
Yes, but each instance must have a unique set of host ports. Use distinct mappings (e.g.,19530:19530and19531:19530) or rely on Docker’s internal DNS to avoid exposing them on the host. - Is the “address already in use” error ever benign?
Only when the process holding the port is the same Milvus instance that is restarting gracefully. In that case, the old process should release the socket before the new one binds; a forced kill or abrupt stop can leave the socket inTIME_WAITstate, which is usually resolved after a few seconds. - How do I differentiate between a Docker‑proxy conflict and a host‑level process?
Thess -ltnpoutput shows the owning binary.docker-proxyindicates a container; any other binary (e.g.,java,python) points to a host process. - What should I do if I need both Milvus and a custom proxy to listen on the same external port?
Place them behind a reverse proxy (e.g., Nginx) that terminates on a single host port and routes traffic based on path or SNI. The reverse proxy then forwards to distinct internal ports, eliminating the direct conflict.