Sandbox cannot resolve ChromaDB host DNS during vector store init

Problem

During development in a sandbox environment the application fails to initialise the ChromaDB vector store. The client library throws a DNS resolution error and aborts before any HTTP request is sent.

Observed log excerpt:

2026-10-07 14:22:31,842 ERROR chromadb.client - Failed to resolve host 'chroma-db.internal.svc.cluster.local': Name or service not known
Traceback (most recent call last):
  File "/app/vector_store.py", line 42, in init_vector_store
    client = chromadb.Client(settings=client_settings)
  File "/usr/local/lib/python3.11/site-packages/chromadb/client/__init__.py", line 87, in __init__
    self._host = self._resolve_host(settings.host)
  File "/usr/local/lib/python3.11/site-packages/chromadb/client/__init__.py", line 112, in _resolve_host
    raise ConnectionError(f"Failed to resolve host '{host}': {e}")
ConnectionError: Failed to resolve host 'chroma-db.internal.svc.cluster.local': Name or service not known

The failure prevents the vector store from being created, causing downstream API calls that rely on embeddings to error out.

Root Cause

The sandbox runs inside a Docker Compose network that does not provide Kubernetes DNS services. The ChromaDB client is configured with the fully‑qualified Kubernetes service name chroma-db.internal.svc.cluster.local. In a non‑Kubernetes environment this name cannot be resolved because:

• Docker’s internal DNS only knows container names and aliases defined in docker-compose.yml.
• The .svc.cluster.local suffix is a Kubernetes DNS zone served by kube-dns/coredns, which is absent in the sandbox.
• The client library performs a synchronous socket.getaddrinfo() call during initialisation; without a resolver entry the call raises EAI_NONAME.

Consequently the DNS lookup fails before any network connection is attempted.

Debug

Step‑by‑step verification that the problem is DNS‑related:

1. Check the configured host in the application settings.

python -c "import json, pathlib; cfg=json.load(open('config.json')); print(cfg['chroma']['host'])"
# Output:
chroma-db.internal.svc.cluster.local

2. Attempt manual resolution from inside the sandbox container.

docker exec -it sandbox_app sh -c "getent hosts chroma-db.internal.svc.cluster.local"
# Expected (if resolvable):
# 10.5.0.2 chroma-db.internal.svc.cluster.local
# Actual:
getent: chroma-db.internal.svc.cluster.local: Name or service not known

3. Inspect the container’s /etc/resolv.conf.

docker exec sandbox_app cat /etc/resolv.conf
# Sample output:
nameserver 127.0.0.11
options ndots:0
search .

The resolver is Docker’s built‑in DNS server; no search domain for .svc.cluster.local is configured.

4. Confirm that the ChromaDB service is reachable via its Docker Compose service name.

docker exec sandbox_app ping -c 1 chroma-db
# Output:
PING chroma-db (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: icmp_seq=0 ttl=64 time=0.123 ms

Resolution works with the Docker service name, confirming the network path is functional.

Solution

Two practical approaches are provided. Choose the one that aligns with your deployment strategy.

Option A – Adjust application configuration for the sandbox

Replace the Kubernetes DNS name with the Docker Compose service name (chroma-db) when the code detects a non‑Kubernetes runtime.

Before (config.json):

{
  "chroma": {
    "host": "chroma-db.internal.svc.cluster.local",
    "port": 8000,
    "ssl": false
  }
}

After (config.json):

{
  "chroma": {
    "host": "chroma-db",
    "port": 8000,
    "ssl": false
  }
}

Alternatively, inject an environment variable that overrides the host at runtime:

# Docker‑compose snippet
services:
  app:
    environment:
      - CHROMA_HOST=chroma-db

In the Python initialisation code:

import os
from chromadb.config import Settings

host = os.getenv("CHROMA_HOST", "chroma-db.internal.svc.cluster.local")
client_settings = Settings(
    chroma_api_impl="rest",
    host=host,
    port=8000,
    ssl=False,
)
client = chromadb.Client(settings=client_settings)

Option B – Provide a local DNS entry for the Kubernetes name

If you prefer to keep the original host string, add a static entry to the container’s /etc/hosts via Docker Compose.

# docker-compose.yml excerpt
services:
  app:
    extra_hosts:
      - "chroma-db.internal.svc.cluster.local:172.20.0.2"

This maps the Kubernetes FQDN to the IP address of the ChromaDB container, allowing the existing configuration to work unchanged.

Verify

After applying either option, repeat the resolution test and start the application.

# Verify DNS resolution
docker exec sandbox_app getent hosts chroma-db.internal.svc.cluster.local
# Expected (Option B):
172.20.0.2 chroma-db.internal.svc.cluster.local

# Or (Option A)
docker exec sandbox_app getent hosts chroma-db
# Expected:
172.20.0.2 chroma-db

Run the vector store initialisation script:

python -c "import vector_store; vector_store.init_vector_store()"
# Expected output:
Vector store initialised successfully – collection 'documents' ready.

Check the application logs for the absence of DNS errors and confirm that subsequent embedding queries succeed.

Prevent

• Externalise service hostnames per environment (Kubernetes vs. Docker) using environment variables or a configuration management tool.
• Include a health‑check endpoint that validates DNS resolution at container start‑up; fail fast if the host cannot be resolved.
• Document the required extra_hosts mapping in the sandbox README to avoid accidental regression.
• Consider using a local DNS server (e.g., dnsmasq) in the sandbox if multiple services share Kubernetes‑style names.
• Automate a sanity test in CI that resolves all external service names against the target environment.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

1. Why does the same code work in my Kubernetes cluster but fail locally?
   Because Kubernetes provides coredns which resolves .svc.cluster.local names. The local Docker network lacks that DNS zone, so the name cannot be resolved.
2. Can I keep the Kubernetes hostname and still run in Docker?
   Yes. Add a static entry via extra_hosts in docker-compose.yml or mount a custom /etc/hosts file that maps the FQDN to the container IP.
3. Is there a way to detect automatically whether the runtime is Kubernetes?
   Check for the presence of the service account token at /var/run/secrets/kubernetes.io/serviceaccount/token or the environment variable KUBERNETES_SERVICE_HOST. Use this to select the appropriate host name.
4. What if the ChromaDB container’s IP changes (e.g., after a redeploy)?
   Static extra_hosts entries become stale. Prefer using the Docker service name (chroma-db) or a DNS alias managed by Docker Compose, which resolves to the current IP automatically.
5. Do I need to enable TLS when switching to the Docker service name?
   Only if the ChromaDB instance is configured for TLS. The example assumes plain HTTP (SSL disabled). Adjust the ssl flag and port accordingly if TLS is required.