Weaviate Connection Timeout During Nightly RAG Ingestion Pipeline

Q: Debug – Step‑by‑Step Investigation 1. Examine pipeline logs

2026-09-24 02:13:45,212 [RAG-INGEST] ERROR WeaviateClient: ConnectionError: HTTPSConnectionPool(host='weaviate.internal', port=443): Max retries exceeded with url: /v1/objects (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c2b1c5d30>, 'Connection to weaviate.internal timed out. (connect timeout=30)')) 2026-09-24 02:13:45,215 [RAG-INGEST] INFO Retry attempt 1/3 2026-09-24 02:14:16,789 [RAG-INGEST] ERROR WeaviateClient: SSLError: SSL: TLSV1_ALERT_PROTOCOL_VERSION 2026-09-24 02:14:16,790 [RAG-INGEST] INFO Retrying after 5 seconds... 2. Verify TLS support on the server openssl s_client -connect weaviate.internal:443 -tls1_3 -servername weaviate.internal Output (truncated):

Q: 4. Inspect connection pool metrics

kubectl exec -it rag-pipeline-0 -- curl -s http://localhost:8000/metrics | grep urllib3_pool_connections weaviate_client_urllib3_pool_connections{host="weaviate.internal"} 10 weaviate_client_urllib3_pool_connections{host="weaviate.internal"} 10 The metric never exceeds the default pool size of 10, indicating queuing.

Q: 2. Increase the urllib3 connection pool size and set a higher timeout

Pass a custom urllib3.PoolManager to the client.

Q: Verify – Confirm the Pipeline Completes Successfully 1. Health‑check the Weaviate endpoint

curl -s -k https://weaviate.internal/v1/.well-known/ready | jq . Expected output:

Q: 2. Run a single‑batch dry‑run

python ingest.py --dry-run --batch-size 500 Log excerpt after fix:

Q: 3. Verify document count in Weaviate

curl -s -k https://weaviate.internal/v1/objects?class=Document | jq '.objects | length' The count should increase by the expected number of new documents (e.g., +30000).

In this article

Table of Contents

Problem – Nightly RAG Ingestion Fails with Weaviate Connection Timeouts

The automated Retrieval‑Augmented Generation (RAG) pipeline runs every night to ingest fresh documents, generate embeddings, and store them in a Weaviate vector store. Since the last deployment, the pipeline aborts during the store_vectors step with repeated connection timeout errors and occasional authentication failures. No new vectors are indexed, causing downstream RAG queries to miss the latest content.

Root Cause – Mismatched TLS Settings and Exhausted Connection Pool

Two intertwined issues were identified:

TLS version mismatch: The Weaviate service was upgraded to v1.23 which defaults to TLSv1.3. The Python weaviate-client library used by the pipeline was pinned to requests==2.28 that, in the container’s OpenSSL 1.1.1k build, disables TLSv1.3 by default. The handshake succeeded only when the client fell back to TLSv1.2, but the server rejected the fallback after a short grace period, leading to SSL_ERROR_SSL: TLS handshake failed and subsequent timeout.
Connection‑pool exhaustion: The pipeline creates a new WeaviateClient instance for each batch of 500 documents. Under the nightly load (≈30 k documents) this spawns ~60 concurrent HTTP connections. The default urllib3 pool size is 10, causing the client to block on poolmanager.connection_from_host and eventually hit the default 30‑second timeout.

Debug – Step‑by‑Step Investigation

1. Examine pipeline logs


2026-09-24 02:13:45,212 [RAG-INGEST] ERROR WeaviateClient: ConnectionError: HTTPSConnectionPool(host='weaviate.internal', port=443): Max retries exceeded with url: /v1/objects (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c2b1c5d30>, 'Connection to weaviate.internal timed out. (connect timeout=30)'))
2026-09-24 02:13:45,215 [RAG-INGEST] INFO  Retry attempt 1/3
2026-09-24 02:14:16,789 [RAG-INGEST] ERROR WeaviateClient: SSLError: SSL: TLSV1_ALERT_PROTOCOL_VERSION
2026-09-24 02:14:16,790 [RAG-INGEST] INFO  Retrying after 5 seconds...

2. Verify TLS support on the server

openssl s_client -connect weaviate.internal:443 -tls1_3 -servername weaviate.internal

Output (truncated):


...
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
...

3. Test client handshake with explicit TLS version

python - <<'PY'
import requests, urllib3
urllib3.disable_warnings()
session = requests.Session()
adapter = urllib3.util.ssl_.SSLContext(urllib3.util.ssl_.PROTOCOL_TLSv1_2)
session.mount('https://', urllib3.poolmanager.ProxyManager('https://weaviate.internal', ssl_context=adapter))
resp = session.get('https://weaviate.internal/v1/.well-known/ready')
print(resp.status_code)
PY

Result: 200 – TLSv1.2 works, confirming the client cannot negotiate TLSv1.3.

4. Inspect connection pool metrics

kubectl exec -it rag-pipeline-0 -- curl -s http://localhost:8000/metrics | grep urllib3_pool_connections


weaviate_client_urllib3_pool_connections{host="weaviate.internal"} 10
weaviate_client_urllib3_pool_connections{host="weaviate.internal"} 10

The metric never exceeds the default pool size of 10, indicating queuing.

5. Reproduce the timeout locally

python - <<'PY'
from weaviate import Client
import time
c = Client("https://weaviate.internal")
start = time.time()
for i in range(60):
    c.batch.add_data_object({"text": f"doc {i}"}, "Document")
c.batch.create_objects()
print("elapsed:", time.time() - start)
PY

Result: elapsed: 45.2 seconds, with a ConnectTimeoutError after the 10th concurrent request.

Solution – Align TLS Versions and Expand the HTTP Connection Pool

1. Pin the client to TLSv1.3 or force TLSv1.2 consistently

Update the container image to use OpenSSL 1.1.1 ≥ 1.1.1l (which enables TLSv1.3) and upgrade requests to 2.31. Alternatively, configure the weaviate-client to use a custom requests.Session with an explicit TLS version.

Before (failing configuration)

# requirements.txt
requests==2.28
weaviate-client==3.5.2

After (TLS‑compatible configuration)

# requirements.txt
requests==2.31
weaviate-client==4.0.0

2. Increase the urllib3 connection pool size and set a higher timeout

Pass a custom urllib3.PoolManager to the client.

Before (default client)

import weaviate

client = weaviate.Client(
    url="https://weaviate.internal",
    auth_client_secret=weaviate.AuthApiKey(api_key=os.getenv("WEAVIATE_API_KEY"))
)

After (expanded pool, explicit timeout)

import weaviate
import urllib3
import ssl

# Create a TLS‑enabled SSLContext that supports TLSv1.3
ssl_ctx = ssl.create_default_context()
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2  # allow 1.2‑1.3

pool_manager = urllib3.PoolManager(
    num_pools=50,               # allow many hosts (we only have one)
    maxsize=100,                # up to 100 concurrent connections per host
    timeout=urllib3.Timeout(connect=60, read=120),
    ssl_context=ssl_ctx,
)

client = weaviate.Client(
    url="https://weaviate.internal",
    auth_client_secret=weaviate.AuthApiKey(api_key=os.getenv("WEAVIATE_API_KEY")),
    additional_headers={"User-Agent": "rag-pipeline/1.0"},
    # inject the custom pool manager
    _pool_manager=pool_manager
)

3. Refactor the pipeline to reuse a single `WeaviateClient` instance

Instantiate the client once at pipeline start and reuse it across all batches. This eliminates the overhead of repeatedly creating connection pools.

Before (per‑batch client)

for batch in batches:
    client = weaviate.Client(url=WEAVIATE_URL, auth_client_secret=auth)
    client.batch.add_data_object(...)
    client.batch.create_objects()

After (single shared client)

client = weaviate.Client(url=WEAVIATE_URL, auth_client_secret=auth)
for batch in batches:
    client.batch.add_data_object(...)
    client.batch.create_objects()

Verify – Confirm the Pipeline Completes Successfully

1. Health‑check the Weaviate endpoint

curl -s -k https://weaviate.internal/v1/.well-known/ready | jq .

Expected output:

{
  "status": "READY"
}

2. Run a single‑batch dry‑run

python ingest.py --dry-run --batch-size 500

Log excerpt after fix:


2026-09-24 02:45:12,001 [RAG-INGEST] INFO  Connected to Weaviate (TLSv1.3) – pool size 100
2026-09-24 02:45:12,015 [RAG-INGEST] INFO  Batch 1/60 indexed in 3.2s
2026-09-24 02:45:15,342 [RAG-INGEST] INFO  Batch 60/60 indexed in 2.9s
2026-09-24 02:45:18,001 [RAG-INGEST] INFO  Nightly ingestion completed in 5m 58s

3. Verify document count in Weaviate

curl -s -k https://weaviate.internal/v1/objects?class=Document | jq '.objects | length'

The count should increase by the expected number of new documents (e.g., +30000).

4. Spot‑check a downstream RAG query

python query_rag.py --question "What new policy was added last night?"

Response should contain references to the newly indexed documents.

Prevent – Operational Guardrails for Future Deployments

Version compatibility matrix: Document supported weaviate-client versions for each Weaviate release, especially when TLS defaults change.
Automated TLS validation: Add a CI step that runs openssl s_client against the staging Weaviate endpoint to ensure the client can negotiate the server’s preferred protocol.
Connection‑pool metrics: Export urllib3_pool_connections and urllib3_pool_maxsize to Prometheus; alert if pool_connections approaches maxsize for >5 minutes.
Batch size tuning: Keep batch size ≤ 1 000 and reuse a single client to avoid excessive concurrent connections.
Secret rotation monitoring: Verify that API‑key authentication failures are logged separately from TLS errors; set an alert on 401 Unauthorized spikes.

FAQ – Common Follow‑up Questions

Why did the pipeline work before the Weaviate upgrade?
The previous Weaviate version defaulted to TLSv1.2, which matched the client’s OpenSSL capabilities. The upgrade switched the default to TLSv1.3, exposing the incompatibility.
Can I keep using TLSv1.2 without upgrading the client?
Yes, by configuring the client’s SSLContext to allow TLSv1.2 only (as shown in the solution). However, it’s recommended to upgrade to TLSv1.3 for better security and performance.
Is increasing the connection pool size always safe?
It is safe as long as the underlying Weaviate instance can handle the concurrent load. Monitor weaviate_http_requests_active and adjust the pool size based on observed CPU/memory usage.
How do I differentiate between TLS handshake failures and authentication errors in logs?
TLS failures surface as SSLError or SSL: TLSV1_ALERT_PROTOCOL_VERSION. Authentication errors appear as 401 Unauthorized from the Weaviate HTTP API. Filtering by exception type or HTTP status code helps isolate the root cause.
What is the recommended way to test TLS compatibility in a CI pipeline?
Use a lightweight script that runs openssl s_client -connect $HOST:$PORT -tls1_3 and checks for a successful handshake (exit code 0). Fail the build if the handshake does not succeed.

Related Topic Hub: Vector Databases Troubleshooting Hub