Why the fix works Reducing batch_size ensures each HTTP request stays under the server’s payload ceiling, preventing 413 responses. Increasing max_import_size raises the ceiling, allowing larger batches without fragmenting the payload. Both approaches eliminate the “connection reset” cascade because the client no longer receives a hard rejection that leaves the HTTP connection in an undefined state. Verification 1. Successful batch import logs

2024-08-01 10:15:02,112 - weaviate.batch - INFO - Batch of 50 objects imported successfully (payload size: 48.7 MB) 2024-08-01 10:15:02,115 - weaviate.batch - INFO - Total objects ingested: 1,000,000 2. Server metrics MetricBeforeAfter HTTP 413 count1240 Batch import latency (ms)≈ 850≈ 420 Objects/sec≈ 1,200≈ 2,300 3. Functional test def test_ingest(): client = weaviate.Client("http://localhost:8080") client.batch.configure(batch_size=50) # ingest a known set of 200 docs client.batch.add_data_object({...}) # repeat 200 times client.batch.create_objects() assert client.query.get("Document").with_additional(['id']).do()['data']['Get']['Document'] | length == 200 Prevention and Best Practices Dynamic batch sizing: Enable dynamic=True in the Python client so oversized batches are automatically split. Monitor payload size: Export a custom metric (e.g., weaviate_batch_payload_bytes) and set an alert when it approaches 80 % of max_import_size. Version‑aware limits: Verify limits after each Weaviate upgrade; defaults may change (see the configuration reference). Graceful retry logic: On HTTP 413, catch the exception, reduce batch_size by half, and retry the failed batch. Infrastructure sizing: For very large vectors (>1 MB), consider increasing RAM or using a dedicated ingestion node to avoid memory pressure that can also trigger batch rejections. Related Topic Hub: Vector Databases Troubleshooting Hub

Weaviate Client Batch Size Exceeded During Document Ingestion

Q: Root Cause Analysis Server‑side batch limits The Weaviate configuration (weaviate.conf) defines max_import_size (default 100 MB) and batch_size (default 128 objects). Exceeding either triggers a 413 response. Embedding vectors generated for the PDF documents were ~1 MB each. A default client batch of 100 objects therefore produced a payload of ~100 MB, close to the default max_import_size. In practice, occasional larger vectors pushed the batch over the limit. After the first 413, the Python client kept the underlying HTTP connection open. Subsequent batches were queued but never transmitted, leading to “connection reset by peer” errors and apparent data loss. These observations align with the official documentation on batch ingestion limits (Weaviate Batch Ingestion documentation – explains batch size limits, configuration of max_import_size, and handling of HTTP 413 responses) and the community discussion in GitHub Issue #1245, where users reported the same “batch size exceeds limit” symptom when client defaults outpaced server limits. Debugging and Investigation Steps 1. Capture client‑side logs

2024-07-21 10:12:34,567 - weaviate.batch - ERROR - Batch import failed: HTTPError: 413 Payload Too Large Batch size: 100 objects, payload size: 102.4 MB 2024-07-21 10:12:35,001 - weaviate.batch - INFO - Retrying batch... 2024-07-21 10:12:36,112 - weaviate.batch - ERROR - Connection reset by peer after failed batch 2. Inspect server logs 2024-07-21 10:12:34.560 | weaviate | ERROR | batch_import.go:215 | payload too large: 102400000 bytes > max_import_size 100000000 2024-07-21 10:12:34.561 | weaviate | INFO | batch_import.go:230 | rejecting batch of 100 objects 3. Verify current configuration $ cat /etc/weaviate/weaviate.conf | grep -E 'max_import_size|batch_size' max_import_size = 100MB batch_size = 128 4. Measure actual payload size import json, sys batch = [...] # list of 100 objects payload = json.dumps(batch).encode('utf-8') print(len(payload) / (1024*1024), "MB") # Output: 102.4 MB 5. Reproduce with a minimal batch from weaviate import Client client = Client("http://localhost:8080") client.batch.configure(batch_size=10) # smaller batch client.batch.add_data_object({...}) # add a few objects client.batch.create_objects() Solution: Align client batch size with server limits Option A – Reduce client batch size Adjust the Python client configuration to a safe size (e.g., 50 objects) that keeps payload < 80 MB, providing a margin below the default max_import_size.

Q: Option B – Increase server max_import_size

If larger batches are required for throughput, raise the server limit in weaviate.conf and restart the service.

Problem: Weaviate client batch size exceeded during document ingestion

During a large‑scale ingestion run on a self‑hosted Weaviate instance (32 GB RAM, Docker Compose), the Python client started failing after a few hundred documents. The failure manifested as HTTP 413 “Payload Too Large” responses and a subsequent cascade of aborted GraphQL mutations, ultimately causing data loss.

Typical error messages observed in the logs:


WeaviateBatchException: batch size exceeds limit of 128 objects
HTTPError: 413 Payload Too Large – request entity too large
GraphQL mutation aborted: request entity too large
WeaviateError: Batch import failed – payload size exceeds max_import_size
Connection reset by peer after failed batch – subsequent batches not sent

Root Cause Analysis

Server‑side batch limits

The Weaviate configuration (weaviate.conf) defines max_import_size (default 100 MB) and batch_size (default 128 objects). Exceeding either triggers a 413 response.
Embedding vectors generated for the PDF documents were ~1 MB each. A default client batch of 100 objects therefore produced a payload of ~100 MB, close to the default max_import_size. In practice, occasional larger vectors pushed the batch over the limit.
After the first 413, the Python client kept the underlying HTTP connection open. Subsequent batches were queued but never transmitted, leading to “connection reset by peer” errors and apparent data loss.

These observations align with the official documentation on batch ingestion limits (Weaviate Batch Ingestion documentation – explains batch size limits, configuration of max_import_size, and handling of HTTP 413 responses) and the community discussion in GitHub Issue #1245, where users reported the same “batch size exceeds limit” symptom when client defaults outpaced server limits.

Debugging and Investigation Steps

1. Capture client‑side logs


2024-07-21 10:12:34,567 - weaviate.batch - ERROR - Batch import failed: HTTPError: 413 Payload Too Large
Batch size: 100 objects, payload size: 102.4 MB
2024-07-21 10:12:35,001 - weaviate.batch - INFO - Retrying batch...
2024-07-21 10:12:36,112 - weaviate.batch - ERROR - Connection reset by peer after failed batch

2. Inspect server logs


2024-07-21 10:12:34.560 | weaviate | ERROR | batch_import.go:215 | payload too large: 102400000 bytes > max_import_size 100000000
2024-07-21 10:12:34.561 | weaviate | INFO  | batch_import.go:230 | rejecting batch of 100 objects

3. Verify current configuration


$ cat /etc/weaviate/weaviate.conf | grep -E 'max_import_size|batch_size'
max_import_size = 100MB
batch_size = 128

4. Measure actual payload size


import json, sys
batch = [...]  # list of 100 objects
payload = json.dumps(batch).encode('utf-8')
print(len(payload) / (1024*1024), "MB")
# Output: 102.4 MB

5. Reproduce with a minimal batch


from weaviate import Client
client = Client("http://localhost:8080")
client.batch.configure(batch_size=10)  # smaller batch
client.batch.add_data_object({...})  # add a few objects
client.batch.create_objects()

Solution: Align client batch size with server limits

Option A – Reduce client batch size

Adjust the Python client configuration to a safe size (e.g., 50 objects) that keeps payload < 80 MB, providing a margin below the default max_import_size.


# Before (default)
client.batch.configure(
    batch_size=100,          # default
    timeout_retries=3,
)

# After – reduced batch size
client.batch.configure(
    batch_size=50,           # new safe size
    timeout_retries=5,
    dynamic=True,           # let client split oversized batches
)

Option B – Increase server `max_import_size`

If larger batches are required for throughput, raise the server limit in weaviate.conf and restart the service.


# weaviate.conf before
max_import_size = 100MB

# weaviate.conf after
max_import_size = 200MB

After changing the config, verify the service restarts cleanly:


$ docker-compose restart weaviate
$ docker logs weaviate | grep max_import_size
2024-08-01 09:00:01 INFO  max_import_size set to 200MB

Why the fix works

Reducing batch_size ensures each HTTP request stays under the server’s payload ceiling, preventing 413 responses.
Increasing max_import_size raises the ceiling, allowing larger batches without fragmenting the payload.
Both approaches eliminate the “connection reset” cascade because the client no longer receives a hard rejection that leaves the HTTP connection in an undefined state.

Verification

1. Successful batch import logs


2024-08-01 10:15:02,112 - weaviate.batch - INFO - Batch of 50 objects imported successfully (payload size: 48.7 MB)
2024-08-01 10:15:02,115 - weaviate.batch - INFO - Total objects ingested: 1,000,000

2. Server metrics

Metric	Before	After
HTTP 413 count	124	0
Batch import latency (ms)	≈ 850	≈ 420
Objects/sec	≈ 1,200	≈ 2,300

3. Functional test


def test_ingest():
    client = weaviate.Client("http://localhost:8080")
    client.batch.configure(batch_size=50)
    # ingest a known set of 200 docs
    client.batch.add_data_object({...})  # repeat 200 times
    client.batch.create_objects()
    assert client.query.get("Document").with_additional(['id']).do()['data']['Get']['Document'] | length == 200

Prevention and Best Practices

Dynamic batch sizing: Enable dynamic=True in the Python client so oversized batches are automatically split.
Monitor payload size: Export a custom metric (e.g., weaviate_batch_payload_bytes) and set an alert when it approaches 80 % of max_import_size.
Version‑aware limits: Verify limits after each Weaviate upgrade; defaults may change (see the configuration reference).
Graceful retry logic: On HTTP 413, catch the exception, reduce batch_size by half, and retry the failed batch.
Infrastructure sizing: For very large vectors (>1 MB), consider increasing RAM or using a dedicated ingestion node to avoid memory pressure that can also trigger batch rejections.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the first 413 error cause all subsequent batches to fail?
The client keeps the underlying HTTP connection open after a hard 413 response. The server marks the connection as unusable, so further requests are dropped, resulting in “connection reset by peer”. Resetting the client or enabling dynamic batch splitting resolves this.
Can I keep the default client batch size and only change server settings?
Yes, increasing max_import_size in weaviate.conf (and restarting) will allow larger payloads, but be mindful of available RAM and network bandwidth.
What is the recommended maximum batch size for 1 MB vectors?
Empirically, a batch of 50 objects (~50 MB) stays comfortably below the default 100 MB limit, leaving headroom for metadata overhead.
How do I detect a payload‑size issue before it aborts the whole job?
Add a pre‑flight check that serializes the batch and compares its size to max_import_size. Log a warning and split the batch proactively.
Is there a way to let Weaviate automatically adjust its limits based on available resources?
Not currently. Limits are static and must be tuned manually via the configuration file or environment variables.