RAG Chunk Overlap Duplicate Embeddings After Increasing Chunk Size On GCP Compute

In this article

Table of Contents

Problem – Overlapping Chunk Configuration Produces Duplicate Embeddings in a RAG Pipeline on GCP Compute

In a production Retrieval‑Augmented Generation (RAG) service running on Google Cloud Compute Engine VMs, engineers observed that increasing the chunk_size parameter caused a sudden rise in duplicate vector embeddings. The vector store returned multiple identical entries for the same passage, degrading retrieval relevance and inflating storage costs.

Symptoms and Impact

Log entries show a sharp increase in Embedding created for chunk messages with identical chunk_id values.
Search results contain the same sentence three‑to‑five times, each with a different doc_id but identical embedding vectors.
Memory usage of the vector store (e.g., FAISS or Qdrant) grew by ~30 % after the change.
Latency of the retrieve() call increased from ~120 ms to >300 ms.

2024-10-26 14:02:13.421 INFO  rag.pipeline - Embedding created for chunk_id=doc123_0 offset=0 length=1024
2024-10-26 14:02:13.426 INFO  rag.pipeline - Embedding created for chunk_id=doc123_0 offset=512 length=1024
2024-10-26 14:02:13.432 INFO  rag.pipeline - Embedding created for chunk_id=doc123_0 offset=1024 length=1024

Root Cause – Misinterpreted Overlap Parameter After Chunk Size Change

The RAG implementation uses a sliding‑window algorithm:

for start in range(0, len(text), chunk_size - overlap):
    end = start + chunk_size
    chunk = text[start:end]

When chunk_size was increased from 512 to 1024, the overlap value remained at 512. The subtraction chunk_size - overlap therefore became 512, which is exactly the previous chunk_size. The loop stepped forward by the old chunk length, causing each new chunk to start halfway through the previous one. Because the overlap was also 512, the two halves overlapped completely, resulting in identical content being emitted twice for each window.

In short, the overlap was larger than the effective stride, producing duplicate chunks that were subsequently embedded.

Investigation and Debugging Steps

Verify configuration values on the VM.

# cat /opt/rag/config.yaml
chunk_size: 1024
overlap: 512
vector_store: qdrant

Inspect the chunking function.

def chunk_text(text, chunk_size, overlap):
    stride = chunk_size - overlap
    if stride <= 0:
        raise ValueError("Stride must be positive")
    chunks = []
    for start in range(0, len(text), stride):
        end = start + chunk_size
        chunks.append(text[start:end])
    return chunks

Run a dry‑run on a sample document.

$ python - <<'PY'
from rag.chunker import chunk_text
text = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" * 40  # 1040 chars
chunks = chunk_text(text, 1024, 512)
print(len(chunks))
print(chunks[0][:60])
print(chunks[1][:60])
PY
2
ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJ...
ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJ...

The two chunks are identical because stride = 512 while chunk_size = 1024, causing the second chunk to start at offset 512, which is exactly the midpoint of the first chunk, and the overlap of 512 repeats the first half.

Check vector store for duplicate vectors.

# qdrant-cli collection list
collections:
  - name: rag_docs
    vectors_count: 12500   # grew after change
# qdrant-cli points search rag_docs --vector  --limit 5
...

Solution – Align Overlap with New Chunk Size or Use Fractional Overlap

The fix consists of either:

Reducing overlap so that stride = chunk_size - overlap remains positive and less than chunk_size, or
Switching to a percentage‑based overlap that scales with the chunk size.

Option A – Fixed Overlap Adjustment

# Before (faulty)
chunk_size: 1024
overlap: 512   # leads to stride 512 → duplicate half‑chunks

# After (correct)
chunk_size: 1024
overlap: 256   # stride = 768, no full duplication

Update config.yaml and restart the service.

# systemctl restart rag-service
# journalctl -u rag-service -f | grep "Embedding created"
2024-10-26 15:10:02.112 INFO  rag.pipeline - Embedding created for chunk_id=doc123_0 offset=0 length=1024
2024-10-26 15:10:02.118 INFO  rag.pipeline - Embedding created for chunk_id=doc123_0 offset=768 length=1024

Option B – Percentage‑Based Overlap (recommended for future scaling)

Introduce a helper that computes overlap as a fraction of chunk_size:

def compute_overlap(chunk_size, ratio=0.25):
    return int(chunk_size * ratio)

# Example usage
chunk_size = 1024
overlap = compute_overlap(chunk_size, 0.25)  # 256
chunks = chunk_text(text, chunk_size, overlap)

Update the configuration schema to accept either an absolute value or a ratio:

# config.yaml
chunk_size: 1024
overlap_ratio: 0.25   # new field, overrides absolute overlap if present

Verification – Confirm No Duplicate Embeddings Remain

Re‑run the dry‑run script and compare chunk hashes.

$ python - <<'PY'
import hashlib, json
from rag.chunker import chunk_text
text = "A"*3000
chunks = chunk_text(text, 1024, 256)
hashes = [hashlib.sha256(c.encode()).hexdigest() for c in chunks]
print(json.dumps(hashes, indent=2))
PY
[
  "e3b0c44298fc1c149afbf4c8996fb924...",
  "5d41402abc4b2a76b9719d911017c592...",
  "7d793037a0760186574b0282f2f435e7..."
]

All hashes are unique.

Query the vector store for duplicate vectors.

# qdrant-cli points list rag_docs --limit 1000 --filter "vector_similarity < 1e-12"
# No results → no near‑identical vectors

Run an end‑to‑end retrieval test.

$ curl -s -X POST http://localhost:8080/retrieve \
  -d '{"query":"What is the policy for data retention?"}' \
  -H "Content-Type: application/json"
{
  "answers": [
    "Data must be retained for at least 7 years according to compliance policy."
  ],
  "source_chunks": [
    {"doc_id":"policy_001","chunk_offset":0}
  ]
}

The response contains a single source chunk per answer, confirming that duplication is gone.

Prevention – Guardrails and Monitoring

Configuration validation. Add a start‑up check that raises an error if overlap >= chunk_size or stride <= 0.

# validate_config.py
if cfg.overlap >= cfg.chunk_size:
    raise ValueError("overlap must be smaller than chunk_size")
if cfg.chunk_size - cfg.overlap <= 0:
    raise ValueError("effective stride must be positive")

Metric alert. Emit a Prometheus gauge rag_duplicate_chunks_total that increments each time a chunk hash collides with a previously stored hash. Alert on a sudden spike (>5 % of total chunks).
Automated tests. Include a unit test that generates a synthetic document, chunks it with the current config, and asserts that len(set(hashes)) == len(hashes).
Documentation. Record the recommended overlap ratio (25 %–30 %) and note that any change to chunk_size must be accompanied by a review of overlap.

FAQ – Common Follow‑Up Questions

Why did the problem appear only after increasing chunk_size?
Because the overlap value remained static. The stride became smaller than the overlap, causing each new window to repeat the previous half of the text.
Can I set overlap to zero?
Yes, but you lose the contextual continuity that helps the LLM retrieve relevant passages across chunk boundaries. A small overlap (10‑30 % of chunk_size) is usually optimal.
How do I detect duplicate embeddings in a large collection without scanning every vector?
Use the vector store’s built‑in deduplication API (e.g., qdrant-cli points search --filter "vector_similarity < 1e-12") or maintain a hash index of the raw chunk text at ingest time.
Is there a performance penalty for using a percentage‑based overlap?
No. The computation is O(1) and the resulting number of chunks is deterministic. It actually prevents the hidden cost of duplicate embeddings.
My pipeline runs inside an air‑gapped network; can I still use external validation tools?
All diagnostics (log inspection, hash comparison, vector‑store queries) are performed locally. Ensure that any third‑party validation scripts are bundled with the deployment artifact.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub