RAG Pipeline Chunk Overlap Causing Duplicate Context Injection in Kubernetes

Problem

The Retrieval‑Augmented Generation (RAG) service deployed on a GPU‑accelerated Kubernetes cluster began exhibiting:

• Excessive token duplication in the vector store, leading to inflated embedding counts.
• Memory pressure on A100/H100 pods, resulting in OOMKilled restarts.
• Embedding latency spikes (>30 s) and a 40 % drop in retrieval accuracy.
• Increased vector_store_ingestion_latency metrics and frequent Embedding batch size exceeded memory limit errors.

Sample log excerpts from the affected pods:

2026-08-15T14:32:07Z ERROR tokenization: Token duplication detected: overlapping chunks produce identical vectors
2026-08-15T14:32:08Z WARN embedding_service: Failed to allocate memory for embedding (requested 2.4 GB, available 2.1 GB)
2026-08-15T14:32:09Z FATAL pod: OOMKilled - container terminated due to memory exhaustion
2026-08-15T14:32:12Z METRIC vector_store_ingestion_latency_seconds 45.8

These symptoms align with real incidents reported in production A100 GPU clusters where a chunk_overlap of 200 tokens on a 512‑token chunk size doubled memory usage per document.

Root Cause Analysis

The RAG service uses a sliding‑window chunker (LangChain/Haystack) configured via a ConfigMap. The chunk_overlap parameter was set to 0.5 × chunk_size (e.g., 200 tokens overlap for a 512‑token chunk). This configuration violates the recommended limit (overlap < chunk_size / 2) documented in:

• LangChain issue #4521 – duplicate token injection and embedding latency.
• Haystack issue #3894 – performance degradation when overlap exceeds half the chunk size.

Consequences of the misconfiguration:

1. Duplicate embeddings: Overlapping windows generate near‑identical token sequences, causing the embedding model to produce redundant vectors. Milvus ingestion logs (milvus #10287) show a proportional increase in insert volume.
2. Memory blow‑up: Each document’s effective token count becomes chunk_size + (num_chunks‑1) * overlap. With 200‑token overlap on 512‑token chunks, memory usage rises to 2‑3× per document, triggering the OOM errors observed.
3. Latency inflation: GPU memory saturation (>95 % utilization) forces the embedding service to serialize batches, leading to the >30 s ingestion latency spikes reported.

Debugging & Investigation

The following steps reproduced the issue and isolated the offending configuration:

1. Inspect the active ConfigMap

kubectl get configmap rag-chunk-config -n rag-namespace -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: rag-chunk-config
data:
  CHUNK_SIZE: "512"
  CHUNK_OVERLAP: "200"   # <-- problematic value
  EMBEDDING_BATCH_SIZE: "64"

2. Verify pod memory usage

kubectl top pod -l app=rag-embed -n rag-namespace

NAME                CPU(cores)   MEMORY(bytes)
rag-embed-0         2.1          12Gi
rag-embed-1         2.3          13Gi   # approaching limit

3. Correlate metrics with recent config change

curl -s http://prometheus.monitoring.svc:9090/api/v1/query?query=vector_store_ingestion_latency_seconds

{
  "status":"success",
  "data":{
    "resultType":"matrix",
    "result":[
      {"metric":{"pod":"rag-embed-0"},"values":[[1692201600,"45.8"],[1692201660,"46.2"]]},
      {"metric":{"pod":"rag-embed-1"},"values":[[1692201600,"44.9"],[1692201660,"45.5"]]}
    ]
  }
}

The latency rise coincided with the Helm chart upgrade that introduced a default overlap of 0.5×chunk_size (see the incident “Production A100 GPU cluster observed OOMKilled pods when chunk_overlap was set to 200 tokens”).

4. Capture a tokenization sample

python - <<'PY'
from langchain.text_splitter import RecursiveCharacterTextSplitter
text = "Lorem ipsum " * 1000  # ~12 000 tokens
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=200)
chunks = splitter.split_text(text)
print(f"Chunks: {len(chunks)}")
print(f"First chunk tokens: {len(chunks[0].split())}")
PY

Chunks: 25
First chunk tokens: 512

Inspecting two adjacent chunks reveals a 200‑token shared region, confirming the duplication.

Solution

Adjust the chunking parameters to keep overlap below chunk_size / 2 and re‑deploy the ConfigMap. The recommended values for a 512‑token window are an overlap of 50‑100 tokens.

Before (problematic)

apiVersion: v1
kind: ConfigMap
metadata:
  name: rag-chunk-config
data:
  CHUNK_SIZE: "512"
  CHUNK_OVERLAP: "200"   # 0.39× chunk size → exceeds safe limit

After (fixed)

apiVersion: v1
kind: ConfigMap
metadata:
  name: rag-chunk-config
data:
  CHUNK_SIZE: "512"
  CHUNK_OVERLAP: "80"    # 0.16× chunk size, well under chunk_size/2

Redeploy steps

1. Update the ConfigMap:

kubectl apply -f rag-chunk-config.yaml

2. Roll out a rolling restart to pick up the new config:

kubectl rollout restart deployment rag-embed -n rag-namespace

3. Optionally adjust the Horizontal Pod Autoscaler (HPA) target to reflect lower memory usage:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-embed-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rag-embed
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Verification

After applying the fix, perform the following checks:

1. Confirm reduced memory consumption

kubectl top pod -l app=rag-embed -n rag-namespace

NAME                CPU(cores)   MEMORY(bytes)
rag-embed-0         2.0          7Gi
rag-embed-1         2.1          7.5Gi

2. Validate ingestion latency

curl -s http://prometheus.monitoring.svc:9090/api/v1/query?query=vector_store_ingestion_latency_seconds

{
  "status":"success",
  "data":{
    "resultType":"matrix",
    "result":[
      {"metric":{"pod":"rag-embed-0"},"values":[[1692205200,"12.3"],[1692205260,"12.1"]]},
      {"metric":{"pod":"rag-embed-1"},"values":[[1692205200,"12.5"],[1692205260,"12.2"]]}
    ]
  }
}

3. Check for duplicate token warnings

kubectl logs -l app=rag-embed -n rag-namespace | grep "Token duplication"

(no output)

4. Run a retrieval test

python - <<'PY'
from rag_service import retrieve
response = retrieve("What is the impact of overlapping chunks?")
print(response)
PY

The retrieval returns a single, non‑redundant context passage, and the generated answer aligns with ground‑truth expectations.

Prevention & Best Practices

• Parameter Guardrails: Enforce CHUNK_OVERLAP < CHUNK_SIZE / 2 via Helm chart validation hooks or admission controllers.
• Resource Requests & Limits: Follow Kubernetes Resource Management guidelines; set memory requests to accommodate the worst‑case token expansion (e.g., request = base + (chunk_size * overlap_factor)).
• Monitoring: Alert on vector_store_ingestion_latency_seconds > 20 s and on OOMKilled events. Use custom Prometheus metrics for embedding_batch_memory_usage.
• Autoscaling: Leverage HPA with custom metrics (GPU memory, embedding latency) as described in the Kubernetes HPA documentation.
• Configuration Management: Store chunking settings in a ConfigMap and reference it in the pod spec; use kubectl diff before applying changes to catch unintended large overlaps.
• Testing: Include unit tests that generate synthetic documents and assert that the total token count after chunking does not exceed chunk_size * (1 + overlap_factor).

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does increasing CHUNK_OVERLAP degrade retrieval accuracy?
   Overlapping windows produce near‑identical embeddings, inflating the vector store with duplicate vectors. Retrieval algorithms then rank multiple identical vectors higher, reducing diversity and causing the model to attend to redundant context, which manifests as lower accuracy.
2. Can I set CHUNK_OVERLAP equal to CHUNK_SIZE for better context continuity?
   No. An overlap equal to the chunk size creates completely duplicated chunks, doubling memory usage and embedding time without adding new information. The effective token count becomes num_chunks × chunk_size, leading to OOM and latency spikes.
3. How do I detect duplicate embeddings in the vector store?
   Milvus emits a warning “Token duplication detected” when identical vectors are inserted consecutively. You can also query for high cosine similarity (>0.99) among vectors belonging to the same document to spot redundancy.
4. What metric should I monitor to catch overlap‑related performance regressions early?
   Track vector_store_ingestion_latency_seconds and a custom gauge embedding_duplicate_ratio (duplicate vectors / total vectors). A sudden rise above 0.1 indicates excessive overlap.
5. Is it safe to rely on GPU memory utilization alone for autoscaling?
   GPU memory is a leading indicator, but you should also consider embedding batch latency and the vector_store_ingestion_latency_seconds metric. Combining these signals prevents scaling decisions based on transient memory spikes caused by temporary overlap misconfigurations.