photo of computer cables

Grafana RAG document loader OOM crash on edge node during embedding

by

Grafana RAG Document Loader OOM Crash on Edge Node During Embedding

Problem

Symptoms and Impact

On edge computing nodes (2 GiB RAM, 2 vCPU) running Grafana 9.x with the RAG Document Loader plugin, the embedding generation step intermittently aborts with an out‑of‑memory (OOM) kill. The failure manifests as:

  • Grafana UI shows a red error banner: Failed to generate embeddings: process killed (OOM)
  • System logs contain Out of memory: Kill process 12345 (node) score 987 or sacrifice child
  • Subsequent dashboard panels that depend on the vector store return empty results.
  • Node becomes unresponsive for ~30 seconds while the OOM killer reclaims memory.

Because the edge node also hosts a local LLM (e.g., llama.cpp) and a lightweight vector database (e.g., Qdrant), the memory pressure is compounded during concurrent embedding jobs.

Root Cause

The RAG Document Loader spawns a child process that loads the entire source document into memory, tokenizes it, and then calls the LLM to produce embeddings for each chunk. The plugin uses the default chunk_size=512 tokens and overlap=64 without any streaming or back‑pressure. On edge hardware the following chain leads to OOM:

  1. Document size: A single PDF of ~30 MiB expands to ~12 MiB of raw text after OCR.
  2. Chunking strategy: With chunk_size=512 tokens (~3 KiB per chunk) the loader creates ~4 000 chunks.
  3. Embedding generation: The plugin batches batch_size=128 and holds all intermediate float32 vectors (128 × 768 ≈ 400 KiB) in a slice before flushing to the vector DB.
  4. Memory allocation pattern: Go’s garbage collector does not reclaim the large []byte buffers until the entire embedding routine returns, causing a peak memory usage of >2 GiB.
  5. Concurrent workloads: Simultaneous dashboard refreshes trigger multiple loader instances, multiplying the peak.

In short, the plugin assumes abundant RAM and does not stream chunks or limit concurrency, which is unsuitable for constrained edge nodes.

Debug

Log Inspection

2024-10-12T14:23:07.321Z WARN  plugin.rag.loader  OOM detected while processing document id=42
2024-10-12T14:23:07.322Z INFO  system          Memory usage: total=2048MiB used=1976MiB free=72MiB
2024-10-12T14:23:07.323Z DEBUG plugin.rag.loader  Chunk count=3987 batch=128
2024-10-12T14:23:07.324Z ERROR kernel          Out of memory: Kill process 11234 (node) score 987 or sacrifice child

Runtime Metrics

Metric Before Crash After Crash
RSS (Grafana process) 1.8 GiB 2.1 GiB (peak)
RSS (LLM process) 0.6 GiB 0.8 GiB
Free system memory 120 MiB <5 MiB

Reproducing the Failure

Running the loader manually on the node reproduces the OOM:

docker run --rm -v $(pwd)/docs:/data \
  grafana/rag-loader:latest \
  --document /data/large-report.pdf \
  --chunk-size 512 \
  --batch-size 128

During execution, top shows the process memory climbing to the node’s limit before the kernel kills it.

Configuration Validation

The default configuration (shown in plugin.yaml) is:

loader:
  chunk_size: 512
  overlap: 64
  batch_size: 128
  max_concurrent_jobs: 4

All values are hard‑coded; there is no fallback for low‑memory environments.

Solution

Implementation Changes

Two complementary changes resolve the OOM while preserving functionality:

  1. Introduce streaming chunk processing – generate and persist embeddings per chunk instead of batching large slices.
  2. Limit concurrency based on available memory – compute a safe max_concurrent_jobs at startup.

Before (batch‑centric code)

// plugin/loader.go (simplified)
func (l *Loader) embedDocument(doc []byte) error {
    chunks := chunkDocument(doc, l.cfg.ChunkSize, l.cfg.Overlap)
    var vectors [][]float32
    for i := 0; i < len(chunks); i += l.cfg.BatchSize {
        batch := chunks[i:min(i+l.cfg.BatchSize, len(chunks))]
        emb, err := l.llm.Embed(batch) // returns [][][]float32
        if err != nil { return err }
        vectors = append(vectors, flatten(emb)...)
    }
    return l.db.Store(vectors)
}

After (streaming + back‑pressure)

// plugin/loader.go (refactored)
func (l *Loader) embedDocument(doc []byte) error {
    chunks := chunkDocument(doc, l.cfg.ChunkSize, l.cfg.Overlap)
    sem := make(chan struct{}, l.cfg.MaxConcurrentJobs) // memory‑aware semaphore
    for _, ch := range chunks {
        sem <- struct{}{}
        go func(c []byte) {
            defer func(){ <-sem }()
            vec, err := l.llm.Embed([]byte{c}) // single‑chunk embed
            if err != nil { log.Error(err); return }
            if err := l.db.Store(vec); err != nil { log.Error(err) }
        }(ch)
    }
    // wait for all goroutines
    for i := 0; i < cap(sem); i++ { <-sem }
    return nil
}

Key differences:

  • Embedding is performed per chunk, eliminating the large vectors slice.
  • A semaphore caps concurrent goroutines; MaxConcurrentJobs is derived from runtime.MemStats at plugin init.

Configuration Adjustments

Deploy a tailored plugin.yaml for edge nodes:

# plugin.yaml (edge profile)
loader:
  chunk_size: 256          # smaller chunks reduce per‑chunk memory
  overlap: 32
  batch_size: 1            # effectively disabled batching
  max_concurrent_jobs: 2   # safe limit for 2 GiB nodes

Deployment Steps

  1. Build the patched plugin: 
    git clone https://github.com/grafana/rag-loader.git
cd rag-loader
git checkout edge-fix
go build -o grafana-rag-loader .
docker build -t grafana/rag-loader:edge .
  2. Replace the existing plugin in Grafana’s plugin directory (e.g., /var/lib/grafana/plugins/rag-loader).
  3. Restart Grafana: 
    systemctl restart grafana-server
  4. Update the dashboard’s data source to reference the new loader configuration.

Verify

Functional Test

Run the loader against a known 30 MiB PDF and monitor memory:

docker run --rm -v $(pwd)/docs:/data grafana/rag-loader:edge \
  --document /data/large-report.pdf \
  --config /etc/grafana/plugin.yaml

Expected output:

2024-10-12T15:01:23.001Z INFO  loader  Starting embedding, chunks=3987
2024-10-12T15:01:23.102Z INFO  loader  Processed chunk 1/3987
...
2024-10-12T15:02:45.210Z INFO  loader  Completed, stored 3987 vectors

System memory should stay below 1 GiB throughout.

Grafana Dashboard Validation

  • Open the RAG panel, trigger a query, and confirm that results appear within the usual latency (< 2 s).
  • Check Grafana logs for INFO loader Completed without any OOM warnings.
  • Verify that qdrant now contains the expected number of vectors (e.g., SELECT COUNT(*) FROM vectors; returns 3987).

Prevent

Monitoring & Alerting

Metric Threshold Alert
Grafana process RSS >1.5 GiB Warning: High memory usage
OOM kill events (kernel) ≥1 per hour Critical: OOM on edge node
Embedding job duration >180 s Info: Potential back‑pressure

Configuration Guardrails

  • Enforce a minimum max_concurrent_jobs of 1 and a maximum of totalRAM/1GiB via a startup script.
  • Provide a separate plugin.yaml.edge that must be explicitly selected for nodes with memory < 4GiB.
  • Add a health‑check endpoint (/api/plugins/rag-loader/health) that returns memory_ok: false if runtime.MemStats.Alloc exceeds 80 % of the limit.

Operational Practices

  • Schedule large document ingestion during off‑peak windows to avoid concurrent dashboard refreshes.
  • Cache frequently used embeddings in a shared vector store to reduce repeated generation.
  • Keep the local LLM model size under 500 MiB on edge nodes; larger models should be offloaded to a central inference service.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  1. Why does the OOM only appear with PDFs larger than 20 MiB?
    Because the default chunking creates thousands of chunks, and the batch buffer holds all intermediate vectors in memory. Smaller files produce fewer chunks, staying under the RAM ceiling.
  2. Can I keep the original batch‑size configuration and still avoid OOM?
    Only if you also reduce max_concurrent_jobs to 1 and increase swap space, but the recommended approach is to stream embeddings per chunk, which eliminates the need for large batches.
  3. Is there a way to offload embedding generation to a remote service?
    Yes. The plugin supports an external embedding endpoint; configure embed_service_url in plugin.yaml and set batch_size to a modest value. This moves the memory burden off the edge node.
  4. What kernel parameters help detect OOM earlier?
    Enable vm.overcommit_memory=2 and set vm.panic_on_oom=1 on the edge node. This forces the kernel to reject allocations that would exceed the available RAM, producing clearer logs.
  5. How do I verify that the new streaming implementation is actually used?
    Enable debug logging (log_level: debug) and look for the line Embedding chunk X/Y without any “batch” messages. Additionally, monitor the process RSS; it should stay flat instead of spiking.