LlamaIndex filter expression parse error during Kubernetes autoscaling

Problem Description

When querying a LlamaIndex microservice deployed in a Kubernetes cluster, the service returns a FilterExpressionParseError for any query that contains complex boolean or range filters. Typical error messages observed in the pod logs are:

FilterExpressionParseError: Unexpected token 'AND' at position 12
ValueError: Could not parse filter expression: syntax error near '>=', expected ':'
JSONDecodeError: Expecting value: line 1 column 1 (char 0) while loading filter configuration

The failure prevents retrieval of indexed documents, causing downstream API endpoints to return HTTP 500 errors. The issue appears only after an autoscaling event (scale‑up or scale‑down) or after a pod restart.

Root Cause Analysis

LlamaIndex stores its vector index and accompanying metadata in a file‑based store (by default SQLite with a JSON schema file). The filter language is defined in the official documentation (LlamaIndex Query Language & Filters) and expects expressions of the form:

field1: "value" AND field2 >= 10

Two independent failure modes converge to produce the parse error:

1. Concurrent writes to the index files – Multiple pods mount the same PersistentVolumeClaim (PVC) and write to the SQLite database and the JSON schema simultaneously. The autoscaling controller may start a new pod while an existing pod is still committing a transaction. Without file‑locking, the SQLite file can become corrupted (GitHub #917) and the JSON schema can be left with half‑written fragments, leading to parsing failures.
2. Stale schema during scale‑up – A read‑only sidecar that snapshots the index may serve an outdated schema.json to a newly created pod. The new pod loads field definitions that do not match the current index, causing the filter parser to encounter unexpected tokens (Reddit discussion).

Both conditions violate the assumptions documented in the Autoscaling Best Practices: the index must be accessed in a mutually exclusive manner, and all pods must see a consistent schema version.

Investigation and Debugging

The following steps isolate the problem:

1. Inspect pod logs for lock errors:

kubectl logs -l app=llama-index -c main | grep "sqlite3.OperationalError"
# Example output:
sqlite3.OperationalError: database is locked

2. Check the integrity of the SQLite file on a running pod:

kubectl exec -it $(kubectl get pod -l app=llama-index -o name | head -n1) -- \
  sqlite3 /data/index.db "PRAGMA integrity_check;"
# Expected output: ok

If the output is anything other than ok, the database is corrupted.

3. Validate the JSON schema file for well‑formedness:

kubectl exec -it $(kubectl get pod -l app=llama-index -o name | head -n1) -- \
  cat /data/schema.json | python -m json.tool
# Expected: pretty‑printed JSON
# Error example:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

4. Capture a failing query and compare it to the documented syntax:

curl -X POST http://llama-index.default.svc.cluster.local/query \
  -H "Content-Type: application/json" \
  -d '{"filter": "category: \"news\" AND rating >= 4"}'
# Response:
{
  "error": "FilterExpressionParseError: Unexpected token 'AND' at position 12"
}

The position points to the token after the first field, indicating that the parser never saw a valid delimiter (colon) because the schema file was incomplete.

Resolution

The fix consists of three parts: enforce exclusive access, synchronize schema updates, and adjust the deployment to avoid simultaneous writes.

1. Switch to a write‑safe storage backend

Configure LlamaIndex to use a remote vector store (e.g., FAISS with Redis metadata) instead of the default SQLite file. This eliminates file‑level contention.

Before (default config)

index = GPTVectorStoreIndex.from_documents(
    documents,
    storage_context=StorageContext.from_defaults(persist_dir="/data")
)

After (Redis‑backed config)

from llama_index.storage import StorageContext
from llama_index.vector_stores import RedisVectorStore

vector_store = RedisVectorStore(host="redis", port=6379, db=0)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = GPTVectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context
)

2. Add file‑locking for SQLite (if staying with file‑based store)
Enable SQLite’s PRAGMA journal_mode=WAL and use a sidecar that provides a POSIX advisory lock.

# Init script executed on container start
sqlite3 /data/index.db "PRAGMA journal_mode=WAL;"

Deploy a initContainer that creates a lock file:

apiVersion: v1
kind: Pod
spec:
  initContainers:
  - name: lockfile
    image: busybox
    command: ["sh", "-c", "flock -n /data/index.lock true"]
    volumeMounts:
    - name: index-volume
      mountPath: /data

3. Enforce single‑writer scaling policy
Modify the HorizontalPodAutoscaler (HPA) to use the scaleTargetRef with a maxReplicas of 1 for the index‑writer deployment, while allowing additional read‑only replicas for query traffic.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llama-index-writer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llama-index-writer
  minReplicas: 1
  maxReplicas: 1
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Validation
After applying the changes, perform the following checks:

Confirm that the index files are no longer locked:


kubectl exec -it $(kubectl get pod -l app=llama-index-writer -o name | head -n1) -- \
  sqlite3 /data/index.db "PRAGMA integrity_check;"
# Output: ok


Run a complex filter query:


curl -X POST http://llama-index.default.svc.cluster.local/query \
  -H "Content-Type: application/json" \
  -d '{"filter": "category: \"news\" AND rating >= 4"}'
# Expected: 200 with matching documents


Check that autoscaling events no longer produce errors:


kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=llama-index-writer
# No recent events with "database is locked" or "FilterExpressionParseError"

Prevention and Best Practices

Use a stateful backend that supports concurrent writers (e.g., Redis, PostgreSQL) for production deployments.
Separate write and read workloads into distinct Deployments with independent scaling policies.
Enable WAL mode and advisory file locks if a file‑based store is unavoidable.
Version the schema file and mount it read‑only; update it via a rolling deployment to guarantee consistency.
Monitor for lock‑related metrics (sqlite3.OperationalError: database is locked) and filter‑parse errors; alert on spikes.
Run periodic integrity checks (e.g., a CronJob that runs PRAGMA integrity_check) and automatically roll back corrupted snapshots.

Related Topic Hub: RAG Systems Troubleshooting Hub
FAQ

Why does the filter parse error appear only after a scale‑up?

Because the new pod reads a partially written schema.json or SQLite file that was being updated by the original writer. The race condition corrupts the in‑memory parser state.
Can I keep using SQLite with multiple pods?

Only if you enforce strict single‑writer semantics and use WAL + file locking. Otherwise, switch to a concurrent‑write‑safe store.
What is the correct syntax for range filters?

According to the official docs, use a colon for equality and comparison operators with a colon separator, e.g., price >= 100 or date: "2023-01-01". Do not mix AND/OR without proper parentheses when combining ranges.
How can I verify which version of the schema a pod is using?

Inspect the file’s hash at startup:


sha256sum /data/schema.json

Compare it against the hash stored in a ConfigMap that is updated during deployments.
Is there a way to automatically recover from a corrupted index?

Implement a sidecar that detects Integrity_check != ok and restores the index from a recent snapshot stored in a separate PVC or object store.