Prometheus RAG Retrieval Empty Results In CI/CD Pipeline

Q: Secondary contributing factors Namespace / label mismatch: In isolated CI namespaces the scrape config may not apply the job="rag" label, as highlighted in the Stack Overflow question “Prometheus RAG query returns no data in CI/CD but works locally”. This leads to PromQL queries that filter on job="rag" returning no series. Remote storage connectivity: Errors such as “prometheus: error reading remote storage: EOF” indicate that a remote‑write endpoint is unreachable, which can also surface as empty results when the remote backend is expected to hold the KB index. Investigation and Debugging Steps 1. Verify PVC binding and content

kubectl get pvc -n ci-test kubectl describe pvc rag-kb-pvc -n ci-test Expected output shows Phase: Bound. If the PVC is Pending, the test pod will start without the KB.

Q: 2. Inspect init container logs

kubectl logs prometheus-0 -c init-copy-kb -n ci-test Look for the error message:

Q: 3. Check Prometheus startup logs for KB loading

kubectl logs prometheus-0 -n ci-test | grep "RAG index" Successful load prints something like:

Q: 4. Validate that the expected labels exist on scraped series

kubectl exec -it prometheus-0 -n ci-test -- curl -s http://localhost:9090/api/v1/label/job/values If rag is missing, the query will never match.

Q: 5. Perform a manual remote‑read query

curl -s -X POST http://prometheus-ci:9090/api/v1/query \ -d 'query=rag_query{job="rag"}' Typical empty response:

Q: 6. Examine Helm release diff between working and failing runs

helm get manifest my-prometheus -n ci-test > manifest.yaml diff manifest.yaml manifest-working.yaml Resolution Fix 1 – Ensure PVC readiness before test execution Introduce an explicit initContainer in the test pod that waits for the PVC to be bound and for the KB files to appear.

Q: Fix 2 – Preserve the KB across StatefulSet recreations

Configure the Prometheus StatefulSet to use a volumeClaimTemplate that does not get deleted on helm upgrade, and set retain reclaim policy on the underlying PV.

Q: Fix 3 – Align scrape labels with test queries

Update the Helm values for the kubernetes_sd_config to inject the job="rag" label on all targets used in CI.

In this article

Table of Contents

Problem Description

During automated integration tests executed in a CI/CD pipeline, a Prometheus instance that is configured for Retrieval‑Augmented Generation (RAG) consistently returns empty result vectors. The same pipeline runs successfully on a developer’s workstation, but in the pipeline the query endpoint responds with messages such as:


retrieval returned empty vector
no data found for query

These symptoms appear despite:

Validated training data being pre‑loaded into the knowledge‑base (KB) directory.
Correct PromQL queries that match the expected job="rag" label set when run locally.
PersistentVolumeClaims (PVCs) defined in the Helm chart for the KB storage.

Root Cause Analysis

Interaction between Prometheus storage and CI orchestration

Prometheus stores the RAG index on a mounted persistent volume (/data/kb) as described in the official Storage Configuration documentation. In a CI environment the following sequence is typical:

Pipeline creates a temporary namespace.
An initContainer copies the knowledge‑base files into the PVC.
Prometheus StatefulSet starts, mounting the same PVC at /data/kb.
Integration tests issue HTTP POST requests to the RAG remote‑read endpoint.

Two independent evidence sources converge on the root cause:

GitHub issue #342 reports that the PVC is sometimes not bound before the test pod starts, leading to “failed to load knowledge base from /data/kb: file not found”.
GitHub issue #12456 shows that when the StatefulSet is recreated (e.g., after a Helm upgrade), the local storage directory is cleared, removing the RAG index.

Consequently, the Prometheus process starts without a populated KB, and any remote‑read query matches no series, producing the empty vector error.

Secondary contributing factors

Namespace / label mismatch: In isolated CI namespaces the scrape config may not apply the job="rag" label, as highlighted in the Stack Overflow question “Prometheus RAG query returns no data in CI/CD but works locally”. This leads to PromQL queries that filter on job="rag" returning no series.
Remote storage connectivity: Errors such as “prometheus: error reading remote storage: EOF” indicate that a remote‑write endpoint is unreachable, which can also surface as empty results when the remote backend is expected to hold the KB index.

Investigation and Debugging Steps

1. Verify PVC binding and content

kubectl get pvc -n ci-test
kubectl describe pvc rag-kb-pvc -n ci-test

Expected output shows Phase: Bound. If the PVC is Pending, the test pod will start without the KB.

2. Inspect init container logs

kubectl logs prometheus-0 -c init-copy-kb -n ci-test

Look for the error message:

failed to load knowledge base from /data/kb: file not found

3. Check Prometheus startup logs for KB loading

kubectl logs prometheus-0 -n ci-test | grep "RAG index"

Successful load prints something like:

RAG index loaded, 1245 documents indexed

4. Validate that the expected labels exist on scraped series

kubectl exec -it prometheus-0 -n ci-test -- curl -s http://localhost:9090/api/v1/label/job/values

If rag is missing, the query will never match.

5. Perform a manual remote‑read query

curl -s -X POST http://prometheus-ci:9090/api/v1/query \
  -d 'query=rag_query{job="rag"}'

Typical empty response:

{
  "status":"success",
  "data":{"resultType":"vector","result":[]}
}

6. Examine Helm release diff between working and failing runs

helm get manifest my-prometheus -n ci-test > manifest.yaml
diff manifest.yaml manifest-working.yaml

Resolution

Fix 1 – Ensure PVC readiness before test execution

Introduce an explicit initContainer in the test pod that waits for the PVC to be bound and for the KB files to appear.

# Before (pipeline step omitted PVC wait)
apiVersion: v1
kind: Pod
metadata:
  name: rag-test
spec:
  containers:
  - name: test-runner
    image: myorg/rag-test:latest
    command: ["./run-tests.sh"]

# After (adds wait logic)
apiVersion: v1
kind: Pod
metadata:
  name: rag-test
spec:
  initContainers:
  - name: wait-for-pvc
    image: busybox
    command: ['sh', '-c', '
      while [ ! -d /data/kb ]; do echo "waiting for KB..."; sleep 2; done
    ']
    volumeMounts:
    - name: kb-volume
      mountPath: /data/kb
  containers:
  - name: test-runner
    image: myorg/rag-test:latest
    command: ["./run-tests.sh"]
    volumeMounts:
    - name: kb-volume
      mountPath: /data/kb
  volumes:
  - name: kb-volume
    persistentVolumeClaim:
      claimName: rag-kb-pvc

The init container blocks the test runner until the directory exists, guaranteeing that Prometheus can read the index.

Fix 2 – Preserve the KB across StatefulSet recreations

Configure the Prometheus StatefulSet to use a volumeClaimTemplate that does not get deleted on helm upgrade, and set retain reclaim policy on the underlying PV.

# Before (default delete policy)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rag-kb-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi

# After (retain policy)
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rag-kb-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    namespace: ci-test
    name: rag-kb-pvc

With Retain, the PV is not wiped when the StatefulSet is recreated, keeping the RAG index intact.

Fix 3 – Align scrape labels with test queries

Update the Helm values for the kubernetes_sd_config to inject the job="rag" label on all targets used in CI.

# values.yaml snippet
scrape_configs:
  - job_name: 'rag'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ['ci-test']
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: rag
        action: keep
      - target_label: job
        replacement: rag

Verification

Redeploy the updated Helm chart and wait for the PVC to reach Bound.
Run the test pod; the init container should report “waiting for KB…” only until the KB appears.

Execute a manual query again:

curl -s -X POST http://prometheus-ci:9090/api/v1/query \
  -d 'query=rag_query{job="rag"}'

Expected result contains at least one vector element:

{
  "status":"success",
  "data":{"resultType":"vector","result":[
    {"metric":{"job":"rag","instance":"..."},"value":[...,"42"]}
  ]}
}

Inspect Prometheus logs for the “RAG index loaded” message.
Confirm that subsequent CI stages can query the same endpoint without failures.

Prevention and Best Practices

Explicit PVC readiness checks: Use init containers or postStart hooks to verify that the KB directory exists before starting dependent services.
Reclaim policy: Set persistentVolumeReclaimPolicy: Retain for KB storage to survive StatefulSet rollouts.
Namespace‑consistent labeling: Ensure that all test pods inherit the same job label defined in the scrape config; otherwise queries will be filtered out.

Monitoring: Add an alert rule that fires when the RAG index load timestamp is older than a configurable threshold:

ALERT RAGIndexStale
  IF time() - prometheus_rag_index_last_successful_load_seconds > 300
  FOR 5m
  LABELS {severity="warning"}
  ANNOTATIONS {
    summary = "RAG index has not been loaded for 5 minutes",
    description = "Check PVC binding and init container logs."
  }

CI pipeline ordering: Place the knowledge‑base loading step in a separate job that produces an artifact (e.g., a ConfigMap) consumed by the Prometheus StatefulSet, guaranteeing that the data exists before the test job runs.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the RAG query work locally but return empty results in CI?
Because the CI namespace uses a fresh PVC that is not yet bound or populated when Prometheus starts, so the RAG index is missing.
What does “retrieval returned empty vector” mean?
Prometheus executed the remote‑read request successfully, but the underlying series set matched no data points, typically due to missing labels or an empty knowledge base.
How can I confirm that the knowledge‑base index is present inside the Prometheus pod?
Run kubectl exec -it prometheus-0 -n ci-test -- ls -l /data/kb and verify that the index files (e.g., index.db) exist.
Is the remote storage configuration relevant to this issue?
Only if the RAG index is stored in a remote backend. In most CI setups the index lives on a local PVC; remote storage errors like “error reading remote storage: EOF” would also produce empty results, but the primary cause here is local storage.
Can I reuse the same PVC across multiple pipeline runs?
Yes, by setting the PVC’s persistentVolumeReclaimPolicy to Retain and ensuring the pipeline does not delete the PVC between stages. Alternatively, create a dedicated “knowledge‑base” namespace that lives longer than individual test jobs.