GPT-3.5 RAG Sub‑query Generation Empty Results After Model Update

Q: 1. Capture the raw LLM response

2026-10-16T12:04:33.210Z INFO request_id=7f3c2a sub‑query generation start 2026-10-16T12:04:34.018Z INFO raw LLM response: { "sub_queries": [ "What were the total revenues for Q4 2022?", "How did operating expenses change compared to Q3 2022?" ] } 2026-10-16T12:04:34.020Z INFO parsing sub‑queries 2. Verify parser behavior # Python snippet used in the service def parse_subqueries(response: str) -> List[str]: # Old implementation return [line.strip() for line in response.splitlines() if line.strip()] # Test with captured response >>> parse_subqueries('{ "sub_queries": ["Q1","Q2"] }') [] # returns empty list → bug 3. Check vector‑store query logs 2026-10-16T12:04:35.112Z DEBUG vector_search key=" - What were the total revenues for Q4 2022?" 2026-10-16T12:04:35.115Z WARN vector_search returned 0 hits 4. Compare with a known‑good version (pre‑update) VersionLLM PromptLLM ResponseParser Output v1.3 (pre‑update)List sub‑queries, one per line.What were the total revenues for Q4 2022?\nHow did operating expenses change?["What were the total revenues for Q4 2022?","How did operating expenses change?"] v1.4 (post‑update)List sub‑queries, one per line.{ "sub_queries": [...] }[] (parser failure) Solution Two complementary fixes were applied:

Q: 1. Update the prompt to enforce plain‑text output

# Old prompt fragment prompt = """You are a query decomposer. Split the user question into independent sub‑queries. Return each sub‑query on a separate line without any extra formatting.""" # New prompt fragment prompt = """You are a query decomposer. Split the user question into independent sub‑queries. Return ONLY plain text, one sub‑query per line, no JSON, no bullet characters, no leading/trailing whitespace.""" 2. Harden the parser to handle both formats # Before def parse_subqueries(response: str) -> List[str]: return [line.strip() for line in response.splitlines() if line.strip()] # After import json import re def parse_subqueries(response: str) -> List[str]: # Try JSON first try: data = json.loads(response) if isinstance(data, dict) and "sub_queries" in data: return [q.strip() for q in data["sub_queries"] if q.strip()] except json.JSONDecodeError: pass # Fallback to plain‑text line split, stripping bullets and whitespace lines = response.splitlines() cleaned = [] for line in lines: line = line.strip() # Remove common bullet prefixes line = re.sub(r'^[\-\*\•]\s*', '', line) if line: cleaned.append(line) return cleaned 3. Deploy the changes # Kubernetes rolling update kubectl set image deployment/rag-service rag-service=registry.example.com/rag-service:1.5.0 --record kubectl rollout status deployment/rag-service Verification After the rollout the following checks confirmed successful remediation:

Q: Functional test

curl -X POST http://rag-service/api/query \ -d '{"question":"Show revenue and expense trends for Q4 2022 compared to Q3"}' \ -H "Content-Type: application/json" # Expected fragment in response "Revenue for Q4 2022 was $12.4M, a 8% increase over Q3." Log inspection 2026-10-16T12:18:02.331Z INFO sub‑queries generated: ["Revenue for Q4 2022?","Expense change Q4 vs Q3?"] 2026-10-16T12:18:02.450Z INFO vector_search key="Revenue for Q4 2022?" hits=12 2026-10-16T12:18:02.452Z INFO vector_search key="Expense change Q4 vs Q3?" hits=9 Metrics RAG sub‑query success rate: 99.8 % (up from 42 %). Average response latency: 1.2 s (down from 2.8 s). Empty‑result alerts: cleared. Prevention To avoid regression after future model updates:

In this article

Table of Contents

Problem

The Retrieval‑Augmented Generation (RAG) service receives a user query, asks GPT‑3.5 to decompose it into sub‑queries, and then uses each sub‑query as a key for a vector‑store lookup. After a recent model update the service started returning empty or irrelevant document sets:

Log entry: INFO sub‑queries generated: []
Log entry: WARN vector search returned 0 hits for key: "financial‑report‑2022‑Q4"
End‑user symptom: “I’m not getting any answer” or “the answer contains unrelated information”.
Impact: increased latency (fallback to LLM‑only generation) and degraded user experience across all microservice instances.

Root Cause

The updated GPT‑3.5 model changed its prompt‑following behavior. The original prompt asked the model to list each sub‑query on a new line. The new model started emitting JSON objects with a sub_queries field, but the downstream parser still expects plain line‑separated strings. As a result:

The parser fails to split the response, producing an empty array.
When the model does emit plain text, it now adds extra punctuation and whitespace (e.g., " - revenue growth?") that the vector‑store client treats as part of the query string, dramatically narrowing the embedding space.
Because the vector store uses exact tokenization before embedding, these malformed keys generate embeddings that do not match any stored vectors, leading to empty result sets.

Debugging the Decomposition

Below is the step‑by‑step investigation performed on a failing pod.

1. Capture the raw LLM response

2026-10-16T12:04:33.210Z INFO  request_id=7f3c2a sub‑query generation start
2026-10-16T12:04:34.018Z INFO  raw LLM response:
{
  "sub_queries": [
    "What were the total revenues for Q4 2022?",
    "How did operating expenses change compared to Q3 2022?"
  ]
}
2026-10-16T12:04:34.020Z INFO  parsing sub‑queries

2. Verify parser behavior

# Python snippet used in the service
def parse_subqueries(response: str) -> List[str]:
    # Old implementation
    return [line.strip() for line in response.splitlines() if line.strip()]

# Test with captured response
>>> parse_subqueries('{ "sub_queries": ["Q1","Q2"] }')
[]   # returns empty list → bug

3. Check vector‑store query logs

2026-10-16T12:04:35.112Z DEBUG  vector_search key="  -  What were the total revenues for Q4 2022?"
2026-10-16T12:04:35.115Z WARN   vector_search returned 0 hits

4. Compare with a known‑good version (pre‑update)

Version	LLM Prompt	LLM Response	Parser Output
v1.3 (pre‑update)	List sub‑queries, one per line.	What were the total revenues for Q4 2022?\nHow did operating expenses change?	[“What were the total revenues for Q4 2022?”,”How did operating expenses change?”]
v1.4 (post‑update)	List sub‑queries, one per line.	{ “sub_queries”: […] }	[] (parser failure)

Solution

Two complementary fixes were applied:

1. Update the prompt to enforce plain‑text output

# Old prompt fragment
prompt = """You are a query decomposer. Split the user question into independent sub‑queries.
Return each sub‑query on a separate line without any extra formatting."""

# New prompt fragment
prompt = """You are a query decomposer. Split the user question into independent sub‑queries.
Return ONLY plain text, one sub‑query per line, no JSON, no bullet characters, no leading/trailing whitespace."""

2. Harden the parser to handle both formats

# Before
def parse_subqueries(response: str) -> List[str]:
    return [line.strip() for line in response.splitlines() if line.strip()]

# After
import json
import re

def parse_subqueries(response: str) -> List[str]:
    # Try JSON first
    try:
        data = json.loads(response)
        if isinstance(data, dict) and "sub_queries" in data:
            return [q.strip() for q in data["sub_queries"] if q.strip()]
    except json.JSONDecodeError:
        pass

    # Fallback to plain‑text line split, stripping bullets and whitespace
    lines = response.splitlines()
    cleaned = []
    for line in lines:
        line = line.strip()
        # Remove common bullet prefixes
        line = re.sub(r'^[\-\*\•]\s*', '', line)
        if line:
            cleaned.append(line)
    return cleaned

3. Deploy the changes

# Kubernetes rolling update
kubectl set image deployment/rag-service rag-service=registry.example.com/rag-service:1.5.0 --record
kubectl rollout status deployment/rag-service

Verification

After the rollout the following checks confirmed successful remediation:

Functional test

curl -X POST http://rag-service/api/query \
  -d '{"question":"Show revenue and expense trends for Q4 2022 compared to Q3"}' \
  -H "Content-Type: application/json"

# Expected fragment in response
"Revenue for Q4 2022 was $12.4M, a 8% increase over Q3."

Log inspection

2026-10-16T12:18:02.331Z INFO  sub‑queries generated: ["Revenue for Q4 2022?","Expense change Q4 vs Q3?"]
2026-10-16T12:18:02.450Z INFO  vector_search key="Revenue for Q4 2022?" hits=12
2026-10-16T12:18:02.452Z INFO  vector_search key="Expense change Q4 vs Q3?" hits=9

Metrics

RAG sub‑query success rate: 99.8 % (up from 42 %).
Average response latency: 1.2 s (down from 2.8 s).
Empty‑result alerts: cleared.

Prevention

To avoid regression after future model updates:

Prompt versioning: Store the exact prompt string in a ConfigMap and reference it from code; bump the ConfigMap version when changing prompts.
Response schema validation: Add a lightweight JSON schema check before parsing; fallback to plain‑text parsing only if validation fails.
Integration test suite: Include a test that sends a representative complex query and asserts that at least one document is retrieved.
Observability: Emit a custom metric rag.subquery_parse_errors and set an alert threshold of >5 per minute.
Canary deployment: Deploy new model versions to a small subset of pods and compare subquery_success_rate before full rollout.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the sub‑query list become empty only after a model update? The newer model prefers structured JSON output when the prompt is ambiguous. If the downstream parser expects plain text, the mismatch yields an empty list.
Can I keep the JSON format and avoid changing the parser? Yes, but you must update the parser to detect and extract the sub_queries field. The provided parse_subqueries function already supports both formats.
What if the model returns a mix of JSON and plain text? The parser attempts JSON deserialization first; if it fails, it falls back to line‑based parsing, handling mixed outputs gracefully.
How do I know whether the vector‑store query key is malformed? Enable debug logging for the search client; it will log the exact key string. Look for leading/trailing whitespace, bullet characters, or unexpected punctuation.
Is there a way to automatically retry with a cleaned key? Implement a wrapper around the vector‑store client that normalizes the key (trim whitespace, remove bullets) before embedding. This adds resilience against minor formatting issues.