RAG decomposition service throws DecompositionFailedException on complex queries

Problem: DecompositionFailedException in RAG Query Decomposition Service

The RAG decomposition microservice consumes user queries from the rag.requests exchange, splits them into sub‑queries, and republishes each sub‑query to topic queues such as rag.subqueries.*. When a complex query (e.g., nested JSON, multi‑entity request, or very long prompt) is processed, the service throws DecompositionFailedException. The exception propagates, the original RabbitMQ message is nacked without requeue, and downstream retrieval/generation services receive no sub‑queries.

Typical observable symptoms include:

• Log entry: DecompositionFailedException: Failed to decompose query – token limit exceeded
• RabbitMQ warning: basic.nack received for delivery-tag 42, requeue=false
• Dead‑letter queue accumulation (if DLX configured) or silent message loss (if no DLX)
• Missing retrieval jobs in monitoring dashboards, leading to increased latency for end‑users

Root Cause Analysis

Why the exception occurs

The decomposition chain (often built on LangChain or LlamaIndex) relies on an LLM call to split the query. Two failure modes dominate:

1. Token‑limit overflow: Complex queries exceed the LLM’s maximum input tokens. The splitter raises DecompositionFailedException with the message “token limit exceeded”.
   Evidence: GitHub issue langchain-ai/langchain#8421 and facebookresearch/llama-index#1234 report identical behavior.
2. Timeout or runtime error in the LLM call: Under high load the LLM service does not respond within the configured 30 s window, causing a TimeoutError that the splitter converts into DecompositionFailedException.
   Evidence: Incident “SaaS analytics tool (Mar 2025)” shows a 406 PRECONDITION_FAILED error after the service timed out.

Both cases leave the consumer without a successful processing path. Because the service does not explicitly ack the original message nor nack with requeue=true, RabbitMQ treats the exception as a hard failure and drops the message (or routes it to a dead‑letter exchange if configured).

Interaction with RabbitMQ

According to the RabbitMQ official documentation on exchanges and routing, the decomposition service must republish sub‑queries to a valid exchange. When the service crashes before publishing, the channel may be closed, leading to errors such as:

Channel closed: 406 PRECONDITION_FAILED - unknown exchange 'rag.subqueries'

Furthermore, the publisher confirms mechanism is often disabled, so the service cannot detect that the sub‑query publish failed.

Investigation and Debugging

Log inspection

2025-05-12 14:03:27,842 ERROR rag.decomposer - DecompositionFailedException: Failed to decompose query – token limit exceeded
2025-05-12 14:03:27,845 WARN rabbitmq - basic.nack received for delivery-tag 57, requeue=false
2025-05-12 14:03:27,850 INFO rabbitmq - Consumer cancelled by server: consumer cancelled due to exception in callback

RabbitMQ metrics

• Check queue.rag.requests.messages_unacknowledged – spikes indicate messages stuck in processing.
• Inspect dead‑letter queue rag.dlq for nacked messages.

Reproduce locally

# Publish a complex query that triggers the failure
curl -X POST http://frontend.local/publish \
  -d '{"query":"Find all orders where customer.address.city = \"San Francisco\" and items contain a nested JSON with specifications > 10"}' \
  -H "Content-Type: application/json"

# Consume with the decomposition service in debug mode
java -jar rag-decomposer.jar --log-level=DEBUG

Observe the stack trace and confirm whether the exception originates from the LLM client or the splitter logic.

Check consumer QoS

High prefetch values can cause back‑pressure, leading to timeouts. Verify the prefetch setting:

channel.basicQos(5); // recommended for low‑latency RAG pipelines

Reference: RabbitMQ Consumer Prefetch and QoS documentation.

Resolution

Code changes

Introduce robust error handling around the decomposition step and ensure proper message acknowledgement.

// Before (simplified)
try {
    List<SubQuery> parts = splitter.decompose(message.getBody());
    for (SubQuery part : parts) {
        channel.basicPublish("rag.subqueries", part.getTopic(), null, part.toBytes());
    }
    channel.basicAck(deliveryTag, false);
} catch (DecompositionFailedException e) {
    logger.error("Decomposition failed", e);
    // No ack/nack -> message lost
}

// After – explicit nack with requeue and fallback publishing
try {
    List<SubQuery> parts = splitter.decompose(message.getBody());
    for (SubQuery part : parts) {
        channel.basicPublish("rag.subqueries", part.getTopic(), null, part.toBytes());
    }
    channel.basicAck(deliveryTag, false);
} catch (DecompositionFailedException e) {
    logger.error("Decomposition failed: {}", e.getMessage());

    // Publish a fallback “fallback” sub‑query or mark as failed
    SubQuery fallback = SubQuery.fallback(message.getBody());
    channel.basicPublish("rag.subqueries.fallback", "", null, fallback.toBytes());

    // Nack with requeue=false so it goes to DLX (if configured)
    channel.basicNack(deliveryTag, false, false);
}

LLM request adjustments

• Increase the LLM token limit or truncate input before splitting.
• Add a retry with exponential back‑off for timeout errors.
• Set a lower maxTokens for the splitter to stay within model limits.

RabbitMQ configuration

Setting	Recommended Value	Rationale
Exchange for sub‑queries	rag.subqueries (topic)	Ensure existence before deployment
Dead‑letter exchange	rag.dlx	Capture failed messages for later analysis
Consumer prefetch	5–10	Prevent channel blockage under load
Publisher confirms	Enabled	Detect publish failures immediately

Configure DLX as per RabbitMQ Dead Letter Exchanges documentation.

Verification

Functional test

# Publish a known complex query
curl -X POST http://frontend.local/publish -d '{"query":""}' -H "Content-Type: application/json"

# Expect sub‑queries in the topic queue
rabbitmqadmin get queue name=rag.subqueries.complex count=10 requeue=false

Successful output should list sub‑query messages with proper routing keys.

Metrics validation

• Confirm queue.rag.requests.messages_unacknowledged drops to near zero.
• Check dead‑letter queue depth – should only contain messages that truly cannot be split after fallback handling.
• Verify that the decomposition service’s consumer_latency_ms metric stays below the SLA threshold (e.g., < 200 ms).

Prevention and Best Practices

• Input sanitization: Truncate or chunk queries exceeding the LLM’s token limit before invoking the splitter.
• Graceful degradation: Always publish a fallback sub‑query or error payload to avoid silent drops.
• Publisher confirms: Enable confirms on the channel to catch routing failures early.
• Dead‑letter handling: Configure a DLX for the rag.subqueries exchange; monitor its depth and set alerts on sudden spikes.
• Consumer QoS: Tune prefetch to match processing capacity; combine with circuit‑breaker patterns for downstream LLM calls.
• Observability: Emit structured logs with correlation IDs (e.g., request_id) and expose Prometheus metrics for decomposition success/failure rates.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the exception only appear for long queries?
   Because the splitter forwards the entire query to the LLM. When the token count exceeds the model’s limit, the LLM returns an error that the splitter translates into DecompositionFailedException. Truncating or chunking the input resolves the issue.
2. Can I rely on RabbitMQ’s automatic requeue after an exception?
   No. By default, a consumer that throws an exception must explicitly basicNack with requeue=true. Otherwise the message is dropped or dead‑lettered, as seen in the “basic.nack … requeue=false” log entries.
3. How do I know whether a sub‑query failed to publish?
   Enable publisher confirms (channel.confirmSelect()) and listen for Basic.Ack or Basic.Nack callbacks. Log a warning if a confirm is not received within the timeout.
4. What is the proper way to configure a dead‑letter exchange for this pipeline?
   Declare a DLX (e.g., rag.dlx) and bind it to the original queue with x-dead-letter-exchange=rag.dlx. Then route dead‑lettered messages to a monitoring queue for manual inspection.
5. Is increasing the consumer prefetch value a good way to improve throughput?
   Higher prefetch can increase throughput but also amplifies the impact of a single failure (more messages stay unacknowledged). For RAG pipelines, a modest prefetch (5–10) balances latency and reliability.