Implementing Query Decomposition and Sub-Question Synthesis for Complex RAG Pipelines

Introduction

Here's a scenario that will feel familiar if you've built anything beyond a basic RAG prototype: a user asks your system, "How did our Q3 marketing spend in the European region compare to Q2, and what was the impact on customer acquisition rates given the new pricing changes we rolled out in July?" You've got a well-tuned vector store, solid embeddings, a good reranker — and your pipeline still returns something mediocre. Not wrong, exactly. Just... thin.

The problem isn't your retrieval infrastructure. It's that you're treating a multi-part, multi-hop question as if it's a single retrieval unit. Standard RAG retrieves documents that are semantically similar to the full query. But a complex question like that one has at least four distinct information needs: Q3 spend data, Q2 spend data, acquisition rate trends, and the July pricing change details. A single embedding query against all four simultaneously will almost certainly miss at least two.

Query decomposition solves this by breaking complex questions into focused sub-questions, retrieving targeted context for each, and then synthesizing those answers into a coherent final response. By the end of this lesson, you'll have built a complete, production-grade query decomposition system that handles real-world complexity.

What you'll learn:

• How to detect when a query needs decomposition and when it doesn't
• How to build a decomposition layer that generates focused, answerable sub-questions
• How to manage parallel retrieval across sub-questions efficiently
• How to synthesize partial answers from multiple sub-queries into a coherent final response
• How to handle dependency ordering when sub-questions rely on each other's answers

---

Prerequisites

You should be comfortable with the fundamentals of RAG — if you need a refresher, Retrieval-Augmented Generation Explained: How RAG Works and When to Use It covers the conceptual foundation. You should also have hands-on experience with embeddings and vector retrieval from Building a Production Document Q&A System with Vector Embeddings. You'll be writing Python throughout this lesson, calling LLMs for both decomposition and synthesis steps.

---

Why Standard RAG Breaks on Complex Queries

Before we build anything, let's understand the failure mode precisely.

When you embed a complex query and do a nearest-neighbor search, you're collapsing multiple information needs into a single vector. The embedding model will weight the most semantically prominent parts of the query and de-emphasize others. In the marketing spend example, the embedding might heavily weight "European region marketing spend" and return documents about regional budget reports — but completely miss the July pricing change documents because that concept got diluted by the surrounding query text.

There are three distinct query types where this pattern breaks down:

Comparative queries — "How did X compare to Y across Z dimensions?" These require retrieving information about X, about Y, and understanding the comparison dimensions — often from completely different document sections.

Multi-hop queries — "What were the downstream effects of the policy change announced after the merger?" You first need to find which merger, then find the policy change, then find its effects. Each step depends on the previous answer.

Aggregative queries — "Summarize the key risks mentioned across all our product roadmap documents." You're not looking for one answer; you're synthesizing across many independent retrievals.

Key insight: A query's complexity isn't just about length or word count. A short question like "Why did we miss guidance after the CFO change?" is a multi-hop query that requires sequential reasoning. Complexity is about the number of distinct retrieval operations needed to answer it fully.

---

The Architecture of a Decomposition Pipeline

The full pipeline has four stages:

1. Complexity detection — Decide if decomposition is warranted
2. Sub-question generation — Break the query into focused atomic questions
3. Parallel retrieval — Fetch context for each sub-question independently
4. Synthesis — Combine sub-answers into a final response

Let's build each stage, then wire them together.

---

Stage 1: Complexity Detection

You don't want to decompose every query. Simple factual questions — "What is our standard payment terms policy?" — don't benefit from decomposition and add unnecessary latency and cost. We need a classifier that routes queries appropriately.

import os
import json
from openai import OpenAI
from dataclasses import dataclass
from typing import Optional

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

COMPLEXITY_SYSTEM_PROMPT = """You are a query analysis assistant. Analyze the user's question and classify its complexity.

Return a JSON object with the following fields:
- needs_decomposition: boolean — true if the question requires multiple distinct information retrievals
- complexity_type: one of ["simple", "comparative", "multi_hop", "aggregative"]
- reasoning: brief explanation of your classification (1-2 sentences)
- estimated_subquestions: integer — estimated number of sub-questions needed (1 if simple)

A question needs decomposition if it:
- Compares two or more entities, time periods, or scenarios
- Requires finding one piece of information to answer another part
- Asks for synthesis across multiple independent topics
- Contains multiple distinct "what/why/how" clauses joined by "and" or "also"

Simple questions that DON'T need decomposition:
- Single-fact lookups
- Definitions
- Questions with a single clear information need"""

@dataclass
class ComplexityAnalysis:
    needs_decomposition: bool
    complexity_type: str
    reasoning: str
    estimated_subquestions: int

def analyze_query_complexity(query: str) -> ComplexityAnalysis:
    """Classify whether a query needs decomposition and what type."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Fast, cheap — fine for classification
        messages=[
            {"role": "system", "content": COMPLEXITY_SYSTEM_PROMPT},
            {"role": "user", "content": f"Analyze this query: {query}"}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    
    result = json.loads(response.choices[0].message.content)
    return ComplexityAnalysis(
        needs_decomposition=result["needs_decomposition"],
        complexity_type=result["complexity_type"],
        reasoning=result["reasoning"],
        estimated_subquestions=result["estimated_subquestions"]
    )

# Test it
query = "How did our Q3 marketing spend in Europe compare to Q2, and what impact did it have on customer acquisition given the July pricing changes?"
analysis = analyze_query_complexity(query)
print(f"Needs decomposition: {analysis.needs_decomposition}")
print(f"Type: {analysis.complexity_type}")
print(f"Reasoning: {analysis.reasoning}")
print(f"Estimated sub-questions: {analysis.estimated_subquestions}")

This uses gpt-4o-mini because classification is a low-stakes, structured task. Save your more capable (and expensive) model calls for decomposition and synthesis. For a deeper look at how structured JSON responses work with LLMs, see Structured Output: Getting JSON, Tables, and Code from LLMs.

Tip: Use temperature=0 for your classification and decomposition steps. You want deterministic, consistent behavior here — creativity is for synthesis.

---

Stage 2: Sub-Question Generation

This is where the real intellectual work happens. Good decomposition isn't just splitting a question on conjunctions — it's identifying the minimal set of focused, independently-answerable questions that together address the original query.

from typing import List
from dataclasses import dataclass, field
from enum import Enum

class DependencyType(Enum):
    INDEPENDENT = "independent"  # Can be answered in parallel
    SEQUENTIAL = "sequential"    # Must wait for another sub-question's answer

@dataclass
class SubQuestion:
    id: str
    question: str
    dependency_type: DependencyType
    depends_on: List[str] = field(default_factory=list)
    context_hint: Optional[str] = None  # Hints for retrieval (e.g., which doc sections to prioritize)

DECOMPOSITION_SYSTEM_PROMPT = """You are an expert at breaking down complex questions into focused sub-questions for a document retrieval system.

Given a complex query, generate the minimal set of sub-questions needed to fully answer it.

Rules for good sub-questions:
1. Each sub-question should be answerable from a single, focused document search
2. Avoid redundancy — don't generate sub-questions that retrieve the same information
3. Identify dependencies — some sub-questions may need another's answer before they can be answered
4. Keep questions specific and concrete — avoid vague sub-questions
5. Use the original query's terminology to maintain semantic consistency

Return a JSON object with:
- subquestions: array of objects, each with:
  - id: string (e.g., "sq1", "sq2")
  - question: the focused sub-question text
  - dependency_type: "independent" or "sequential"
  - depends_on: array of sub-question IDs this depends on (empty if independent)
  - context_hint: optional string hinting at what kind of document/section to search

Original query context helps — think about what documents would need to be retrieved to fully answer."""

def decompose_query(
    query: str,
    complexity_analysis: ComplexityAnalysis,
    additional_context: str = ""
) -> List[SubQuestion]:
    """Break a complex query into focused, retrievable sub-questions."""
    
    user_prompt = f"""Query to decompose: {query}

Query type: {complexity_analysis.complexity_type}
Expected complexity: {complexity_analysis.estimated_subquestions} sub-questions

{f'Additional context: {additional_context}' if additional_context else ''}

Generate the sub-questions needed to fully answer this query."""

    response = client.chat.completions.create(
        model="gpt-4o",  # Use capable model for decomposition
        messages=[
            {"role": "system", "content": DECOMPOSITION_SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    
    result = json.loads(response.choices[0].message.content)
    subquestions = []
    
    for sq in result["subquestions"]:
        subquestions.append(SubQuestion(
            id=sq["id"],
            question=sq["question"],
            dependency_type=DependencyType(sq["dependency_type"]),
            depends_on=sq.get("depends_on", []),
            context_hint=sq.get("context_hint")
        ))
    
    return subquestions

# Test with our example query
subquestions = decompose_query(query, analysis)
for sq in subquestions:
    dep_info = f" (depends on: {sq.depends_on})" if sq.depends_on else " (independent)"
    print(f"{sq.id}: {sq.question}{dep_info}")

For our marketing spend query, you'd typically get output like:

sq1: What was the total marketing spend in the European region in Q3? (independent)
sq2: What was the total marketing spend in the European region in Q2? (independent)
sq3: What pricing changes were rolled out in July? (independent)
sq4: What were the customer acquisition rates in Q2 vs Q3 in Europe? (independent)
sq5: How did the July pricing changes affect customer acquisition rates? (depends on: sq3, sq4)

Notice that sq5 is marked as sequential with dependencies. That's crucial — it needs the specific pricing changes from sq3 and the acquisition rate data from sq4 before it can be meaningfully answered. This structure lets us parallelize sq1 through sq4 and then run sq5 after.

Warning: Decomposition can go too granular. If you find your system generating 8-10 sub-questions for most queries, your decomposition prompt is being over-eager. This dramatically increases latency and cost. Aim for 2-5 sub-questions for typical complex queries, and tune your prompt to discourage excessive splitting.

---

Stage 3: Parallel Retrieval with Dependency Management

Now we retrieve context for each sub-question. Independent questions run in parallel; sequential ones wait for their dependencies.

import asyncio
from typing import Dict, Tuple
import numpy as np

# Assume you have a vector store client — we'll abstract it
# In production this might be Pinecone, Weaviate, pgvector, etc.
class VectorStoreClient:
    """Abstract interface for your vector store."""
    
    def retrieve(
        self, 
        query: str, 
        top_k: int = 5,
        filter_metadata: Optional[Dict] = None
    ) -> List[Dict]:
        """Returns list of {'content': str, 'metadata': dict, 'score': float}"""
        raise NotImplementedError

@dataclass
class SubQuestionResult:
    sub_question: SubQuestion
    retrieved_chunks: List[Dict]
    answer: str
    confidence: float

async def retrieve_for_subquestion(
    sq: SubQuestion,
    vector_store: VectorStoreClient,
    resolved_dependencies: Dict[str, SubQuestionResult],
    top_k: int = 5
) -> Tuple[str, List[Dict]]:
    """
    Retrieve relevant chunks for a sub-question.
    If it has dependencies, augment the query with resolved answers.
    """
    retrieval_query = sq.question
    
    # For sequential questions, enrich the query with dependency answers
    if sq.depends_on and resolved_dependencies:
        dep_context = "\n".join([
            f"Known: {resolved_dependencies[dep_id].answer}"
            for dep_id in sq.depends_on
            if dep_id in resolved_dependencies
        ])
        # The retrieval query incorporates what we already know
        retrieval_query = f"{sq.question}\n\nContext from related questions:\n{dep_context}"
    
    # Run retrieval (use asyncio.to_thread if your client is synchronous)
    chunks = await asyncio.to_thread(
        vector_store.retrieve,
        retrieval_query,
        top_k
    )
    
    return sq.id, chunks

async def answer_subquestion(
    sq: SubQuestion,
    chunks: List[Dict],
    resolved_dependencies: Dict[str, SubQuestionResult],
    original_query: str
) -> SubQuestionResult:
    """Generate a focused answer for a single sub-question given retrieved chunks."""
    
    context_text = "\n\n---\n\n".join([
        f"Source: {chunk['metadata'].get('source', 'Unknown')}\n{chunk['content']}"
        for chunk in chunks
    ])
    
    dep_answers = ""
    if sq.depends_on and resolved_dependencies:
        dep_answers = "\n\nAnswers from related sub-questions:\n" + "\n".join([
            f"- {resolved_dependencies[dep_id].sub_question.question}\n  Answer: {resolved_dependencies[dep_id].answer}"
            for dep_id in sq.depends_on
            if dep_id in resolved_dependencies
        ])
    
    system_prompt = """You are answering a focused sub-question as part of answering a larger complex query.
    
Answer ONLY the specific sub-question asked. Be precise and cite the source documents when possible.
If the context doesn't contain sufficient information, say so explicitly — don't hallucinate.
Rate your confidence from 0.0 to 1.0 at the end in format: [CONFIDENCE: X.X]"""
    
    user_prompt = f"""Original query (for context): {original_query}

Sub-question to answer: {sq.question}

Retrieved context:
{context_text}
{dep_answers}

Answer the sub-question based on the retrieved context."""

    response = await asyncio.to_thread(
        client.chat.completions.create,
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        temperature=0.1
    )
    
    answer_text = response.choices[0].message.content
    
    # Parse confidence score
    confidence = 0.5  # default
    if "[CONFIDENCE:" in answer_text:
        try:
            conf_str = answer_text.split("[CONFIDENCE:")[-1].strip().rstrip("]")
            confidence = float(conf_str)
            answer_text = answer_text.split("[CONFIDENCE:")[0].strip()
        except (ValueError, IndexError):
            pass
    
    return SubQuestionResult(
        sub_question=sq,
        retrieved_chunks=chunks,
        answer=answer_text,
        confidence=confidence
    )

async def execute_retrieval_plan(
    subquestions: List[SubQuestion],
    vector_store: VectorStoreClient,
    original_query: str,
    top_k: int = 5
) -> Dict[str, SubQuestionResult]:
    """
    Execute retrieval for all sub-questions, respecting dependency ordering.
    Independent questions run in parallel; sequential ones wait for dependencies.
    """
    resolved: Dict[str, SubQuestionResult] = {}
    
    # Topologically sort: process independent questions first
    remaining = list(subquestions)
    
    while remaining:
        # Find all questions whose dependencies are fully resolved
        ready = [
            sq for sq in remaining
            if all(dep in resolved for dep in sq.depends_on)
        ]
        
        if not ready:
            raise ValueError(
                f"Circular dependency detected in sub-questions: "
                f"{[sq.id for sq in remaining]}"
            )
        
        # Retrieve context for all ready questions in parallel
        retrieval_tasks = [
            retrieve_for_subquestion(sq, vector_store, resolved, top_k)
            for sq in ready
        ]
        retrieval_results = await asyncio.gather(*retrieval_tasks)
        
        # Answer all ready questions in parallel
        answer_tasks = [
            answer_subquestion(sq, chunks, resolved, original_query)
            for sq, (_, chunks) in zip(ready, retrieval_results)
        ]
        answers = await asyncio.gather(*answer_tasks)
        
        # Mark as resolved
        for result in answers:
            resolved[result.sub_question.id] = result
            remaining.remove(result.sub_question)
    
    return resolved

The topological sort here is doing real work. It ensures that sequential sub-questions only run when their dependencies are complete, while maximizing parallelism for everything else. For complex pipelines like this, you may want to look at Orchestrating Parallel LLM Calls: Batching, Concurrency, and Async Patterns for High-Throughput Production Pipelines for more sophisticated concurrency patterns.

Note: The asyncio.to_thread() wrappers handle synchronous clients (like the standard OpenAI Python client) gracefully in an async context. If you're using an async client like AsyncOpenAI, drop these wrappers and await directly.

---

Stage 4: Answer Synthesis

With all sub-question answers in hand, we synthesize them into a single, coherent final response. This isn't just concatenation — it's an LLM call that understands the original intent and weaves the partial answers together.

def synthesize_final_answer(
    original_query: str,
    resolved_subquestions: Dict[str, SubQuestionResult],
    subquestion_order: List[SubQuestion]
) -> str:
    """
    Synthesize all sub-question answers into a coherent final response.
    """
    
    # Build the synthesis context — order matters for readability
    sub_answers_text = ""
    for sq in subquestion_order:
        result = resolved_subquestions[sq.id]
        confidence_label = "high" if result.confidence > 0.7 else "medium" if result.confidence > 0.4 else "low"
        sub_answers_text += f"\n**{sq.question}**\n"
        sub_answers_text += f"Answer (confidence: {confidence_label}): {result.answer}\n"
    
    synthesis_prompt = f"""You have answered several sub-questions to address a complex user query. 
Now synthesize these answers into a single, comprehensive, well-structured response.

Original query: {original_query}

Sub-question answers:
{sub_answers_text}

Synthesis instructions:
- Address the original query directly and completely
- Integrate the sub-answers naturally — don't just list them
- When sub-answers have low confidence, note the uncertainty
- If sub-answers conflict, acknowledge the discrepancy
- Use appropriate structure (paragraphs, bullet points) based on the content
- Be concise — don't repeat information unnecessarily
- Do not mention the sub-question decomposition process to the user"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system", 
                "content": "You are an expert analyst synthesizing research findings into clear, actionable answers."
            },
            {"role": "user", "content": synthesis_prompt}
        ],
        temperature=0.2
    )
    
    return response.choices[0].message.content

The last instruction in that prompt — "do not mention the sub-question decomposition process" — matters. Users don't care about your pipeline architecture. They asked a question and want an answer. The decomposition should be an implementation detail.

---

Wiring It All Together

Now we build the complete pipeline orchestrator that handles both simple and complex queries:

import time
import logging

logger = logging.getLogger(__name__)

@dataclass
class RAGResponse:
    answer: str
    original_query: str
    decomposed: bool
    subquestions: Optional[List[SubQuestion]]
    resolved_results: Optional[Dict[str, SubQuestionResult]]
    total_chunks_retrieved: int
    latency_ms: float

class QueryDecompositionRAG:
    def __init__(
        self,
        vector_store: VectorStoreClient,
        top_k_per_subquestion: int = 5,
        enable_decomposition: bool = True
    ):
        self.vector_store = vector_store
        self.top_k = top_k_per_subquestion
        self.enable_decomposition = enable_decomposition
    
    async def answer(
        self, 
        query: str,
        force_decompose: bool = False
    ) -> RAGResponse:
        start_time = time.time()
        
        # Step 1: Analyze complexity
        complexity = analyze_query_complexity(query)
        logger.info(f"Query classified as: {complexity.complexity_type}, "
                   f"decomposition needed: {complexity.needs_decomposition}")
        
        should_decompose = (
            self.enable_decomposition and 
            (force_decompose or complexity.needs_decomposition)
        )
        
        if not should_decompose:
            # Fast path: standard single-retrieval RAG
            chunks = self.vector_store.retrieve(query, top_k=self.top_k)
            answer = self._simple_answer(query, chunks)
            
            return RAGResponse(
                answer=answer,
                original_query=query,
                decomposed=False,
                subquestions=None,
                resolved_results=None,
                total_chunks_retrieved=len(chunks),
                latency_ms=(time.time() - start_time) * 1000
            )
        
        # Step 2: Decompose the query
        subquestions = decompose_query(query, complexity)
        logger.info(f"Decomposed into {len(subquestions)} sub-questions")
        
        # Step 3: Execute retrieval plan (parallel where possible)
        resolved = await execute_retrieval_plan(
            subquestions,
            self.vector_store,
            query,
            self.top_k
        )
        
        # Step 4: Synthesize final answer
        final_answer = synthesize_final_answer(query, resolved, subquestions)
        
        total_chunks = sum(
            len(result.retrieved_chunks) for result in resolved.values()
        )
        
        return RAGResponse(
            answer=final_answer,
            original_query=query,
            decomposed=True,
            subquestions=subquestions,
            resolved_results=resolved,
            total_chunks_retrieved=total_chunks,
            latency_ms=(time.time() - start_time) * 1000
        )
    
    def _simple_answer(self, query: str, chunks: List[Dict]) -> str:
        """Standard RAG answer for simple queries."""
        context = "\n\n".join([c["content"] for c in chunks])
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Answer the question based on the provided context. If the context is insufficient, say so."},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}
            ],
            temperature=0.1
        )
        return response.choices[0].message.content

# Usage
async def main():
    vector_store = YourVectorStoreClient()  # Your actual implementation
    rag = QueryDecompositionRAG(vector_store, top_k_per_subquestion=5)
    
    response = await rag.answer(
        "How did our Q3 marketing spend in Europe compare to Q2, "
        "and what impact did it have on customer acquisition given the July pricing changes?"
    )
    
    print(f"Decomposed: {response.decomposed}")
    print(f"Sub-questions asked: {len(response.subquestions or [])}")
    print(f"Total chunks retrieved: {response.total_chunks_retrieved}")
    print(f"Latency: {response.latency_ms:.0f}ms")
    print(f"\nAnswer:\n{response.answer}")

---

Hands-On Exercise: Building a Financial Document Analyzer

Let's apply what you've built to a realistic scenario: a financial document analyzer that lets analysts ask complex questions across earnings reports, 10-K filings, and analyst notes.

Your task is to extend the system above with the following features:

Step 1 — Add metadata-aware retrieval. Modify retrieve_for_subquestion to accept a date range filter. When a sub-question mentions a specific quarter (e.g., "Q3 2023"), the retrieval should filter to documents from that period. The SubQuestion.context_hint field is where the decomposition layer should put this information.

Update the decomposition prompt to include:

- context_hint: JSON string with optional fields:
  - time_period: e.g., "Q3 2023" or "July 2023"
  - document_type: e.g., "earnings_report", "10k", "analyst_note"
  - company: company name if the question is about a specific entity

Step 2 — Add a confidence-gated synthesis. Modify synthesize_final_answer to behave differently based on aggregate confidence. If more than half the sub-question answers have confidence below 0.5:

• Flag the response as low-confidence
• Include explicit uncertainty language in the synthesis prompt
• Add a "⚠️ Low confidence — retrieved context may be incomplete" prefix to the answer

Step 3 — Build a simple CLI test harness. Write a script that accepts queries from stdin, runs them through your pipeline, and prints:

• Whether decomposition fired
• Each sub-question and its confidence score
• The final answer

Test it with these three queries:

1. "What is our standard refund policy?" (should NOT decompose)
2. "How did revenue growth compare between the software and hardware segments in FY2023 vs FY2022?" (SHOULD decompose — comparative)
3. "Which product line had the highest margin improvement after the cost restructuring that followed the Q2 2022 earnings miss?" (SHOULD decompose — multi-hop)

---

Common Mistakes & Troubleshooting

Problem: Sub-questions are semantically too similar and retrieve duplicate chunks.

This happens when your decomposition isn't specific enough. A comparative query might generate "What were Q3 sales figures?" and "What were the Q3 revenue numbers?" — functionally the same retrieval.

Fix: Add this instruction to your decomposition prompt: "Ensure each sub-question targets distinctly different information. If two sub-questions would retrieve the same documents, merge them into one."

You can also add a post-decomposition deduplication step that embeds the sub-questions and removes any with cosine similarity above 0.92.

---

Problem: Sequential sub-questions don't use dependency context effectively.

Sometimes the sequential sub-question still retrieves off-target documents because the dependency answer isn't integrated into the retrieval query well.

Fix: In retrieve_for_subquestion, be more deliberate about query construction. Instead of appending raw dependency answers, extract key entities and facts:

# Instead of this:
retrieval_query = f"{sq.question}\n\nContext: {dep_answer}"

# Do this — extract entities first:
extraction_response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{
        "role": "user",
        "content": f"Extract the key named entities, dates, and facts from this text as a comma-separated list:\n{dep_answer}"
    }],
    temperature=0
)
key_facts = extraction_response.choices[0].message.content
retrieval_query = f"{sq.question} [relevant entities: {key_facts}]"

---

Problem: Synthesis is just listing sub-answers instead of integrating them.

This is a prompt engineering issue. Your synthesis prompt needs to explicitly discourage list-style output for certain query types.

Fix: Add query-type-specific synthesis instructions:

synthesis_style = {
    "comparative": "Write a comparative analysis with explicit before/after or entity-vs-entity structure.",
    "multi_hop": "Write a narrative that traces the chain of causation or events logically.",
    "aggregative": "Write a structured summary with clear categories or themes."
}.get(complexity_type, "Write a comprehensive, integrated answer.")

---

Problem: Latency is unacceptably high for complex queries.

Decomposition adds at least 2-3 extra LLM calls (classify, decompose, synthesize) plus parallel retrieval. In the worst case, a multi-hop query with 5 sub-questions might add 3-4 seconds of wall time.

Mitigations:

• Use gpt-4o-mini for classification and decomposition (it's 10x cheaper and 2x faster for structured tasks)
• Cache decomposition results for identical or near-identical queries — see Implementing LLM Response Caching with Redis: Semantic Deduplication, TTL Strategies, and Cache Invalidation Patterns for semantic caching strategies
• Set a maximum sub-question count (4-5 is usually sufficient) and enforce it in your prompt
• For time-sensitive applications, stream the synthesis response while retrieval is completing — Building Streaming AI Interfaces: Real-Time Response Delivery covers this pattern

Warning: Be careful about caching decomposition outputs too aggressively. The same question phrased differently might warrant different decompositions if the documents in your index have changed. Use short TTLs (minutes to hours) for decomposition caches, or tie cache invalidation to index updates.

---

Problem: The system incorrectly decomposes simple questions.

Your classifier has a false positive rate — it's marking simple questions as complex. This leads to unnecessary decomposition overhead.

Diagnose: Add logging of every classification decision. After a week, review the false positives. Common patterns:

• Questions with "and" that are actually simple ("What are the terms and conditions?")
• Questions with multiple clauses that are still single-retrieval ("How long has the company been operating and where is it based?")

Fix: Add explicit negative examples to your classification prompt, or fine-tune a small classifier on your logged data. Also consider setting a token threshold — questions under ~20 tokens rarely benefit from decomposition.

---

Evaluation: How Do You Know It's Working?

Implementing this without measurement is guesswork. You need to evaluate both retrieval quality per sub-question and synthesis quality on the final answer.

For each sub-question, track:

• Chunk relevance score: Average retrieval score from your vector store
• Sub-answer confidence: The self-reported confidence from the answer step
• Coverage: Whether the answer actually addresses the sub-question (you can use an LLM judge for this)

For the final answer, use standard RAG evaluation metrics:

• Faithfulness: Does the answer stick to what's in the retrieved context?
• Answer relevance: Does it address the original query?
• Information completeness: Did decomposition actually surface more relevant information vs. baseline single-retrieval?

That last metric is the key one. If your decomposition pipeline isn't consistently outperforming single-retrieval on complex queries, something is wrong with your decomposition quality or synthesis step. Consider pairing this with the evaluation framework from Testing and Evaluating LLM Applications: A Comprehensive Guide to Quality Assurance.

Key insight: The ROI on query decomposition is highly query-dependent. On a dataset of simple factual queries, it will perform worse than standard RAG (more latency, more cost, same or worse answers). The gain only materializes on genuinely complex, multi-part questions. Segment your eval set by query complexity before drawing conclusions.

---

When to Use Query Decomposition vs. Alternatives

Query decomposition isn't always the right tool. Here's how it compares:

Hybrid search with better chunking — If your retrieval is missing information because chunks are too large or document structure is poor, fix your chunking strategy before adding decomposition complexity. Many "complex query" failures are actually chunking failures.

HyDE (Hypothetical Document Embeddings) — For comparative queries where you know what the answer should look like, HyDE can improve single-shot retrieval without the full decomposition overhead.

Knowledge graph traversal — For multi-hop queries that follow entity relationships (e.g., "What did the CEO who replaced the founder do about the supply chain issues?"), a knowledge graph approach can be more reliable than sequential sub-question decomposition. This is covered in detail in Building a Knowledge Graph-Augmented RAG System.

Reranking — Sometimes decomposition isn't needed; your retrieval is finding the right documents but ranking them poorly. Adding a reranking layer is lower overhead and worth trying first.

The general rule: try retrieval quality improvements (chunking, hybrid search, reranking) before adding architectural complexity like decomposition. Decomposition shines specifically when the query has genuinely independent information needs that can't be satisfied with a single retrieval.

---

Summary & Next Steps

You've built a complete query decomposition system with four production-ready stages: complexity classification, dependency-aware sub-question generation, parallel retrieval with topological ordering, and answer synthesis. The key ideas to carry forward:

• Not every query needs decomposition. Build the classifier and use it; the fast path matters for cost and latency.
• Dependency ordering is not optional. Multi-hop queries fail silently if you ignore sequential dependencies — you get answers that don't incorporate what was learned in prior steps.
• Synthesis quality determines user experience. The best decomposition in the world is wasted if your synthesis step just concatenates answers. Invest in your synthesis prompt.
• Measure the right thing. Compare decomposed complex queries against single-retrieval on the same complex queries, not against each other on a mixed dataset.

From here, consider extending this system in a few directions:

• Query reformulation — After synthesis, use the full answer to reformulate and retrieve once more. Sometimes synthesis reveals gaps.
• Iterative decomposition — If a sub-question answer has very low confidence, automatically generate follow-up sub-questions to close the gap
• User-facing transparency — Some applications benefit from showing users the decomposition ("I'm looking at three things to answer this..."). This is especially useful for enterprise analytics tools where trust and auditability matter.
• Persistent memory integration — For conversational applications, the sub-question answers from a previous turn can seed the context for the next, dramatically improving follow-up query handling. Implementing Structured Agent Memory covers this pattern in depth.