Query Routing in RAG: How to Direct Questions to the Right Data Source or Retrieval Strategy

Introduction

Imagine you've built a RAG (Retrieval-Augmented Generation) system for a mid-sized e-commerce company. You have a product catalog database, a customer support knowledge base, a returns policy document, and a real-time inventory API. A user types: "Can I return the wireless headphones I bought last Tuesday?" — and your system tries to search all four data sources simultaneously, gets confused by irrelevant results from the product catalog, and spits out a generic answer that doesn't mention the actual return deadline.

This is the problem query routing solves. Instead of blindly throwing every question at every data source and hoping something useful comes back, a well-designed RAG system figures out which data source — or which retrieval strategy — is actually appropriate for each specific question. It's the difference between a customer service rep who reads the right manual before answering versus one who reads all four manuals at random and then wings it.

By the end of this lesson, you'll understand exactly how query routing works, why it matters, and how to build one yourself using Python and a language model. You'll go from zero to a working router that can intelligently direct questions to the right data sources.

What you'll learn:

• What query routing is and why naive RAG systems fail without it
• The two main routing strategies: keyword/rule-based routing and LLM-based semantic routing
• How to design a routing schema that maps question types to data sources
• How to implement a functional query router in Python using an LLM
• How to combine routing with fallback strategies for resilience

---

Prerequisites

• Basic Python familiarity (you should know what a function and a dictionary are)
• A conceptual understanding of RAG — you know that it involves retrieving context and feeding it to an LLM to generate answers
• An OpenAI API key, or familiarity with another LLM provider (the code here uses OpenAI, but the concepts translate directly)
• pip installed so you can install Python packages

If you haven't worked with RAG at all yet, spend 20 minutes reading a RAG intro first — this lesson builds on that foundation.

---

What Query Routing Actually Is

Let's start from first principles. In a basic RAG system, the flow looks like this:

1. User asks a question
2. System searches a vector store (a database of text chunks stored as numerical embeddings) for relevant chunks
3. Those chunks get passed to an LLM along with the question
4. The LLM generates an answer

This works beautifully when you have one data source. The moment you have multiple sources — or multiple retrieval methods — you have a decision problem. Should you search the policy documents? The product database? Should you call a live API? Should you even retrieve anything, or can the LLM answer from general knowledge?

Query routing is the component in your pipeline that makes this decision. It sits between the user's question and the retrieval step, examines what the user is actually asking, and decides where to send that question — before any retrieval happens.

Think of it like a hospital triage nurse. When a patient walks in, the nurse doesn't immediately send everyone to surgery. They assess the situation: Is this a broken bone? Go to radiology. Chest pain? Cardiology. Minor cut? Walk-in clinic. The triage step saves enormous time and resources by routing cases to the right specialist upfront.

---

Why Naive Routing Fails (And What It Costs You)

Before building the solution, let's understand the failure modes clearly — because knowing why something breaks helps you design something that doesn't.

Problem 1: Retrieval noise degrades answer quality. If you search all your data sources for every question, you pull in irrelevant chunks. An LLM trying to answer "What's your refund policy?" doesn't need product specifications cluttering its context window. Those irrelevant chunks can distract the model, dilute the relevant signal, and produce worse answers.

Problem 2: Latency and cost spiral. Searching three vector stores and calling two APIs for every single query is expensive and slow. If 60% of your questions are about return policies and they all hit every data source, you're doing 5x the work you need to.

Problem 3: Some questions shouldn't retrieve at all. If a user asks "What's 15% of $240?", you don't need to search anything. Sending that to a retrieval system wastes time and might actually hurt the answer if the retrieved chunks confuse the model.

Problem 4: Different questions need different retrieval strategies. A question about a specific order number needs a structured database lookup, not semantic vector search. A question about a vague concept like "your best headphones for commuting" needs semantic similarity search, not a keyword match. Routing isn't just about which data source — it's also about how to retrieve.

---

The Two Main Routing Approaches

There are two broad families of routing strategies, and you'll often use them together.

Rule-Based Routing

Rule-based routing uses explicit logic — keywords, patterns, or simple classifiers — to make routing decisions. It's fast, deterministic, and cheap.

def simple_rule_router(query: str) -> str:
    query_lower = query.lower()
    
    # Check for order-specific questions
    if any(word in query_lower for word in ["order", "tracking", "shipment", "delivery"]):
        return "orders_database"
    
    # Check for return/refund questions
    if any(word in query_lower for word in ["return", "refund", "exchange", "money back"]):
        return "policy_documents"
    
    # Check for product questions
    if any(word in query_lower for word in ["product", "specs", "features", "review", "price"]):
        return "product_catalog"
    
    # Default fallback
    return "general_knowledge"

# Test it
print(simple_rule_router("How do I return my headphones?"))  
# → policy_documents

print(simple_rule_router("Where is my order #84521?"))       
# → orders_database

This is great for getting started. But notice the brittleness: what happens when someone asks "Can I get my money back for the broken speaker I ordered?" — that hits both "money back" (policy) and "ordered" (orders database). And what about "I'd like to send back the item I purchased" — the word "return" isn't there, but the intent is clearly about returns.

This is where LLM-based routing becomes valuable.

LLM-Based Semantic Routing

LLM-based routing uses a language model to understand the intent behind a question, not just its surface keywords. You give the LLM a description of your available data sources and ask it to classify where the question should go.

This handles paraphrasing, ambiguity, and complex multi-intent questions far better than keyword rules.

Tip: LLM-based routing adds latency (usually 200–500ms for a fast model). For production systems, use a small, fast model like GPT-4o-mini or a locally hosted classifier specifically for routing — not a full-scale model.

---

Designing Your Routing Schema

Before writing routing code, you need to define your schema — the structured description of what each data source contains and what kinds of questions belong there.

This is more important than it looks. A vague schema produces vague routing decisions. Be specific about what each source handles, and give examples of the kinds of questions it should receive.

Here's a schema for our e-commerce example:

ROUTING_SCHEMA = {
    "routes": [
        {
            "name": "policy_documents",
            "description": "Company policies including returns, refunds, warranties, "
                           "shipping timelines, and terms of service. Use this for "
                           "questions about what the company allows or how processes work.",
            "example_questions": [
                "What is your return policy?",
                "How long does a refund take?",
                "Is there a restocking fee?",
                "Can I exchange a gift?"
            ]
        },
        {
            "name": "orders_database",
            "description": "Real-time data about specific customer orders, including "
                           "order status, shipping tracking, delivery estimates, and "
                           "purchase history. Use this when the user references a "
                           "specific order or asks about the status of something they bought.",
            "example_questions": [
                "Where is my package?",
                "When will my order arrive?",
                "What did I order last month?",
                "Has my order shipped yet?"
            ]
        },
        {
            "name": "product_catalog",
            "description": "Product information including specifications, pricing, "
                           "availability, comparisons, and reviews. Use this for "
                           "questions about what products exist or their features.",
            "example_questions": [
                "What's the battery life on the XR-500 headphones?",
                "Do you sell waterproof speakers?",
                "What's the difference between the Pro and Standard model?"
            ]
        },
        {
            "name": "general_knowledge",
            "description": "Questions that don't require retrieving company-specific "
                           "information and can be answered from general knowledge. "
                           "Use this for calculations, general advice, or questions "
                           "about topics unrelated to company data.",
            "example_questions": [
                "What's the best way to clean headphones?",
                "How does Bluetooth 5.0 work?",
                "What are good headphones for running?"
            ]
        }
    ]
}

Notice how each route has a name, a detailed description, and example_questions. Those examples aren't just for documentation — you'll use them in your LLM prompt to help it make better decisions.

---

Building the LLM-Based Router

Now let's build the actual router. We'll use OpenAI's API with structured output to get a reliable, parseable routing decision.

First, install the dependency:

pip install openai

Now let's build the router:

import json
from openai import OpenAI

client = OpenAI()  # Uses OPENAI_API_KEY from environment

def build_routing_prompt(schema: dict) -> str:
    """Convert our schema into a clear prompt for the router LLM."""
    
    routes_description = ""
    for route in schema["routes"]:
        examples = "\n    - ".join(route["example_questions"])
        routes_description += f"""
Route: {route["name"]}
Description: {route["description"]}
Example questions:
    - {examples}

"""
    
    return f"""You are a query routing system for a customer service platform.
Your job is to analyze a user's question and decide which data source should handle it.

Available routes:
{routes_description}

Rules:
- Choose exactly ONE route that best matches the question's primary intent.
- If a question could fit multiple routes, choose the one that addresses the core need.
- Respond with valid JSON only, in this exact format:
{{"route": "route_name", "confidence": 0.0-1.0, "reasoning": "brief explanation"}}
"""

def route_query(query: str, schema: dict) -> dict:
    """
    Route a query to the appropriate data source.
    Returns a dict with 'route', 'confidence', and 'reasoning'.
    """
    system_prompt = build_routing_prompt(schema)
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Fast, cheap — perfect for routing
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Route this query: {query}"}
        ],
        temperature=0,  # We want deterministic routing decisions
        response_format={"type": "json_object"}
    )
    
    result = json.loads(response.choices[0].message.content)
    return result


# Test our router
test_queries = [
    "Can I return the wireless headphones I bought last Tuesday?",
    "Where is my order #84521?",
    "What's the difference between the HD-Pro and HD-Standard?",
    "How do I pair Bluetooth headphones with my laptop?",
    "I want to send back the item I purchased — it arrived broken"
]

for query in test_queries:
    result = route_query(query, ROUTING_SCHEMA)
    print(f"\nQuery: {query}")
    print(f"→ Route: {result['route']} (confidence: {result['confidence']:.0%})")
    print(f"→ Reasoning: {result['reasoning']}")

Running this should produce output like:

Query: Can I return the wireless headphones I bought last Tuesday?
→ Route: policy_documents (confidence: 90%)
→ Reasoning: The user wants to know if a return is possible — this requires policy information.

Query: Where is my order #84521?
→ Route: orders_database (confidence: 100%)
→ Reasoning: The user references a specific order number and wants status — requires real-time order data.

Query: I want to send back the item I purchased — it arrived broken
→ Route: policy_documents (confidence: 85%)
→ Reasoning: Despite mentioning a purchase, the core need is about initiating a return process.

Notice how the last example correctly routes to policy_documents even though it mentions an order — because the intent is about returning something, not tracking it. A keyword router would have been confused by "purchased."

---

Adding Multi-Route Support and Fallbacks

Some questions genuinely need multiple data sources. "Can I still return my order #84521 if it's already shipped?" needs both the order status and the return policy. Let's extend the router to handle this:

def route_query_multi(query: str, schema: dict, max_routes: int = 2) -> dict:
    """
    Route a query, potentially returning multiple routes for complex questions.
    """
    system_prompt = build_routing_prompt(schema)
    
    multi_prompt = system_prompt + f"""

For complex questions that genuinely require multiple data sources, you may return 
up to {max_routes} routes, ordered by priority.
Respond with valid JSON in this format:
{{"routes": ["primary_route", "secondary_route"], "confidence": 0.0-1.0, "reasoning": "explanation"}}
Only use multiple routes when truly necessary — most questions need just one.
"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": multi_prompt},
            {"role": "user", "content": f"Route this query: {query}"}
        ],
        temperature=0,
        response_format={"type": "json_object"}
    )
    
    result = json.loads(response.choices[0].message.content)
    
    # Normalize: if result has 'route' instead of 'routes', wrap it
    if "route" in result and "routes" not in result:
        result["routes"] = [result["route"]]
    
    return result


def execute_retrieval(routes: list, query: str) -> dict:
    """
    Stub function showing how you'd dispatch to actual retrievers.
    In a real system, each branch calls your actual data source.
    """
    results = {}
    
    for route in routes:
        if route == "policy_documents":
            # results[route] = your_vector_store.search(query, collection="policies")
            results[route] = f"[Retrieved policy chunks for: {query}]"
        
        elif route == "orders_database":
            # results[route] = your_db.query("SELECT * FROM orders WHERE ...")
            results[route] = f"[Retrieved order records for: {query}]"
        
        elif route == "product_catalog":
            # results[route] = your_catalog_api.search(query)
            results[route] = f"[Retrieved product data for: {query}]"
        
        elif route == "general_knowledge":
            results[route] = None  # No retrieval needed
    
    return results


# Test the full pipeline
complex_query = "Can I still return order #84521 if it's already been shipped?"
routing_result = route_query_multi(complex_query, ROUTING_SCHEMA)

print(f"Query: {complex_query}")
print(f"Routes: {routing_result['routes']}")
print(f"Reasoning: {routing_result['reasoning']}")

context = execute_retrieval(routing_result["routes"], complex_query)
print(f"Retrieved context from: {list(context.keys())}")

Warning: Multi-route retrieval increases latency and cost proportionally. Be conservative — only enable multi-routing for question types where you've identified genuine need. Start with single routing and add multi-routing only where users are actually getting bad answers.

---

Adding Confidence Thresholds and Fallbacks

A routing system without a fallback is fragile. What happens when the confidence is low? What if the LLM returns an invalid route name? You need graceful degradation.

VALID_ROUTES = {route["name"] for route in ROUTING_SCHEMA["routes"]}
DEFAULT_FALLBACK_ROUTE = "general_knowledge"
CONFIDENCE_THRESHOLD = 0.6

def safe_route_query(query: str, schema: dict) -> dict:
    """
    Route a query with validation and fallback handling.
    """
    try:
        result = route_query(query, schema)
        
        # Validate the returned route exists
        if result.get("route") not in VALID_ROUTES:
            print(f"Warning: Router returned unknown route '{result.get('route')}'. Using fallback.")
            result["route"] = DEFAULT_FALLBACK_ROUTE
            result["confidence"] = 0.0
            result["fallback_triggered"] = True
        
        # Check confidence threshold
        if result.get("confidence", 0) < CONFIDENCE_THRESHOLD:
            print(f"Warning: Low confidence ({result['confidence']:.0%}). Consider multi-source retrieval.")
            result["low_confidence"] = True
        
        return result
    
    except json.JSONDecodeError:
        # Router returned non-JSON output
        print("Error: Router returned invalid JSON. Using fallback.")
        return {
            "route": DEFAULT_FALLBACK_ROUTE,
            "confidence": 0.0,
            "reasoning": "Fallback due to routing error",
            "fallback_triggered": True
        }
    
    except Exception as e:
        print(f"Error in routing: {e}. Using fallback.")
        return {
            "route": DEFAULT_FALLBACK_ROUTE,
            "confidence": 0.0,
            "reasoning": "Fallback due to unexpected error",
            "fallback_triggered": True
        }

This pattern — try, validate, threshold check, fallback — is what separates a demo from something production-worthy.

---

Hands-On Exercise

Let's put everything together with a realistic scenario. You're building a RAG system for an HR department. They have three data sources:

1. HR Policies — documents about PTO, parental leave, performance reviews, code of conduct
2. Benefits Database — structured data about health plans, 401k details, dental/vision options
3. Org Chart API — real-time data about who reports to whom, team structures, manager contacts

Your task:

Step 1: Define a ROUTING_SCHEMA dictionary for this HR system. Write descriptions and at least three example questions for each route. Add a general_knowledge route as a fallback.

Step 2: Copy the route_query function from this lesson and run it against these five test queries:

• "How many PTO days do I get in my first year?"
• "Who is the VP of Engineering?"
• "What's the difference between the PPO and HMO health plans?"
• "Can I cash out unused vacation days?"
• "What does a typical performance review look like?"

Step 3: For each result, evaluate whether the routing decision makes sense. If any routing decision surprises you, adjust the schema description for that route and re-run.

Step 4: Add a sixth test query that you'd expect to be ambiguous — something that could reasonably go to two different routes. Check what the router decides, and whether the reasoning makes sense.

Stretch goal: Modify the schema to add a self_service_portal route for questions about how to actually do things in HR systems (submitting a PTO request, updating direct deposit, enrolling in benefits). Run all six queries again and see how the routing changes.

---

Common Mistakes & Troubleshooting

Mistake 1: Writing vague route descriptions. If your description for policy_documents just says "company policies," the LLM won't have enough signal to route confidently. Be specific about what kinds of policies and what kinds of questions. The more concrete the description, the better the routing.

Mistake 2: Using temperature > 0 for routing. Routing decisions should be deterministic. If you use temperature 0.7, the same question might route differently on different calls. Always use temperature=0 for routing.

Mistake 3: Not validating route names. LLMs occasionally hallucinate. If your schema has policy_documents but the LLM returns policies_db, your downstream code will break. Always validate against your known route list.

Mistake 4: Routing before cleaning the query. If the user types "idk where my thing is can you help??", the router will have a harder time. A simple preprocessing step — stripping excess punctuation, lowercasing, maybe expanding common abbreviations — improves routing accuracy noticeably.

Mistake 5: Forgetting that routing adds latency. Every LLM call takes time. If your router uses the same large model as your answer generation, you've doubled your base latency. Use the smallest, fastest model that achieves acceptable accuracy for routing. GPT-4o-mini, Claude Haiku, or a fine-tuned local classifier are all good options.

Mistake 6: Treating routing as all-or-nothing. If your confidence threshold is 0.6 and a result comes back at 0.55, don't just fail. Consider routing to two sources in parallel and letting the answer generation step reconcile. Uncertainty in routing is a signal, not a crisis.

---

Summary & Next Steps

You've now built a complete understanding of query routing in RAG systems. Let's recap what you've learned:

• Why routing matters: Without it, RAG systems waste resources, retrieve irrelevant context, and produce lower-quality answers as you scale to multiple data sources.
• Two routing approaches: Rule-based routing is fast and cheap but brittle. LLM-based semantic routing handles paraphrase and ambiguity far better.
• Schema design: Your routing descriptions and example questions directly determine routing quality — invest time in writing them well.
• Implementation: A routing LLM call uses temperature=0, structured JSON output, and a clear system prompt describing each available route.
• Production resilience: Always validate route names, set confidence thresholds, and implement fallbacks for unexpected failures.

Where to go next:

• Metadata filtering: Combine routing with metadata filters inside your vector store — route to a source, then filter by date, category, or user permissions within that source
• Query rewriting: Before routing, rewrite ambiguous queries into cleaner forms using an LLM — this significantly improves routing accuracy
• Evaluating your router: Build a labeled test set of 50–100 queries with expected routes and measure your router's accuracy. Track this as your schema evolves
• Hybrid retrieval strategies: Beyond choosing which source, route to how to retrieve — keyword search vs. semantic search vs. structured SQL depending on question type

The routing layer is one of the highest-leverage improvements you can make to a RAG system. A well-designed router doesn't just save compute — it fundamentally changes the quality ceiling of your answers.