RAG systems fail in predictable ways — and most developers spend hours guessing when they should be diagnosing. This lesson breaks down the five core retrieval failure modes, shows you how to distinguish them from generation failures, and gives you concrete diagnostic tools to find and fix the root cause every time.
You've built a RAG system. You've loaded your documents, created embeddings, stood up a vector database, and wired everything to an LLM. You test it with a few queries and it works beautifully. Then you put it in front of real users and the wheels start coming off. The system confidently answers a question about Q3 revenue using a document from two years ago. It returns three nearly identical chunks about the same topic while completely missing the one paragraph that actually answers the question. It gives a perfect response to "what is our refund policy" but returns garbage when someone asks "can I get my money back?"
If any of that sounds familiar, you're not dealing with a broken system — you're dealing with retrieval failure modes, and they're almost universal in RAG pipelines that haven't been deliberately hardened against them. The good news is that most failures fall into a small number of recognizable categories, and each category has a specific diagnostic approach and set of fixes.
By the end of this lesson, you'll be able to identify the root cause of a bad RAG response, trace it back to the specific failure mode responsible, and apply targeted remediation rather than guessing. This is the difference between productive debugging and randomly tweaking parameters and hoping things improve.
What you'll learn:
You should have a basic working understanding of how RAG pipelines are structured — documents are chunked, embedded into vectors, stored in a vector database, and retrieved by comparing query embeddings to stored vectors. If you haven't built one yet, work through Building Your First RAG Pipeline first. Familiarity with Python is helpful for reading the code examples, but not strictly required.
Before we get into specific failure modes, you need to internalize one critical principle: a bad final answer might not be a retrieval problem at all. RAG systems have two moving parts — retrieval and generation — and they fail in completely different ways.
When an LLM produces a wrong answer, many developers immediately start tweaking their embeddings or changing their similarity threshold. But sometimes the retrieval was perfect, and the LLM simply misread, hallucinated over, or ignored the context it was given. Misdiagnosing this wastes enormous time.
Here's the first thing to add to any RAG system you're debugging: log your retrieved chunks before they hit the LLM. This single step makes the boundary between retrieval and generation visible.
def rag_query(query: str, retriever, llm, k: int = 5) -> dict:
# Step 1: Retrieve
retrieved_chunks = retriever.get_relevant_documents(query)
# Step 2: Log what was retrieved — ALWAYS do this during debugging
print(f"\n--- Retrieved {len(retrieved_chunks)} chunks for query: '{query}' ---")
for i, chunk in enumerate(retrieved_chunks):
print(f"\n[Chunk {i+1}] Source: {chunk.metadata.get('source', 'unknown')}")
print(f"Score: {chunk.metadata.get('score', 'N/A')}")
print(f"Content preview: {chunk.page_content[:200]}...")
# Step 3: Generate
context = "\n\n".join([c.page_content for c in retrieved_chunks])
response = llm.invoke(f"Context:\n{context}\n\nQuestion: {query}")
return {
"answer": response,
"retrieved_chunks": retrieved_chunks,
"query": query
}
Now you can manually inspect what was actually retrieved and ask two separate questions: "Did we get the right chunks?" and "Did the LLM do something sensible with them?" These questions have different answers and different remedies.
Key insight: Always examine your retrieved chunks manually before you start tweaking anything. At least 40% of the time, what looks like a generation problem is actually perfect retrieval paired with a poorly structured prompt, and vice versa.
This is the most common retrieval failure and the easiest to misunderstand. It happens when the meaning of a user's query and the meaning of the relevant document chunk don't overlap enough in embedding space for the similarity search to surface the right content.
Think about what's actually happening under the hood. When you call an embedding model, it converts text into a point in a high-dimensional vector space — the idea being that semantically similar text lands near each other. But "near" is defined by the patterns the embedding model was trained on. If your users use different vocabulary than your documents, the relevant chunk might sit frustratingly far away from the query vector.
A classic example: your internal documentation always calls it "order cancellation policy," but a customer asks "can I return something I bought?" The words don't overlap, and depending on your embedding model, the semantic similarity might not be strong enough to bridge that gap.
To diagnose this, probe your retriever directly with multiple phrasings of the same question:
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
import numpy as np
def probe_semantic_coverage(vectorstore, question_variants: list[str], k: int = 3):
"""
Test whether different phrasings of the same question retrieve the same chunks.
Wide variance in results → semantic mismatch failure mode.
"""
results_by_variant = {}
for variant in question_variants:
docs = vectorstore.similarity_search_with_score(variant, k=k)
results_by_variant[variant] = [
{"content": doc.page_content[:150], "score": score}
for doc, score in docs
]
# Print comparison
for variant, results in results_by_variant.items():
print(f"\nQuery: '{variant}'")
for r in results:
print(f" Score: {r['score']:.4f} | {r['content'][:100]}...")
return results_by_variant
# Test it
variants = [
"can I return something I bought?",
"what is your return policy?",
"order cancellation policy",
"how do I get a refund?",
"I want my money back"
]
probe_semantic_coverage(vectorstore, variants)
If the top retrieved chunks change dramatically across these variants, or the scores drop sharply on the more colloquial phrasings, you have a semantic mismatch problem.
Remediation options in order of effort:
This failure mode is sneaky because your system can retrieve exactly the right location in the document and still return useless content. It happens when the answer to a question spans a chunk boundary, meaning the crucial sentence or data point ends up split across two chunks and neither chunk alone contains a complete answer.
Consider a 10-K financial document. A question like "What were the primary risk factors cited for the revenue decline in the Americas segment?" might require context that spans multiple paragraphs — a topic sentence in one chunk, the explanation two paragraphs later, and the quantitative support in yet another. Fixed-size chunking with no regard for document structure slices right through this.
You can spot this failure mode when retrieved chunks look like they're almost right — they're clearly from the correct section of the document, but they're missing the beginning or end of the relevant passage.
def inspect_chunk_boundaries(vectorstore, query: str, k: int = 5):
"""
Retrieve chunks and show their source positions to spot boundary issues.
If chunks are clustered in the same source with adjacent chunk_ids,
you likely have a boundary-splitting problem.
"""
docs = vectorstore.similarity_search_with_score(query, k=k)
for i, (doc, score) in enumerate(docs):
meta = doc.metadata
print(f"\nChunk {i+1} | Score: {score:.4f}")
print(f" Source: {meta.get('source', '?')}")
print(f" Chunk index: {meta.get('chunk_index', '?')}")
print(f" Char start: {meta.get('start_char', '?')}")
print(f" Length: {len(doc.page_content)} chars")
print(f" First 100 chars: {doc.page_content[:100]}")
print(f" Last 100 chars: {doc.page_content[-100:]}")
If you see chunks that end mid-sentence, start with lowercase letters or conjunctions ("and the primary reason was..."), or have adjacent chunk indices from the same source all appearing in the top-k, chunk boundaries are your problem.
Tip: Storing chunk metadata — specifically
source,chunk_index, and character offsets — at indexing time costs almost nothing and pays enormous dividends during debugging. Make this a standard practice from day one.
Remediation options:
The Chunking Strategies: How to Split Documents for Better Retrieval lesson walks through these approaches with hands-on examples.
Your vector index might contain things it shouldn't: headers, footers, page numbers, boilerplate legal text that repeats identically across hundreds of documents, navigation menus extracted from HTML, or auto-generated table-of-contents entries. These chunks embed just fine and often match queries reasonably well — but they contain no actual useful information.
This failure mode is particularly cruel because it steals slots from your top-k results. If you retrieve 5 chunks and 3 of them are boilerplate ("This document is confidential and intended only for the named recipient..."), you've effectively cut your useful context window by 60%.
You can catch this by auditing a random sample of your index:
import random
def audit_index_sample(vectorstore, sample_size: int = 50):
"""
Pull random documents from the index and categorize them.
Look for patterns: short fragments, repeated content, boilerplate.
"""
# This approach depends on your vectorstore's API
# Here shown for Chroma
collection = vectorstore._collection
all_docs = collection.get(limit=sample_size, include=["documents", "metadatas"])
short_chunks = []
potential_boilerplate = []
for doc, meta in zip(all_docs["documents"], all_docs["metadatas"]):
word_count = len(doc.split())
if word_count < 20:
short_chunks.append({"content": doc, "meta": meta})
# Flag chunks that look like headers/footers
if doc.strip().endswith("...") or doc.count("\n") > word_count * 0.3:
potential_boilerplate.append({"content": doc, "meta": meta})
print(f"Sample size: {sample_size}")
print(f"Very short chunks (<20 words): {len(short_chunks)}")
print(f"Potential boilerplate/structural: {len(potential_boilerplate)}")
print("\n--- Sample short chunks ---")
for item in short_chunks[:5]:
print(f" '{item['content']}'")
return short_chunks, potential_boilerplate
Warning: Index pollution compounds over time. If your system ingests new documents automatically, boilerplate patterns accumulate silently. A clean index at launch becomes a polluted one six months later. Build filtering logic into your ingestion pipeline, not as an afterthought.
Remediation options:
This failure mode has two opposite presentations, which makes it confusing to diagnose.
Too greedy (low threshold / no threshold): Your system returns chunks even when they have very low similarity to the query. The retriever is asked for k=5 results and it returns 5 — even if the 4th and 5th results have similarity scores of 0.42 when your good results cluster around 0.85. These low-quality stragglers dilute the context passed to the LLM and can actively mislead it.
Too strict (high threshold): Your system filters out results below a certain similarity score, and legitimate answers get cut because they express the relevant content in unexpected vocabulary. The LLM receives no context, falls back on its training data, and hallucinates.
To diagnose this, plot the score distribution for a representative set of queries:
import statistics
def analyze_score_distribution(vectorstore, test_queries: list[str], k: int = 10):
"""
For each test query, retrieve k results and inspect score distribution.
Look for: bimodal distributions, unexpectedly low top-k scores,
or large gaps between the "good" and "filler" results.
"""
all_scores = []
for query in test_queries:
docs_with_scores = vectorstore.similarity_search_with_score(query, k=k)
scores = [score for _, score in docs_with_scores]
print(f"\nQuery: '{query[:60]}...'")
print(f" Scores: {[f'{s:.3f}' for s in scores]}")
print(f" Top score: {max(scores):.3f} | Bottom score: {min(scores):.3f}")
print(f" Gap (top vs bottom): {max(scores) - min(scores):.3f}")
all_scores.extend(scores)
print(f"\nOverall statistics across all queries:")
print(f" Mean score: {statistics.mean(all_scores):.3f}")
print(f" Stdev: {statistics.stdev(all_scores):.3f}")
print(f" Proportion of scores below 0.5: "
f"{sum(1 for s in all_scores if s < 0.5) / len(all_scores):.1%}")
Look for a large gap between the top results and the bottom results in each query's score list. A gap of more than 0.2 often signals that the bottom results are noise.
Note: Similarity scores are not standardized across different vector databases or embedding models. A score of 0.7 might be excellent in one system and mediocre in another. Always calibrate your thresholds empirically against your specific setup rather than copying thresholds from tutorials.
Remediation options:
This is the most subtle failure mode and the hardest to fix. It happens when the queries your users ask are systematically different in structure, style, or scope from the documents in your index — and no amount of retrieval tuning fully bridges that gap.
The classic case: your documents are dense, technical reference material written in formal prose, and your users ask short, conversational questions. The embedding model tries its best, but there's a fundamental mismatch in the kind of text being compared. A short query like "memory leak?" doesn't embed into the same neighborhood as a five-paragraph explanation of heap allocation failures, even though one is clearly the answer to the other.
A subtler variant: your users ask questions that require synthesizing across multiple documents ("compare our Q2 and Q3 margins by region") but your retrieval returns individual chunks, none of which alone contains a complete answer. No individual chunk will have a high similarity score for a comparative question.
Diagnosing this requires looking at query patterns across many failures, not just individual cases:
def categorize_query_failures(failed_queries: list[dict]) -> dict:
"""
Classify failed queries by likely failure category.
failed_queries: list of {"query": str, "retrieved_chunks": list, "top_score": float}
"""
categories = {
"low_confidence": [], # top score < 0.6
"multi_hop_required": [], # query contains comparison/aggregation language
"short_query": [], # query is fewer than 5 words
"temporal": [], # query contains time references
}
multi_hop_signals = ["compare", "difference between", "vs", "both",
"and also", "summarize across", "how many total"]
temporal_signals = ["last year", "this quarter", "recently", "current",
"latest", "previous", "before", "after"]
for item in failed_queries:
query = item["query"].lower()
if item["top_score"] < 0.6:
categories["low_confidence"].append(item)
if any(signal in query for signal in multi_hop_signals):
categories["multi_hop_required"].append(item)
if len(query.split()) < 5:
categories["short_query"].append(item)
if any(signal in query for signal in temporal_signals):
categories["temporal"].append(item)
for cat, items in categories.items():
print(f"{cat}: {len(items)} queries")
return categories
If you find that failures cluster around multi-hop queries, that points toward architectural solutions like Agentic RAG: Building Self-Correcting Retrieval Pipelines That Query, Reflect, and Retry or Query Routing in RAG: How to Direct Questions to the Right Data Source or Retrieval Strategy. If failures cluster around short, ambiguous queries, query expansion is your lever.
Now that you understand the five failure modes, here's a decision tree you can work through whenever a RAG system returns bad results:
Step 1: Is this a retrieval problem or a generation problem? → Log the retrieved chunks. Read them yourself. Could a human answer the question from these chunks? If yes → generation problem (check your prompt). If no → retrieval problem.
Step 2: Were the right chunks retrieved at all? → Search for the answer manually in your document corpus. Does it exist? If it doesn't exist, that's a data gap, not a retrieval failure. If it exists, continue.
Step 3: What were the similarity scores for the retrieved chunks? → High scores but wrong content → semantic mismatch or index pollution. → Correct-looking content but low scores → threshold problem or vocabulary gap. → No results above threshold → threshold too strict, or severe vocabulary mismatch.
Step 4: Do the retrieved chunks look fragmented? → Chunks ending mid-sentence, starting with conjunctions, or containing only part of a relevant passage → chunk boundary problem.
Step 5: Do multiple phrasings of the same question return very different results? → Yes → semantic mismatch / vocabulary gap. Try query expansion.
Step 6: Does the failure type cluster with certain query patterns? → Comparative, multi-hop, or temporal queries failing systematically → query-document distribution mismatch requiring architectural changes.
For this exercise, you'll deliberately trigger and diagnose retrieval failures using a small document corpus.
Setup: Take any 10–15 page PDF you have access to (a company report, product documentation, or long article). Ingest it with a simple fixed-size chunker at 500 characters with no overlap. Create a Chroma vector store with OpenAI embeddings.
Task 1 — Trigger a chunk boundary failure: Find a question whose answer spans two paragraphs in the source document. Ask it and inspect the retrieved chunks. Verify that you can see the answer split across chunk boundaries. Then re-ingest with 100-character overlap and compare results.
Task 2 — Trigger a semantic mismatch:
Write five paraphrases of one question, ranging from formal document language to casual colloquial phrasing. Use the probe_semantic_coverage function from earlier. Measure how much the top retrieved chunk changes across variants.
Task 3 — Audit for index pollution:
Use the audit_index_sample function on your index. Count chunks under 30 words. Identify any repeated structural text. Re-ingest after adding a minimum length filter and compare the result quality on your hardest queries.
Document your findings: For each task, write down which failure mode was triggered, how you diagnosed it, what you changed, and whether the fix worked. This process of deliberate diagnosis → targeted fix → measured improvement is the core skill this lesson is building.
"My similarity scores are all between 0.95 and 0.99 but results are still bad." This happens when cosine similarity is computed on normalized vectors and you're using a model that collapses many different meanings into similar vector regions. Very high scores don't guarantee relevance — they just guarantee geometric proximity. Switch to a more capable embedding model or add reranking as a second-pass filter.
"Adding more chunks (increasing k) makes things worse." More chunks mean more noise in the context. If your precision is low (many retrieved chunks are irrelevant), increasing k makes the signal-to-noise ratio worse, not better. Focus on improving precision before increasing recall. Contextual Compression in RAG: Filtering and Compressing Retrieved Chunks Before Passing to the LLM addresses exactly this problem.
"The system works great for some users and terribly for others." This is a strong signal of query-document distribution mismatch. The users it works for are probably using vocabulary close to your document language. Analyze the failing queries for vocabulary patterns and consider query expansion or hybrid search for those patterns.
"Retrieval looks right but the final answer is still wrong." Go back to your prompt. Read the Prompt Engineering for RAG: How to Structure System Prompts That Ground LLM Responses in Retrieved Context lesson. The most common cause of this pattern is a prompt that allows the LLM to use its prior knowledge when context doesn't perfectly match the question — it should be instructed to say "I don't know" instead of hallucinating.
Warning: Don't treat retrieval failure modes as one-time problems to fix at launch. User queries evolve, documents change, and new edge cases emerge continuously. Build logging and monitoring into your production system from the start. Evaluating RAG Systems: Precision, Recall, and Faithfulness gives you the evaluation framework to track these failures over time.
RAG retrieval failures are frustrating precisely because they're invisible without deliberate instrumentation. The system appears to work — it always returns something — but that something is wrong in ways that are hard to see without logging and manual inspection.
The five failure modes you've learned to recognize are:
The diagnostic workflow is always the same: separate retrieval from generation, then systematically rule out failure modes starting with the most visible (score inspection) and working toward the most structural (distribution mismatch).
From here, the logical next steps are to deepen your knowledge in the areas where your system needs the most work. If you're seeing vocabulary gaps, explore Query Expansion in RAG and Hybrid Search. If your failures cluster around complex, multi-part questions, look at Agentic RAG. And if you want to build ongoing visibility into how your system is performing across all query types, make Evaluating RAG Systems: Precision, Recall, and Faithfulness your next stop.