LLMs are powerful reasoners, but they can't answer questions about your data — unless you give them a way to look things up. This lesson explains exactly how RAG works, walks you through building a real pipeline in Python, and shows you when to use it versus other approaches.
Imagine you hire a brilliant consultant who has read every business book, academic paper, and industry report published before 2023. They're articulate, fast, and remarkably good at synthesizing ideas. But when you ask them about your company's Q3 earnings, the proprietary customer churn analysis your team just finished, or a news event from last week — they're lost. They'll either admit ignorance or, worse, confidently make something up. That's the fundamental problem with large language models (LLMs) used in isolation: they're extraordinarily capable reasoners, but their knowledge is frozen at a point in time and entirely disconnected from your data.
Retrieval-Augmented Generation, or RAG, is the architecture that solves this. It gives your LLM a library card — a way to look things up before it answers, pulling relevant information from your documents, databases, or knowledge bases and using that information to ground its response. The result is an AI assistant that can reason about your specific data, stay current, and — critically — tell you where its answers came from.
By the end of this lesson, you'll understand exactly how RAG works under the hood, when it's the right tool to reach for, and how to build a simple working RAG pipeline from scratch. You won't just know the buzzword — you'll be able to use it.
What you'll learn:
To understand why RAG matters, you need to understand two hard limits that all LLMs share.
Limit 1: The training cutoff. Every LLM is trained on a snapshot of data collected up to a certain date. GPT-4's knowledge, for example, doesn't include anything published after its training cutoff. If your users ask about recent regulatory changes, last quarter's results, or a product that launched six months ago, the model simply doesn't know.
Limit 2: The context window. LLMs read and generate text inside a "context window" — think of it as the model's working memory. Modern models have impressively large windows (sometimes over 100,000 tokens), but they're still finite. You can't just paste your entire 500-page internal wiki into every prompt and hope for the best.
A natural first instinct is to solve this through fine-tuning — taking a pretrained model and continuing to train it on your own data. Fine-tuning does work well for teaching a model how to behave (tone, format, domain-specific reasoning patterns), but it's a poor fit for teaching a model facts. Facts need to be updated constantly, and retraining a model every time your documentation changes is prohibitively expensive and slow. Fine-tuning also doesn't make the model reliably cite sources, which matters enormously in professional settings.
RAG sidesteps both problems. Instead of baking knowledge into the model's weights, it retrieves knowledge at inference time — meaning every time a user asks a question. The model doesn't need to "remember" your documents; it just needs to be able to read and reason about them when they're placed in front of it.
Every RAG system, from the simplest prototype to a production-grade enterprise search tool, follows the same three-stage architecture. Understanding these stages deeply is what separates someone who can copy-paste a tutorial from someone who can actually design and debug these systems.
Before your system can retrieve anything, it needs to know what's available. Indexing is the offline process of taking your source documents — PDFs, web pages, database records, support tickets, whatever — and converting them into a form that can be searched efficiently.
This process involves three sub-steps:
Chunking: You break your documents into smaller pieces. A 40-page policy document gets split into paragraphs or sections of, say, 500 words each. Why? Because you want to retrieve the relevant portion of a document, not an entire 40-page file. Chunk size is a tuning decision that matters — too small and you lose context; too large and you return too much noise.
Embedding: Each chunk is passed through an embedding model, which converts text into a list of numbers called a vector. A vector is a mathematical representation of meaning. The magic of good embedding models is that chunks with similar meaning end up with vectors that are close together in mathematical space — even if they use completely different words. The phrase "the contract was terminated" and "the agreement was cancelled" will have very similar vectors.
Storing: Those vectors get stored in a vector database — a specialized database built to answer the question "which vectors are closest to this query vector?" efficiently. Popular options include Pinecone, Weaviate, Chroma, and pgvector (an extension for PostgreSQL). The original text of each chunk is stored alongside its vector so you can retrieve it later.
When a user asks a question, the retrieval stage kicks in. Here's what happens:
This is faster than it sounds. Modern vector databases can search millions of stored chunks in milliseconds using algorithms like approximate nearest neighbor search. The key insight is that you're not doing keyword matching — you're doing semantic matching. A question about "how do I cancel my subscription" will retrieve chunks about "account termination procedures" even if the word "cancel" never appears in the document.
Now you have both the user's question and a handful of relevant document chunks. You combine them into a single prompt — usually called the augmented prompt — and send that to the LLM.
A typical augmented prompt looks something like this:
You are a helpful assistant. Answer the user's question using only the
information provided in the context below. If the answer is not in the
context, say so.
CONTEXT:
[Chunk 1 text here]
[Chunk 2 text here]
[Chunk 3 text here]
USER QUESTION:
What is the process for requesting a contract amendment?
The LLM reads the context you've provided, reasons over it, and generates an answer. Because the relevant information is sitting right there in the prompt, the model doesn't need to "remember" anything from training — it just needs to read and synthesize. And because you know exactly which chunks were used, you can display citations to the user.
Let's make this concrete. We'll build a minimal RAG pipeline that lets you ask questions about a small set of text documents. We'll use chromadb as our local vector database, sentence-transformers for embedding, and the OpenAI API for generation.
First, install the dependencies:
pip install chromadb sentence-transformers openai
import chromadb
from sentence_transformers import SentenceTransformer
# Initialize a local Chroma vector database
client = chromadb.Client()
collection = client.create_collection("company_policies")
# Initialize an embedding model
embedder = SentenceTransformer("all-MiniLM-L6-v2")
# Simulated document chunks — in production, you'd load these from files
documents = [
"Employees may request up to 15 days of paid time off per calendar year. "
"Requests must be submitted at least two weeks in advance via the HR portal.",
"Contract amendments must be reviewed by the Legal department and signed "
"by both parties. The amendment process typically takes 5-10 business days.",
"The company reimbursement policy covers travel, lodging, and meals up to "
"$75 per day. Receipts must be submitted within 30 days of the expense.",
"Remote work arrangements are available to employees who have completed "
"at least six months of employment. Approval is granted by the direct manager.",
]
# Embed all documents and store them
embeddings = embedder.encode(documents).tolist()
collection.add(
documents=documents,
embeddings=embeddings,
ids=[f"doc_{i}" for i in range(len(documents))]
)
print(f"Indexed {len(documents)} document chunks.")
def retrieve(query: str, top_k: int = 2) -> list[str]:
query_embedding = embedder.encode([query]).tolist()
results = collection.query(
query_embeddings=query_embedding,
n_results=top_k
)
# results["documents"] is a list of lists; flatten the top result set
return results["documents"][0]
# Test retrieval
user_question = "How do I get reimbursed for a business trip?"
retrieved_chunks = retrieve(user_question)
print("Retrieved chunks:")
for i, chunk in enumerate(retrieved_chunks, 1):
print(f"\n[{i}] {chunk}")
You should see the reimbursement policy chunk returned as the top result, even though the question used the word "reimbursed" and the document used "reimbursement policy."
from openai import OpenAI
openai_client = OpenAI(api_key="your-api-key-here")
def answer_with_rag(question: str) -> str:
# Retrieve relevant context
chunks = retrieve(question)
context = "\n\n".join(chunks)
# Build the augmented prompt
augmented_prompt = f"""You are a helpful HR assistant. Answer the employee's
question using only the information provided in the context below. If the
answer is not covered in the context, say "I don't have information about that."
CONTEXT:
{context}
QUESTION:
{question}
ANSWER:"""
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": augmented_prompt}],
temperature=0 # Zero temperature for factual Q&A — more on this below
)
return response.choices[0].message.content
# Ask a question
answer = answer_with_rag("What is the daily meal allowance for business travel?")
print(answer)
The model should answer with "$75 per day" and mention the receipt submission requirement — information it pulled directly from your documents, not from its training data.
Why temperature=0? Temperature controls how "creative" the model is in choosing its next word. For factual retrieval tasks, you want deterministic, grounded answers. Setting temperature to 0 tells the model to always pick the highest-probability next token, reducing hallucination risk. For creative writing or brainstorming tasks, you'd want a higher value.
RAG is a powerful tool, but it's not the right tool for every problem. Here's a practical framework for deciding.
Use RAG when:
Don't reach for RAG when:
Watch out for this misconception: RAG doesn't eliminate hallucination. The model can still reason incorrectly over the retrieved context, misinterpret passages, or hallucinate when the retrieved chunks are only partially relevant. RAG reduces hallucination by grounding answers in real text — it doesn't eliminate the fundamental behavior.
Now it's your turn to extend the pipeline you built above.
The challenge: Your HR assistant currently answers from a flat list of documents. Make it smarter in two ways.
Part 1 — Add a new domain: Add five more document chunks from a different domain — say, a simplified IT help desk policy covering topics like password resets, software installation requests, and VPN access. Re-index the full collection, then test queries from both domains. Verify that the retriever correctly distinguishes HR questions from IT questions.
Part 2 — Surface citations: Modify the retrieve function to also return the chunk IDs or a document label (e.g., "HR Policy - Section 3: Reimbursement"). Update the augmented prompt to ask the model to reference which source it used in its answer. Print both the answer and the source labels for every response.
Stretch goal: Add a simple fallback. If the highest similarity score returned by Chroma falls below a threshold (e.g., 0.4), have the system respond with "I couldn't find relevant information about that in our knowledge base" instead of passing low-quality context to the LLM. This prevents the model from hallucinating answers when the retrieval step fails.
Mistake 1: Chunks that are too large or too small
If your chunks are 3,000 words each, you'll retrieve entire sections of documents and flood the context window with irrelevant text. If they're 50 words each, you'll retrieve snippets with no surrounding context and the model won't have enough to work with. A good starting point is 300–500 words per chunk with a 50-word overlap between adjacent chunks so ideas don't get cut off at the boundary.
Mistake 2: Using a different embedding model at query time than at index time
This is a silent, catastrophic failure. If you index your documents with all-MiniLM-L6-v2 and then accidentally query with a different model, the vectors are in different mathematical spaces — similarity scores become meaningless. Always use the same model for both indexing and retrieval, and make this explicit in your configuration.
Mistake 3: Ignoring the quality of the source documents
RAG is a "garbage in, garbage out" system. If your source documents are poorly written, out of date, or contradictory, the model will faithfully surface that bad information. Curating and maintaining your knowledge base is not optional maintenance — it's core product work.
Mistake 4: Treating retrieved context as ground truth in the prompt
Don't write your prompt as "The following information is correct: [context]." The retrieved chunks are relevant candidates, not verified facts. A better framing is "Answer based on the following context" — this keeps the model's reasoning appropriately calibrated rather than forcing it to accept potentially flawed source material without question.
Mistake 5: Not evaluating retrieval and generation separately
When a RAG system gives a wrong answer, the bug could be in the retrieval (the wrong chunks were returned) or in the generation (the right chunks were returned but the model reasoned incorrectly over them). Log both the retrieved chunks and the final answer during testing. If the chunks are wrong, tune your chunking strategy, embedding model, or retrieval parameters. If the chunks are right but the answer is wrong, tune your prompt.
You now understand RAG from first principles. Here's the mental model to carry forward: a RAG system is a librarian (retrieval) paired with a synthesizer (generation). The librarian finds the relevant pages; the synthesizer reads them and writes a coherent answer. Neither works well without the other.
The three-stage pipeline — index, retrieve, generate — is the backbone of nearly every production RAG system, whether it's a simple internal chatbot or a sophisticated enterprise search platform. The core components are embeddings (which convert text to mathematical meaning), vector databases (which search that meaning efficiently), and augmented prompts (which give the LLM the context it needs to answer accurately).
Where to go next:
The best way to build intuition for this is to build something real. Pick a set of documents you actually work with — meeting notes, process documentation, a team wiki — and build a RAG pipeline around them. The edge cases and failure modes you'll encounter in real data will teach you more than any tutorial.