Large language models are powerful but suffer from frozen knowledge and no access to your private data. This lesson explains exactly how Retrieval-Augmented Generation solves both problems — covering the full pipeline from document ingestion to vector search to grounded LLM generation, with working code and real design tradeoffs.
Imagine you've just deployed a customer-facing chatbot powered by GPT-4. It's impressive for about three days — until a user asks about your company's new refund policy (updated last Tuesday), or the specifics of a contract clause buried in a 200-page PDF your legal team uploaded to SharePoint. The model confidently answers... and it's completely wrong. Not because the model is bad. Because the model genuinely doesn't know. Its knowledge was frozen at a training cutoff months or years ago, and it has never seen your internal documents.
This is the problem Retrieval-Augmented Generation (RAG) was built to solve. Rather than relying solely on what a language model memorized during training, RAG systems dynamically pull relevant information from an external knowledge base at the moment a question is asked, then hand that information to the model so it can generate a grounded, accurate response. It's the difference between asking a colleague to answer from memory versus asking them to look it up first.
By the end of this lesson, you'll understand how RAG works from the ground up — the data flow, the architectural components, the tradeoffs — well enough to make real design decisions and begin building systems of your own.
What you'll learn:
This article is written for practitioners who are comfortable reading code snippets but are new to RAG and language model systems. You don't need prior machine learning experience. Familiarity with the concept of an API and basic Python syntax will help you follow the examples.
Large language models (LLMs) like GPT-4, Claude, or Llama are trained on enormous corpora of text — hundreds of billions of words scraped from books, websites, code repositories, and more. During training, the model learns statistical patterns in that text, which lets it do remarkable things: write code, explain concepts, summarize documents, answer questions.
But that training process has two hard constraints that matter enormously in practice.
First: a frozen knowledge cutoff. Training is expensive. You run it once (or periodically), and then the model is deployed. Whatever happened after the training data was collected, the model simply doesn't know. Ask a model trained in early 2024 about a law passed in late 2024, and you'll get either an admission of ignorance or, more dangerously, a confident guess.
Second: no access to private information. Your organization's internal wikis, HR policies, product documentation, Slack conversations, and contracts never made it into the training data. The model can't answer questions about them because it has never seen them.
The naive response to both problems is fine-tuning — retraining or adapting the model on your proprietary data. But fine-tuning is expensive, slow, doesn't update easily, and often doesn't actually improve factual recall the way you'd hope. The model might absorb the style of your documents without reliably learning their content. We explore this tradeoff in depth in Retrieval-Augmented Generation vs. Fine-Tuning: When to Use Each Approach and Why.
RAG takes a different approach entirely: rather than baking knowledge into the model's weights, it looks knowledge up at runtime and shows it to the model as part of the prompt. The model's job isn't to remember — it's to reason over what you hand it.
Key insight
RAG doesn't make the LLM smarter. It gives the LLM better information to work with. This distinction matters for how you debug and improve these systems.
Strip away all the complexity and a RAG system does exactly two things in sequence:
Here's what that looks like in the simplest possible pseudocode:
def answer_question(user_question, knowledge_base):
# Step 1: Retrieve
relevant_chunks = knowledge_base.search(user_question, top_k=5)
# Step 2: Generate
prompt = f"""
Use the following context to answer the question.
Context:
{relevant_chunks}
Question: {user_question}
Answer:
"""
response = llm.complete(prompt)
return response
That's the whole idea. Everything else — vector databases, embedding models, chunking strategies, reranking — is about making these two steps work well at scale, with real documents, for real queries.
A production RAG system has two distinct phases: an indexing phase (which runs once, offline) and a query phase (which runs in real time, for every user request).
Before you can retrieve anything, you have to prepare your knowledge base. This involves three sub-steps.
1. Document ingestion and preprocessing
You start with raw source documents — PDFs, Word files, HTML pages, database records, Confluence pages. These need to be loaded and cleaned: extracting text from PDFs, stripping HTML tags, handling encoding issues, discarding boilerplate headers and footers. This preprocessing step matters more than people expect, because garbage in means garbage retrieved. For a detailed treatment of this stage, see Document Ingestion Pipelines: Loading, Cleaning, and Preprocessing Text for RAG.
2. Chunking
An LLM can only process a limited amount of text at once — this is called its context window, measured in tokens (roughly, word-pieces). You can't stuff a 50-page policy document into a single prompt. So you split documents into smaller pieces called chunks — usually a few hundred to a couple thousand words each, often with some overlap between adjacent chunks so you don't split ideas mid-thought.
The right chunk size is not obvious. Chunks too small lose context; chunks too large flood the prompt with irrelevant text. Getting chunking right is a surprisingly impactful design decision, explored fully in Chunking Strategies: How to Split Documents for Better Retrieval.
3. Embedding
Each chunk is converted into a vector — a list of numbers, typically with hundreds or thousands of dimensions — using a model called an embedding model. These aren't arbitrary numbers. They're a mathematical representation of the chunk's meaning, positioned in a high-dimensional space such that chunks with similar meaning end up close to each other.
For example, the chunks "cancellation within 30 days entitles you to a full refund" and "customers who cancel in the first month receive their money back" would have very similar vector representations, even though they share almost no exact words. This is the core property that makes semantic retrieval possible: you're matching by meaning, not by keyword.
To understand how this works under the hood, Embeddings Explained: How Text Becomes Vectors for Semantic Search walks through the mechanics from scratch.
These vectors are then stored in a vector database — a specialized data store optimized for finding the closest vectors to a given query vector. Popular options include Pinecone, Weaviate, and pgvector. Each has tradeoffs worth understanding before you commit to one for a production system.
Note
The embedding model you use at indexing time and the embedding model you use at query time must be the same model. If you switch models later, you'll need to re-embed and re-index everything. This is a common and painful mistake.
This is what happens every time a user submits a question.
1. Query embedding
The user's question is converted into a vector using the same embedding model you used during indexing.
2. Similarity search
The vector database searches its index for the stored chunk vectors that are most similar to the query vector. "Similar" here is measured mathematically — typically using cosine similarity, which captures the angle between vectors in high-dimensional space. The database returns the top-k most similar chunks (k is usually somewhere between 3 and 20, depending on your design).
For a deeper look at how these similarity measures work and when to prefer one over another, see Understanding Similarity Search: Cosine, Dot Product, and Euclidean Distance for Vector Retrieval.
3. Prompt construction
The retrieved chunks are assembled into a prompt. The structure matters: typically you provide a system instruction that tells the model to use the provided context and not to speculate beyond it, followed by the retrieved chunks, followed by the user's question.
4. LLM generation
The LLM reads the prompt and generates a response grounded in the retrieved context. If the retrieval was good, the response is accurate and citable. If retrieval failed — wrong chunks came back, or nothing relevant was found — the model may hallucinate anyway or correctly say "I don't have enough information."
Here's a minimal but realistic implementation using Python with the OpenAI API:
from openai import OpenAI
import numpy as np
client = OpenAI()
def embed(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
def cosine_similarity(vec_a: list, vec_b: list) -> float:
a = np.array(vec_a)
b = np.array(vec_b)
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def retrieve(query: str, indexed_chunks: list[dict], top_k: int = 3) -> list[str]:
query_embedding = embed(query)
scored = [
(chunk["text"], cosine_similarity(query_embedding, chunk["embedding"]))
for chunk in indexed_chunks
]
scored.sort(key=lambda x: x[1], reverse=True)
return [text for text, score in scored[:top_k]]
def generate_answer(query: str, context_chunks: list[str]) -> str:
context = "\n\n---\n\n".join(context_chunks)
prompt = f"""You are a helpful assistant. Use ONLY the following context to answer
the question. If the answer isn't in the context, say so.
Context:
{context}
Question: {query}
Answer:"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
# Example usage
query = "What is the refund policy for annual subscriptions?"
relevant_chunks = retrieve(query, indexed_chunks)
answer = generate_answer(query, relevant_chunks)
print(answer)
This in-memory approach works for learning and prototyping with a small corpus. For anything beyond a few hundred documents, you'd replace the retrieve function with calls to a proper vector database.
Semantic vector search is powerful, but it has real limitations you need to understand early.
It struggles with exact lookups. If a user asks "What does section 4.2.1 of the contract say?", semantic search may return chunks that are topically related to contracts but miss the specific section. For this kind of query, keyword search (traditional inverted index, like Elasticsearch) is more reliable. This is why many production RAG systems combine both approaches — a technique called hybrid search, covered in Hybrid Search: Combining Keyword and Semantic Search for Better Results.
It also struggles with multi-hop reasoning — questions that require connecting information from multiple documents. "Which of our vendors are based in jurisdictions that our updated compliance policy now restricts?" requires finding the vendor list, finding the compliance policy update, understanding both, and then joining the information. Pure vector retrieval typically won't surface both pieces together. This is where more advanced architectures like Graph RAG come in.
Warning
High cosine similarity between a query and a chunk doesn't guarantee that the chunk actually answers the question. A chunk about "refund timelines" might be very similar to a query about "refund eligibility" — same topic, different question. Retrieval quality is not the same as answer quality.
Basic RAG — embed, retrieve, generate — is just the starting point. As you build real systems, you'll encounter (and need) variations:
Naive RAG is what we've described: embed everything, retrieve top-k by similarity, stuff into prompt. Fast to build, good baseline, but brittle for edge cases.
Advanced RAG adds pre-retrieval and post-retrieval steps. Pre-retrieval might include query rewriting (transforming the user's question into a form that retrieves better) or query expansion. Post-retrieval might include reranking — running a more expensive model over the retrieved chunks to reorder them by actual relevance before constructing the prompt. Reranking Retrieved Results: Implementing Cross-Encoders to Improve RAG Accuracy covers this in detail.
Modular RAG breaks the pipeline into swappable components — different retrievers for different data sources, routing logic that decides which retrieval strategy to use for which query type. This is what most production systems eventually evolve toward.
Agentic RAG goes further: rather than a single retrieval-then-generate pass, an agent can decide to retrieve, evaluate what it got, decide whether to retrieve again with a refined query, and iterate until it's confident in the answer. This introduces planning and self-correction loops.
Tip
Don't reach for Agentic RAG on day one. The added complexity is only justified once you've hit the ceiling of simpler approaches and understand specifically where they're failing. Start with Naive RAG, instrument it, and let failure modes guide your architectural decisions.
There's a physical limit to how much text you can pass to an LLM at once. As of 2024, models range from about 8,000 tokens (roughly 6,000 words) to 128,000 tokens or more. This sounds generous, but it has significant implications for RAG design.
If you retrieve 10 chunks of 500 words each, that's 5,000 words just for context — before the system prompt, the user question, and the response. You're burning budget fast. And counterintuitively, stuffing more context into the prompt doesn't always improve quality. Models can get "lost in the middle" — paying less attention to information in the middle of a long context than to information at the beginning or end.
This is why chunk size, top-k retrieval count, and prompt design all need to be calibrated together. You can read more about how these constraints shape system design in Understanding LLM Context Windows: How Token Limits Shape RAG Design Decisions.
Key insight
More retrieved context is not always better. A tightly relevant three-chunk context often outperforms a sprawling fifteen-chunk context. Precision in retrieval matters as much as recall.
Let's build the simplest possible working RAG system from scratch. You'll need Python and the OpenAI library (pip install openai numpy).
Setup: Create a small in-memory knowledge base from these three "documents" representing a fictional company's HR policy:
documents = [
{
"id": "doc1",
"text": "Employees are entitled to 20 days of paid vacation per year. "
"Unused vacation days can be carried over up to 5 days into the following year. "
"Vacation requests must be submitted at least 2 weeks in advance."
},
{
"id": "doc2",
"text": "Remote work is permitted for all roles that do not require on-site presence. "
"Employees must be available during core hours of 10am to 3pm local time. "
"Remote work arrangements require manager approval."
},
{
"id": "doc3",
"text": "Health insurance coverage begins on the first day of the month following your start date. "
"The company covers 80% of the premium for individual coverage. "
"Dental and vision plans are available as optional add-ons."
}
]
Task 1: Using the embed() and retrieve() functions from earlier, pre-compute embeddings for each document and store them alongside the text.
Task 2: Ask the system these three questions and examine the results:
Task 3: Ask a question the knowledge base can't answer: "What's the parental leave policy?" Observe what happens. Does the model admit it doesn't know, or does it hallucinate? If it hallucinated, refine your system prompt to enforce grounding.
Stretch goal: Add a fourth document that partially answers the parental leave question, re-index, and verify that retrieval now surfaces it.
Mistake 1: The retrieval looks right but the answer is wrong. Check whether the retrieved chunks actually contain the answer. Often they're topically similar but don't address the specific question. This is a retrieval precision problem. Try adjusting chunk size, adding a reranker, or improving your chunking strategy.
Mistake 2: The model ignores the context and answers from its training data. Your system prompt isn't strong enough. Add explicit instructions: "You MUST base your answer only on the provided context. If the context does not contain the answer, say 'I don't have information about that.'" Then test adversarially.
Mistake 3: Embedding a query at search time gives different results than expected. Embedding models are sensitive to phrasing. "refund" and "money back" might embed differently than you expect, especially with weaker models. Consider query expansion or using a stronger embedding model.
Mistake 4: Everything works in development, nothing works in production. You built on clean test documents and are now hitting real-world PDFs with scanned text, inconsistent formatting, and embedded tables. The preprocessing pipeline needs to be much more robust. This is where document ingestion engineering becomes critical work.
Mistake 5: Indexing once and forgetting. Your documents change. Policies update. New products ship. A RAG system with a stale index is a hallucination factory. Build update mechanisms from the start, even if they're simple scheduled re-indexes.
RAG addresses two fundamental limitations of LLMs — frozen knowledge and no access to private data — by separating the act of knowing (retrieval) from the act of reasoning (generation). The pipeline has two phases: an offline indexing phase where documents are chunked, embedded, and stored in a vector database; and an online query phase where the user's question is embedded, relevant chunks are retrieved, and an LLM generates a grounded response.
The core concepts to hold onto:
The right next step depends on what you want to do next. If you want to build a working system end-to-end, Building Your First RAG Pipeline is the natural continuation. If you want to understand a production-ready version with real documents, head to Building a RAG Pipeline with Your Own Documents. And if you're curious about where RAG fits into the broader world of AI agents that can take actions and make decisions, AI Agents: From Concept to Implementation will orient you in that direction.
RAG is not a magic fix for LLM limitations, and it's not trivial to build well. But it's the most practical and reliable architecture we have for grounding AI systems in real, current, private knowledge — and understanding it deeply puts you ahead of the vast majority of people deploying these systems today.