Single-turn RAG breaks the moment users start asking follow-up questions. This deep-dive lesson teaches you how to build genuine conversational memory into your RAG pipeline — with query reformulation, rolling history compression, session persistence, and retrieval deduplication that actually works in production.
Picture this: a financial analyst is using your internal knowledge assistant to research a merger. She asks about the target company's revenue trends, then asks "how does that compare to their main competitor?" Then she asks "what was the debt situation again?" Then "given all that, what's the acquisition risk?" Each question builds on the last. Without any memory of the conversation, your RAG system treats every message as if it arrived from a stranger — forcing her to repeat context, re-explain relationships, and essentially start over with every turn. The assistant is technically answering questions correctly while being practically useless.
This is the core problem of stateful multi-turn RAG: making retrieval-augmented systems that genuinely participate in conversations, not just answer isolated queries. Single-turn RAG is well-understood. You embed a question, retrieve relevant chunks, and pass them to an LLM. But conversations have dynamics that completely break that simple model. References become ambiguous ("it," "that," "the one we discussed"), topics drift and return, context accumulates faster than any context window can hold it, and users expect the system to remember — not just mechanically — but intelligently, surfacing the right prior context at the right time.
By the end of this lesson, you'll have the skills to architect and implement a production-grade stateful RAG system. We'll go deep on the actual engineering: how to represent session state, how to reformulate queries that depend on prior turns, how to selectively compress conversation history without losing fidelity, and how to make smart decisions about what to keep in active memory versus what to retrieve lazily.
What you'll learn:
This lesson assumes you're already comfortable with the fundamentals of RAG pipelines. If you need a grounding in how basic retrieval-augmented generation works, start with Building Your First RAG Pipeline before continuing. You should also have familiarity with LLM API calls, vector similarity search, and embedding models. Familiarity with token budget management — specifically how context windows constrain design — will be particularly important; if that's fuzzy, read Understanding LLM Context Windows: How Token Limits Shape RAG Design Decisions first.
You'll need Python 3.10+, access to an OpenAI-compatible LLM API, and a vector database (code examples use a generic interface, but adapt to whatever you're running in production).
Before writing any code, let's be precise about what "stateful" actually means in the context of a RAG pipeline. Every turn in a conversation involves at minimum:
A naive stateful implementation just concatenates all prior messages into the context window and calls it done. This works for a handful of turns, then falls apart. Here's what the naive version looks like:
from openai import OpenAI
client = OpenAI()
def naive_multiturn_rag(user_message: str, history: list[dict], retriever) -> str:
# Retrieve chunks for the current query
chunks = retriever.retrieve(user_message)
context = "\n".join([chunk.text for chunk in chunks])
# Build messages: system + all prior history + current turn
messages = [
{"role": "system", "content": f"Answer using this context:\n{context}"}
]
messages.extend(history)
messages.append({"role": "user", "content": user_message})
response = client.chat.completions.create(
model="gpt-4o",
messages=messages
)
assistant_message = response.choices[0].message.content
history.append({"role": "user", "content": user_message})
history.append({"role": "assistant", "content": assistant_message})
return assistant_message
This has three compounding problems. First, retrieval is done on the raw user message, which may be a pronoun-laden fragment like "what about their Q3?" — a terrible embedding query. Second, history grows without bound. Third, the retrieved context is fixed at the current turn with no connection to what was retrieved in prior turns.
Let's build up a proper solution piece by piece.
The first step toward a real solution is representing session state as a first-class data structure rather than a flat list of messages. What we actually need to track:
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import uuid
@dataclass
class RetrievedChunk:
chunk_id: str
text: str
source: str
score: float
turn_retrieved: int # which turn this was retrieved in
@dataclass
class ConversationTurn:
turn_number: int
user_message: str
reformulated_query: str # the query actually sent to retrieval
retrieved_chunks: list[RetrievedChunk]
assistant_response: str
timestamp: datetime
entities_mentioned: list[str] # extracted named entities
topics: list[str] # topic labels for this turn
@dataclass
class SessionState:
session_id: str
user_id: str
created_at: datetime
turns: list[ConversationTurn] = field(default_factory=list)
# Accumulated entity memory: entity -> last-mentioned context
entity_memory: dict[str, str] = field(default_factory=dict)
# Topic progression through the session
topic_history: list[str] = field(default_factory=list)
# Chunks seen across all turns (deduplicated)
seen_chunk_ids: set[str] = field(default_factory=set)
# Compressed summary of earlier turns (populated when history gets long)
history_summary: Optional[str] = None
summary_covers_turns: int = 0 # how many turns the summary covers
def current_turn_number(self) -> int:
return len(self.turns)
def recent_turns(self, n: int = 5) -> list[ConversationTurn]:
return self.turns[-n:]
def get_active_messages(self, max_turns: int = 10) -> list[dict]:
"""Build the message list for LLM context, respecting the window."""
messages = []
for turn in self.turns[-max_turns:]:
messages.append({"role": "user", "content": turn.user_message})
messages.append({"role": "assistant", "content": turn.assistant_response})
return messages
This structure is the foundation of everything that follows. Notice that we're tracking what was retrieved and when — this matters enormously for deduplication and for building a coherent retrieval strategy across turns.
Key insight: A conversation is not just a sequence of messages — it's a sequence of retrieval events. Each turn shifts what the user is focusing on, which should shift what you retrieve. Tracking retrieval history separately from message history lets you reason about the information landscape of the session, not just the dialogue.
The single biggest performance leverage point in multi-turn RAG is query reformulation: transforming the user's raw message into a query that will retrieve relevant documents even when the message contains unresolved references, elliptical phrases, or implicit dependencies on prior context.
Consider this exchange:
If you send turn 4's message directly to your vector store, you'll get garbage back. The embedding for "which is more promising for enterprise deployment?" captures almost none of the actual topic. What you need is something like: "Comparison of Constitutional AI (Anthropic) and OpenAI's RLHF safety approach for enterprise AI deployment suitability."
Here's how to implement query reformulation using the LLM itself:
def reformulate_query(
current_message: str,
session: SessionState,
client: OpenAI,
model: str = "gpt-4o-mini"
) -> str:
"""
Given the current user message and session state, produce a
self-contained retrieval query.
"""
# Build a compact representation of recent context
recent = session.recent_turns(n=4)
context_summary = ""
if session.history_summary:
context_summary = f"Earlier conversation summary: {session.history_summary}\n\n"
recent_dialogue = "\n".join([
f"User: {t.user_message}\nAssistant: {t.assistant_response[:300]}..."
for t in recent
])
entity_context = ""
if session.entity_memory:
entity_context = "Key entities in this conversation: " + ", ".join(
f"{k} ({v})" for k, v in list(session.entity_memory.items())[-10:]
)
reformulation_prompt = f"""You are a query reformulation engine for a retrieval system.
Given a conversation context and a new user message, rewrite the user message into a complete, self-contained search query that:
1. Resolves all pronouns and references ("it", "they", "that", "the one we discussed")
2. Makes the topic explicit even if the user was elliptical
3. Includes relevant entities and concepts from prior context that are needed for retrieval
4. Is optimized for semantic similarity search (not conversational)
5. Is a single paragraph, 1-3 sentences maximum
{context_summary}
Recent conversation:
{recent_dialogue}
{entity_context}
New user message: "{current_message}"
Reformulated query (return ONLY the query, no explanation):"""
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": reformulation_prompt}],
temperature=0.1,
max_tokens=200
)
return response.choices[0].message.content.strip()
This is a lightweight LLM call — use a smaller, faster model (GPT-4o-mini, Claude Haiku, Gemini Flash) to keep latency down. The reformulated query then goes to your vector store instead of the raw message.
Warning: Query reformulation adds a second LLM call per turn. At scale this adds both latency and cost. Profile your P95 latency carefully. One mitigation: implement caching for reformulation when the user message is already self-contained (simple heuristic: if it's longer than 80 characters and doesn't contain pronouns, skip reformulation). Another option is to parallelize reformulation and retrieval — reformulate while also firing the raw query, then merge results.
Let's also implement entity extraction so the session state stays rich:
import json
def extract_entities_and_topics(
user_message: str,
assistant_response: str,
client: OpenAI,
model: str = "gpt-4o-mini"
) -> tuple[list[str], list[str]]:
"""Extract named entities and topic labels from a completed turn."""
prompt = f"""Extract named entities and topic labels from this exchange.
User: {user_message}
Assistant: {assistant_response[:500]}
Return JSON with:
- "entities": list of named entities (companies, people, products, standards, regulations)
- "topics": list of 1-3 high-level topic labels (e.g. "financial risk", "model architecture")
Return ONLY valid JSON, nothing else."""
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.0,
response_format={"type": "json_object"}
)
result = json.loads(response.choices[0].message.content)
return result.get("entities", []), result.get("topics", [])
Here's the mathematical reality: GPT-4o has a 128K token context window. Sounds enormous. But consider that a typical assistant response is 300-600 tokens, a user message is 50-200 tokens, and your retrieved chunks (say, 5 chunks of 512 tokens each) add another 2,560 tokens per turn. By turn 15, you're already at 45,000+ tokens just from conversation history and retrieved context. Add your system prompt, and you're burning most of the window — which degrades response quality (LLMs struggle with the "lost in the middle" problem at high context loads) and dramatically increases cost.
The solution is hierarchical history management: keep recent turns verbatim, compress older turns into summaries, and discard or externalize the rest.
class HistoryManager:
"""
Manages conversation history with rolling compression.
Strategy:
- Keep last N turns verbatim (working memory)
- Summarize turns beyond N into a rolling summary
- Track which chunk IDs have been seen to avoid redundant retrieval
"""
def __init__(
self,
client: OpenAI,
verbatim_turns: int = 6,
summary_model: str = "gpt-4o-mini"
):
self.client = client
self.verbatim_turns = verbatim_turns
self.summary_model = summary_model
def maybe_compress(self, session: SessionState) -> None:
"""
If the session has more turns than verbatim_turns,
compress the oldest turns into the session's history_summary.
"""
turns_to_summarize = len(session.turns) - self.verbatim_turns
if turns_to_summarize <= session.summary_covers_turns:
return # Nothing new to summarize
# Get the turns we need to add to the summary
new_turns_start = session.summary_covers_turns
new_turns_end = len(session.turns) - self.verbatim_turns
turns_to_add = session.turns[new_turns_start:new_turns_end]
if not turns_to_add:
return
turns_text = "\n".join([
f"Turn {t.turn_number}:\n"
f"User asked: {t.user_message}\n"
f"Key information discussed: {t.assistant_response[:400]}"
for t in turns_to_add
])
existing_summary = session.history_summary or "No prior summary."
compression_prompt = f"""You are maintaining a conversation summary for a RAG assistant.
Existing summary of earlier turns:
{existing_summary}
New turns to incorporate:
{turns_text}
Update the summary to include the new turns. The summary must:
1. Preserve all specific facts, numbers, entity names, and decisions
2. Capture what questions were asked and what was concluded
3. Note any unresolved questions or open threads
4. Be structured (use a few short paragraphs, not bullets)
5. Not exceed 600 words
Updated summary:"""
response = self.client.chat.completions.create(
model=self.summary_model,
messages=[{"role": "user", "content": compression_prompt}],
temperature=0.1,
max_tokens=800
)
session.history_summary = response.choices[0].message.content.strip()
session.summary_covers_turns = new_turns_end
def build_context_window(
self,
session: SessionState,
retrieved_chunks: list[RetrievedChunk],
system_prompt_template: str
) -> list[dict]:
"""
Build the full message list to send to the LLM,
respecting token budgets.
"""
# Deduplicate retrieved chunks against what we've seen
new_chunks = [
c for c in retrieved_chunks
if c.chunk_id not in session.seen_chunk_ids
]
# Build retrieved context string
context_parts = []
if new_chunks:
context_parts.append("=== Retrieved Context (Current Turn) ===")
for chunk in new_chunks:
context_parts.append(
f"[Source: {chunk.source}]\n{chunk.text}"
)
# Include chunks from recent turns if still relevant
# (simplified: include chunks from last 2 turns)
recent_chunk_texts = []
for turn in session.recent_turns(n=2):
for chunk in turn.retrieved_chunks:
if chunk.chunk_id not in {c.chunk_id for c in new_chunks}:
recent_chunk_texts.append(
f"[Source: {chunk.source}, retrieved turn {chunk.turn_retrieved}]\n{chunk.text}"
)
if recent_chunk_texts:
context_parts.append("=== Context From Recent Turns ===")
context_parts.extend(recent_chunk_texts[:3]) # cap at 3 to control tokens
retrieved_context = "\n\n".join(context_parts)
# Build system message with history summary if available
history_context = ""
if session.history_summary:
history_context = f"""
=== Conversation History Summary ===
{session.history_summary}
"""
system_message = system_prompt_template.format(
retrieved_context=retrieved_context,
history_context=history_context
)
messages = [{"role": "system", "content": system_message}]
# Add verbatim recent turns
for turn in session.recent_turns(self.verbatim_turns):
messages.append({"role": "user", "content": turn.user_message})
messages.append({"role": "assistant", "content": turn.assistant_response})
return messages
Tip: When compressing history, the quality of your summary is critical — a poor summary that drops numerical facts or conflates entities will silently corrupt the conversation's accuracy. Consider using a slightly larger model for summarization (GPT-4o) even if you use smaller models for reformulation. The cost difference is small because summarization is infrequent.
This approach connects closely to the broader topic of memory architectures — if you want to go deeper on the difference between how working memory and long-term episodic memory should be structured for AI systems, Tool Memory vs. Retrieval Memory in AI Agents provides an excellent companion framework.
Standard RAG retrieves fresh chunks for every query. In a multi-turn context, this misses two important opportunities: avoiding redundant retrieval (fetching the same chunks you already have) and leveraging context carryover (using what you know from prior turns to inform retrieval for the current turn).
Once a chunk has been retrieved and passed to the LLM, there's usually limited value in retrieving it again on the next turn. Track seen chunk IDs in your session state and filter them from new results:
def retrieve_with_deduplication(
query: str,
session: SessionState,
retriever,
top_k: int = 8,
final_k: int = 5
) -> list[RetrievedChunk]:
"""
Retrieve more than needed, then filter out chunks already seen.
"""
# Over-retrieve to account for deduplication loss
raw_results = retriever.retrieve(query, top_k=top_k)
# Filter chunks already in the session
novel_results = [
r for r in raw_results
if r.chunk_id not in session.seen_chunk_ids
]
# If deduplication left us with too few, include some high-scoring seen chunks
if len(novel_results) < 2 and raw_results:
top_seen = [r for r in raw_results if r.chunk_id in session.seen_chunk_ids]
novel_results = novel_results + top_seen[:2]
# Track what we're about to add
selected = novel_results[:final_k]
for chunk in selected:
session.seen_chunk_ids.add(chunk.chunk_id)
return selected
Sometimes a user's message is genuinely ambiguous — it could be asking about the current topic, a prior topic, or something bridging both. In these cases, firing multiple retrieval queries and merging results dramatically improves recall. This is especially important in longer sessions where the user circles back to earlier topics.
from collections import defaultdict
def multi_query_retrieval(
primary_query: str,
session: SessionState,
retriever,
client: OpenAI,
top_k_per_query: int = 5
) -> list[RetrievedChunk]:
"""
Generate 2-3 retrieval queries covering different interpretations,
then merge and rank results.
"""
recent_topics = session.topic_history[-6:] if session.topic_history else []
expansion_prompt = f"""Given this primary retrieval query and recent conversation topics,
generate 2 additional alternative queries that might capture different aspects the user could be asking about.
Primary query: {primary_query}
Recent topics: {', '.join(recent_topics)}
Return a JSON object with key "alternative_queries" containing a list of 2 query strings.
Return ONLY valid JSON."""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": expansion_prompt}],
response_format={"type": "json_object"},
temperature=0.3
)
result = json.loads(response.choices[0].message.content)
alt_queries = result.get("alternative_queries", [])
all_queries = [primary_query] + alt_queries[:2]
# Retrieve for each query
chunk_scores: dict[str, float] = defaultdict(float)
chunk_objects: dict[str, RetrievedChunk] = {}
for i, query in enumerate(all_queries):
chunks = retriever.retrieve(query, top_k=top_k_per_query)
for rank, chunk in enumerate(chunks):
# RRF-style scoring: weight primary query higher
query_weight = 1.5 if i == 0 else 1.0
rrf_score = query_weight / (rank + 60)
chunk_scores[chunk.chunk_id] += rrf_score
chunk_objects[chunk.chunk_id] = chunk
# Sort by accumulated score and deduplicate
ranked = sorted(chunk_scores.items(), key=lambda x: x[1], reverse=True)
return [chunk_objects[cid] for cid, _ in ranked[:7]]
This integrates naturally with contextual compression — after merging multi-query results, you can apply a compressor to trim each chunk to only the sentences relevant to the user's current question, controlling token usage even when you retrieve broadly.
Now let's assemble the complete pipeline, wiring together all the pieces:
class StatefulRAGPipeline:
SYSTEM_TEMPLATE = """You are a knowledgeable research assistant with access to retrieved documents.
{history_context}
{retrieved_context}
Instructions:
- Answer based on the retrieved context and conversation history
- If you're referencing something discussed earlier in the conversation, acknowledge it
- If the retrieved context doesn't cover the question, say so clearly
- Be specific: cite sources when possible
- Do not repeat large blocks of text from prior responses; the user can scroll up"""
def __init__(
self,
retriever,
client: OpenAI,
history_manager: HistoryManager,
llm_model: str = "gpt-4o"
):
self.retriever = retriever
self.client = client
self.history_manager = history_manager
self.llm_model = llm_model
def create_session(self, user_id: str) -> SessionState:
return SessionState(
session_id=str(uuid.uuid4()),
user_id=user_id,
created_at=datetime.now()
)
def process_turn(
self,
user_message: str,
session: SessionState
) -> tuple[str, SessionState]:
"""
Process one conversational turn, updating session state.
Returns (assistant_response, updated_session).
"""
turn_number = session.current_turn_number() + 1
# Step 1: Compress history if needed
self.history_manager.maybe_compress(session)
# Step 2: Reformulate the query
if session.turns: # Only reformulate if there's prior context
reformulated = reformulate_query(user_message, session, self.client)
else:
reformulated = user_message # First turn, no context to resolve
# Step 3: Retrieve with deduplication + multi-query
retrieved_chunks = multi_query_retrieval(
primary_query=reformulated,
session=session,
retriever=self.retriever,
client=self.client
)
# Attach turn number to chunks
for chunk in retrieved_chunks:
chunk.turn_retrieved = turn_number
# Step 4: Build context window
messages = self.history_manager.build_context_window(
session=session,
retrieved_chunks=retrieved_chunks,
system_prompt_template=self.SYSTEM_TEMPLATE
)
# Step 5: Add current user message
messages.append({"role": "user", "content": user_message})
# Step 6: Generate response
response = self.client.chat.completions.create(
model=self.llm_model,
messages=messages,
temperature=0.3
)
assistant_response = response.choices[0].message.content
# Step 7: Extract entities and topics
entities, topics = extract_entities_and_topics(
user_message, assistant_response, self.client
)
# Step 8: Update entity memory
for entity in entities:
# Store the context in which this entity appeared
session.entity_memory[entity] = f"Discussed in turn {turn_number}: {user_message[:100]}"
session.topic_history.extend(topics)
# Step 9: Record the turn
completed_turn = ConversationTurn(
turn_number=turn_number,
user_message=user_message,
reformulated_query=reformulated,
retrieved_chunks=retrieved_chunks,
assistant_response=assistant_response,
timestamp=datetime.now(),
entities_mentioned=entities,
topics=topics
)
session.turns.append(completed_turn)
return assistant_response, session
Note: In production, you'd want to serialize the
SessionStateto a database between requests (Redis for hot sessions, PostgreSQL for durable storage). For high-traffic deployments, the session object should be designed for efficient serialization — keep retrieved chunk texts in a separate cache keyed by chunk_id, and store only references in the session state.
A stateful RAG system requires a persistence layer. The simplest pattern that works at production scale uses a two-tier store:
SessionState as JSON or use Pickle. Set TTL to 2 hours.import json
import redis
import psycopg2
from dataclasses import asdict
class SessionStore:
def __init__(self, redis_url: str, pg_dsn: str):
self.redis = redis.from_url(redis_url)
self.pg_dsn = pg_dsn
self.hot_ttl = 7200 # 2 hours
def save(self, session: SessionState) -> None:
"""Save to both Redis (hot) and Postgres (durable)."""
session_data = self._serialize(session)
# Hot cache
self.redis.setex(
f"session:{session.session_id}",
self.hot_ttl,
json.dumps(session_data)
)
# Durable store (upsert)
with psycopg2.connect(self.pg_dsn) as conn:
with conn.cursor() as cur:
cur.execute("""
INSERT INTO sessions (session_id, user_id, data, updated_at)
VALUES (%s, %s, %s, NOW())
ON CONFLICT (session_id) DO UPDATE
SET data = EXCLUDED.data, updated_at = NOW()
""", (session.session_id, session.user_id, json.dumps(session_data)))
def load(self, session_id: str) -> Optional[SessionState]:
"""Load from Redis first, fall back to Postgres."""
# Try hot cache first
raw = self.redis.get(f"session:{session_id}")
if raw:
return self._deserialize(json.loads(raw))
# Fall back to durable store
with psycopg2.connect(self.pg_dsn) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT data FROM sessions WHERE session_id = %s",
(session_id,)
)
row = cur.fetchone()
if row:
session = self._deserialize(row[0])
# Repopulate hot cache
self.redis.setex(
f"session:{session_id}",
self.hot_ttl,
json.dumps(row[0])
)
return session
return None
def _serialize(self, session: SessionState) -> dict:
# Convert set to list for JSON serialization
data = asdict(session)
data['seen_chunk_ids'] = list(session.seen_chunk_ids)
data['created_at'] = session.created_at.isoformat()
for turn in data['turns']:
turn['timestamp'] = turn['timestamp'] if isinstance(
turn['timestamp'], str) else turn['timestamp'].isoformat()
return data
def _deserialize(self, data: dict) -> SessionState:
# Reconstruct the dataclass from the dict
data['seen_chunk_ids'] = set(data.get('seen_chunk_ids', []))
data['created_at'] = datetime.fromisoformat(data['created_at'])
# ... reconstruct nested dataclasses
# (full implementation left as exercise)
return SessionState(**data)
Security note: in multi-tenant environments, always validate that the requesting user matches the user_id in the session before loading. A session ID should be a UUID that's cryptographically random — never sequential integers that users could enumerate. For enterprise deployments, this intersects directly with access control patterns covered in Enterprise RAG: Security, Permissions, and Multi-Tenant Architecture.
Users don't stay on one topic. They might spend six turns on revenue analysis, pivot to regulatory risk, then come back to revenue. The naive history representation loses the thread when topics jump around.
The solution is to implement a topic signal in your reformulation prompt. When you detect (via topic history) that the current message seems to be returning to an earlier topic, explicitly signal the reformulator:
def detect_topic_return(session: SessionState, current_topics: list[str]) -> bool:
"""Check if current topics appeared much earlier in the session."""
if not session.topic_history or not current_topics:
return False
recent_topics = set(session.topic_history[-4:])
earlier_topics = set(session.topic_history[:-4])
for topic in current_topics:
if topic in earlier_topics and topic not in recent_topics:
return True
return False
When a topic return is detected, you can use the summary of earlier turns (which should have captured that topic's prior discussion) as additional context during reformulation.
What happens when retrieved documents in turn 8 contradict something retrieved in turn 3? Or when the user corrects themselves: "Wait, I meant the 2023 figures, not 2022"?
Handle corrections by scanning the user's message for correction signals ("actually," "I meant," "sorry, that should be," "let me clarify") and explicitly flagging these in your system prompt:
CORRECTION_SIGNALS = [
"actually", "i meant", "sorry", "let me clarify",
"wait,", "correction:", "that's wrong", "not what i said"
]
def detect_correction(user_message: str) -> bool:
lower = user_message.lower()
return any(signal in lower for signal in CORRECTION_SIGNALS)
When a correction is detected, add a note in the context window: "Note: the user has issued a correction in this turn. Prioritize the corrected information over prior context."
At 50+ turns, even your rolling summary gets large. Implement a two-level summary: a "deep summary" covering turns 1-30 (compressed once and rarely updated) and an "active summary" covering turns 30 to present. This keeps the summarization cost manageable.
For sessions that span multiple days (a research project that continues over a week), the active context should be re-bootstrapped at session resumption: load the full summary, load the last 3 turns verbatim, and explicitly tell the LLM in the system prompt how long this conversation has been ongoing and what its main arc has been.
Warning: Do not keep the full verbatim history of a 50-turn session in the context window even if your model's context length technically permits it. LLM attention is not uniform — models degrade significantly when relevant information is buried in the middle of a very long context. A concise summary is often better than a complete transcript because it forces the most important facts to the surface. This is the "lost in the middle" problem documented in several LLM benchmarks.
Evaluating multi-turn RAG is harder than evaluating single-turn because quality degrades across turns, and the failure modes are different. Standard metrics like faithfulness and answer relevance still apply, but you need additional metrics:
Coreference resolution accuracy: Does the reformulated query correctly resolve what "it," "they," and "that" refer to? You can evaluate this by having human raters (or a judge LLM) score whether the reformulated query is semantically equivalent to what a fully explicit human would have asked.
Context carryover score: Take a multi-turn conversation, strip every other turn, and ask: does the answer to the last question still make sense? If the system is properly carrying context, it should. If it's just doing turn-by-turn retrieval, it won't.
Session coherence: Over a 20-turn session, is the assistant's understanding of the entities and topics consistent? Sample 5 turns at random, extract the key entities and their properties as mentioned in those turns, and check for contradictions.
Retrieval novelty: What percentage of retrieved chunks in each turn are new (not seen before in the session)? A healthy system should show declining novelty as the session matures (you've already seen the most relevant chunks), but should not drop to zero (there should always be fresh context when the topic shifts). Track this per-session and alarm if novelty goes to zero too early — it often indicates the reformulation is broken.
The broader evaluation framework for RAG systems — precision, recall, faithfulness at the chunk level — still applies here. If you need a refresher on those metrics, Evaluating RAG Systems: Precision, Recall, and Faithfulness is the place to start.
Build a complete stateful RAG system and stress-test it with the following conversation scenario. Use a document corpus of your choice (regulatory filings, technical documentation, or research papers work well).
Scenario: A compliance analyst researching a financial regulation
Design a 15-turn conversation that includes:
Your deliverables:
The goal is to identify where your implementation leaks information, wastes tokens, or fails to carry context properly. Real improvement happens when you find the specific failure mode in your own implementation.
Mistake: Using raw user messages for retrieval in all turns
Symptom: Retrieval quality drops dramatically after turn 3. Users have to repeat themselves to get relevant results.
Fix: Always reformulate for turns after the first. The cost is 50-100ms and a fraction of a cent. The quality improvement is dramatic.
Mistake: Including full assistant responses in the verbatim history
Assistant responses can be 600+ tokens each. Including 6 turns of verbatim history means up to 3,600 tokens from assistant responses alone, most of which is elaboration rather than key facts.
Fix: When building the history for the context window, truncate prior assistant responses to their first 200-300 tokens, or summarize them to one sentence. The LLM doesn't need to re-read its own prior outputs in detail — it needs the key facts.
Mistake: Not tracking which chunks have been retrieved
Symptom: The same highly-relevant chunk appears in the context window turn after turn, consuming tokens for information the LLM already has and has already used.
Fix: Track seen_chunk_ids in session state and deduplicate as shown above.
Mistake: Compressing history too aggressively or too early
Symptom: The system "forgets" specific facts (exact figures, specific names) that were discussed in compressed turns. Users get frustrated when the assistant gives subtly wrong information that contradicts what was discussed earlier.
Fix: Test your summarizer by feeding it a transcript and comparing the summary against specific facts from that transcript. Any specific number, proper noun, or decision that's mentioned in the original should survive the summary. If your summary model drops these, use a stronger model for summarization, or consider a hybrid approach: keep a structured "fact ledger" (key-value pairs of entity: fact) alongside the prose summary.
Mistake: One session store for all users
Symptom: Works fine in testing, catastrophically fails in production when two users happen to have operations that interleave in the session store.
Fix: Always namespace session keys by user ID and session ID. Make session load/save atomic (use Redis transactions or database transactions). Never allow one user's request to read or modify another user's session.
Mistake: Not handling session resumption gracefully
Symptom: When a user returns after 2 days, the system either fails (session expired) or picks up exactly where it left off without acknowledging the time gap.
Fix: On session load, check the timestamp of the last turn. If it's more than X hours ago, add a system note: "This session was last active [N days ago]. The conversation previously covered: [summary]. Resume naturally." This prevents jarring continuity breaks.
Tip: Consider implementing adaptive retrieval strategies per turn: early turns (exploratory, broad questions) benefit from retrieving more chunks with wider semantic search; later turns (specific, narrow questions) benefit from fewer, higher-precision chunks with metadata filtering. The turn number and question complexity together are good signals for adjusting your retrieval configuration.
We've built a complete architecture for stateful multi-turn RAG that addresses the real engineering challenges: query reformulation that resolves conversational references, rolling history compression that respects token budgets, retrieval deduplication that avoids wasting context on chunks the model has already seen, session persistence with a two-tier hot/cold store, and entity memory that tracks the key objects of a conversation across many turns.
The key mental model to take away: a multi-turn RAG session is managing multiple overlapping information streams — the conversation history, the retrieved document corpus, and the accumulated session state — and your job is to present the LLM with the most signal-dense possible context window at every turn, not just the most complete one. Completeness is the enemy of clarity when your window is finite.
For your next steps, consider how this stateful framework interacts with more advanced RAG patterns. If your conversations often cross topic boundaries that map to different document sources, query routing strategies become critical — your reformulated query should carry enough signal for the router to make good decisions. If your sessions involve complex multi-hop reasoning, look at Agentic RAG patterns where the retrieval itself can iterate and self-correct, extending naturally into the multi-turn case. And if you're deploying this in production at scale, latency becomes your primary concern — the multiple LLM calls (reformulation, extraction, compression) all add up, and retrieval latency optimization techniques apply directly to the retrieval steps in this pipeline.