Implementing Hypothetical Document Embeddings (HyDE) from Scratch: Generating Synthetic Answers to Improve Sparse and Dense Retrieval

Introduction

Here's a problem you've probably run into: you build a RAG pipeline, tune your chunking, pick a good embedding model, and still find that retrieval underperforms on certain query types. The user asks something like "What are the main risks of using stablecoins as collateral in DeFi lending protocols?" — a sophisticated, abstract question — and your vector search returns chunks about basic stablecoin mechanics instead of the nuanced risk analysis buried in your corpus.

The problem isn't your embedding model. It's the fundamental mismatch between the linguistic style of a question and the linguistic style of an answer. Questions are short, interrogative, and often abstract. Documents are declarative, dense, and concrete. Even the best dense retrieval models struggle with this gap, especially when users ask in ways that don't share vocabulary with the document's surface text.

Hypothetical Document Embeddings (HyDE) attacks this problem at its root. Instead of embedding the question and searching for similar documents, you ask an LLM to hallucinate a plausible answer, embed that synthetic answer, and use that embedding to search your corpus. The hallucinated document lives in the same representational neighborhood as real documents, closing the query-document gap without any fine-tuning or index changes.

By the end of this lesson, you'll be able to build a fully functional HyDE retrieval pipeline from scratch — and understand exactly when to use it, when to skip it, and how to combine it with other retrieval strategies.

What you'll learn:

• The theoretical foundation of HyDE and why query-document embedding asymmetry hurts retrieval
• How to implement HyDE for both dense (vector) and sparse (BM25) retrieval pipelines
• How to generate and aggregate multiple hypothetical documents for more robust retrieval
• How to evaluate whether HyDE actually improves your specific use case
• How to combine HyDE with hybrid search and reranking for production-grade pipelines

Prerequisites

You should already be comfortable with:

• Dense retrieval fundamentals — how embeddings work and why cosine similarity finds relevant chunks (see Embeddings Explained: How Text Becomes Vectors for Semantic Search)
• Basic RAG pipeline architecture, including chunking, indexing, and generation (Building Your First RAG Pipeline)
• Python, OpenAI API (or equivalent), and a vector store you can query programmatically
• BM25 at a conceptual level — keyword overlap scoring using term frequency and inverse document frequency

The Query-Document Gap: Why Retrieval Fails on Abstract Questions

Before writing a single line of code, let's understand the failure mode precisely. When you embed a user query, you're encoding its semantic intent. When you embed a document chunk, you're encoding its semantic content. These two things often overlap — but not always.

Consider a biomedical corpus. A clinician asks: "Are there contraindications for prescribing ACE inhibitors during the second trimester of pregnancy?"

The relevant document might say: "Use of ACE inhibitors during the second and third trimesters of pregnancy is associated with fetal renal dysfunction, oligohydramnios, and neonatal death. These agents are contraindicated after the first trimester."

That document is a direct answer. But the embedding similarity between the question and this chunk is lower than you'd expect because the question is formulated as a possibility ("are there contraindications?") while the document is assertive ("use of ACE inhibitors is associated with..."). The vocabulary overlaps, but the syntactic frame is completely different.

This gap widens dramatically in three scenarios:

1. Expert queries on specialized corpora — questions use domain shorthand while documents use formal language
2. Conceptual questions — "what are the implications of X" when documents describe X factually without framing implications explicitly
3. Sparse retrieval (BM25) — keyword matching entirely breaks down when the question uses different vocabulary than the answer

HyDE's insight is elegant: generate a document that sounds like an answer, and let the LLM's language model distribution bridge the vocabulary gap for you.

Key insight: HyDE isn't asking the LLM for the correct answer — it's asking it to generate text that occupies the same semantic space as real answers. A factually wrong hypothetical document can still point you to the right real document if it uses the right vocabulary and discourse structure.

Setting Up Your Environment

We'll build everything using Python with OpenAI for generation, sentence-transformers for embeddings, rank_bm25 for sparse retrieval, and chromadb as a lightweight vector store. Install dependencies:

pip install openai sentence-transformers rank-bm25 chromadb numpy

Our running example throughout this lesson will be a corpus of financial regulatory documents — SEC filings, Fed policy statements, and Basel III accord summaries. This is a realistic scenario where HyDE shines: documents are dense and formal, but user queries are often exploratory and abstract.

Building the Baseline Retriever

Let's start with a standard dense retriever so we have a baseline to compare against.

import chromadb
import numpy as np
from sentence_transformers import SentenceTransformer
from typing import List, Dict, Any

# Initialize embedding model and vector store
embedder = SentenceTransformer("BAAI/bge-large-en-v1.5")
client = chromadb.Client()
collection = client.create_collection(
    name="financial_docs",
    metadata={"hnsw:space": "cosine"}
)

# Sample corpus: financial regulatory documents
corpus = [
    {
        "id": "doc_001",
        "text": "Basel III introduced the Net Stable Funding Ratio (NSFR), requiring banks to maintain a stable funding profile relative to their assets. Banks must hold sufficient stable funding to cover required stable funding over a one-year period. This rule applies to institutions with total consolidated assets exceeding $100 billion.",
        "source": "Basel III Summary"
    },
    {
        "id": "doc_002", 
        "text": "The leverage ratio under Basel III is defined as Tier 1 capital divided by total exposure. Institutions must maintain a minimum leverage ratio of 3%, with an additional 50 basis point buffer for global systemically important banks (G-SIBs). This constraint operates independently of risk-weighted capital requirements.",
        "source": "Basel III Capital Requirements"
    },
    {
        "id": "doc_003",
        "text": "Liquidity Coverage Ratio (LCR) requires banking institutions to hold high-quality liquid assets (HQLA) sufficient to cover net cash outflows over a 30-day stress period. Level 1 assets include central bank reserves and sovereign debt. Level 2A and 2B assets face haircuts of 15% and 25-50% respectively.",
        "source": "LCR Implementation Guide"
    },
    {
        "id": "doc_004",
        "text": "Stablecoin issuers face unique regulatory scrutiny. The President's Working Group report recommends that stablecoin issuers be subject to insured depository institution requirements, ensuring reserve assets are held in safe, liquid instruments. This addresses concerns about reserve adequacy and run risk.",
        "source": "PWG Stablecoin Report"
    },
    {
        "id": "doc_005",
        "text": "Procyclicality in financial regulation refers to regulatory mechanisms that amplify economic cycles rather than dampen them. Risk-weighted capital requirements under Basel II were found to be procyclical because asset risk weights declined during booms and rose during downturns, reducing available capital precisely when it was needed.",
        "source": "FSB Procyclicality Analysis"
    }
]

def index_documents(corpus: List[Dict], collection):
    """Embed and index all documents into the vector store."""
    texts = [doc["text"] for doc in corpus]
    ids = [doc["id"] for doc in corpus]
    metadatas = [{"source": doc["source"]} for doc in corpus]
    
    embeddings = embedder.encode(texts, normalize_embeddings=True).tolist()
    
    collection.add(
        documents=texts,
        embeddings=embeddings,
        ids=ids,
        metadatas=metadatas
    )
    print(f"Indexed {len(corpus)} documents")

def baseline_retrieve(query: str, collection, n_results: int = 3) -> List[Dict]:
    """Standard dense retrieval using raw query embedding."""
    query_embedding = embedder.encode(query, normalize_embeddings=True).tolist()
    
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )
    
    return [
        {
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        }
        for i in range(len(results["documents"][0]))
    ]

# Index our corpus
index_documents(corpus, collection)

Now let's test our baseline with a query that tends to trip up standard retrieval:

test_query = "What happens to bank capital requirements when the economy enters a recession?"

baseline_results = baseline_retrieve(test_query, collection)
for i, result in enumerate(baseline_results):
    print(f"\nResult {i+1} (distance: {result['distance']:.4f})")
    print(f"Source: {result['source']}")
    print(f"Text: {result['text'][:200]}...")

The correct answer lives in doc_005 (procyclicality), but baseline retrieval often surfaces doc_002 (leverage ratio) instead because the query's vocabulary ("recession," "economy") doesn't overlap well with the document's vocabulary ("procyclicality," "risk-weighted").

Implementing Core HyDE

Now for the main event. HyDE has three steps:

1. Send the query to an LLM with a prompt instructing it to write a hypothetical answer
2. Embed the hypothetical answer (not the query)
3. Use that embedding to search the corpus

from openai import OpenAI

openai_client = OpenAI()  # assumes OPENAI_API_KEY is set

def generate_hypothetical_document(
    query: str,
    domain_context: str = "financial regulation and banking",
    max_tokens: int = 250,
    temperature: float = 0.7
) -> str:
    """
    Generate a hypothetical document that would answer the query.
    
    The key is the prompt framing: we want document-like text,
    not conversational text. "Write a passage" not "Answer this question."
    """
    prompt = f"""You are an expert in {domain_context}. 

Write a concise passage from a technical document that directly answers the following question. 
Write it as if it were an excerpt from an authoritative report or textbook — declarative, 
factual, and precise. Do not preface it with "Here is a passage" or similar. 
Just write the passage itself.

Question: {query}

Passage:"""

    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": prompt}
        ],
        max_tokens=max_tokens,
        temperature=temperature
    )
    
    return response.choices[0].message.content.strip()

def hyde_retrieve(
    query: str,
    collection,
    domain_context: str = "financial regulation and banking",
    n_results: int = 3
) -> Dict[str, Any]:
    """
    HyDE retrieval: generate hypothetical doc, embed it, search corpus.
    Returns results plus the hypothetical document for inspection.
    """
    # Step 1: Generate hypothetical document
    hypothetical_doc = generate_hypothetical_document(query, domain_context)
    
    # Step 2: Embed the hypothetical document (not the query)
    hyde_embedding = embedder.encode(hypothetical_doc, normalize_embeddings=True).tolist()
    
    # Step 3: Search the corpus with the hypothetical embedding
    results = collection.query(
        query_embeddings=[hyde_embedding],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )
    
    retrieved = [
        {
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        }
        for i in range(len(results["documents"][0]))
    ]
    
    return {
        "hypothetical_document": hypothetical_doc,
        "results": retrieved
    }

Let's run HyDE on the same query and compare:

hyde_output = hyde_retrieve(
    "What happens to bank capital requirements when the economy enters a recession?",
    collection
)

print("=== HYPOTHETICAL DOCUMENT ===")
print(hyde_output["hypothetical_document"])
print("\n=== RETRIEVED RESULTS ===")
for i, result in enumerate(hyde_output["results"]):
    print(f"\nResult {i+1} (distance: {result['distance']:.4f})")
    print(f"Source: {result['source']}")
    print(f"Text: {result['text'][:200]}...")

The hypothetical document might read something like: "During economic downturns, risk-weighted capital requirements exhibit procyclical behavior. As asset valuations decline and default probabilities rise, risk weights increase, effectively raising the amount of regulatory capital banks must hold at precisely the moment when raising capital is most difficult and costly..."

That text now shares vocabulary with doc_005. The embedding naturally clusters with the procyclicality document. This is HyDE working exactly as designed.

Note: The hypothetical document will sometimes be factually wrong. That's fine — you're not serving it to the user. You're using it purely as an embedding vehicle. Factual errors don't necessarily degrade retrieval quality, but consistently wrong domain knowledge in the hypothetical might. We'll address this with multiple hypotheticals next.

Multi-Hypothetical HyDE: Averaging for Robustness

A single hypothetical document can get unlucky — the LLM might generate a plausible but stylistically unusual response that happens to miss the embedding neighborhood of your target documents. The solution: generate several hypothetical documents and average their embeddings.

def generate_multiple_hypotheticals(
    query: str,
    domain_context: str,
    n_hypotheticals: int = 5,
    temperature: float = 0.8
) -> List[str]:
    """
    Generate multiple diverse hypothetical documents.
    Higher temperature increases diversity, which helps with averaging.
    """
    hypotheticals = []
    for _ in range(n_hypotheticals):
        doc = generate_hypothetical_document(
            query, 
            domain_context, 
            temperature=temperature
        )
        hypotheticals.append(doc)
    return hypotheticals

def multi_hyde_retrieve(
    query: str,
    collection,
    domain_context: str = "financial regulation and banking",
    n_hypotheticals: int = 5,
    n_results: int = 3
) -> Dict[str, Any]:
    """
    Multi-hypothetical HyDE: average embeddings from multiple synthetic docs.
    This reduces variance from any single unlucky generation.
    """
    # Generate multiple hypothetical documents
    hypotheticals = generate_multiple_hypotheticals(
        query, domain_context, n_hypotheticals
    )
    
    # Embed all of them
    all_embeddings = embedder.encode(
        hypotheticals, 
        normalize_embeddings=True
    )  # shape: (n_hypotheticals, embedding_dim)
    
    # Average the embeddings, then renormalize
    # This centroid represents the "average" semantic location
    # across all hypothetical documents
    averaged_embedding = np.mean(all_embeddings, axis=0)
    averaged_embedding = averaged_embedding / np.linalg.norm(averaged_embedding)
    
    # Search with the averaged embedding
    results = collection.query(
        query_embeddings=[averaged_embedding.tolist()],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )
    
    retrieved = [
        {
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source"],
            "distance": results["distances"][0][i]
        }
        for i in range(len(results["documents"][0]))
    ]
    
    return {
        "hypothetical_documents": hypotheticals,
        "results": retrieved
    }

Tip: The renormalization step after averaging is important. When you average unit vectors, the result is no longer unit length (unless all vectors were identical). For cosine similarity search, you want a unit vector query. Skip normalization and you'll get subtly wrong similarity scores.

The averaged embedding acts as a centroid in semantic space — it's pulled toward the region where multiple plausible answers cluster. Outlier generations (the LLM going off-script) get diluted by the consensus of other hypotheticals.

HyDE for Sparse Retrieval (BM25)

Here's something the original HyDE paper explores that many practitioners miss: you can apply the same idea to BM25 and other sparse retrievers. Instead of averaging embeddings, you concatenate (or pool) the hypothetical documents and run keyword matching against the expanded text.

from rank_bm25 import BM25Okapi
import re

def tokenize(text: str) -> List[str]:
    """Simple tokenizer — lowercase, split on non-alphanumeric."""
    return re.findall(r'\b[a-z0-9]+\b', text.lower())

# Build BM25 index over our corpus
corpus_texts = [doc["text"] for doc in corpus]
tokenized_corpus = [tokenize(text) for text in corpus_texts]
bm25 = BM25Okapi(tokenized_corpus)

def baseline_bm25_retrieve(query: str, n_results: int = 3) -> List[Dict]:
    """Standard BM25 retrieval using the raw query."""
    tokenized_query = tokenize(query)
    scores = bm25.get_scores(tokenized_query)
    top_indices = np.argsort(scores)[::-1][:n_results]
    
    return [
        {
            "text": corpus[i]["text"],
            "source": corpus[i]["source"],
            "bm25_score": scores[i]
        }
        for i in top_indices
    ]

def hyde_bm25_retrieve(
    query: str,
    domain_context: str = "financial regulation and banking",
    n_hypotheticals: int = 3,
    n_results: int = 3
) -> Dict[str, Any]:
    """
    HyDE for sparse retrieval.
    
    Strategy: generate hypothetical documents, concatenate them into a single
    expanded "query document," and use its tokens for BM25 matching.
    This dramatically expands vocabulary coverage.
    """
    hypotheticals = generate_multiple_hypotheticals(
        query, domain_context, n_hypotheticals, temperature=0.7
    )
    
    # Concatenate the original query + all hypotheticals
    # This preserves original query terms while adding generated vocabulary
    expanded_query_text = query + " " + " ".join(hypotheticals)
    tokenized_expanded = tokenize(expanded_query_text)
    
    scores = bm25.get_scores(tokenized_expanded)
    top_indices = np.argsort(scores)[::-1][:n_results]
    
    return {
        "hypothetical_documents": hypotheticals,
        "expanded_token_count": len(set(tokenized_expanded)),
        "results": [
            {
                "text": corpus[i]["text"],
                "source": corpus[i]["source"],
                "bm25_score": scores[i]
            }
            for i in top_indices
        ]
    }

Let's compare BM25 with and without HyDE:

sparse_query = "What happens to bank capital requirements when the economy enters a recession?"

print("=== BASELINE BM25 ===")
baseline_sparse = baseline_bm25_retrieve(sparse_query)
for r in baseline_sparse:
    print(f"Score: {r['bm25_score']:.3f} | {r['source']}")

print("\n=== HYDE BM25 ===")
hyde_sparse = hyde_bm25_retrieve(sparse_query)
print(f"Expanded token vocabulary: {hyde_sparse['expanded_token_count']} unique tokens")
for r in hyde_sparse["results"]:
    print(f"Score: {r['bm25_score']:.3f} | {r['source']}")

The baseline BM25 will score poorly on the procyclicality document because "recession" doesn't appear in doc_005 — the document uses "economic downturns" and "booms." The HyDE-expanded query will include both phrasings, unlocking BM25 matches that were invisible before.

Warning: Concatenating long hypothetical documents into your BM25 query bloats the token count dramatically. BM25 scoring (specifically the document length normalization) assumes queries are short. Very long "queries" can distort scoring in subtle ways. Monitor BM25 score distributions when using HyDE — if scores cluster strangely, consider truncating or sampling from the hypothetical text rather than using it in full.

Building a Hybrid HyDE Pipeline

The most robust production setup combines HyDE-augmented dense retrieval with HyDE-augmented sparse retrieval, then merges results. This is where HyDE really shines — combining it with Hybrid Search: Combining Keyword and Semantic Search for Better Results gives you the best of both worlds.

from collections import defaultdict

def reciprocal_rank_fusion(
    result_lists: List[List[Dict]], 
    k: int = 60,
    id_key: str = "source"
) -> List[Dict]:
    """
    Combine multiple ranked lists using Reciprocal Rank Fusion.
    k=60 is the standard constant that balances top-rank emphasis.
    """
    rrf_scores = defaultdict(float)
    doc_store = {}
    
    for result_list in result_lists:
        for rank, doc in enumerate(result_list, start=1):
            doc_id = doc[id_key]
            rrf_scores[doc_id] += 1.0 / (k + rank)
            doc_store[doc_id] = doc
    
    # Sort by RRF score, highest first
    sorted_ids = sorted(rrf_scores.keys(), key=lambda x: rrf_scores[x], reverse=True)
    
    return [
        {**doc_store[doc_id], "rrf_score": rrf_scores[doc_id]}
        for doc_id in sorted_ids
    ]

def hybrid_hyde_retrieve(
    query: str,
    collection,
    domain_context: str = "financial regulation and banking",
    n_hypotheticals: int = 3,
    n_results: int = 5
) -> Dict[str, Any]:
    """
    Full hybrid HyDE pipeline:
    1. HyDE dense retrieval (averaged embeddings)
    2. HyDE sparse retrieval (BM25 with expanded query)
    3. RRF fusion of both result sets
    """
    # Generate hypotheticals once and share them across both retrievers
    hypotheticals = generate_multiple_hypotheticals(
        query, domain_context, n_hypotheticals
    )
    
    # --- Dense HyDE ---
    all_embeddings = embedder.encode(hypotheticals, normalize_embeddings=True)
    averaged_embedding = np.mean(all_embeddings, axis=0)
    averaged_embedding = averaged_embedding / np.linalg.norm(averaged_embedding)
    
    dense_results_raw = collection.query(
        query_embeddings=[averaged_embedding.tolist()],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )
    dense_results = [
        {
            "text": dense_results_raw["documents"][0][i],
            "source": dense_results_raw["metadatas"][0][i]["source"],
            "distance": dense_results_raw["distances"][0][i]
        }
        for i in range(len(dense_results_raw["documents"][0]))
    ]
    
    # --- Sparse HyDE ---
    expanded_query_text = query + " " + " ".join(hypotheticals)
    tokenized_expanded = tokenize(expanded_query_text)
    bm25_scores = bm25.get_scores(tokenized_expanded)
    top_sparse_indices = np.argsort(bm25_scores)[::-1][:n_results]
    sparse_results = [
        {
            "text": corpus[i]["text"],
            "source": corpus[i]["source"],
            "bm25_score": bm25_scores[i]
        }
        for i in top_sparse_indices
    ]
    
    # --- Fusion ---
    fused_results = reciprocal_rank_fusion([dense_results, sparse_results])
    
    return {
        "hypothetical_documents": hypotheticals,
        "dense_results": dense_results,
        "sparse_results": sparse_results,
        "fused_results": fused_results[:n_results]
    }

Notice the efficiency optimization: we generate the hypotheticals once and reuse them for both dense and sparse retrieval. This halves your LLM API costs compared to running two separate HyDE calls.

Crafting Effective HyDE Prompts

The quality of your hypothetical documents depends heavily on your generation prompt. Here are the key levers:

Domain specificity matters more than you'd expect. A generic prompt produces generic hypotheticals. Telling the model "you are an expert in Basel III capital adequacy frameworks" produces documents with the specific vocabulary that will match your corpus.

Instructing document-like style is crucial. If you ask "answer this question," you get conversational text. If you ask "write a passage from a technical report," you get declarative, document-style text that lives in the same embedding space as your corpus.

Length calibration. Hypothetical documents should be roughly the same length as your typical corpus chunks. If your chunks are 300 tokens, generate hypotheticals around 200-300 tokens. Too short and they lack vocabulary coverage; too long and they introduce noise.

def build_hyde_prompt(
    query: str,
    domain: str,
    chunk_style: str = "technical report",  # or "academic paper", "legal document", etc.
    approximate_length: str = "2-3 paragraphs"
) -> str:
    """
    Customizable HyDE prompt builder.
    Adapt chunk_style to match your corpus document type.
    """
    return f"""You are an expert in {domain}.

Write {approximate_length} from a {chunk_style} that directly and precisely answers 
the following question. Use terminology and phrasing that would appear in authoritative 
documents on this topic. Be specific and technical. Write only the passage — no preamble, 
no "Here is..." framing.

Question: {query}

Passage:"""

Tip: Look at your actual corpus documents and deliberately match their writing style in your HyDE prompt. If your documents are SEC filings that use passive voice and precise legal terminology, tell the model to write that way. If your corpus is conversational Slack exports, ask for informal prose. The closer the style match, the better your embedding will align.

Evaluating HyDE: Does It Actually Help?

HyDE improves retrieval on many workloads but not all. You need to measure it, not assume it. The right evaluation framework is Evaluating RAG Systems: Precision, Recall, and Faithfulness, but here's a focused version for comparing retrieval strategies:

from typing import Callable

def evaluate_retrieval_strategy(
    strategy_fn: Callable,
    test_cases: List[Dict],
    top_k: int = 3
) -> Dict[str, float]:
    """
    Evaluate a retrieval function on labeled test cases.
    
    Each test case: {"query": str, "relevant_sources": List[str]}
    relevant_sources are the sources that should appear in top-k results.
    """
    hits = 0
    reciprocal_ranks = []
    
    for case in test_cases:
        results = strategy_fn(case["query"])
        retrieved_sources = [r["source"] for r in results[:top_k]]
        relevant = set(case["relevant_sources"])
        
        # Hit Rate: did at least one relevant doc appear in top-k?
        if any(source in relevant for source in retrieved_sources):
            hits += 1
        
        # Mean Reciprocal Rank: where did the first relevant doc appear?
        rr = 0.0
        for rank, source in enumerate(retrieved_sources, start=1):
            if source in relevant:
                rr = 1.0 / rank
                break
        reciprocal_ranks.append(rr)
    
    return {
        "hit_rate@k": hits / len(test_cases),
        "mrr": np.mean(reciprocal_ranks),
        "n_queries": len(test_cases)
    }

# Build labeled test cases for our financial corpus
test_cases = [
    {
        "query": "What happens to bank capital requirements when the economy enters a recession?",
        "relevant_sources": ["FSB Procyclicality Analysis"]
    },
    {
        "query": "How much liquidity must banks hold for a one-month stress scenario?",
        "relevant_sources": ["LCR Implementation Guide"]
    },
    {
        "query": "What regulatory framework governs digital dollar reserve requirements?",
        "relevant_sources": ["PWG Stablecoin Report"]
    },
    {
        "query": "What leverage constraints apply to systemically important financial institutions?",
        "relevant_sources": ["Basel III Capital Requirements"]
    }
]

# Wrap strategies to return consistent format
def run_baseline(query):
    return baseline_retrieve(query, collection)

def run_hyde(query):
    return hyde_retrieve(query, collection)["results"]

def run_hybrid_hyde(query):
    return hybrid_hyde_retrieve(query, collection)["fused_results"]

print("Baseline dense:", evaluate_retrieval_strategy(run_baseline, test_cases))
print("HyDE dense:", evaluate_retrieval_strategy(run_hyde, test_cases))
print("Hybrid HyDE:", evaluate_retrieval_strategy(run_hybrid_hyde, test_cases))

Run this evaluation on your actual corpus with 50-100 labeled queries — not just 4 toy examples. Results vary significantly by domain, corpus size, and embedding model.

Hands-On Exercise: HyDE for a Legal Document Corpus

Build a complete HyDE pipeline for a corpus of employment law documents. This exercise will test your understanding of domain adaptation and evaluation.

Setup:

1. Create a corpus of 20+ chunks from publicly available employment law resources (EEOC guidelines, FLSA regulations, NLRA summaries). You can use the requests library to fetch and chunk Wikipedia articles on these topics as a starting point.

2. Index these documents using the same ChromaDB pattern from this lesson. For guidance on preprocessing steps before indexing, see Document Ingestion Pipelines: Loading, Cleaning, and Preprocessing Text for RAG.

3. Build a HyDE prompt specifically tuned to legal documents. Legal text is highly specific — your prompt should instruct the model to write in the style of regulatory guidance, citing specific statutes and thresholds where applicable.

Core task:

Write a function legal_hyde_retrieve(query: str) -> List[Dict] that:

• Generates 4 hypothetical legal documents
• Averages their embeddings
• Returns top-3 chunks with source attribution

Test it against these queries:

• "What are an employer's obligations when an employee reports sexual harassment?"
• "How many employees must a company have before FMLA applies?"
• "What constitutes a protected concerted activity under labor law?"

Stretch goal:

Add a post-retrieval step that passes the retrieved chunks and original query to GPT-4o-mini with a structured prompt, then evaluate whether HyDE-retrieved chunks produce better final answers than baseline-retrieved chunks. Use Prompt Engineering for RAG: How to Structure System Prompts That Ground LLM Responses in Retrieved Context for guidance on structuring that generation prompt.

Common Mistakes & Troubleshooting

Mistake 1: Using HyDE on factual lookup queries

HyDE adds latency (an LLM call) and costs money. If your query is "What is the capital of France?" or "What's the order number for invoice #4521?", standard retrieval works fine. HyDE helps with abstract, inferential, and conceptual queries — not lookup queries.

Consider combining HyDE with Query Routing in RAG: How to Direct Questions to the Right Data Source or Retrieval Strategy to route simple queries to standard retrieval and complex queries to HyDE.

Mistake 2: Not matching hypothetical length to chunk length

If your corpus chunks are 100 tokens and your hypothetical documents are 600 tokens, the embedding will be pulled toward features of long documents — section structure, transitional language, repetition. Generate hypotheticals at roughly the same length as your corpus chunks.

Mistake 3: Temperature too low

With temperature=0.0, all 5 hypothetical documents will be nearly identical, defeating the purpose of averaging. Use temperature between 0.6 and 0.9 to get meaningful diversity across hypotheticals.

Mistake 4: HyDE confidently hallucinates and misleads retrieval

Sometimes the LLM will generate a hypothetical that's not just factually wrong but semantically wrong — it captures a plausible-sounding but incorrect answer concept that points retrieval in the wrong direction entirely. This is more common in highly specialized domains where the LLM's pretraining doesn't cover the corpus well.

Mitigation: add the original query embedding as one of the vectors you average in. This anchors the search near the original query intent:

# Add original query embedding to the mix before averaging
query_embedding = embedder.encode(query, normalize_embeddings=True)
all_embeddings_with_query = np.vstack([query_embedding, all_embeddings])
averaged_embedding = np.mean(all_embeddings_with_query, axis=0)
averaged_embedding = averaged_embedding / np.linalg.norm(averaged_embedding)

Mistake 5: Ignoring latency cost

HyDE adds one LLM call (or more) to your retrieval path. On GPT-4o-mini, that's roughly 300-800ms. If you're generating 5 hypotheticals in parallel, you still pay the latency of one call but reduce sequential time. On time-sensitive pipelines, consider caching hypotheticals for frequent query patterns. See Retrieval Latency Optimization: Indexing Strategies, ANN Tuning, and Caching Layers for Sub-100ms RAG in Production for caching strategies.

Mistake 6: Applying HyDE without a comparative evaluation

HyDE is not universally better. On short, well-keyword-matched queries against small corpora, it often performs worse than baseline dense retrieval because the hypothetical document introduces vocabulary that doesn't appear in your corpus. Always A/B test with your actual query distribution.

Warning: Hallucination in hypothetical documents is a feature when it's controlled (expanding vocabulary) and a bug when it's systematic (confidently wrong domain model). Monitor the quality of your generated hypotheticals by logging them alongside retrieval results. If you see the hypotheticals consistently missing key domain terminology, update your HyDE prompt with few-shot examples of good hypothetical documents from your domain.

When to Use HyDE vs. Alternatives

HyDE is one of several query-side augmentation techniques. Here's how it compares:

HyDE vs. Query Expansion (multi-query): Multi-query generates several rewordings of the original question and retrieves for each. HyDE generates an answer rather than more questions. HyDE tends to win on highly conceptual queries; multi-query tends to win when the query phrasing is the problem (ambiguous or ill-formed questions). They can be combined. See Query Expansion in RAG: Hypothetical Document Embeddings and Multi-Query Retrieval for a direct comparison.

HyDE vs. Reranking: Reranking operates after retrieval to re-score the candidates. HyDE operates at retrieval to get better candidates in the first place. They're complementary — use HyDE to get better candidates, then Reranking Retrieved Results: Implementing Cross-Encoders to Improve RAG Accuracy to rerank them.

HyDE vs. Fine-tuned embeddings: Fine-tuning your embedding model on your domain can close the query-document gap at the representation level. This is more expensive upfront but cheaper at inference time (no LLM call per query). HyDE is the right choice when you don't have labeled training data or need a quick improvement without retraining. For a deep dive on the tradeoffs, see Fine-Tuning Embedding Models on Domain-Specific Data to Improve Retrieval Accuracy in RAG Pipelines.

Summary & Next Steps

HyDE is a genuinely clever technique that reframes the retrieval problem: instead of asking "what documents are similar to this question?" you ask "what documents are similar to a plausible answer to this question?" That reframing bridges the linguistic gap between queries and documents without touching your index or fine-tuning your embeddings.

Here's what you built in this lesson:

• A baseline dense retriever using sentence-transformers and ChromaDB
• A single-hypothetical HyDE retriever
• A multi-hypothetical HyDE retriever with averaged embeddings
• A HyDE-augmented BM25 sparse retriever with expanded query vocabulary
• A full hybrid HyDE pipeline with RRF fusion
• An evaluation framework to measure whether HyDE actually helps on your workload

The decision of whether to deploy HyDE comes down to your query distribution and latency budget. If your users ask abstract, conceptual questions against a dense technical corpus, HyDE typically delivers meaningful recall improvements. If your queries are short and keyword-rich, the LLM overhead isn't justified.

Where to go from here:

• Combine HyDE with contextual compression to trim retrieved chunks before generation — see Contextual Compression in RAG: Filtering and Compressing Retrieved Chunks Before Passing to the LLM
• Build an adaptive retrieval system that switches between baseline, HyDE, and hybrid strategies based on query complexity — Adaptive Retrieval: Dynamically Adjusting Chunk Count and Search Strategy Based on Query Complexity at Runtime
• Explore how retrieval failures manifest and how to diagnose them systematically — Retrieval Failure Modes: Why RAG Systems Return Wrong or Irrelevant Results and How to Diagnose Them