Before you build a RAG pipeline, you need to understand what "similar" actually means in vector space. This lesson breaks down cosine similarity, dot product, and Euclidean distance from first principles — with working Python code and real retrieval scenarios — so you can make the right choice every time.
Imagine you're building a customer support chatbot. A user types: "My order hasn't shown up yet." Your knowledge base contains an article titled "Tracking Your Shipment." These two phrases share zero words in common, yet they're clearly talking about the same thing. How does your AI system know to retrieve that article? The answer lies in similarity search — the process of finding vectors that are mathematically close to each other in high-dimensional space.
Before you can build anything meaningful with Retrieval-Augmented Generation (RAG) or AI agents, you need to understand how similarity search actually works under the hood. The three metrics you'll encounter everywhere — cosine similarity, dot product, and Euclidean distance — are not interchangeable. Choosing the wrong one is a silent killer: your pipeline will run without errors, but your retrieval quality will be quietly terrible.
By the end of this lesson, you'll know exactly what each metric measures, when to use each one, how to compute them by hand and in Python, and how to reason about tradeoffs when building real retrieval systems.
What you'll learn:
You should be comfortable reading Python code and have a basic sense of what machine learning models do. You do not need any prior knowledge of linear algebra — we'll build everything from the ground up using plain language. If you know what a list of numbers is, you're ready.
Let's start at the foundation. A vector is simply an ordered list of numbers. That's it. When you see [0.2, -0.8, 0.5, 0.1], that's a vector with four elements. Each element is called a dimension, so this vector lives in four-dimensional space.
In AI systems, text is converted into vectors using a process called embedding. An embedding model reads a piece of text — a word, a sentence, or an entire document — and outputs a vector, typically with hundreds or thousands of dimensions. The magic is that the embedding model is trained so that similar meanings produce similar vectors. "My order hasn't arrived" and "Where is my shipment?" end up as vectors that are very close together in this high-dimensional space, even though they look nothing alike as strings.
Think of it like a map. Cities that are physically close to each other appear near each other on the map. Embedding models create a kind of "meaning map" where semantically similar ideas are placed near each other in vector space.
So when you do a similarity search, you're asking: "Given my query vector, which stored vectors are closest to it?" That's a geometry problem. And geometry has multiple ways of measuring "closeness."
Before diving into formulas, let's anchor each metric to an intuition:
These three questions sound similar but give meaningfully different answers in the real world. Let's work through each one.
You've been using Euclidean distance your whole life without knowing its name. When you measure the distance between two cities on a flat map, you're using it. On a two-dimensional grid, if point A is at coordinates (1, 2) and point B is at (4, 6), the Euclidean distance is:
distance = √((4-1)² + (6-2)²)
= √(9 + 16)
= √25
= 5
The general formula for two vectors A and B with n dimensions is:
distance = √( (A₁-B₁)² + (A₂-B₂)² + ... + (Aₙ-Bₙ)² )
In Python, let's implement this from scratch and then compare it to NumPy's built-in:
import numpy as np
def euclidean_distance(a, b):
"""Compute Euclidean distance between two vectors."""
diff = [x - y for x, y in zip(a, b)]
squared_diffs = [d ** 2 for d in diff]
return sum(squared_diffs) ** 0.5
# Example: embedding fragments for customer support topics
order_tracking = [0.8, 0.6, 0.1, 0.3]
shipment_delayed = [0.7, 0.5, 0.2, 0.4]
password_reset = [0.1, 0.2, 0.9, 0.7]
print(euclidean_distance(order_tracking, shipment_delayed)) # 0.245 — close!
print(euclidean_distance(order_tracking, password_reset)) # 1.166 — far apart
# NumPy shorthand
a = np.array(order_tracking)
b = np.array(shipment_delayed)
c = np.array(password_reset)
print(np.linalg.norm(a - b)) # Same result: 0.245
print(np.linalg.norm(a - c)) # Same result: 1.166
Lower Euclidean distance means more similar. A distance of 0 means the vectors are identical.
Here's a critical problem you'll run into in real embeddings. Imagine two documents about shipping:
Embedding models often produce longer (higher magnitude) vectors for longer, denser content. Even if both documents are semantically very similar, Document B's vector might be much longer than Document A's. Euclidean distance will punish that length difference even when the direction of the vectors is nearly identical.
Key insight: Euclidean distance is sensitive to vector magnitude. If your vectors haven't been normalized to a consistent length, it can give misleading similarity scores.
Cosine similarity sidesteps the magnitude problem entirely. Instead of measuring the straight-line distance between two points, it measures the angle between two vectors. If two vectors point in the same direction, the angle between them is 0° and the cosine similarity is 1.0 (maximum). If they're perpendicular (completely unrelated directions), the angle is 90° and the cosine similarity is 0. If they point in exactly opposite directions, the similarity is -1.
The formula is:
cosine_similarity(A, B) = (A · B) / (|A| × |B|)
Where:
A · B is the dot product of A and B (we'll define this shortly)|A| is the magnitude (length) of vector A|B| is the magnitude of vector BThe division by the product of magnitudes is exactly what normalizes away size — it forces the result to live between -1 and 1 regardless of how long the vectors are.
Let's implement it:
import numpy as np
def cosine_similarity(a, b):
"""Compute cosine similarity between two vectors."""
a = np.array(a, dtype=float)
b = np.array(b, dtype=float)
dot_product = np.dot(a, b)
magnitude_a = np.linalg.norm(a)
magnitude_b = np.linalg.norm(b)
if magnitude_a == 0 or magnitude_b == 0:
return 0.0 # Guard against zero vectors
return dot_product / (magnitude_a * magnitude_b)
# Back to our customer support example
order_tracking = np.array([0.8, 0.6, 0.1, 0.3])
shipment_delayed = np.array([0.7, 0.5, 0.2, 0.4])
password_reset = np.array([0.1, 0.2, 0.9, 0.7])
# Now introduce a "scaled up" version of the shipment vector
shipment_scaled = shipment_delayed * 10 # Same direction, 10x magnitude
print(f"order_tracking vs shipment_delayed: {cosine_similarity(order_tracking, shipment_delayed):.4f}")
print(f"order_tracking vs shipment_scaled: {cosine_similarity(order_tracking, shipment_scaled):.4f}")
print(f"order_tracking vs password_reset: {cosine_similarity(order_tracking, password_reset):.4f}")
Output:
order_tracking vs shipment_delayed: 0.9917
order_tracking vs shipment_scaled: 0.9917 ← identical! magnitude doesn't matter
order_tracking vs password_reset: 0.4714
This is the power of cosine similarity. shipment_delayed and shipment_scaled score identically because they point in the same direction. The magnitude was irrelevant.
Best practice: Cosine similarity is the default choice for text embedding similarity search. It's what most embedding models are designed and evaluated against. When in doubt, start here.
The dot product (also called the inner product or scalar product) of two vectors is computed by multiplying corresponding elements and summing the results:
A · B = A₁×B₁ + A₂×B₂ + ... + Aₙ×Bₙ
You can see that the dot product is the numerator of the cosine similarity formula. On its own, it measures how aligned two vectors are while caring about how long they are. A high dot product score means either: the vectors are pointing in similar directions, one or both vectors are large, or both.
import numpy as np
def dot_product_similarity(a, b):
"""Compute raw dot product between two vectors."""
return np.dot(np.array(a), np.array(b))
query = np.array([0.6, 0.4, 0.1])
result_a = np.array([0.5, 0.3, 0.1]) # moderately aligned, short
result_b = np.array([5.0, 3.0, 1.0]) # same direction as a, 10x magnitude
print(f"Query · result_a: {dot_product_similarity(query, result_a):.4f}") # 0.43
print(f"Query · result_b: {dot_product_similarity(query, result_b):.4f}") # 4.3
cosine_a = cosine_similarity(query, result_a)
cosine_b = cosine_similarity(query, result_b)
print(f"Cosine with result_a: {cosine_a:.4f}") # 0.9994
print(f"Cosine with result_b: {cosine_b:.4f}") # 0.9994 — identical
Notice that the dot product considers result_b 10 times more similar to the query than result_a, while cosine similarity treats them as equally similar. Neither answer is wrong — they're measuring different things.
Dot product becomes the right choice when the magnitude of a vector carries meaningful information. The most common scenario is when embedding models output vectors where the length encodes something like confidence or relevance weight. OpenAI's text-embedding-3 models, for example, are trained with a dot product objective — meaning they're specifically optimized so that higher-magnitude vectors represent stronger, more confident representations.
Warning: Using dot product on raw, un-normalized embeddings from models trained with a cosine objective will produce misleading rankings. Always check your embedding model's documentation to see what similarity metric it was trained against.
If your embedding model returns unit-normalized vectors (all vectors have magnitude 1), then cosine similarity and dot product give the exact same ranking. Unit-normalized means |A| = 1 for all vectors, so the denominator in the cosine formula is always 1 × 1 = 1, reducing cosine similarity to just the dot product.
Let's put all three metrics to work on a realistic retrieval scenario: a knowledge base for a SaaS product, where you store embeddings for different support articles.
import numpy as np
# Simulated 6-dimensional embeddings for support articles
# In reality these would be 768 or 1536 dimensions from an embedding model
articles = {
"billing_invoices": np.array([0.9, 0.8, 0.1, 0.05, 0.1, 0.05]),
"billing_refunds": np.array([0.85, 0.75, 0.15, 0.1, 0.05, 0.1]),
"account_password": np.array([0.1, 0.05, 0.9, 0.8, 0.1, 0.15]),
"account_2fa": np.array([0.05, 0.1, 0.85, 0.75, 0.15, 0.1]),
"api_authentication": np.array([0.15, 0.1, 0.7, 0.8, 0.8, 0.7]),
}
# User query: "How do I get a refund for my subscription?"
query = np.array([0.88, 0.78, 0.12, 0.08, 0.08, 0.12])
def rank_articles(query, articles, metric_fn, higher_is_better=True):
scores = {}
for name, vec in articles.items():
scores[name] = metric_fn(query, vec)
return sorted(scores.items(), key=lambda x: x[1], reverse=higher_is_better)
# Euclidean distance (lower = more similar)
euclidean_fn = lambda a, b: np.linalg.norm(a - b)
# Cosine similarity (higher = more similar)
cosine_fn = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# Dot product (higher = more similar)
dot_fn = lambda a, b: np.dot(a, b)
print("=== Euclidean Distance (lower = better) ===")
for name, score in rank_articles(query, articles, euclidean_fn, higher_is_better=False):
print(f" {name:<25} {score:.4f}")
print("\n=== Cosine Similarity (higher = better) ===")
for name, score in rank_articles(query, articles, cosine_fn):
print(f" {name:<25} {score:.4f}")
print("\n=== Dot Product (higher = better) ===")
for name, score in rank_articles(query, articles, dot_fn):
print(f" {name:<25} {score:.4f}")
Run this and you'll observe that cosine similarity and Euclidean distance often agree on rankings here, because the vectors are of similar magnitudes. That's often true with normalized embeddings — and it's another reason cosine similarity is the dominant default.
In production, you don't compute similarity against every stored vector one by one. That would be far too slow at scale. Instead, vector databases like Pinecone, Qdrant, Weaviate, and ChromaDB build approximate nearest neighbor (ANN) indexes that allow fast retrieval across millions of vectors.
When you create a collection or index in these databases, you have to specify your distance metric upfront. Here's how that configuration looks in ChromaDB, a popular open-source vector database:
import chromadb
client = chromadb.Client()
# Create a collection using cosine similarity
collection = client.create_collection(
name="support_articles",
metadata={"hnsw:space": "cosine"} # Options: "cosine", "l2" (Euclidean), "ip" (inner product/dot product)
)
# Add documents (ChromaDB handles embedding if you provide an embedding function)
collection.add(
documents=[
"How to request a refund for your subscription",
"Resetting your account password",
"Understanding your invoice and billing cycle",
],
ids=["doc1", "doc2", "doc3"]
)
# Query
results = collection.query(
query_texts=["I want my money back"],
n_results=2
)
print(results["documents"])
Important: The distance metric you specify in your vector database must match the metric your embedding model was trained with. Mixing these up is one of the most common causes of poor RAG retrieval performance.
Let's put everything together in a mini retrieval system. Your goal: build a simple semantic search function using NumPy, test it with three different metrics, and observe where they agree and disagree.
Setup:
import numpy as np
# Pretend these are real embeddings (in practice, use sentence-transformers or OpenAI)
# We'll use random unit-normalized vectors to simulate realistic behavior
np.random.seed(42)
def make_embedding(seed_vector, noise_level=0.1):
"""Create a unit-normalized embedding with controlled noise."""
vec = np.array(seed_vector, dtype=float)
vec += np.random.randn(len(vec)) * noise_level
return vec / np.linalg.norm(vec) # normalize to unit length
# Knowledge base: job descriptions for a recruiting tool
job_postings = {
"senior_ml_engineer": make_embedding([0.9, 0.8, 0.2, 0.1, 0.3, 0.1]),
"junior_data_analyst": make_embedding([0.4, 0.3, 0.7, 0.2, 0.1, 0.5]),
"data_scientist": make_embedding([0.7, 0.6, 0.5, 0.3, 0.4, 0.2]),
"devops_engineer": make_embedding([0.1, 0.2, 0.1, 0.9, 0.8, 0.3]),
"frontend_developer": make_embedding([0.1, 0.1, 0.3, 0.5, 0.9, 0.8]),
}
# Candidate resume summary (simulated embedding)
candidate_query = make_embedding([0.75, 0.65, 0.45, 0.25, 0.35, 0.25])
Your tasks:
make_embedding to not normalize the vectors (remove the last line), then re-run. Observe how dot product and cosine rankings diverge.This exercise directly illustrates why understanding normalization matters for choosing your metric.
Mistake 1: Using the wrong metric for your embedding model.
This is the single most common retrieval quality issue. OpenAI's text-embedding-ada-002 was trained with cosine similarity. Google's newer models often use dot product. Always check the model card or documentation before configuring your vector store.
Mistake 2: Forgetting that Euclidean distance and cosine similarity have opposite conventions. With Euclidean distance, smaller is better. With cosine similarity, larger is better. It sounds obvious, but sorting results in the wrong direction is an embarrassingly easy bug to ship to production.
Mistake 3: Computing cosine similarity on zero vectors. If a document or query somehow produces an all-zero embedding (can happen with empty strings or edge cases in preprocessing), dividing by zero will give you NaN values. Always add a guard clause, as shown in the implementation above.
Mistake 4: Assuming normalized vectors are always a good idea. Normalization is great for cosine similarity. But if you're using a model that encodes confidence or relevance in vector magnitude — and you're using dot product — normalizing your vectors first destroys that signal.
Mistake 5: Re-indexing your entire vector store after changing the metric. If you build an index with cosine and then switch to Euclidean without rebuilding, you'll get meaningless results. The index structure itself encodes the distance metric assumption. There's no shortcut here — change the metric, rebuild the index.
Let's tie everything together:
Where to go next:
sentence-transformers, text-embedding-3-small, and open-source alternatives like nomic-embed-text produce these vectors and how to evaluate their quality for your domain.Understanding the math behind similarity search isn't academic — it directly determines whether your RAG system retrieves the right context or quietly returns garbage. You now have the foundation to make these decisions intentionally.