Your embedding model determines what "similar" means in your RAG system — and it's the single biggest lever over retrieval quality. This lesson compares OpenAI, Sentence Transformers, and Cohere side by side with working code, a decision framework, and a hands-on evaluation harness you can run on your own data.
You're building a RAG pipeline for your company's internal knowledge base — hundreds of PDFs, Confluence pages, and Slack exports. You've set up your vector store, written your chunking logic, and now you hit the decision that quietly determines whether your whole system works or flops: which embedding model do you use?
Pick the wrong one and you'll wonder why your retrieval keeps surfacing tangentially related documents while missing the obvious answers. Pick the right one and suddenly your system feels almost magical — users ask questions in natural language and get back exactly the right passages. The embedding model is the single biggest lever you have over retrieval quality, and yet most tutorials treat it as an afterthought, defaulting to whatever the first code example shows.
By the end of this lesson, you'll know how embedding models actually work, what makes one model better than another for a given use case, and how to make an informed, practical choice between the three dominant options in the RAG ecosystem: OpenAI's text-embedding models, the open-source Sentence Transformers family, and Cohere's embed API. You'll also have working code for all three so you can run your own comparisons.
What you'll learn:
You should be comfortable with Python and have a basic understanding of what a RAG system does. If embeddings are a new concept, spend 10 minutes with Embeddings Explained: How Text Becomes Vectors for Semantic Search before continuing. You'll also find it useful to have read Building Your First RAG Pipeline to understand the broader context in which your embedding model operates.
Before we compare specific models, let's make sure we share a clear mental model of what an embedding model is doing.
An embedding model takes a piece of text — a sentence, a paragraph, a document chunk — and converts it into a list of numbers called a vector. You can think of this vector as a set of coordinates that place the text in a vast multi-dimensional space. The key property that makes this useful is semantic proximity: texts that mean similar things end up close together in that space, even if they use completely different words.
For example, the query "How do I reset my password?" and the document chunk "To change your credentials, navigate to Account Settings and click the key icon" have very different words but nearly identical meaning. A good embedding model pushes both of these close together in vector space. When your RAG pipeline does a similarity search, it finds the document chunk because of that proximity.
Key insight
Your embedding model determines what "similar" means in your RAG system. A weak model will cluster documents by surface word overlap rather than meaning. A strong model clusters them by semantic intent. This is why retrieval quality lives and dies on embedding choice.
The dimension of the output vector (e.g., 768, 1536, 3072 numbers) and how those dimensions were learned during training are what differentiate one model from another. Larger dimensions can capture more nuance but also cost more to store and compare. The training data and training objective matter even more: a model trained primarily on web crawl data will encode general-purpose semantics, while one trained on biomedical literature will understand "myocardial infarction" and "heart attack" as equivalent far better than a general model would.
Understanding this sets up everything that follows. When we say one model is "better" than another, we always mean: better for a specific domain and use case based on its training data, output dimensions, and the kind of similarity it was trained to preserve.
OpenAI's embedding API is the most commonly used in the RAG ecosystem, largely because it pairs naturally with GPT-4 in tutorials and documentation. The current generation models are text-embedding-3-small (1536 dimensions) and text-embedding-3-large (3072 dimensions).
How you use them:
from openai import OpenAI
client = OpenAI(api_key="your-api-key")
def embed_openai(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
response = client.embeddings.create(
input=texts,
model=model
)
return [item.embedding for item in response.data]
# Embed a document chunk
chunks = [
"The refund policy allows returns within 30 days of purchase.",
"Customers must provide a receipt to process any exchange."
]
embeddings = embed_openai(chunks)
print(f"Dimensions: {len(embeddings[0])}") # 1536
Key characteristics:
dimensions parameter. This can significantly reduce storage costs.text-embedding-3-small is extremely cheap (~$0.02 per million tokens), making cost rarely the deciding factor for most organizations.Warning
OpenAI embeddings require sending your text to an external API. For organizations handling sensitive data — healthcare records, legal documents, financial reports — this may violate data governance policies. Always check with your security and compliance team before using any cloud API for embedding.
Sentence Transformers is a Python library built on top of HuggingFace's Transformers that provides hundreds of pre-trained embedding models you can run locally. The most common models for RAG are from the all-MiniLM and all-mpnet families, plus the newer BAAI/bge and intfloat/e5 series which consistently top leaderboards.
from sentence_transformers import SentenceTransformer
# Load a model locally (downloads on first run, cached afterward)
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
def embed_sentence_transformers(texts: list[str]) -> list[list[float]]:
# BGE models work best with an instruction prefix for queries
# Documents are embedded as-is
embeddings = model.encode(texts, normalize_embeddings=True)
return embeddings.tolist()
chunks = [
"The refund policy allows returns within 30 days of purchase.",
"Customers must provide a receipt to process any exchange."
]
embeddings = embed_sentence_transformers(chunks)
print(f"Dimensions: {len(embeddings[0])}") # 768
Tip
For BGE and E5 models, embed your queries with a prefix like "Represent this sentence for searching relevant passages: " but embed your document chunks without any prefix. This asymmetric approach is how these models were trained and significantly improves retrieval quality.
Key characteristics:
Cohere occupies an interesting middle ground: a managed API like OpenAI's, but with some distinctive architectural choices that make it genuinely competitive in specific scenarios.
import cohere
co = cohere.Client("your-api-key")
def embed_cohere(
texts: list[str],
input_type: str = "search_document" # or "search_query"
) -> list[list[float]]:
response = co.embed(
texts=texts,
model="embed-english-v3.0",
input_type=input_type
)
return response.embeddings
# When indexing documents:
chunk_embeddings = embed_cohere(chunks, input_type="search_document")
# When embedding a user query at search time:
query = "What is the return window for purchases?"
query_embedding = embed_cohere([query], input_type="search_query")
print(f"Dimensions: {len(chunk_embeddings[0])}") # 1024
Notice the input_type parameter — this is Cohere's explicit acknowledgment that the vector representing a question should be in a slightly different part of the embedding space than the vector representing a document chunk that answers it. This is called asymmetric embedding, and it's one of Cohere's strongest differentiators.
Key characteristics:
search_query vs search_document distinction is built into the API. This often yields measurably better retrieval for Q&A use cases.embed-multilingual-v3.0 supports 100+ languages in a single embedding space, meaning a French query can retrieve an English document. This is a significant advantage for global enterprises.text-embedding-3-large.This is often the most important constraint and should be evaluated first. If your documents contain PII, protected health information, privileged legal content, or trade secrets, a managed API may be off the table entirely. In that case, Sentence Transformers running on your own infrastructure is the default answer, and the comparison between OpenAI and Cohere becomes moot.
For organizations in heavily regulated sectors, this single factor makes the decision: run open-source models locally.
General-purpose embedding models are trained on diverse internet text. They work well for common language but struggle with specialized vocabulary. Consider a radiology department building a RAG system over medical imaging reports. The term "adenopathy" needs to be close to "lymph node enlargement" in embedding space — a general model may not reliably encode this relationship.
Here's a practical test: take 10 queries your users will actually ask and 10 document chunks that correctly answer them. For each query-chunk pair, compute the cosine similarity with each candidate model. The model that produces consistently higher similarity scores for true pairs (and lower for non-matching pairs) is the better fit.
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
def evaluate_retrieval_quality(queries, relevant_chunks, embed_fn):
"""
For each query, compute similarity to its relevant chunk.
Higher average = better model for this domain.
"""
scores = []
for query, chunk in zip(queries, relevant_chunks):
q_vec = np.array(embed_fn([query]))
c_vec = np.array(embed_fn([chunk]))
score = cosine_similarity(q_vec, c_vec)[0][0]
scores.append(score)
print(f"Mean similarity: {np.mean(scores):.4f}")
print(f"Min similarity: {np.min(scores):.4f}")
return scores
# Run this with your actual data for each model
Key insight
Don't benchmark models on generic datasets. Use your queries and your documents. A model that tops the MTEB leaderboard might underperform on your company's internal jargon, while a domain-specific model from HuggingFace might be exactly what you need.
If your users will query in multiple languages or your documents span multiple languages, the picture changes considerably:
embed-multilingual-v3.0 is the strongest managed API option for cross-lingual retrieval.intfloat/multilingual-e5-large is the strongest open-source option, trained on 94 languages.If you're building a high-traffic production system, API latency matters. Retrieval Latency Optimization covers this in depth, but from an embedding perspective:
For high-throughput RAG applications — think a customer support system handling thousands of queries per minute — local inference with a fast model like BAAI/bge-small-en-v1.5 often wins on raw performance.
Vector dimensions directly determine storage requirements. At 1 million document chunks:
| Model | Dimensions | Storage (float32) |
|---|---|---|
| OpenAI text-embedding-3-small | 1536 | ~6 GB |
| OpenAI text-embedding-3-small (compressed to 256) | 256 | ~1 GB |
| BAAI/bge-base-en-v1.5 | 768 | ~3 GB |
| Cohere embed-english-v3.0 | 1024 | ~4 GB |
| Cohere embed-english-v3.0 (int8) | 1024 | ~1 GB |
OpenAI's dimension reduction and Cohere's quantization features become significant advantages when you're indexing millions of chunks. For more on how this interacts with your vector store design, see Indexing Strategies for RAG.
Here's a practical set of questions to guide your choice:
Is your data sensitive or subject to regulatory constraints? → Yes: Use Sentence Transformers, run locally. → No: Continue.
Do your users query in multiple languages, or do your documents span multiple languages?
→ Yes: Use Cohere multilingual or multilingual-e5-large.
→ No: Continue.
Is your domain highly specialized (legal, medical, scientific, financial)? → Yes: Benchmark domain-specific Sentence Transformers models. Consider fine-tuning an embedding model on your data. → No: Continue.
Do you need the fastest possible time-to-working-prototype with minimal infrastructure?
→ Yes: OpenAI text-embedding-3-small — install the SDK, add your API key, done.
→ No: Continue.
Are you optimizing for retrieval quality in a Q&A or search use case with English content?
→ Cohere's asymmetric embeddings or BAAI/bge-large-en-v1.5 are both excellent choices. Benchmark both.
Note
These aren't mutually exclusive in production. A sophisticated RAG system might use a fast, small local model for coarse candidate retrieval and a high-quality API model for reranking. See Reranking Retrieved Results for more on this pattern.
Let's build a simple but real comparison harness. You'll need openai, sentence-transformers, cohere, numpy, and scikit-learn installed.
Create a file called embedding_comparison.py:
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from openai import OpenAI
from sentence_transformers import SentenceTransformer
import cohere
# --- Setup ---
openai_client = OpenAI(api_key="your-openai-key")
cohere_client = cohere.Client("your-cohere-key")
st_model = SentenceTransformer("BAAI/bge-base-en-v1.5")
# --- Your test data (replace with your actual domain content) ---
queries = [
"What is the penalty for early contract termination?",
"How do I escalate a billing dispute?",
"When does the service level agreement apply to outages?",
]
# The "correct" document chunk for each query
relevant_chunks = [
"Early termination of the service agreement incurs a fee equal to two months of the current subscription rate.",
"Billing disputes must be submitted via the support portal within 60 days of the invoice date. Unresolved disputes escalate to the billing team lead.",
"The SLA guarantees 99.9% uptime. Unplanned outages exceeding 15 minutes trigger automatic service credits.",
]
# Distractor chunks — NOT the answer to any query
distractor_chunks = [
"Our platform supports SSO via SAML 2.0 and OAuth 2.0.",
"Data is encrypted at rest using AES-256.",
"The mobile app is available on iOS 14+ and Android 10+.",
]
all_chunks = relevant_chunks + distractor_chunks
# --- Embedding functions ---
def embed_openai(texts):
response = openai_client.embeddings.create(
input=texts, model="text-embedding-3-small"
)
return np.array([item.embedding for item in response.data])
def embed_st(texts):
return st_model.encode(texts, normalize_embeddings=True)
def embed_cohere_docs(texts):
r = cohere_client.embed(texts=texts, model="embed-english-v3.0", input_type="search_document")
return np.array(r.embeddings)
def embed_cohere_query(texts):
r = cohere_client.embed(texts=texts, model="embed-english-v3.0", input_type="search_query")
return np.array(r.embeddings)
# --- Evaluation ---
def evaluate_model(embed_query_fn, embed_doc_fn, label):
print(f"\n{'='*40}")
print(f"Model: {label}")
doc_vecs = embed_doc_fn(all_chunks)
correct_retrievals = 0
for i, query in enumerate(queries):
query_vec = embed_query_fn([query])
sims = cosine_similarity(query_vec, doc_vecs)[0]
top_idx = np.argmax(sims)
is_correct = top_idx == i # correct chunk is at index i
if is_correct:
correct_retrievals += 1
print(f" Query: '{query[:50]}...'")
print(f" Top match index: {top_idx} ({'✓ CORRECT' if is_correct else '✗ WRONG'})")
print(f" Similarity score: {sims[top_idx]:.4f}")
print(f" Accuracy: {correct_retrievals}/{len(queries)}")
# Run comparisons
evaluate_model(embed_openai, embed_openai, "OpenAI text-embedding-3-small")
evaluate_model(embed_st, embed_st, "BAAI/bge-base-en-v1.5")
evaluate_model(embed_cohere_query, embed_cohere_docs, "Cohere embed-english-v3.0")
Run this with your actual domain queries and chunks — not the generic examples above. The model that retrieves the correct chunk most consistently is your winner.
Tip
For a rigorous evaluation, use at least 50 query-chunk pairs drawn from realistic user questions. Three examples can tell you if a model is catastrophically wrong, but it can't reliably distinguish good from great. For a framework to do this rigorously, see Evaluating RAG Systems: Precision, Recall, and Faithfulness.
Mistake 1: Using the same embedding model for queries and documents when you shouldn't
BGE and E5 models expect different treatment for queries versus documents (instruction prefixes on queries). If you forget this, your similarity scores will be systematically lower and retrieval will suffer. Always check the model card on HuggingFace for the recommended usage pattern.
Mistake 2: Switching embedding models after indexing
Every document in your vector store must be embedded with the same model using the same settings. If you change models — even just upgrading from text-embedding-3-small to text-embedding-3-large — you must re-embed every single document. Embedding spaces from different models are incompatible. Build model selection into your system design early.
Mistake 3: Ignoring chunk size when benchmarking
Your embedding model doesn't operate in isolation — it works on chunks produced by your chunking strategy. A model that works well on 512-token chunks may perform differently on 128-token chunks. Always benchmark embedding models with the actual chunk size you plan to use in production. See Chunking Strategies: How to Split Documents for Better Retrieval for guidance on chunk size selection.
Mistake 4: Assuming the leaderboard winner is the right choice
The MTEB (Massive Text Embedding Benchmark) is useful for general guidance, but it evaluates on standardized public datasets. Your use case has specific vocabulary, query patterns, and document structure that may not match those benchmarks. Always validate on your own data.
Mistake 5: Forgetting to normalize embeddings before similarity search
Many models produce vectors that aren't unit-normalized by default. If you're using cosine similarity, always normalize first. Most vector databases handle this transparently, but if you're computing similarities manually (as in the exercise above), ensure your vectors are normalized. Sentence Transformers has a normalize_embeddings=True parameter for this; for OpenAI and Cohere, the vectors are already normalized.
Warning
If you see very low similarity scores (below 0.3) even for obvious query-document matches, check normalization first. Un-normalized vectors will skew cosine similarity calculations and make retrieval look far worse than it actually is.
You've now got a clear picture of the three dominant embedding model families and the specific dimensions that should drive your choice:
text-embedding-3-small/large): Best for fast prototyping, English-centric use cases, and teams without GPU infrastructure. The dimension reduction feature is genuinely useful at scale.BAAI/bge-large-en-v1.5 and intfloat/e5-large-v2 are consistently strong all-around choices.The meta-lesson is that there's no universal winner. Embedding model selection is a decision you make based on your specific constraints: regulatory environment, document domain, language requirements, latency budget, and team infrastructure capabilities. The evaluation harness in the exercise above gives you a repeatable way to make that decision with data rather than intuition.
From here, there are several natural directions to deepen your knowledge. If your domain vocabulary is specialized, Fine-Tuning Embedding Models on Domain-Specific Data will show you how to adapt any of these models to your content. If you're finding that no single embedding model captures everything you need, Hybrid Search: Combining Keyword and Semantic Search for Better Results shows how to layer keyword search on top of semantic retrieval for a more robust system. And once you've settled on your embedding strategy, Prompt Engineering for RAG will help you make the most of the chunks your model retrieves.