Embedding Metadata Enrichment: Attaching Document Attributes at Ingestion to Power Filtered and Faceted Retrieval in RAG Pipelines

Introduction

Imagine you've built a RAG system for a large financial services company. The vector store contains thousands of documents: regulatory filings, internal policy memos, product documentation, and market research reports — all ingested together. A compliance officer asks: "What does our current AML policy say about transaction monitoring thresholds?" Your retrieval system dutifully fetches the ten most semantically similar chunks. But three of them are from a policy document that was superseded two years ago, two are from a product brochure that mentions thresholds only in passing, and one is from a competitor analysis report that never should have been surfaced in the first place. The LLM synthesizes all of this into a confident, detailed — and dangerously wrong — answer.

The problem isn't your embedding model. The problem is that you treated every document as a featureless blob of text. In the real world, documents carry rich structural attributes: who wrote them, when, for what department, at what classification level, covering which products. When you strip all of that away at ingestion time, you lose the ability to reason about which documents should even be considered for a given query. Metadata enrichment is how you give your retrieval system a memory for those attributes, enabling it to pre-filter the candidate pool before semantic search ever runs.

By the end of this lesson, you'll know how to design and attach metadata at ingestion time so that your RAG system can perform filtered and faceted retrieval — returning not just semantically relevant chunks, but contextually appropriate ones.

What you'll learn:

• What metadata enrichment is and why it's architecturally necessary in production RAG
• How to identify which document attributes to capture and how to model them as structured fields
• How to attach metadata to embeddings at ingestion time using LangChain and Chroma/Pinecone
• How pre-filtering with metadata differs from post-filtering, and when each matters
• How to design a metadata schema that supports faceted retrieval across multiple dimensions simultaneously

---

Prerequisites

You should be comfortable with the basic mechanics of a RAG pipeline — if terms like "embedding," "vector store," and "retrieval" are unfamiliar, start with RAG Fundamentals: Build Your First Retrieval-Augmented Generation System and Embeddings Explained: How Text Becomes Vectors for Semantic Search first. You should also be comfortable with Python at a beginner level — reading and writing dictionaries, lists, and function calls.

---

What Metadata Actually Is (and Isn't)

Let's be precise, because this term gets used loosely. In the context of a vector store, metadata refers to structured, non-vector attributes stored alongside each embedding. Think of it like attaching a label to the back of a photograph. The photograph itself (the embedding) captures visual content — what's in the scene, the mood, the composition. But the label on the back says: "Taken June 2019, Nairobi, during the product launch event, by Sarah." That label doesn't change what the photo looks like, but it makes it findable in ways that visual similarity alone cannot achieve.

In a vector store, each stored object typically has three things:

1. The embedding vector — a high-dimensional array of numbers representing semantic content
2. The document text — the original chunk, used to populate the LLM's context window
3. The metadata dict — a flat or nested dictionary of structured attributes

Common metadata fields for enterprise documents might include:

• source_file — the filename or path
• document_type — "policy", "contract", "report", "faq"
• department — "compliance", "engineering", "sales"
• author — person or team responsible
• created_date — ISO 8601 date string
• effective_date — when the document took effect (crucial for policy documents)
• expiry_date — when it was superseded or expired
• jurisdiction — "US", "EU", "APAC"
• product_line — which products the document pertains to
• classification — "public", "internal", "confidential"
• version — document version or revision number

What metadata is not is a replacement for the semantic content of your chunks. You're not summarizing or paraphrasing the document into the metadata. You're attaching structured facts about the document as a whole — and sometimes about the specific chunk — that allow you to filter before or alongside vector search.

---

Designing Your Metadata Schema

The most consequential decision in metadata enrichment isn't the code — it's the schema. A poorly designed schema either doesn't capture enough to be useful or captures so much that it becomes impossible to maintain consistently.

Start with your query patterns

The right question to ask is: "What attributes would a user reasonably want to filter by when asking questions of this corpus?" Think about real queries your users will make:

• "What does our current (not historical) refund policy say?" → needs document_type, effective_date, expiry_date
• "Show me only engineering runbooks for the payments service" → needs department, document_type, product_line
• "What are the EU regulatory requirements for data retention?" → needs jurisdiction, document_type
• "What did the Q3 2023 market analysis say about churn?" → needs document_type, reporting_period, created_date

Notice how each of these queries has an implicit structured constraint. Without metadata, the LLM has to infer currency and relevance from the text alone — which it often gets wrong.

Keep fields filterable

Vector databases have constraints on what field types support filtering. Most support:

• Strings and string arrays (exact match, $in operators)
• Numbers (range comparisons: $gte, $lte)
• Booleans (true/false flags like is_current_version)

Avoid storing long free-text strings in metadata fields you intend to filter on — those belong in the document content itself. Dates should be stored as ISO 8601 strings ("2023-09-15") or as Unix timestamps (integers) depending on what your vector store supports for range filtering.

Chunk-level vs. document-level metadata

Most metadata applies at the document level — the author, department, and created_date are the same for every chunk derived from a single source file. But some attributes apply at the chunk level:

• chunk_index — the position of this chunk within the document (useful for debugging and for parent document retrieval)
• section_heading — the heading under which this chunk falls
• page_number — especially relevant for PDF sources
• contains_table — a boolean flag if the chunk contains structured tabular data

A good schema carries document-level metadata down to every chunk, then augments it with chunk-level attributes.

---

The Ingestion Pipeline: Where Metadata Gets Attached

Understanding when metadata is attached is as important as understanding what metadata to attach. The attachment happens during ingestion — specifically, when you create Document objects (in LangChain's terminology) before they're embedded and stored.

Here's the high-level flow:

Raw files → Load → Clean → Chunk → Attach Metadata → Embed → Store

The metadata attachment step sits between chunking and embedding. You're annotating each chunk with its contextual attributes before the vector store ever sees it. Let's build this out concretely.

Step 1: Loading documents with source metadata

from langchain.document_loaders import PyPDFLoader, DirectoryLoader
import os

def load_policy_documents(directory: str) -> list:
    """Load all PDFs from a directory, preserving file metadata."""
    documents = []
    for filename in os.listdir(directory):
        if not filename.endswith(".pdf"):
            continue
        filepath = os.path.join(directory, filename)
        loader = PyPDFLoader(filepath)
        pages = loader.load()
        # Each page already has basic metadata: source, page
        documents.extend(pages)
    return documents

raw_docs = load_policy_documents("./policy_documents/")
print(raw_docs[0].metadata)
# {'source': './policy_documents/aml_policy_v3.pdf', 'page': 0}

Out of the box, loaders attach minimal metadata — typically just the source path and page number. That's a starting point, not a destination.

Step 2: Building a metadata enrichment function

The real work happens in a function that takes a raw document and its filename, then attaches domain-specific attributes. In practice, you might derive these from a metadata registry (a CSV or database that maps filenames to attributes), from filename parsing conventions, or from automated extraction.

from datetime import datetime
import re

# Simulated metadata registry — in production, this might be a database
DOCUMENT_REGISTRY = {
    "aml_policy_v3.pdf": {
        "document_type": "policy",
        "department": "compliance",
        "effective_date": "2024-01-01",
        "expiry_date": None,
        "is_current_version": True,
        "jurisdiction": "US",
        "version": 3,
        "author": "Compliance Team",
        "classification": "internal",
        "product_line": None,
    },
    "aml_policy_v2.pdf": {
        "document_type": "policy",
        "department": "compliance",
        "effective_date": "2022-06-01",
        "expiry_date": "2023-12-31",
        "is_current_version": False,
        "jurisdiction": "US",
        "version": 2,
        "author": "Compliance Team",
        "classification": "internal",
        "product_line": None,
    },
    "payments_runbook_q4.pdf": {
        "document_type": "runbook",
        "department": "engineering",
        "effective_date": "2023-10-01",
        "expiry_date": None,
        "is_current_version": True,
        "jurisdiction": None,
        "version": 1,
        "author": "Payments Engineering",
        "classification": "internal",
        "product_line": "payments",
    },
}

def enrich_document_metadata(doc, registry: dict) -> dict:
    """
    Given a LangChain Document and a registry dict,
    return an enriched metadata dict.
    """
    source_path = doc.metadata.get("source", "")
    filename = os.path.basename(source_path)
    
    # Start with existing metadata from the loader
    enriched = dict(doc.metadata)
    
    # Pull in registry attributes
    registry_entry = registry.get(filename, {})
    enriched.update(registry_entry)
    
    # Add chunk-level computed fields
    enriched["source_filename"] = filename
    enriched["ingestion_timestamp"] = datetime.utcnow().isoformat()
    
    # Normalize None to empty string for vector stores that don't handle None well
    for key, value in enriched.items():
        if value is None:
            enriched[key] = ""
    
    return enriched

Step 3: Chunking and propagating metadata

When you split documents into chunks, the metadata must propagate to every child chunk. LangChain's text splitters do this automatically — each chunk inherits the parent Document's metadata dictionary. You can then add chunk-specific fields on top.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

def chunk_and_enrich(raw_docs: list, registry: dict) -> list:
    """
    Split documents into chunks and attach full enriched metadata to each chunk.
    """
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=512,
        chunk_overlap=64,
        separators=["\n\n", "\n", ". ", " "],
    )
    
    enriched_chunks = []
    
    for doc in raw_docs:
        # Enrich the document-level metadata first
        enriched_meta = enrich_document_metadata(doc, registry)
        enriched_doc = Document(page_content=doc.page_content, metadata=enriched_meta)
        
        # Split into chunks — metadata propagates automatically
        chunks = splitter.split_documents([enriched_doc])
        
        # Add chunk-level attributes
        for i, chunk in enumerate(chunks):
            chunk.metadata["chunk_index"] = i
            chunk.metadata["total_chunks_in_doc"] = len(chunks)
            # Generate a stable doc_id from filename
            source_filename = chunk.metadata.get("source_filename", "unknown")
            chunk.metadata["doc_id"] = source_filename.replace(".pdf", "").lower()
        
        enriched_chunks.extend(chunks)
    
    return enriched_chunks

all_chunks = chunk_and_enrich(raw_docs, DOCUMENT_REGISTRY)
print(f"Total chunks: {len(all_chunks)}")
print(f"Sample metadata: {all_chunks[0].metadata}")

This approach is discussed further in Document Ingestion Pipelines: Loading, Cleaning, and Preprocessing Text for RAG and pairs naturally with the chunking strategies covered in Chunking Strategies: How to Split Documents for Better Retrieval.

Step 4: Storing enriched chunks in a vector store

With Chroma (a local vector store great for development), storing metadata is seamless:

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=all_chunks,
    embedding=embedding_model,
    collection_name="company_policies",
    persist_directory="./chroma_db",
)

vectorstore.persist()
print("Ingestion complete. Metadata attached to all chunks.")

Chroma stores the metadata dict verbatim alongside each embedding. When you query later, those fields are available for filtering.

---

Filtered Retrieval: Using Metadata at Query Time

Attaching metadata is only half the story. The payoff comes at query time, when you use those attributes to constrain what the vector search engine considers.

Pre-filtering vs. post-filtering

There are two ways to apply metadata constraints:

Pre-filtering (also called index-time filtering): The vector store discards non-matching documents before running approximate nearest-neighbor (ANN) search. Only the matching subset is searched. This is faster when the filter is highly selective (removes most of the index) but can degrade ANN quality if too few candidates remain.

Post-filtering: ANN search runs across the full index, then results are filtered afterward. This preserves ANN quality but means you may retrieve 100 candidates only to keep 2 — wasting compute and potentially returning too few results.

Most production vector stores (Pinecone, Weaviate, Qdrant) support pre-filtering with inverted indexes on metadata fields. Chroma also supports pre-filtering. For most use cases, pre-filtering is what you want — particularly when your metadata fields are high-cardinality enough to meaningfully narrow the search space. This topic is explored in depth in Metadata Filtering in RAG: Using Structured Attributes to Narrow Retrieval Before Vector Search.

Basic filtered retrieval in Chroma

# Query only current policy documents in the compliance department
results = vectorstore.similarity_search(
    query="transaction monitoring thresholds for wire transfers",
    k=5,
    filter={
        "$and": [
            {"document_type": {"$eq": "policy"}},
            {"department": {"$eq": "compliance"}},
            {"is_current_version": {"$eq": True}},
        ]
    }
)

for doc in results:
    print(f"Source: {doc.metadata['source_filename']}")
    print(f"Effective: {doc.metadata['effective_date']}")
    print(f"Content: {doc.page_content[:200]}\n")

Without the filter, this query might surface outdated policy versions or irrelevant engineering runbooks. With the filter, you've constrained the search to only current compliance policies before semantic similarity is even computed.

Faceted retrieval: filtering across multiple dimensions

Faceted retrieval is the practice of combining multiple independent filter dimensions simultaneously — analogous to the sidebar filters on an e-commerce site where you filter by category, price range, brand, and rating all at once. For RAG, this means applying compound filters that respect multiple metadata attributes.

def retrieve_with_facets(
    query: str,
    document_type: str = None,
    department: str = None,
    jurisdiction: str = None,
    product_line: str = None,
    current_only: bool = True,
    k: int = 5,
) -> list:
    """
    Perform faceted retrieval: build a dynamic filter from provided facets
    and run filtered semantic search.
    """
    conditions = []
    
    if document_type:
        conditions.append({"document_type": {"$eq": document_type}})
    if department:
        conditions.append({"department": {"$eq": department}})
    if jurisdiction:
        conditions.append({"jurisdiction": {"$eq": jurisdiction}})
    if product_line:
        conditions.append({"product_line": {"$eq": product_line}})
    if current_only:
        conditions.append({"is_current_version": {"$eq": True}})
    
    # Build the filter expression
    if len(conditions) == 0:
        search_filter = None
    elif len(conditions) == 1:
        search_filter = conditions[0]
    else:
        search_filter = {"$and": conditions}
    
    return vectorstore.similarity_search(
        query=query,
        k=k,
        filter=search_filter,
    )

# "What are the EU data retention requirements in our current policies?"
eu_policy_results = retrieve_with_facets(
    query="data retention requirements",
    document_type="policy",
    jurisdiction="EU",
    current_only=True,
    k=5,
)

# "What does the payments team's runbook say about incident escalation?"
payments_runbook_results = retrieve_with_facets(
    query="incident escalation procedure",
    document_type="runbook",
    department="engineering",
    product_line="payments",
    k=3,
)

This faceted pattern is what separates a system that feels like a competent assistant from one that feels like a broken search engine. The user doesn't have to say "only look at current EU policies" in their query text — the application layer knows the user's context (their role, the jurisdiction they're working in, the product they're responsible for) and translates that into structured filter constraints before the query even reaches the vector store.

---

Automated Metadata Extraction

Not every organization has a clean metadata registry mapping filenames to attributes. Real-world ingestion pipelines often deal with unstructured documents where attributes must be extracted from content.

LLM-assisted metadata extraction

You can use a fast, cheap LLM call to extract structured metadata from document headers, titles, and first pages:

import json
from openai import OpenAI

client = OpenAI()

def extract_metadata_with_llm(document_text: str) -> dict:
    """
    Use an LLM to extract structured metadata from the first 1000 chars of a document.
    """
    prompt = f"""Extract structured metadata from the following document excerpt.
Return a JSON object with these fields (use null if not found):
- document_type: one of ["policy", "runbook", "report", "contract", "faq", "other"]
- department: the responsible department or team
- effective_date: ISO 8601 date string (YYYY-MM-DD) or null
- jurisdiction: geographic jurisdiction (e.g., "US", "EU") or null
- author: author name or team name or null
- version: version number as integer or null

Document excerpt:
{document_text[:1000]}

Return only the JSON object, no explanation."""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    
    try:
        return json.loads(response.choices[0].message.content)
    except json.JSONDecodeError:
        return {}  # Graceful fallback — don't crash ingestion on extraction failure

Hybrid extraction: heuristics first, LLM as fallback

import re

def extract_date_from_filename(filename: str) -> str | None:
    """Try to parse a date from common filename patterns."""
    patterns = [
        r"(\d{4}-\d{2}-\d{2})",      # 2024-01-15
        r"(\d{4}_\d{2}_\d{2})",      # 2024_01_15
        r"_([A-Z][a-z]{2}\d{4})",    # Jan2024
        r"_Q([1-4])_(\d{4})",        # Q3_2023
    ]
    for pattern in patterns:
        match = re.search(pattern, filename)
        if match:
            return match.group(0).lstrip("_")
    return None

def extract_department_from_path(filepath: str) -> str | None:
    """Infer department from directory structure."""
    known_departments = ["compliance", "engineering", "sales", "legal", "hr", "finance"]
    path_lower = filepath.lower()
    for dept in known_departments:
        if dept in path_lower:
            return dept
    return None

This layered approach — heuristics for what's reliably parseable, LLM for what isn't — gives you the best combination of speed, cost, and accuracy.

---

Hands-On Exercise

You're building a RAG system for a legal team that manages contracts across multiple jurisdictions and practice areas. Here's what to do:

Step 1: Create a contracts/ directory with three mock text files:

• nda_us_acme_2024.txt — "This Non-Disclosure Agreement is effective January 1, 2024. Governing law: State of Delaware, USA. Parties: Acme Corp and Wicked Smart Data Inc. Confidentiality period: 3 years. Practice area: corporate."
• gdpr_data_processing_eu_2023.txt — "Data Processing Agreement under GDPR. Effective date: May 25, 2023. Jurisdiction: European Union. Practice area: privacy. This agreement governs the processing of personal data..."
• employment_contract_uk_2024.txt — "Employment Agreement. Effective: March 1, 2024. Jurisdiction: United Kingdom. Practice area: employment. Role: Senior Data Engineer. Probation period: 90 days."

Step 2: Write a metadata enrichment function that:

• Parses jurisdiction from the filename (look for _us_, _eu_, _uk_)
• Parses document_type from the filename prefix (nda, gdpr, employment)
• Parses year from the filename as an integer
• Marks all as classification: "confidential" and is_current_version: True

Step 3: Ingest these documents into a Chroma collection with the enriched metadata attached.

Step 4: Write three faceted queries:

1. All EU documents about data processing
2. All US documents
3. All 2024 documents across any jurisdiction

Step 5: Inspect the metadata on the returned chunks and verify the filters are working correctly.

This exercise will force you to think about the relationship between naming conventions, metadata schema, and query-time filter expressions.

---

Common Mistakes & Troubleshooting

Mistake 1: Storing metadata in the document text instead of the metadata dict

Some engineers concatenate metadata like "[SOURCE: AML Policy v3] [DEPT: Compliance]" into the chunk text itself, hoping the LLM will use it for context. This pollutes your embeddings (the model now encodes structural labels as semantic content), wastes tokens, and makes filtering impossible since the vector store can't filter on embedded strings. Keep metadata in the metadata dict, period.

Mistake 2: Using inconsistent field values

If some documents have department: "Compliance" and others have department: "compliance" or department: "COMPLIANCE", your equality filters will silently miss documents. Normalize all string metadata values to lowercase during ingestion. Add a validation step that checks against an allowed enum before writing to the store.

Mistake 3: Not planning for metadata updates

Documents get revised. Policies get superseded. If you mark v2 as is_current_version: True and later ingest v3, you need to update v2's record to is_current_version: False. Most vector stores don't support in-place metadata updates — you may need to delete and re-insert. Design your ingestion pipeline with this update lifecycle in mind from day one. See Indexing Strategies for RAG: How to Build and Update Your Vector Store as Documents Change for strategies.

Mistake 4: Over-filtering to the point of zero results

Compound filters on sparse metadata fields can return zero candidates, causing your RAG system to silently fail or fall back to "I don't know." Always implement a fallback: if filtered search returns fewer than k/2 results, either relax the filter (drop one condition at a time) or flag the retrieval as low-confidence. Log zero-result queries — they're a signal that your metadata schema is misaligned with real query patterns.

Mistake 5: Ignoring the metadata size limits of your vector store

Pinecone has a 40KB metadata limit per vector. If you embed large text blobs (like full section text) in the metadata dict, you'll hit this limit. Store only structured, filterable attributes in metadata. Long text belongs in the document content field or in a separate document store referenced by ID.

---

Summary & Next Steps

You've learned that metadata enrichment isn't an optional optimization — it's a foundational design decision that determines whether your RAG system can serve the complex, context-dependent queries that real users actually ask. By attaching structured document attributes at ingestion time, you enable:

• Pre-filtered retrieval that narrows the candidate pool before semantic search runs
• Faceted retrieval that combines multiple independent filter dimensions simultaneously
• Version-aware retrieval that surfaces only current documents when currency matters
• Permission-scoped retrieval as a building block for access control

The key decisions you need to make for any new RAG project are:

1. What attributes do users implicitly filter by when asking questions?
2. What's the source of truth for those attributes (registry, filename conventions, LLM extraction)?
3. How will you handle metadata updates when documents change?
4. What's your fallback when filters return too few results?

From here, the natural next topics to explore are how metadata filtering interacts with security and multi-tenancy — covered in Enterprise RAG: Security, Permissions, and Multi-Tenant Architecture — and how to combine metadata filtering with keyword search in Hybrid Search: Combining Keyword and Semantic Search for Better Results. Both of these build directly on the metadata infrastructure you've now learned to design and attach.