Wicked Smart Data
LearnInsightsAboutContact
Sign InLet's Build
LearnInsightsAboutContact
Sign InLet's Build
Wicked Smart Data

Intelligence, automation, and expert execution — plus an elite library of free knowledge. We turn complexity into competitive advantage.

Start a conversation

Platform

  • Learning Paths
  • Insights
  • RSS Feed

Company

  • About
  • Contact
  • Work With Us

Legal

  • Privacy Policy
  • Terms of Service

© 2026 Wicked Smart Data. All rights reserved.

Intelligence · Automation · Advantage

All Insights
AI & Machine Learning

Multimodal RAG: Retrieving and Reasoning Over Text, Tables, and Images in a Unified Pipeline

Most RAG pipelines silently ignore tables, charts, and diagrams — the parts of documents that often contain the most critical data. This lesson builds a complete multimodal RAG pipeline that extracts, indexes, and retrieves text, tables, and images together, then feeds them to a vision-capable LLM for accurate, grounded answers.

⚡ Practitioner21 min readAug 19, 2026Updated Sep 23, 2026
Multimodal RAG: Retrieving and Reasoning Over Text, Tables, and Images in a Unified Pipeline
On this page
  • Introduction
  • Prerequisites
  • Understanding the Multimodal Document Problem
  • Stage 1: Document Parsing and Extraction
  • Setting Up the Extractor
  • Precision Table Extraction with pdfplumber
  • Handling Images
  • Stage 2: The Two Indexing Strategies
  • Strategy A: Summary Embedding with Raw Content Storage
  • Strategy B: Direct CLIP Embedding (When to Use It)
  • Stage 3: The Retrieval and Generation Chain
  • Putting It All Together: The Full Pipeline
  • Hands-On Exercise: Annual Report Analysis System
  • Common Mistakes & Troubleshooting
  • Mistake 1: Tables Rendering as Garbage Text
  • Mistake 2: Retrieving Wrong Modalities
  • Mistake 3: Images Not Attached to LLM Calls
  • Mistake 4: Indexing Summaries but Retrieving Summaries (Double Summary Problem)
  • Mistake 5: Context Window Overflow
  • Performance Considerations and Production Readiness
  • Indexing Cost and Latency
  • Persistent Vector Storage
  • When Not to Use Multimodal RAG
  • Summary & Next Steps
  • What to Explore Next
  • Multimodal RAG: Retrieving and Reasoning Over Text, Tables, and Images in a Unified Pipeline

    Introduction

    You've built a solid RAG pipeline. It chunks your PDFs, embeds the text, retrieves relevant passages, and feeds them to an LLM. Then a stakeholder asks: "Can you also get the quarterly revenue figures from the table on page 12, and explain what's shown in the architecture diagram on page 7?" Your pipeline returns blank stares. The numbers are in a table that got mangled into a string of whitespace-separated digits. The diagram never made it into the index at all.

    This is the wall that practitioners hit when real-world documents show up. Annual reports, research papers, technical specifications, medical records, product catalogs — these aren't clean prose. They're layered compositions of text, tables, charts, schematics, and screenshots. A text-only RAG pipeline ignores anywhere from 20% to 80% of the meaningful information in those documents, depending on the domain. Multimodal RAG closes that gap by building a retrieval and reasoning pipeline that treats text, tables, and images as first-class citizens — each parsed appropriately, embedded in a way that preserves their semantics, and retrieved together when a query demands it.

    By the end of this lesson, you'll have built a working multimodal RAG pipeline from scratch. You'll understand not just the mechanics but the design tradeoffs at every stage — where to summarize vs. where to embed raw content, how to handle retrieval across modalities, and how to feed heterogeneous context to an LLM that can actually reason over it.

    What you'll learn:

    • How to extract and structure text, tables, and images from complex documents using unstructured and pdfplumber
    • Two distinct strategies for multimodal indexing: the summary-embedding approach vs. the direct image-embedding approach, and when to use each
    • How to build a unified vector store that retrieves across modalities using a docstore + retriever pattern
    • How to construct a multimodal reasoning chain that passes mixed context (text, table markdown, image data) to a vision-capable LLM
    • How to evaluate and debug multimodal retrieval failures

    Prerequisites

    You should be comfortable with:

    • Standard RAG pipelines (chunking, embedding, vector retrieval, generation)
    • Python at an intermediate level — we'll use LangChain, but you don't need to be an expert
    • Basic familiarity with LLMs and the concept of vision-language models (GPT-4o, Claude 3, Gemini)
    • A working environment with Python 3.10+

    You'll need API access to OpenAI (for embeddings and GPT-4o) or a comparable vision-capable LLM. We'll use unstructured, pdfplumber, langchain, chromadb, and Pillow. Install everything upfront:

    pip install unstructured[all-docs] pdfplumber langchain langchain-openai \
        langchain-chroma chromadb pillow open-clip-torch torch tiktoken
    

    Understanding the Multimodal Document Problem

    Before writing a single line of code, you need to understand what you're actually dealing with when a PDF or document arrives in your pipeline.

    A typical PDF is not a structured data format — it's a set of rendering instructions. Text may be stored as positioned character sequences with no paragraph structure. Tables may be encoded as a grid of text boxes with no relationship metadata. Images are embedded as binary blobs. When a standard PDF parser runs, it tries to reconstruct meaning from these primitives, and it frequently fails in interesting ways.

    Consider a financial report. You might have:

    1. Prose sections: Executive summary, MD&A, footnotes — these are well-suited to standard text chunking
    2. Data tables: Income statements, segment breakdowns, five-year summaries — these have a row-column structure where the relationships between cells carry the meaning
    3. Charts and graphs: Revenue trends, market share pie charts — the information is encoded visually and doesn't exist as text at all
    4. Embedded diagrams: Org charts, product architecture diagrams — similarly image-only
    5. Mixed-content pages: A page that's 60% text and 40% a figure with a caption

    Each of these requires a different extraction and representation strategy. The core insight of multimodal RAG is that you don't have to find a single representation that works for all of them — you build a pipeline that handles each correctly and then retrieves across all of them at query time.


    Stage 1: Document Parsing and Extraction

    The extraction stage is where most multimodal RAG systems either succeed or fail irreparably. Bad extraction means no downstream fix will save you.

    We'll use two tools in combination: unstructured for high-level document parsing and element classification, and pdfplumber for precision table extraction. This combination covers the majority of real-world PDF structures.

    Setting Up the Extractor

    import pdfplumber
    from unstructured.partition.pdf import partition_pdf
    from unstructured.documents.elements import Table, Image, CompositeElement, NarrativeText, Title
    from pathlib import Path
    import base64
    import io
    from PIL import Image as PILImage
    
    def extract_document_elements(pdf_path: str) -> dict:
        """
        Extract text, tables, and images from a PDF, returning a structured
        dictionary with each element type separated.
        """
        path = Path(pdf_path)
        
        # Use unstructured for overall document parsing
        # strategy="hi_res" uses a layout model to detect element boundaries
        # This is slower but dramatically more accurate for mixed-layout documents
        raw_elements = partition_pdf(
            filename=str(path),
            strategy="hi_res",
            infer_table_structure=True,
            extract_images_in_pdf=True,
            extract_image_block_types=["Image", "Table"],
            extract_image_block_output_dir=str(path.parent / "extracted_images"),
            chunking_strategy="by_title",
            max_characters=4000,
            new_after_n_chars=3800,
            combine_text_under_n_chars=2000,
        )
        
        text_elements = []
        table_elements = []
        image_elements = []
        
        for element in raw_elements:
            if isinstance(element, (NarrativeText, Title, CompositeElement)):
                # Filter out very short elements — usually headers or artifacts
                if len(str(element)) > 100:
                    text_elements.append({
                        "type": "text",
                        "content": str(element),
                        "metadata": element.metadata.to_dict() if hasattr(element, 'metadata') else {}
                    })
            elif isinstance(element, Table):
                table_elements.append({
                    "type": "table",
                    "content": element.metadata.text_as_html,
                    "text_content": str(element),
                    "metadata": element.metadata.to_dict() if hasattr(element, 'metadata') else {}
                })
        
        return {
            "text": text_elements,
            "tables": table_elements,
            "image_paths": list(Path(path.parent / "extracted_images").glob("*.png"))
                if (path.parent / "extracted_images").exists() else []
        }
    

    Why strategy="hi_res"? The default fast strategy uses PDF text layer extraction, which completely misses scanned documents and mis-orders text in complex layouts. hi_res runs a document layout model (similar to LayoutParser) that understands visual structure. It's 5-10x slower but produces dramatically better results for anything other than simple text-heavy PDFs.

    Precision Table Extraction with pdfplumber

    unstructured does a reasonable job with tables, but for financial or scientific documents where table accuracy is critical, pdfplumber's explicit table detection gives you better control:

    def extract_tables_with_pdfplumber(pdf_path: str) -> list[dict]:
        """
        Extract tables with precise cell-level structure using pdfplumber.
        Returns tables as both markdown strings and raw cell data.
        """
        tables = []
        
        with pdfplumber.open(pdf_path) as pdf:
            for page_num, page in enumerate(pdf.pages, start=1):
                # Find tables using pdfplumber's table detection algorithm
                # You can tune these settings for your document type
                table_settings = {
                    "vertical_strategy": "lines",
                    "horizontal_strategy": "lines",
                    "snap_tolerance": 3,
                    "join_tolerance": 3,
                }
                
                page_tables = page.extract_tables(table_settings)
                
                for table_idx, raw_table in enumerate(page_tables):
                    if not raw_table or len(raw_table) < 2:
                        continue
                        
                    # Convert to markdown for LLM consumption
                    markdown_table = table_to_markdown(raw_table)
                    
                    tables.append({
                        "type": "table",
                        "content": markdown_table,
                        "raw_cells": raw_table,
                        "page": page_num,
                        "table_index": table_idx,
                        "source": pdf_path
                    })
        
        return tables
    
    def table_to_markdown(table_data: list[list]) -> str:
        """Convert raw cell data to GitHub-flavored markdown table."""
        if not table_data:
            return ""
        
        # Clean None values
        cleaned = [
            [cell.strip() if cell else "" for cell in row]
            for row in table_data
        ]
        
        if len(cleaned) == 0:
            return ""
        
        # Header row
        header = "| " + " | ".join(cleaned[0]) + " |"
        separator = "| " + " | ".join(["---"] * len(cleaned[0])) + " |"
        
        # Data rows
        data_rows = [
            "| " + " | ".join(row) + " |"
            for row in cleaned[1:]
        ]
        
        return "\n".join([header, separator] + data_rows)
    

    Handling Images

    For images, you have two paths: extract them for visual embedding or generate textual summaries. We'll implement both and choose at indexing time:

    import openai
    import base64
    
    def encode_image_to_base64(image_path: str) -> str:
        """Encode an image file to base64 string for API transmission."""
        with open(image_path, "rb") as f:
            return base64.standard_b64encode(f.read()).decode("utf-8")
    
    def summarize_image_with_vision(
        image_path: str, 
        client: openai.OpenAI,
        context_hint: str = ""
    ) -> str:
        """
        Use GPT-4o to generate a rich textual description of an image.
        This description becomes the searchable representation of the image.
        """
        image_data = encode_image_to_base64(image_path)
        
        system_prompt = """You are an expert document analyst. When given an image from a 
        business or technical document, provide a comprehensive description that captures:
        1. The type of visual element (chart, diagram, photo, screenshot, etc.)
        2. All data, labels, legends, and values visible in the image
        3. The key insight or conclusion the visual conveys
        4. Any trends, comparisons, or relationships shown
        
        Be specific enough that someone could answer data questions using only your description."""
        
        user_prompt = f"Describe this document image in detail.{' Context: ' + context_hint if context_hint else ''}"
        
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": system_prompt},
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": user_prompt},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": f"data:image/png;base64,{image_data}",
                                "detail": "high"
                            }
                        }
                    ]
                }
            ],
            max_tokens=1000
        )
        
        return response.choices[0].message.content
    

    The summary trade-off: Generating summaries at index time costs money and time but produces excellent retrieval because the description is written in the same natural language as your queries. The alternative — using CLIP embeddings to embed images directly — is cheaper but retrieves based on visual similarity, which doesn't always align with semantic query intent. For document understanding tasks, summaries almost always outperform direct image embeddings.


    Stage 2: The Two Indexing Strategies

    Now that you have structured extractions, you need to index them for retrieval. There are two distinct architectural approaches, and understanding the tradeoff determines which you should use.

    Strategy A: Summary Embedding with Raw Content Storage

    This is the approach we'll build fully. The idea is elegant: you embed a summary or clean representation of each element (text chunk, table markdown, image description), but you store the original content in a separate docstore. The vector index finds the right elements; the docstore returns the full content to the LLM.

    This pattern is sometimes called "retrieval with rich context" — the retrieval signal is optimized for search, while the generation input is optimized for reasoning.

    from langchain_openai import OpenAIEmbeddings, ChatOpenAI
    from langchain_chroma import Chroma
    from langchain.storage import InMemoryStore
    from langchain.schema.document import Document
    from langchain.retrievers.multi_vector import MultiVectorRetriever
    import uuid
    
    def build_multimodal_index(
        text_elements: list[dict],
        table_elements: list[dict], 
        image_summaries: list[dict],  # {"summary": str, "image_path": str}
        collection_name: str = "multimodal_rag"
    ) -> MultiVectorRetriever:
        """
        Build a MultiVectorRetriever that stores summaries in the vector index
        and full content in an in-memory docstore.
        """
        
        embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
        vectorstore = Chroma(
            collection_name=collection_name,
            embedding_function=embeddings
        )
        docstore = InMemoryStore()
        
        retriever = MultiVectorRetriever(
            vectorstore=vectorstore,
            docstore=docstore,
            id_key="doc_id",
            search_kwargs={"k": 6}  # Retrieve more candidates in multimodal settings
        )
        
        # --- Index Text Elements ---
        text_docs = []
        text_summaries = []
        
        for element in text_elements:
            doc_id = str(uuid.uuid4())
            
            # The "summary" for text is the text itself (it's already natural language)
            summary_doc = Document(
                page_content=element["content"],
                metadata={
                    "doc_id": doc_id,
                    "type": "text",
                    "source": element["metadata"].get("filename", "unknown"),
                    "page": element["metadata"].get("page_number", 0)
                }
            )
            
            # The stored content is also the text (no transformation needed)
            full_doc = Document(
                page_content=element["content"],
                metadata={"type": "text", "doc_id": doc_id}
            )
            
            text_summaries.append(summary_doc)
            text_docs.append((doc_id, full_doc))
        
        # --- Index Table Elements ---
        table_docs = []
        table_summaries = []
        
        for element in table_elements:
            doc_id = str(uuid.uuid4())
            
            # Embed the markdown representation — it's human-readable and searchable
            summary_doc = Document(
                page_content=element["content"],  # markdown table
                metadata={
                    "doc_id": doc_id,
                    "type": "table",
                    "page": element.get("page", 0),
                    "source": element.get("source", "unknown")
                }
            )
            
            # Store the markdown table for direct LLM consumption
            full_doc = Document(
                page_content=f"[TABLE]\n{element['content']}\n[/TABLE]",
                metadata={"type": "table", "doc_id": doc_id}
            )
            
            table_summaries.append(summary_doc)
            table_docs.append((doc_id, full_doc))
        
        # --- Index Image Elements ---
        image_docs = []
        image_summary_docs = []
        
        for item in image_summaries:
            doc_id = str(uuid.uuid4())
            
            # Embed the vision-generated summary
            summary_doc = Document(
                page_content=item["summary"],
                metadata={
                    "doc_id": doc_id,
                    "type": "image",
                    "image_path": item["image_path"]
                }
            )
            
            # Store the image path + summary together
            # At generation time, we'll re-encode the image for the LLM
            full_doc = Document(
                page_content=item["summary"],
                metadata={
                    "type": "image",
                    "doc_id": doc_id,
                    "image_path": item["image_path"]
                }
            )
            
            image_summary_docs.append(summary_doc)
            image_docs.append((doc_id, full_doc))
        
        # Add everything to the retriever
        all_summary_docs = text_summaries + table_summaries + image_summary_docs
        all_full_docs = text_docs + table_docs + image_docs
        
        retriever.vectorstore.add_documents(all_summary_docs)
        retriever.docstore.mset(all_full_docs)
        
        return retriever
    

    Strategy B: Direct CLIP Embedding (When to Use It)

    For use cases where visual similarity matters — product catalogs, image search, brand monitoring — you'd embed images directly using CLIP and query with image inputs or image-text pairs. This is beyond our current scope, but here's a sketch:

    # When you'd use CLIP instead of summaries:
    # - You want to query with "find images similar to this image"
    # - Your images are photographs, not charts/diagrams
    # - You need sub-second retrieval without LLM summarization latency
    # - Visual aesthetics matter more than semantic content
    
    # The summary approach is almost always better for:
    # - Charts, graphs, diagrams, screenshots
    # - Any image where the data/labels matter for question answering
    # - When queries are text-based (which they almost always are)
    

    Stage 3: The Retrieval and Generation Chain

    With the index built, you need a chain that:

    1. Embeds the query
    2. Retrieves relevant documents across all modalities
    3. Constructs a mixed-modality prompt
    4. Calls a vision-capable LLM with the right inputs

    This is where many implementations fall apart — people retrieve documents correctly but then flatten everything to text for the LLM, throwing away the images they worked so hard to extract.

    from langchain_core.runnables import RunnableLambda, RunnablePassthrough
    from langchain_core.messages import HumanMessage
    from langchain_core.output_parsers import StrOutputParser
    
    def build_multimodal_chain(retriever: MultiVectorRetriever):
        """
        Build a chain that retrieves mixed-modality context and constructs
        a vision-capable LLM prompt with text, table, and image inputs.
        """
        
        llm = ChatOpenAI(model="gpt-4o", max_tokens=2000)
        
        def split_retrieved_docs(docs: list[Document]) -> dict:
            """
            Separate retrieved documents by type for different prompt handling.
            """
            text_parts = []
            table_parts = []
            image_parts = []
            
            for doc in docs:
                doc_type = doc.metadata.get("type", "text")
                if doc_type == "text":
                    text_parts.append(doc.page_content)
                elif doc_type == "table":
                    table_parts.append(doc.page_content)
                elif doc_type == "image":
                    image_parts.append({
                        "summary": doc.page_content,
                        "image_path": doc.metadata.get("image_path", "")
                    })
            
            return {
                "text_context": "\n\n---\n\n".join(text_parts),
                "table_context": "\n\n".join(table_parts),
                "images": image_parts
            }
        
        def build_prompt(inputs: dict) -> list:
            """
            Construct the multimodal message list for the LLM.
            GPT-4o accepts interleaved text and image_url content blocks.
            """
            query = inputs["query"]
            context = inputs["context"]
            
            # Build the text portion of the prompt
            system_text = """You are an expert analyst with access to document excerpts 
            that may include text passages, data tables, and images. Answer the question 
            using all provided context. When referencing tables, be precise about values. 
            When referencing images, describe what visual evidence supports your answer.
            If the context doesn't contain enough information, say so clearly."""
            
            # Compose context string
            context_parts = []
            
            if context["text_context"]:
                context_parts.append(f"## Text Passages\n\n{context['text_context']}")
            
            if context["table_context"]:
                context_parts.append(f"## Data Tables\n\n{context['table_context']}")
            
            if context["images"]:
                context_parts.append(
                    f"## Images\n\n{len(context['images'])} image(s) attached below. "
                    f"Image summaries:\n" + 
                    "\n".join([f"- {img['summary'][:200]}..." for img in context["images"]])
                )
            
            full_context = "\n\n".join(context_parts)
            
            # Build the content list with interleaved text and images
            content = [
                {
                    "type": "text",
                    "text": f"{system_text}\n\n## Retrieved Context\n\n{full_context}\n\n## Question\n\n{query}"
                }
            ]
            
            # Attach actual images for visual reasoning
            for img_data in context["images"]:
                if img_data["image_path"] and Path(img_data["image_path"]).exists():
                    try:
                        img_b64 = encode_image_to_base64(img_data["image_path"])
                        content.append({
                            "type": "image_url",
                            "image_url": {
                                "url": f"data:image/png;base64,{img_b64}",
                                "detail": "high"
                            }
                        })
                    except Exception as e:
                        print(f"Warning: Could not attach image {img_data['image_path']}: {e}")
            
            return [HumanMessage(content=content)]
        
        # Compose the chain
        chain = (
            {
                "query": RunnablePassthrough(),
                "context": retriever | RunnableLambda(split_retrieved_docs)
            }
            | RunnableLambda(build_prompt)
            | llm
            | StrOutputParser()
        )
        
        return chain
    

    Putting It All Together: The Full Pipeline

    Here's the complete orchestration function that ties every stage together. This is the code you'd actually run against a new document:

    def build_multimodal_rag_pipeline(pdf_path: str) -> callable:
        """
        Full pipeline: parse document → extract elements → index → build chain.
        Returns a callable that accepts a question and returns an answer.
        """
        client = openai.OpenAI()
        
        print(f"[1/4] Extracting document elements from {pdf_path}...")
        elements = extract_document_elements(pdf_path)
        tables = extract_tables_with_pdfplumber(pdf_path)
        
        print(f"      Found: {len(elements['text'])} text chunks, "
              f"{len(tables)} tables, "
              f"{len(elements['image_paths'])} images")
        
        print("[2/4] Generating image summaries...")
        image_summaries = []
        for img_path in elements["image_paths"]:
            # Skip tiny images — likely logos or artifacts, not meaningful figures
            img = PILImage.open(img_path)
            if img.width < 100 or img.height < 100:
                continue
            
            print(f"      Summarizing {img_path.name}...")
            summary = summarize_image_with_vision(str(img_path), client)
            image_summaries.append({
                "summary": summary,
                "image_path": str(img_path)
            })
        
        print(f"      Summarized {len(image_summaries)} meaningful images")
        
        print("[3/4] Building multimodal vector index...")
        # Combine tables from both extraction methods (prefer pdfplumber for structure)
        all_tables = tables if tables else elements["tables"]
        
        retriever = build_multimodal_index(
            text_elements=elements["text"],
            table_elements=all_tables,
            image_summaries=image_summaries
        )
        
        print("[4/4] Assembling generation chain...")
        chain = build_multimodal_chain(retriever)
        
        print("Pipeline ready. Ask away.\n")
        return chain
    
    # Usage
    pipeline = build_multimodal_rag_pipeline("annual_report_2024.pdf")
    
    answer = pipeline.invoke(
        "What was the YoY revenue growth by segment, and does the chart on page 8 show any anomalies?"
    )
    print(answer)
    

    Hands-On Exercise: Annual Report Analysis System

    Now let's apply this to a real scenario. Download a publicly available annual report — the Apple 10-K or any company's SEC filing works well. These documents are ideal test cases because they contain all three modalities in meaningful proportions.

    Exercise goals:

    1. Run the full pipeline on a real document
    2. Test retrieval quality across modalities
    3. Identify where retrieval succeeds and fails

    Step 1: Download and prepare your document

    # Download Apple's 2023 10-K (publicly available)
    curl -o apple_10k_2023.pdf \
      "https://s2.q4cdn.com/470004039/files/doc_earnings/2023/ar/_10-k-2023-(as-filed).pdf"
    

    Step 2: Run extraction and check what you got

    elements = extract_document_elements("apple_10k_2023.pdf")
    tables = extract_tables_with_pdfplumber("apple_10k_2023.pdf")
    
    # Inspect what was found
    print(f"Text chunks: {len(elements['text'])}")
    print(f"Tables: {len(tables)}")
    print(f"Images: {len(elements['image_paths'])}")
    
    # Look at a table to verify extraction quality
    print("\nSample table (first found):")
    print(tables[0]["content"][:500] if tables else "No tables found")
    
    # Check an extracted image exists
    if elements["image_paths"]:
        print(f"\nFirst extracted image: {elements['image_paths'][0]}")
    

    Step 3: Build the pipeline and run targeted queries

    pipeline = build_multimodal_rag_pipeline("apple_10k_2023.pdf")
    
    # Test 1: Pure text retrieval
    q1 = "What are Apple's primary risk factors related to supply chain?"
    print("Q1:", pipeline.invoke(q1))
    
    # Test 2: Table retrieval
    q2 = "What were Apple's net sales by product category for fiscal year 2023?"
    print("Q2:", pipeline.invoke(q2))
    
    # Test 3: Cross-modal reasoning
    q3 = "Summarize Apple's revenue trends and explain any visualizations that support the narrative."
    print("Q3:", pipeline.invoke(q3))
    

    Step 4: Evaluate retrieval quality

    # Check what was actually retrieved for a given query
    from langchain_core.runnables import RunnablePassthrough
    
    def inspect_retrieval(retriever, query: str):
        """See exactly what documents are being retrieved for a query."""
        docs = retriever.invoke(query)
        
        print(f"Query: {query}")
        print(f"Retrieved {len(docs)} documents:")
        print("-" * 50)
        
        for i, doc in enumerate(docs, 1):
            doc_type = doc.metadata.get("type", "unknown")
            print(f"\n[{i}] Type: {doc_type.upper()}")
            print(f"    Content preview: {doc.page_content[:200]}...")
        
        return docs
    
    # Run inspection
    retrieved = inspect_retrieval(
        retriever, 
        "What was Apple's gross margin percentage?"
    )
    

    What to look for: You should see a mix of types returned. If every result is type "text" when you're asking about financial figures, your table extraction likely failed and needs debugging. If image summaries never appear, check that your image extraction directory isn't empty.


    Common Mistakes & Troubleshooting

    Mistake 1: Tables Rendering as Garbage Text

    Symptom: Your "tables" look like "Revenue 94052 85777 97278 Product Services" — a flattened string with no structure.

    Cause: You're using fast-mode PDF extraction, which reads the text layer left-to-right without understanding column structure.

    Fix: Switch to strategy="hi_res" in partition_pdf, or use pdfplumber's extract_tables() which detects cell boundaries from line geometry.

    # Check if table extraction worked
    for table in tables[:3]:
        lines = table["content"].split("\n")
        has_separator = any("---" in line for line in lines)
        if not has_separator:
            print("WARNING: Table may not have been parsed correctly")
            print("Raw content:", table["content"][:300])
    

    Mistake 2: Retrieving Wrong Modalities

    Symptom: Queries about financial data return image summaries; queries about charts return prose text.

    Cause: All your embeddings are in the same vector space, and the similarity landscape doesn't match the modality boundaries you expect.

    Fix: Add modality metadata to your documents and use filtered retrieval when you have strong signal about what type of content a query needs:

    # Metadata-filtered retrieval for explicit table queries
    from langchain_core.vectorstores import VectorStoreRetriever
    
    table_retriever = retriever.vectorstore.as_retriever(
        search_kwargs={
            "k": 4,
            "filter": {"type": "table"}
        }
    )
    
    # Use the filtered retriever for clearly quantitative queries
    # Use the full retriever for ambiguous queries
    

    Mistake 3: Images Not Attached to LLM Calls

    Symptom: The LLM says "Based on the image summary..." instead of actually reasoning over the image. Or it describes something inconsistent with what's actually in the image.

    Cause: You're passing the summary text to the LLM but not the actual image bytes. The LLM is reasoning from your summary, not from the visual.

    Fix: Always re-encode and attach the image at generation time. Check the build_prompt function — verify that content has both text blocks and image_url blocks before sending:

    def debug_prompt_content(content: list) -> None:
        """Quick diagnostic to verify prompt contains actual images."""
        text_blocks = sum(1 for block in content if block.get("type") == "text")
        image_blocks = sum(1 for block in content if block.get("type") == "image_url")
        print(f"Prompt contains: {text_blocks} text blocks, {image_blocks} image blocks")
    

    Mistake 4: Indexing Summaries but Retrieving Summaries (Double Summary Problem)

    Symptom: The LLM produces correct-sounding but vague answers — it seems to be reasoning from summaries of summaries.

    Cause: In build_multimodal_index, if you store the summary as both the vector content and the docstore content, the retriever returns the summary at generation time instead of the original content.

    Fix: The docstore content should always be the original (the raw markdown table, the full image path + original summary). Only the vector embedding should use the summary or clean representation.

    Mistake 5: Context Window Overflow

    Symptom: API errors about token limits, or truncated responses that cut off mid-answer.

    Cause: Retrieving 6 documents that each contain a large table or high-detail image description can easily hit 32K tokens.

    Fix: Be strategic about search_kwargs["k"]. For multimodal retrieval, 4-6 total documents is usually right. Also truncate table content if it's very large:

    def truncate_table_for_context(table_markdown: str, max_rows: int = 20) -> str:
        """Keep first N data rows of large tables to control token usage."""
        lines = table_markdown.split("\n")
        if len(lines) <= max_rows + 2:  # header + separator + data rows
            return table_markdown
        
        header_lines = lines[:2]  # header + separator
        data_lines = lines[2:max_rows + 2]
        truncation_note = f"| ... | (table truncated, {len(lines) - max_rows - 2} more rows) |"
        
        return "\n".join(header_lines + data_lines + [truncation_note])
    

    Performance Considerations and Production Readiness

    Indexing Cost and Latency

    For a 100-page document with 20 images, expect:

    • Extraction: 5-15 minutes (hi_res strategy runs an ML model per page)
    • Image summarization: ~1-2 minutes at ~$0.01-0.03 per image with GPT-4o
    • Embedding: ~30 seconds for the full document's text and table elements

    This means multimodal RAG is a batch indexing operation, not a real-time one. Build your pipeline with this in mind — index documents on ingest, store the index persistently, and serve queries from the pre-built index.

    Persistent Vector Storage

    For production, swap Chroma with a persistent vector store and replace InMemoryStore with a Redis or PostgreSQL-backed docstore:

    # Persistent Chroma
    vectorstore = Chroma(
        collection_name=collection_name,
        embedding_function=embeddings,
        persist_directory="./chroma_multimodal_db"
    )
    
    # For the docstore in production, consider:
    from langchain.storage import RedisStore
    docstore = RedisStore(redis_url="redis://localhost:6379", key_prefix="multimodal_rag:")
    

    When Not to Use Multimodal RAG

    Multimodal RAG adds real complexity. You should use it when:

    • Your documents are genuinely mixed-modality and the non-text content contains critical information
    • Users will ask questions that require data from tables or interpretation of visuals
    • You have a budget for image summarization at index time

    You should not use it when:

    • Your documents are primarily text (standard RAG is simpler and just as good)
    • Images are decorative (logos, stock photos) and carry no informational content
    • Latency is critical — multimodal pipelines are inherently slower due to image processing

    Summary & Next Steps

    You've built a complete multimodal RAG pipeline that handles the full spectrum of document content. Let's recap the key architectural decisions you made and why they matter:

    Extraction: You used hi_res strategy and pdfplumber together — the former for layout understanding, the latter for precision table cell detection. This combination handles the majority of real-world document structures.

    Indexing strategy: You chose summary-embedding over direct image embedding. Summaries written in natural language retrieve better against text queries, which is what your users will send 95% of the time.

    Retrieval pattern: The MultiVectorRetriever pattern — separate vector index from docstore — gives you the best of both worlds: optimized search representations and full-fidelity content for generation.

    Generation: You constructed interleaved text+image prompts for GPT-4o, which actually looks at images rather than just reading your summaries about them.

    What to Explore Next

    1. Agentic multimodal retrieval: Instead of a single retrieval call, build an agent that can decide which modality to query based on the question — reducing noise in retrieved context

    2. ColPali and late interaction models: These models embed document page images directly and support query-document interactions at the patch level — a completely different architecture that eliminates the extraction stage entirely

    3. Structured output from table reasoning: Use LLM function calling to return table data as structured JSON rather than natural language, enabling downstream computation on retrieved values

    4. Evaluation frameworks: Set up RAGAS or a custom evaluation harness to measure retrieval recall by modality — you may find that table retrieval lags text retrieval and needs a domain-specific re-ranker

    The multimodal document understanding space is moving fast. ColPali-style approaches in particular threaten to make extraction-based pipelines obsolete for many use cases. Understanding both the extraction-based architecture you built here and where it breaks down will help you evaluate when to adopt the next generation of approaches.

    Work With Us

    From insight to implementation

    Reading is the start. When you're ready to build the data, automation, or AI systems behind it, our team turns strategy into shipped results.

    Let's Build

    RAG & AI Agents

    Previous

    Indexing Strategies for RAG: How to Build and Update Your Vector Store as Documents Change

    Next

    Guardrails for RAG Pipelines: Implementing Input Validation, Output Filtering, and Policy Enforcement in Production

    Related Insights

    AI & Machine LearningExpert

    Implementing Plan-and-Execute Agents: Separating Task Planning from Execution to Build Reliable Long-Horizon LLM Workflows

    31 min
    AI & Machine LearningPractitioner

    Building a LangGraph Agent with Persistent State: Implementing Nodes, Edges, and Conditional Branching for Multi-Step AI Workflows

    22 min
    AI & Machine LearningPractitioner

    Tone and Voice Control in AI Prompts: How to Adjust Formality, Style, and Persona for Different Business Audiences

    24 min

    On this page

    • Introduction
    • Prerequisites
    • Understanding the Multimodal Document Problem
    • Stage 1: Document Parsing and Extraction
    • Setting Up the Extractor
    • Precision Table Extraction with pdfplumber
    • Handling Images
    • Stage 2: The Two Indexing Strategies
    • Strategy A: Summary Embedding with Raw Content Storage
    • Strategy B: Direct CLIP Embedding (When to Use It)
    • Stage 3: The Retrieval and Generation Chain
    • Putting It All Together: The Full Pipeline
    • Hands-On Exercise: Annual Report Analysis System
    • Common Mistakes & Troubleshooting
    • Mistake 1: Tables Rendering as Garbage Text
    • Mistake 2: Retrieving Wrong Modalities
    • Mistake 3: Images Not Attached to LLM Calls
    • Mistake 4: Indexing Summaries but Retrieving Summaries (Double Summary Problem)
    • Mistake 5: Context Window Overflow
    • Performance Considerations and Production Readiness
    • Indexing Cost and Latency
    • Persistent Vector Storage
    • When Not to Use Multimodal RAG
    • Summary & Next Steps
    • What to Explore Next