Most RAG pipelines silently ignore tables, charts, and diagrams — the parts of documents that often contain the most critical data. This lesson builds a complete multimodal RAG pipeline that extracts, indexes, and retrieves text, tables, and images together, then feeds them to a vision-capable LLM for accurate, grounded answers.
You've built a solid RAG pipeline. It chunks your PDFs, embeds the text, retrieves relevant passages, and feeds them to an LLM. Then a stakeholder asks: "Can you also get the quarterly revenue figures from the table on page 12, and explain what's shown in the architecture diagram on page 7?" Your pipeline returns blank stares. The numbers are in a table that got mangled into a string of whitespace-separated digits. The diagram never made it into the index at all.
This is the wall that practitioners hit when real-world documents show up. Annual reports, research papers, technical specifications, medical records, product catalogs — these aren't clean prose. They're layered compositions of text, tables, charts, schematics, and screenshots. A text-only RAG pipeline ignores anywhere from 20% to 80% of the meaningful information in those documents, depending on the domain. Multimodal RAG closes that gap by building a retrieval and reasoning pipeline that treats text, tables, and images as first-class citizens — each parsed appropriately, embedded in a way that preserves their semantics, and retrieved together when a query demands it.
By the end of this lesson, you'll have built a working multimodal RAG pipeline from scratch. You'll understand not just the mechanics but the design tradeoffs at every stage — where to summarize vs. where to embed raw content, how to handle retrieval across modalities, and how to feed heterogeneous context to an LLM that can actually reason over it.
What you'll learn:
unstructured and pdfplumberYou should be comfortable with:
You'll need API access to OpenAI (for embeddings and GPT-4o) or a comparable vision-capable LLM. We'll use unstructured, pdfplumber, langchain, chromadb, and Pillow. Install everything upfront:
pip install unstructured[all-docs] pdfplumber langchain langchain-openai \
langchain-chroma chromadb pillow open-clip-torch torch tiktoken
Before writing a single line of code, you need to understand what you're actually dealing with when a PDF or document arrives in your pipeline.
A typical PDF is not a structured data format — it's a set of rendering instructions. Text may be stored as positioned character sequences with no paragraph structure. Tables may be encoded as a grid of text boxes with no relationship metadata. Images are embedded as binary blobs. When a standard PDF parser runs, it tries to reconstruct meaning from these primitives, and it frequently fails in interesting ways.
Consider a financial report. You might have:
Each of these requires a different extraction and representation strategy. The core insight of multimodal RAG is that you don't have to find a single representation that works for all of them — you build a pipeline that handles each correctly and then retrieves across all of them at query time.
The extraction stage is where most multimodal RAG systems either succeed or fail irreparably. Bad extraction means no downstream fix will save you.
We'll use two tools in combination: unstructured for high-level document parsing and element classification, and pdfplumber for precision table extraction. This combination covers the majority of real-world PDF structures.
import pdfplumber
from unstructured.partition.pdf import partition_pdf
from unstructured.documents.elements import Table, Image, CompositeElement, NarrativeText, Title
from pathlib import Path
import base64
import io
from PIL import Image as PILImage
def extract_document_elements(pdf_path: str) -> dict:
"""
Extract text, tables, and images from a PDF, returning a structured
dictionary with each element type separated.
"""
path = Path(pdf_path)
# Use unstructured for overall document parsing
# strategy="hi_res" uses a layout model to detect element boundaries
# This is slower but dramatically more accurate for mixed-layout documents
raw_elements = partition_pdf(
filename=str(path),
strategy="hi_res",
infer_table_structure=True,
extract_images_in_pdf=True,
extract_image_block_types=["Image", "Table"],
extract_image_block_output_dir=str(path.parent / "extracted_images"),
chunking_strategy="by_title",
max_characters=4000,
new_after_n_chars=3800,
combine_text_under_n_chars=2000,
)
text_elements = []
table_elements = []
image_elements = []
for element in raw_elements:
if isinstance(element, (NarrativeText, Title, CompositeElement)):
# Filter out very short elements — usually headers or artifacts
if len(str(element)) > 100:
text_elements.append({
"type": "text",
"content": str(element),
"metadata": element.metadata.to_dict() if hasattr(element, 'metadata') else {}
})
elif isinstance(element, Table):
table_elements.append({
"type": "table",
"content": element.metadata.text_as_html,
"text_content": str(element),
"metadata": element.metadata.to_dict() if hasattr(element, 'metadata') else {}
})
return {
"text": text_elements,
"tables": table_elements,
"image_paths": list(Path(path.parent / "extracted_images").glob("*.png"))
if (path.parent / "extracted_images").exists() else []
}
Why
strategy="hi_res"? The default fast strategy uses PDF text layer extraction, which completely misses scanned documents and mis-orders text in complex layouts.hi_resruns a document layout model (similar to LayoutParser) that understands visual structure. It's 5-10x slower but produces dramatically better results for anything other than simple text-heavy PDFs.
unstructured does a reasonable job with tables, but for financial or scientific documents where table accuracy is critical, pdfplumber's explicit table detection gives you better control:
def extract_tables_with_pdfplumber(pdf_path: str) -> list[dict]:
"""
Extract tables with precise cell-level structure using pdfplumber.
Returns tables as both markdown strings and raw cell data.
"""
tables = []
with pdfplumber.open(pdf_path) as pdf:
for page_num, page in enumerate(pdf.pages, start=1):
# Find tables using pdfplumber's table detection algorithm
# You can tune these settings for your document type
table_settings = {
"vertical_strategy": "lines",
"horizontal_strategy": "lines",
"snap_tolerance": 3,
"join_tolerance": 3,
}
page_tables = page.extract_tables(table_settings)
for table_idx, raw_table in enumerate(page_tables):
if not raw_table or len(raw_table) < 2:
continue
# Convert to markdown for LLM consumption
markdown_table = table_to_markdown(raw_table)
tables.append({
"type": "table",
"content": markdown_table,
"raw_cells": raw_table,
"page": page_num,
"table_index": table_idx,
"source": pdf_path
})
return tables
def table_to_markdown(table_data: list[list]) -> str:
"""Convert raw cell data to GitHub-flavored markdown table."""
if not table_data:
return ""
# Clean None values
cleaned = [
[cell.strip() if cell else "" for cell in row]
for row in table_data
]
if len(cleaned) == 0:
return ""
# Header row
header = "| " + " | ".join(cleaned[0]) + " |"
separator = "| " + " | ".join(["---"] * len(cleaned[0])) + " |"
# Data rows
data_rows = [
"| " + " | ".join(row) + " |"
for row in cleaned[1:]
]
return "\n".join([header, separator] + data_rows)
For images, you have two paths: extract them for visual embedding or generate textual summaries. We'll implement both and choose at indexing time:
import openai
import base64
def encode_image_to_base64(image_path: str) -> str:
"""Encode an image file to base64 string for API transmission."""
with open(image_path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def summarize_image_with_vision(
image_path: str,
client: openai.OpenAI,
context_hint: str = ""
) -> str:
"""
Use GPT-4o to generate a rich textual description of an image.
This description becomes the searchable representation of the image.
"""
image_data = encode_image_to_base64(image_path)
system_prompt = """You are an expert document analyst. When given an image from a
business or technical document, provide a comprehensive description that captures:
1. The type of visual element (chart, diagram, photo, screenshot, etc.)
2. All data, labels, legends, and values visible in the image
3. The key insight or conclusion the visual conveys
4. Any trends, comparisons, or relationships shown
Be specific enough that someone could answer data questions using only your description."""
user_prompt = f"Describe this document image in detail.{' Context: ' + context_hint if context_hint else ''}"
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "text", "text": user_prompt},
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{image_data}",
"detail": "high"
}
}
]
}
],
max_tokens=1000
)
return response.choices[0].message.content
The summary trade-off: Generating summaries at index time costs money and time but produces excellent retrieval because the description is written in the same natural language as your queries. The alternative — using CLIP embeddings to embed images directly — is cheaper but retrieves based on visual similarity, which doesn't always align with semantic query intent. For document understanding tasks, summaries almost always outperform direct image embeddings.
Now that you have structured extractions, you need to index them for retrieval. There are two distinct architectural approaches, and understanding the tradeoff determines which you should use.
This is the approach we'll build fully. The idea is elegant: you embed a summary or clean representation of each element (text chunk, table markdown, image description), but you store the original content in a separate docstore. The vector index finds the right elements; the docstore returns the full content to the LLM.
This pattern is sometimes called "retrieval with rich context" — the retrieval signal is optimized for search, while the generation input is optimized for reasoning.
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain.storage import InMemoryStore
from langchain.schema.document import Document
from langchain.retrievers.multi_vector import MultiVectorRetriever
import uuid
def build_multimodal_index(
text_elements: list[dict],
table_elements: list[dict],
image_summaries: list[dict], # {"summary": str, "image_path": str}
collection_name: str = "multimodal_rag"
) -> MultiVectorRetriever:
"""
Build a MultiVectorRetriever that stores summaries in the vector index
and full content in an in-memory docstore.
"""
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Chroma(
collection_name=collection_name,
embedding_function=embeddings
)
docstore = InMemoryStore()
retriever = MultiVectorRetriever(
vectorstore=vectorstore,
docstore=docstore,
id_key="doc_id",
search_kwargs={"k": 6} # Retrieve more candidates in multimodal settings
)
# --- Index Text Elements ---
text_docs = []
text_summaries = []
for element in text_elements:
doc_id = str(uuid.uuid4())
# The "summary" for text is the text itself (it's already natural language)
summary_doc = Document(
page_content=element["content"],
metadata={
"doc_id": doc_id,
"type": "text",
"source": element["metadata"].get("filename", "unknown"),
"page": element["metadata"].get("page_number", 0)
}
)
# The stored content is also the text (no transformation needed)
full_doc = Document(
page_content=element["content"],
metadata={"type": "text", "doc_id": doc_id}
)
text_summaries.append(summary_doc)
text_docs.append((doc_id, full_doc))
# --- Index Table Elements ---
table_docs = []
table_summaries = []
for element in table_elements:
doc_id = str(uuid.uuid4())
# Embed the markdown representation — it's human-readable and searchable
summary_doc = Document(
page_content=element["content"], # markdown table
metadata={
"doc_id": doc_id,
"type": "table",
"page": element.get("page", 0),
"source": element.get("source", "unknown")
}
)
# Store the markdown table for direct LLM consumption
full_doc = Document(
page_content=f"[TABLE]\n{element['content']}\n[/TABLE]",
metadata={"type": "table", "doc_id": doc_id}
)
table_summaries.append(summary_doc)
table_docs.append((doc_id, full_doc))
# --- Index Image Elements ---
image_docs = []
image_summary_docs = []
for item in image_summaries:
doc_id = str(uuid.uuid4())
# Embed the vision-generated summary
summary_doc = Document(
page_content=item["summary"],
metadata={
"doc_id": doc_id,
"type": "image",
"image_path": item["image_path"]
}
)
# Store the image path + summary together
# At generation time, we'll re-encode the image for the LLM
full_doc = Document(
page_content=item["summary"],
metadata={
"type": "image",
"doc_id": doc_id,
"image_path": item["image_path"]
}
)
image_summary_docs.append(summary_doc)
image_docs.append((doc_id, full_doc))
# Add everything to the retriever
all_summary_docs = text_summaries + table_summaries + image_summary_docs
all_full_docs = text_docs + table_docs + image_docs
retriever.vectorstore.add_documents(all_summary_docs)
retriever.docstore.mset(all_full_docs)
return retriever
For use cases where visual similarity matters — product catalogs, image search, brand monitoring — you'd embed images directly using CLIP and query with image inputs or image-text pairs. This is beyond our current scope, but here's a sketch:
# When you'd use CLIP instead of summaries:
# - You want to query with "find images similar to this image"
# - Your images are photographs, not charts/diagrams
# - You need sub-second retrieval without LLM summarization latency
# - Visual aesthetics matter more than semantic content
# The summary approach is almost always better for:
# - Charts, graphs, diagrams, screenshots
# - Any image where the data/labels matter for question answering
# - When queries are text-based (which they almost always are)
With the index built, you need a chain that:
This is where many implementations fall apart — people retrieve documents correctly but then flatten everything to text for the LLM, throwing away the images they worked so hard to extract.
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_core.messages import HumanMessage
from langchain_core.output_parsers import StrOutputParser
def build_multimodal_chain(retriever: MultiVectorRetriever):
"""
Build a chain that retrieves mixed-modality context and constructs
a vision-capable LLM prompt with text, table, and image inputs.
"""
llm = ChatOpenAI(model="gpt-4o", max_tokens=2000)
def split_retrieved_docs(docs: list[Document]) -> dict:
"""
Separate retrieved documents by type for different prompt handling.
"""
text_parts = []
table_parts = []
image_parts = []
for doc in docs:
doc_type = doc.metadata.get("type", "text")
if doc_type == "text":
text_parts.append(doc.page_content)
elif doc_type == "table":
table_parts.append(doc.page_content)
elif doc_type == "image":
image_parts.append({
"summary": doc.page_content,
"image_path": doc.metadata.get("image_path", "")
})
return {
"text_context": "\n\n---\n\n".join(text_parts),
"table_context": "\n\n".join(table_parts),
"images": image_parts
}
def build_prompt(inputs: dict) -> list:
"""
Construct the multimodal message list for the LLM.
GPT-4o accepts interleaved text and image_url content blocks.
"""
query = inputs["query"]
context = inputs["context"]
# Build the text portion of the prompt
system_text = """You are an expert analyst with access to document excerpts
that may include text passages, data tables, and images. Answer the question
using all provided context. When referencing tables, be precise about values.
When referencing images, describe what visual evidence supports your answer.
If the context doesn't contain enough information, say so clearly."""
# Compose context string
context_parts = []
if context["text_context"]:
context_parts.append(f"## Text Passages\n\n{context['text_context']}")
if context["table_context"]:
context_parts.append(f"## Data Tables\n\n{context['table_context']}")
if context["images"]:
context_parts.append(
f"## Images\n\n{len(context['images'])} image(s) attached below. "
f"Image summaries:\n" +
"\n".join([f"- {img['summary'][:200]}..." for img in context["images"]])
)
full_context = "\n\n".join(context_parts)
# Build the content list with interleaved text and images
content = [
{
"type": "text",
"text": f"{system_text}\n\n## Retrieved Context\n\n{full_context}\n\n## Question\n\n{query}"
}
]
# Attach actual images for visual reasoning
for img_data in context["images"]:
if img_data["image_path"] and Path(img_data["image_path"]).exists():
try:
img_b64 = encode_image_to_base64(img_data["image_path"])
content.append({
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{img_b64}",
"detail": "high"
}
})
except Exception as e:
print(f"Warning: Could not attach image {img_data['image_path']}: {e}")
return [HumanMessage(content=content)]
# Compose the chain
chain = (
{
"query": RunnablePassthrough(),
"context": retriever | RunnableLambda(split_retrieved_docs)
}
| RunnableLambda(build_prompt)
| llm
| StrOutputParser()
)
return chain
Here's the complete orchestration function that ties every stage together. This is the code you'd actually run against a new document:
def build_multimodal_rag_pipeline(pdf_path: str) -> callable:
"""
Full pipeline: parse document → extract elements → index → build chain.
Returns a callable that accepts a question and returns an answer.
"""
client = openai.OpenAI()
print(f"[1/4] Extracting document elements from {pdf_path}...")
elements = extract_document_elements(pdf_path)
tables = extract_tables_with_pdfplumber(pdf_path)
print(f" Found: {len(elements['text'])} text chunks, "
f"{len(tables)} tables, "
f"{len(elements['image_paths'])} images")
print("[2/4] Generating image summaries...")
image_summaries = []
for img_path in elements["image_paths"]:
# Skip tiny images — likely logos or artifacts, not meaningful figures
img = PILImage.open(img_path)
if img.width < 100 or img.height < 100:
continue
print(f" Summarizing {img_path.name}...")
summary = summarize_image_with_vision(str(img_path), client)
image_summaries.append({
"summary": summary,
"image_path": str(img_path)
})
print(f" Summarized {len(image_summaries)} meaningful images")
print("[3/4] Building multimodal vector index...")
# Combine tables from both extraction methods (prefer pdfplumber for structure)
all_tables = tables if tables else elements["tables"]
retriever = build_multimodal_index(
text_elements=elements["text"],
table_elements=all_tables,
image_summaries=image_summaries
)
print("[4/4] Assembling generation chain...")
chain = build_multimodal_chain(retriever)
print("Pipeline ready. Ask away.\n")
return chain
# Usage
pipeline = build_multimodal_rag_pipeline("annual_report_2024.pdf")
answer = pipeline.invoke(
"What was the YoY revenue growth by segment, and does the chart on page 8 show any anomalies?"
)
print(answer)
Now let's apply this to a real scenario. Download a publicly available annual report — the Apple 10-K or any company's SEC filing works well. These documents are ideal test cases because they contain all three modalities in meaningful proportions.
Exercise goals:
Step 1: Download and prepare your document
# Download Apple's 2023 10-K (publicly available)
curl -o apple_10k_2023.pdf \
"https://s2.q4cdn.com/470004039/files/doc_earnings/2023/ar/_10-k-2023-(as-filed).pdf"
Step 2: Run extraction and check what you got
elements = extract_document_elements("apple_10k_2023.pdf")
tables = extract_tables_with_pdfplumber("apple_10k_2023.pdf")
# Inspect what was found
print(f"Text chunks: {len(elements['text'])}")
print(f"Tables: {len(tables)}")
print(f"Images: {len(elements['image_paths'])}")
# Look at a table to verify extraction quality
print("\nSample table (first found):")
print(tables[0]["content"][:500] if tables else "No tables found")
# Check an extracted image exists
if elements["image_paths"]:
print(f"\nFirst extracted image: {elements['image_paths'][0]}")
Step 3: Build the pipeline and run targeted queries
pipeline = build_multimodal_rag_pipeline("apple_10k_2023.pdf")
# Test 1: Pure text retrieval
q1 = "What are Apple's primary risk factors related to supply chain?"
print("Q1:", pipeline.invoke(q1))
# Test 2: Table retrieval
q2 = "What were Apple's net sales by product category for fiscal year 2023?"
print("Q2:", pipeline.invoke(q2))
# Test 3: Cross-modal reasoning
q3 = "Summarize Apple's revenue trends and explain any visualizations that support the narrative."
print("Q3:", pipeline.invoke(q3))
Step 4: Evaluate retrieval quality
# Check what was actually retrieved for a given query
from langchain_core.runnables import RunnablePassthrough
def inspect_retrieval(retriever, query: str):
"""See exactly what documents are being retrieved for a query."""
docs = retriever.invoke(query)
print(f"Query: {query}")
print(f"Retrieved {len(docs)} documents:")
print("-" * 50)
for i, doc in enumerate(docs, 1):
doc_type = doc.metadata.get("type", "unknown")
print(f"\n[{i}] Type: {doc_type.upper()}")
print(f" Content preview: {doc.page_content[:200]}...")
return docs
# Run inspection
retrieved = inspect_retrieval(
retriever,
"What was Apple's gross margin percentage?"
)
What to look for: You should see a mix of types returned. If every result is type "text" when you're asking about financial figures, your table extraction likely failed and needs debugging. If image summaries never appear, check that your image extraction directory isn't empty.
Symptom: Your "tables" look like "Revenue 94052 85777 97278 Product Services" — a flattened string with no structure.
Cause: You're using fast-mode PDF extraction, which reads the text layer left-to-right without understanding column structure.
Fix: Switch to strategy="hi_res" in partition_pdf, or use pdfplumber's extract_tables() which detects cell boundaries from line geometry.
# Check if table extraction worked
for table in tables[:3]:
lines = table["content"].split("\n")
has_separator = any("---" in line for line in lines)
if not has_separator:
print("WARNING: Table may not have been parsed correctly")
print("Raw content:", table["content"][:300])
Symptom: Queries about financial data return image summaries; queries about charts return prose text.
Cause: All your embeddings are in the same vector space, and the similarity landscape doesn't match the modality boundaries you expect.
Fix: Add modality metadata to your documents and use filtered retrieval when you have strong signal about what type of content a query needs:
# Metadata-filtered retrieval for explicit table queries
from langchain_core.vectorstores import VectorStoreRetriever
table_retriever = retriever.vectorstore.as_retriever(
search_kwargs={
"k": 4,
"filter": {"type": "table"}
}
)
# Use the filtered retriever for clearly quantitative queries
# Use the full retriever for ambiguous queries
Symptom: The LLM says "Based on the image summary..." instead of actually reasoning over the image. Or it describes something inconsistent with what's actually in the image.
Cause: You're passing the summary text to the LLM but not the actual image bytes. The LLM is reasoning from your summary, not from the visual.
Fix: Always re-encode and attach the image at generation time. Check the build_prompt function — verify that content has both text blocks and image_url blocks before sending:
def debug_prompt_content(content: list) -> None:
"""Quick diagnostic to verify prompt contains actual images."""
text_blocks = sum(1 for block in content if block.get("type") == "text")
image_blocks = sum(1 for block in content if block.get("type") == "image_url")
print(f"Prompt contains: {text_blocks} text blocks, {image_blocks} image blocks")
Symptom: The LLM produces correct-sounding but vague answers — it seems to be reasoning from summaries of summaries.
Cause: In build_multimodal_index, if you store the summary as both the vector content and the docstore content, the retriever returns the summary at generation time instead of the original content.
Fix: The docstore content should always be the original (the raw markdown table, the full image path + original summary). Only the vector embedding should use the summary or clean representation.
Symptom: API errors about token limits, or truncated responses that cut off mid-answer.
Cause: Retrieving 6 documents that each contain a large table or high-detail image description can easily hit 32K tokens.
Fix: Be strategic about search_kwargs["k"]. For multimodal retrieval, 4-6 total documents is usually right. Also truncate table content if it's very large:
def truncate_table_for_context(table_markdown: str, max_rows: int = 20) -> str:
"""Keep first N data rows of large tables to control token usage."""
lines = table_markdown.split("\n")
if len(lines) <= max_rows + 2: # header + separator + data rows
return table_markdown
header_lines = lines[:2] # header + separator
data_lines = lines[2:max_rows + 2]
truncation_note = f"| ... | (table truncated, {len(lines) - max_rows - 2} more rows) |"
return "\n".join(header_lines + data_lines + [truncation_note])
For a 100-page document with 20 images, expect:
hi_res strategy runs an ML model per page)This means multimodal RAG is a batch indexing operation, not a real-time one. Build your pipeline with this in mind — index documents on ingest, store the index persistently, and serve queries from the pre-built index.
For production, swap Chroma with a persistent vector store and replace InMemoryStore with a Redis or PostgreSQL-backed docstore:
# Persistent Chroma
vectorstore = Chroma(
collection_name=collection_name,
embedding_function=embeddings,
persist_directory="./chroma_multimodal_db"
)
# For the docstore in production, consider:
from langchain.storage import RedisStore
docstore = RedisStore(redis_url="redis://localhost:6379", key_prefix="multimodal_rag:")
Multimodal RAG adds real complexity. You should use it when:
You should not use it when:
You've built a complete multimodal RAG pipeline that handles the full spectrum of document content. Let's recap the key architectural decisions you made and why they matter:
Extraction: You used hi_res strategy and pdfplumber together — the former for layout understanding, the latter for precision table cell detection. This combination handles the majority of real-world document structures.
Indexing strategy: You chose summary-embedding over direct image embedding. Summaries written in natural language retrieve better against text queries, which is what your users will send 95% of the time.
Retrieval pattern: The MultiVectorRetriever pattern — separate vector index from docstore — gives you the best of both worlds: optimized search representations and full-fidelity content for generation.
Generation: You constructed interleaved text+image prompts for GPT-4o, which actually looks at images rather than just reading your summaries about them.
Agentic multimodal retrieval: Instead of a single retrieval call, build an agent that can decide which modality to query based on the question — reducing noise in retrieved context
ColPali and late interaction models: These models embed document page images directly and support query-document interactions at the patch level — a completely different architecture that eliminates the extraction stage entirely
Structured output from table reasoning: Use LLM function calling to return table data as structured JSON rather than natural language, enabling downstream computation on retrieved values
Evaluation frameworks: Set up RAGAS or a custom evaluation harness to measure retrieval recall by modality — you may find that table retrieval lags text retrieval and needs a domain-specific re-ranker
The multimodal document understanding space is moving fast. ColPali-style approaches in particular threaten to make extraction-based pipelines obsolete for many use cases. Understanding both the extraction-based architecture you built here and where it breaks down will help you evaluate when to adopt the next generation of approaches.