Designing Multi-Modal Prompt Pipelines: Combining Text, Images, and Structured Data Inputs to Solve Complex Business Problems

A logistics company's operations team is drowning. Every morning, they receive 200+ delivery exception reports — each one a different format: a driver's handwritten note photographed on a phone, a structured CSV export from their warehouse management system, a scanned PDF of a damaged goods form, and a terse text description from the customer. A human analyst has to reconcile all four inputs and decide: escalate, refund, reroute, or dismiss. It takes four hours per day. Two analysts. Neither loves the job.

This is exactly the class of problem that multi-modal prompt pipelines were built to solve — not by replacing judgment entirely, but by doing the reconciliation, normalization, and triage work so your analysts can focus on the handful of cases that genuinely need human attention. The catch is that wiring these pipelines together properly is not trivial. You're not just submitting a photo to ChatGPT and asking "what do you see?" You're designing a coordinated system where different input types flow through different processing stages, outputs from one stage become inputs to another, and the whole thing has to be reliable enough to run unsupervised in production.

By the end of this lesson, you'll know how to design and implement exactly that kind of system. We'll work through real examples, cover the architectural decisions that actually matter, and be honest about where these pipelines break down.

What you'll learn:

• How modern vision-language models process and reason about image, text, and structured data inputs simultaneously
• How to architect multi-stage prompt pipelines that chain image extraction, structured parsing, and synthesis steps
• How to normalize heterogeneous inputs into a unified context object before passing them to a reasoning model
• How to handle error states, confidence scoring, and fallback logic when one modality produces unreliable output
• How to balance cost, latency, and quality across a production multi-modal pipeline

---

Prerequisites

You should be comfortable with the mechanics of prompting before diving into pipeline architecture. If you haven't yet, review Prompt Engineering Fundamentals for Data Professionals and Prompt Chaining: Breaking Complex Tasks into Steps — this lesson builds directly on those foundations. You should also understand how context windows and token budgets work, since multi-modal inputs consume context aggressively; Tokens, Context Windows, and Input Limits: What Data Professionals Need to Know Before Building AI Workflows covers this in depth.

---

What "Multi-Modal" Actually Means in Production

The term "multi-modal" gets thrown around loosely. Let's be precise about what it means when you're building business systems, because the architectural implications differ depending on which combination of modalities you're working with.

Text is the native modality of every major LLM. When we say "text," we mean natural language — emails, reports, chat transcripts, customer feedback, unstructured notes. Text is the lowest-friction input because models were trained overwhelmingly on it.

Images in the context of vision-capable models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and their successors) means raster images submitted either as base64-encoded content or as URLs. The model doesn't "read" images the way it reads text — it processes them through a separate vision encoder that produces patch embeddings, which are then fused with the language model's token stream. This matters because the model's understanding of image content is probabilistic and spatially coarse. It can read printed text in images quite well, interpret charts and diagrams reasonably well, and describe physical scenes adequately — but its spatial precision degrades as image complexity increases.

Structured data — CSVs, JSON objects, database exports, API payloads — are technically just text when they reach the model, but they carry semantic structure that you need to explicitly manage. A 50,000-row CSV cannot and should not be pasted into a prompt. The architecture question is always: what do you compute externally, and what do you ask the model to reason over?

PDFs and documents deserve their own mention because they're a hybrid: they may contain text, embedded images, tables, and mixed layouts. Before a PDF can be used as LLM input, it has to be decomposed — extracted text, rendered page images, or both. This pre-processing step is where most production pipelines get their first serious bug.

---

The Architecture Pattern: Intake, Normalize, Enrich, Synthesize

Every robust multi-modal pipeline follows a four-stage pattern. The names may vary across teams, but the logical structure is consistent. Understanding each stage clearly will prevent a whole class of design mistakes.

Stage 1: Intake

Intake is the boundary layer between your raw inputs and your pipeline. Its job is purely mechanical: receive inputs, validate that they exist and are in an acceptable format, and route them to the appropriate pre-processing path.

At minimum, intake should:

• Confirm input completeness (did all expected modalities arrive?)
• Validate file types and sizes
• Flag obviously corrupt or empty inputs early, before you spend money on API calls
• Assign a correlation ID so you can trace a specific work item through every downstream stage

This sounds like boring plumbing, but failing to implement it properly is one of the most common production failures. A pipeline that silently proceeds with a missing image input and then hallucinates conclusions about it is worse than one that fails loudly.

Stage 2: Normalize

Normalization transforms each modality into a form the model can actually use. For each input type, this means something different:

For images: Resize to the model's optimal resolution (most vision models are tuned around 1024x1024 or similar; sending a 12-megapixel phone photo wastes tokens and can degrade performance), convert to a consistent format (JPEG or PNG), and optionally run a cheap pre-processing step like OCR if text extraction is the primary goal.

For structured data: Don't send the raw CSV. Compute summary statistics, extract the relevant slice, convert to a compact representation (JSON is generally more token-efficient than CSV for small datasets; Markdown tables work well for human-readable intermediate results).

For PDFs: Decide your extraction strategy — pure text extraction (fast, cheap, loses layout), page rendering (preserves layout, expensive), or hybrid. Tools like pypdf, pdfplumber, and pymupdf each have different strengths for this step.

For text: Normalize encoding, strip boilerplate (headers, footers, repeated disclaimers), and chunk if necessary.

Stage 3: Enrich

Enrichment is where you add computed context that the model couldn't derive itself from the raw inputs. Examples:

• You've extracted line items from an invoice image. Before sending to the synthesis model, you join against your product catalog to add canonical product names and category codes.
• You've parsed structured data about a customer. Before sending to the reasoning model, you query your CRM for their tier, lifetime value, and open tickets.
• You've extracted text from a scanned document. Before synthesis, you run a quick named-entity recognition pass to tag the persons, organizations, and dates mentioned.

Enrichment is a force multiplier. It turns a generic reasoning model into a domain-aware one without fine-tuning, by injecting the business context the model needs to make accurate decisions. This is related to retrieval-augmented generation but applied at the pipeline stage level rather than just at query time.

Stage 4: Synthesize

Synthesis is the final model call where all normalized, enriched inputs are combined into a unified prompt and the model performs the target task: classification, extraction, summarization, decision recommendation, narrative generation, etc.

The synthesis prompt is the one most teams obsess over, but it's only as good as what you fed it. Garbage in from the normalization and enrichment stages produces sophisticated-sounding garbage out.

---

Designing the Context Object

One of the most useful concepts in multi-modal pipeline design is the context object — a single structured payload that aggregates all normalized and enriched inputs before the synthesis call. Designing this object well is an art.

Here's an example for the delivery exception use case from the introduction:

{
  "exception_id": "EX-2024-09182",
  "ingestion_timestamp": "2024-11-14T07:43:00Z",
  "modality_confidence": {
    "driver_photo_ocr": 0.91,
    "warehouse_csv_parse": 1.0,
    "customer_note": 1.0,
    "damaged_goods_form_ocr": 0.74
  },
  "driver_report": {
    "extracted_text": "Arrived 14:32. Gate locked. Left package at side entrance per dispatcher. Signature unavailable.",
    "handwriting_clarity": "moderate",
    "extraction_method": "vision_ocr"
  },
  "warehouse_data": {
    "shipment_id": "SHP-884421",
    "origin_facility": "DFW-3",
    "scheduled_delivery": "2024-11-14",
    "package_weight_kg": 4.2,
    "declared_value_usd": 340.00,
    "fragile_flag": true,
    "prior_exceptions_30d": 0
  },
  "customer_note": "Package shown as delivered but I never received it. Need resolution today.",
  "damaged_goods_form": {
    "extracted_text": "Item: Electronic components. Condition on arrival: [ILLEGIBLE]. Reported by: [ILLEGIBLE]",
    "form_completeness": "partial",
    "extraction_confidence": 0.74
  },
  "enrichment": {
    "customer_tier": "Gold",
    "customer_ltv_usd": 28400,
    "open_tickets": 0,
    "account_age_days": 847,
    "sla_breach_threshold_hours": 4
  }
}

Notice several design decisions embedded in this structure:

Confidence scores per modality. The damaged goods form had poor OCR confidence (0.74). The synthesis prompt can be written to explicitly handle low-confidence inputs differently — perhaps flagging the case for human review rather than making a definitive recommendation.

Extraction method metadata. Tracking how each field was derived lets you audit failures and improve your pipeline over time.

Partial extraction gracefully represented. The damaged goods form fields are partial, but the object still exists with "form_completeness": "partial" rather than erroring out. The synthesis model can work with incomplete information as long as it knows the information is incomplete.

Enrichment segregated. Customer tier, LTV, and SLA threshold are clearly labeled as enrichment — data joined from other systems. This helps when debugging: if the synthesis model makes a wrong call, you can immediately see whether it was because the extracted data was wrong or because the enrichment data was wrong.

---

Writing the Synthesis Prompt for Multi-Modal Inputs

With a well-structured context object, your synthesis prompt becomes much cleaner. You're not doing data wrangling in the prompt — you're asking the model to reason over pre-organized information.

Here's a synthesis prompt for the delivery exception case:

You are a delivery exception triage specialist for a logistics company.
You will receive a structured exception report containing information
from multiple sources: a driver field report, warehouse shipment data,
a customer complaint, and a damaged goods form. Some fields may have
low confidence scores or partial data; use your judgment about how
much weight to give them.

Your task is to:
1. Assess what likely happened based on all available evidence
2. Recommend one of: [ESCALATE, REFUND, REROUTE, DISMISS, MANUAL_REVIEW]
3. Provide a 2-3 sentence rationale for your recommendation
4. Identify any data gaps that prevent a confident determination
5. Flag if SLA breach is imminent based on ingestion_timestamp and
   sla_breach_threshold_hours

Guidelines:
- If modality_confidence for any source is below 0.80, treat that
  source as supporting evidence only, not primary evidence
- Gold and Platinum tier customers should receive ESCALATE or REFUND
  recommendations when evidence is ambiguous, not DISMISS
- If prior_exceptions_30d > 2, flag for fraud review

Return your response as valid JSON matching this schema:
{
  "recommendation": "<ESCALATE|REFUND|REROUTE|DISMISS|MANUAL_REVIEW>",
  "confidence": <0.0-1.0>,
  "rationale": "<string>",
  "data_gaps": ["<string>"],
  "sla_breach_risk": <boolean>,
  "fraud_flag": <boolean>
}

Exception report:
<EXCEPTION_DATA>

Then at runtime, you replace <EXCEPTION_DATA> with your serialized context object.

Notice what this prompt does well:

• It provides role context (triage specialist)
• It explicitly instructs the model on how to handle low-confidence inputs
• It encodes business rules (Gold tier treatment, fraud flag logic)
• It specifies a structured output format
• It's focused — one decision per call

What it doesn't do: handle the image inputs directly. In a real API call using a model like GPT-4o or Claude 3.5 Sonnet, you'd submit the actual images alongside this text prompt if you needed the model to re-examine them. But notice that we've already done image-to-text extraction in the normalization stage. For most structured reasoning tasks, you're better off having a dedicated vision call extract the image content into text first, then having a separate reasoning call work with that text. This is cleaner, more auditable, and usually cheaper.

This separation of concerns — vision extraction as its own stage, reasoning as a separate stage — is a core architectural principle that more experienced practitioners apply consistently. For more on how to force consistent JSON output from the reasoning stage, see Structured Output and JSON Mode: How to Force AI to Return Machine-Readable Data for Downstream Automation.

---

The Vision Extraction Stage in Detail

Let's go deeper on the vision extraction stage, since it's where most multi-modal pipelines first encounter serious reliability issues.

When to Use Vision vs. Dedicated OCR

Not all image inputs should go directly to a vision-language model. Here's a decision framework:

Use a dedicated OCR tool (Tesseract, AWS Textract, Google Document AI, Azure Form Recognizer) when:

• The document is machine-printed with predictable layout
• You need high-volume throughput (dedicated OCR is 5-20x cheaper per page)
• The primary extraction target is text, not visual understanding
• You need character-level confidence scores
• You're processing financial documents, IDs, or other forms where regulatory accuracy requirements apply

Use a vision-language model when:

• The image requires interpretation beyond text extraction (charts, diagrams, handwriting)
• Layout context is semantically important (e.g., a table where row/column relationships matter but aren't represented in standard OCR output)
• You're processing heterogeneous document types that don't fit a template
• The extraction task requires judgment (e.g., "is this damage consistent with impact or water?")

Hybrid approach: Run dedicated OCR to extract text, then pass both the original image and the OCR'd text to the vision model with an instruction to validate and correct the OCR output. This is more expensive but significantly improves accuracy for complex documents.

import anthropic
import base64
from pathlib import Path

def extract_from_damaged_goods_form(image_path: str, ocr_text: str) -> dict:
    """
    Hybrid extraction: pass both raw image and OCR pre-extraction
    to the vision model for validation and correction.
    """
    client = anthropic.Anthropic()
    
    image_data = base64.standard_b64encode(
        Path(image_path).read_bytes()
    ).decode("utf-8")
    
    extraction_prompt = f"""
    You are extracting structured data from a damaged goods report form.
    
    I'm providing you with:
    1. The original form image
    2. A preliminary OCR extraction (which may contain errors)
    
    Preliminary OCR output:
    {ocr_text}
    
    Please examine the image carefully and extract the following fields:
    - item_description: What item is described
    - condition_on_arrival: The condition noted (exact wording)
    - reported_by: Name/ID of person who completed the form
    - report_date: Date on the form (ISO 8601 format if possible)
    - damage_type: [PHYSICAL, WATER, MISSING_CONTENTS, UNKNOWN]
    - form_completeness: Percentage of required fields filled (0-100)
    
    Where the OCR output conflicts with what you see in the image,
    trust the image. Where text is genuinely illegible, use null
    and note the field as illegible.
    
    Return as JSON only, no explanation.
    """
    
    message = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/jpeg",
                            "data": image_data,
                        },
                    },
                    {
                        "type": "text",
                        "text": extraction_prompt
                    }
                ],
            }
        ],
    )
    
    # Parse and add extraction metadata
    import json
    result = json.loads(message.content[0].text)
    result["extraction_method"] = "hybrid_vision_ocr"
    result["model_used"] = "claude-opus-4-5"
    result["input_tokens"] = message.usage.input_tokens
    
    return result

---

Handling Structured Data at Scale

The challenge with structured data in multi-modal pipelines isn't the model's ability to reason over it — modern LLMs handle JSON and tabular data well. The challenge is the volume. You cannot paste a 50,000-row dataset into a prompt.

The right approach depends on the task:

Task: Reasoning over aggregates

If you need to answer questions about a large dataset (e.g., "which SKUs had more than 3 exceptions this month?"), compute the aggregation in code and pass only the result to the model.

import pandas as pd
import json

def prepare_exception_summary(df: pd.DataFrame, shipment_id: str) -> str:
    """
    Transform a large exceptions dataset into a concise
    model-ready summary for a specific shipment context.
    """
    # Historical context for this shipment's route
    route_stats = df[df['route_id'] == df[df['shipment_id'] == shipment_id]['route_id'].iloc[0]]
    
    summary = {
        "shipment_id": shipment_id,
        "route_exception_rate_30d": round(
            len(route_stats[route_stats['date'] >= pd.Timestamp.now() - pd.Timedelta(days=30)]) 
            / len(route_stats) * 100, 1
        ),
        "most_common_exception_types": (
            route_stats['exception_type']
            .value_counts()
            .head(3)
            .to_dict()
        ),
        "avg_resolution_hours": round(route_stats['resolution_hours'].mean(), 1),
        "driver_exception_count_30d": len(
            df[(df['driver_id'] == df[df['shipment_id'] == shipment_id]['driver_id'].iloc[0]) &
               (df['date'] >= pd.Timestamp.now() - pd.Timedelta(days=30))]
        )
    }
    
    return json.dumps(summary)

The model now receives a 200-token JSON object with exactly the context it needs, instead of a 2-million-token raw dataset it can't process anyway.

Task: Row-level reasoning over many records

If you need to classify, enrich, or extract from individual rows of a large dataset, you need a batch processing architecture. Each row (or small batch of rows) becomes an independent pipeline invocation. This is where async processing, rate limiting, and cost control become serious engineering concerns.

For this pattern, look at combining the multi-modal pipeline with ideas from Agentic AI Workflows: Designing Multi-Step Autonomous Pipelines That Plan, Act, and Self-Correct, which covers orchestration patterns for running large numbers of model calls systematically.

Task: Schema validation and anomaly detection

Sometimes structured data itself is the subject of the analysis. You need the model to identify inconsistencies, fill gaps, or flag anomalies. Here, the right input is a sample plus schema, not the full dataset:

You are auditing a warehouse shipment record for data quality issues.

Schema definition:
{
  "shipment_id": "string, format SHP-NNNNNN",
  "weight_kg": "float, must be > 0",
  "declared_value_usd": "float, must be > 0",
  "fragile_flag": "boolean",
  "origin_facility": "string, one of: [DFW-1, DFW-2, DFW-3, LAX-1, LAX-2, ORD-1]",
  "scheduled_delivery": "date, ISO 8601, must be >= shipment_date"
}

Business rules:
- Shipments with fragile_flag=true and weight_kg > 20 require supervisor sign-off
- Declared value > $500 requires insurance documentation flag

Record to audit:
{
  "shipment_id": "SHP-884421",
  "weight_kg": -4.2,
  "declared_value_usd": 340.00,
  "fragile_flag": true,
  "origin_facility": "DFW-3",
  "scheduled_delivery": "2024-11-14"
}

Identify all data quality issues, schema violations, and business rule
violations. Return as JSON array of issue objects with fields:
field, issue_type, description, severity (ERROR|WARNING|INFO).

This pattern essentially uses the model as a smart validator — something it does exceptionally well because it can combine schema logic, business rules, and semantic understanding in ways that pure code-based validators cannot.

---

Managing Confidence and Fallback Logic

A production multi-modal pipeline must have explicit confidence handling at every stage. Here's a practical framework.

Confidence Tiers

Confidence	Range	Handling
High	0.85 – 1.0	Proceed automatically
Medium	0.65 – 0.84	Proceed with flag for sampling/audit
Low	0.40 – 0.64	Request secondary confirmation or human review
Failure	< 0.40	Route to manual queue, alert pipeline owner

These thresholds are a starting point — calibrate them against your business's specific cost-of-error calculations. For a fraud detection pipeline, you might push the High threshold to 0.95. For a content tagging pipeline, 0.75 might be plenty.

Computing Confidence

Confidence scores don't come for free — you have to design for them. Three approaches:

Model self-reported confidence: Ask the model to include a confidence score in its output JSON. This is the simplest approach but the least reliable — models are notoriously poorly calibrated about their own uncertainty. It's a useful signal but shouldn't be your only one.

Multi-sample consistency: Run the same extraction multiple times (at temperature > 0) and measure agreement across runs. High agreement → high confidence. Expensive, but reliable. Good for high-value decisions.

Cross-modal validation: If the same piece of information appears in multiple modalities (e.g., a weight in both the image of a label and the structured CSV), compare them. Disagreement is a confidence signal.

def cross_modal_confidence(
    image_extracted_weight: float | None,
    csv_weight: float | None,
    tolerance_pct: float = 0.05
) -> tuple[float, str]:
    """
    Compute confidence score from cross-modal validation of weight field.
    Returns (confidence_score, explanation).
    """
    if image_extracted_weight is None and csv_weight is None:
        return 0.0, "Weight missing from both modalities"
    
    if image_extracted_weight is None:
        return 0.6, "Weight from CSV only; image extraction failed"
    
    if csv_weight is None:
        return 0.6, "Weight from image only; CSV field empty"
    
    pct_diff = abs(image_extracted_weight - csv_weight) / csv_weight
    
    if pct_diff <= tolerance_pct:
        return 0.95, f"Cross-modal agreement within {tolerance_pct*100}%"
    elif pct_diff <= 0.15:
        return 0.75, f"Minor cross-modal discrepancy: {pct_diff*100:.1f}%"
    else:
        return 0.40, f"Significant cross-modal conflict: image={image_extracted_weight}, csv={csv_weight}"

---

Cost and Latency Architecture

Multi-modal pipelines have a cost structure that looks different from single-call prompting. You need to think about it explicitly.

Model Tier Selection Per Stage

Not every stage requires your most capable (and expensive) model. A common optimization:

Stage	Task	Model Tier
Image OCR extraction	Text extraction from clean forms	Small vision model or dedicated OCR
Chart/diagram interpretation	Complex visual reasoning	Large vision model
Structured data validation	Schema checking, anomaly detection	Mid-tier reasoning model
Final synthesis/decision	Multi-source reasoning with business rules	Large reasoning model

Running a dedicated GPT-4o call to OCR a printed invoice when AWS Textract would do it at 1/10th the cost and latency is a common waste of budget. Reserve your expensive model calls for the tasks that actually need them.

Parallelism

Normalization stages for different modalities can run in parallel. There's no reason to wait for image extraction to complete before you start structured data processing. In Python with asyncio or a workflow orchestrator like Prefect or Dagster, this is straightforward to implement.

import asyncio
from typing import Any

async def normalize_all_modalities(
    image_path: str,
    csv_data: bytes,
    customer_note: str,
    shipment_id: str
) -> dict[str, Any]:
    """
    Run all normalization tasks in parallel, then combine results.
    """
    
    # These run concurrently
    results = await asyncio.gather(
        extract_from_image(image_path),          # Vision API call
        parse_and_summarize_csv(csv_data),        # Local computation
        validate_note_encoding(customer_note),    # Local computation
        fetch_enrichment_data(shipment_id),       # CRM/database call
        return_exceptions=True
    )
    
    image_result, csv_result, note_result, enrichment_result = results
    
    # Handle partial failures gracefully
    return {
        "driver_report": image_result if not isinstance(image_result, Exception) else None,
        "warehouse_data": csv_result if not isinstance(csv_result, Exception) else None,
        "customer_note": note_result if not isinstance(note_result, Exception) else None,
        "enrichment": enrichment_result if not isinstance(enrichment_result, Exception) else None,
        "pipeline_errors": [
            str(r) for r in results if isinstance(r, Exception)
        ]
    }

Parallelizing the normalization stage typically reduces end-to-end latency by 40-60% for pipelines with 3+ modalities.

Caching

Image extraction results should be cached aggressively. If the same document image is re-submitted (which happens frequently in retry scenarios), re-running the vision API call wastes money and time. Store extracted content by a hash of the source image and cache for the duration of the case's lifecycle.

---

Evaluation: How Do You Know Your Pipeline Is Working?

This is where many teams struggle. With a single-call prompt, evaluation is relatively contained. With a multi-stage pipeline, failures can originate in any stage, and the synthesis model's output quality depends on everything upstream.

You need evaluation at two levels:

Stage-level evaluation: Test each extraction and transformation stage independently. For vision extraction stages, maintain a golden dataset of 50-100 documents with manually verified extractions. Run new prompt versions and model versions against this dataset before deploying. Track accuracy, precision/recall for key fields, and false null rates (fields returned as null/illegible that a human can read).

Pipeline-level evaluation: Evaluate end-to-end outcomes against ground truth. For the delivery exception use case, this means comparing AI recommendations against what experienced human analysts decided for the same cases. Track agreement rate, and more importantly, analyze the disagreement cases — they reveal systematic biases in your pipeline.

For a thorough approach to building this kind of evaluation infrastructure, see Designing AI Evaluation Frameworks: How to Benchmark, Test, and Monitor LLM Performance in Production Workflows.

---

Hands-On Exercise

Scenario: A retail company receives supplier invoices in three forms: scanned paper invoices (photographed with phones), structured CSV exports from their EDI system, and email summary notes from account managers. Your task is to design and implement the normalization and synthesis stages of a pipeline that matches these three inputs against purchase orders in a database and flags discrepancies.

Step 1: Design the Context Object

Before writing any code, sketch the context object your synthesis prompt will receive. Include:

• Fields extracted from the invoice image (vendor name, invoice number, line items, total)
• Fields from the EDI CSV (PO number, ordered quantities, unit prices)
• Structured summary of the email note (mentioned issues, urgency signal)
• Enrichment from the PO database (expected total, payment terms, vendor tier)
• Confidence scores for each modality

Step 2: Write the Vision Extraction Prompt

Write a prompt for the invoice image extraction stage. Your prompt should:

• Specify exactly which fields to extract
• Handle partial/obscured invoices gracefully
• Return structured JSON with a completeness score
• Include explicit instructions for how to handle currency formatting variations (USD, EUR, GBP with different decimal conventions)

Test your prompt against three sample invoice images of varying quality. Evaluate field-level accuracy.

Step 3: Write the Synthesis Prompt

Write the synthesis prompt that takes your complete context object and:

• Identifies whether the invoice matches the PO (MATCH, PARTIAL_MATCH, MISMATCH, INSUFFICIENT_DATA)
• Quantifies any discrepancy in dollar terms
• Recommends action (APPROVE, HOLD_FOR_REVIEW, REJECT, REQUEST_CORRECTION)
• Adjusts recommendation based on vendor tier (preferred vendors get more lenient matching thresholds)

Step 4: Add Confidence Handling

Extend your synthesis prompt to include explicit instructions for handling cases where modality confidence is below 0.75. The model should down-weight those sources and increase the probability of a HOLD_FOR_REVIEW recommendation.

Step 5: Measure and Iterate

Using 20 historical invoices with known outcomes (you'll need to create synthetic ones for this exercise), evaluate your pipeline's recommendation accuracy. Where does it fail? Is the failure pattern in the extraction stage or the synthesis stage? How do you know?

---

Common Mistakes and Troubleshooting

Mistake 1: Using vision for everything that contains an image

Symptom: High per-call costs, slow throughput, no improvement in accuracy over cheaper alternatives.

Fix: Route documents to vision APIs only when visual understanding adds value. Machine-printed forms with consistent layouts belong in dedicated OCR pipelines.

Mistake 2: Over-stuffing the synthesis prompt

Symptom: Synthesis model ignores some inputs, produces inconsistent outputs, seems to "forget" earlier context.

Fix: If your context object exceeds ~10,000 tokens, you're likely including information the model doesn't need for this specific decision. Trim mercilessly. If you genuinely need all that context, split into multiple synthesis calls and aggregate results.

Mistake 3: No cross-modal validation

Symptom: Pipeline produces confident wrong answers when one modality contains an error.

Fix: Identify fields that appear in multiple modalities and implement explicit cross-modal comparison in your normalization stage. Surface disagreements in the context object so the synthesis model can reason about them.

Mistake 4: Brittle prompt templates that break on novel document types

Symptom: Pipeline works well in testing, fails in production when a new vendor uses an unusual invoice format.

Fix: Build in an explicit "structured extraction failed, falling back to free-form extraction" path. Have your vision extraction prompt include a parse_success boolean. When it's false, trigger a different extraction prompt that's more permissive but produces a less structured output, then route to human review. See also From Vague to Precise: How to Diagnose and Fix Prompts That Return Unhelpful AI Responses for systematic prompt debugging approaches.

Mistake 5: Ignoring the failure modes of structured data parsing

Symptom: The CSV from one facility uses commas as decimal separators (European convention), another uses periods. Your parser outputs wrong numbers silently.

Fix: Validate all structured data outputs before they enter the context object. Use range checks, type checks, and spot-check a sample of computed fields against raw source data regularly.

Mistake 6: Not handling timeouts for parallel stages

Symptom: When the CRM enrichment call is slow, the entire pipeline hangs, even though the other modalities were processed successfully.

Fix: Set explicit timeouts for every parallel stage. Treat timeout as a partial failure — log it, set the affected modality's confidence to null, and proceed. A pipeline that produces a good answer in 8 seconds is better than one that produces a perfect answer in 30 seconds, or one that times out entirely.

---

Summary and Next Steps

Multi-modal prompt pipelines are the right architecture when your business problems live at the intersection of different information formats — and most real business problems do. The principles we've covered give you a complete design framework:

1. The four-stage pattern (Intake → Normalize → Enrich → Synthesize) separates concerns cleanly and makes each stage independently testable and improvable.

2. The context object is your central design artifact. It aggregates normalized inputs, carries confidence metadata, and becomes the single input to your synthesis prompt.

3. Vision extraction is a specialized sub-problem. Know when to use vision models versus dedicated OCR, use hybrid approaches for complex documents, and cache results aggressively.

4. Structured data belongs outside the prompt until you've done the computation. Pass summaries and relevant slices, not raw datasets.

5. Confidence scoring and fallback logic are not optional in production. Design for partial failures from the start.

6. Evaluation must be multi-level. Test stages independently and test end-to-end outcomes against business ground truth.

The next area to explore from here is scaling these pipelines to run autonomously across large volumes of work. Agentic AI Workflows covers orchestration patterns for autonomous multi-step execution. If you're building the API infrastructure to call multiple models programmatically, Integrating AI APIs into Business Workflows: Call OpenAI, Claude, and Gemini Programmatically is your practical reference. And when your pipeline is in production and you need to systematically improve it, Prompt Versioning and Iteration Management gives you the tooling framework to do that without breaking what's already working.

Multi-modal pipelines are genuinely hard to build well — but the business problems they solve are genuinely valuable. The four hours per day your logistics team spends reconciling exception reports is not a rounding error. It's exactly the kind of problem worth engineering seriously.