Your quarterly business review deck is sitting on your desk. It has 47 slides, three embedded Excel charts, a scanned table from a vendor report, and a PDF of regional sales data that someone exported from Tableau at 2am and absolutely refused to send as a CSV. Your manager wants a written summary with key insights and anomaly flags by end of day. Two years ago, that was a half-day job involving copy-paste gymnastics, manual chart reading, and a lot of squinting. Today, you can feed those files directly into an AI model and get structured analysis in minutes — if you know how to prompt for it correctly.
That "if" is doing serious work in that sentence. Multimodal AI — models that can process images, PDFs, charts, and text together — is genuinely powerful, but most practitioners are using it at 20% capacity. They drag a screenshot into ChatGPT, type "what does this show?" and get back a vague paragraph that tells them less than reading the chart themselves would have. The problem isn't the model. It's the prompting strategy. Multimodal prompting has its own craft, its own failure modes, and its own set of techniques that are completely different from pure text prompting.
By the end of this lesson, you'll be able to build structured analysis pipelines around visual and document inputs — extracting data tables from charts, pulling structured fields from PDFs, identifying anomalies in dashboards, and generating report-ready narratives from mixed-media inputs. You'll understand why the model succeeds or fails with different input types and how to compensate for those failure modes in your prompts.
What you'll learn:
You should already be comfortable with:
Before you can prompt well, you need a mental model of what's actually happening when you attach a file to an AI conversation. This understanding will explain a lot of otherwise mysterious behavior.
When you upload an image — whether it's a screenshot of a dashboard, a photo of a printed report, or an exported chart — the model receives it as a grid of pixels that gets encoded by a vision component into a vector representation. The model then attends to both that encoded image and your text prompt simultaneously. It is not reading text in the image the way OCR software does. It is pattern-matching visual features and correlating them with its training data about what charts, documents, and visual layouts tend to contain.
This has important practical implications:
What models are good at visually:
What models struggle with visually:
PDFs are a different animal entirely. Most multimodal interfaces handle PDFs in one of two ways: they either render each page as an image (losing the underlying text layer), or they extract the text content first and pass that to the model alongside the images. This distinction matters enormously. If your PDF is a native digital document with real text, the model can read it precisely. If it's a scanned PDF — essentially a photograph of a document — the model is working from pixel patterns, and precision drops significantly.
Tip: When you're working with a PDF and getting imprecise extractions, try asking the model directly: "Are you reading the text of this document or interpreting it as an image?" The answer (and the quality of responses to follow-up questions) will tell you which mode you're in.
Understanding this distinction shapes every prompting decision you'll make. When precision matters, you compensate. When approximation is fine, you can prompt more loosely.
The single biggest upgrade you can make to your multimodal prompting practice is adding explicit output structure to every extraction request. Vague questions get vague answers. Structured requests get structured data.
Compare these two prompts applied to an uploaded bar chart showing monthly revenue by product line:
Weak prompt:
What does this chart show?
Strong prompt:
This is a bar chart showing monthly revenue by product line.
Please extract the following as a JSON object:
- chart_type: the type of chart
- time_period: the date range shown (if visible)
- categories: list of product lines shown in the legend
- data_table: for each product line, list the approximate monthly values in order
- observations: 3 specific observations about trends, peaks, or anomalies you notice
- data_confidence: rate your confidence in the numerical extractions as "high", "medium", or "low" and explain why
If you cannot read a specific value precisely, give a range estimate (e.g., "between 42,000 and 45,000").
The second prompt does several critical things. It tells the model what kind of chart it's looking at (reducing the cognitive load of classification). It specifies exactly what you want out. It uses JSON formatting so the output is immediately usable in downstream code. It explicitly asks for confidence ratings, which forces the model to flag uncertainty rather than hallucinate precise-looking numbers. And it gives permission to express ranges instead of false precision.
Let's look at what this might produce for a chart showing quarterly SaaS revenue:
{
"chart_type": "grouped bar chart",
"time_period": "Q1 2023 - Q4 2023",
"categories": ["Enterprise", "SMB", "Self-Serve"],
"data_table": {
"Enterprise": [1200000, 1350000, 1480000, 1620000],
"SMB": [680000, 710000, 695000, 740000],
"Self-Serve": [290000, 310000, 305000, 285000]
},
"observations": [
"Enterprise revenue shows consistent quarter-over-quarter growth, increasing approximately 35% from Q1 to Q4",
"SMB revenue is relatively flat with minor fluctuation, suggesting a plateau in this segment",
"Self-Serve revenue declined slightly in Q4 compared to Q3, which may warrant investigation"
],
"data_confidence": "medium — the y-axis gridlines are at $200K intervals and bars don't always align precisely to them. Enterprise values are estimated by interpolation and may vary ±$30K."
}
That output is immediately useful. You can parse it, validate the numbers against source data you have access to, pipe the observations into a report template, or use the confidence rating to decide whether you need to pull the original data before acting on the numbers.
PDFs in the wild come in three flavors, and your prompting strategy should match the type:
Type 1: Native digital PDFs (exported from Excel, Tableau, PowerPoint, or generated by software) — These have an underlying text and structure layer. Modern AI APIs can extract from these with high precision.
Type 2: Scanned physical documents — Essentially image files. Accuracy varies based on scan quality, font clarity, and page complexity.
Type 3: Hybrid PDFs — Native document with embedded images of charts or tables. Common with consulting reports and annual filings. The text portions extract cleanly; the visual elements need image-processing prompts.
For a hybrid PDF — say, a vendor's 30-page market analysis report — here's a prompting sequence that works well:
First prompt (document overview):
I've attached a market analysis PDF report. Before we extract specific data, please:
1. Tell me how many pages it has
2. List the major sections and their approximate page numbers
3. Identify which sections contain data tables vs. charts vs. prose narrative
4. Note any sections where the content appears to be images rather than readable text
This will help me know where to focus our extraction work.
This reconnaissance prompt is invaluable. You find out the document structure before you commit to a specific extraction strategy. If section 4 is all charts rendered as images, you'll prompt differently than if it's a clean data table.
Second prompt (targeted table extraction):
Please extract the data table from section 3 (Market Share by Region, pages 12-14).
Format the output as a markdown table first, then as a Python dictionary where:
- Keys are region names
- Values are dictionaries with keys: market_share_pct, yoy_change_pct, absolute_revenue_usd
If any cells are footnoted or marked with asterisks, include a "footnotes" key explaining them.
If you cannot read a cell value clearly, use null and note it.
Asking for markdown table and Python dictionary gives you a human-readable version for verification alongside a machine-usable version. The null instruction is important — it forces explicit missing values rather than guesses.
Third prompt (cross-referencing):
Looking at the table you extracted from section 3 and the executive summary in section 1,
are there any discrepancies between the numbers cited in the summary prose and the
detailed table? List any you find.
This cross-referencing prompt is one of the highest-value multimodal techniques. Consulting reports and annual filings frequently have inconsistencies between summary callouts and detailed appendix data. AI catches these faster than any human reviewer.
There's a meaningful difference between asking an AI to describe a chart and asking it to analyze one. Description is "the blue line goes up over time." Analysis is "the blue line's growth rate appears to be accelerating — the slope in Q3-Q4 is steeper than in Q1-Q2, which might indicate the marketing campaign launched in August is producing results."
To get analysis, you need to provide context and ask specific analytical questions. The model cannot know what your marketing campaign launch date was unless you tell it.
Here's a prompting pattern for substantive chart analysis:
[Attached: screenshot of a line chart showing weekly website sessions from January to December]
Context for analysis:
- This is our e-commerce site's weekly session data for 2023
- We ran a paid search campaign from March 15 to April 30
- We published a viral blog post on July 8 (approximately week 27)
- We experienced a server outage on September 19-20 (approximately week 38)
Please analyze this chart with that context in mind:
1. Do the session spikes appear to correlate with any of the events I mentioned?
2. Identify any trends or patterns NOT explained by the events I listed
3. Estimate the baseline weekly sessions before the campaign, during peak, and in December
4. Flag any data points that look anomalous even accounting for the events I described
This prompt structure — context block followed by specific analytical questions — consistently outperforms open-ended "analyze this chart" prompts. You're not constraining the model's analysis; you're giving it the domain knowledge it needs to do meaningful work.
Warning: Be careful not to bias the model toward confirming your hypothesis. If you write "please confirm that the March campaign drove the spike," you'll get confirmation whether or not the data actually supports it. Instead, ask "do the spikes appear to correlate?" and let the model make the call.
A full dashboard screenshot — the kind you might export from Tableau, Power BI, or Looker — is one of the more challenging multimodal inputs because it contains multiple charts, filter states, KPI tiles, and text elements all competing for the model's attention.
The key technique here is panel isolation combined with a master summary pass. Instead of prompting for everything at once, you break the analysis into phases.
Phase 1: Inventory the dashboard
[Attached: full dashboard screenshot]
Please identify and list every visual element in this dashboard:
- KPI tiles (list each metric name and its displayed value)
- Charts (list each chart, its type, and what metric/dimension it shows)
- Tables (list each table and its column headers)
- Filter controls (list any filters that appear to be active)
- Date range shown (if visible)
Number each element. We'll refer to them by number in follow-up prompts.
Phase 2: Deep-dive individual panels
Focus only on chart #3 from your list (the regional performance heatmap).
- Which regions are in the top quartile of performance?
- Which are in the bottom quartile?
- Is the color scale showing absolute values or normalized scores?
- What does the darkest cell represent and what does the lightest represent?
Phase 3: Synthesize across panels
Based on all the elements you've analyzed in this dashboard, please write a 3-paragraph executive summary:
Paragraph 1: Overall business performance vs. what the KPI tiles suggest the targets are
Paragraph 2: Geographic and segment-level patterns across the charts
Paragraph 3: The 2-3 most important things a senior executive should act on based on this data
Maintain a professional, direct tone. Flag anywhere your analysis is based on an uncertain reading.
This phased approach yields dramatically better results than a single prompt because it gives the model manageable cognitive chunks instead of asking it to simultaneously parse every element and synthesize across them.
Real-world data work involves imperfect inputs. Someone faxed a table. A photo was taken of a whiteboard with bad lighting. An old annual report from 2009 is only available as a scanned PDF with coffee stains. You still need to extract value from these.
Several techniques improve outcomes with degraded inputs:
Describe what you know about the document before asking:
[Attached: photo of a handwritten table on a whiteboard]
This is a photo of a capacity planning table written on a whiteboard during a planning session.
The table has 5 columns: Product SKU, Current Capacity, Proposed Capacity, Investment Required ($K), and Owner.
There are approximately 8 rows of data.
The handwriting may be difficult to read in places.
Please extract what you can read into a JSON array. For any cell that is illegible, use "ILLEGIBLE" as the value.
By telling the model the schema (column names, approximate row count), you dramatically improve extraction accuracy. The model isn't guessing at structure — it's filling in known slots.
Ask for a confidence breakdown:
After extracting the table, please go through each row and rate each cell's confidence as:
- Clear: you could read it unambiguously
- Likely: you're fairly confident but not certain
- Guess: you made a best guess from context
- Illegible: you could not determine the value
Return this as a separate "confidence_matrix" array parallel to your data array.
A confidence matrix like this is a practical quality control tool. When you get the extraction back, you immediately know which cells need human verification before you trust them.
Use context from surrounding content:
The SKU codes in this table should all follow the format: [2-letter category code]-[4-digit number]-[region code].
For example: EL-4421-NE or AP-3301-SW.
If you can partially read a SKU but it seems to follow this pattern, please reconstruct it and flag it as "Reconstructed."
Domain-specific pattern constraints help the model make intelligent inferences rather than random guesses.
Individual multimodal prompts are useful. Chained multimodal prompt sequences are transformative. Here's how to think about building pipelines.
Consider a monthly reporting workflow for a retail chain with 40 locations. Every month, regional managers submit PDF reports with embedded charts, summary tables, and narrative commentary. You need to synthesize all of this into a national summary. Here's a chain that handles this:
Step 1: Per-report extraction (run once per PDF)
[Attached: Regional Manager Report - Southeast Region - October 2024.pdf]
You are a retail analytics assistant. Extract the following from this regional report:
{
"region": string,
"reporting_period": string,
"kpis": {
"total_revenue": number,
"revenue_vs_plan_pct": number,
"revenue_vs_prior_year_pct": number,
"average_transaction_value": number,
"customer_count": number,
"store_count": integer
},
"top_performing_stores": [list of store names/IDs],
"underperforming_stores": [list of store names/IDs],
"manager_commentary_summary": string (2-3 sentences max),
"flagged_issues": [list of any issues the manager explicitly flagged],
"charts_present": [list of chart types in the report]
}
If a KPI is not present in the report, use null. Do not infer or estimate KPIs not explicitly stated.
Step 2: Aggregation prompt (run once after all per-report extractions)
I'm going to provide you with JSON extractions from 8 regional retail reports for October 2024.
[Paste all 8 JSON outputs here]
Please:
1. Aggregate total national revenue and calculate what % each region contributes
2. Identify which regions are above plan vs. below plan
3. Find stores that appear in multiple regions' underperforming lists (they won't — each store is in one region — but flag any single region with more than 2 underperforming stores)
4. Synthesize the manager commentary themes — what issues are showing up across multiple regions?
5. Produce a national executive summary table: Region | Revenue | vs Plan | vs PY | Status
Step 3: Narrative generation
Based on the aggregated analysis, write the executive narrative section of the national monthly report.
Format:
- Opening paragraph: national headline performance (3-4 sentences)
- Regional performance section: brief bullet for each region (1-2 sentences each)
- Issues requiring attention: numbered list of cross-regional or high-severity issues
- Recommendations: 3 specific, actionable recommendations based on the data patterns
Tone: professional, direct, appropriate for C-level readership. No hedging language.
This pipeline takes what was a half-day synthesis job and reduces it to minutes of prompting time, with the human's effort concentrated on validation rather than mechanical extraction.
Tip: Save each step's output to a file as you go. If the model session times out or you need to iterate on step 3 without re-running steps 1 and 2, you'll thank yourself.
Multimodal models can and do hallucinate numbers. This is the fact you cannot ignore when using AI for data extraction. The model might confidently report "Q3 revenue was $1.84M" when the chart actually shows $1.48M — a transposition that looks totally reasonable and is completely wrong. Before you trust extracted numbers in a business context, you need a validation strategy.
Validation technique 1: Sum-check
If you've extracted a table of values that should sum to a known total (like market share percentages that should add to 100%, or regional revenues that should match a reported national total), always check the sum.
The regional revenue figures you extracted were:
- Northeast: $2.1M
- Southeast: $1.8M
- Midwest: $1.4M
- West: $2.6M
- Total: $7.9M
The executive summary on page 1 states national revenue was $8.2M.
These don't reconcile. Please re-examine the regional figures — are there any regions you may have missed, or any values that might be off?
Validation technique 2: Relative magnitude check
If you know the business well enough to have expectations about relative magnitudes, challenge the model on them.
You extracted that the Self-Serve tier revenue ($4.2M) was higher than Enterprise tier ($3.8M).
In our business context, Enterprise has historically been our largest revenue segment by a significant margin.
Does this match what you see in the chart, or is it possible the tier labels in the legend were mismatched?
Validation technique 3: Re-extraction with a different prompt
Extract the same data twice, using different prompt framings, and compare results. Discrepancies signal uncertain readings.
First extraction prompt:
"What is the value of the blue bar in month 6?"
Second extraction prompt:
"Looking at the bar chart, list all 12 monthly values for the Enterprise segment."
If month 6 in the second extraction doesn't match what the first extraction reported, you have a conflict to investigate.
Validation technique 4: Explicit uncertainty elicitation
At the end of any numerical extraction, ask directly:
Now go back through the numbers you extracted and tell me:
- Which 3 values are you most confident in, and why?
- Which 3 values are you least confident in, and why?
- Are there any values where you made a judgment call that I should double-check against source data?
Models that are uncertain tend to reveal it when asked directly in this way. If the model says "I'm least confident in Q2 SMB revenue because the bar is slightly below the $700K gridline but I can't tell if it's $680K or $650K," that's actionable information — you know exactly where to go back to the source.
Once you've developed a prompting approach that works for a recurring report type, you should systematize it into a reusable template. This is the difference between a useful technique and a scalable workflow.
Here's a template structure for a monthly financial report analysis:
## SYSTEM CONTEXT
You are a financial data analyst assistant. You extract structured data from financial reports with precision, flag uncertainty explicitly, and produce analysis ready for executive review.
## REPORT METADATA
Report Type: [Monthly P&L / QBR / Board Report / Vendor Analysis — choose one]
Company: [Company name]
Period: [Reporting period]
Known Benchmarks: [List any numbers you already know that can serve as validation anchors]
## EXTRACTION INSTRUCTIONS
From the attached [document/chart/dashboard], extract:
[Paste your structured schema here]
## ANALYSIS INSTRUCTIONS
After extraction, provide:
1. [Specific analysis question 1]
2. [Specific analysis question 2]
3. [Specific analysis question 3]
## OUTPUT FORMAT
Return your response in two sections:
### EXTRACTED DATA
[JSON or table format]
### ANALYSIS
[Narrative paragraphs addressing the analysis questions]
## VALIDATION NOTE
After completing the above, list any values where your confidence is below "high" and explain why.
With this template, anyone on your team can run a consistent analysis by filling in the brackets. The system context maintains analytical rigor. The validation note requirement is non-negotiable — it's baked in so it can't be skipped.
Store your templates in a shared doc, a Notion database, or — if you're building this into a Python workflow — as string constants in a prompts module.
This exercise ties everything together. You'll need to find or create a multi-element input set that simulates a realistic QBR package. If you don't have access to real business data, you can use:
Your task:
Part 1 — Document Reconnaissance (15 minutes)
Upload your document and run a reconnaissance prompt. Ask the model to inventory all visual and data elements, identify the document structure, and flag any sections that appear to be images rather than readable text. Save this output.
Part 2 — Structured Extraction (20 minutes)
Identify the 3-5 most important data tables or charts in the document. For each one, write a targeted extraction prompt using the structured JSON format covered in this lesson. Include a confidence rating request in each prompt. Save all outputs.
Part 3 — Cross-Reference Validation (10 minutes)
Find at least one number that appears in both a detailed chart/table and in a summary section (executive summary, cover page KPIs, etc.). Ask the model to compare the two occurrences and flag any discrepancy. Document the result.
Part 4 — Narrative Generation (15 minutes)
Using your extracted data as input (paste the JSON into the prompt rather than re-submitting the document), ask the model to write a 3-paragraph executive summary suitable for a stakeholder audience. Specify the tone and the audience role.
Reflection questions:
Mistake 1: Uploading tiny or compressed images
Models read compressed JPEG charts with significant loss of detail. Small text, fine gridlines, and close-together bars all degrade badly. Always export charts at the highest resolution available, or use PNG instead of JPEG. If you're screenshotting a dashboard, use a full-resolution display and zoom in before capturing.
Mistake 2: Asking for everything in one prompt
"Analyze this entire 40-page report" is an instruction to produce mediocre analysis of everything. Break the work into targeted prompts. Focused inputs yield focused outputs.
Mistake 3: Not providing context
A chart without context is an unlabeled mystery. The model doesn't know your industry, your business cycle, your competitive landscape, or what counts as a good number vs. a bad one. Provide that context in your prompt and your analysis quality will jump immediately.
Mistake 4: Treating extracted numbers as gospel
We've covered this, but it bears repeating: any number extracted from a visual should be considered an estimate until validated against source data. Build verification steps into your workflows, especially for numbers that will appear in external reports.
Mistake 5: Ignoring model refusals or caveats
Sometimes a model will say "I'm not certain about this value" or decline to extract a specific element. Pay attention to these signals. They're often accurate. Pressing the model to just give you a number anyway produces hallucinations dressed up as analysis.
Mistake 6: Forgetting about chart legends
Many chart reading errors happen at the legend level. The model misidentifies which color corresponds to which category, and then assigns all subsequent values to the wrong series. In your extraction prompts, explicitly ask the model to state which color/pattern corresponds to which category before extracting values.
Before extracting the data values, please first describe the legend:
list each category and the color or pattern associated with it in the chart.
This one addition catches a surprising number of legend misreadings before they cascade into wrong extractions.
Troubleshooting: The model says it can't see the image
This usually means the file upload didn't succeed, the model you're using doesn't have vision enabled, or you've hit a context length limit. Check: did the file upload show a thumbnail or confirmation? Is vision turned on for your model (in the API, this means using the correct message format with image_url type)? Is the file under the size limit (typically 20MB for most platforms)?
Troubleshooting: Wildly wrong numbers on every extraction
If numerical extractions are consistently far off, the issue is likely image quality or image size. Try: exporting at higher resolution, using a PDF version instead of an image if available, cropping to just the relevant chart before uploading, or asking the model to describe what it sees before extracting (sometimes this reveals it's seeing a very different image than you think you sent).
Multimodal prompting is not a single technique — it's a discipline with distinct skills for different input types. Here's what you've built over this lesson:
For images and charts: You now use structured extraction prompts with explicit schemas, confidence ratings, and legend verification. You know how to provide business context that elevates description to analysis. You understand what models see accurately vs. approximately.
For PDFs: You know to run reconnaissance first, distinguish between native and scanned documents, and use cross-referencing prompts to catch inconsistencies between summary and detail layers.
For dashboards: You use a phased approach — inventory, panel deep-dive, synthesis — instead of expecting one prompt to handle everything.
For validation: You have four concrete techniques (sum-check, magnitude check, re-extraction, explicit uncertainty elicitation) to catch hallucinations before they propagate.
For scaling: You know how to chain prompts into pipelines and build reusable templates that make recurring workflows consistent and efficient.
The next skill to develop is automating these workflows in Python using the OpenAI, Anthropic, or Google Gemini APIs, which let you process batches of documents programmatically — passing files, parsing JSON outputs, and routing results into databases or report templates without manual copy-pasting. That's the bridge from "impressive technique" to "production capability."
You should also look at specialized document intelligence tools like Azure Document Intelligence, AWS Textract, or LlamaParse, which are purpose-built for structured extraction from documents and often outperform general vision models on precision-critical tasks. The best workflows often combine these specialized tools for extraction with general LLMs for synthesis and narrative generation.
The fundamental skill you've built here — understanding why the model succeeds and fails and designing prompts that compensate for those failure modes — transfers directly to those tools and any multimodal system you encounter going forward.
Learning Path: Intro to AI & Prompt Engineering