Automating Image-Based UI Interactions in Power Automate Desktop: Using Screen Scraping, Image Recognition, and Coordinate-Based Actions When Selectors Fail

Introduction

You've built a solid automation. It runs perfectly in testing. Then one day it fails — not because the underlying application changed, but because a Windows update shifted the rendering engine, a virtualized desktop session is running at a different DPI, or someone at the help desk changed the theme on the target machine. Your selectors, which worked beautifully before, now return nothing. The UI element that Power Automate Desktop was tracking by CSS attribute or accessibility tree path simply isn't being found anymore.

This is the moment where most RPA developers either give up on the target application or fall into the trap of "let me just re-record the whole thing." Neither approach addresses the root problem. What you actually need is a fundamentally different automation strategy — one that doesn't rely on the application exposing a well-structured accessibility interface. Instead, it works the way a human would: by looking at the screen, recognizing visual patterns, and interacting based on what's visible rather than what's programmatically queryable.

This lesson teaches you exactly that. We'll go deep on Power Automate Desktop's image recognition capabilities, screen scraping infrastructure, and coordinate-based click mechanics. By the end, you'll have a complete mental model for when each technique is appropriate, how to implement it reliably in production, and what to do when even image-based approaches need to be defended against environmental variability. You'll be building automations that can handle legacy Citrix-delivered apps, custom-rendered WPF controls, Java applications with broken accessibility trees, and any other nightmare scenario where traditional selectors simply don't reach.

What you'll learn:

• How Power Automate Desktop's image recognition engine works internally and what affects match confidence
• How to capture, tune, and deploy image-based click, wait, and conditional actions
• How to use screen scraping (OCR) alongside image recognition for hybrid text-plus-visual workflows
• How to calculate and use absolute and relative screen coordinates safely across different display configurations
• How to build fallback logic that gracefully degrades from selector-based to image-based to coordinate-based interaction

---

Prerequisites

This is an expert-level lesson. You should already be comfortable with:

• Building and debugging selector-based desktop flows (see UI Elements and Selectors in Power Automate Desktop: Building Automations That Don't Break if you need a refresher)
• Using error handling blocks — you'll need them here (see Error Handling in Desktop Flows: On Block Error, Retry Policies, and Recovery Screenshots)
• Understanding the difference between attended and unattended execution contexts, because image-based approaches behave differently between them (see Attended vs Unattended RPA: Choosing a Run Mode and Configuring Machines in Power Automate)
• Basic familiarity with variables and flow control in Power Automate Desktop

---

Why Selectors Fail (and What You're Actually Replacing)

Before we get tactical, it's worth understanding precisely why you're in this situation. Power Automate Desktop's standard UI automation works by querying a structured accessibility interface. On Windows, this is exposed through UI Automation (UIA), MSAA (Microsoft Active Accessibility), or in the case of web apps, the DOM. When you use "Click UI Element," the engine traverses a tree of control objects, matches the element by the selector you defined, gets the element's current bounding rectangle, and synthesizes a click at the center of that rectangle.

This fails in several well-understood scenarios:

Citrix and RDP virtualization: When the application is rendered on a remote machine and streamed to your screen as a video feed, there is no local accessibility tree. The pixels you see are exactly that — pixels. There is no control hierarchy to query.

Custom-rendered controls: Some applications — particularly older Java-based ERP systems, custom WPF applications with owner-drawn controls, or applications built with third-party UI frameworks — don't properly implement accessibility APIs. The control exists visually but returns nothing meaningful from a UIA query.

Flash, Silverlight, and browser plugin content: Legacy web applications that relied on browser plugins rendered their own visual output without exposing accessibility structures. While these platforms are dying, you'll still encounter them in enterprise environments.

Dynamic IDs and unstable attributes: Some applications (particularly poorly-built web apps or auto-generated thick client UIs) change control IDs, names, or class attributes on every session or even on every render cycle. The selector was valid when you recorded it; it's invalid ten minutes later.

Protected and obfuscated applications: Security-hardened financial or compliance applications sometimes actively block accessibility tree access as part of their security posture.

In all of these cases, the solution space splits into three techniques you can combine: image recognition (find this visual pattern on screen), OCR-based screen scraping (read text from a region of screen pixels), and coordinate-based actions (click at a specific pixel location, calculated either statically or derived from a recognized image's position).

---

How Power Automate Desktop's Image Recognition Engine Works

Power Automate Desktop's image recognition is based on template matching — specifically, a variant of normalized cross-correlation. When you capture an image template and use it in an action, the engine takes a screenshot of the current screen (or a specified region), then slides your template image across the screenshot pixel by pixel, computing a similarity score at each position. When the score exceeds your specified tolerance threshold, it reports a match and returns the screen coordinates of the matching region's bounding box.

Understanding this mechanism explains several non-obvious behaviors:

Scaling sensitivity: Template matching operates at the pixel level, which means that if your template was captured at 150% DPI and the target machine runs at 100% DPI, the template will be scaled differently than the screen content. The match will fail not because the button is gone but because its pixels have been scaled to a different size.

Color sensitivity: Normalized cross-correlation is sensitive to color values. A Windows theme change — from light mode to dark mode, or even from one accent color scheme to another — can change the pixel values of UI chrome elements enough to drop match confidence below threshold.

Anti-aliasing and subpixel rendering: Text and icon edges rendered with ClearType or other subpixel antialiasing techniques produce slightly different pixel patterns depending on the background color they're rendered against. If you capture a template of a button label against a white background and the button's background color changes, the antialiased pixels around the text edges will differ from your template.

Hardware acceleration and GPU rendering: Some applications use GPU-accelerated rendering pipelines that can produce slightly different pixel outputs depending on the GPU driver version and hardware. Two machines running the same application version may produce subtly different pixel patterns.

This is why template matching has a tolerance (or similarity) setting — it's not just a convenience feature, it's essential to handle real-world environmental variation. The challenge is setting it correctly: too low (strict) and you get false negatives; too high (loose) and you get false positives.

---

Capturing Image Templates That Actually Work

The quality of your image template determines everything downstream. A poorly captured template will either fail to match consistently or match things you didn't intend. Here's how to do it right.

Choosing What to Capture

The goal is to capture the smallest, most visually unique region that unambiguously identifies what you're looking for. Counter-intuitively, bigger is not better. A large template that includes surrounding UI chrome (title bars, borders, adjacent controls) is more likely to fail when those surrounding elements change. A tightly cropped image of just the distinctive part of the target element is more resilient.

For a button, capture just the button's visual content — its icon and label text — not the button's border or shadow. For a specific menu item, capture just the text and any distinctive icon. For a grid cell indicator, capture the cell content itself.

Avoid capturing images that include:

• Window borders or shadows (theme-dependent)
• Text that could change (dynamic labels, counters, timestamps)
• Partially transparent or gradient regions where the exact pixel values depend on what's behind them
• UI elements that appear differently when focused vs. unfocused

Using the Image Capture Tool

In Power Automate Desktop, you access image capture through the "Wait for image" or "Move mouse to image" or "Click image on screen" actions. When you configure one of these actions and click the capture button, PAD will minimize itself and let you drag a selection rectangle on the screen.

Be precise with your selection rectangle. Don't include extra whitespace around your target. Every pixel of background you include in the template is a pixel that must match precisely (within tolerance) against the screen. If the application has a 1-pixel shadow that shifts slightly, or if font antialiasing affects the edge pixels of a label, those "extra" pixels around your intended target will reduce match confidence for no benefit.

After capture, PAD shows you the captured template in the action configuration dialog. This is your moment to review it. The captured image should be:

• Sharp and clear, not blurry
• Captured at the same DPI that will be in use during automation execution
• Representing the element in its expected state during the automation (not selected, not hovered, not disabled — unless you're specifically looking for that state)

DPI and Multi-Monitor Considerations

This is where many expert-level automations still break. If you capture templates on a 4K monitor at 150% DPI scaling, then deploy the automation to a machine running at 1080p at 100% DPI scaling, the templates will be a different pixel size than what appears on screen. The match will fail.

There are two approaches to handling this:

Approach 1 - Normalize your execution environment: Configure the automation machine to use a fixed DPI setting, and capture all templates at that setting. For unattended RPA, this means standardizing the display configuration of your machine group. This is the cleanest solution.

Approach 2 - Capture at multiple scales: For each critical image template, capture it at multiple DPI settings and use conditional logic to select which template to use based on the detected screen resolution. This is more complex but allows a single flow to run across heterogeneous machine configurations.

---

The Core Image-Based Action Set

Power Automate Desktop provides several actions that work with image recognition. Here's the full picture of what each one does and when to use it.

Wait for Image on Screen

This is your synchronization workhorse. Before you interact with anything using image-based techniques, you need to confirm the application is in the state you expect. "Wait for image on screen" blocks execution until the template image appears (or disappears, if you configure it that way) on the screen.

Configuration parameters that matter in production:

Search for image on entire screen vs. foreground window vs. specific region: Use the most restrictive option that works. If you're looking for a button in a specific application window, search within the foreground window rather than the entire screen. If you know the button is always in a specific part of the window, define a search region. Narrowing the search region both improves performance (less screen area to scan) and reduces false positive risk.

Wait timeout: Don't use PAD's default timeout blindly. Think about how long this application state legitimately takes to appear under normal conditions, then add a reasonable buffer. If a report generation process takes 30–90 seconds normally, set your timeout to 120 seconds and use error handling to surface meaningful errors when it exceeds that.

Occurrence index: If the template might appear multiple times on screen (multiple identical buttons in a list, for example), the occurrence index lets you specify which one to use.

# Example action configuration (pseudo-notation):
Action: Wait for image on screen
Template: [Captured image of "Submit" button]
Wait for image to: Appear
Search on: Foreground Window
Timeout: 30 seconds
Tolerance: 0.85
On error: Fail with error message "Submit button did not appear within 30 seconds"

Click Image on Screen

Once you've confirmed the image is present, "Click image on screen" performs a click at a position relative to the matched image. The key feature here is the relative position offset. You don't have to click the exact center of the matched image — you can specify an X and Y offset from the image's top-left corner.

This is enormously useful when the target you want to click is consistently positioned relative to a visually distinctive reference image, but the target itself isn't distinctive enough to use as a template. For example: a row of data in a grid where each row has identical-looking checkboxes, but the first column of each row contains a unique order number. You can't use the checkbox as a template (they all look the same), but you can use OCR to find the row for a specific order number, get its Y coordinate, and then click at a fixed X offset that corresponds to the checkbox column.

Move Mouse to Image

This action moves the mouse cursor to the matched image position without clicking. It's useful for triggering hover states — tooltips, dropdown reveals, sub-menus — before taking the next action.

Take Screenshot of Image Region

After matching an image, you can capture the region around it as a new screenshot for further processing. This feeds naturally into OCR workflows where you use a visually distinctive header or label to find a region, then OCR that region to read dynamic text content.

If Image Exists on Screen

This is the conditional form of image recognition — it checks for the image without waiting and returns a boolean result. Use this for branching logic:

If image "ErrorDialog.png" exists on screen
    Then:
        Click image "OK_button.png"
        Log "Error dialog dismissed"
Else:
    Continue with main flow

---

Screen Scraping and OCR: Reading What You Can't Select

Image recognition tells you where something is on screen. OCR tells you what it says. For many real-world automation scenarios involving difficult applications, you need both.

Power Automate Desktop's OCR capabilities go beyond simple "extract text from image" — you can scope OCR to a specific region, and you can combine it with image recognition to OCR a dynamically positioned region. The lesson on Extracting Text from PDFs, Images, and Scanned Documents with OCR in Power Automate Desktop covers OCR in depth. Here we'll focus on the specific patterns for combining OCR with image recognition in UI automation.

Pattern 1: Anchor Image + OCR Region

This is the most powerful pattern for reading dynamic text from applications that don't expose accessible text. The idea:

1. Use "Wait for image on screen" to find a static visual anchor (a column header, a section label, an icon that always appears next to the data you want)
2. Get the anchor image's screen coordinates from the action output
3. Calculate a region offset from those coordinates to define where the actual text is
4. Run OCR on that calculated region

Here's what this looks like in practice. Imagine you're automating a legacy warehouse management system. The application shows an order status next to a small status indicator icon. The icon is visually distinctive (a green circle for open orders). You need to read the order number that appears to the right of this icon.

# Step 1: Find the anchor image
Action: Wait for image on screen
Template: [StatusIcon_Green.png]
Store matched image location in: StatusIconLocation
# StatusIconLocation contains X, Y coordinates of the match top-left corner

# Step 2: Calculate the text region
# Assume the order number text is always 30px to the right and spans 120px wide
Set TextRegionX = StatusIconLocation.X + 30
Set TextRegionY = StatusIconLocation.Y - 5
Set TextRegionWidth = 120
Set TextRegionHeight = 25

# Step 3: OCR the calculated region
Action: Extract text with OCR from screen region
Region: [TextRegionX, TextRegionY, TextRegionWidth, TextRegionHeight]
Store result in: OrderNumberText

Pattern 2: OCR First, Then Coordinate-Based Interaction

Sometimes the opposite flow makes sense. You OCR a region to find a piece of text you care about, then use the returned position of that text to calculate where to click.

PAD's OCR actions can return not just the extracted text but also the position of text elements within the OCR'd region. This lets you find the bounding box of a specific string within a complex block of text, and then derive click coordinates from it.

This pattern is particularly useful for table-like layouts in legacy applications where rows are text-rendered but interaction is coordinate-based:

1. OCR the entire table region
2. Parse the OCR output to find the row containing your target record
3. Use the Y coordinate of that text match to calculate the row's center
4. Use a fixed X coordinate (pre-measured for the action column) for the click

---

Coordinate-Based Actions: The Nuclear Option and How to Use It Safely

Pure coordinate-based interaction — "click at pixel (847, 312)" — is often described as the last resort of RPA, and for good reason. Hardcoded pixel coordinates are brittle in almost every dimension: they break when the window moves, when screen resolution changes, when the application layout changes, when the OS scales differently.

But with the right engineering, coordinate-based actions become a legitimate tool rather than a hack. The key is that coordinates should almost never be static — they should be calculated from a combination of discovered positions, known offsets, and validated assumptions.

Relative vs. Absolute Coordinates

Power Automate Desktop supports both absolute screen coordinates (measured from the top-left corner of the primary display) and coordinates relative to a specific window or region.

Absolute coordinates are fragile for most use cases but have legitimate applications: if you're automating an application that always opens maximized on a known fixed-resolution display, and the layout is completely deterministic, absolute coordinates for specific elements can be reliable. This applies particularly in unattended automation with controlled infrastructure — if you manage the machines and guarantee a fixed resolution and window state, absolute coordinates become much more defensible.

Relative coordinates (relative to a matched image or a found window) are far more resilient. The "Move mouse to image" action, for example, lets you specify a relative offset from the matched image's center. This gives you coordinate-based precision combined with image recognition's self-orienting behavior.

Measuring Offsets Precisely

When you need to measure offsets between a visual anchor and an interaction target, the best tool is a screen ruler or screenshot with pixel coordinates. In practice:

1. Take a screenshot of the application in the target state
2. Open the screenshot in a graphics tool (Paint, GIMP, Photoshop) that displays pixel coordinates as you move your cursor
3. Note the pixel position of your image recognition anchor's center
4. Note the pixel position of the interaction target's center
5. Calculate the X and Y difference — these are your offset values

Do this measurement on the same machine, at the same resolution, at the same DPI settings that will be used during automation execution. Otherwise you're measuring offsets that won't translate correctly.

Using the Send Mouse Click and Move Mouse Actions

The primary coordinate-based actions in PAD are:

Move mouse: Moves the cursor to specified X, Y coordinates. Accepts both absolute and relative-to-window coordinates. Can also receive a variable as the coordinate, which is how you implement the pattern of "move to a calculated position."

Send mouse click: Performs a click (left, right, double, middle) at the current cursor position or at specified coordinates. This is the terminal action after a Move mouse call in scenarios where you need fine control over the timing between mouse movement and click.

Scroll mouse wheel at position: For applications where scrolling is required to reveal content, this lets you scroll at a specific coordinate — useful for virtual scrolling lists where the scrollable region doesn't respond to standard scroll actions.

Building Dynamic Coordinate Calculation into Your Flow

Here's a complete pattern that demonstrates dynamic coordinate calculation for a grid-based legacy application where each row has a clickable "Edit" region at a fixed X position but variable Y positions depending on where the data appears:

# Variables established from measurement:
# EditColumnX = 847 (fixed X coordinate of the Edit button column)
# RowHeight = 22 (pixel height of each row)
# FirstRowY = 210 (Y coordinate of the first data row's center)

# After using OCR to find target record at row index TargetRowIndex:
Set ClickY = FirstRowY + (TargetRowIndex * RowHeight)
Set ClickX = EditColumnX

Action: Move mouse
X: ClickX
Y: ClickY

Action: Wait (200 milliseconds)  # Allow any hover state rendering

Action: Send mouse click
Button: Left click

This pattern turns coordinate calculation into a mathematical model of the application's layout. When the application's layout changes (say, the header section gets taller), you update FirstRowY and the rest of the model adapts automatically.

---

Handling Tolerance, False Positives, and Confidence Scoring

In production, image recognition failures come in two flavors: false negatives (the image is there but wasn't matched) and false positives (the image wasn't there but something else matched). Both are dangerous, but false positives can be particularly harmful because the flow proceeds thinking it found the right element.

Setting Tolerance Correctly

PAD's image recognition tolerance is typically expressed as a value between 0 and 1, where 1 requires a perfect pixel match and 0 would match anything. In practice:

• 0.95 – 1.0: Use only when you have absolute control over the rendering environment and can guarantee pixel-perfect consistency. Brittle in most real-world scenarios.
• 0.85 – 0.94: Good starting point for controlled environments. Handles minor antialiasing variation and minor color differences from font rendering.
• 0.70 – 0.84: Handles moderate environmental variation, appropriate for applications where you expect theme or DPI differences but have constrained execution environments.
• Below 0.70: You're accepting significant visual difference between template and target. Carefully validate that you're not getting false positives at this tolerance level.

The calibration process: start at 0.90, run the automation against your target application across the range of machines and configurations it will encounter, identify the tolerance at which you start getting false negatives, and set your production threshold 0.05 below that floor.

Restricting Search Region to Prevent False Positives

The single most effective technique for eliminating false positives is constraining the search region. If the button you're looking for is always in the top-right quadrant of a specific application window, configure your image recognition actions to only search that region.

This also dramatically improves performance. Scanning the full screen for an image match involves thousands of template comparison operations. Restricting to a 200×200 pixel region reduces the comparison count by orders of magnitude.

Building a Validation Layer

For critical interactions, add a validation step after image recognition but before clicking. This could be:

• A secondary image recognition check for a confirming visual element
• An OCR read of nearby text to confirm the application is in the expected state
• A window title or foreground window check to confirm focus is where expected

# Primary match
Action: Wait for image on screen
Template: [SubmitButton.png]
Store location in: ButtonLocation

# Validation: confirm we're in the right dialog
Action: If image exists on screen
Template: [OrderConfirmationDialog_Header.png]
If image exists:
    # Safe to proceed
    Action: Click image on screen [SubmitButton.png]
Else:
    # Unexpected state - surface an error
    Action: Throw error "Submit button found but confirmation dialog header not present - unexpected application state"

---

Building Fallback Chains: Selectors → Image Recognition → Coordinates

The most resilient production automations don't commit to a single interaction technique. They implement a fallback chain that tries the most reliable method first and gracefully degrades to more flexible (but more brittle) methods only when necessary.

Here's the architectural pattern for a three-tier fallback:

# Tier 1: Try selector-based interaction (most reliable when it works)
On block error (set ErrorOccurred = True):
    Action: Click UI element [Selector: #SubmitButton]
    Set InteractionSucceeded = True

# Tier 2: If selector failed, try image recognition
If ErrorOccurred = True OR InteractionSucceeded = False:
    Set ErrorOccurred = False
    On block error (set ErrorOccurred = True):
        Action: Wait for image on screen [SubmitButton.png] timeout: 5s
        Action: Click image on screen [SubmitButton.png]
        Set InteractionSucceeded = True

# Tier 3: If image recognition failed, try coordinate-based
If ErrorOccurred = True OR InteractionSucceeded = False:
    # Log the degradation for monitoring
    Action: Log "WARNING: Falling back to coordinate-based click for Submit button"
    Action: Move mouse X: SubmitButtonX Y: SubmitButtonY
    Action: Send mouse click (left)
    Set InteractionSucceeded = True

# Validate result regardless of which tier was used
Action: Wait for image on screen [SubmissionConfirmation.png] timeout: 10s

This pattern has some important properties. It tries the most fragile method last and the most reliable method first. It logs when degradation occurs — this is critical for operations teams to know that image recognition or coordinate-based fallback is being triggered, because it's a signal that the environment has drifted and the primary automation strategy needs updating. And it always validates success at the end, regardless of which interaction tier was used.

For a deeper look at how to structure reusable blocks of logic like this, see Subflows and Reusable Logic in Power Automate Desktop — this three-tier fallback pattern is an excellent candidate for encapsulation in a subflow that accepts the UI element, image template, and coordinates as input variables.

---

Practical Application: Automating a Citrix-Delivered Application

Let's work through a realistic end-to-end scenario. You're automating a procurement workflow in a Citrix-delivered SAP environment. The Citrix client renders the SAP GUI as a pixel stream — no UI Automation tree access. Standard SAP scripting (which works beautifully in direct SAP GUI installations, as covered in Automating SAP GUI Interactions with Power Automate Desktop) is not available in this configuration.

Your task: Log into the Citrix-delivered SAP, navigate to transaction ME23N (Display Purchase Order), enter a PO number from a data table, and extract the vendor name and total value from the displayed PO header.

Step 1: Establish a Stable Citrix Session State

Before any image recognition work, you need the Citrix window in a predictable state. Maximize it programmatically:

Action: Get window [Citrix Receiver / Citrix Workspace application window]
Action: Set window state to: Maximized
Action: Wait 2 seconds

Maximize the window before every session start. This eliminates coordinate drift caused by window position variability.

Step 2: Image Recognition for SAP Login Fields

Capture image templates for:

• The SAP login dialog (the logo or a distinctive header element — use as a state confirmation image)
• The "Client" field label (to anchor your first interaction)

Action: Wait for image on screen [SAPLogonDialog.png]
# Confirms we're at the SAP logon screen

Action: Wait for image on screen [ClientFieldLabel.png]
Store location in: ClientLabelLocation

# Click into the Client field (known offset from the label)
Set ClientFieldX = ClientLabelLocation.X + 80
Set ClientFieldY = ClientLabelLocation.Y
Action: Move mouse X: ClientFieldX Y: ClientFieldY
Action: Send mouse click (left)
Action: Send keys "100"  # Client number

# Tab to next field
Action: Send keys "{Tab}"
Action: Send keys "your_username"
Action: Send keys "{Tab}"
Action: Send keys "%PasswordVariable%"
Action: Send keys "{Enter}"

Step 3: Navigate to ME23N

After login, you'll be at the SAP Easy Access menu. The command field (where you type transaction codes) is at a predictable position in a maximized SAP window, but use image recognition to confirm first:

Action: Wait for image on screen [SAPEasyAccessIcon.png] timeout: 30s
# Confirms successful login

Action: Wait for image on screen [CommandFieldBorder.png]
Store location in: CommandFieldLocation

Set CommandX = CommandFieldLocation.X + 20
Set CommandY = CommandFieldLocation.Y + 10
Action: Move mouse X: CommandX Y: CommandY
Action: Send mouse click (left)
Action: Send keys "/nME23N{Enter}"

Step 4: Enter PO Number and Extract Data

Action: Wait for image on screen [ME23N_PONumberFieldLabel.png] timeout: 15s
Store location in: POLabelLocation

Set POFieldX = POLabelLocation.X + 120
Set POFieldY = POLabelLocation.Y
Action: Move mouse X: POFieldX Y: POFieldY
Action: Send mouse click (left)
Action: Send keys {Control}A  # Select all in field
Action: Send keys CurrentPONumber  # From your data table loop
Action: Send keys "{Enter}"

# Wait for PO to load
Action: Wait for image on screen [ME23N_HeaderDataSection.png] timeout: 20s

# Use OCR to extract vendor name from the vendor field region
Action: Wait for image on screen [VendorFieldLabel.png]
Store location in: VendorLabelLocation

Set VendorTextX = VendorLabelLocation.X + 100
Set VendorTextY = VendorLabelLocation.Y - 3
Set VendorTextW = 200
Set VendorTextH = 20

Action: Extract text with OCR from screen region
Region: [VendorTextX, VendorTextY, VendorTextW, VendorTextH]
Store in: ExtractedVendorName

This workflow layers every technique: image recognition for application state confirmation and field location, coordinate calculation for interaction with known offsets from visual anchors, OCR for data extraction from rendered text, and keyboard input for data entry.

---

Performance Considerations for Image-Based Automation

Image recognition is computationally heavier than selector-based interaction. A single full-screen image match operation involves capturing a screenshot (~2-10ms) and then performing template matching across millions of pixel locations (variable, but can be 50-500ms for a full 1080p screen). When you have dozens of image recognition steps in a flow, this adds up.

Optimization strategies:

Minimize search region aggressively: Constraining search to a 300×200 pixel region instead of a 1920×1080 screen can reduce matching time by 20-40x. Do this wherever you have any knowledge of where the target image will appear.

Use "Wait for image" timeout as a diagnostic signal: If you're setting a 10-second timeout but the image consistently appears in under 1 second, that's fine. But if you're seeing the flow regularly consume 5+ seconds waiting for images, that's a signal that either your template tolerance needs tuning or there's a rendering performance issue in the target application.

Cache image locations: If you've found an image's position and the application hasn't scrolled or resized, the position is still valid for subsequent operations on the same element. Store it in a variable and use coordinate-based subsequent interactions rather than re-running image recognition on every step.

Parallelize state checks where possible: While PAD doesn't offer true parallelism within a single flow, you can structure your "check application state" image recognition steps to be as sequential-efficient as possible by ordering them from most-likely to least-likely state.

---

Hands-On Exercise

Build a complete automation for the following scenario:

Scenario: You have access to a local installation of the Windows Calculator application. Your goal is to build a flow that:

1. Launches Calculator and maximizes it
2. Uses image recognition (not UI elements) to verify it's in Standard mode
3. Switches it to Scientific mode using keyboard shortcuts, then verifies using image recognition
4. Uses a combination of image recognition and coordinate-based clicks to calculate sin(45)
5. Uses OCR to read the result from the display region and stores it in a variable
6. Logs the result and closes Calculator

Steps to build this:

1. Launch Calculator using the "Launch application" action, then maximize its window.

2. Capture an image template of the Calculator's standard mode layout (capture a distinctive portion of the standard mode button layout that won't appear in scientific mode).

3. Add a "Wait for image on screen" action using your template to confirm standard mode.

4. Send Ctrl+2 (the keyboard shortcut to switch to Scientific mode). Wait 500ms.

5. Capture a second template from scientific mode (the degree/radian indicator, or the sin/cos/tan button row — these only appear in scientific mode).

6. Add a "Wait for image on screen" action to confirm scientific mode loaded.

7. Use image recognition to locate the "4" button and click it with an offset to also click "5" (or use separate image templates for each digit button).

8. Use image recognition to locate and click the "sin" function button.

9. Capture a template of a pixel region around the result display area, then OCR that region to extract the result.

10. Add a "Write to text file" or "Display message" action to output the extracted result.

11. Close Calculator.

Challenge extension: Wrap your image recognition interactions in an error-handling block that captures a screenshot on failure and writes it to a diagnostics folder. Then intentionally break one of your templates (edit the image in Paint to change a few pixels) and verify that your error handling fires correctly.

---

Common Mistakes & Troubleshooting

Mistake 1: Capturing Images During Development on a Different Machine Than Execution

If you build your flow on your developer workstation (maybe a 4K display at 150% DPI) and deploy to a standard 1080p machine at 100% DPI, every single image template will fail. The solution is to always capture templates in the execution environment or a machine that exactly replicates it.

Mistake 2: Using Too Large a Capture Region

A template that includes surrounding UI (window chrome, adjacent controls, background gradients) will fail more often than a tightly cropped template. When in doubt, crop tighter.

Mistake 3: Not Accounting for Application Load Time

Image recognition failures that seem intermittent are often timing failures. The application is still rendering when your "Wait for image" action runs, so the image isn't fully formed yet. Add explicit "Wait" actions (100-300ms) after navigation steps before your image recognition checks, and ensure your timeout values are generous enough to cover slow rendering scenarios.

Mistake 4: Forgetting that Screen Scraping is Session-Aware

For unattended flows, the machine must have an active desktop session — not just a user logged in but also a visible desktop. If the machine locks its screen, image recognition and OCR will fail (they see the lock screen, not your application). Design your unattended architecture accordingly, using machine configurations that keep the session unlocked. See Managing Machines and Machine Groups for Scalable Unattended Automation in Power Automate for proper machine configuration.

Mistake 5: Using OCR Without Specifying the Right OCR Engine

PAD supports both Windows-native OCR and Tesseract OCR. They have different strengths: Windows-native OCR handles printed text in UI applications well; Tesseract can be configured for different languages and document types. For Citrix-delivered applications with SAP-style text rendering, Windows-native OCR typically performs better. Always test both before committing to one.

Mistake 6: Not Validating Interaction Success

With selector-based interaction, PAD can often confirm that a click succeeded because the element returned confirmation. With image recognition and coordinate-based clicks, the click simply fires at a location. Validate that the interaction produced the expected result by checking for a subsequent expected state (a confirmation dialog appearing, the form clearing, the next screen loading).

Troubleshooting: "Image not found" errors that are inconsistent

This is almost always one of: timing (add more wait time before the check), tolerance (adjust up slightly), template size (crop tighter or add more distinctive content), or a genuine application state issue (the application sometimes shows a loading spinner or intermediate dialog). Enable screenshot capture on error to see exactly what the screen looks like when the match fails.

Troubleshooting: Matches the wrong element

Increase the tolerance to make matching stricter. Restrict the search region. Consider adding more distinctive content to your template. If two elements genuinely look identical, you need a different strategy — use the anchor-plus-offset pattern to select among identical-looking elements by position rather than appearance.

---

Summary & Next Steps

You now have a complete framework for automation scenarios where standard selectors fail. The key mental model to carry forward: image recognition, OCR, and coordinate-based actions aren't inferior techniques to be used only as a last resort — they're a different layer of the automation stack that operates on what's visually rendered rather than what's programmatically exposed. The skill is knowing when to deploy each layer and how to combine them into automation that's as resilient as the environment requires.

The principles we covered:

• Image recognition works through template matching and is sensitive to scale, color, and rendering environment — tune your templates and tolerance for the specific execution environment
• OCR combined with image recognition enables reading dynamic text from applications that don't expose accessible text interfaces
• Coordinate-based clicks should always be derived from discovered positions and known offsets, never hardcoded as static values
• Fallback chains that gracefully degrade from selector → image → coordinate give you maximum resilience without sacrificing reliability for environments where selectors do work
• Always validate interaction success after image-based or coordinate-based clicks — don't assume the click hit the right target

Where to go from here:

• For multi-application workflows where you're combining image-based techniques with standard automation across Excel, browsers, and desktop apps in a single flow, see Automating Multi-Application Workflows in Power Automate Desktop
• For advanced scripting techniques that can complement image-based automation (particularly PowerShell for querying display configuration), see Scripting Inside Desktop Flows: Running PowerShell, Python, and VBScript Actions
• For building monitoring and alerting around flows that use these brittle-by-nature techniques, see Monitoring and Troubleshooting Desktop Flow Runs at Scale
• For the recorder-based approach to initially capturing interactions (which you may want to use as a starting point before converting to image-based), see Capturing and Replaying Mouse Clicks and Keystrokes with the Power Automate Desktop Recorder