Most web scraping tutorials cover the happy path. This lesson covers production reality: dynamic tables, pagination loops that detect the last page automatically, session timeouts mid-run, and writing thousands of extracted rows to clean CSV and Excel output files — without losing a single row when something goes wrong.
You've been handed a task that sounds deceptively simple: pull the last 90 days of vendor invoices from the company's procurement portal, cross-reference them with the supplier list, and drop everything into a structured file that finance can use for reconciliation. The portal has no API. There's no export button. The data lives across 47 pages of a web table, the page count changes every month, and the "Next" button sometimes goes missing entirely when you hit the last page. Welcome to real-world web scraping with Power Automate Desktop.
Most tutorials cover the happy path — extract a static table, save it to a file, done. That's not what production automation looks like. What you actually face is dynamic pagination where the total page count is only visible at runtime, tables that rebuild themselves in the DOM after JavaScript executes, stale element references that cause your selectors to break mid-loop, and the need to write thousands of rows to a structured output without corrupting your data or running out of memory. This lesson covers all of it.
By the end of this article, you'll have built a complete, production-grade multi-page scraping flow that handles dynamic table structures, navigates pagination reliably, recovers from common failures, and writes clean structured output to both CSV and Excel. You'll understand not just what to configure, but why certain approaches succeed where others silently fail.
What you'll learn:
Before working through this lesson, you should be comfortable with:
Before writing a single action, you need to understand the failure modes. Web scraping loops in Power Automate Desktop break in predictable ways, and most of them stem from two root causes: selector fragility and state mismanagement.
Selector fragility happens when you capture a UI element against the DOM as it exists at capture time, then try to use that element again after the page has re-rendered. Every time you click "Next Page," the browser tears down and rebuilds the table. If your next-button selector was built using positional attributes (like nth-of-type or absolute XPath positions) rather than stable semantic attributes (like a specific aria-label, data-testid, or meaningful class name), it will silently fail or, worse, click the wrong thing.
State mismanagement shows up when you accumulate data incorrectly. A common mistake is creating a new DataTable inside the loop — which means you overwrite your previous page's results on every iteration instead of appending to a single collection. Another variant is writing to the output file inside the loop, which means you either overwrite the file on every page or append rows with the headers repeated on every pass.
We'll address both of these systematically as we build the flow. The architecture we'll use keeps the DataTable outside the loop, writes to the file exactly once at the end, and uses stable semantic selectors with deliberate waits to handle DOM re-rendering.
Key insight
The most reliable web scraping flows treat the browser like an unreliable API — you always wait for confirmation that the next state has loaded before interacting with it. Never click "Next" and immediately try to extract. Always wait for a DOM signal that the new page's content is present.
We'll build against a realistic scenario: a vendor management portal that lists purchase orders in a paginated table. The table has columns for PO Number, Vendor Name, Amount, Issue Date, and Status. The portal shows 25 rows per page, with a footer that displays "Page X of Y" and a "Next" button that disappears on the last page.
Start by opening a new flow and adding the Launch new Microsoft Edge action (or Launch new Chrome action if your environment uses Chrome). Set the URL to your portal's login page and configure it to wait for the page to load completely.
After the browser launches, you'll handle authentication. For this scenario, assume credentials come from a secure variable (not hardcoded — if you need to understand credential management, review Handling Credentials Securely in Desktop Flows: Sensitive Variables and Azure Key Vault). Navigate to the PO listing page once authenticated.
Now set up your foundational variables before entering any loop:
# Variables to initialize before the loop
Set variable CurrentPage to 1
Set variable TotalPages to 1 # Will be updated after first extraction
Set variable HasNextPage to True
Create new data table: AllExtractedData
Columns: PONumber, VendorName, Amount, IssueDate, Status
The AllExtractedData DataTable is critical — this is your accumulator. It lives outside any loop and persists across every page iteration. Every page's rows get appended to this single structure.
Warning
Do not initialize AllExtractedData inside your pagination loop. If you do, you'll recreate an empty table on every iteration and lose every previous page's data. This is the single most common scraping mistake in Power Automate Desktop, and it produces no error — just silently incomplete output.
With the browser on the first page of results, it's time to configure the extraction. Use the Extract data from web page action. When you click "Live web helper" and hover over the table, Power Automate Desktop will attempt to detect the table structure automatically.
Here's where you make a crucial configuration decision: Extraction Mode.
When PAD detects a table, it offers two approaches under the hood. If you let it auto-detect the entire table as a structured object, it will generate an extraction that grabs all visible rows and their cell values in a single pass. This is what you want for tabular data. However, if your table is rendered by JavaScript (which is true of most modern procurement portals, React apps, and Angular-based systems), the auto-detect may grab an empty shell if the JavaScript hasn't finished populating the rows by the time PAD queries the DOM.
To handle JavaScript-rendered tables reliably:
In the Live web helper, click the first data cell in the table (not the header). PAD will prompt you to select more data to establish a pattern. Click the same column in the second row. PAD will infer the pattern and ask if you want to extract the entire list — select yes. Then extend the selection to include all five columns by clicking each column header in the extraction UI and mapping it to a column name.
The output of this action will be a DataTable variable. Name it PageData.
Tip
When mapping columns in the extraction UI, always assign explicit column names (PONumber, VendorName, Amount, IssueDate, Status) rather than accepting the default Column1, Column2 names. If the table ever gains a column upstream, your named mappings survive the change; positional defaults will silently shift your data into the wrong columns.
After each extraction, you need to merge PageData (the current page's rows) into AllExtractedData (your accumulator). Power Automate Desktop doesn't have a built-in "append DataTable to DataTable" action, so you'll use a For each loop over PageData rows and the Add row to data table action.
For each CurrentRow in PageData:
Add row to data table AllExtractedData with values:
Column 'PONumber' = CurrentRow['PONumber']
Column 'VendorName' = CurrentRow['VendorName']
Column 'Amount' = CurrentRow['Amount']
Column 'IssueDate' = CurrentRow['IssueDate']
Column 'Status' = CurrentRow['Status']
This inner loop runs for every row on every page. For a 25-row table across 47 pages, that's 1,175 iterations of this inner loop — plus the outer pagination loop. This is entirely within PAD's performance envelope; a flow this size typically completes in 3–8 minutes depending on network latency and page load time.
One important optimization: if you're dealing with very large extractions (thousands of rows per page, hundreds of pages), consider batching your writes. Instead of accumulating everything in memory, write to the output file every N pages and reset the accumulator. We'll cover the output writing mechanism shortly, but keep this architecture option in mind.
This is the most architecturally interesting part of the flow. You have two general strategies for pagination detection:
Strategy A: Read total page count on first load, loop a fixed number of times. This works when the "Page X of Y" indicator is reliable and the total count is stable during your run.
Strategy B: Check for the presence of the "Next" button on each iteration. This works when the total page count is unknown, hidden, or subject to change during the run.
For production use, Strategy B is generally more resilient. Here's why: if the portal adds new records while your flow is running (which happens with live procurement systems), the page count can increase mid-run. Strategy A would stop early. Strategy B will continue as long as a Next button exists.
However, we can combine both for belt-and-suspenders reliability:
# Outer pagination loop
Loop while HasNextPage = True:
# 1. Wait for table content to be present
Wait for web page element 'TableFirstDataCell' to exist
Timeout: 20 seconds
On timeout: Set HasNextPage to False, Break loop
# 2. Extract current page data
Extract data from web page → PageData
# 3. Append rows to accumulator
For each CurrentRow in PageData:
Add row to AllExtractedData...
# 4. Check for Next button existence
If web page element 'NextPageButton' exists:
Click web element 'NextPageButton'
Wait for web page element 'TableFirstDataCell' to not exist
Wait for web page element 'TableFirstDataCell' to exist
Increment CurrentPage by 1
Else:
Set HasNextPage to False
The double-wait pattern after clicking Next is the key to reliable pagination. The first wait (for element to not exist) confirms that the old table has been torn down. The second wait (for element to exist) confirms the new table has been rendered. Without both, you risk extracting the previous page's data again because the DOM transition happened faster than your extraction action fired.
Warning
Never use a fixed Wait action (like "Wait 3 seconds") between page navigations. Fixed waits are both unreliable (3 seconds might not be enough on a slow network day) and inefficient (3 seconds is always too long when the page loads in 0.8 seconds). Always wait for a specific DOM condition using Wait for web page content or element existence checks.
Let's also handle the edge case where PageData comes back empty. Some portals render a table shell with zero rows as a valid response — for example, a filtered result set with no matches. If you don't check for this, you'll loop forever (the Next button may still exist as a phantom element) or accumulate hundreds of empty rows in your output.
Add a row count check immediately after extraction:
If count of rows in PageData = 0:
Set HasNextPage to False
Break loop (or log a warning)
Your Next button selector deserves careful construction. By default, PAD's recorder will grab whatever attributes are most prominent in the DOM at capture time — which is often a combination of CSS class and position. This breaks the moment the portal team updates their CSS or restructures their navigation markup.
Build a more resilient selector for UI elements and selectors in Power Automate Desktop using these priority guidelines:
aria-label attributes. A button with aria-label="Next page" is highly stable — it exists for accessibility compliance and doesn't change with visual redesigns.data-testid or data-automation-id attributes if they exist. These are placed explicitly for testing and automation and are the most stable identifiers available.:nth-child(3) or absolute XPath expressions like /html/body/div[2]/div[1]/nav/button[2].In the PAD UI element editor, you can manually edit the generated selector by switching to "Custom selector" mode and writing a CSS selector or XPath directly. For a pagination button with aria-label="Go to next page", your CSS selector would be:
button[aria-label="Go to next page"]
Or as XPath:
//button[@aria-label="Go to next page"]
For the table content detection wait (the signal that new content has loaded), target something specific to the table data itself rather than a structural element. A good candidate is a cell in the first data row — for example, the PO number cell, which will always contain a value when the table is populated:
table#purchase-orders tbody tr:first-child td:first-child
This selector says: within the purchase-orders table body, find the first cell of the first row. It only matches when actual data rows exist, not when the table shell is empty.
Some modern portals don't use traditional pagination at all — they use infinite scroll, where new rows are appended to the same table as the user scrolls down. The extraction strategy is completely different here.
For infinite scroll portals:
End key to the browser) to trigger the next batch of rows.Set PreviousRowCount to 0
Loop while True:
Extract data from web page → PageData
Set CurrentRowCount to count of rows in PageData
If CurrentRowCount = PreviousRowCount:
Break loop # No new rows loaded, we're at the bottom
Set PreviousRowCount to CurrentRowCount
Scroll down on web page
Wait for element 'LoadingSpinner' to not exist
Wait 1 second # Small buffer for DOM stabilization
With infinite scroll, you extract the entire accumulated table on each iteration rather than just the new page's rows. This means you want PageData to be your final output (not an accumulator) — or you'll need to extract only the newly added rows by slicing the DataTable based on row count deltas.
Note
The infinite scroll approach involves repeated full-table extractions that grow in size with each scroll. For a table that ultimately contains 5,000 rows loaded in 50-row batches, you'll perform 100 extractions where each one is larger than the last. This is O(n²) in extraction cost. For very large infinite-scroll tables, consider extracting only new rows by keeping track of the last extracted row index and slicing accordingly.
Once your pagination loop completes and AllExtractedData contains every row from every page, it's time to write the output. For reading and writing to CSV and text files in Power Automate Desktop, the cleanest approach is to use the Write to CSV file action directly against your DataTable — this action handles quoting, delimiter handling, and optional headers in a single step.
Configure the action as follows:
AllExtractedDataC:\RPA\Output\PO_Extract_%CurrentDateTime%.csvTo build the timestamped filename, create a DateTime variable before the loop:
Get current date and time → RunTimestamp
Format datetime RunTimestamp using format 'yyyyMMdd_HHmmss' → TimestampString
Set OutputFilePath to 'C:\RPA\Output\PO_Extract_' + TimestampString + '.csv'
This produces filenames like PO_Extract_20241115_143022.csv, which are sortable chronologically and prevent overwriting previous runs.
Tip
Always write your output file to a dedicated folder, not to the Desktop or Downloads. Use a path like C:\RPA\Output\ that your automation machine's service account has explicit write permissions to. On unattended machines, Desktop paths are session-dependent and may resolve to unexpected locations — this is a common source of "file not found" errors in production.
For stakeholders who need the data in Excel format with formatted columns, use the Excel actions instead of (or in addition to) the CSV write. The automating Excel with Power Automate Desktop workflow for writing a DataTable to a new workbook looks like this:
Launch Excel with blank workbook → ExcelInstance
Write to Excel worksheet:
Excel instance: ExcelInstance
Value to write: AllExtractedData
Write mode: Write on specified cell
Start column: A
Start row: 1
Write column names as first row: Yes
After writing the data, apply basic formatting so the output is immediately usable:
# Set column widths (use the 'Resize column/row' action or a macro)
Set cell value in column A row 1 to 'PO Number'
Set cell value in column B row 1 to 'Vendor Name'
# ... etc.
# Auto-fit columns
Run Excel macro: 'Columns("A:E").AutoFit()'
# Format Amount column as currency
Run Excel macro: 'Range("C2:C" & Cells(Rows.Count,"C").End(xlUp).Row).NumberFormat = "$#,##0.00"'
Save the workbook to a file:
Save Excel workbook as:
Excel instance: ExcelInstance
Document format: Excel workbook (.xlsx)
File path: C:\RPA\Output\PO_Extract_%TimestampString%.xlsx
Then close Excel:
Close Excel:
Excel instance: ExcelInstance
Before closing: Do not save document (already saved above)
A multi-page scraping flow that runs unattended needs error handling at multiple layers. A single unhandled exception mid-loop can lose everything you've accumulated. Design your error handling with these layers:
Layer 1: Per-action error handling for the extraction itself. Wrap the Extract data from web page action in an On Block Error block. If extraction fails, log the current page number and attempt a page refresh before retrying:
On block error:
If error action is 'Extract data from web page':
Log 'Extraction failed on page ' + CurrentPage
Refresh web page
Wait for element 'TableFirstDataCell' to exist (timeout 30s)
Retry the block (max 2 retries)
Else:
Write partial results to CSV (don't lose accumulated data)
Re-throw error
Layer 2: Per-page failure isolation. If a page consistently fails after retries, don't abort the entire flow. Skip that page, record it in a separate error log, and continue to the next page. Your output will be slightly incomplete, but 46 out of 47 pages is far more useful than zero.
Layer 3: Session timeout handling. Long-running scraping flows frequently encounter session timeouts from the portal. Build a session check into the loop — if the page redirects to a login page (detectable by checking the current URL or looking for a login form element), re-authenticate and navigate back to the correct page.
Get current URL of web page → CurrentURL
If CurrentURL contains 'login' or CurrentURL contains 'session-expired':
# Re-authenticate
Call subflow: AuthenticateToPortal
# Navigate back to the correct page
Navigate browser to: BaseURL + '?page=' + CurrentPage
Wait for element 'TableFirstDataCell' to exist
Factoring your authentication into a subflow for reusable logic is especially useful here — you can call it from both the initial setup and the session-recovery branch without duplicating actions.
Key insight
Write your accumulated DataTable to a checkpoint file every 10 pages. If the flow crashes on page 43 of 47, you can recover from the checkpoint rather than starting over. The overhead of writing a 1,000-row CSV file every 10 iterations is negligible compared to the cost of re-running a 45-minute flow from scratch.
When you're scraping tens of thousands of rows across hundreds of pages, the naive approach starts to show performance problems. Here are the optimizations that matter:
Minimize DOM wait time. Your wait-for-element timeouts represent the worst-case load time you're willing to accept. Setting every timeout to 30 seconds means a slow page costs you 30 seconds even when the content loads in 2. Use adaptive waits: start with a short timeout (5 seconds), catch the timeout exception, and retry with a longer window. Most pages load fast; only a few will need the full timeout.
Disable browser extensions and unnecessary rendering. If you're running headless or via an unattended machine, launch the browser with arguments that disable GPU rendering, animations, and image loading. Images are irrelevant for data extraction and consume bandwidth and rendering time. For Chrome:
Launch Chrome with arguments: '--disable-extensions --blink-settings=imagesEnabled=false --disable-gpu'
Batch your Excel writes. If you're writing to Excel, don't use the "Write to Excel worksheet" action row by row in a loop — this is extremely slow due to COM interop overhead on each call. Always write the complete DataTable in a single call after your loop completes. A single 5,000-row DataTable write to Excel takes about 2–3 seconds; 5,000 individual row writes take several minutes.
Use PAD's built-in DataTable operations instead of loops where possible. The Filter data table and Sort data table actions operate on the entire table at once in native code, far faster than iterating rows with a For Each loop in PAD's interpreted action engine.
Build the following complete flow against a publicly available paginated data source. We'll use the GitHub repository search results page, which paginates at 10 results per page and has a reliable "Next" navigation link.
Objective: Scrape the top 5 pages of GitHub repository search results for "power automate," extracting repository name, owner, star count, and last updated date, then write the results to a timestamped CSV file.
Step 1: Initialize variables and launch browser
Set CurrentPage to 1
Set HasNextPage to True
Create data table AllRepos with columns: RepoName, Owner, Stars, LastUpdated
Get current datetime → RunTimestamp
Format RunTimestamp as 'yyyyMMdd_HHmmss' → TimestampString
Set OutputPath to 'C:\RPA\Output\GitHub_Repos_' + TimestampString + '.csv'
Launch new Microsoft Edge
Navigate to: https://github.com/search?q=power+automate&type=repositories
Step 2: Build the pagination loop with a maximum page guard
Loop while HasNextPage = True AND CurrentPage <= 5:
Wait for web page element matching CSS 'li.repo-list-item' to exist (timeout 15s)
Extract data from web page:
Target: The repository list
Map columns: RepoName, Owner, Stars, LastUpdated
Output → PageData
If count of rows in PageData = 0:
Set HasNextPage to False
Break
For each Row in PageData:
Add row to AllRepos with:
RepoName = Row['RepoName']
Owner = Row['Owner']
Stars = Row['Stars']
LastUpdated = Row['LastUpdated']
If web element matching CSS 'a[rel="next"]' exists:
Click web element 'a[rel="next"]'
Wait for web element 'li.repo-list-item' to not exist (timeout 10s)
Wait for web element 'li.repo-list-item' to exist (timeout 15s)
Increment CurrentPage by 1
Else:
Set HasNextPage to False
End Loop
Step 3: Write output
Write AllRepos to CSV file:
File: OutputPath
Encoding: UTF-8
Include column names: Yes
Separator: Comma
Display message: 'Extraction complete. ' + count of rows in AllRepos + ' repositories written to ' + OutputPath
Expected outcome: A CSV file containing up to 50 rows (10 per page × 5 pages) with clean column headers and no duplicate data. Run it twice; you should get two separate timestamped files with identical content (assuming search results haven't changed).
Extension challenge: Modify the flow to accept the search term as an input variable passed from a cloud flow trigger, making the scraper reusable for any GitHub search query. Explore triggering desktop flows from cloud flows to understand how to wire up the input parameter.
Problem: Flow extracts the same page's data on every iteration
This is the stale-extraction problem. Your extraction action is firing before the DOM has updated with the new page's content. Solution: Add the double-wait pattern (wait for old content to disappear, then wait for new content to appear) between the Next button click and the extraction. Also verify your wait is targeting a dynamic element — not a structural container that persists across page loads.
Problem: AllExtractedData has rows with all empty values
The column names in your Add row to data table action don't match the column names in AllExtractedData. Column name matching in PAD's DataTable actions is case-sensitive. If you defined the table with column PONumber and you're referencing poNumber or PO Number, the value goes nowhere and the cell stays empty. Double-check every column reference.
Problem: Next button click succeeds but the page doesn't change
Some portals require the Next button to be scrolled into view before it can be clicked. Add a Scroll web page to element action before the click. Alternatively, the button may be intercepted by an overlay (a cookie banner, a promotional modal) that appeared after initial page load. Use element existence checks to detect and dismiss these overlays.
Problem: Extraction captures header row as a data row
This happens when the extraction pattern was defined by clicking the header cells rather than the data cells. Re-run the extraction setup and ensure your first click is on a data cell (row 2 or below), not the <th> header cell. Alternatively, after extraction, filter out the first row if its values match your expected column headers.
Problem: CSV file contains garbled characters for vendor names
Encoding mismatch. Set the CSV write encoding to UTF-8 with BOM (UTF-8 with BOM option). The BOM causes Excel to correctly interpret the file as UTF-8 when opened directly. Without the BOM, Excel may interpret the file as your system's default code page, which corrupts accented characters and non-Latin scripts.
Problem: Flow fails on page 1 of 1 (single-page results)
Your loop assumes a Next button will always eventually appear. If the result set fits on one page, the Next button never exists, HasNextPage gets set to False immediately, and the loop exits after one iteration — which is correct behavior. But if you have a minimum-pages guard (CurrentPage <= 5) without the Next-button check, you'll attempt to scrape pages that don't exist. Ensure the Next-button existence check always takes priority over the page counter limit.
Problem: Session expires midway through a long scraping run
Add a URL check at the start of each loop iteration. If the URL contains session-expiry indicators (login, timeout, session-expired), call your authentication subflow and re-navigate to the correct paginated URL. For portals that use token-based auth with short expiry windows, you may need to proactively re-authenticate every N pages rather than reactively detecting the expiry.
Warning
Some portals actively detect and block bot-like behavior — specifically, navigation that happens too fast or too consistently timed. If you encounter CAPTCHA challenges or IP-based blocking during scraping, you'll need to add randomized delays (within a reasonable range, like 1–3 seconds per page) and ensure you're running from a known, whitelisted IP address. Check your organization's terms of service and acceptable use policies before automating portal interactions.
When this flow moves from development to production, several additional considerations apply.
Output file management: Your flow creates a new timestamped file on every run. After 30 days of daily runs, you have 30 output files. Build a cleanup step that deletes output files older than a retention period (e.g., 14 days). Use the Get files in folder action filtered by creation date, then Delete file for anything outside the window.
Run scheduling and concurrency: If this flow is scheduled to run twice daily, ensure it can't overlap with itself. Use a lock file mechanism — write a scraping.lock file at the start, delete it at the end. If the lock file already exists when the flow starts, it means a previous run is still in progress; exit gracefully rather than starting a second concurrent extraction.
Monitoring and alerting: For production unattended flows, emit structured log entries at key milestones (pages extracted, row counts, errors encountered) and write them to a log file or database table. This gives you visibility into whether the flow is running correctly without having to watch it manually. Pair this with a cloud flow trigger that sends an email or Teams notification if the output file isn't created within the expected window. The monitoring patterns for this are covered in detail in the monitoring and troubleshooting desktop flow runs at scale guide.
Machine and credential hygiene: Ensure the machine running the flow has a stable, dedicated account with the minimum permissions required — browser access, write access to the output folder, and nothing more. Don't run scraping flows under a shared admin account.
You now have a complete, production-ready architecture for multi-page web scraping in Power Automate Desktop. The key principles that make this work reliably at scale:
aria-label, data-testid, and text content over positional CSS or XPath.From here, there are several natural directions to take this further. If your scraping targets require JavaScript interaction beyond simple clicking — dropdown selections, date range filters, checkbox toggles — those interaction patterns build directly on the foundation we've established here. If you're scraping from virtual environments or Citrix-hosted applications, the selectors and wait strategies need adaptation; the Internet Explorer and Citrix hosted applications article covers those specialized contexts.
For flows that need to scrape multiple different portals with different structures, refactor the portal-specific extraction logic into dedicated subflows with standardized input/output contracts. This gives you a scraping framework rather than a one-off flow — each new portal is a new subflow, and the orchestration loop stays stable.
Finally, if you need this extraction to run on demand in response to a business trigger — a new row in SharePoint, a form submission, a scheduled cloud flow — the connection between your desktop scraping flow and Power Automate cloud flows is straightforward to configure and opens up a much richer set of automation scenarios.
Power Automate Desktop & RPA
Automating Mainframe Terminal Sessions in Power Automate Desktop: Connecting via TN3270 and TN5250 Emulators, Navigating Green Screen Menus, and Extracting Structured Data for Modern System Integration
Implementing Dynamic Selector Repair and Fallback Strategies in Power Automate Desktop: Detecting Broken UI Elements at Runtime and Switching to Alternative Identification Methods Without Flow Failure