Building a Resilient Unattended RPA Orchestration Framework in Power Automate Desktop: Queue-Driven Job Dispatch, Machine Load Balancing, and Automated Recovery for 24/7 Production Bot Fleets
Most RPA deployments fail not because the bots are bad, but because the orchestration layer wasn't designed for production. This expert lesson walks you through building a complete queue-driven, self-healing bot fleet using Power Automate Work Queues, machine groups, and layered monitoring cloud flows. You'll leave with a blueprint for 24/7 automation that recovers from failures automatically and scales without babysitting.
🔥 Expert30 min readSep 22, 2026Updated Sep 22, 2026
Building a Resilient Unattended RPA Orchestration Framework in Power Automate Desktop: Queue-Driven Job Dispatch, Machine Load Balancing, and Automated Recovery for 24/7 Production Bot Fleets
Introduction
Picture this: it's 2:47 AM on a Tuesday, and your finance team's month-end close automation should have processed 4,200 invoice records overnight. Instead, three of your five bots silently failed at midnight when a legacy ERP system threw an unexpected session timeout. The fourth bot is running, but it's been stuck on the same record for 90 minutes because a PDF renderer crashed. The fifth bot finished its queue slice hours ago and has been idling ever since. By the time your team arrives at 8 AM, you have six hours of catch-up work, a furious CFO, and no audit trail explaining what happened.
This scenario isn't hypothetical — it's the standard failure mode of RPA deployments that were designed for happy-path execution and bolted-on error handling as an afterthought. True 24/7 production automation requires a fundamentally different architecture: one where the orchestration layer is as carefully engineered as the bots themselves. In this lesson, we're going to build that architecture using Power Automate's native capabilities — work queues, machine groups, cloud flow orchestrators, and structured recovery flows — combined with some disciplined design patterns that separate the amateurs from the engineers.
By the end of this lesson, you'll have a complete blueprint for a production-grade unattended RPA fleet that can scale horizontally, recover from failures automatically, rebalance work across available machines, and give your operations team full visibility without requiring anyone to babysit it at 3 AM.
What you'll learn:
How to design a queue-driven job dispatch system using Power Automate Work Queues as the single source of truth for bot work
How to configure machine groups and implement intelligent load balancing strategies that maximize bot utilization
How to build multi-layer error handling with automated recovery, including retry logic, poison message handling, and escalation paths
How to architect a monitoring and alerting cloud flow that detects fleet health degradation and self-heals where possible
How to structure your desktop flows, subflows, and cloud flow orchestrators so the entire system is maintainable and testable
Prerequisites
This is an expert-level lesson. You should already be comfortable with:
You'll also need: Power Automate Premium licenses for your bot machines, an environment with Dataverse enabled (required for Work Queues), and at least two machines registered in a machine group.
The Architecture Before You Write a Single Action
The most expensive mistake in enterprise RPA is building the automation first and the orchestration second. You end up retrofitting queue logic into flows that were designed assuming sequential, single-machine execution. So let's start with architecture.
A production-grade unattended fleet has four distinct layers:
Layer 1: Work Queue (the source of truth)
All work items — invoice records, file paths, transaction IDs, whatever your bots process — live in a Work Queue. Nothing gets processed that isn't in the queue. Nothing gets marked complete unless the queue item is updated. The queue is the contract between the business process and the bot fleet.
Layer 2: Dispatcher (the cloud flow orchestrator)
A scheduled cloud flow polls the queue, checks machine group availability, and fires desktop flow runs. It doesn't do any business logic itself — its only job is to match available work to available machines intelligently.
Layer 3: Executor (the desktop flow)
The desktop flow picks up a single queue item, does the actual work (interacts with applications, processes data, writes outputs), and reports success or failure back to the queue. It's stateless — it shouldn't care which machine it runs on or what the previous item was.
Layer 4: Monitor (the health watchdog)
A separate cloud flow runs on a short interval (every 5-15 minutes), checks for stalled runs, orphaned queue items, machine offline events, and overall fleet health. It triggers recovery actions when it detects problems, and it sends alerts when human intervention is genuinely required.
This separation of concerns is what makes the system resilient. If your dispatcher has a bug, your executors keep finishing in-flight work. If a machine goes offline, the monitor detects it and the dispatcher routes new work elsewhere. Each layer can be tested and debugged independently.
Key insight
The Work Queue is not just a convenience — it's the state machine for your entire operation. Every queue item should have a status that accurately reflects reality at all times: Queued, Processing, Processed, Exception, or OnHold. If a bot crashes mid-item, that item must automatically return to Queued or move to Exception — never stay in Processing forever. Designing your queue transitions correctly is more important than any single piece of bot logic.
Setting Up Work Queues as Your Operational Backbone
Power Automate Work Queues live in Dataverse and are accessible through both cloud flows and, via HTTP actions, from desktop flows. Let's set up the queue structure properly.
Navigate to make.powerautomate.com, open your target environment, and find Monitor > Work queues in the left navigation. Create a new queue called InvoiceProcessingQueue. Give it a meaningful description — your operations team will thank you at 11 PM six months from now.
The key configuration decisions at queue creation:
Priority handling: Enable priority support if your items have different urgency. High-priority items (emergency reprocessing requests, end-of-day cutoff items) should be able to jump ahead of bulk batch items. Set three priority levels: 1 (High), 2 (Normal), 3 (Low).
Expiry behavior: Set a sensible expiry for queue items. An invoice that was urgent at 6 AM is probably not worth processing at 8 PM the same day. Expired items move to a reportable state without clogging your active queue.
Callback URL: Leave this blank for now — we'll handle status callbacks through the cloud flow layer rather than Dataverse triggers.
Structuring Queue Item Payloads
Each queue item's value (a JSON string) should contain everything the executor needs — no database lookups, no config file reads during execution if you can avoid it. Here's a well-structured payload for an invoice processing scenario:
Notice retryCount and lastErrorMessage in the payload. You're going to update these fields when an item fails and gets re-queued. This gives your monitor flow and your operations team a running history without needing a separate logging table for basic retry tracking.
Loading Items into the Queue
Your upstream data pipeline — whatever creates the work — should populate the queue via a cloud flow using the Add item to work queue action. This is typically triggered by a schedule, a SharePoint file arrival, a Dataverse record creation, or a manual trigger for ad-hoc processing.
Trigger: Recurrence (Daily at 6:00 AM)
→ List rows from SharePoint (Invoices_Pending table)
→ Apply to each (invoice row):
→ Compose: Build JSON payload from row columns
→ Add item to work queue:
Queue: InvoiceProcessingQueue
Name: invoice['InvoiceNumber']
Value: outputs('Compose')
Priority: if(equals(invoice['IsUrgent'], true), 1, 2)
Expiry: addHours(utcNow(), 12)
Warning
Never add duplicate items to a work queue without a deduplication check. If your upstream trigger fires twice (Power Automate's at-least-once delivery guarantee means this can happen), you'll process the same invoice twice. Before adding, query the queue for existing items with the same itemId using the List work queue items action with a filter like _workqueueid_value eq 'your-queue-id' and workqueueitemkey eq 'INV-2024-089341' and statecode eq 0. Only add if the result is empty.
Building the Dispatcher: Intelligent Queue-to-Machine Routing
The dispatcher is a cloud flow that runs on a schedule — typically every 2-5 minutes for a busy queue, or every 10-15 minutes for lower-volume operations. Its job is to check how many queue items are waiting and how many machines are available, then fire the right number of desktop flow runs.
Here's the dispatcher logic in detail:
Trigger: Recurrence (Every 3 minutes)
Step 1: Get queue depth
→ List work queue items
Queue: InvoiceProcessingQueue
Status: Queued
→ Set variable: QueuedItemCount = length(body('List_work_queue_items')?['value'])
Step 2: Check if work exists
→ Condition: QueuedItemCount greater than 0
→ If No: Terminate (nothing to do)
Step 3: Get machine group status
→ HTTP GET to Power Automate API:
URI: https://api.powerautomate.com/providers/Microsoft.ProcessSimple/
environments/{envId}/machineGroups/{groupId}
→ Parse machine group response
→ Set variable: AvailableMachines = count of machines with status 'Available'
Step 4: Calculate dispatch count
→ Set variable: MaxConcurrentRuns = 5 (your license limit)
→ Get currently running desktop flows via HTTP
→ Set variable: ActiveRuns = count of in-progress runs
→ Set variable: DispatchCount = min(
QueuedItemCount,
sub(MaxConcurrentRuns, ActiveRuns),
AvailableMachines
)
Step 5: Fire runs
→ Apply to each (range(0, DispatchCount)):
→ Run desktop flow (unattended):
Desktop flow: InvoiceProcessor_Main
Machine group: InvoiceProcessingMachineGroup
Run mode: Unattended
(No input parameters — bot will self-serve from queue)
→ Delay: 10 seconds (stagger starts to avoid simultaneous app launches)
The stagger delay in Step 5 is something most documentation doesn't mention. When five bots all start simultaneously and all try to launch SAP or a browser at the same second, you get race conditions on shared resources — license server connections, shared network drives, application initialization locks. A 10-second stagger adds only 40 seconds to your overall dispatch time for 5 bots, but it dramatically reduces startup failures.
Tip
The Power Automate API for checking machine group status requires an Azure AD OAuth token. Use a service principal with the Flow Machine role to authenticate. Store the client secret in Azure Key Vault and retrieve it via the secure credential pattern rather than hardcoding it in your cloud flow. This is not optional in production — hardcoded secrets in cloud flow definitions are visible to anyone with edit access to the flow.
Why Bots Self-Serve from the Queue (Not Dispatcher-Assigned)
You might be wondering why the dispatcher fires runs without assigning specific queue items to specific bots. This is a deliberate design choice.
If the dispatcher pre-assigns items ("Bot 1 gets items 1-100, Bot 2 gets items 101-200"), you introduce coupling problems: if Bot 1 runs faster than expected, it finishes and idles while Bot 2 is still working. If Bot 1 crashes halfway through, you have to figure out which of "its" items were processed.
Instead, each bot run opens its own Dataverse session and calls Dequeue item from work queue at runtime. This atomically marks one item as Processing and returns it to the bot. When the bot finishes, it marks the item Processed. This is a pull model, and it naturally implements work-stealing load balancing: fast machines process more items, slow machines process fewer, and there's no coordinator overhead.
The Executor Desktop Flow: Stateless, Resilient, Verbose
The executor desktop flow is where your actual business logic lives. Its structure should follow a strict pattern regardless of what application it's automating.
Main Flow Structure
Main:
→ Call Subflow: Initialize_Environment
→ Loop: While True
→ Call Subflow: Dequeue_Item
Output: CurrentItem (JSON), QueueItemId
→ If CurrentItem is empty:
Exit loop (no more work)
→ On Block Error (label: ItemProcessingBlock):
→ Call Subflow: Handle_Item_Error
→ Continue next iteration
→ Call Subflow: Parse_Item_Payload
Input: CurrentItem
Outputs: InvoiceNumber, VendorId, Amount, DocumentPath, RetryCount, ...
→ Call Subflow: Process_Invoice
Inputs: all parsed fields
Output: ProcessingResult
→ Call Subflow: Complete_Queue_Item
Inputs: QueueItemId, ProcessingResult
→ Call Subflow: Cleanup_Environment
Let's look at each critical subflow.
Subflow: Initialize_Environment
This runs once before the loop. It launches your target applications, authenticates, and verifies the environment is healthy before the bot touches any queue items.
Initialize_Environment:
→ Set variable: MaxItemsPerRun = 50 (prevent infinite loops)
→ Set variable: ItemsProcessed = 0
→ Launch SAP GUI application
→ Wait for SAP login screen (timeout: 30 seconds)
→ On Block Error (label: SAPLaunchError):
→ Log to custom Dataverse table: "SAP failed to launch on [MachineName]"
→ Send HTTP POST to monitoring webhook
→ Stop flow with error: "Environment initialization failed"
→ Call Subflow: SAP_Login
Inputs: Username (from Key Vault), Password (from Key Vault)
→ Verify SAP home screen is visible
→ Set variable: EnvironmentReady = True
The MaxItemsPerRun counter is important. Without it, a bot will process queue items forever in a single run, which means no other dispatcher-triggered run can start on that machine (the machine is occupied). In practice, set this to a number that represents 45-55 minutes of work per bot run. This way, the dispatcher's 3-minute polling cycle can start fresh runs on machines as they free up, giving you natural load redistribution opportunities.
Subflow: Dequeue_Item
This is where the bot atomically claims its next item of work:
Dequeue_Item:
→ If ItemsProcessed >= MaxItemsPerRun:
→ Set output: CurrentItem = ""
→ Return (signal the main loop to exit cleanly)
→ HTTP POST to Dataverse:
URI: /api/data/v9.2/workqueues({queueId})/Microsoft.Dynamics.CRM.DequeueWorkQueueItem
Body: {"worksessionid": "%MachineName%_%Timestamp%"}
→ Parse response:
→ If response is empty or status 204: Set CurrentItem = "" (queue empty)
→ If response contains item:
Set CurrentItem = response['value']
Set QueueItemId = response['workqueueitemid']
→ Increment ItemsProcessed by 1
The worksessionid is a custom string that identifies which machine/run claimed this item. This is invaluable for debugging — when you see an item stuck in Processing, your logs immediately tell you which machine had it.
Subflow: Handle_Item_Error
This is the most important subflow in the entire system, and most RPA developers don't build it robustly enough.
Handle_Item_Error:
→ Set variable: ErrorMessage = LastActionError
→ Set variable: ErrorDetails = GetLastError() (PAD system variable)
→ Parse CurrentItem JSON → Get RetryCount
→ Set variable: NewRetryCount = RetryCount + 1
→ If NewRetryCount < 3:
→ Update queue item via HTTP PATCH:
Status: Queued (re-queue for retry)
Priority: 1 (bump priority so it retries soon)
Value: UpdatedJSON with NewRetryCount and ErrorMessage
→ Log: "Item [InvoiceNumber] re-queued for retry [NewRetryCount] of 3"
→ Else:
→ Update queue item via HTTP PATCH:
Status: Exception
ExceptionMessage: ErrorMessage
Value: UpdatedJSON with final error details
→ HTTP POST to Teams webhook:
Message: "POISON ITEM: [InvoiceNumber] failed 3 times.
Last error: [ErrorMessage].
Manual review required."
→ Log to audit table: full error details + screenshot path
→ Take screenshot: Save to "\\fileserver01\bot_screenshots\[ItemId]_[Timestamp].png"
→ Continue (don't terminate the bot — process next item)
Key insight
The critical distinction here is between transient failures (network blip, SAP session expired, PDF locked by another process) and persistent failures (bad data in the queue item, target application fundamentally broken, logic bug). Your retry logic handles transient failures. Your poison item handling routes persistent failures to human attention. The 3-retry threshold is a starting point — tune it based on your actual failure patterns. Some processes warrant 5 retries; others should go straight to exception on first failure if the error type indicates bad data.
Machine Load Balancing: Beyond Round-Robin
Power Automate's default machine group behavior is approximately round-robin: the service routes new runs to the machine that has been waiting longest. This is fine for simple scenarios but breaks down in production for several reasons:
Machines have different performance profiles (some are on faster hardware, some have more RAM)
Some bot runs are heavier than others
Machines can be degraded — running slowly due to other processes — without being "offline"
Here's how to implement smarter load balancing at the orchestration level.
Machine Performance Scoring
Create a Dataverse table called BotMachineMetrics with columns: MachineName, LastRunDurationSeconds, AverageRunDuration7Day, SuccessRate7Day, LastHealthCheck, CurrentStatus.
In your desktop flow, at the start of each run, log the machine name and start time. At the end, log the duration and outcome. Your dispatcher cloud flow reads this table to calculate a performance score:
Machines with higher scores get priority in dispatch decisions. You implement this by using the Run desktop flow action's machine targeting — instead of targeting the entire machine group, you target specific machines in priority order.
Warning
Don't over-engineer the performance scoring to the point where your dispatcher becomes brittle. A score that leans too heavily on recent run duration can cause "thrashing" — the dispatcher constantly routes to one machine until it slows down, then swings to another. Use a rolling average over at least 7 days to smooth out noise, and cap the routing differential: your best machine shouldn't get more than 2x the work of your worst machine unless the performance gap is dramatic.
Graceful Machine Rotation for Maintenance
In a 24/7 fleet, you need to patch and restart machines without stopping automation. Build a MaintenanceSchedule Dataverse table with entries like: MachineName, MaintenanceStartUTC, MaintenanceEndUTC, MaintenanceType (Patch/Restart/Manual).
Your dispatcher checks this table before routing to any machine:
For each machine in machine group:
→ Query MaintenanceSchedule where MachineName = machine
AND MaintenanceStartUTC < utcNow()
AND MaintenanceEndUTC > utcNow()
→ If record exists: Skip this machine
→ Else: Include in eligible machines list
When you need to patch BOTMACHINE03, you create a maintenance window entry, and within one dispatcher polling cycle, no new work is routed to that machine. In-flight work completes normally (the desktop flow finishes its current item), and the machine goes idle naturally. No abrupt terminations, no lost queue items.
The Monitoring Flow: Your Automated Site Reliability Engineer
The monitoring flow is the piece most teams skip, and it's why they end up with operations teams whose entire job is watching a dashboard and manually triggering reruns. Build it correctly and your bots effectively run themselves.
Health Check Schedule
Run the monitoring flow every 5 minutes. At this frequency, you catch problems within one processing cycle rather than discovering them when a business deadline has already passed.
Monitor_Main (Recurrence: Every 5 minutes):
Check 1: Stalled Processing Items
→ List work queue items where Status = Processing
AND StatusChangedOn < addMinutes(utcNow(), -30)
→ For each stalled item:
→ Get MachineName from worksessionid
→ Check if machine has an active desktop flow run via HTTP
→ If no active run found:
→ Reset item to Queued status (the bot died without cleanup)
→ Log: "Orphaned item recovered: [ItemId] from [MachineName]"
→ If active run exists but runtime > 90 minutes:
→ Flag for human review (bot may be in infinite loop)
→ Send alert: "Bot run exceeded 90min SLA on [MachineName]"
Check 2: Queue Depth Alert
→ Get count of Queued items
→ If count > ThresholdForAlert (e.g., 200):
→ Check how many bots are running
→ If all available machines are already running:
→ Alert operations: "Queue depth critical, need additional capacity"
→ If machines are idle (dispatcher may have failed):
→ Trigger dispatcher cloud flow immediately (don't wait for schedule)
Check 3: Exception Rate Monitoring
→ Count items moved to Exception in last 30 minutes
→ If exception rate > 10% of processing rate:
→ Alert: "High exception rate detected — possible systemic failure"
→ Check if exceptions share same error message pattern
→ If systemic (all same error):
→ Pause queue (set all running bots to drain current items)
→ Alert: "Queue paused — investigate [ErrorPattern] before resuming"
Check 4: Machine Offline Detection
→ For each machine in machine group:
→ If LastHeartbeat > 15 minutes ago:
→ Mark machine as degraded in BotMachineMetrics
→ Alert: "Machine [MachineName] may be offline"
→ Attempt to start machine via Azure VM API (if cloud-hosted)
Tip
The "Queue Pause on Systemic Failure" check (Check 3) is the automation equivalent of a circuit breaker. If your target application is down — SAP is having a P1 outage, for instance — you don't want 5 bots hammering retry attempts and filling your exception queue with noise. Pausing cleanly means all your queue items stay in Queued status, ready to resume when the outage resolves. Implement the pause by setting a global variable in your Dataverse environment settings table that your dispatcher reads at the start of each polling cycle.
Automated Recovery Actions
For the monitor to actually self-heal rather than just alert, it needs to take action. The most important recovery actions are:
Orphaned item recovery: When a bot machine crashes (power outage, Windows blue screen, PAD service crash), any Processing queue items are stuck. Your monitor's 30-minute stall check finds these and resets them to Queued, preventing permanent data loss. This is the most common overnight failure mode and it's fully automatable.
Bot restart on machine restart: Use the Azure VM API (if your bots run on Azure VMs) or a Wake-on-LAN flow (for on-premises) to restart machines that have gone offline. After restart, the machine automatically re-registers with the Power Automate service, and your dispatcher routes new work to it on the next polling cycle.
Dispatcher self-healing: Cloud flows can fail too. Add a separate ultra-lightweight "watchdog" flow that runs every 15 minutes and simply checks whether the dispatcher flow ran successfully in the last 10 minutes. If not, it triggers the dispatcher directly. Yes, this is a watchdog watching a dispatcher that watches bots — the depth of monitoring is proportional to the business criticality of the automation.
Handling the Hard Cases: Poison Messages and Partial Processing
In any high-volume queue system, you will encounter items that no bot can successfully process. These are "poison messages" — they'll keep failing and consuming retry budget unless you handle them explicitly. We covered the basic poison item flow above, but let's go deeper.
Detecting Poison Patterns Early
Instead of waiting for an item to fail 3 times, build a pre-processing validation subflow that checks for known bad patterns before attempting application interaction:
Subflow: Validate_Item
→ Parse payload fields
→ Validate: InvoiceNumber matches pattern "^[0-9]{10}$"
→ Validate: Amount > 0 AND Amount < 10000000
→ Validate: DocumentPath is accessible (Check file exists action)
→ Validate: DueDate is valid date and not in past by more than 90 days
→ Validate: VendorId exists in vendor lookup table (cached in-memory)
→ If any validation fails:
→ Immediately set item to Exception (don't waste a retry)
→ Include specific validation failure message
→ Route to data quality correction queue (separate Dataverse queue)
→ Return: IsValid = False
→ Return: IsValid = True
The vendor lookup table "cached in-memory" deserves explanation. At the start of each bot run, in Initialize_Environment, you fetch the current vendor ID list from your ERP or Dataverse once, store it in a data table variable, and use it for all item validations in that run. This avoids 50 individual API calls during a 50-item processing run — a significant performance gain that also reduces load on your source system.
Partial Processing Recovery
Some items involve multi-step processes where failure mid-way leaves the target system in an inconsistent state. For example: your bot reads an invoice, creates the SAP posting document, sends an approval email, then marks the invoice as submitted. If it crashes after creating the SAP document but before sending the email, retrying the whole item will create a duplicate SAP document.
The solution is checkpoint-based processing with idempotency checks:
Subflow: Process_Invoice (with checkpoints):
→ Check SAP for existing document: InvoiceNumber + "_POSTED"
→ If SAP document already exists:
→ Set checkpoint: SAP_COMPLETE = True
→ Log: "SAP step already complete, resuming from email step"
→ Else:
→ Create SAP posting (your actual SAP automation)
→ Update queue item value: Add "checkpoints": {"SAP_COMPLETE": true}
→ Check email audit log: Was approval email sent for this invoice?
→ If not sent:
→ Send approval email
→ Update queue item value: Add checkpoint "EMAIL_SENT": true
→ Else:
→ Log: "Email step already complete, skipping"
→ Mark invoice as submitted in source system
→ Update queue item: Status = Processed
For your SAP interactions in this pattern, the key is that your SAP queries are read operations before write operations — you're always checking state before acting. This makes each step idempotent: running it twice produces the same result as running it once. See Automating SAP GUI Interactions with Power Automate Desktop for the specific SAP action patterns you'll use for the state-check queries.
Credentials, Security, and Compliance in a 24/7 Fleet
In unattended automation running at 3 AM, credential security is not an afterthought — it's a core design requirement. Your bots need to authenticate to target applications, and those credentials need to be rotated periodically without taking down the fleet.
Zero-Downtime Credential Rotation
Store all bot credentials in Azure Key Vault with versioning enabled. In your Initialize_Environment subflow, retrieve credentials fresh at the start of each run (not cached from the previous run):
Subflow: SAP_Login:
→ HTTP GET: Azure Key Vault secret "bot-sap-username"
→ HTTP GET: Azure Key Vault secret "bot-sap-password"
→ Type username into SAP login field (using sensitive variable)
→ Type password into SAP password field (using sensitive variable)
→ Clear sensitive variables from memory after login
When your security team rotates the SAP password, they update the Key Vault secret. The next time any bot starts a new run, it gets the new credential automatically. No redeployment, no bot downtime, no coordination required.
Warning
Never log sensitive variables, even to your audit table. Power Automate Desktop's screenshot feature captures screen content — if your bot takes a recovery screenshot while the SAP password is visible on screen (e.g., in a "wrong password" error dialog that echoes the input), that screenshot is now in your audit trail. Build your error handling to clear sensitive variables before taking recovery screenshots, and configure SAP to mask password fields. This is the kind of edge case that creates compliance findings.
Monitoring and Troubleshooting Desktop Flow Runs at Scale
Even with excellent automated recovery, your operations team needs visibility into fleet status. The monitoring infrastructure you've built generates data — now you need to surface it.
Operations Dashboard Structure
Build a Power BI dashboard (or a Canvas App if you want in-platform) that surfaces:
Real-time queue metrics: Items queued, processing, processed, and in exception — updated every 5 minutes from Dataverse
Machine fleet status: Each machine's current status, last heartbeat, today's success rate, average processing time
Throughput trend: Items processed per hour over the last 24 hours (lets you spot performance degradation before it becomes a crisis)
Exception drill-down: All exception items with their error messages, sortable by frequency (lets you spot systemic issues at a glance)
SLA tracking: What percentage of items were processed within their expiry window
For the operational alerting, integrate your monitor flow with Teams and email. But be deliberate about alert fatigue — route truly urgent alerts (systemic failure, queue pause, machine offline) to Teams channels watched by on-call staff. Route informational alerts (single item exceptions, single machine warning) to an email digest that gets reviewed each morning.
This exercise has you build a minimal but complete version of the orchestration framework using a simulated data processing scenario.
Scenario: You have a set of product records in a SharePoint list that need to be validated against a web catalog and written to an Excel tracking file. You'll build the full stack: queue loading, dispatcher, executor, and basic monitoring.
Step 1: Set up the Work Queue
In your test environment, create a Work Queue named ProductValidationQueue. Enable priority support. Set expiry to 4 hours.
Step 2: Build the Queue Loader Cloud Flow
Create a scheduled cloud flow (every hour). It should:
List items from a SharePoint list Products_Pending filtered to Status eq 'New'
For each item, check the queue for an existing item with the same ProductID as the name
If not already queued, add to work queue with payload: {"productId": item ID, "sku": item SKU, "catalogUrl": constructed URL, "retryCount": 0}
Update the SharePoint item's Status to "Queued"
Step 3: Build the Executor Desktop Flow
The desktop flow should:
Have a Dequeue_Item subflow using HTTP POST to the Dataverse Work Queue dequeue endpoint
Have a Process_Product subflow that uses browser automation to navigate to the catalog URL and extract the product name, price, and availability (if you have a real catalog to test against, use it; otherwise simulate with a lookup table)
Write results to an Excel file using Excel automation — append a new row with product ID, extracted data, and timestamp
Mark each item as Processed or Exception appropriately
Loop until the queue is empty or 20 items are processed, whichever comes first
Step 4: Build the Simple Dispatcher
A cloud flow on a 5-minute recurrence that:
Checks queue depth (items in Queued status)
If > 0, fires one desktop flow run (unattended on your registered machine)
Includes a condition: don't fire if an existing run is already active (use the Power Automate API to check)
Step 5: Add Basic Monitoring
A cloud flow on a 10-minute recurrence that:
Lists all Processing items older than 20 minutes
For each: logs to a SharePoint list BotAuditLog with the item details and a "Stalled" flag
Sends you an email if any stalled items are found
Validation: Load 10 items into the SharePoint list. Let the system run for 30 minutes. Verify: all items reach Processed or Exception status, the Excel file has correct data for each processed item, and your audit log shows the run activity.
Common Mistakes & Troubleshooting
Mistake: Bot marks item Processing but crashes before completing — item is stuck forever
This is why the stall-detection monitor is non-negotiable. The immediate fix for a stuck item is to PATCH the item status back to Queued via Dataverse API. The permanent fix is the monitor flow's orphan recovery logic. If you're seeing this frequently, check whether your desktop flow has a cleanup path that runs even on crash — use the On Error action at the Main flow level (not just subflows) to attempt a queue item reset.
Mistake: Dispatcher fires too many concurrent runs, overwhelming target application
Symptoms: high error rate from application login failures, SAP license exhaustion errors, database connection pool errors. Fix: reduce MaxConcurrentRuns in your dispatcher, and add the stagger delay between run launches. Also check whether your target application has explicit concurrency limits (SAP often licenses by concurrent sessions — a count of 5 bots all needing separate sessions can be a licensing issue).
Mistake: Queue items expire because dispatcher isn't running
If your dispatcher cloud flow itself has an error and stops running, no new work gets dispatched. Your items sit in Queued status until they hit their expiry time and are automatically moved to a terminal state. Fix: implement the watchdog flow that monitors the dispatcher. Also, set your cloud flow's error notification settings to alert immediately on failure (not the default which often requires multiple failures). Review run history and debugging techniques to diagnose cloud flow failures quickly.
Mistake: Retry logic re-queues items at normal priority, causing a retry storm
When 50 items fail simultaneously (systemic issue) and all get re-queued at high priority, your next dispatch cycle tries to process 50 high-priority items that will probably fail again for the same reason. This burns through your retry budget rapidly. Fix: when re-queuing for retry, add an exponential backoff — set the ScheduledFor property on the queue item to addMinutes(utcNow(), pow(2, retryCount) * 5). Retry 1 waits 5 minutes, retry 2 waits 10 minutes, retry 3 waits 20 minutes.
Mistake: Desktop flow uses hardcoded delays instead of wait-for-condition logic
In web automation and application automation, Wait 3 seconds works on the developer's machine and breaks in production when the application is slower under load. Replace every Wait action with a conditional wait: "Wait until element exists with timeout 30 seconds." Your processing time becomes resilient to application performance variation, and you stop getting false-positive failures caused by timing.
Mistake: All error handling sends the bot to Exception state on first failure
If your On Block Error in the item processing block always marks the item as Exception (not Queued for retry), you lose all the benefits of the retry system. Check every error handler: is it distinguishing between transient and permanent failures? Timeout errors, network errors, and "element not found" errors (often transient) should retry. "Invalid data" and "business rule violation" errors (permanent) should go straight to Exception.
Summary & Next Steps
You've built a production-grade orchestration framework that treats 24/7 reliability as a first-class design requirement rather than an afterthought. Let's recap the key architectural decisions and why they matter:
Queue-driven dispatch means your work items have independent lifecycle management from your bot runs. Items survive bot crashes, machine restarts, and dispatcher failures. The queue is the single source of truth.
Self-service dequeue (pull model) means load balancing is automatic and emergent — fast machines process more, slow or failed machines process less, without any centralized coordinator making routing decisions.
Layered error handling — item-level retry, poison item routing, and circuit-breaker patterns — means that individual failures are contained, systemic failures are detected quickly, and the system degrades gracefully rather than catastrophically.
Monitor-first operations means your ops team is reading summaries and approving escalations rather than watching dashboards and manually triggering reruns.
Where to go from here:
The natural extension of this framework is adding dynamic scaling — automatically spinning up additional Azure VMs when queue depth exceeds a threshold and shutting them down when the queue clears. This requires Azure automation runbooks integrated with your monitoring cloud flow, and it's the difference between "we have 5 bots" and "we have the compute we need right now."
You should also explore how parallel branching and concurrency control in your cloud flow layer can accelerate the dispatcher's queue-loading phase when you're bulk-loading thousands of items at schedule time.
Finally, consider deploying this framework across environments using proper ALM practices — your dispatcher, executor, and monitor flows should all be solution-aware and deployable via pipelines so promoting from dev to test to production is a one-click operation, not a manual rebuild.
The bots are awake. The queue is moving. Your job now is to make sure it keeps moving — and this framework gives you the foundation to do exactly that.
