Designing Idempotent Flows: Preventing Duplicate Processing Under At-Least-Once Delivery

Introduction

Picture this: your order management flow has been running flawlessly for three months. Then one Thursday afternoon, a transient network blip causes a connector timeout. Power Automate retries the failed action — as it should — but the downstream ERP system received the original request just fine. Now you've got a duplicate sales order sitting in the system, and your operations team is spending their Friday untangling it. Nobody is happy.

This is the at-least-once delivery problem, and it's not a Power Automate bug — it's a fundamental property of distributed systems. Any durable messaging or retry system will occasionally deliver the same message more than once. The guarantee you get is that your message will be processed, not that it will be processed exactly once. Your job as a flow designer is to make your automation safe to run multiple times with the same input and still produce the same result. That property has a name: idempotency.

By the end of this lesson, you'll know how to architect flows that can safely handle retries, duplicate triggers, and redelivered messages without creating ghost records, doubled invoices, or mismatched state. We'll build a realistic order processing scenario that demonstrates each technique, and you'll walk away with patterns you can drop into your next production integration.

What you'll learn:

• Why at-least-once delivery is unavoidable and what triggers duplicate execution in Power Automate
• How to design an idempotency key strategy using Dataverse, SharePoint lists, or Azure Table Storage as a deduplication store
• How to implement "check-then-act" patterns with proper concurrency guards
• How to make downstream HTTP calls idempotent using conditional headers and correlation IDs
• How to test your idempotency logic before it gets tested for you in production

---

Prerequisites

You should be comfortable building multi-step cloud flows, working with expressions, and handling errors. Specifically, you'll get the most from this lesson if you've already worked through Master Error Handling and Retry Patterns in Power Automate for Bulletproof Flows and have some experience with Dataverse or HTTP connectors. Familiarity with Implementing Parallel Branching and Concurrency Control in Power Automate to Maximize Throughput and Prevent Race Conditions will also be useful when we discuss locking.

---

Why At-Least-Once Delivery Is Unavoidable

Before we build anything, let's be precise about why this happens. Power Automate's connectors communicate over HTTP. When your flow calls an action — say, creating a record in an ERP via a custom connector — one of three things can happen from the flow engine's perspective:

1. The call succeeds and returns a 2xx response. All good.
2. The call fails with a clear 4xx error. No retry, the action is marked failed.
3. The call times out, or returns a 5xx, or the network drops. The engine doesn't know if the downstream system received and processed the request.

In scenario 3, Power Automate's built-in retry policy kicks in. It re-sends the request. This is the right behavior — you'd rather have a duplicate than lose a transaction — but it means your downstream systems need to be ready for it.

The same logic applies when Power Automate itself is the consumer. If you're processing messages from a queue — like Azure Service Bus — the broker holds the message until your flow completes successfully and calls "Complete." If the flow crashes mid-execution after processing but before completing, the message becomes available again for redelivery.

There are also less obvious sources of duplication beyond retries:

• A user double-clicking a button that triggers an instant flow
• Two near-simultaneous Dataverse webhook deliveries for the same record update (Power Platform webhooks have at-least-once semantics)
• A scheduled flow that overlaps with a manual run of the same flow
• Reprocessing during incident recovery — someone re-queues events to replay a failed batch

Each of these can be solved with the same family of patterns. Let's build them.

---

The Idempotency Key: Your Foundation

An idempotency key is a stable, unique identifier for a logical operation — not just a record. The core rule is: if you've seen this key before and processed it successfully, skip the work and return the same result.

Choosing the right key depends on your trigger:

Trigger type	Good idempotency key
Service Bus message	Message MessageId property
Dataverse record created/updated	Row GUID + version number or modifiedon timestamp
Webhook from external system	Correlation ID or event ID from the payload
Scheduled batch	YYYY-MM-DD-HH + batch name
Button/instant flow	Client-provided GUID passed as an input parameter

For our running example, we'll process incoming purchase orders from an external supplier portal. Each order payload contains an orderId (e.g., "ORD-2024-88172") and a supplierId. The combination of these two fields — or ideally a dedicated correlationId field the supplier provides — is our idempotency key.

Computing a Stable Key

When the upstream system provides a correlation ID, use it directly. When it doesn't, derive one deterministically from the payload. In Power Automate expressions, you can construct a compound key like this:

concat(triggerBody()?['supplierId'], '-', triggerBody()?['orderId'])

Store this in a Compose action early in your flow and reference it everywhere as outputs('Compose_IdempotencyKey'). That single Compose action becomes the canonical key for the entire flow run.

If your incoming data is less structured — say, a file drop or a form submission — hash the relevant stable fields using a Compose action with a SHA-256 expression isn't natively available, but you can call a lightweight Azure Function or use the guid() function seeded from a stable string via a workaround. The simplest production approach is to require upstream systems to provide explicit correlation IDs; put that in your integration design contract.

---

Building a Deduplication Store

You need somewhere to record that a given idempotency key has been processed. The choice of store matters: it needs to support atomic check-and-write operations (or at least give you a way to detect conflicts), it needs to be durable, and it needs to be queryable quickly.

Three solid options in the Power Platform ecosystem:

Dataverse (recommended for most scenarios): Create a custom table called Flow Processing Log with a column cr_idempotency_key marked as an alternate key. Alternate keys in Dataverse enforce uniqueness at the database level, which is exactly what you want. You can then use "upsert" semantics — try to create a row; if the alternate key constraint fires, you know it's a duplicate.

SharePoint List: Easier to set up, but slower and lacking true atomic upsert. Use with caution in high-throughput scenarios, and add a "Processing Status" column to track in-progress vs. completed.

Azure Table Storage: Excellent for high-volume, low-latency deduplication. Each row has a PartitionKey + RowKey pair that must be unique; an insert that violates this returns a 409 Conflict, which you can catch and handle. Access it via the HTTP connector with proper authentication.

For this lesson, we'll use Dataverse because it integrates cleanly with Power Automate's native connector and supports alternate keys.

Setting Up the Dataverse Table

In your solution, create a new table named Flow Processing Log with these columns:

• Idempotency Key (Text, required, marked as Alternate Key) — cr_idempotencykey
• Flow Name (Text) — cr_flowname
• Status (Choice: Pending, Completed, Failed) — cr_status
• Trigger Payload Hash (Text) — cr_payloadhash (optional but useful for debugging)
• Completed On (Date/Time) — cr_completedon
• Result Payload (Multiline Text) — cr_resultpayload (store the response if callers need idempotent responses)

---

The Check-Then-Act Pattern

With the store in place, here's the basic pattern:

1. Compute the idempotency key
2. Query the log table — does a record exist for this key with Status = Completed?
3. If yes: short-circuit the flow, optionally return the cached result
4. If no: proceed with processing, then write the Completed record

In Power Automate, implement this as follows:

Step 1 — Compose the key:

Compose action: Idempotency Key
Value: concat(triggerBody()?['supplierId'], '-', triggerBody()?['orderId'])

Step 2 — Query Dataverse:

List rows action (Dataverse)
Table: Flow Processing Logs
Filter rows: cr_idempotencykey eq 'outputs('Compose_IdempotencyKey')'
Top count: 1

Step 3 — Condition:

Condition: empty(outputs('List_rows')?['body/value']) is equal to false
  AND first(outputs('List_rows')?['body/value'])?['cr_status'] is equal to 'Completed'

If true (already processed): Add a "Terminate" action with Status = Succeeded and a message like "Duplicate request: already processed". Optionally use a Compose to surface the cached result payload.

If false (new request): Continue into your main processing logic.

---

Closing the Race Condition with a Pending Record

The race condition window — the gap between "check" and "write the completion record" — is where duplicates sneak through under concurrent load. The solution is to write a Pending record before you do any work, not after.

Here's the revised pattern:

1. Compute the idempotency key
2. Attempt to create a Pending record in the log table (using the alternate key)
3. If creation fails with a conflict (409), a concurrent instance already claimed this key — short-circuit
4. If creation succeeds, you now hold the "lock" — proceed with processing
5. On success, update the log record to Completed
6. On failure, update the log record to Failed (allowing retry decisions)

In Dataverse, implement step 2 as an "Add a new row" action targeting the Flow Processing Log table. Set cr_idempotencykey to your computed key and cr_status to Pending.

Configure the action's "Run after" settings to allow it to be the first real action after your key computation. Then wrap it in a try/catch using a parallel branch or scope:

Scope: Claim Processing Slot

• Action: Add a new row (Flow Processing Log) with status Pending
• Configure "Run after" on the next scope to handle failure:

Scope: Claim Processing Slot
  └─ Add a new row (Flow Processing Log)
       cr_idempotencykey: outputs('Compose_IdempotencyKey')
       cr_status: Pending
       cr_flowname: 'Purchase Order Processor'

Condition: result('Scope_Claim_Processing_Slot') is equal to 'Failed'
  If yes:
    Terminate (Succeeded) — "Duplicate: another instance is processing this key"
  If no:
    Continue to main processing logic

The Dataverse alternate key constraint ensures that if two flow instances both try to insert a Pending record for the same key, exactly one succeeds and the other gets a 400/412 error — which causes the Scope to fail, which triggers the short-circuit condition.

Handling the Failed State

A Pending record that never gets updated to Completed or Failed represents a flow that crashed mid-execution. You need a cleanup strategy. Two approaches:

Time-based expiry: A scheduled flow runs every 15 minutes, queries for Flow Processing Log records with status = Pending and createdon older than 10 minutes, and updates them to Failed. This frees up the slot for retry.

Retry-aware cleanup: When a new instance arrives for a key that has a Pending record older than your maximum expected processing time, it takes over — updates the record back to Pending with its own run ID, then proceeds.

For most flows, the time-based expiry approach is simpler and sufficient. For mission-critical flows, build the retry-aware version.

---

Making Downstream HTTP Calls Idempotent

Your flow can be perfectly idempotent internally, but if the downstream API isn't, you're just pushing the problem downstream. When calling external APIs via the HTTP connector, use two techniques:

Idempotency-Key Header

Many APIs — particularly in payments (Stripe, Adyen) and modern ERP systems — support an Idempotency-Key header. Send the same key on retry and the server returns the same response without re-executing the operation. In your HTTP action:

{
  "method": "POST",
  "uri": "https://erp.contoso.com/api/orders",
  "headers": {
    "Content-Type": "application/json",
    "Idempotency-Key": "@{outputs('Compose_IdempotencyKey')}",
    "X-Correlation-Id": "@{workflow()?['run']['name']}"
  },
  "body": {
    "orderId": "@{triggerBody()?['orderId']}",
    "supplierId": "@{triggerBody()?['supplierId']}",
    "lineItems": "@{triggerBody()?['lineItems']}"
  }
}

The X-Correlation-Id carries the flow run name, which appears in your run history and makes it easy to correlate a Power Automate run to a specific API call in downstream logs.

Conditional Updates (PUT vs. POST)

Prefer PUT over POST where the API supports it. PUT is semantically idempotent: PUT /orders/ORD-2024-88172 with the same body applied twice leaves the resource in the same state. POST to /orders creates a new resource each time. Design your integration contracts to use resource-oriented endpoints, and you get idempotency largely for free from REST semantics.

When you must use POST, add a query parameter or header that acts as a deduplication key and document the expected behavior in your connector definition. If you're building a custom connector, this is worth covering in depth — the custom connectors lesson walks through how to define headers and parameters that flow through to every operation.

---

Idempotent Dataverse Operations

When your flow writes to Dataverse, use upsert instead of create-or-update pairs. The "Update a row" Dataverse action with the alternate key set will update if the record exists and fail cleanly if it doesn't. The "Add a new row" action will create if the key is absent.

The cleanest approach: use Dataverse's native upsert capability via the Dataverse connector's "Update a row" action, selecting the alternate key as the identifier. This collapses your check-then-create-or-update logic into a single atomic operation.

For related records — like adding order line items after creating the header — use a transaction-like pattern: create all child records before updating the parent to "Confirmed" status. The parent status acts as a commit flag. If you see a parent in "Pending" state, child records may be incomplete. Your duplicate-detection logic checks the parent status, not the existence of child records.

If you're building incremental sync pipelines against Dataverse, the Dataverse change tracking lesson covers how to use version tokens and delta tokens to ensure you're never reprocessing the same change twice — a natural complement to the patterns here.

---

Structured Flow Architecture for Idempotent Processing

Here's what the complete, production-ready flow structure looks like. The nesting of Scopes is deliberate — it makes the control flow readable and gives you clean run-after configurations.

Trigger: When a message is received (Service Bus)

├─ Compose: IdempotencyKey
│    Value: concat(body('Get_message')?['supplierId'], '-', body('Get_message')?['orderId'])

├─ Scope: Claim Processing Slot
│    └─ Add a new row (Flow Processing Log)
│         cr_idempotencykey: @{outputs('Compose_IdempotencyKey')}
│         cr_status: Pending
│         cr_flowname: Purchase Order Processor

├─ Condition: Did_Claim_Fail
│    Check: result('Scope_Claim_Processing_Slot') == 'Failed'
│    Yes branch:
│    │  ├─ Complete message (Service Bus) ← don't re-queue, we're handling it elsewhere
│    │  └─ Terminate (Succeeded) "Duplicate: key already claimed"
│    No branch: (continue)

├─ Scope: Main Processing
│    ├─ HTTP: POST to ERP (with Idempotency-Key header)
│    ├─ Add a row: Order Tracking (Dataverse)
│    └─ Send notification (Teams/Email)

├─ Condition: Did_Main_Processing_Fail
│    Check: result('Scope_Main_Processing') == 'Failed'
│    Yes branch:
│    │  ├─ Update row (Flow Processing Log) → Status: Failed
│    │  ├─ Abandon message (Service Bus) ← requeue for retry
│    │  └─ Terminate (Failed) with error detail
│    No branch: (continue)

├─ Update row (Flow Processing Log)
│    Status: Completed
│    cr_completedon: @{utcNow()}
│    cr_resultpayload: @{outputs('HTTP_POST_to_ERP')?['body']}

└─ Complete message (Service Bus)

Notice the deliberate separation of Service Bus "Complete" vs. "Abandon" based on the outcome. When the claim fails (duplicate), you complete the message — it's been handled. When main processing fails, you abandon the message — let it be redelivered and retried. This is the correct interaction between your idempotency layer and the message broker.

---

Environment Promotion Considerations

Your Flow Processing Log table and its alternate key configuration need to be part of your solution. If you export the solution without the table or without the alternate key definition, the flow will deploy to production without its deduplication spine.

Make sure your solution includes:

• The Flow Processing Log table with all columns
• The alternate key definition on cr_idempotencykey
• Any environment-specific configuration (like the Service Bus connection reference) managed via environment variables

The ALM and environment promotion lesson covers how to package dependent tables and ensure they're deployed in the right order. Environment variables for things like the log table retention policy (how long to keep Completed records before archiving) belong in the solution as environment variables, not hardcoded in the flow logic.

---

Hands-On Exercise

Build a purchase order idempotency flow from scratch. Here's the scenario: your company receives purchase order payloads via a SharePoint list (a common low-infrastructure alternative to Service Bus for mid-volume scenarios). Each item has a Title field (the order ID), a SupplierID field, and a Payload JSON field.

Part 1 — Set up the deduplication table

In Dataverse, create the Flow Processing Log table as described earlier. Set up the alternate key on cr_idempotencykey. Test it manually by creating two rows with the same key value and confirming the second fails.

Part 2 — Build the flow

Create an automated flow triggered by "When an item is created" on your SharePoint list.

First Compose action:

Name: Idempotency Key
Value: concat(triggerBody()?['SupplierID'], '-', triggerBody()?['Title'])

Add the Scope for claiming the processing slot. Configure the failure condition. In the main processing scope, simulate your ERP call with an HTTP action pointing to https://httpbin.org/post — this echoes your request back and you can verify the idempotency key header is being sent correctly:

{
  "method": "POST",
  "uri": "https://httpbin.org/post",
  "headers": {
    "Idempotency-Key": "@{outputs('Compose_Idempotency_Key')}",
    "Content-Type": "application/json"
  },
  "body": "@{triggerBody()?['Payload']}"
}

Complete the processing log update on success and failure branches.

Part 3 — Test idempotency

Add two identical SharePoint items (same Title and SupplierID values) within a 30-second window. Watch the flow run history. You should see two runs — one with a Completed status in your log table, one that short-circuits with a "Duplicate" termination. Check the Flow Processing Log in Dataverse to confirm there's exactly one record per key.

Part 4 — Test the Pending race condition

Add a 2-minute delay (using a Delay action) inside your main processing scope between the Pending claim and the Completed update. While the first run is sleeping, manually add a duplicate SharePoint item. Observe the second run's behavior — it should see the Pending record, fail its claim attempt (if you've configured the alternate key correctly), and self-terminate.

---

Common Mistakes & Troubleshooting

"My alternate key isn't preventing duplicates." Verify the alternate key is actually published in Dataverse — navigate to your table's Keys section in the maker portal and confirm the key status shows as Active, not Pending. It takes a few minutes to activate after creation and won't enforce uniqueness until it does.

"The flow is short-circuiting on the first run, not the second." You're likely querying the log table before it's had time to index the new alternate key, or there's a caching issue in the Dataverse connector. Check the exact filter expression in your List rows action. The OData filter cr_idempotencykey eq 'YOUR_KEY' requires the column's logical name (not display name).

"Duplicate runs are both reaching main processing." Your claim scope is succeeding for both — meaning the alternate key isn't configured, or you're using the wrong column in the Add a new row action. Add logging to both runs by capturing workflow()?['run']['name'] in a Compose at the start of each scope and write it to the log record. This lets you see exactly which run wrote the Pending record.

"Abandoned Service Bus messages are being processed multiple times and the log isn't catching them." This is the correct behavior — abandoned messages are intended for retry. The issue is usually that your main processing scope is failing consistently (not transiently). Look at the failure reason in your monitoring telemetry. The monitoring and alerting setup covers how to surface this.

"Completed log records are filling up my Dataverse storage." Implement a retention flow: a scheduled flow that deletes Flow Processing Log records where cr_status eq 'Completed' and cr_completedon lt @{addDays(utcNow(), -30)}. Run it nightly. For very high-volume scenarios, consider routing old records to an Azure Storage Archive before deletion.

"I need idempotency on child flows, not just parent flows." Pass the idempotency key as an input parameter to each child flow. The child flow performs its own claim against the shared Flow Processing Log table, using a compound key like concat(parentKey, '-', childFlowName). This ensures each logical unit of work within a parent flow is independently idempotent. The child flows architecture lesson discusses how to design input/output contracts for child flows, which is where you'd define the idempotency key parameter.

---

Summary & Next Steps

Idempotency isn't a nice-to-have in production automation — it's the difference between a system that recovers gracefully from transient failures and one that requires manual cleanup after every blip. The patterns you've built here form a reusable foundation:

• Stable idempotency keys derived from payload business identifiers, not runtime metadata
• Atomic claim-before-process using Dataverse alternate keys to close the race condition window
• Status lifecycle management (Pending → Completed/Failed) that supports both deduplication and retry decisions
• Idempotency headers on outbound HTTP calls to propagate the guarantee downstream
• Retention policies to keep your log table from becoming a storage liability

These patterns compose well with each other and with other resilience techniques. If you're processing high volumes, look at batching and chunking strategies to understand how to apply idempotency at the batch level, not just the individual message level. And if you're governing flows at scale — tracking which flows have idempotency controls and which don't — the CoE Toolkit and governance lesson shows you how to audit flow patterns across your tenant.

The next challenge after deduplication is observability: knowing when duplicates are arriving, how often, and whether your idempotency layer is performing correctly under production load. That's the job of your monitoring pipeline — and it's where resilient integration patterns meet operational maturity.