Manual refreshes don't scale. Learn how to schedule Dataflow Gen2 refreshes automatically and chain them into a Fabric Data Pipeline alongside Copy Activities and Spark Notebooks — so your data is ready before your team arrives each morning.
You've built a Dataflow Gen2 that pulls sales data from your company's REST API, cleans it, and writes tidy Delta tables to your lakehouse. It works beautifully — when you remember to click Refresh manually. But data doesn't wait for you to remember, and your reporting team certainly doesn't. What you need is a system that wakes up at 6:00 AM every morning, runs your dataflow, then triggers downstream transformations, and finishes by the time your analysts arrive with their coffee.
That's exactly what this lesson teaches. We'll cover two complementary mechanisms in Microsoft Fabric: standalone scheduled refreshes on Dataflow Gen2, and weaving that dataflow into a Fabric Data Pipeline so it becomes one step in a fully automated end-to-end chain. You'll understand not just how to configure each option, but when each one makes sense — and what happens when things go wrong.
What you'll learn:
Before diving in, you should be comfortable with the basics of Dataflow Gen2 — building Power Query queries, setting output destinations, and running a manual refresh. If you need a refresher, the article on Ingesting Data with Dataflow Gen2: Power Query Skills in Microsoft Fabric covers that ground thoroughly.
You'll also want a Fabric-enabled workspace to follow along. If you haven't set one up yet, Fabric Capacities and Workspaces: F SKUs, Trials, and Setting Up Your First Workspace will get you there.
A basic understanding of Fabric Data Pipelines is helpful but not required — we'll explain each pipeline concept as it appears.
Before touching any UI, let's establish a mental model that will save you confusion later.
Model 1: Scheduled refresh on the Dataflow Gen2 itself
Fabric lets you attach a schedule directly to a dataflow, similar to how you'd schedule a refresh on a Power BI dataset. At the configured time, Fabric wakes up the dataflow engine, runs all your Power Query queries, and writes the results to your output destination. Nothing else is involved.
This model is simple and appropriate when your dataflow is a self-contained unit — it doesn't depend on other processes completing first, and nothing downstream needs to wait for it to finish.
Model 2: Dataflow Gen2 activity inside a Fabric Data Pipeline
A Fabric Data Pipeline is an orchestration tool — think of it as a conductor that coordinates multiple instruments (activities). When you add a Dataflow Gen2 activity to a pipeline, the pipeline controls when the dataflow runs, waits for it to complete, inspects its success or failure, and then decides what to do next.
This model is right when your dataflow is one step in a larger process. For example: first copy raw files from Azure Blob Storage, then run the dataflow to transform them, then execute a Spark notebook to compute aggregates, then refresh the Power BI semantic model. The pipeline holds all of that together.
Key insight
The two models aren't mutually exclusive by accident — they solve different problems. Standalone scheduling is simple to configure and has no pipeline overhead. Pipeline orchestration is more powerful and gives you sequencing, error handling, and conditional logic. Choose based on what your workflow actually needs.
Let's start with the simpler model. Assume you have a Dataflow Gen2 named SalesIngestion_DF in your workspace. It connects to a REST API, applies some transformations, and writes to a lakehouse table called raw_sales.
In your workspace, find SalesIngestion_DF in the item list. Hover over it to reveal the context menu (the three-dot ellipsis that appears to the right), then click Settings. This opens the dataflow's settings panel.
In the settings panel, look for the Scheduled refresh section. You'll see a toggle that is off by default. Slide it to On.
Once enabled, you'll see options for:
For a morning data load scenario, you'd configure:
Click Apply to save. Fabric will now automatically trigger this dataflow every morning at 6:00 AM Eastern.
Warning
Dataflow Gen2 scheduled refreshes consume your Fabric capacity. If your capacity is paused (which can happen on trial or low-tier SKUs to save cost), the scheduled refresh will fail silently. Monitor your refreshes in the Monitoring Hub to catch this early. The article on Monitoring Fabric Capacity Usage and Pipeline Activity with the Monitoring Hub shows you exactly how.
After your first scheduled run, go back to the dataflow's settings and look for the Refresh history section. You'll see a log of each run with a timestamp and a status — Success, Failed, or Cancelled. If a run failed, clicking into it will show you the error message, which is your starting point for debugging.
Imagine your sales data arrives as a CSV file dropped into Azure Blob Storage at 5:45 AM every morning by an upstream vendor. Your dataflow reads that file, cleans the data, and loads it to your lakehouse. You schedule the dataflow for 6:00 AM — 15 minutes of buffer seems fine.
But one morning the vendor's process runs late. The file doesn't arrive until 6:10 AM. Your dataflow ran on schedule, found stale data, and loaded yesterday's numbers again. Nobody noticed until the afternoon meeting.
Standalone scheduling is time-based, not event-based or dependency-aware. It can't say "run after the file arrives" or "run only after the preceding pipeline step succeeds." That's the gap that Fabric Data Pipelines fill.
Note
There is no native event-based trigger in Fabric Data Pipelines at the time of writing (unlike Azure Data Factory, which supports storage event triggers). The typical workaround is to use a scheduled pipeline with a short enough interval, or to invoke pipelines via the Fabric REST API from an external system that knows when the upstream data is ready.
Now let's build the orchestrated version. We'll construct a pipeline with three activities in sequence:
This covers a realistic pattern you'll encounter constantly in production environments.
In your Fabric workspace, click New item, then select Data pipeline. Give it a descriptive name like DailySales_Orchestration_PL. The pipeline canvas opens — a blank grid where you'll arrange activities.
Click the Add activity button (or simply click on the canvas and select from the activity picker that appears). Select Copy data. A rectangle labelled "Copy data" appears on the canvas.
Click the activity to select it, then use the bottom panel to configure it:
Files/raw/sales_vendor/)For a deeper walkthrough of Copy Activity configuration, see Orchestrating Loads with Fabric Data Pipelines: Copy Activities, Parameters, and Schedules.
With your Copy activity on the canvas, you now want to add a Dataflow Gen2 activity that runs after the copy succeeds.
Click Add activity again and select Dataflow. A new activity block appears on the canvas.
Now you need to connect them with a dependency arrow. Hover over the Copy activity block — small green and red dots appear on its edges. These are the On Success (green) and On Failure (red) connectors. Click and drag from the green On Success dot on the Copy activity to the Dataflow activity block. A green arrow now connects them, which tells the pipeline: "Only run the Dataflow activity if the Copy activity succeeds."
Click the Dataflow activity to select it, then in the bottom configuration panel:
That's it. The pipeline will now call your dataflow as a managed activity, wait for it to complete, and inspect its exit status before proceeding.
Tip
When a Dataflow Gen2 runs as a pipeline activity, it does not run its own standalone schedule at the same time — the two are independent. This means if you have both configured, the dataflow could run twice: once from its own schedule and once from the pipeline. Unless you intentionally want that, disable the standalone schedule on any dataflow you're orchestrating through a pipeline.
Click Add activity and select Notebook. Connect it to the Dataflow activity using another green On Success arrow.
In the Notebook activity's Settings tab:
DailySalesAggregates_NB)If your notebook accepts parameters (like the date to process), you can pass them here via the Base parameters section. For a complete treatment of writing and configuring Spark notebooks, see Transforming Data with Spark Notebooks in Microsoft Fabric: PySpark for Lakehouse Tables.
Your canvas now shows three activities connected left to right: Copy → Dataflow → Notebook. This is your sequential execution chain.
Real-world pipelines fail. The vendor API is temporarily unavailable. The dataflow times out because a query ran longer than expected. A transient network issue interrupts the copy. Your pipeline needs to handle these gracefully rather than silently stopping.
Click your Dataflow Gen2 activity on the canvas. In the bottom panel, select the General tab. You'll see:
2 or 3 for transient failures60 or 120; this gives the upstream system time to recover before retryingWith retry set to 2 and interval at 60 seconds, if your dataflow fails on the first attempt, Fabric will wait 60 seconds and try again — up to two more times — before declaring the activity failed.
What should happen if the dataflow fails even after retries? You have options:
To add a failure branch, hover over the Dataflow activity and drag from its red dot (On Failure) to a new activity. This creates a parallel path that only executes when the dataflow fails.
Warning
Don't connect failure paths back into success paths on the same pipeline run — this creates circular dependency issues. Failure handling branches should terminate cleanly, either by sending a notification or writing a log record, not by looping back into the main flow.
For a comprehensive look at pipeline-level scheduling options and notification configuration, the article on Scheduling and Automating Fabric Data Pipeline Runs with Activity-Level Retries, Alerts, and Email Notifications goes much deeper on this topic.
Now that your pipeline orchestrates the full sequence, the pipeline — not the individual dataflow — owns the schedule.
In the pipeline canvas, click the Schedule button in the top toolbar (it looks like a clock icon). This opens a scheduling panel very similar to the dataflow's standalone scheduler:
Click Apply. Your pipeline will now run on this schedule, executing all three activities in sequence each morning.
Tip
When you schedule at the pipeline level, you're giving yourself a single control point for the entire workflow. If you need to pause the morning run (say, for a holiday), you disable one schedule — not three separate dataflow, notebook, and copy schedules scattered across your workspace.
After your first scheduled run, you'll want to confirm everything worked end to end. In your workspace, click Monitoring hub in the left navigation panel. You'll see a list of recent activity runs, including your pipeline.
Click on the pipeline run to see a detailed view. Each activity (Copy, Dataflow, Notebook) appears with its own duration, status, and — critically — an input/output payload you can inspect. If the Dataflow activity shows "Succeeded" with a duration of 3 minutes and 22 seconds, you know the Power Query engine completed successfully within the pipeline context.
If the Dataflow activity shows "Failed," click the error icon next to it. You'll see the error message surfaced from the dataflow engine — the same information you'd see in standalone refresh history, but now embedded in the pipeline context so you can trace exactly where in the chain things broke.
Work through this sequence to build a complete scheduled, chained workflow:
Step 1: In your workspace, open your existing Dataflow Gen2 (or create a simple one that reads a public CSV — try the NYC Yellow Taxi dataset — and writes it to a lakehouse table). Confirm it runs successfully with a manual refresh.
Step 2: Disable any existing standalone schedule on the dataflow to avoid double-runs.
Step 3: Create a new Data Pipeline named Exercise_Orchestration_PL. Add a single Dataflow Gen2 activity, connect it to your dataflow, and configure a retry of 2 with a 30-second interval.
Step 4: Run the pipeline manually using the Run button in the toolbar. Watch the activity status indicators — they'll show "Queued," then "In Progress," then "Succeeded" (or "Failed"). Note how long the dataflow activity takes compared to a standalone manual refresh.
Step 5: Add a second activity after the dataflow. If you have a notebook, use a Notebook activity. If not, add a simple Wait activity (found in the activity picker under "Control flow") and set it to wait 10 seconds — this simulates a downstream step without needing real code. Connect it via an On Success arrow from the Dataflow activity.
Step 6: Configure a daily schedule on the pipeline for a time 5-10 minutes from now. Save the pipeline, then wait and verify the pipeline runs automatically. Check the Monitoring Hub to confirm both activities show as Succeeded.
Step 7: Deliberately break the dataflow by changing its output destination to a table name with illegal characters (like bad table!). Save the dataflow, then run the pipeline again. Observe how the failure surfaces in the pipeline run detail view, and how the Notebook/Wait activity is skipped because its predecessor failed.
Mistake: Running both a standalone schedule and a pipeline schedule simultaneously
This is the most common confusion. If your dataflow has its own 6:00 AM schedule and your pipeline also triggers it at 6:00 AM, you'll get two concurrent dataflow runs competing for capacity. The second run may queue behind the first or fail with a concurrency error. Always disable the standalone schedule when using pipeline orchestration.
Mistake: Scheduling the pipeline too early before upstream data arrives
If the Copy activity runs but finds no new file in Blob Storage (because the vendor is late), it may succeed vacuously or fail depending on your configuration. Design your Copy activity to fail explicitly when no source file is found — use the Enable logging and Fault tolerance settings on the Source tab to control this behavior.
Mistake: Forgetting time zone configuration
Fabric's default time zone is UTC. If you configure a 6:00 AM refresh without specifying a time zone, it will run at 6:00 AM UTC — which may be 1:00 AM or 11:00 PM in your local time zone. Always explicitly set the time zone.
Mistake: Not checking dataflow output destination permissions
When a Dataflow Gen2 runs inside a pipeline, it authenticates using the credentials of the dataflow's owner (the person who last saved and published it). If that person doesn't have write access to the lakehouse table, the pipeline activity will fail with a permissions error. Confirm the dataflow owner has Contributor access to the target lakehouse.
Tip
If you're building pipelines that others will maintain, consider using a service principal or a shared account as the dataflow owner to avoid the "Bob left the company and now all the pipelines are broken" problem. The article on Securing and Governing Microsoft Fabric: Workspace Roles, Item Permissions, and OneLake Data Access covers identity and permission models in detail.
Mistake: Assuming pipeline schedule and dataflow schedule have the same precision
The standalone dataflow scheduler and the pipeline scheduler both have scheduling precision to the minute, but the actual run time can drift by a few minutes depending on capacity availability. Don't design workflows that depend on activities completing at an exact wall-clock minute. Design for completion windows, not specific timestamps.
You now have two tools in your orchestration toolkit. Standalone scheduled refreshes on Dataflow Gen2 are straightforward and appropriate when your dataflow is self-contained and doesn't depend on upstream signals. Fabric Data Pipelines give you sequencing, dependency management, retry logic, and failure branching — essential when your dataflow is one step in a larger process.
The pattern we built — Copy → Dataflow → Notebook — is a fundamental building block of a medallion architecture in Fabric. The Copy activity brings raw data in, the Dataflow cleans and shapes it, and the Notebook computes business-level aggregates. If you want to see how this fits into a broader architectural approach, Implementing the Medallion Architecture in Microsoft Fabric: Bronze, Silver, and Gold Layers extends these ideas into a full multi-layer design.
From here, consider exploring:
Orchestration is where data engineering gets interesting. You're no longer just transforming data — you're building systems that work reliably while you sleep.
Microsoft Fabric Fundamentals