Learn how to build reliable, automated data pipelines in Microsoft Fabric from the ground up. This hands-on lesson covers Copy Activities, dynamic parameters, activity chaining with dependencies, and scheduled triggers — everything you need to load data without manual intervention.
Imagine you're a data engineer at a mid-sized retail company. Every morning at 6 AM, your analytics team needs fresh sales data loaded from your transactional database into your Fabric lakehouse — before the merchandising team starts their daily planning session. You could log in each morning, manually copy files, and run transformations by hand. Or you could build a pipeline that does all of that automatically, reliably, and without you losing any sleep.
That's the promise of Microsoft Fabric Data Pipelines. They're the orchestration layer of the Fabric platform — the part that answers the question "how does data actually get from Point A to Point B, on schedule, without human intervention?" Where tools like Dataflow Gen2 handle the transformation of data, pipelines handle the movement and coordination — triggering activities in sequence, passing runtime values dynamically, and running on whatever schedule your business demands.
By the end of this lesson, you'll know how to build a real, functional data pipeline in Microsoft Fabric from scratch. You'll move data using Copy Activities, make your pipelines flexible with parameters, and set up automated schedules so that fresh data appears in your lakehouse without you lifting a finger.
What you'll learn:
Before diving in, you should have a Fabric workspace set up and ready to use. If you haven't done that yet, start with the lesson on Fabric Capacities and Workspaces: F SKUs, Trials, and Setting Up Your First Workspace. You should also have at least a basic lakehouse created — the lesson on Building Your First Lakehouse in Microsoft Fabric: Files, Tables, and the SQL Analytics Endpoint will get you there. No pipeline experience is required.
A data pipeline is a directed sequence of activities that execute in a defined order. Think of it like a recipe: first you gather ingredients (extract data from a source), then you prepare them (transform or move the data), then you serve the dish (write the data to a destination). The pipeline coordinates all of that, making sure each step happens at the right time and in the right order.
Fabric Data Pipelines are built on the same engine as Azure Data Factory, Microsoft's longtime cloud ETL (Extract, Transform, Load) service. If you've used Azure Data Factory before, you'll feel right at home. If you haven't, don't worry — Fabric's interface makes the concepts very accessible.
Inside a pipeline, you work with activities. Each activity is a discrete unit of work: copying data, running a notebook, executing a stored procedure, waiting for a condition, and so on. You arrange activities on a canvas, connect them with dependency arrows, and the pipeline engine executes them in order.
Note
Pipelines in Fabric are a distinct tool from Dataflow Gen2. Dataflows are excellent for transforming data using a visual, Power Query-based interface. Pipelines are better suited for orchestration — moving large volumes of data, chaining multiple steps, handling conditional logic, and scheduling complex workflows. In practice, you'll often use both: a pipeline that triggers a Dataflow Gen2 activity as one of its steps.
The most fundamental pipeline activity is the Copy Activity, which we'll focus on first.
The Copy Activity is exactly what it sounds like: it copies data from a source to a sink (the destination). It handles the connection to both ends, reading from one system and writing to another.
What makes the Copy Activity powerful is the breadth of connectors it supports. Your source could be:
Your sink can be your Fabric lakehouse (Files section or Tables section), a Fabric warehouse, Azure Blob Storage, and more.
For our retail scenario, let's say the source is a CSV file sitting in Azure Blob Storage — a common pattern where an upstream system drops a daily export. The sink will be our Fabric lakehouse's Files section, from which a downstream process will load the data into a Delta table.
Key insight
The Copy Activity does movement, not transformation. It's optimized to move data fast and reliably. If you need to reshape, filter, or enrich data during the move, you have two options: use a lightweight mapping inside the Copy Activity, or use a separate transformation step (a Dataflow Gen2 activity or a Notebook activity) after the Copy Activity completes.
Let's walk through creating a pipeline step by step.
In your Fabric workspace, click the + New button and select Data pipeline from the list. Give your pipeline a descriptive name — something like pl_load_daily_sales (the pl_ prefix is a common naming convention that makes it easy to identify pipelines in a busy workspace). Click Create.
You'll land on the pipeline canvas. It's a blank white space with a toolbar at the top and a properties panel on the right. This is where you'll assemble your activities.
Click Add activity in the toolbar (or right-click on the canvas) and select Copy data. A Copy Activity block appears on the canvas. Click on it to select it, and notice that the properties panel below the canvas updates to show configuration tabs: General, Source, Sink, Mapping, and Settings.
Start with the General tab. Give this activity a meaningful name, like Copy Sales CSV to Lakehouse Files. This name appears in run logs, so clarity here pays off when you're debugging at 7 AM because a pipeline failed overnight.
Click the Source tab. You need to create a Linked Service — which is Fabric's term for a reusable connection definition. Think of it as a saved set of credentials and connection details for a specific system.
Click + New next to the Connection field. A panel slides in showing you connector categories. For our example, select Azure Blob Storage, then fill in:
Click Test connection to verify Fabric can reach the storage account, then click Create.
Back in the Source tab, set the File path to point to your container and file. For a daily export that lands at a predictable path like sales-exports/2024-01-15/sales.csv, you'd enter the container name and file path. We'll make this dynamic with parameters shortly.
Set the File format to DelimitedText and configure whether your file has a header row (almost always yes for CSV exports).
Click the Sink tab. Click + New next to the Connection field, but this time select Microsoft Fabric → Lakehouse. You'll be prompted to choose your workspace and then your lakehouse from a dropdown. Select the lakehouse you want to load into.
Set Root folder to Files (as opposed to Tables — we're landing the raw CSV first, before any Delta table processing). Set the File path to a folder like raw/sales/ and set the file name to sales.csv.
Tip
Landing raw files in the Files section of your lakehouse before loading them into Delta tables is a widely-used pattern called "landing zone" or "raw zone" storage. It gives you a recoverable copy of the source data exactly as it arrived, which is invaluable for debugging and reprocessing. The OneLake Explained article covers how data is stored in your lakehouse in more depth.
Click Validate in the toolbar to check your pipeline for configuration errors. If validation passes, click Run to execute the pipeline manually and confirm it works before adding more complexity.
A pipeline that only ever copies one specific file from one specific path isn't very reusable. What if you have 12 regional sales files, each in a different subfolder? What if you want to use the same pipeline logic for both a daily load and a historical backfill?
This is where parameters come in. A parameter is a named placeholder whose value you supply at runtime — either manually when you trigger the pipeline, or programmatically from a parent pipeline or a schedule.
Click somewhere on the blank canvas (not on any activity) to deselect everything. The properties panel at the bottom now shows the pipeline-level properties. Click the Parameters tab.
Click + New to add a parameter. Add these two:
| Name | Type | Default Value |
|---|---|---|
SourceFolderDate |
String | 2024-01-15 |
SourceFileName |
String | sales.csv |
The default value is what the pipeline uses if you trigger it without specifying a value — useful for testing.
Now click on your Copy Activity and go back to the Source tab. Instead of typing a hardcoded file path, you'll reference your parameters using Fabric's expression language.
Click the File path field. You'll notice a small Add dynamic content link or button appears below the field. Click it. This opens the expression editor, a text area where you can write expressions using Fabric's expression syntax.
To build a path like sales-exports/2024-01-15/sales.csv dynamically, your expression would be:
@concat('sales-exports/', pipeline().parameters.SourceFolderDate, '/', pipeline().parameters.SourceFileName)
Let's unpack this:
@ tells Fabric that what follows is an expression, not a literal stringconcat() joins multiple strings togetherpipeline().parameters.SourceFolderDate retrieves the current value of the SourceFolderDate parameter at runtimeYou can use the same approach in the Sink tab to write output files to date-partitioned folders:
@concat('raw/sales/', pipeline().parameters.SourceFolderDate, '/')
Warning
The expression language in Fabric Pipelines is case-sensitive. pipeline().parameters.SourceFolderDate is not the same as pipeline().parameters.sourceFolderDate. A mismatched case is one of the most common sources of "parameter not found" errors, and it can be maddening to spot. Always copy parameter names directly from the Parameters tab rather than typing them by hand in expressions.
When you click Run with parameters defined, Fabric will prompt you to provide values for each parameter. You can override the defaults or accept them. When you trigger from a schedule or from a parent pipeline, you pass parameter values in the trigger configuration — we'll cover that in the scheduling section.
Real pipelines rarely have just one step. After copying the raw CSV into your lakehouse Files section, you probably want to run a Notebook that reads that file and writes it into a properly typed Delta table. Then maybe run a stored procedure that updates a load audit log.
You chain activities by drawing dependency arrows between them.
Click Add activity → Notebook. A new activity block appears on the canvas. In the General tab, name it Transform Sales to Delta Table. In the Settings tab, select your workspace and then the Notebook you want to execute.
You can also pass parameters to notebooks. If your notebook is parameterized (it has a cell tagged as a "parameters" cell in the notebook editor), you can supply those parameter values from the pipeline under the Base parameters section of the Settings tab.
Hover over the right edge of your Copy Activity block. You'll see a small arrow appear. Click and drag that arrow to the Notebook activity block. This creates a dependency — a line that says "only run the Notebook activity after the Copy Activity finishes."
When you draw a dependency, it defaults to On success — the Notebook only runs if the Copy Activity succeeded. Right-click the dependency arrow to see other options:
Tip
Use "On failure" dependencies to build notification branches. Add a Web Activity after a critical step that calls a webhook or Azure Logic App URL, and connect it with an "On failure" dependency. This way, your team gets an automatic alert any time that step breaks — without you having to check manually.
A simple three-step pipeline for our retail scenario looks like this in terms of activity flow:
[Copy Sales CSV] --On success--> [Transform to Delta Table] --On success--> [Log Load Completion]
If the copy fails, neither downstream activity runs. If the transformation fails, the logging step doesn't run. This guarantees that your audit log only records genuine successful loads.
A pipeline you have to run manually is useful. A pipeline that runs itself is a superpower. Fabric provides two scheduling mechanisms: Scheduled triggers and Event-based triggers (the latter, which fires when a file arrives in storage, is more advanced and beyond scope here). We'll focus on scheduled triggers.
In the pipeline toolbar, click the Schedule button (it looks like a clock icon). A configuration panel appears.
Toggle the schedule On. Then configure:
Repeat: Choose Daily, Weekly, Monthly, or a custom interval in minutes or hours. For our 6 AM daily sales load, choose Daily.
Time: Set to 06:00 AM. Make sure to set the correct Time zone — this is a commonly overlooked setting. If your Fabric capacity is in a US East region but your business runs on Pacific time, loading at 6 AM UTC will have your data arriving at 10 PM the night before local time. Always verify the time zone.
Start date: Set this to today or a future date. Fabric won't trigger runs for dates in the past.
End date (optional): If this pipeline should only run for a defined period — say, during a migration project — set an end date. Otherwise, leave it open-ended.
Click Apply.
Note
Fabric schedules are attached to the pipeline itself, not to a separate trigger object like in Azure Data Factory. This makes them simpler to manage, but it also means if you clone a pipeline, you need to reconfigure the schedule on the clone manually.
Here's where parameters and schedules combine into something genuinely powerful. Instead of hardcoding 2024-01-15 as your SourceFolderDate parameter default, you can make the pipeline calculate yesterday's date automatically every time it runs.
You don't configure this in the schedule itself — you set a dynamic default value for your parameter using an expression. Change the default value of SourceFolderDate to:
@formatDateTime(addDays(utcNow(), -1), 'yyyy-MM-dd')
This expression:
utcNow()addDays(..., -1) — because at 6 AM we want yesterday's complete data2024-01-15 with formatDateTime(..., 'yyyy-MM-dd')Now every morning when the schedule fires, the pipeline automatically targets the previous day's file with zero manual intervention.
Key insight
Dynamic expressions on parameter defaults are evaluated at pipeline run time, not at pipeline save time. This means your date expression will always calculate relative to when the pipeline runs, not when you wrote it. This is the correct behavior, but it trips people up when they test a pipeline manually weeks before it goes into production — the date in the parameter will be today's date, not the date they were testing for.
After your pipeline runs (manually or on schedule), you need to know if it succeeded. In your Fabric workspace, find your pipeline and look for the View run history option (accessible from the pipeline's context menu in the workspace list, or from within the pipeline editor via the Run → View run history button).
The run history shows each execution with a status (Succeeded, Failed, In Progress), the start time, duration, and a link to detailed activity-level logs. Click on any run to see how each activity performed individually — how many rows were copied, whether the notebook completed without errors, and so on.
For the Copy Activity specifically, the activity details show you a Data read and Data written count, along with throughput in MB/s. If a copy that normally takes 30 seconds suddenly takes 10 minutes, that throughput number will tell you immediately whether it's a network issue, a source performance problem, or a data volume spike.
Build a complete pipeline that demonstrates everything covered in this lesson. Here's a realistic scenario you can adapt to your own data:
Goal: Create a pipeline that copies a CSV file from Azure Blob Storage (or from a lakehouse Files folder as a stand-in source) to a destination folder in a lakehouse, using date-based folder organization, and runs automatically on a daily schedule.
Step 1: Create a new pipeline named pl_daily_raw_ingest.
Step 2: Add two pipeline parameters:
LoadDate (String) with default @formatDateTime(addDays(utcNow(), -1), 'yyyy-MM-dd')SourceFile (String) with default transactions.csvStep 3: Add a Copy Activity. Configure the source to read from a container/folder path using the expression @concat('exports/', pipeline().parameters.LoadDate, '/', pipeline().parameters.SourceFile). Configure the sink to write to your lakehouse Files section at @concat('raw/transactions/', pipeline().parameters.LoadDate, '/').
Step 4: Add a second activity — a Notebook activity if you have a notebook ready, or a Web Activity pointing to a public echo endpoint like https://httpbin.org/post as a stand-in. Connect it to the Copy Activity with an "On success" dependency.
Step 5: Validate the pipeline, then trigger a manual run. Supply a specific LoadDate value that matches a folder and file that actually exists in your source. Verify the file appears in the expected lakehouse path after the run.
Step 6: Enable a daily schedule for 7:00 AM in your local time zone. Verify the schedule is saved by reopening the schedule panel.
Step 7: Check the run history after your manual trigger completes. Find the Copy Activity details and note the rows copied and throughput figures.
"The copy activity fails with 'file not found'"
Nine times out of ten, this is a path expression error. Open the activity input in the run details (available in the run history for each activity) — it shows you exactly what value your expression resolved to at runtime. Compare it against what the file path actually looks like in your storage. A missing slash, an extra subfolder level, or a date format mismatch (2024/01/15 vs 2024-01-15) will cause this.
"My schedule isn't triggering" First, confirm the schedule is toggled On — it's easy to configure it and forget to enable it. Second, check that the start date isn't in the future. Third, verify the time zone setting. Fourth, check that your Fabric capacity isn't paused (a paused capacity won't execute pipelines). The lesson on Fabric Capacities and Workspaces covers capacity states.
"Parameter values aren't what I expected" Remember that parameters passed explicitly at trigger time override default values. If you're seeing unexpected values, check whether the run was triggered manually (where you might have typed a value) versus from a schedule (which uses defaults). The run history activity inputs panel will always show you the resolved value.
"The notebook activity fails even though the notebook runs fine manually" Notebooks in pipelines run as the pipeline's identity (usually a service principal or the workspace's managed identity), not as your personal user account. If the notebook accesses a lakehouse or external resource, that identity needs access. Also, make sure your notebook has a parameters cell if you're passing parameters from the pipeline — without the parameters cell tag, Fabric doesn't know where to inject the values.
"My pipeline is slow even though the files are small" Each pipeline run has a startup overhead of roughly 20-40 seconds for the orchestration layer to initialize. For very small, frequent loads, this overhead dominates the total runtime. If latency matters, consider whether Ingesting Data with Dataflow Gen2 might be a more efficient fit for small-volume, transformation-heavy scenarios.
You've covered a lot of ground. Let's recap what you can now do:
Where should you go from here? The natural next step is to understand the data stores your pipelines are loading into. If you're deciding between a lakehouse and a warehouse for your destination, the lesson on Fabric Lakehouse vs Warehouse: Choosing the Right Store for Your Workload will sharpen that decision. You should also explore the full breadth of what Microsoft Fabric offers beyond pipelines — the platform is designed so that pipelines, notebooks, dataflows, and Direct Lake reports work together as an integrated whole.
The pipeline patterns you've learned here — parameterized Copy Activities, success/failure branches, and scheduled triggers — are the foundation of every serious data engineering workflow in Fabric. As your pipelines grow more complex, you'll add ForEach loops to process batches of files, If Condition activities for branching logic, and Lookup activities to drive dynamic behavior from configuration tables. But all of that complexity is built on the same building blocks you've just mastered.