Orchestrating Multi-Notebook Workflows in Microsoft Fabric: Using Pipeline Notebook Activities, Activity Dependencies, and Output Variables to Chain PySpark Transformations Across Medallion Layers | Wicked Smart Data
Orchestrating Multi-Notebook Workflows in Microsoft Fabric: Using Pipeline Notebook Activities, Activity Dependencies, and Output Variables to Chain PySpark Transformations Across Medallion Layers
Learn how to wire Fabric PySpark notebooks into a production pipeline using Notebook activities, On Success/On Failure dependencies, and mssparkutils.notebook.exit() output variables. Build a complete Bronze→Silver→Gold medallion pipeline that passes context between stages and handles failures gracefully.
⚡ Practitioner20 min readSep 22, 2026Updated Sep 22, 2026
Orchestrating Multi-Notebook Workflows in Microsoft Fabric: Using Pipeline Notebook Activities, Activity Dependencies, and Output Variables to Chain PySpark Transformations Across Medallion Layers
Introduction
You've got three PySpark notebooks that each do one job well: the first lands raw JSON from an API into your Bronze lakehouse table, the second cleans and conforms the data into Silver, and the third aggregates it into a Gold fact table ready for Power BI. Running them one at a time by hand works fine when you're building — but that's not a production data pipeline. Production means scheduling, dependency enforcement, error handling, and passing context between stages so each notebook knows exactly what the previous one processed.
This is where Fabric's pipeline Notebook activity becomes essential. Instead of treating notebooks as standalone experiments, you wire them together inside a Data Pipeline, define success/failure dependencies between them, and use output variables — values that one notebook computes and passes forward — to give downstream notebooks the context they need. The result is a fully orchestrated, end-to-end medallion workflow that runs on a schedule, fails gracefully, and tells you exactly where and why something went wrong.
By the end of this lesson, you'll have built a real multi-notebook pipeline that processes retail sales data through Bronze → Silver → Gold layers, with proper dependency chaining and output variables flowing between stages.
What you'll learn:
How to add Notebook activities to a Fabric Data Pipeline and configure them correctly
How to define activity dependencies (On Success, On Failure, On Completion, On Skip) to control execution flow
How to use mssparkutils.notebook.exit() to produce output values from a notebook
How to reference those output values in downstream activities using dynamic expressions
How to structure a three-layer medallion pipeline that handles partial failures without rerunning clean stages
Throughout this lesson, we'll build an orchestrated pipeline for a fictional retailer called Contoso Outdoor. Every night, their point-of-sale system drops a new batch of transaction files into a storage location. The pipeline needs to:
Bronze notebook — Ingest raw CSV files into a Delta table, record a row count and the batch timestamp
Silver notebook — Clean, deduplicate, and conform the data; receive the batch timestamp from Bronze so it only processes today's records
Gold notebook — Aggregate sales by product category and region; only run if Silver succeeds; write the output row count back as a variable
This is a realistic pattern. Let's build it from the notebooks up, then wire them together in a pipeline.
Building the Three Notebooks
Before you can orchestrate notebooks, they need to be well-structured for orchestration. That means: accepting parameters, doing one clearly bounded job, and exiting with a meaningful output value.
The Bronze Notebook: Raw Ingestion
Create a notebook called nb_bronze_sales_ingest. Attach it to your lakehouse. The first cell should declare parameters — this is what Fabric's pipeline will inject values into:
# Cell 1 — Parameter cell (toggle "Parameters" on this cell in the toolbar)
source_path = "Files/raw/sales/"
batch_date = "2024-01-15" # Default; pipeline will override this
Note
To make a cell a parameter cell in Fabric notebooks, select the cell, then click the three-dot menu on the right side of the cell and choose "Toggle parameter cell." You'll see a small "Parameters" label appear at the bottom of the cell. This is how Fabric knows which values to inject when the pipeline calls the notebook.
# Cell 2 — Ingest raw CSV files
from pyspark.sql import functions as F
from datetime import datetime
raw_df = spark.read.option("header", True).option("inferSchema", True).csv(
f"abfss://your-workspace@onelake.dfs.fabric.microsoft.com/your-lakehouse.Lakehouse/{source_path}"
)
# Stamp the ingestion date onto every row
raw_df = raw_df.withColumn("ingestion_date", F.lit(batch_date)) \
.withColumn("ingested_at", F.current_timestamp())
row_count = raw_df.count()
print(f"Raw rows ingested: {row_count}")
That last cell is critical. mssparkutils.notebook.exit() takes a string — so we serialize a dictionary as JSON. This becomes the output value the pipeline can read. We're passing batch_date forward so Silver doesn't have to guess which date to process, and bronze_row_count so a downstream audit notebook (or a simple log) can confirm the numbers add up end-to-end.
The Silver Notebook: Cleansing and Conformation
Create nb_silver_sales_clean:
# Cell 1 — Parameters
batch_date = "2024-01-15"
bronze_row_count = 0 # Will be passed from pipeline; used for validation
# Cell 2 — Read from Bronze, filter to today's batch
from pyspark.sql import functions as F
import json
bronze_df = spark.read.format("delta").table("bronze_sales_raw")
# Only process the current batch
batch_df = bronze_df.filter(F.col("ingestion_date") == batch_date)
print(f"Bronze rows for batch {batch_date}: {batch_df.count()}")
Using replaceWhere on the ingestion_date partition lets you safely re-run the Silver notebook for a given batch without wiping historical data from other dates. This is idempotent — if Silver fails halfway through and the pipeline retries, you won't end up with duplicated or corrupted data.
You now have three well-structured, parameter-aware notebooks. Each one does a bounded job, accepts injection of context, and produces a structured output. Time to wire them together.
Creating the Pipeline and Adding Notebook Activities
In your Fabric workspace, create a new Data Pipeline — call it pl_sales_medallion_daily. You'll be working in the pipeline canvas.
To add a Notebook activity:
From the Activities tab in the ribbon at the top, click Notebook. A Notebook activity tile appears on the canvas.
Click the activity to select it, then in the Settings panel below the canvas, set the Notebook field to nb_bronze_sales_ingest.
Rename the activity to something readable — double-click the activity title on the canvas and type Bronze_Ingest.
Repeat this process for the Silver and Gold notebooks, naming them Silver_Clean and Gold_Aggregate respectively.
At this point, all three activities exist on the canvas but aren't connected. They'd run in parallel if you triggered the pipeline now, which is exactly what you don't want.
Warning
If you leave Notebook activities unconnected on the canvas, Fabric treats them as independent parallel branches. Bronze, Silver, and Gold would all fire simultaneously — and Silver would try to read data that Bronze hasn't written yet. Always wire up your dependencies before testing.
Configuring Activity Dependencies
Dependencies in Fabric pipelines are set by drawing connection lines between activities. Each connection carries a condition that defines when the downstream activity fires.
The four dependency conditions are:
Condition
Meaning
On Success
Run only if the upstream activity succeeded
On Failure
Run only if the upstream activity failed (useful for alerting or cleanup)
On Completion
Run regardless of success or failure
On Skip
Run if the upstream activity was skipped (because its own dependency wasn't met)
For our medallion pipeline, the primary chain is straightforward:
Bronze_Ingest → On Success → Silver_Clean
Silver_Clean → On Success → Gold_Aggregate
To draw a dependency: hover over the Bronze_Ingest activity until you see a small arrow appear on its right edge. Click and drag that arrow to the Silver_Clean activity. A connection line appears. By default it's set to On Success — you can confirm or change this by clicking the small condition label on the connection line itself.
Do the same between Silver_Clean and Gold_Aggregate.
Adding a Failure Branch
Now let's add a realistic failure handler. Create a fourth activity — a Web activity (or another Notebook activity) — called Notify_Failure. This could call a Logic App endpoint, Teams webhook, or simply log the failure to a Delta table.
Connect Silver_Clean → On Failure → Notify_Failure. Now if Silver crashes, the pipeline branches: the On Success path to Gold is skipped, but the On Failure path to your notification fires.
Tip
You can have multiple outgoing connections from a single activity with different conditions. A Silver activity might have an On Success connection to Gold AND an On Failure connection to a notification activity simultaneously. Both connections coexist — they just fire under different circumstances.
This gives you a pipeline that looks like a diamond for the failure branch: Bronze feeds Silver, Silver either feeds Gold (success) or feeds the notifier (failure). Gold's path naturally ends; the notifier's path naturally ends. Pipeline run completes when all active branches are resolved.
Passing Parameters Into Notebook Activities
Every Notebook activity in a pipeline has a Base parameters section in its Settings panel. This is where you inject values into the notebook's parameter cell. For Bronze_Ingest, you want to inject batch_date dynamically based on when the pipeline runs.
Click Bronze_Ingest to select it. In the Settings panel, scroll to Base parameters and click + New. Add:
Name: batch_date
Value: Click the field, then select Add dynamic content (the blue link that appears)
This expression evaluates to today's date in ISO format every time the pipeline runs. If you're testing, you can hardcode 2024-01-15 temporarily.
For Silver_Clean, add batch_date the same way. You'll also want to start threading through the Bronze output, but let's handle that in the next section.
Note
The parameter name you use in Base parameters must exactly match the variable name in the notebook's parameter cell. If your notebook has batch_date = "2024-01-15" in the parameter cell, the pipeline parameter name must also be batch_date. Capitalization matters.
Here's where the real power of chained notebooks comes in. When a notebook calls mssparkutils.notebook.exit("some string"), that string becomes accessible in the pipeline via the activity's output.
The dynamic expression to read the exit value from Bronze_Ingest is:
Now Silver knows how many rows Bronze produced, and can include that in its own logging or validation logic. Silver might flag a warning if its row count is dramatically lower than Bronze's (suggesting aggressive filtering that deserves investigation).
Similarly, for Gold_Aggregate, wire in Silver's output:
The exitValue is always a string, even if you pass a number from the notebook. If you try to use @activity('Bronze_Ingest').output.result.exitValue directly as a numeric parameter, you'll get a type mismatch. Always either parse it with json() to extract specific fields, or convert it explicitly with int() in the expression: @int(json(activity('Bronze_Ingest').output.result.exitValue).bronze_row_count).
Using Pipeline Variables for Cross-Activity State
Sometimes you need to store intermediate values not just for one downstream notebook, but for use by multiple activities — including non-Notebook activities like a logging step at the very end of the pipeline.
Fabric pipelines support Variables (not to be confused with parameters — variables are mutable, parameters are fixed per run). Define a variable by clicking on the pipeline canvas background (not on any activity), then in the Settings panel at the bottom, navigate to the Variables tab and add:
batch_date (String)
bronze_row_count (Integer)
silver_row_count (Integer)
gold_row_count (Integer)
After Bronze_Ingest succeeds, add a Set Variable activity between Bronze and Silver. Connect Bronze → On Success → Set Variable → On Success → Silver.
Repeat this pattern after Silver to capture silver_row_count, and after Gold to capture gold_row_count.
At the very end of the pipeline — after Gold succeeds — add a final Notebook activity called Log_Audit_Summary that receives all three counts as parameters. This notebook writes a single-row audit record to a pipeline_audit_log Delta table, giving you a complete lineage of every run: when it happened, what batch it processed, and how many rows moved through each layer.
Validating the Exit Value Structure
One of the most common sources of confusion when working with notebook exit values is that the structure of activity().output has more layers than you'd expect. It's worth understanding the full shape:
Notice that exitValue is a string, not an object — even though we passed JSON from the notebook. The json() function in the pipeline expression language deserializes it into a proper object so you can use dot notation to access fields.
If your notebook crashes without calling mssparkutils.notebook.exit(), the exitValue will either be absent or empty. Trying to evaluate @json(activity('Bronze_Ingest').output.result.exitValue).batch_date in that scenario will cause a downstream expression evaluation error. We'll cover how to handle this gracefully in the Troubleshooting section.
Hands-On Exercise: Build the Complete Pipeline
Let's put everything together. In your Fabric workspace:
Step 1: Create the notebooks
Create all three notebooks (nb_bronze_sales_ingest, nb_silver_sales_clean, nb_gold_sales_aggregate) using the code from the earlier sections. Attach each to your lakehouse. For testing purposes, you'll need some sample data — create a CSV file at Files/raw/sales/sales_2024_01_15.csv in your lakehouse with these columns: transaction_id, transaction_date, product_category, region, sale_amount, quantity. Drop in 20-30 rows of realistic retail data.
Step 2: Test each notebook individually
Run Bronze manually with batch_date = "2024-01-15" hardcoded. Verify the Delta table bronze_sales_raw appears in your lakehouse explorer. Check that the notebook's exit cell runs without error — in the Fabric notebook interface, the output of the last cell will show your JSON string. If you see an error about mssparkutils not being found, make sure you're running in Fabric (not locally), as mssparkutils is a Fabric/Synapse runtime built-in.
Do the same for Silver and Gold.
Step 3: Create the pipeline
Create pl_sales_medallion_daily. Add the four main activities:
Click Run (the Debug button for a test run, or Trigger Now for a full run). The pipeline monitoring view shows each activity's status in real time. Click on any activity to see its input parameters and output values — including the exitValue from each notebook.
After a successful run, verify:
bronze_sales_raw has rows with ingestion_date = '2024-01-15'
silver_sales_clean has the cleaned version
gold_sales_daily has the aggregated summary rows
Step 7: Test the failure path
Deliberately break Silver by changing the table name in Cell 3 to something that doesn't exist. Re-run the pipeline. Observe that Bronze succeeds, Silver fails, and the Notify_Failure activity fires while Gold is skipped.
Tip
When testing failure scenarios, use Debug mode rather than Trigger Now. Debug runs don't count toward your capacity billing the same way triggered runs do, and they give you a faster feedback loop. After a debug run, the output panel shows the full activity run history including status, duration, input, and output for every activity.
Common Mistakes & Troubleshooting
"The expression could not be evaluated" on downstream parameters
Cause: You're referencing activity('Bronze_Ingest').output.result.exitValue in a Silver parameter, but Bronze failed and its exit value is empty or missing.
Fix: Wrap your expression in a coalesce with a safe fallback, or add a conditional using the if() function:
This evaluates to 0 if Bronze didn't succeed, rather than throwing an expression error.
Notebook parameter cell not receiving values
Cause: Either the cell isn't marked as a parameter cell, or the parameter name in the pipeline doesn't match the variable name in the notebook.
Fix: In the notebook, select the relevant cell, open its three-dot menu, and confirm "Toggle parameter cell" is active — you'll see a Parameters tag at the bottom of the cell. Then double-check that the name in the pipeline's Base parameters matches exactly (case-sensitive).
`mssparkutils.notebook.exit()` doesn't appear to work
Cause: This is the correct module path in Fabric's Spark runtime, but if you're getting a NameError, it's possible your Spark session is stale or you're running a very old runtime version.
Fix: Try importing explicitly: from notebookutils import mssparkutils. In Fabric's current runtime, mssparkutils is available globally, but an explicit import resolves any namespace issues.
Exit value is truncated or missing keys
Cause: The JSON string you pass to mssparkutils.notebook.exit() is too long (there's a size limit, typically around 8KB), or a Python variable contained a None value that serialized as null, breaking your downstream int() cast.
Fix: Keep exit payloads small — pass counts, dates, and status flags, not dataframes or long lists. Guard against None before serializing:
output_payload = json.dumps({
"batch_date": batch_date or "",
"bronze_row_count": int(row_count) if row_count else 0,
"status": "success"
})
Activities run in parallel when they should be sequential
Cause: You forgot to draw connection lines between activities, or accidentally drew an On Completion dependency instead of On Success.
Fix: Click each connection line to inspect its condition. It appears as a small colored label on the line: green = On Success, red = On Failure, grey = On Completion, blue = On Skip. Verify each one matches your intent.
Pipeline succeeds but Gold table is empty
Cause: The batch_date flowing into Gold doesn't match the ingestion_date values in the Silver table, so the filter returns zero rows.
Fix: Add a debug print in Gold's notebook to display batch_date and a sample of ingestion_date values from Silver. The most common culprit is a timezone offset: pipeline().TriggerTime uses UTC, but your data might use a local timezone. Normalize both sides to the same timezone before filtering.
Warning
Be careful with @formatDateTime(pipeline().TriggerTime, 'yyyy-MM-dd') in pipelines that run at or near midnight in UTC. If your business day ends at 11 PM Eastern (4 AM UTC), a pipeline triggered at 1 AM UTC on the 16th will use 2024-01-16 as the batch date even though the business considers that data part of January 15th. Parameterize this at the pipeline level and let the scheduler or trigger pass the correct business date explicitly.
Performance Considerations
A few things worth knowing as you scale this pattern:
Spark session startup is the dominant cost for short notebooks. If your Bronze notebook runs in 30 seconds of actual computation but takes 2.5 minutes end-to-end, most of that is Spark session initialization. For medallion pipelines where each layer is fast, consider whether combining Bronze + Silver into one notebook (with a clear internal structure) would reduce total runtime. The tradeoff is lost granularity in pipeline monitoring.
Parallel branches for independent workloads. If you have multiple product lines or regions that are processed independently through the medallion layers, you can run them as parallel branches in the pipeline rather than sequential. The dependency structure would be: a single Bronze activity that ingests everything, followed by multiple independent Silver/Gold pairs that each filter to their partition. Use pipeline variables to track each branch's outcome.
Delta table optimization. Large Gold tables that are queried by Power BI in Direct Lake mode benefit from periodic Delta table optimization. Add a fifth activity at the end of your pipeline — a Notebook activity that calls OPTIMIZE gold_sales_daily ZORDER BY (transaction_date, product_category). This runs once per batch and keeps your Direct Lake reports fast without requiring a separate maintenance job.
Summary & Next Steps
You've now built a production-ready multi-notebook pipeline. The key skills you've practiced:
Notebook activities are the bridge between Fabric's orchestration layer (pipelines) and its compute layer (Spark). They let you treat any notebook as a callable, parameterized unit of work.
Activity dependencies control execution flow with precision — not just "run after," but "run only if the previous step succeeded," giving you clean failure isolation between medallion layers.
mssparkutils.notebook.exit() is your output channel from a notebook back to the pipeline. Keep exit payloads small and well-structured JSON.
Dynamic expressions like @json(activity('...').output.result.exitValue).field_name let you thread values from one stage to the next, so each notebook has exactly the context it needs without hardcoding anything.
Explore Fabric Git integration to version-control your notebooks and pipeline definitions across dev, test, and prod environments
The medallion pattern orchestrated through pipelines is the backbone of production data engineering in Fabric. Once this pattern is in your hands, every subsequent project starts with a strong, reliable foundation.
