Using Notebook Variables and Parameters in Microsoft Fabric: Passing Pipeline Values to PySpark for Dynamic Lakehouse Ingestion

Introduction

Imagine you've built a Spark notebook that reads sales data from a CSV file and writes it into a lakehouse Delta table. It works perfectly — for January. But now your pipeline needs to run that same notebook for February, March, and every month going forward. Do you create a separate notebook for each month? Do you manually edit the file path every time? Neither option scales, and both are the kind of thing that makes data engineers quietly suffer.

The answer is notebook parameters: a mechanism that lets a pipeline pass dynamic values into a running notebook at execution time. Instead of hardcoding file paths, table names, date ranges, or environment names into your PySpark code, you define those as parameters that the calling pipeline provides. The notebook becomes a reusable tool rather than a one-time script, and your pipeline becomes the conductor that tells it exactly what to process, when, and where to put the results.

By the end of this lesson, you'll be able to build a PySpark notebook that accepts external parameters, wire it into a Fabric data pipeline that supplies those values dynamically, and apply this pattern to real ingestion scenarios like date-partitioned file loading and environment-aware routing. This is one of the highest-leverage skills you can develop in Fabric — it's what turns notebooks from experiments into production-grade infrastructure.

What you'll learn:

• How Fabric notebooks receive parameters from pipelines using the notebookutils parameter cell
• How to define, type-check, and use notebook variables safely in PySpark
• How to configure a Notebook activity in a data pipeline to pass dynamic parameter values
• How to build a realistic, reusable ingestion notebook that handles date-partitioned files
• How to test parameters interactively before wiring up the pipeline

---

Prerequisites

You should have a working Fabric workspace and at least one lakehouse before starting. If you're new to either of those, the articles on setting up your first workspace and building your first lakehouse will get you there quickly.

You should also be comfortable with the basics of writing PySpark in a Fabric notebook. If not, Transforming Data with Spark Notebooks in Microsoft Fabric covers everything you need. A basic familiarity with Fabric data pipelines is helpful — the article on orchestrating loads with Fabric data pipelines is the right background if you haven't worked with pipelines yet.

---

How Notebook Parameters Work in Fabric

Before writing a single line of code, let's understand the mechanics. When a Fabric pipeline runs a notebook, it needs some way to inject values into that notebook's execution. Fabric uses a concept borrowed directly from Databricks and Jupyter tooling: a parameter cell.

A parameter cell is an ordinary notebook code cell that you tag with a special designation. When the notebook runs standalone (you click Run All yourself), the cell executes normally and your hardcoded default values apply. When a pipeline invokes the notebook and passes parameters, Fabric injects a new cell immediately after the parameter cell that overrides those variables with the pipeline's values. Your notebook code below that point picks up the overridden values automatically.

This is elegant because it means the same notebook is runnable in two modes:

• Interactive mode: default values kick in, so you can develop and debug normally
• Pipeline mode: injected values override the defaults, so the pipeline drives the behavior

The parameter cell itself is just Python code — you assign your variables their default values there. The only special thing about it is the tag you apply to mark it.

---

Creating a Parameter Cell in a Fabric Notebook

Let's build a notebook from scratch. In your Fabric workspace, create a new notebook and attach it to your lakehouse. Give it a descriptive name like nb_ingest_sales_partition.

In the first code cell, write your parameter assignments:

# Default parameter values — these get overridden when called from a pipeline
source_folder = "raw/sales/2024/01"
target_table  = "sales_bronze"
load_month    = "2024-01"
overwrite_mode = "append"

Now tag this cell as the parameter cell. In the Fabric notebook UI, look at the top-right corner of the cell — you'll see a three-dot menu (the ellipsis button). Click it, then select Toggle parameter cell. When you do this successfully, the cell will display a small "Parameters" label at the bottom-left of the cell block. That label is your confirmation.

That's genuinely all it takes to make a cell into a parameter cell. The rest of your notebook code simply uses those variable names, and they'll work correctly whether you're running interactively or being called from a pipeline.

---

Writing the Notebook Logic Using Parameters

Now let's write the actual ingestion logic that consumes those parameters. Add a second cell below the parameter cell:

from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, IntegerType, DateType

print(f"Starting ingestion run:")
print(f"  Source folder : {source_folder}")
print(f"  Target table  : {target_table}")
print(f"  Load month    : {load_month}")
print(f"  Write mode    : {overwrite_mode}")

This logging step is worth more than it looks. When your pipeline runs dozens of notebook executions in a month, those print statements become the audit trail you'll rely on to diagnose issues. Always log your parameter values at the top.

Add a third cell for reading the source files:

# Build the full path using the parameter
source_path = f"Files/{source_folder}"

# Read all CSV files in the source folder
df = (
    spark.read
    .option("header", "true")
    .option("inferSchema", "true")
    .csv(source_path)
)

print(f"Rows read: {df.count()}")
df.printSchema()

Notice we're using source_folder directly from the parameter cell. The Files/ prefix is how Fabric Spark notebooks address the lakehouse file system — it maps to the Files section of the attached lakehouse.

Add a fourth cell that adds a load metadata column and writes to the target table:

# Add a tracking column so we know which pipeline run loaded each row
df_with_meta = df.withColumn("load_month", F.lit(load_month)) \
                 .withColumn("ingested_at", F.current_timestamp())

# Write to the lakehouse Delta table using the parameterized mode
(
    df_with_meta
    .write
    .format("delta")
    .mode(overwrite_mode)
    .saveAsTable(target_table)
)

print(f"Write complete. Table: {target_table}, Mode: {overwrite_mode}")

This is a fully functional ingestion notebook. It reads from a parameterized path, enriches the data with metadata, and writes to a parameterized table name. For a deeper look at the write patterns available — append vs overwrite vs merge — see Writing Data from a Spark Notebook to a Fabric Lakehouse Delta Table.

---

Type Safety and Input Validation

One important thing to understand: parameters passed from a pipeline arrive as strings, even if the pipeline passes a number or boolean. This catches a lot of people off guard. If your pipeline sends "3" expecting PySpark to treat it as the integer 3, it won't automatically — you'll get string behavior.

Add a validation cell immediately after your parameter cell to handle this:

# Parameter cell output is always string — coerce and validate
source_folder  = str(source_folder).strip()
target_table   = str(target_table).strip()
load_month     = str(load_month).strip()
overwrite_mode = str(overwrite_mode).strip().lower()

# Validate overwrite_mode is one of the allowed values
allowed_modes = {"append", "overwrite"}
if overwrite_mode not in allowed_modes:
    raise ValueError(f"overwrite_mode must be one of {allowed_modes}, got: '{overwrite_mode}'")

# Validate load_month format
import re
if not re.match(r"^\d{4}-\d{2}$", load_month):
    raise ValueError(f"load_month must be in YYYY-MM format, got: '{load_month}'")

print("Parameter validation passed.")

This might feel like defensive overkill on a small project. It isn't. When a pipeline misconfigures a parameter at 2am and your notebook writes an empty table in "append" mode forever, you'll wish it had failed loud and early instead.

---

Adding the Notebook to a Pipeline

Now let's wire this notebook into a pipeline that passes parameters dynamically. In your Fabric workspace, open or create a data pipeline. If you're building a new one, name it something like pl_ingest_sales_monthly.

Step 1: Add a Notebook activity

In the pipeline canvas, click the Add activity button (or drag from the Activities pane) and select Notebook. This adds a Notebook activity tile to the canvas.

Step 2: Configure the Settings tab

Click on the Notebook activity to select it, then click the Settings tab in the properties panel below the canvas. Use the Notebook dropdown to select your nb_ingest_sales_partition notebook from the workspace.

Step 3: Add base parameters

Still on the Settings tab, scroll down to find the Base parameters section. This is where you define what to pass into the notebook. Click + New to add a parameter entry.

Add four parameters matching the names you defined in the parameter cell:

Name	Type	Value
source_folder	String	@concat('raw/sales/', formatDateTime(pipeline().parameters.report_month, 'yyyy/MM'))
target_table	String	sales_bronze
load_month	String	@pipeline().parameters.report_month
overwrite_mode	String	append

Notice that the source_folder and load_month values use dynamic expressions (the @ prefix tells the pipeline engine to evaluate the expression rather than treat it as a literal string). This is how the pipeline's own parameters feed into the notebook's parameters.

Step 4: Define the pipeline parameter

For those expressions to work, your pipeline needs its own report_month parameter. In the pipeline canvas, click on an empty area to deselect any activity, then look at the Parameters tab in the bottom properties panel. Click + New and add:

• Name: report_month
• Type: String
• Default value: 2024-01

---

Testing the Parameter Flow End to End

Before you schedule anything, test manually. In the pipeline editor, click Run (the play button in the toolbar) to trigger a debug run. A dialog will appear asking you to provide parameter values — enter 2024-02 for report_month and click OK.

Watch the activity output in the pipeline run monitor. Once the Notebook activity turns green, click the activity tile and then the glasses icon to open the run output. You'll see the print statements your notebook logged:

Starting ingestion run:
  Source folder : raw/sales/2024/02
  Target table  : sales_bronze
  Load month    : 2024-02
  Write mode    : append

If those values match what you passed in, congratulations — the parameter flow works. If you see the default values from your parameter cell instead (raw/sales/2024/01), it means the base parameters weren't configured correctly in the pipeline activity settings. Double-check that the names in the base parameters section exactly match the variable names in the parameter cell.

---

A More Realistic Pattern: ForEach Loops with Notebook Parameters

One of the most powerful applications of parameterized notebooks is combining them with a ForEach activity in the pipeline. This lets you run the same notebook multiple times in parallel, each with different parameter values.

Suppose you need to backfill six months of sales data. Instead of running your pipeline six times manually, you define a ForEach loop that iterates over a list of months and invokes the notebook once per month.

In your pipeline, add a ForEach activity. Set its Items property to:

@createArray('2023-08','2023-09','2023-10','2023-11','2023-12','2024-01')

Inside the ForEach activity, add your Notebook activity. In the Notebook activity's base parameters, set the load_month value to:

@item()

And set source_folder to:

@concat('raw/sales/', replace(item(), '-', '/'))

The @item() expression gives you the current value from the ForEach loop's iteration. With batch count set to 3 in the ForEach settings, Fabric will run three notebook instances in parallel at a time, which is a sensible balance between speed and Spark cluster resource usage.

This pattern is explored in depth in the context of incremental loads in Incrementally Loading Data into a Fabric Lakehouse with Watermarks and Pipeline Lookup Activities, which pairs nicely with the parameterization approach here.

---

Hands-On Exercise

Work through this end-to-end scenario in your own Fabric environment:

Scenario: Your company receives daily customer transaction files dropped into your lakehouse Files section. Each file is named transactions_YYYYMMDD.csv and lives under raw/transactions/YYYY/MM/. You need a parameterized pipeline that ingests any given day's file on demand.

Step 1: Upload a sample file. Create a small CSV with columns transaction_id, customer_id, amount, and transaction_date. Save it as transactions_20240315.csv and upload it to Files/raw/transactions/2024/03/ in your lakehouse. You can do this by expanding the Files node in the Lakehouse Explorer and using the Upload button.

Step 2: Create a notebook named nb_ingest_transactions_daily. Define a parameter cell with these defaults:

transaction_date = "20240315"
source_root      = "raw/transactions"
target_table     = "transactions_bronze"

Step 3: Write the notebook body to:

• Derive the year and month from transaction_date using Python string slicing (transaction_date[:4] for year, transaction_date[4:6] for month)
• Build the full file path as Files/{source_root}/{year}/{month}/transactions_{transaction_date}.csv
• Read the CSV, add ingested_at and transaction_date columns, and write to target_table

Step 4: Test it interactively by clicking Run All with the default parameters. Confirm the table appears in your lakehouse.

Step 5: Create a pipeline named pl_ingest_transactions_daily. Add a pipeline parameter run_date (String, default 20240315). Add a Notebook activity pointing to your notebook, and wire transaction_date → @pipeline().parameters.run_date.

Step 6: Trigger a manual run with run_date = 20240316 (you'll need to upload a second file for that date, or let the notebook fail gracefully and verify it received the correct parameter value in the logs).

---

Common Mistakes & Troubleshooting

The notebook uses default values instead of pipeline values

The most common cause is a mismatch between parameter names. Open the Notebook activity's Settings tab in your pipeline and compare the names in the base parameters section character-by-character with the variable names in your notebook's parameter cell. They must be identical, including case.

The parameter cell isn't being recognized

If you tagged the cell correctly but the pipeline still doesn't inject values, check whether you've saved the notebook after tagging. In Fabric notebooks, unsaved changes aren't visible to the pipeline. Use the autosave indicator in the top bar or press Ctrl+S.

Type errors when using parameters in PySpark operations

Remember: all incoming parameters are strings. If you need an integer (say, for a row limit or partition count), explicitly cast it in your validation cell:

max_rows = int(max_rows)  # convert from string after parameter injection

The Files/ path isn't found

When you reference Files/source_folder in PySpark, the notebook must be attached to a lakehouse that contains that path. If you detach and re-attach the lakehouse, or work across multiple lakehouses, the path may not resolve. Verify the lakehouse attachment in the notebook's Lakehouse panel on the left side of the notebook UI.

ForEach loop runs all iterations sequentially even with batch count set

The ForEach sequential checkbox overrides the batch count. If your iterations are running one at a time, check whether the Sequential checkbox is enabled in the ForEach activity settings and uncheck it.

---

Summary & Next Steps

You've now learned the full parameter injection pattern for Fabric notebooks: how to define a parameter cell with sensible defaults, how to validate and type-coerce incoming values safely, and how to configure a pipeline's Notebook activity to supply those values dynamically via expressions. You've also seen how ForEach loops amplify this pattern by letting a single notebook handle batch processing across a range of inputs.

This is a foundational pattern in production Fabric architectures. Once your notebooks are parameterized, they stop being scripts tied to one particular file or date, and start being reusable processing units that your pipelines can orchestrate intelligently.

Here's where to go next:

• Explore the medallion architecture in Fabric to see how parameterized notebooks fit into a layered bronze/silver/gold design — typically you'll have a separate parameterized notebook for each layer transformation.
• Learn how to create and manage lakehouses with notebooks for more advanced file reading patterns including reading across multiple lakehouses via OneLake paths.
• If you're also parameterizing Dataflow Gen2 queries (a similar pattern for Power Query-based ingestion), see Parameterizing Dataflow Gen2 Queries with Pipeline Integration.
• Once your Delta tables are landing reliably, consider optimizing their performance with OPTIMIZE, VACUUM, and Z-Order — especially important when you're appending daily partitions and query patterns are date-range heavy.