Incrementally Loading Data into a Fabric Lakehouse with Watermarks and Pipeline Lookup Activities

Full table loads don't scale — and sooner or later every production pipeline needs an incremental strategy. Learn how to build a complete watermark-based incremental load pattern in Microsoft Fabric using Lookup activities, parameterized Copy activities, and PySpark notebook watermark updates.

⚡ Practitioner20 min readSep 22, 2026Updated Sep 22, 2026

On this page

Incrementally Loading Data into a Fabric Lakehouse with Watermarks and Pipeline Lookup Activities

Introduction

Every production data pipeline eventually runs into the same wall: your source system has millions of rows, your lakehouse is growing by tens of millions, and running a full extract every night is starting to feel like boiling the ocean. The full load that took 4 minutes on day one now takes 47 minutes—and it's only going to get worse. Worse still, you're hammering the source database with a full table scan every time, which your DBA is quietly resenting.

The answer is incremental loading: instead of pulling everything, you pull only what's new or changed since the last time you looked. The challenge is keeping track of where "last time" ended. That's where watermarks come in. A watermark is simply a stored value—usually a timestamp or an integer ID—that marks your high-water point in the source data. Every pipeline run reads the watermark, fetches rows above it, loads them into the lakehouse, and then updates the watermark to the new maximum. It's elegant, reliable, and scales to billions of rows without drama.

In this lesson, you'll build a complete incremental load pattern in Microsoft Fabric using pipeline Lookup activities to read and write watermarks, a Copy activity to extract only the new data, and a Fabric Lakehouse table to store the watermarks themselves. By the end, you'll have a reusable, production-grade pattern you can adapt to any source system.

What you'll learn:

What a watermark is and why it's the right tool for high-volume incremental loads
How to create a watermark table in a Fabric Lakehouse and query it via the SQL Analytics Endpoint
How to wire up Lookup activities in a Fabric data pipeline to read and write watermark values
How to parameterize a Copy activity to fetch only rows newer than the last watermark
How to handle edge cases: first runs, time zone drift, late-arriving data, and partial failures

Prerequisites

This lesson assumes you're already comfortable with the following:

You have a Fabric workspace with at least Contributor access. If you're just getting started, see Fabric Capacities and Workspaces: F SKUs, Trials, and Setting Up Your First Workspace.
You've created at least one Lakehouse and understand the difference between Files and Tables. The Building Your First Lakehouse in Microsoft Fabric: Files, Tables, and the SQL Analytics Endpoint lesson covers the essentials.
You understand what a Fabric data pipeline is and how Copy activities work. If not, read Orchestrating Loads with Fabric Data Pipelines: Copy Activities, Parameters, and Schedules first.
You have a source system with a reliable change-tracking column—a timestamp like ModifiedDate or an auto-incrementing integer like OrderID. This pattern does not work on sources with no change-tracking column whatsoever.

The Watermark Pattern: How It Actually Works

Before touching the UI, let's make the logic crystal clear. Here's the complete flow of a single pipeline run:

Read the old watermark — Query the watermark table for the current high-water value for this source table (e.g., 2024-11-15 02:00:00).
Query the source for the new maximum — Before you pull data, ask the source what the highest value of the change column currently is (e.g., 2024-11-16 01:58:43). This becomes your new watermark.
Extract rows between old and new — Run the Copy activity with a WHERE ModifiedDate > '2024-11-15 02:00:00' AND ModifiedDate <= '2024-11-16 01:58:43'.
Write data to the lakehouse — Append those rows to your Delta table.
Update the watermark — Write 2024-11-16 01:58:43 back to the watermark table.

Key insight

You fetch the new maximum watermark before extracting data, not after. This bounds your query window to a fixed range. If the extraction fails halfway through, you haven't updated the watermark—so the next run will retry the same window cleanly. This is what makes the pattern safe to re-run.

Why two lookups instead of one? Because the alternative—pulling everything greater than the old watermark with no upper bound—creates a race condition. New rows could be inserted during your copy job, partially included in the extract, and then your new watermark would be set to a point that leaves gaps. Bounding both ends eliminates that risk.

Setting Up the Watermark Table

The watermark table lives in your lakehouse. You'll use the SQL Analytics Endpoint to create it with a T-SQL script, which means you can version-control it and re-run it in other environments.

Navigate to your lakehouse, then click SQL analytics endpoint in the top-right menu to switch to the SQL view. Open a new query window and run this DDL:

CREATE TABLE dbo.pipeline_watermarks (
    source_table_name   NVARCHAR(200)  NOT NULL,
    watermark_column    NVARCHAR(100)  NOT NULL,
    watermark_value     DATETIME2(7)   NOT NULL,
    last_updated_utc    DATETIME2(7)   NOT NULL,
    CONSTRAINT pk_pipeline_watermarks PRIMARY KEY (source_table_name)
);

Then seed it with an initial row for your first source table. The watermark value here is set far enough in the past to capture your full historical load on the first run—adjust it to match your business requirements:

INSERT INTO dbo.pipeline_watermarks 
    (source_table_name, watermark_column, watermark_value, last_updated_utc)
VALUES 
    ('sales.Orders', 'ModifiedDate', '2000-01-01 00:00:00', GETUTCDATE());

Note

The source_table_name field is a logical key, not necessarily a real schema-qualified table name. You could use it as a pipeline identifier like 'crm_contacts_daily' if you're pulling from a REST API rather than a relational table. The naming convention is up to you, but be consistent.

If you're loading multiple tables, insert one row per table:

INSERT INTO dbo.pipeline_watermarks 
    (source_table_name, watermark_column, watermark_value, last_updated_utc)
VALUES 
    ('sales.Customers', 'LastModified', '2000-01-01 00:00:00', GETUTCDATE()),
    ('inventory.Products', 'UpdatedAt', '2000-01-01 00:00:00', GETUTCDATE()),
    ('hr.Employees', 'ModifiedDate', '2000-01-01 00:00:00', GETUTCDATE());

Warning

The SQL Analytics Endpoint on a Lakehouse supports reading Delta tables via T-SQL, but DML write operations (INSERT, UPDATE, DELETE) are not supported directly through the endpoint. For the watermark pattern, you'll update the watermark table from within the pipeline using a Script activity or a stored procedure approach—which we'll cover shortly. The CREATE TABLE DDL above works because it creates a Delta table via the SQL endpoint; writes go through the pipeline.

This limitation is actually important to understand. Your lakehouse tables are Delta tables stored in OneLake. When you run CREATE TABLE via the SQL endpoint, Fabric creates a proper Delta table in the Tables/ section of your lakehouse. You can read it from T-SQL, PySpark, and Spark SQL. But to write to it from a pipeline, you'll use a Notebook activity or the Lakehouse's REST API rather than a raw SQL command. More on this in the pipeline build section.

To learn more about how Delta tables work in OneLake and why they're readable from so many compute engines, see OneLake Explained: One Copy of Data, Delta Tables, and Shortcuts.

Building the Pipeline: Activity by Activity

Open your Fabric workspace and create a new Data pipeline. Name it something descriptive: pl_incremental_load_orders. The naming convention matters once you have dozens of pipelines.

Step 1: Add a Parameter for Source Table Name

Before adding activities, add a pipeline parameter so the same pipeline can serve multiple source tables. Click the blank canvas background (not any activity), then select the Parameters tab at the bottom of the screen. Add these parameters:

Parameter Name	Type	Default Value
`p_source_table`	String	`sales.Orders`
`p_source_schema`	String	`sales`
`p_source_table_name`	String	`Orders`

Parameterizing at this level means you can call this pipeline from a parent orchestration pipeline, passing different table names for each invocation. This is how you scale one pattern to twenty tables.

Step 2: Lookup Activity — Read the Old Watermark

Drag a Lookup activity onto the canvas. Name it Lookup_OldWatermark.

In the Settings tab:

Data store type: Workspace
Workspace data store type: Lakehouse
Lakehouse: Select your control lakehouse (the one containing the watermarks table)
Use query: Select Query
Query:

SELECT watermark_value 
FROM pipeline_watermarks 
WHERE source_table_name = '@{pipeline().parameters.p_source_table}'

Notice the dynamic expression syntax @{...}. This is Fabric pipeline expression language, which interpolates the parameter value at runtime.

First row only: Set to true — you're expecting exactly one row.

Tip

Always use First row only: true on Lookup activities that should return a single record. If it's false and your query returns multiple rows, the output is an array you need to iterate over, which isn't what you want here. Setting it to true also makes the downstream expression syntax cleaner—you reference activity('Lookup_OldWatermark').output.firstRow.watermark_value rather than indexing into an array.

Step 3: Lookup Activity — Read the New Watermark Maximum

Add a second Lookup activity. Name it Lookup_NewWatermark. Connect it from the success output of Lookup_OldWatermark (drag the green arrow).

This activity queries the source system—not the lakehouse—for the current maximum value of the change-tracking column. Configure the linked service to point to your source SQL database (Azure SQL, SQL Server, or another compatible system).

Use query: Select Query
Query:

SELECT MAX(ModifiedDate) AS new_watermark_value 
FROM sales.Orders

If you've fully parameterized the table name, this becomes:

SELECT MAX(ModifiedDate) AS new_watermark_value 
FROM @{pipeline().parameters.p_source_schema}.@{pipeline().parameters.p_source_table_name}

First row only: true

Warning

If your source table is completely empty or no rows have changed since the last run, MAX(ModifiedDate) returns NULL. You must handle this case downstream—either with an If Condition activity that skips the Copy if the new watermark is null or equal to the old watermark, or by defaulting to the old watermark value. We'll cover this in the edge cases section.

Step 4: Copy Activity — Extract the Delta

Add a Copy activity. Name it Copy_IncrementalData. Connect it from the success output of Lookup_NewWatermark.

Source settings:

Link to your source database
Use query mode, with the following SQL:

SELECT *
FROM sales.Orders
WHERE ModifiedDate > '@{activity('Lookup_OldWatermark').output.firstRow.watermark_value}'
  AND ModifiedDate <= '@{activity('Lookup_NewWatermark').output.firstRow.new_watermark_value}'

This is the bounded window query. Everything greater than the old mark, up to and including the new mark.

Sink settings:

Data store type: Workspace → Lakehouse
Root folder: Tables
Table name: bronze_orders (or whatever your target table is)
Write method: Append

Key insight

Choosing Append as the write method means every pipeline run adds rows to the Delta table rather than overwriting it. For most incremental patterns this is correct. If your source supports updates (not just inserts), you'll need a separate merge/upsert step after the copy—typically done in a PySpark notebook that runs as a subsequent activity in the same pipeline. That post-copy transformation is where the medallion architecture pattern becomes critical: the Copy activity writes raw appended data to bronze, and a notebook promotes deduplicated/merged records to silver.

Step 5: Notebook Activity — Update the Watermark

Here's where we handle the limitation from earlier: you can't easily run a UPDATE statement against a lakehouse Delta table from a SQL Script activity in a Fabric pipeline (at least not reliably against the lakehouse's own tables). The most robust solution is a short PySpark notebook.

Create a new notebook in your workspace named nb_update_watermark. The notebook takes two parameters: source_table and new_watermark_value. Here's the full notebook code:

# Cell 1 — Parameters cell (toggle "Parameters" on this cell)
source_table = "sales.Orders"
new_watermark_value = "2024-01-01 00:00:00"
lakehouse_name = "your_lakehouse_name"

# Cell 2 — Update the watermark using Delta merge
from delta.tables import DeltaTable
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, TimestampType
from datetime import datetime

# Parse incoming watermark value
new_wm_ts = datetime.strptime(new_watermark_value, "%Y-%m-%d %H:%M:%S.%f") \
            if '.' in new_watermark_value \
            else datetime.strptime(new_watermark_value, "%Y-%m-%d %H:%M:%S")

# Build a single-row DataFrame with the update
update_df = spark.createDataFrame([
    (source_table, new_wm_ts, datetime.utcnow())
], ["source_table_name", "watermark_value", "last_updated_utc"])

# Reference the existing Delta table
watermark_table_path = f"Tables/pipeline_watermarks"
dt = DeltaTable.forName(spark, f"{lakehouse_name}.pipeline_watermarks")

# Merge: update if source_table_name matches, insert if new
dt.alias("target") \
  .merge(
      update_df.alias("source"),
      "target.source_table_name = source.source_table_name"
  ) \
  .whenMatchedUpdate(set={
      "watermark_value": "source.watermark_value",
      "last_updated_utc": "source.last_updated_utc"
  }) \
  .whenNotMatchedInsertAll() \
  .execute()

print(f"Watermark updated for {source_table} to {new_wm_ts}")

Tip

Mark Cell 1 as a Parameters cell in the notebook (click the three dots on the cell and toggle "Parameters"). This lets you pass values into the notebook from the pipeline's Notebook activity without hardcoding them. When the pipeline invokes the notebook, it injects its own values for source_table and new_watermark_value, overriding the defaults in the parameters cell.

Back in the pipeline, add a Notebook activity after the Copy activity. Configure it:

Notebook: nb_update_watermark
Base parameters:
- source_table: @{pipeline().parameters.p_source_table}
- new_watermark_value: @{activity('Lookup_NewWatermark').output.firstRow.new_watermark_value}

Connect it from the success output of the Copy activity.

For more on writing PySpark notebooks in Fabric, see Transforming Data with Spark Notebooks in Microsoft Fabric: PySpark for Lakehouse Tables.

Handling Edge Cases Properly

A pattern that only works when everything goes right isn't production-ready. Here are the scenarios you will encounter.

Edge Case 1: No New Data

When Lookup_NewWatermark returns NULL (empty source table or no changes), passing NULL into the Copy activity query will either throw a SQL error or pull unexpected results. Add an If Condition activity between Lookup_NewWatermark and the Copy activity.

Expression:

@and(
    not(equals(activity('Lookup_NewWatermark').output.firstRow.new_watermark_value, null)),
    greater(
        activity('Lookup_NewWatermark').output.firstRow.new_watermark_value,
        activity('Lookup_OldWatermark').output.firstRow.watermark_value
    )
)

Put the Copy activity and Notebook activity inside the True branch. The False branch can contain a Set Variable or simply be empty — the pipeline succeeds without copying anything, which is correct behavior.

Edge Case 2: First Run with Historical Data

When you seed the watermark with 2000-01-01, the first run pulls your entire history. This is intentional — but it can be slow. Two strategies:

Pre-load history separately — Run a one-time full copy first, then manually set the watermark to today before activating the incremental pipeline.
Batch the first run — Add a loop with date-range chunks if the initial load is too large for a single copy.

Edge Case 3: Time Zone Mismatches

If your source database stores timestamps in local time (Eastern, Pacific, etc.) but your pipeline or lakehouse operates in UTC, your watermark comparisons will drift by the UTC offset—potentially missing rows or double-loading them around daylight saving transitions.

The fix: standardize everything to UTC at the source query level:

SELECT MAX(CONVERT(DATETIME2, SWITCHOFFSET(CAST(ModifiedDate AS DATETIMEOFFSET), '+00:00'))) 
    AS new_watermark_value
FROM sales.Orders

And store watermarks in UTC consistently. This is non-negotiable for any cross-timezone pipeline.

Edge Case 4: Late-Arriving Data

Some source systems allow records to be inserted or updated with a ModifiedDate that's in the past—ETL retries, data corrections, or slow-arriving feeds from third-party systems. These rows will fall below your current watermark and get silently missed.

The pragmatic solution: add a configurable lookback buffer. Instead of using the exact old watermark value, subtract a buffer period in your source query:

SELECT *
FROM sales.Orders
WHERE ModifiedDate > DATEADD(HOUR, -2, 
    '@{activity('Lookup_OldWatermark').output.firstRow.watermark_value}')
  AND ModifiedDate <= 
    '@{activity('Lookup_NewWatermark').output.firstRow.new_watermark_value}'

This re-fetches the last two hours of data on every run, catching late arrivals at the cost of some duplicate rows in bronze. Your silver layer merge/upsert handles deduplication downstream.

Scaling to Multiple Tables with a Parent Pipeline

Once the single-table pipeline works, you'll almost certainly want to run the same pattern across a dozen tables. The right approach is a parent-child pipeline architecture:

Parent pipeline reads a configuration table (your pipeline_watermarks table works for this) to get the list of active source tables.
ForEach activity iterates over the list.
Each iteration calls the child pipeline (pl_incremental_load_orders) with the appropriate parameters.

In the parent pipeline:

Add a Lookup activity that queries all active tables:

SELECT source_table_name, watermark_column 
FROM pipeline_watermarks
ORDER BY source_table_name

Set First row only to false this time — you want all rows.

Then add a ForEach activity:

Items: @activity('Lookup_AllWatermarks').output.value
Sequential vs Parallel: Start with Sequential (one table at a time) to avoid overwhelming the source system. Once you've validated the pattern, switch to Parallel with a batch count of 3–5 depending on source system capacity.

Inside the ForEach, add an Execute Pipeline activity:

Invoked pipeline: pl_incremental_load_orders
Parameters:
- p_source_table: @{item().source_table_name}

Tip

Keep the child pipeline generic and the parent pipeline table-driven. When you need to add a new source table, you just insert a row into pipeline_watermarks—you don't touch any pipeline code. This is the difference between a system that's easy to operate and one that requires a developer every time business requirements change.

You can schedule the parent pipeline to run on whatever cadence your business needs. For scheduling guidance, see Scheduling and Automating Fabric Data Pipeline Runs with Activity-Level Retries, Alerts, and Email Notifications.

Monitoring Your Incremental Loads

An incremental pipeline that silently fails — or silently copies zero rows when it should have copied thousands — is worse than no pipeline at all. Build observability in from the start.

After the Notebook (watermark update) activity, add a final Set Variable or Append Variable activity that captures the rows copied. The Copy activity output includes a rowsCopied property you can log:

@activity('Copy_IncrementalData').output.rowsCopied

Write this to a pipeline_run_log table in your lakehouse alongside the run timestamp, source table name, old watermark, new watermark, and status. Over time this table becomes your pipeline health dashboard.

The Fabric Monitoring Hub gives you pipeline run history and activity-level duration stats at the workspace level. For deep dives into what's available, see Monitoring Fabric Capacity Usage and Pipeline Activity with the Monitoring Hub.

Note

The rowsCopied value from the Copy activity reflects rows written to the sink, not rows read from the source. If your source query returns 10,000 rows but a sink error truncates the write, rowsCopied will be less than your source count. Always validate both sides when troubleshooting discrepancies.

Hands-On Exercise

Build a complete incremental load pipeline for an AdventureWorks-style orders scenario. You'll use a freely available Azure SQL sample database as the source, or simulate one with a static Azure SQL table you create.

Setup:

Create or use an existing Fabric Lakehouse. Name it lh_adventure_works.
Using the SQL Analytics Endpoint, run the CREATE TABLE and INSERT scripts from the "Setting Up the Watermark Table" section above, creating a watermark row for 'sales.SalesOrderHeader' with watermark_column = 'ModifiedDate' and watermark_value = '2014-01-01 00:00:00' (this covers the AdventureWorks date range).
Create a linked service in your pipeline to connect to your Azure SQL source (AdventureWorks sample or equivalent).

Build the pipeline:

Create pl_incremental_load_salesorderheader with the three parameters from this lesson.
Add Lookup_OldWatermark querying pipeline_watermarks for 'sales.SalesOrderHeader'.
Add Lookup_NewWatermark querying SELECT MAX(ModifiedDate) AS new_watermark_value FROM Sales.SalesOrderHeader on the source.
Add the If Condition to gate on null/no-new-data.
In the True branch, add the Copy activity writing to Tables/bronze_salesorderheader with Append.
Create nb_update_watermark with the PySpark merge code.
Add the Notebook activity, passing the new watermark value.

Validate:

Run the pipeline manually. Check rowsCopied in the output panel.
Query the lakehouse table: SELECT COUNT(*), MIN(ModifiedDate), MAX(ModifiedDate) FROM bronze_salesorderheader.
Check the watermark table: SELECT * FROM pipeline_watermarks.
Run the pipeline a second time. Confirm rowsCopied is 0 (no new changes since the first run).
Insert a test row into your source table with ModifiedDate = GETUTCDATE(). Run the pipeline again. Confirm exactly 1 row is copied and the watermark advances.

Once you've validated the pattern end-to-end, wire it up to a Power BI report via the lakehouse's SQL Analytics Endpoint to see your data in near-real-time. The Connecting a Power BI Semantic Model to a Fabric Lakehouse in Direct Lake Mode lesson shows you how.

Common Mistakes & Troubleshooting

Mistake: Using >= instead of > for the lower bound

Your source query uses WHERE ModifiedDate >= old_watermark. Since you set the watermark to the maximum of the last run, the boundary row itself gets re-copied every time. Multiply this by twenty tables and a year of daily runs, and you have significant duplicate data accumulating in bronze. Always use strict greater-than (>) for the lower bound.

Mistake: Updating the watermark before confirming the copy succeeded

If you put the watermark update activity parallel to or before the Copy activity, a failed copy leaves the watermark advanced — meaning the next run skips the failed data permanently. Always chain: Lookup → Lookup → Copy → (on success) → Update Watermark. Never update on failure paths.

Mistake: Treating integer IDs as equivalent to timestamps

Auto-incrementing integer keys look like perfect watermarks, and often they are — but gaps in sequences (caused by rolled-back transactions) mean MAX(ID) can skip values. If a transaction inserts ID 1005, 1006, 1007 then rolls back 1006, your watermark advances past 1005 and the next run misses 1007 if it was inserted during the rollback window. Timestamp-based watermarks with a lookback buffer are generally safer.

Mistake: Expression syntax errors in the Copy activity source query

The dynamic expression @{activity('Lookup_OldWatermark').output.firstRow.watermark_value} returns a .NET DateTime object, which when interpolated into a SQL string might not include the correct format for your database. If you see CONVERSION FAILED errors on the source, explicitly format the value:

@{formatDateTime(activity('Lookup_OldWatermark').output.firstRow.watermark_value, 'yyyy-MM-dd HH:mm:ss')}

Mistake: Forgetting to set the Lakehouse name in the notebook

The line DeltaTable.forName(spark, f"{lakehouse_name}.pipeline_watermarks") requires the lakehouse to be attached to the notebook session. If you see table not found errors, verify that the lakehouse is attached to the notebook (check the Explorer pane on the left side of the notebook editor) and that the lakehouse_name variable matches exactly.

Troubleshooting: Pipeline runs but no data appears in the lakehouse

Check in order:

Did the If Condition evaluate to False? The Copy was skipped because new watermark ≤ old watermark.
Is the source query returning the expected rows when you run it directly in your source database?
Is the sink table name correct? The Copy activity creates the table if it doesn't exist — look for a table with a slightly different name (e.g., camelCase vs snake_case).
Did the Copy activity succeed (green check) but rowsCopied shows 0? That's a source query issue returning an empty result set, not a pipeline failure.

Summary & Next Steps

You now have a complete, production-grade incremental load pattern built on three core concepts: a watermark table that tracks where you left off, Lookup activities that read and write those watermarks, and a bounded window query that precisely extracts only what's new. The pattern is safe to re-run (watermarks only advance on success), scalable to multiple tables via a parent-child pipeline architecture, and observable through row-count logging.

The key engineering decisions that make this pattern reliable:

Bound both ends of the extraction window to avoid race conditions
Update the watermark only on successful copy to preserve re-run safety
Handle null new watermarks to avoid errors on quiet source tables
Use a lookback buffer if your source has late-arriving data characteristics

Where to go from here:

For sources where you don't control the schema and can't rely on a ModifiedDate column, explore Database Mirroring in Microsoft Fabric: Replicating Azure SQL and Snowflake into OneLake — Fabric's mirroring feature handles change tracking at the CDC level, eliminating the need for watermarks entirely on supported sources.
Once your incremental bronze data is landing reliably, build the silver and gold transformation layers using PySpark notebooks or Dataflow Gen2. See Implementing the Medallion Architecture in Microsoft Fabric: Bronze, Silver, and Gold Layers for the full pattern.
To connect your incrementally-loaded lakehouse tables to Power BI without waiting for a scheduled refresh, read Direct Lake Mode in Power BI: How It Works and When to Use It over Import and DirectQuery.

The incremental load pattern you've built here is the backbone of most production lakehouses. Get comfortable with it, automate the scaffolding for new tables, and you'll spend far less time babysitting pipelines and far more time delivering analytics value.

Incrementally Loading Data into a Fabric Lakehouse with Watermarks and Pipeline Lookup Activities

Incrementally Loading Data into a Fabric Lakehouse with Watermarks and Pipeline Lookup Activities

Introduction

Prerequisites

The Watermark Pattern: How It Actually Works

Setting Up the Watermark Table

Building the Pipeline: Activity by Activity

Step 1: Add a Parameter for Source Table Name

Step 2: Lookup Activity — Read the Old Watermark

Step 3: Lookup Activity — Read the New Watermark Maximum

Step 4: Copy Activity — Extract the Delta

Step 5: Notebook Activity — Update the Watermark

Handling Edge Cases Properly

Edge Case 1: No New Data

Edge Case 2: First Run with Historical Data

Edge Case 3: Time Zone Mismatches

Edge Case 4: Late-Arriving Data

Scaling to Multiple Tables with a Parent Pipeline

Monitoring Your Incremental Loads

Hands-On Exercise

Common Mistakes & Troubleshooting

Summary & Next Steps

Related Insights

Writing Your First PySpark Notebook in Microsoft Fabric: Reading CSV Files from OneLake, Exploring Data with DataFrames, and Saving Results as a Delta Table

Writing Your First PySpark Notebook in Microsoft Fabric: Reading CSV Files from OneLake, Transforming Data with DataFrames, and Writing Delta Tables to a Lakehouse

Writing Your First PySpark Notebook in Microsoft Fabric: Reading CSV Files from OneLake, Transforming Data with DataFrames, and Saving Results as a Delta Table