Implementing Incremental Refresh for Direct Lake Semantic Models in Microsoft Fabric: Configuring Delta Table Partitioning, Framing Policies, and Triggering Refresh via the XMLA Endpoint | Wicked Smart Data
Implementing Incremental Refresh for Direct Lake Semantic Models in Microsoft Fabric: Configuring Delta Table Partitioning, Framing Policies, and Triggering Refresh via the XMLA Endpoint
Direct Lake incremental refresh isn't a checkbox — it requires aligning Delta table partitioning, TMDL framing policies, and XMLA-triggered refresh operations into a precise orchestration chain. This lesson gives you the full architecture, the code, and the troubleshooting knowledge to implement it correctly in production.
🔥 Expert26 min readSep 22, 2026Updated Sep 23, 2026
Implementing Incremental Refresh for Direct Lake Semantic Models in Microsoft Fabric: Configuring Delta Table Partitioning, Framing Policies, and Triggering Refresh via the XMLA Endpoint
Introduction
Here's a scenario you've probably lived through: your Power BI semantic model works beautifully when the dataset is small, but as your lakehouse grows to hundreds of millions of rows, the nightly full refresh starts running for 90 minutes, hammering your Fabric capacity, and occasionally timing out. Your stakeholders are getting stale data, your capacity metrics are screaming, and the refresh window is too narrow. The obvious answer — incremental refresh — feels like it should just work with Direct Lake, but when you go looking for the "Enable Incremental Refresh" button in Power BI Desktop or the Fabric portal, you find that the familiar Import-mode wizard doesn't apply. The rules are different here, and most of the documentation leaves out the hard parts.
Direct Lake semantic models don't refresh data at all in the traditional sense — they read directly from Delta tables in OneLake. That's both the power and the complexity. When you implement incremental refresh for a Direct Lake model, you're really doing two distinct things: structuring your Delta tables so that only recent partitions need to be "framed" into the model during a refresh cycle, and defining a policy that tells the Analysis Services engine which partitions are hot (incrementally refreshed) and which are cold (archived after a full historical load). Those two concerns — the Delta layer and the semantic model layer — have to be aligned precisely, and misaligning them is where almost everyone hits problems.
By the end of this lesson, you'll have a complete, production-ready understanding of how to implement incremental refresh end-to-end for Direct Lake semantic models. You'll know how to partition Delta tables correctly, how to define and deploy framing policies using Tabular Model Definition Language (TMDL), and how to trigger partition refresh operations programmatically through the XMLA endpoint using SSMS and Python. You'll also understand the failure modes, the capacity implications, and the edge cases that the official documentation glosses over.
What you'll learn:
How Direct Lake framing works and why it's fundamentally different from Import mode incremental refresh
How to design Delta table partitioning schemes that align with Analysis Services partition boundaries
How to write and deploy incremental refresh policies using TMDL via the XMLA endpoint
How to trigger selective partition refreshes programmatically using SSMS and the semantic-link Python library
How to troubleshoot framing failures, partition mismatches, and capacity overruns
Prerequisites
You should arrive at this lesson with solid footing in a few areas. You need to understand how Direct Lake mode works conceptually — if you're shaky there, read Direct Lake Mode in Power BI: How It Works and When to Use It over Import and DirectQuery before continuing. You also need to know how Delta tables are structured in a Fabric lakehouse, including partitioning and the transaction log — OneLake Explained: One Copy of Data, Delta Tables, and Shortcuts covers that. Familiarity with the XMLA endpoint (connecting via SSMS or Python) is assumed, as is comfort with PySpark for writing partitioned Delta tables.
From an infrastructure standpoint, you need a Fabric capacity of at least F64 (or Premium P1/EM3) to use the XMLA endpoint for write operations. F2 and F4 trials support XMLA reads but not writes, which means you can inspect models but cannot push policy changes or trigger partition refreshes programmatically.
How Direct Lake Framing Actually Works
Before we write a single line of code, we need to be precise about what "refresh" means in Direct Lake. This distinction shapes every decision you'll make.
In Import mode, refresh means: query the source, bring data into the VertiPaq in-memory engine, compress it, and swap the old data for the new. In Direct Lake mode, there is no data movement into the engine at all — the semantic model reads Parquet files from OneLake directly via the storage abstraction layer. So what does a Direct Lake "refresh" actually do?
It performs framing. Framing is the process by which the Analysis Services engine reads the Delta transaction log, identifies the current set of Parquet files that represent each table (respecting the Delta snapshot isolation guarantee), and registers those file pointers in its internal metadata. The engine then knows exactly which files to read when a query touches a given table. Framing does not copy data. It creates a stable, point-in-time view of the Delta table that query processing can rely on.
This has a profound implication for incremental refresh: the "cost" of a full framing operation scales with the number of Delta table files (and therefore the complexity of the transaction log), not the raw row count. A table with 10 billion rows stored in well-optimized, large Parquet files can frame faster than a table with 100 million rows stored across thousands of tiny files created by poorly managed micro-batch writes. If you've read about optimizing Delta table performance in a Fabric lakehouse, you know that running OPTIMIZE and VACUUM regularly directly reduces your framing latency.
Incremental refresh in Direct Lake narrows the framing scope. Instead of re-framing an entire table on every refresh cycle, you define partitions — each corresponding to a slice of the Delta table, typically by date range — and only re-frame the partitions that could have received new or changed data. Historical partitions are framed once and then declared "processed," meaning the engine won't re-examine them until you explicitly tell it to.
Key insight
Incremental refresh in Direct Lake is not about moving less data — it's about reducing the framing cost by scoping which Delta table partitions the engine needs to re-examine on each cycle. The efficiency gains come from reducing metadata operations, not from avoiding data transfer.
Designing Your Delta Table Partitioning Strategy
The foundation of any workable incremental refresh implementation is a correctly partitioned Delta table. The Analysis Services partition boundaries for your semantic model table must map cleanly onto the physical partitioning of your Delta table. If they don't align, you'll either re-frame more data than necessary or end up with broken partitions that fall back to full scans.
The Date-Based Partition Column
The canonical pattern is to partition your Delta table by a date-derived column — typically year and month. The partition column needs to be a column that exists in the table and whose values are stable for historical records. For an orders fact table in a retail context, order_date is a natural choice. For an events table in a streaming context, event_date works. The key constraint: the partition column must be of integer, date, or string type. Delta Lake doesn't care about semantics, but Analysis Services will.
For incremental refresh to work correctly, you need a column that Analysis Services can use in a RangeStart/RangeEnd filter. The convention is to use a datetime column, but in practice you'll often work with integer surrogate keys (like a DateKey of format YYYYMMDD) or a proper date column. The TMDL framing policy you'll write later references this column by name, so it needs to be queryable from the semantic model's perspective.
Here's how you'd write a well-partitioned fact table from PySpark in a Fabric notebook:
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, year, month, to_date
from delta.tables import DeltaTable
spark = SparkSession.builder.getOrCreate()
# Load your silver-layer orders table
silver_orders = spark.read.format("delta").load(
"abfss://your-workspace@onelake.dfs.fabric.microsoft.com/silver.Lakehouse/Tables/orders"
)
# Add a partition-friendly date column if not already present
# We'll use a year_month integer like 202401 for efficient partition pruning
gold_orders = silver_orders.withColumn(
"order_date", to_date(col("order_timestamp"))
).withColumn(
"partition_year", year(col("order_date")).cast("int")
).withColumn(
"partition_month", month(col("order_date")).cast("int")
)
# Write to gold lakehouse with physical partitioning on year and month
(
gold_orders.write
.format("delta")
.mode("overwrite")
.partitionBy("partition_year", "partition_month")
.option("overwriteSchema", "true")
.option("delta.columnMapping.mode", "name") # Required for column renaming later
.save("abfss://your-workspace@onelake.dfs.fabric.microsoft.com/gold.Lakehouse/Tables/fact_orders")
)
Notice a few things about this write:
Two-level partitioning (year + month) is the sweet spot for most use cases. Year-only gives you too-large partitions that defeat the purpose. Day-level gives you too many partitions and makes the Delta transaction log unwieldy. Month-level is the right granularity for most incremental refresh scenarios where you're refreshing the current month's partition daily.
Column mapping mode (delta.columnMapping.mode = "name") is enabled here because Direct Lake semantic models benefit from it — it allows you to rename columns in the Delta table without rewriting files, which matters when you need to clean up partition column naming in a live model.
After writing, run OPTIMIZE on recently written partitions to consolidate small files. This is critical — incremental appends from pipeline runs create many small Parquet files per partition, and each file creates metadata entries that the framing process must enumerate.
from delta.tables import DeltaTable
delta_table = DeltaTable.forPath(
spark,
"abfss://your-workspace@onelake.dfs.fabric.microsoft.com/gold.Lakehouse/Tables/fact_orders"
)
# Optimize only the current month's partition — don't touch historical ones
current_year = 2025
current_month = 1
delta_table.optimize().where(
f"partition_year = {current_year} AND partition_month = {current_month}"
).executeCompaction()
Warning
Running OPTIMIZE without a partition filter on a large table will create a massive Delta transaction log entry and significantly increase framing time on the next refresh. Always scope OPTIMIZE operations to the partitions that received new writes.
Understanding the TMDL Framing Policy
Now that your Delta table is correctly partitioned, you need to tell the Analysis Services engine how to map its partition boundaries onto those Delta table partitions. This is done through the model's incremental refresh policy, which you define in TMDL (Tabular Model Definition Language) and deploy via the XMLA endpoint.
TMDL is the successor to TMSL (Tabular Model Scripting Language). For Direct Lake semantic models in Fabric, TMDL is the authoritative way to define incremental refresh policies programmatically. The Fabric portal UI exposes some of these settings for Import-mode models, but for Direct Lake the XMLA endpoint is your control plane.
The Policy Structure
An incremental refresh policy for a Direct Lake model has several key components:
Let me walk you through the semantics of each field, because getting these wrong is the most common source of broken incremental refresh:
rollingWindowGranularity and rollingWindowPeriods: These define the total historical window the model will maintain. "month" + 3 means the model keeps 3 months of data available via partitioned incremental refresh. Data older than 3 months is either dropped (if you're not archiving) or kept in a permanent "historical" partition.
incrementalGranularity and incrementalPeriods: These define the hot zone — the range of time that gets re-framed on every refresh cycle. "month" + 2 means the last 2 months are always re-framed. This matters because data corrections often arrive for the previous month, and you want those corrections to be picked up without a full historical reload.
incrementalPeriodsOffset: This is subtle but important. A value of 0 means the incremental window includes the current incomplete period (today's partial month). A value of -1 means it starts from the completed previous period. For most fact tables where you're loading completed-day data, -1 gives you cleaner partition boundaries.
sourceExpression: This is the name of the lakehouse table as referenced in the model's data source. For Direct Lake, this is literally the table name in the lakehouse, not a DAX expression.
pollingExpression: This M expression is evaluated to detect whether the source has changed since the last refresh. For Direct Lake, = DateTime.LocalNow() is typically used as a no-op that forces re-evaluation, since the Delta transaction log is the authoritative change detector.
Note
The pollingExpression in Direct Lake behaves differently from Import mode. In Import, it prevents unnecessary data movement when the source hasn't changed. In Direct Lake, the engine always re-frames partitions in the incremental window regardless of whether the polling expression suggests changes — it's cheap enough that the guard isn't needed in the same way.
Connecting to the XMLA Endpoint
To deploy your framing policy and trigger partition refreshes, you need to connect to your Fabric semantic model via the XMLA endpoint. The XMLA endpoint URL follows this format:
powerbi://api.powerbi.com/v1.0/myorg/Your Workspace Name
You can find the exact URL in the Fabric portal by navigating to your workspace settings. This is a standard Analysis Services connection string, which means you can use SSMS, Tabular Editor, the mssqlserver Python libraries, or the Microsoft.AnalysisServices.AdomdClient NuGet package.
Warning
Write operations through the XMLA endpoint (which includes deploying refresh policies and triggering partition refreshes) require the workspace to be on Premium/Fabric capacity, and you must have at least Contributor access to the workspace. Read-only XMLA connections work on lower-capacity SKUs but you won't be able to push changes.
Connecting with SSMS
In SSMS, select "Connect to Server" and set the server type to "Analysis Services." Enter your XMLA endpoint URL as the server name. For authentication, use "Azure Active Directory - Universal with MFA" and enter your Microsoft account. Once connected, you'll see your workspace listed as a catalog, and semantic models appear as databases.
Deploying the Refresh Policy via XMLA
The most reliable way to deploy an incremental refresh policy to an existing Direct Lake model is to use a TMSL createOrReplace command. Here's a complete example targeting a fact_orders table in a model called Sales Analytics:
Execute this in an SSMS XMLA query window by right-clicking the database, selecting "New Query > XMLA," pasting the JSON, and clicking Execute. A successful response looks like:
After deploying the policy, the partition structure visible in SSMS under the table will change. You'll see that the original single partition has been replaced by multiple partitions named with a date-range suffix pattern like fact_orders-202501 for January 2025. These partitions are what you'll target with selective refresh operations.
Triggering Incremental Refresh Programmatically
Deploying the policy is a one-time operation (per model, per table). The ongoing work is triggering refresh operations on a schedule — typically from a Fabric Data Pipeline. You have two good options: SSMS for ad-hoc operations, and Python (semantic-link or adomd) for pipeline automation.
Triggering via SSMS (Ad-Hoc)
For ad-hoc operations or debugging, you can use a TMSL refresh command that targets specific partitions:
The type field here is "full" because you want to re-frame the entire partition, not a subset of it. In Direct Lake context, this means: re-read the Delta transaction log for files that belong to this partition's date range and update the internal file registry.
You can also use "type": "automatic" to let the engine decide which partitions need refreshing based on the policy definition, but for production automation you typically want explicit control.
Triggering via Python
For pipeline integration, Python gives you repeatable, parameterizable control. The semantic-link library (sempy) available in Fabric notebooks provides a clean interface:
import sempy.fabric as fabric
import json
from datetime import datetime, timedelta
# Identify the partitions to refresh
# Convention: re-frame current month and previous month
today = datetime.utcnow()
current_period = f"{today.year}{today.month:02d}"
previous_month = today.replace(day=1) - timedelta(days=1)
previous_period = f"{previous_month.year}{previous_month.month:02d}"
workspace_name = "Data Platform - Production"
dataset_name = "Sales Analytics"
table_name = "fact_orders"
partitions_to_refresh = [
f"{table_name}-{previous_period}",
f"{table_name}-{current_period}"
]
# Build the TMSL refresh command
refresh_command = {
"refresh": {
"type": "full",
"objects": [
{
"database": dataset_name,
"table": table_name,
"partition": partition_name
}
for partition_name in partitions_to_refresh
]
}
}
# Execute via the XMLA endpoint
client = fabric.PowerBIRestClient()
# Use the ExecuteQueries endpoint for XMLA operations
response = fabric.execute_xmla(
workspace=workspace_name,
dataset=dataset_name,
xmla_command=json.dumps(refresh_command)
)
print(f"Refresh triggered for partitions: {partitions_to_refresh}")
print(f"Response: {response}")
Tip
If you're running this Python code from a Fabric notebook, you don't need to manage authentication — the notebook runtime inherits the workspace identity. If you're running this from an external scheduler or Azure Function, you'll need a service principal with the Fabric API permissions and an OAuth token exchange before making XMLA calls.
Alternatively, the Microsoft.AnalysisServices.AdomdClient approach gives you finer-grained control and better error handling for production scenarios:
import clr
import sys
# This approach works when running from a .NET-capable environment
# For Fabric notebooks, sempy is preferred
clr.AddReference("Microsoft.AnalysisServices.AdomdClient")
from Microsoft.AnalysisServices.AdomdClient import AdomdConnection, AdomdCommand
connection_string = (
"Data Source=powerbi://api.powerbi.com/v1.0/myorg/Data Platform - Production;"
"Initial Catalog=Sales Analytics;"
"User ID=app:your-service-principal-id@your-tenant-id;"
"Password=your-client-secret;"
)
tmsl_command = json.dumps(refresh_command)
with AdomdConnection(connection_string) as conn:
conn.Open()
cmd = AdomdCommand(tmsl_command, conn)
cmd.CommandTimeout = 3600 # 1 hour timeout for large partition refreshes
result = cmd.ExecuteNonQuery()
print(f"Command executed. Result: {result}")
Integrating with a Fabric Data Pipeline
For production orchestration, you'll call the semantic model refresh from a Fabric Data Pipeline using a Web Activity that calls the Fabric REST API, or by embedding a Fabric notebook (containing the sempy-based Python above) as a Notebook Activity in the pipeline.
The Notebook Activity approach is cleaner because it handles authentication automatically and gives you access to pipeline parameters. Your pipeline looks like:
Load Activity: Runs your PySpark transformation notebook that writes the current month's Delta table partition
Optimize Activity: Runs a notebook that calls delta_table.optimize().where(...) on the current partition
Refresh Activity: Runs the sempy notebook that triggers the XMLA partition refresh
You can pass the year and month as pipeline parameters into each notebook activity, making the whole flow date-aware and rerunnable. The pattern for parameterizing notebooks is covered in using notebook variables and parameters in Microsoft Fabric.
Key insight
Always run OPTIMIZE on the Delta partition before triggering the XMLA refresh. If you trigger the refresh first, the engine frames the unoptimized file set. Running OPTIMIZE afterward creates a new Delta snapshot, but the model is already framed to the old snapshot and won't pick up the compacted files until the next refresh cycle. The order matters.
Managing the Historical Partition
When you first deploy an incremental refresh policy to an existing model table, the engine needs to build out the full partition structure. This initial setup refresh is different from ongoing incremental refreshes — it's heavier because it must process all historical partitions.
You trigger this with a processRecalc or full table refresh:
This will be slow — potentially hours for a large table — but it only needs to happen once. After the initial full refresh, all historical partitions older than your rolling window boundary get marked as processed and won't be touched again unless you explicitly request it.
The "historical" partition is a special construct. In the XMLA partition list, you'll see it named with a GUID suffix like fact_orders-historicaldata-bc6c9a7e.... This partition covers all data outside the rolling window and is framed once. If your historical data is truly immutable, this is perfect. If you occasionally need to restate historical data (corrections, late-arriving records from months ago), you'll need to explicitly trigger a refresh of the historical partition, which is expensive.
For tables with occasional historical corrections, consider a two-tier approach: keep a wider incremental window (6-12 months) so that late-arriving records for the previous few months are automatically picked up without touching the historical partition. Accept that corrections older than your rolling window require a planned, off-peak full historical refresh.
Hands-On Exercise
This exercise walks you through setting up incremental refresh on a fact_transactions table in a Direct Lake semantic model for a hypothetical financial services use case. You have 3 years of transaction data, roughly 800 million rows, and new transactions arriving daily. The business requirement is that the semantic model reflects previous-day transactions by 7 AM.
Step 1: Prepare the Delta Table
In a Fabric notebook attached to your gold lakehouse, write the following:
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_date, year, month, dayofmonth
from delta.tables import DeltaTable
spark = SparkSession.builder.getOrCreate()
GOLD_PATH = "abfss://prod-workspace@onelake.dfs.fabric.microsoft.com/gold.Lakehouse/Tables/fact_transactions"
# Read from silver (assume this exists and is well-formed)
SILVER_PATH = "abfss://prod-workspace@onelake.dfs.fabric.microsoft.com/silver.Lakehouse/Tables/transactions"
df = spark.read.format("delta").load(SILVER_PATH)
# Ensure partition columns exist
df = df.withColumn("transaction_date", to_date(col("created_at"))) \
.withColumn("partition_year", year(col("transaction_date")).cast("int")) \
.withColumn("partition_month", month(col("transaction_date")).cast("int"))
# Write with physical partitioning
(
df.write
.format("delta")
.mode("overwrite")
.partitionBy("partition_year", "partition_month")
.option("overwriteSchema", "true")
.option("delta.columnMapping.mode", "name")
.option("delta.minReaderVersion", "2")
.option("delta.minWriterVersion", "5")
.save(GOLD_PATH)
)
print("Initial write complete. Running OPTIMIZE on recent partitions...")
dt = DeltaTable.forPath(spark, GOLD_PATH)
current_year = 2025
# Optimize the last 3 months
for month_num in [10, 11, 12]:
dt.optimize().where(
f"partition_year = {current_year} AND partition_month = {month_num}"
).executeCompaction()
print(f"Optimized {current_year}-{month_num:02d}")
Step 2: Connect to the Semantic Model via SSMS
Open SSMS and connect to Analysis Services using your XMLA endpoint URL
Expand the database tree to find your Financial Reporting semantic model
Right-click the database and select "New Query > XMLA"
Step 3: Deploy the Refresh Policy
Paste and execute the following TMSL. Note the rolling window of 36 months (3 years to cover your full history) with a 2-month incremental hot zone:
Schedule this for off-peak hours. Monitor it in the SSMS "Activity Monitor" or via the Fabric Monitoring Hub.
Step 5: Verify Partition Structure
After the full refresh completes, run this DMV query in SSMS to inspect the resulting partitions:
SELECT
[ID],
[Name],
[Description],
[State],
[Type]
FROM $System.TMSCHEMA_PARTITIONS
WHERE [TableID] IN (
SELECT [ID] FROM $System.TMSCHEMA_TABLES
WHERE [Name] = 'fact_transactions'
)
ORDER BY [Name]
Create a Fabric notebook with the following and add it as a Notebook Activity in your daily pipeline:
import sempy.fabric as fabric
import json
from datetime import datetime, timedelta
def get_refresh_partitions(table_name: str, hot_periods: int = 2) -> list[str]:
"""
Returns the list of partition names that should be refreshed
based on the current date and the number of hot periods.
"""
partitions = []
today = datetime.utcnow()
for i in range(hot_periods):
# Work backwards from current month
target_date = today.replace(day=1) - timedelta(days=30 * i)
period_str = f"{target_date.year}{target_date.month:02d}"
partitions.append(f"{table_name}-{period_str}")
return partitions
workspace = "Data Platform - Production"
dataset = "Financial Reporting"
table = "fact_transactions"
refresh_targets = get_refresh_partitions(table, hot_periods=2)
refresh_command = {
"refresh": {
"type": "full",
"objects": [
{"database": dataset, "table": table, "partition": p}
for p in refresh_targets
]
}
}
print(f"Triggering refresh for: {refresh_targets}")
result = fabric.execute_xmla(
workspace=workspace,
dataset=dataset,
xmla_command=json.dumps(refresh_command)
)
print("Refresh complete.")
Common Mistakes and Troubleshooting
Mistake 1: Partition Column Not in the Delta Physical Layout
The most common failure is defining a refresh policy referencing a date column that exists in the table schema but is not used as the Delta physical partition column. The semantic model will create partitions based on the policy, but each partition will still scan the entire Delta table because the file layout doesn't match the partition filter.
How to diagnose: Check the Delta table's partition columns with:
If the output is [] or lists different columns than what your policy references, you need to rewrite the table with correct partitioning.
Mistake 2: Too Many Small Files in the Incremental Partition
If your daily pipeline appends rows to the Delta table without running OPTIMIZE, each day's load creates new Parquet files. After a month of daily loads, your current month's partition might have 30 small files instead of 1-3 large ones. Framing all 30 files takes longer and consumes more memory from the Analysis Services engine.
Fix: Always run delta_table.optimize().where(partition_filter).executeCompaction() on the current partition before triggering the XMLA refresh.
Mistake 3: XMLA Write Timeout on Large Partitions
When the historical partition covers years of data, the initial full refresh can time out through the XMLA connection if your client has a short timeout configured. The operation continues server-side, but your client disconnects.
Fix: Set a long timeout (CommandTimeout = 7200 for 2 hours) when connecting programmatically, and always verify completion via the DMV query from Step 5 of the exercise rather than relying on client-side confirmation.
Mistake 4: Refreshing the Historical Partition Accidentally
If you trigger a refresh command at the table level (not partition level) after the initial setup, you'll re-process the historical partition along with the hot-zone partitions. This defeats the entire purpose of incremental refresh.
Fix: Always target specific partitions by name in your TMSL refresh command after the initial full load. Verify partition names via the DMV query first.
Warning
Running a full table refresh ("type": "full" on the table object without specifying partitions) after an incremental refresh policy is deployed will re-process all partitions including the historical one. This is the equivalent of clearing your incremental refresh cache. For a 36-month historical partition on a large table, this can take hours.
Mistake 5: Policy Deployed but No Partitions Created
If you deploy the refresh policy but the partition structure doesn't change in the SSMS tree, it usually means the TMSL command succeeded but the subsequent framing operation was skipped because the model detected no change (the polling expression returned the same value).
Fix: Trigger an explicit full table refresh immediately after deploying the policy for the first time to force the partition structure to build out.
Mistake 6: Fallback to DirectQuery
Direct Lake models fall back to DirectQuery when the framing process encounters a condition it can't handle — such as too many Delta table versions in the transaction log, missing statistics, or a V-Order violation. When this happens, your partition refresh might "succeed" in the XMLA sense, but queries will be slower than expected.
How to detect: In the Fabric portal, your model's connection mode will show "DirectQuery" in the performance analyzer or through the model properties. Cross-reference with whether VACUUM has been run recently (too few transaction log entries is also a problem — very aggressive VACUUM can remove snapshots the engine was relying on).
Tip
Monitor your Fabric capacity usage during incremental refreshes using the Monitoring Hub. XMLA partition refresh operations consume CU seconds from your capacity, and if you're scheduling many partition refreshes simultaneously across multiple models, you can saturate the capacity and cause queuing delays.
Capacity and Scaling Considerations
Incremental refresh for Direct Lake is not free from a capacity standpoint, even though it's lighter than full Import-mode refresh. Each partition refresh consumes Analysis Services processing capacity. On an F64, you can typically process 4-6 partitions concurrently without impacting query performance significantly. On F128 and above, you have considerably more headroom.
For very large deployments — 10+ semantic models each with large fact tables — consider staggering your refresh schedules so that XMLA partition operations don't overlap. The Fabric capacity smoothing model (which averages CU consumption over 5-minute windows) helps with short bursts, but sustained XMLA processing at high concurrency will still be visible in your capacity metrics.
You now have a complete, production-grade understanding of how to implement incremental refresh for Direct Lake semantic models. The key mental model to carry forward: incremental refresh in Direct Lake is about reducing framing cost by scoping partition enumeration, not about moving less data. Your Delta table's physical partitioning must align with the Analysis Services partition boundaries defined in your TMDL refresh policy. The orchestration sequence — write to Delta, OPTIMIZE, then trigger XMLA refresh — must be treated as a strict dependency chain, not a loosely ordered set of steps.
The skills that compound on top of this lesson:
Deeper Delta optimization: Understanding V-Order and Z-Order and how they affect Direct Lake query performance after framing goes deeper into the storage layer than we covered here — revisit optimizing Delta table performance in a Fabric Lakehouse with fresh eyes now that you understand the framing relationship
Security on refreshed data: Row-level security applies at query time in Direct Lake, not at the partition level, but it interacts with partition structure in non-obvious ways — worth reviewing as you harden the production setup
Incremental refresh is one of those features that feels optional until you have a table with a few hundred million rows and a 2-hour refresh window. Building it correctly from the start, with aligned partitioning and explicit XMLA control, is the difference between a semantic model that scales to production and one that becomes a daily operational headache.
