Implementing Power BI Hybrid Tables with Incremental Refresh and Real-Time DirectQuery Partitions to Minimize Latency on Enterprise Premium Datasets

Introduction

Imagine you're the lead BI architect at a logistics company. Every morning, your warehouse operations team opens Power BI dashboards expecting to see shipment statuses updated to the minute — but instead they're looking at data that's twelve hours old. The underlying dataset is a 500-million-row fact table covering three years of order history. You can't import everything every hour; the refresh would run for forty minutes, hammer the source database, and still leave a gap. But switching the entire table to DirectQuery would kill query performance for the historical trend analysis that finance depends on.

This is the exact architectural tension that hybrid tables were designed to resolve. A hybrid table combines the best of both worlds: Import mode partitions hold months or years of historical data — fast, in-memory, compressed, cached — while a DirectQuery partition covers the most recent period with no latency at all. Users get sub-second performance on historical slices and real-time freshness on today's data, all from a single seamless table.

By the end of this lesson, you'll be able to design and implement a production-grade hybrid table strategy for large enterprise datasets. You'll understand the mechanics at the partition level, configure incremental refresh policies with real-time DirectQuery overlays, troubleshoot the common failure modes, and make architecture decisions that will hold up as your data volumes grow.

What you'll learn:

• How Power BI partitions work internally and why hybrid tables are a fundamentally different architecture than simple incremental refresh
• Step-by-step configuration of incremental refresh policies with the "Get the latest data in real time with DirectQuery" option enabled
• How the Analysis Services engine resolves queries across Import and DirectQuery partitions at runtime
• DAX and data source design considerations that dramatically affect hybrid table performance
• Operational patterns for monitoring, managing, and evolving hybrid table deployments in Premium environments

---

Prerequisites

Before diving in, you should already be comfortable with the following:

• Power BI Premium or Premium Per User (PPU) licensing — hybrid tables are not available on Pro-only workspaces
• Working knowledge of incremental refresh configuration in Power BI Desktop; if you need a refresher, the lesson on designing and implementing Power BI incremental refresh for large-scale enterprise datasets covers the foundational setup
• Familiarity with how Import and DirectQuery modes behave differently; the article on composite models and DirectQuery: when to use which in Power BI is a solid companion read
• Basic understanding of the XMLA endpoint and Tabular Object Model (TOM) for partition inspection
• Access to a Power BI Premium capacity (P1 or higher, or Premium Per User) and a data source that supports DirectQuery (Azure SQL Database, Synapse Analytics, SQL Server, etc.)

---

How Power BI Partitions Work Under the Hood

Before you can implement hybrid tables intelligently, you need to understand what's actually happening inside the Analysis Services engine that Power BI Premium runs on.

Every table in a Power BI dataset is composed of one or more partitions. In a standard Import table, there's typically a single partition containing all the data compressed in the VertiPaq columnar engine. When you enable incremental refresh, Power BI splits that single partition into multiple date-ranged partitions — a "historical" partition covering older periods and one or more "recent" partitions covering configurable rolling windows.

When you enable the real-time DirectQuery option on top of incremental refresh, Power BI adds one more partition: a DirectQuery partition with no data stored locally at all. This partition passes queries directly to the source system at query time. The result is a table that physically looks like this in the engine:

Table: SalesOrders
├── Partition: SalesOrders_2022 (Import, VertiPaq)
├── Partition: SalesOrders_2023 (Import, VertiPaq)
├── Partition: SalesOrders_2024_Q1 (Import, VertiPaq)
├── Partition: SalesOrders_2024_Q2 (Import, VertiPaq)
├── Partition: SalesOrders_2024_Q3 (Import, VertiPaq)
├── Partition: SalesOrders_2024_Oct (Import, VertiPaq)
├── Partition: SalesOrders_2024_Nov (Import, VertiPaq)
├── Partition: SalesOrders_DirectQuery (DirectQuery, no local data)

When a DAX query hits the table, the Analysis Services formula engine performs partition elimination — it inspects filter context to determine which partitions the query could possibly touch and routes accordingly. A query asking for last week's sales goes only to the DirectQuery partition. A query asking for Q2 2023 goes only to the corresponding Import partition. A query spanning several years hits a mix of both Import and DirectQuery partitions and the engine merges the results seamlessly.

Key insight: The formula engine's partition elimination logic is the secret weapon of hybrid tables. It's not magic — it's a deterministic, predicate-based filter on the OrderDate (or whatever your range key is) column. This means your date-range filters must be sargable — the engine needs to be able to statically determine partition membership from the filter predicates. If your DAX generates dynamic date calculations that the engine can't resolve at compile time, you'll accidentally fan out to all partitions including the DirectQuery one.

This architecture also explains why hybrid tables require Premium. The partition management APIs, the XMLA endpoint access for inspecting partition state, and the capacity resources needed to maintain multiple partition types simultaneously are all Premium features.

---

Designing Your Data Source for Hybrid Tables

The single most important thing you can do before touching Power BI Desktop is ensure your source database is structured to support efficient DirectQuery queries on your most recent data.

The DirectQuery partition will issue SQL queries against your source system every time a user's report touches the real-time period. In a busy enterprise environment, that could be hundreds of queries per hour. Here's what your source needs:

Indexing Strategy

Your date/datetime column — the same column you'll use as the RangeStart/RangeEnd filter in Power Query — must have a covering index or be part of a clustered columnstore index. For a table like dbo.SalesOrders in Azure SQL Database:

-- For row-store tables: non-clustered covering index on the date column
CREATE NONCLUSTERED INDEX IX_SalesOrders_OrderDateUTC
ON dbo.SalesOrders (OrderDateUTC)
INCLUDE (OrderID, CustomerID, ProductID, Quantity, UnitPrice, TotalAmount, StatusCode)
WHERE OrderDateUTC >= '2024-01-01'; -- Filtered index for recent data only

-- Or for high-volume OLAP scenarios: clustered columnstore
-- (columnstore handles range scans on datetime extremely well)
CREATE CLUSTERED COLUMNSTORE INDEX CCI_SalesOrders
ON dbo.SalesOrders;

The filtered index approach is particularly elegant for hybrid tables because it mirrors the partition boundary — the database index and the Power BI DirectQuery partition cover exactly the same date range.

View or Stored Procedure as Source

Rather than pointing Power BI directly at a raw transaction table, expose a view that enforces a consistent column set and data type contract:

CREATE VIEW dbo.vw_SalesOrders_Analytics AS
SELECT
    OrderID,
    CAST(OrderDateUTC AS DATE) AS OrderDate,      -- Consistent date type
    CAST(OrderDateUTC AS DATETIME2) AS OrderDateUTC,
    CustomerID,
    ProductID,
    SalesRegionID,
    Quantity,
    UnitPrice,
    TotalAmount,
    CAST(StatusCode AS VARCHAR(20)) AS StatusCode,
    CAST(CreatedBy AS NVARCHAR(100)) AS CreatedBy,
    LastModifiedUTC
FROM dbo.SalesOrders
WHERE IsDeleted = 0;  -- Enforce soft-delete filter at the source

This view ensures that when the incremental refresh M query runs (with RangeStart/RangeEnd filters folding down to the source), and when the DirectQuery partition issues real-time SQL, they both hit the same semantic object with the same data contract.

Warning: Be extremely careful with soft deletes. If a record is deleted (marked IsDeleted = 1) after it was already imported into an Import partition, that partition will not automatically reflect the change — it's already cached in VertiPaq. The DirectQuery partition will correctly exclude the record. This means for a brief period your historical Import partitions may show "ghost" records. Plan your partition refresh frequency accordingly, or handle soft deletes through a dedicated status dimension rather than exclusion.

---

Configuring Incremental Refresh with Real-Time DirectQuery in Power BI Desktop

Now let's walk through the actual implementation. We'll use a sales orders scenario with an Azure SQL Database source.

Step 1: Define RangeStart and RangeEnd Parameters

Open Power BI Desktop and navigate to the Power Query Editor via Transform Data. Create two parameters with these exact specifications:

• Name: RangeStart
  • Type: Date/Time
  • Current Value: 1/1/2022 12:00:00 AM (or your historical floor)
• Name: RangeEnd
  • Type: Date/Time
  • Current Value: 12/31/2024 11:59:59 PM (a recent date for development testing)

The names must be exactly RangeStart and RangeEnd — these are reserved parameter names that Power BI's incremental refresh engine looks for specifically.

Step 2: Write a Foldable M Query

Your Power Query expression must fold the date filter all the way to the source. If it doesn't fold, the engine downloads all data to Power BI and then filters in memory — completely defeating the purpose. Here's a properly constructed query:

let
    Source = Sql.Database(
        "yourserver.database.windows.net",
        "YourDatabase",
        [
            Query = null,
            CommandTimeout = null,
            CreateNavigationProperties = true
        ]
    ),
    SalesOrders = Source{[Schema="dbo", Item="vw_SalesOrders_Analytics"]}[Data],
    FilteredRows = Table.SelectRows(
        SalesOrders,
        each [OrderDateUTC] >= RangeStart and [OrderDateUTC] < RangeEnd
    )
in
    FilteredRows

The critical part is [OrderDateUTC] >= RangeStart and [OrderDateUTC] < RangeEnd. This is the exact predicate that Power BI's refresh engine will inject per-partition values into. Make sure you can verify query folding by right-clicking the last step in Power Query Editor and choosing "View Native Query" — you should see SQL with actual date predicates.

Tip: If you see "View Native Query" is greyed out, your query is not folding. Common causes include using Table.AddColumn or certain M functions before the filter step, or joining to a local table. Restructure the query so that all transformations happen after the filter, or push them into the source view.

Step 3: Load the Data and Open Incremental Refresh Settings

Close the Power Query Editor and apply your changes. In the table list in the Fields pane, right-click on your SalesOrders table and select Incremental refresh.

You'll see the incremental refresh configuration dialog. Here are the settings that matter most for a hybrid table configuration:

Archive data starting: Set this to how far back your full dataset goes. For our logistics example: 3 Years. This is the total span that will be maintained across all Import partitions.

Incrementally refresh data starting: Set this to the rolling window that gets refreshed on each cycle. For most enterprise scenarios: 7 Days to 30 Days depending on your source data latency. Going too short means more frequent full re-imports of the recent window; going too long means stale data in more recent Import partitions.

Get the latest data in real time with DirectQuery: Check this box. This is the hybrid table toggle. When enabled, Power BI will maintain an additional DirectQuery partition covering the period from the most recent Import partition boundary to "now."

You may also see an option to detect data changes — this is an optimization where Power BI checks a "last modified" watermark column to avoid re-importing unchanged partitions. For high-volume tables this is worth configuring:

• Column: Select your LastModifiedUTC column
• This causes Power BI to check the max value of this column in each partition range and skip re-import if nothing changed

Step 4: Publish to Premium Workspace

Local development with hybrid tables has a significant limitation: you cannot fully test hybrid table behavior in Power BI Desktop because the Desktop runtime doesn't manage multiple partitions. The hybrid behavior only activates after publishing to a Premium workspace.

Before publishing, verify your workspace is assigned to a Premium or PPU capacity. You can check this in the Power BI service by navigating to Workspace Settings — a diamond icon next to the workspace name confirms Premium assignment.

Publish the dataset. After publishing, the first refresh will be a full historical load, which can take substantial time depending on your data volume. A 500-million-row dataset over 3 years might take 30-45 minutes on a P1 capacity, depending on parallelism and source throughput.

---

Inspecting Partition State via the XMLA Endpoint

After the first refresh completes, you should verify that the partition structure was created correctly. The XMLA endpoint is your window into the actual engine state.

Connect to your Premium workspace via SQL Server Management Studio (SSMS) or Azure Data Studio using the XMLA endpoint URL (found in the workspace settings under Premium). Then run the following XMLA/DMV query:

SELECT
    [ID] AS PartitionName,
    [Name] AS PartitionDisplayName,
    [Mode] AS StorageMode,
    [State] AS PartitionState,
    [DataCoverageExpression],
    [RowsCount]
FROM $SYSTEM.TMSCHEMA_PARTITIONS
WHERE [TableID] IN (
    SELECT [ID] FROM $SYSTEM.TMSCHEMA_TABLES WHERE [Name] = 'SalesOrders'
)
ORDER BY [Name];

You should see output with multiple Import partitions and one DirectQuery partition. The DirectQuery partition will have StorageMode = 2 (DirectQuery) and RowsCount = 0 (no cached data). The Import partitions will have StorageMode = 1 (Import) and actual row counts.

If the DirectQuery partition is missing, the hybrid configuration didn't apply — most often because the workspace is not on Premium capacity or the dataset was published without the checkbox enabled.

Note: The XMLA endpoint also allows you to manually trigger partition-level refreshes through Tabular Editor or custom TMSL scripts. This is valuable for scenarios where you need to re-process a specific historical partition (e.g., after a source data correction) without triggering a full dataset refresh. The lesson on implementing the Power BI XMLA endpoint for advanced dataset management covers this in depth.

---

How Query Execution Actually Works at Runtime

Understanding the runtime execution model is essential for diagnosing performance problems and writing DAX that performs well against hybrid tables.

Partition Elimination in Practice

When a user opens a report page showing "This Month's Revenue," the formula engine evaluates the DAX, determines the filter context includes a date range that falls within the DirectQuery partition's coverage period, and generates a SQL query against the source. The user sees data current to within seconds.

When the same user navigates to a page showing "Year-over-Year Revenue — Last 3 Years," the formula engine determines the filter spans multiple Import partitions and the DirectQuery partition. It:

1. Scans the Import partitions using VertiPaq storage engine (in-memory, fast)
2. Issues a SQL query to the source for the DirectQuery partition's date range
3. Merges the results from all partitions in the formula engine
4. Returns the unified result set to the report

This merge step is where things can get complicated. The formula engine handles it gracefully for simple aggregations (SUM, COUNT, AVERAGE), but for complex DAX patterns — particularly those involving CALCULATE with table filters, USERELATIONSHIP, or RANKX — the behavior can surprise you.

DAX Patterns That Cause Problems

Time intelligence functions are the most common source of unexpected DirectQuery fan-out. Consider this measure:

Revenue YTD = 
TOTALYTD(
    [Total Revenue],
    'Date'[Date]
)

TOTALYTD expands to a filter from January 1 of the current year to the current filter context date. If the current date is in November 2024, this filter spans from January 2024 through November 2024 — crossing potentially several Import partition boundaries AND the DirectQuery partition. The formula engine will correctly handle this, but it means the DirectQuery partition gets hit even for "historical" YTD calculations that you might have expected to stay entirely in Import.

CALCULATE with row-level filters can also cause unexpected DirectQuery hits:

-- This looks innocent but can cause DirectQuery fan-out
High Value Orders = 
CALCULATE(
    [Order Count],
    SalesOrders[TotalAmount] > 10000
)

If the engine can't push TotalAmount > 10000 into the DirectQuery SQL (e.g., because TotalAmount is a calculated column rather than a native column), it may materialize more data from DirectQuery than you expect.

Key insight: Write DAX measures that are partition-aware. If you know a measure should only ever query historical data, add an explicit date filter that keeps it within the Import partition range. For example, CALCULATE([Revenue], DATESBETWEEN('Date'[Date], DATE(2020,1,1), DATE(2023,12,31))) will never touch the DirectQuery partition, keeping it fast and source-friendly.

The DirectQuery SQL Generated

To understand what SQL the DirectQuery partition generates, enable the monitoring Power BI performance with Premium Metrics app and examine query traces. You can also use SQL Server Profiler or Azure SQL's Query Store to capture the incoming SQL from Power BI.

A typical DirectQuery query generated by a hybrid table might look like:

SELECT
    CAST([OrderDate] AS DATE) AS [c1],
    SUM([TotalAmount]) AS [c2],
    COUNT(*) AS [c3]
FROM [dbo].[vw_SalesOrders_Analytics]
WHERE
    [OrderDateUTC] >= '2024-11-01T00:00:00'
    AND [OrderDateUTC] < GETUTCDATE()
GROUP BY
    CAST([OrderDate] AS DATE)

Notice the upper bound uses a dynamic expression (GETUTCDATE() or similar) — this is how the DirectQuery partition always captures "everything after the last Import partition's cutoff." This also means the partition boundary is a soft concept at query time; the engine dynamically determines what "now" means for each query execution.

---

Gateway Configuration for Hybrid Tables

If your data source is on-premises (SQL Server, Oracle, etc.) rather than in Azure, you need an on-premises data gateway. Hybrid tables impose specific requirements on gateway configuration that differ from standard Import refresh.

Why Hybrid Tables Are Different

With a standard Import dataset, the gateway only fires during scheduled refresh windows. With a hybrid table, the DirectQuery partition routes queries through the gateway every time a user interacts with data in the real-time period. This transforms the gateway from a batch worker to a continuously active query relay.

For a team of 50 users with reports open all day, you might see thousands of DirectQuery queries per hour flowing through the gateway. A single-node gateway installation that's fine for batch refresh will buckle under this load.

Warning: Never use the Personal Gateway for hybrid table deployments. It's not designed for concurrent DirectQuery load. You need the standard On-Premises Data Gateway in enterprise mode with multiple nodes in a cluster for resilience and load distribution. The comparison article on choosing the right refresh architecture elaborates on the architecture trade-offs.

For an enterprise hybrid table deployment with significant user load, configure your gateway cluster with:

• Minimum 2 nodes for high availability (4 recommended for heavy DirectQuery load)
• Dedicated VMs with 16+ cores and 64+ GB RAM per node
• Network latency under 5ms to the data source (co-locate with your database if possible)
• Gateway service account with read-only access to the source view

Connection Pooling

The gateway maintains connection pools to your data source. For DirectQuery-heavy workloads, ensure the source database's max connections limit is set high enough to accommodate gateway pool demands:

-- For Azure SQL Database, check current connection limits
SELECT
    databasename = DB_NAME(),
    connection_limit = (
        SELECT TOP 1 value_in_use 
        FROM sys.configurations 
        WHERE name = 'max connections'
    )

For Synapse Analytics or Azure SQL with a heavy hybrid table DirectQuery load, configure the gateway's Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config to tune connection pool size — though for most enterprise scenarios the defaults handle up to 200 concurrent connections per node adequately.

---

Refresh Scheduling and Partition Lifecycle Management

The hybrid table partition lifecycle is more nuanced than standard incremental refresh because you now have two fundamentally different partition types evolving over time.

The Rolling Window Mechanism

Each time a scheduled refresh runs, Power BI's refresh engine:

1. Evaluates the current date
2. Determines which Import partitions fall within the "incrementally refresh" window (the shorter rolling period you configured)
3. Re-imports those partitions from the source
4. Evaluates the "archive" period and drops any Import partitions older than the cutoff
5. Leaves the DirectQuery partition unchanged (it needs no refresh — it queries live)

This means the DirectQuery partition's effective coverage changes with each refresh. Before a refresh, it covers (say) the last 8 days. After the refresh re-imports the last 7 days from Import, the DirectQuery partition still covers from the Import boundary to now — but since Import has now been updated to yesterday, the DirectQuery partition shrinks back to approximately the last 24 hours of gap.

This sliding window behavior is actually elegant: you always have real-time data in DirectQuery, and the most recent "settled" data gets promoted into Import regularly.

Refresh Frequency Considerations

How often should you schedule refresh? The answer depends on your latency requirements and source database capacity:

Scenario	Recommended Refresh Frequency	Notes
Financial reporting, daily close	Once daily (overnight)	DirectQuery covers the current trading day
Operations dashboard	Every 30 minutes	Minimizes Import/DirectQuery overlap period
Near-real-time IOT analytics	Every 15 minutes (with XMLA)	Requires custom XMLA refresh scripting
Regulatory reporting (end-of-day)	Once daily	Longer DirectQuery window is acceptable

For the highest-frequency refresh scenarios, scheduled refresh through the Power BI service UI is limited to 48 refreshes per day (every 30 minutes). To go faster, you need to trigger refreshes programmatically via the Power BI REST API:

import requests
import json

def trigger_dataset_refresh(workspace_id, dataset_id, access_token):
    url = f"https://api.powerbi.com/v1.0/myorg/groups/{workspace_id}/datasets/{dataset_id}/refreshes"
    headers = {
        "Authorization": f"Bearer {access_token}",
        "Content-Type": "application/json"
    }
    body = {
        "notifyOption": "MailOnFailure",
        "retryCount": 2,
        "type": "Full"   # For incremental refresh, "Full" still only processes incremental partitions
    }
    response = requests.post(url, headers=headers, json=body)
    return response.status_code, response.json() if response.content else {}

Tip: When triggering refreshes via the REST API for hybrid table datasets, the "type": "Full" parameter doesn't override your incremental refresh policy — the policy always takes precedence. "Full" simply means "evaluate all partitions according to policy" rather than a subset. This is often counterintuitive coming from standard dataset refresh patterns.

---

Advanced Patterns: Aggregations with Hybrid Tables

Hybrid tables pair exceptionally well with Power BI Aggregations to create a three-tier performance architecture:

• Tier 1: Pre-aggregated Import partition (fastest — answers summary queries in milliseconds)
• Tier 2: Detailed Import partitions (fast — in-memory VertiPaq for historical detail)
• Tier 3: DirectQuery partition (live — real-time data, slightly higher latency)

The article on implementing Power BI aggregations to optimize query performance on billion-row datasets covers aggregation configuration in detail. Here's how it integrates with hybrid tables:

You create an aggregation table (SalesOrders_Agg) containing pre-aggregated summaries by day and product category, stored entirely in Import mode. When users query high-level KPIs like "Total Revenue by Month," the formula engine hits the aggregation table — bypassing both the Import detail partitions AND the DirectQuery partition entirely.

The DirectQuery partition only fires when users drill into individual order details for today's data, where no aggregation exists yet. This pattern reduces DirectQuery query volume by 80-90% for typical executive dashboard usage patterns.

---

Security Considerations with Hybrid Tables

Row-level security (RLS) behaves differently across Import and DirectQuery partitions, and this difference can have significant security implications.

RLS on Import Partitions

For Import partitions, RLS filters are applied in the formula engine after data is loaded from VertiPaq. The security filter is a DAX expression evaluated against cached data:

// RLS role: Sales Region Filter
[SalesRegionID] = LOOKUPVALUE(
    UserRegionMapping[RegionID],
    UserRegionMapping[UserEmail], USERPRINCIPALNAME()
)

RLS on the DirectQuery Partition

For the DirectQuery partition, Power BI translates the RLS DAX filter into a SQL predicate that's appended to every DirectQuery query. This means the database executes the filter — the data is never transmitted to Power BI unfiltered.

This is actually a security improvement for your real-time data: sensitive rows never leave the database. However, it also means your RLS expression must be translatable to SQL. Complex DAX involving RELATED(), multi-table lookups, or DAX-only functions may not fold correctly to DirectQuery.

Warning: Always verify your RLS rules work correctly against the DirectQuery partition specifically. Test by impersonating users with restricted roles in the Power BI service (not just Desktop) and checking that the DirectQuery SQL (visible in database query logs) contains the expected WHERE clause. A broken RLS translation that silently fails to apply the filter on the DirectQuery partition is a serious data security risk.

The complete treatment of row-level security in Power BI covers testing and validation patterns in detail.

---

Monitoring Hybrid Table Performance in Production

Once your hybrid table is live, you need to monitor two distinct performance surfaces: Import partition refresh health, and DirectQuery query latency.

Refresh Health Monitoring

Configure refresh failure alerts in the Power BI service — if an incremental refresh fails partway through, you may end up with Import partitions that have stale data while the DirectQuery partition covers a larger-than-expected gap. The article on implementing scheduled refresh and refresh failure alerting shows how to set up proactive alerting.

Use the Premium Metrics app to track:

• CPU time per refresh: Should be proportional to the number of partitions refreshed (not the total dataset size)
• Memory usage during refresh: Each partition is processed in-memory; monitor for memory pressure spikes
• Refresh duration trend: A suddenly longer refresh often indicates the incremental window is widening or source query performance degraded

DirectQuery Latency Monitoring

In your source database, set up a lightweight monitoring query that captures DirectQuery patterns:

-- Azure SQL Database: Monitor Power BI DirectQuery patterns
SELECT TOP 100
    qs.execution_count,
    qs.total_elapsed_time / qs.execution_count AS avg_elapsed_ms,
    qs.max_elapsed_time AS max_elapsed_ms,
    SUBSTRING(qt.text, (qs.statement_start_offset/2)+1,
        ((CASE qs.statement_end_offset
            WHEN -1 THEN DATALENGTH(qt.text)
            ELSE qs.statement_end_offset
        END - qs.statement_start_offset)/2)+1) AS query_text,
    qs.last_execution_time
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) qt
WHERE
    qt.text LIKE '%vw_SalesOrders_Analytics%'
    AND qs.last_execution_time > DATEADD(HOUR, -1, GETUTCDATE())
ORDER BY qs.total_elapsed_time DESC;

If average DirectQuery latency exceeds 2-3 seconds, your report users will feel it as sluggishness. Target under 500ms for a good experience, which requires appropriate indexing on the source and a low-latency network path from gateway to database.

---

Hands-On Exercise

This exercise will take you from a flat Import table to a fully functional hybrid table with real-time DirectQuery coverage. You'll need a Power BI Premium or PPU workspace and a database with at least a few months of date-ranged data.

Setup: Create a Sample Source Table

If you don't have a suitable source, create one in Azure SQL Database:

-- Create a reasonably sized fact table for testing
CREATE TABLE dbo.SalesTransactions (
    TransactionID BIGINT IDENTITY(1,1) PRIMARY KEY,
    OrderDateUTC DATETIME2 NOT NULL,
    CustomerID INT NOT NULL,
    ProductID INT NOT NULL,
    RegionCode NVARCHAR(10) NOT NULL,
    Quantity INT NOT NULL,
    UnitPrice DECIMAL(10,2) NOT NULL,
    TotalAmount AS (CAST(Quantity AS DECIMAL(10,2)) * UnitPrice) PERSISTED,
    StatusCode NVARCHAR(20) NOT NULL DEFAULT 'COMPLETED',
    LastModifiedUTC DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME(),
    IsDeleted BIT NOT NULL DEFAULT 0
);

-- Create index to support DirectQuery and incremental refresh
CREATE NONCLUSTERED INDEX IX_SalesTransactions_OrderDateUTC
ON dbo.SalesTransactions (OrderDateUTC)
INCLUDE (CustomerID, ProductID, RegionCode, Quantity, UnitPrice, TotalAmount, StatusCode);

-- Populate with 2 years of data (approximately 1M rows)
WITH DateSeries AS (
    SELECT DATEADD(MINUTE, n.n, '2023-01-01') AS TxDateTime
    FROM (
        SELECT TOP 1000000 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) - 1 AS n
        FROM sys.all_objects a CROSS JOIN sys.all_objects b
    ) n
)
INSERT INTO dbo.SalesTransactions (OrderDateUTC, CustomerID, ProductID, RegionCode, Quantity, UnitPrice, StatusCode)
SELECT
    TxDateTime,
    ABS(CHECKSUM(NEWID())) % 10000 + 1,
    ABS(CHECKSUM(NEWID())) % 500 + 1,
    CHOOSE(ABS(CHECKSUM(NEWID())) % 5 + 1, 'NORTH','SOUTH','EAST','WEST','CENTRAL'),
    ABS(CHECKSUM(NEWID())) % 10 + 1,
    CAST(ABS(CHECKSUM(NEWID())) % 990 + 10 AS DECIMAL(10,2)),
    'COMPLETED'
FROM DateSeries;

Exercise Steps

Step 1: Connect Power BI Desktop to dbo.SalesTransactions using the SQL Server connector. In the Power Query Editor, verify query folding works by right-clicking "Applied Steps" and checking Native Query.

Step 2: Create the RangeStart (Date/Time, value: 1/1/2023) and RangeEnd (Date/Time, value: 12/31/2024) parameters. Modify the M query to filter OrderDateUTC using these parameters.

Step 3: Add a simple calculated column for OrderDate as Date.From([OrderDateUTC]) to give yourself a clean date column for the date table relationship.

Step 4: Right-click the table in the Fields pane and open Incremental Refresh. Configure:

• Archive: 2 years
• Refresh incrementally: 7 days
• Enable "Get the latest data in real time with DirectQuery"
• Enable "Only refresh complete days" if your data has a clear daily close pattern

Step 5: Publish to your Premium workspace and trigger the first refresh. Monitor it in the Refresh History view.

Step 6: After refresh completes, connect to the XMLA endpoint in SSMS and run the partition DMV query from earlier. Confirm you see both Import and DirectQuery partitions.

Step 7: Build a simple report with two pages:

• Page 1: "Today's Transactions" — a card showing total count of transactions with OrderDate = TODAY()
• Page 2: "Historical Trend" — a line chart showing monthly transaction counts over the full 2-year period

On Page 1, observe that the number updates within seconds of inserting new rows in the source table (the DirectQuery partition picks them up immediately). On Page 2, observe the fast query performance despite the large historical dataset (Import partitions).

---

Common Mistakes & Troubleshooting

Mistake 1: The DirectQuery Partition Doesn't Appear After Publishing

Symptom: XMLA inspection shows only Import partitions; no DirectQuery partition exists.

Causes and fixes:

• The workspace is not on Premium or PPU capacity — verify and move the workspace
• The dataset was published before the checkbox was enabled — re-enable in Desktop and republish
• The Power Query query doesn't fold — fix the M query for proper folding, then republish

Mistake 2: Reports Show a Gap or Duplicate Data at the Partition Boundary

Symptom: Data for the current day appears double-counted, or there's a missing hour of data at the boundary between Import and DirectQuery.

Cause: The partition boundary is defined by RangeEnd which is set to "now" relative to the last refresh. If the refresh runs at 2:00 PM, the last Import partition covers through approximately 2:00 PM. The DirectQuery partition covers from that point forward. But if a report user has a date filter that rounds to "today" (a full calendar day), they may aggregate across both partitions, summing data correctly — or if the boundary calculation has a timezone mismatch, you get a gap or overlap.

Fix: Ensure all datetime handling uses a consistent timezone, preferably UTC throughout. Convert to local timezone only at the report/measure level, never at the M query or partition boundary level.

Mistake 3: DirectQuery Queries Are Extremely Slow

Symptom: Reports that show today's data take 10+ seconds to load.

Diagnosis: Check source database query logs. If the SQL generated by Power BI does full table scans, your index strategy is inadequate for the DirectQuery pattern.

Fix: Add or optimize covering indexes on OrderDateUTC. If using Azure SQL, consider enabling the Query Store and Automatic Plan Correction to prevent plan regressions caused by Power BI's parameterized query patterns.

Mistake 4: Incremental Refresh Is Reimporting All Historical Partitions

Symptom: Every scheduled refresh takes as long as a full refresh.

Cause: The RangeStart/RangeEnd filter isn't folding, so Power BI can't determine partition boundaries from the query. All partitions are marked for re-import.

Fix: Verify query folding in Desktop before publishing. The easiest diagnostic is to check the refresh history in the service and look at "Rows refreshed per partition" — if all partitions show row counts greater than zero, they're all being processed.

Mistake 5: RLS Doesn't Apply to the DirectQuery Partition

Symptom: Users in restricted RLS roles can see data from the current day that they shouldn't.

Cause: The RLS DAX expression contains functions that don't translate to SQL (e.g., LOOKUPVALUE against a local table that isn't exposed to DirectQuery).

Fix: Redesign the RLS filter to use simpler expressions that fold to SQL, or replicate the user-mapping logic in the source database as a security view, then reference the view in the RLS expression.

---

Summary & Next Steps

Hybrid tables are one of the most architecturally sophisticated features in the Power BI Premium ecosystem. The core insight is elegant: data that's old enough to be "settled" gets imported and compressed into fast VertiPaq partitions, while data that's actively changing gets served live from the source through a DirectQuery partition — and the formula engine merges the two transparently for users.

Getting it right requires attention at every layer:

• Source design: Proper indexing, views with consistent contracts, soft-delete handling
• Power Query: Foldable M expressions that the engine can parameterize per-partition
• Power BI config: Correct incremental refresh policy with the DirectQuery toggle enabled
• Gateway: Cluster sizing for concurrent DirectQuery load, not just batch refresh throughput
• DAX: Measures that respect partition boundaries and avoid unnecessary DirectQuery fan-out
• Monitoring: Separate tracking of Import refresh health and DirectQuery latency

The business result — genuinely real-time data in a dataset that also handles years of historical analysis efficiently — is worth the implementation complexity.

Where to go from here:

• Explore implementing large format datasets with hybrid tables for scenarios where your datasets exceed 10GB and need large dataset storage format enabled alongside hybrid tables
• Study implementing Power BI query scale-out with read-only replicas to learn how to distribute the query load across multiple replicas — especially useful when hybrid tables drive high concurrent DirectQuery usage
• Examine monitoring Power BI Premium performance with Premium Metrics to build operational dashboards that surface partition-level refresh and query performance details
• Review implementing Power BI dataset certification and endorsement workflows to ensure your hybrid table datasets are discoverable and trusted across the enterprise as authoritative data sources

Hybrid tables are a production commitment, not a feature to enable and forget. Invest in monitoring, document your partition boundaries clearly for your team, and build your DAX patterns with explicit awareness of which partitions each measure will touch. Do that, and you'll have an analytics platform that keeps both your data engineers and your business users genuinely happy.