When Power Query evaluates a complex pipeline, it decides what to refresh, when, and how many times — unless you architect it otherwise. This lesson teaches you how to take control of evaluation order, buffer volatile sources like APIs and SharePoint lists, and build pipelines that are deterministic and reliable in production.

Picture this: your Power BI report refreshes overnight, and by morning, the sales dashboard is showing mismatched totals. The transaction fact table loaded successfully, but the customer dimension it joins to pulled stale data from a cached source. The numbers don't reconcile, nobody can explain why, and the business meeting starts in forty minutes. You've been there, or you know someone who has.
The root cause in most of these scenarios isn't bad transformation logic — it's uncontrolled refresh dependency order and poorly isolated volatile sources. When Power Query evaluates a pipeline with a dozen interdependent queries, it makes its own decisions about what to evaluate, when, and how many times unless you deliberately architect it to do otherwise. The M engine is lazy and non-deterministic in ways that can surprise even experienced practitioners. A volatile source — a live API, a SharePoint list, a database view with session-level logic — evaluated at an arbitrary point in the refresh cycle can produce results that are inconsistent with queries evaluated moments earlier.
By the end of this lesson, you'll know exactly how to take control. You'll understand how the M engine builds and traverses dependency graphs, how to use staging queries and Table.Buffer to establish deterministic evaluation checkpoints, how to isolate volatile sources so they contribute exactly one read to the pipeline, and how to structure your query workspace so refresh order is something you design rather than something you discover after the fact.
What you'll learn:
Table.Buffer and connection-only queriesYou should be comfortable loading and transforming data in Power Query, and you should have a working understanding of how queries reference one another. If you haven't read about understanding query dependencies and evaluation order in Power Query, do that first — this lesson builds directly on those mechanics. Familiarity with the multi-stage staging architecture pattern will also help you place these techniques in context.
Before you can control evaluation order, you need to understand what drives it. The M engine doesn't execute queries sequentially from top to bottom in your query list. It builds a directed acyclic graph (DAG) of dependencies by analyzing which queries reference which other queries, then evaluates leaf nodes first and works inward toward the queries that are actually loaded to the data model.
Consider a simple three-query pipeline:
RawSales → CleanedSales → FactSales
When Power BI refreshes FactSales, the engine traces back through the dependency graph, identifies CleanedSales and then RawSales as prerequisites, and evaluates them in that order. So far so good — that's intuitive.
The trouble starts when you have a broader graph:
RawSales ──────────────┐
├──► FactSales
CleanedSales ──────────┘
│
└──► CustomerDim ──► FactSales
Now FactSales depends on both CleanedSales and CustomerDim. The engine may evaluate these branches in parallel or in an order you didn't anticipate. If CleanedSales pulls from a live database view and CustomerDim also pulls from the same view (or a related one), you have no guarantee that both branches see the same snapshot of the source data. This is the core of the volatile source problem.
Key insight: The M engine optimizes for performance, not for your mental model of the pipeline. It will fold queries to the source, combine steps, and evaluate branches opportunistically unless you explicitly tell it where to stop and materialize results.
There's a second issue layered on top of this: query folding. When a downstream query references an upstream query and both can fold to the same data source, the engine may collapse the entire dependency chain into a single, more complex query sent directly to the source. This is usually wonderful for performance, but it means your "staging" query isn't staging anything — it's a phantom that the engine erases in translation. We'll address this directly when we cover buffering.
Let's ground this in a realistic scenario. You're building a pipeline for a retail client with the following sources:
Orders table that is stable and folding-friendly/inventory/current endpoint that returns live stock levels, changes by the minute, and has rate limitsProductCatalog list managed by merchandising, updated irregularlyYour pipeline has several downstream consumers for the API data: a LowStockAlerts query, a ReplenishmentSuggestions query, and an InventorySnapshot query. Each of these references a CurrentInventory staging query that calls the API.
Here's what happens without proper isolation. Each downstream query that references CurrentInventory may cause the engine to re-evaluate it independently, because without buffering, CurrentInventory is just a recipe — not a cached result. The API gets called three times, potentially returning different results on each call (stock levels changed between calls), and you blow through your rate limit budget.
Warning: In Power Query, referencing an unbuffered query does not guarantee that query is evaluated only once. The M engine treats each reference as a fresh expression evaluation unless the result is explicitly materialized. This is the single most important thing to understand about volatile source isolation.
The fix has two parts: materializing the volatile source into memory exactly once, and ensuring all downstream consumers read from that materialized copy rather than re-querying the source.
Table.Buffer is your primary tool for creating evaluation checkpoints. When you wrap a table expression in Table.Buffer, you're telling the M engine: evaluate this expression completely, hold the result in memory, and serve all subsequent references from that in-memory copy.
Here's how CurrentInventory should be structured:
// CurrentInventory (connection-only — do not load to model)
let
Source = Json.Document(
Web.Contents(
"https://api.example.com/inventory/current",
[
Headers = [
Authorization = "Bearer " & InventoryApiKey,
#"Content-Type" = "application/json"
],
Timeout = #duration(0, 0, 0, 30)
]
)
),
AsTable = Table.FromRecords(
Record.ToList(Source[data]),
type table [
ProductId = text,
SKU = text,
StockOnHand = Int64.Type,
ReservedQty = Int64.Type,
LastUpdated = datetime
]
),
// This is the critical line — materialize here, once
Buffered = Table.Buffer(AsTable)
in
Buffered
With this in place, every downstream query that references CurrentInventory will read from Buffered. The API is called exactly once per refresh cycle.
Tip: Set
CurrentInventoryas a connection-only query (not loaded to the data model). Its job is to be a shared, buffered intermediate. Loading it to the model wastes memory and creates a redundant table that confuses report authors.
Now your downstream consumers look like this:
// LowStockAlerts
let
Source = CurrentInventory,
Filtered = Table.SelectRows(
Source,
each [StockOnHand] - [ReservedQty] < 10
),
AddedUrgencyFlag = Table.AddColumn(
Filtered,
"UrgencyLevel",
each if [StockOnHand] - [ReservedQty] = 0
then "Critical"
else "Low",
type text
)
in
AddedUrgencyFlag
// InventorySnapshot
let
Source = CurrentInventory,
AddedSnapshotDate = Table.AddColumn(
Source,
"SnapshotDate",
each DateTime.LocalNow(),
type datetime
),
Typed = Table.TransformColumnTypes(
AddedSnapshotDate,
{{"StockOnHand", Int64.Type}, {"ReservedQty", Int64.Type}}
)
in
Typed
Both queries reference CurrentInventory, but because it's buffered, neither triggers a second API call. You've effectively decoupled the source read from the transformation logic.
Table.Buffer is powerful but misunderstood. It only materializes the result when the engine actually evaluates that step — and that evaluation happens in the context of whatever query is being refreshed. If you have five queries that all reference CurrentInventory and those five queries load independently to the data model, each load triggers its own evaluation context, and Table.Buffer within CurrentInventory only prevents re-evaluation within a single evaluation context.
This is a crucial distinction. Let me illustrate:
Scenario A — One loaded query, multiple internal references:
FactSales references both LowStockAlerts and InventorySnapshot. When FactSales refreshes, the engine evaluates its dependency graph in one context. CurrentInventory is buffered once, both branches read from the buffer. ✅
Scenario B — Two separately loaded queries:
LowStockAlerts and InventorySnapshot both load to the data model as independent tables. Each refresh triggers its own evaluation context. CurrentInventory is evaluated once per context, so the API is called twice. ❌
The solution for Scenario B is to restructure so that the volatile source is consumed by a single loaded query that produces all necessary outputs, or — better — to reconsider whether both downstream tables genuinely need to be separate loads.
Key insight:
Table.Buffereliminates within-context re-evaluation. It does not coordinate across independent evaluation contexts (i.e., separately refreshed queries). If you have multiple loaded queries that trace back to the same volatile source, you need an architectural solution, not just a buffering solution.
For genuine cross-context isolation, the production-grade answer is to land the volatile source to a persistent store (a database staging table, a SharePoint list, an Azure Blob) as a separate scheduled process before the Power Query pipeline runs. Power Query is then pulling from a stable snapshot rather than a live source. This is particularly relevant when you're working with parameterized queries and dynamic data sources where the source endpoint itself might change.
The M engine resolves dependencies based on query references. If Query B references Query A, then A is guaranteed to be evaluated before B. You can exploit this to enforce a specific evaluation sequence even when there's no natural data dependency between two queries.
The pattern is called a dependency signal or phantom dependency. You create a calculated value from an upstream query — typically a row count, a maximum date, or a validation flag — and reference it in a downstream query as a guard condition.
Here's a concrete example. Suppose you need DimProduct to finish loading before FactSales begins, even though FactSales doesn't directly join to DimProduct in the Power Query pipeline (maybe that join happens in the data model):
// DimProduct (loaded to model)
let
Source = Sql.Database("prod-sql-01", "RetailDW"),
ProductTable = Source{[Schema="dbo", Item="Product"]}[Data],
Filtered = Table.SelectRows(ProductTable, each [IsActive] = true),
Renamed = Table.RenameColumns(Filtered, {
{"ProductKey", "ProductID"},
{"ProductName", "Name"},
{"CategoryCode", "Category"}
})
in
Renamed
// DimProduct_LoadSignal (connection-only — do not load)
let
// Force DimProduct to evaluate, capture a signal value
RowCount = Table.RowCount(DimProduct),
Signal = if RowCount >= 0 then true else error "DimProduct failed to load"
in
Signal
Now in FactSales:
// FactSales (loaded to model)
let
// Reference the signal to enforce evaluation order
_Guard = DimProduct_LoadSignal, // This forces DimProduct to evaluate first
Source = Sql.Database("prod-sql-01", "RetailDW"),
SalesTable = Source{[Schema="dbo", Item="SalesFact"]}[Data],
DateFiltered = Table.SelectRows(
SalesTable,
each [TransactionDate] >= #date(2024, 1, 1)
),
// The guard is used in a condition that's always true,
// but the engine must evaluate it to know that
GuardedResult = if _Guard then DateFiltered else error "Dependency failed"
in
GuardedResult
The engine cannot evaluate FactSales without first evaluating DimProduct_LoadSignal, which in turn requires DimProduct. You've encoded a load-order dependency without introducing circular references or changing the data itself.
Warning: Don't use
List.First({DimProduct})or other indirect tricks that the query optimizer might fold away. The guard pattern works becauseTable.RowCount(DimProduct)actually forces table materialization. Test your guard patterns with query diagnostics turned on to confirm the dependency is being respected.
Dependency management gets exponentially harder as your query count grows. Fifteen queries without a clear naming convention and grouping strategy become impossible to reason about. By the time you're troubleshooting a refresh failure at 6 AM, you want the dependency structure to be self-documenting.
A naming convention that mirrors the evaluation layers is your first defense. The staging architecture pattern maps naturally onto a query naming scheme:
| Prefix | Purpose | Load to model? |
|---|---|---|
src_ |
Raw source connection, minimal transformation | No |
stg_ |
Cleaned and typed intermediate | No |
buf_ |
Buffered volatile sources | No |
dim_ |
Dimension tables | Yes |
fact_ |
Fact tables | Yes |
sig_ |
Load order signal queries | No |
Using this convention for the inventory pipeline:
buf_CurrentInventory (connection-only, buffered API result)
src_Orders (connection-only, raw SQL read)
src_ProductCatalog (connection-only, raw SharePoint read)
stg_Orders (connection-only, cleaned Orders)
stg_ProductCatalog (connection-only, cleaned catalog)
sig_DimProduct (connection-only, signals DimProduct loaded)
dim_Product (loaded to model)
fact_Sales (loaded to model)
fact_LowStockAlerts (loaded to model)
fact_InventorySnapshot (loaded to model)
Power Query groups in the Queries pane reinforce this structure visually. Create groups called "Sources," "Staging," "Buffers," "Signals," and "Output," and drag queries into the appropriate group. This isn't decorative — it forces you to think about what layer each query belongs to, which surfaces architectural mistakes early.
Tip: Make a rule that any query in the "Signals" group is connection-only, contains no business logic, and references exactly one upstream query. If you find yourself adding transformation steps to a signal query, extract those steps back to the appropriate staging layer.
SharePoint lists are a perfect example of a volatile, unreliable source. They're edited by humans, subject to permission changes, throttled by the service, and prone to schema drift when someone adds a new column on Tuesday without telling IT. Let's build a proper isolation wrapper.
// src_ProductCatalog (connection-only)
let
Source = SharePoint.Tables(
"https://yourcompany.sharepoint.com/sites/Merchandising",
[ApiVersion = 15]
),
ProductList = Source{[Title = "ProductCatalog"]}[Items],
// Select only the columns we expect — guards against schema drift
SafeColumns = Table.SelectColumns(
ProductList,
{
"Title", // Product name
"SKU",
"CategoryCode",
"ListPrice",
"IsActive",
"EffectiveDate",
"Modified"
},
MissingField.Ignore // Don't error if a column is absent
)
in
SafeColumns
Notice MissingField.Ignore — this is your first line of defense against schema drift. It's discussed in depth in the lesson on handling dynamic schema changes, but the short version is: an unexpected missing column shouldn't take down your entire pipeline. You want to degrade gracefully.
The staging layer adds type enforcement and business cleaning:
// stg_ProductCatalog (connection-only)
let
Source = src_ProductCatalog,
// Enforce types explicitly — SharePoint returns everything as text
Typed = Table.TransformColumnTypes(Source, {
{"Title", type text},
{"SKU", type text},
{"CategoryCode", type text},
{"ListPrice", Currency.Type},
{"IsActive", type logical},
{"EffectiveDate", type date},
{"Modified", type datetime}
}),
// Drop inactive products
ActiveOnly = Table.SelectRows(Typed, each [IsActive] = true),
// Rename to model-friendly names
Renamed = Table.RenameColumns(ActiveOnly, {
{"Title", "ProductName"},
{"Modified", "LastModifiedDate"}
}),
// Buffer here — the SharePoint source is volatile and slow
Buffered = Table.Buffer(Renamed)
in
Buffered
The buffer happens at the staging layer, not the source layer. This is intentional: you want to apply cleaning and type coercion before materializing, so the in-memory copy is the clean version, not the raw SharePoint mess.
Now dim_Product simply reads from stg_ProductCatalog:
// dim_Product (loaded to model)
let
Source = stg_ProductCatalog,
// Add a surrogate key
WithIndex = Table.AddIndexColumn(Source, "ProductKey", 1, 1, Int64.Type),
// Reorder for readability
Reordered = Table.ReorderColumns(
WithIndex,
{"ProductKey", "SKU", "ProductName", "CategoryCode", "ListPrice", "EffectiveDate", "LastModifiedDate"}
)
in
Reordered
This separation means that if you ever need to change the SharePoint connection logic — different site, different list name, updated authentication — you change it in src_ProductCatalog and the rest of the pipeline is unaffected. If you need to change the business rules for what makes a product "active," you change stg_ProductCatalog. The performance implications of this buffering approach are worth reviewing — specifically, note that Table.Buffer prevents folding on everything downstream of it, so place buffers as far downstream as possible while still protecting the volatile source.
In a production pipeline, a volatile source failing should not necessarily cascade to a full refresh failure if you can substitute a fallback. The try...otherwise pattern in M gives you this capability.
// buf_CurrentInventory (connection-only)
let
FetchAttempt = try Json.Document(
Web.Contents(
"https://api.example.com/inventory/current",
[
Headers = [Authorization = "Bearer " & InventoryApiKey],
Timeout = #duration(0, 0, 0, 30)
]
)
),
// If the API fails, fall back to the previous day's snapshot from SQL
InventoryData = if FetchAttempt[HasError] then
let
Fallback = Sql.Database("prod-sql-01", "RetailDW"),
SnapTable = Fallback{[Schema="staging", Item="InventorySnapshot"]}[Data],
LatestDate = List.Max(SnapTable[SnapshotDate]),
LatestSnapshot = Table.SelectRows(SnapTable, each [SnapshotDate] = LatestDate)
in
LatestSnapshot
else
let
Parsed = Table.FromRecords(
Record.ToList(FetchAttempt[Value][data])
)
in
Parsed,
Buffered = Table.Buffer(InventoryData)
in
Buffered
Warning: When using
try...otherwisewithWeb.Contents, be aware that not all HTTP errors are caught the same way. Timeout errors are typically caught, but some credential or network errors may surface before thetrywrapper can catch them. Always test your fallback paths deliberately, not just theoretically. The lesson on debugging and error handling in M covers this in full detail.
The fallback pattern also illustrates why having a persistent snapshot mechanism matters. If your pipeline can't gracefully degrade to yesterday's data, any API blip causes a complete refresh failure. For pipelines that run in Power BI Service on a scheduled refresh, a full failure means stakeholders wake up to no data at all — which is often worse than slightly stale data.
When your refresh behavior doesn't match your expectations, Query Diagnostics is the tool that shows you what the engine actually did versus what you thought it would do.
To enable Query Diagnostics in Power Query Editor, go to the Tools menu and select Start Diagnostics. Run your refresh, then select Stop Diagnostics. This generates a detailed trace table showing every evaluation, its duration, whether it folded to the source, and how many times each step was evaluated.
Look specifically for:
Repeated source reads: If buf_CurrentInventory shows multiple calls to the API endpoint, your buffer isn't working as expected. This usually means the buffer is being defeated by the evaluation context problem described earlier — you have multiple independently loaded queries all tracing back to the same buffered source.
Unexpected folding: If a step you expected to evaluate in Power Query (like a Table.Buffer application) is folding back to SQL, the engine is collapsing your isolation layer. Check the DataSourceQuery column in the diagnostics output to see exactly what SQL is being generated.
Long-pole evaluations: The diagnostics output includes timing. If one source evaluation is taking 85% of your refresh time and it's being called multiple times, that's your optimization target.
Tip: Save your diagnostics results to an Excel file or database table before you start optimizing. It's easy to lose track of your baseline, and having a before/after comparison is the only reliable way to know whether your changes actually helped.
Build the following pipeline from scratch, then validate it using Query Diagnostics.
Scenario: You're connecting to two data sources — a SQL Server orders table (stable) and a live REST API that returns current exchange rates (volatile, rate-limited to 10 calls per minute). You need to produce three output tables:
FactOrders — orders with amounts in USDFactOrdersEUR — orders with amounts converted to EURExchangeRateLog — a snapshot of the rates used in this refresh cycleStep 1: Create a buf_ExchangeRates query using Table.Buffer that fetches from https://api.exchangerate-api.com/v4/latest/USD (a real, free endpoint) and materializes the result. Wrap it in try...otherwise that returns a hardcoded table with USD=1.0, EUR=0.92 if the API fails.
Step 2: Create a src_Orders connection-only query pointing to a SQL Server orders table, selecting only the columns you need.
Step 3: Create stg_Orders that cleans types, renames columns, and adds a CurrencyConversion join key.
Step 4: Create FactOrders by joining stg_Orders to buf_ExchangeRates to produce USD amounts.
Step 5: Create FactOrdersEUR by referencing buf_ExchangeRates for the EUR rate, but applying it to the same stg_Orders data.
Step 6: Enable Query Diagnostics before refreshing. After refresh, examine how many times the exchange rate API was called. It should be exactly once.
Validation: In the diagnostics output, you should see a single external call to the exchange rate API. Both FactOrders and FactOrdersEUR should show that they read from the buffered in-memory result, not from the live source.
If you see two API calls, Table.Buffer isn't materializing the way you expect. Check whether FactOrders and FactOrdersEUR are both loaded to the data model (they're separate evaluation contexts). If so, restructure so that a single loaded query produces both, or accept two API calls and monitor your rate limit budget.
Mistake 1: Buffering too early
Placing Table.Buffer on the raw source before type enforcement and cleaning means you're holding dirty data in memory. Worse, if the raw source returns different schemas on different days, the buffer materializes the bad schema before your cleaning steps can catch it. Buffer after cleaning, not before.
Mistake 2: Assuming connection-only means "not evaluated"
Setting a query to connection-only means it's not loaded to the data model as a separate table. It absolutely is evaluated when another query references it. Connection-only is a load destination setting, not an evaluation setting. Don't confuse the two.
Mistake 3: Creating circular signal dependencies
The guard pattern works with unidirectional signals. If FactSales signals DimProduct and DimProduct somehow signals FactSales, you have a circular reference that will cause the engine to error out. Always draw your dependency graph before implementing guard queries, even informally on paper.
Mistake 4: Forgetting that Table.Buffer defeats query folding
Once you buffer a table, the engine can no longer fold subsequent operations back to the SQL source. This is the price of isolation. If your SQL source is powerful and your transformation is complex, you might be better off doing more work in a SQL view or stored procedure before Power Query touches it — which is exactly the approach covered in connecting to SQL Server with native queries.
Mistake 5: Not testing fallback paths
The try...otherwise fallback looks great in code review and fails silently in production because nobody ever actually took the API down to test what happens. Build a test version of your query that forces the error condition (change the URL to something invalid) and verify the fallback path produces usable data.
Troubleshooting: Refresh succeeds locally, fails in Power BI Service
If your pipeline works in Power Query Editor but fails on scheduled refresh in Power BI Service, the most likely culprits are credential scope (a data source credential isn't available to the gateway), privacy level conflicts (a merge between a public source and a private source triggers the firewall), or timeout thresholds that are more aggressive in the service than in your local session. Check the privacy levels and credential management guide if you're dealing with firewall errors, and the scheduling and refresh failure diagnostics guide for everything else.
The discipline of dependency orchestration in Power Query separates pipelines that work in development from pipelines that stay reliable in production. The core ideas to carry forward:
Table.Buffer materializes results within a single evaluation context, eliminating within-context re-reads of volatile sourcesTable.Buffer does not coordinate across themTable.RowCount or similar expressions give you explicit control over cross-query evaluation ordersrc_, stg_, buf_, sig_, plus output prefixes) makes dependency structure self-documentingtry...otherwise wrappers on volatile sources enable graceful degradation rather than full pipeline failureThe patterns in this lesson compose naturally with incremental refresh strategies. Once you understand how to isolate a volatile source and buffer its result at a checkpoint, you're ready to think about automating incremental data refreshes with persistent state and change tracking — which takes the volatile source problem one step further by eliminating the need to re-read historical data entirely.
If you're building pipelines that are shared across a team, consider pairing these dependency patterns with a reusable function library approach so that your buffering and fallback wrappers are centralized and version-controlled rather than copy-pasted into every project.