Using OneLake Shortcuts to Query Data in ADLS Gen2 and Amazon S3 Without Copying It
OneLake Shortcuts let you query data in ADLS Gen2 and Amazon S3 directly from Microsoft Fabric — no copying, no pipelines, no duplication. This deep-dive lesson covers the internal architecture, credential configuration, Spark and T-SQL querying patterns, performance optimization, and the security model you must understand before deploying shortcuts in production.
🔥 Expert29 min readSep 22, 2026Updated Sep 22, 2026
Using OneLake Shortcuts to Query Data in ADLS Gen2 and Amazon S3 Without Copying It
Introduction
Here's a scenario that plays out constantly in large organizations: your data engineering team has spent years building a carefully managed Azure Data Lake Storage Gen2 environment. Petabytes of Parquet and Delta files, governed by fine-grained ACLs, organized into bronze/silver/gold zones. Now the business wants to start using Microsoft Fabric — specifically, they want to run Spark notebooks against that data, expose it to Power BI through Direct Lake, and join it with new data landing in OneLake. The instinctive answer is "copy everything into Fabric." But that answer is expensive, operationally risky, and almost always wrong.
OneLake Shortcuts change the equation entirely. A shortcut is a symbolic link — a pointer inside OneLake's filesystem that transparently redirects reads to an external storage location, whether that's ADLS Gen2, Amazon S3, Google Cloud Storage, or another OneLake location. From the perspective of any Fabric compute engine — Spark, the SQL Analytics Endpoint, a Dataflow Gen2 — the data appears to live inside your lakehouse. No data movement. No duplication. No synchronization jobs to babysit. The external storage remains the authoritative source, and Fabric reads from it as needed.
By the end of this lesson, you'll understand exactly how shortcuts work at the filesystem level, how to create and configure them securely for ADLS Gen2 and Amazon S3, how to query shortcut-backed data using Spark and T-SQL, and where shortcuts break down so you can avoid the pitfalls before they cost you.
What you'll learn:
The internal architecture of OneLake shortcuts — how the namespace indirection works and why it matters for performance
How to create shortcuts pointing to ADLS Gen2 containers and Amazon S3 buckets, including credential configuration
How Fabric compute engines (Spark, SQL Analytics Endpoint) resolve and execute queries against shortcut data
Performance characteristics and optimization strategies for shortcut-backed tables vs. native OneLake data
Security model for shortcuts: what credentials are used, who can see what, and where cross-tenant access gets complicated
Either an ADLS Gen2 storage account with some Parquet or Delta data in it, or an AWS S3 bucket — or both
For ADLS Gen2: either an Azure AD service principal with Storage Blob Data Reader on the relevant container, or account key
For S3: an AWS IAM access key pair with s3:GetObject and s3:ListBucket on the target bucket
How OneLake Shortcuts Work Under the Hood
Before touching the UI, you need a mental model of what a shortcut actually is at the filesystem layer, because that model will explain almost every behavior you'll see — including the confusing ones.
OneLake is built on top of Azure Data Lake Storage Gen2 under the hood, but it exposes a unified namespace through the OneLake service layer rather than giving you direct ADLS access. Every Fabric item — lakehouses, warehouses, semantic models — lives in a hierarchy that looks like this:
When you create a shortcut inside Files/ or Tables/, you're registering a mapping in the OneLake metadata service. The mapping says: "Any read request for onelake://.../Tables/sales_history/ should be resolved by fetching from abfss://raw@contosodatalake.dfs.core.windows.net/sales/history/." The data never moves. The OneLake service stores only the pointer definition and the encrypted credentials needed to authenticate to the external store.
When Spark or the SQL endpoint executes a query that touches a shortcut path, the compute engine resolves the path through OneLake's metadata layer first. OneLake returns either a redirect URL or proxies the data through its own I/O path, depending on the source type and the specific Fabric workload. For ADLS Gen2 to OneLake reads, when both are in the same Azure region, the I/O path is highly optimized — Azure network fabric handles this as intra-datacenter traffic, which is effectively free from a cost perspective and fast from a latency perspective.
Key insight
Shortcuts don't just save you the cost of copying data. They preserve your existing governance model. If you apply a delete on the source ADLS Gen2 blob, it's immediately reflected through the shortcut. There's no stale copy to worry about. This is fundamentally different from ingestion patterns using Fabric Data Pipelines where you're explicitly moving bytes.
For Amazon S3, the architecture adds latency because reads cross from Azure to AWS infrastructure. The OneLake service issues authenticated requests to S3 on behalf of the Fabric compute engine. You're going cross-cloud, which means you're subject to internet egress from AWS (which AWS does charge for), latency in the tens to hundreds of milliseconds per request, and the usual reliability characteristics of cross-cloud networking.
Understanding this topology is critical for making good architectural decisions. Shortcuts to ADLS Gen2 in the same region as your Fabric capacity are almost free. Shortcuts to S3 in a region geographically close to your Azure region are workable for moderate query volumes. Shortcuts to S3 in a distant region on queries scanning hundreds of gigabytes will be painfully slow and cost real money in AWS egress fees.
Creating an ADLS Gen2 Shortcut
Let's walk through creating a shortcut to an ADLS Gen2 container. We'll use a realistic scenario: a retail organization that has a transactions Delta table living in an existing data lake, stored at abfss://silver@retailcorp-lake.dfs.core.windows.net/transactions/. The Fabric team wants to query this data in a lakehouse alongside new data flowing in from Dataflow Gen2.
Step 1: Open Your Lakehouse and Start the Shortcut Wizard
Navigate to your Fabric workspace and open the lakehouse where you want the shortcut to live. In the Explorer pane on the left, you'll see two top-level nodes: Files and Tables. Right-click on Tables and select New shortcut.
Alternatively, use the Get data menu in the ribbon and select New shortcut from there. Both routes take you to the same shortcut creation dialog.
Note
You can create shortcuts under both Files/ and Tables/. The distinction matters: shortcuts under Tables/ are treated as Delta tables by the SQL Analytics Endpoint and will appear in the lakehouse schema as queryable tables. Shortcuts under Files/ are accessible via Spark and the Files API, but won't automatically appear as SQL tables. If your external data is a proper Delta table (with a _delta_log/ directory), always create the shortcut under Tables/ so the SQL endpoint picks it up.
Step 2: Choose the Source Type
The shortcut wizard presents source options: Microsoft OneLake, Azure Data Lake Storage Gen2, Amazon S3, Google Cloud Storage, and Dataverse. Select Azure Data Lake Storage Gen2.
Step 3: Configure the Connection
You'll be prompted to configure a connection. Fabric stores connections as reusable objects that can be shared across multiple shortcuts. The fields you need to fill in:
URL: This is the DFS endpoint of your storage account. Format: https://<storageaccount>.dfs.core.windows.net
For our example: https://retailcorplake.dfs.core.windows.net
Authentication kind: You have three options:
Organizational account (Azure AD): Uses your own identity delegated to Fabric. Clean from a governance perspective, but the identity doing the shortcut creation needs Storage Blob Data Reader at minimum, and the shortcut will fail for other users if they don't also have access to the underlying ADLS resource. This model works well inside a single organization where Azure AD is the identity provider for both Fabric and ADLS.
Service Principal: Fabric authenticates to ADLS using a registered Azure AD application. You provide the tenant ID, client ID, and client secret. This is the most common production choice because the service principal identity can be tightly scoped, the credentials rotate independently of user accounts, and access persists regardless of which user runs a query in Fabric.
Account Key: The ADLS Gen2 storage account key. Simple, but it grants full access to the entire storage account and has no expiration unless you rotate it manually. Only use this for development or when you have no other choice.
For production, configure a Service Principal. Here's what that looks like:
The service principal needs at minimum Storage Blob Data Reader on the specific container (not the whole storage account — scope it down). If you need the shortcut to also handle Delta log writes (for the MERGE operation), it needs Storage Blob Data Contributor on the path. For read-only shortcuts, Reader is sufficient and safer.
Step 4: Specify the Shortcut Path
After the connection is configured and validated, you specify which part of the ADLS Gen2 hierarchy to point at. The wizard shows a folder browser. Navigate to the container and path. For our example:
Container:silver
Path within container:/transactions/
Then give the shortcut a name. This name becomes the table name visible inside the lakehouse. Use the same name as the underlying folder if possible — transactions — to avoid confusion. Click Create.
Step 5: Verify the Shortcut Appears
Back in the lakehouse explorer, expand Tables. You should see transactions listed with a small shortcut icon (a curved arrow). Click on it. The lakehouse schema preview should show the column names and data types inferred from the Delta table's schema.
Warning
If your ADLS Gen2 data is Parquet files rather than a Delta table, the SQL endpoint won't automatically infer a schema and the shortcut won't appear as a SQL table even if you place it under Tables/. You'd need to create it under Files/ and then run a CREATE TABLE statement in a Spark notebook pointing to the shortcut path, or convert the data to Delta format at the source. The SQL Analytics Endpoint is Delta-aware; it is not a generic Parquet catalog.
Creating an Amazon S3 Shortcut
The S3 shortcut creation process follows the same wizard flow, but the authentication model is different and the trade-offs are more significant. Let's use a scenario where a company's marketing team has audience data landing in an S3 bucket managed by their ad tech vendor: s3://marketing-audiences/processed/.
IAM Configuration in AWS
Before touching Fabric, you need to set up the right IAM permissions in AWS. Create an IAM user (or ideally a role if you're using cross-account federation, though Fabric's S3 shortcut implementation uses access keys at the time of writing, so IAM user keys are the practical choice).
Scope the s3:GetObject permission to just the prefix you're exposing (processed/*), not the whole bucket. This matters because OneLake shortcuts pass through these credentials to all Fabric users who query through the shortcut — so the credentials you store in the Fabric connection define the blast radius if those credentials are ever compromised.
Warning
There is a common misconception that Fabric workspace permissions control what data users can access through a shortcut. They don't — not completely. A Fabric workspace member who has permission to query a lakehouse will be able to read anything accessible through the shortcut, regardless of their own AWS IAM permissions. The shortcut runs with the credentials stored at shortcut creation time. This is a critical governance point: design your IAM policies to expose only what you intend to expose, not to rely on Fabric row-level security to restrict shortcut data.
Shortcut Wizard for S3
In the shortcut wizard, select Amazon S3. The connection fields:
Bucket URL:https://marketing-audiences.s3.amazonaws.com or in path-style format https://s3.amazonaws.com/marketing-audiences. Fabric accepts both.
Authentication: Access Key / Secret Key pair from the IAM user you created above.
After the connection validates, specify the path within the bucket:
Bucket:marketing-audiences
Path:/processed/
Name the shortcut audience_segments (or whatever makes sense for your lakehouse schema). Create it.
Verifying S3 Data Accessibility
After creating the shortcut, test immediately by running a cell in a Spark notebook:
Watch the execution time on that first cell. Cross-cloud reads are noticeably slower than intra-Azure reads. A 1GB scan from S3 that would take 2-3 seconds from ADLS Gen2 in the same region might take 15-30 seconds from S3, depending on your AWS region and network conditions.
Querying Shortcut Data with Spark
Once your shortcuts are in place, Spark treats them as first-class paths inside your lakehouse. This is where the abstraction pays off — you write the same code you'd write for native OneLake data.
Reading Delta Shortcuts
For a Delta table shortcut under Tables/, you can reference it either by its logical table name or by its full ABFS path:
# Using the logical path (preferred)
df = spark.read.format("delta").load("Tables/transactions")
# Equivalent: using the full OneLake ABFS path
# (useful when running the same notebook code against multiple workspaces)
df = spark.read.format("delta") \
.load("abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/RetailLakehouse.Lakehouse/Tables/transactions")
# Count and inspect
print(f"Row count: {df.count():,}")
df.printSchema()
df.limit(10).display()
The display() call (Fabric's enhanced version of show()) will materialize data from the external source. If your shortcut points to ADLS Gen2 in the same Azure region as your Fabric capacity, this should feel nearly as fast as querying native OneLake data.
Joining Shortcut Data with Native Lakehouse Tables
The real power of shortcuts becomes obvious when you join external data with data that lives natively in OneLake. Here's a realistic example: join the transactions shortcut (from ADLS Gen2) with a customers table that lives natively in the lakehouse, to build a denormalized fact table:
from pyspark.sql import functions as F
# Load shortcut-backed Delta table from ADLS Gen2
transactions = spark.read.format("delta").load("Tables/transactions")
# Load native OneLake Delta table
customers = spark.read.format("delta").load("Tables/customers")
# Join and compute customer lifetime value
clv = transactions \
.join(customers, on="customer_id", how="inner") \
.groupBy("customer_id", "customer_name", "region") \
.agg(
F.count("transaction_id").alias("transaction_count"),
F.sum("transaction_amount").alias("lifetime_value"),
F.max("transaction_date").alias("last_purchase_date")
) \
.withColumn("value_tier",
F.when(F.col("lifetime_value") >= 10000, "Platinum")
.when(F.col("lifetime_value") >= 2500, "Gold")
.when(F.col("lifetime_value") >= 500, "Silver")
.otherwise("Bronze")
)
# Write result to a native OneLake Delta table (the gold layer)
clv.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("customer_lifetime_value")
Notice what's happening here: Spark reads from two sources simultaneously — an ADLS Gen2 shortcut and native OneLake storage — and writes the result to a new native OneLake table. The compute engine handles the join and aggregation in memory on the Spark cluster. The shortcut abstraction means you didn't have to write any custom I/O code for the ADLS Gen2 source; it just works as a table.
Tip
When joining a large shortcut table with a small native table, Spark's optimizer may choose a broadcast join, sending the small table to all executors rather than shuffling the large one. For shortcuts pointing to external storage, this is almost always the right choice — you want to minimize the data scanned from the external source. You can hint this explicitly: customers_bc = F.broadcast(customers) and then join with transactions.join(customers_bc, ...).
Working with Non-Delta Parquet Data via Shortcuts
If your ADLS Gen2 or S3 data is plain Parquet (no Delta log), you can still query it effectively, but you lose Delta's versioning, schema enforcement, and ACID guarantees. You can also take advantage of partition pruning if your data is partitioned with Hive-style directory names:
# Parquet data partitioned by year and month
# Directory structure: Files/sales_data/year=2024/month=01/part-00000.parquet
sales_raw = spark.read \
.format("parquet") \
.load("Files/sales_data")
# Filter pushdown will prune partitions based on directory structure
q4_sales = sales_raw.filter(
(F.col("year") == 2024) & (F.col("month").isin([10, 11, 12]))
)
q4_sales.explain("formatted") # Verify partition pruning in the plan
Run explain("formatted") and look for PartitionFilters in the output. If partition pruning is working, you'll only see the relevant partition directories being read. This is important when your shortcut points to a large external dataset — every byte you don't read from the external store saves both time and (for S3) egress cost.
Querying Shortcut Data with the SQL Analytics Endpoint
Shortcuts that live under Tables/ and point to valid Delta tables are automatically queryable through the SQL Analytics Endpoint without any additional configuration. This means you can use T-SQL to query them, and — critically — Power BI can connect to them via Direct Lake mode.
Basic T-SQL Queries
Open the SQL Analytics Endpoint for your lakehouse (switch from "Lakehouse" to "SQL analytics endpoint" in the top-right mode selector). In the query editor:
-- Query the shortcut-backed table directly
SELECT
region,
COUNT(*) AS transaction_count,
SUM(transaction_amount) AS total_revenue,
AVG(transaction_amount) AS avg_transaction_value
FROM transactions -- This is the ADLS Gen2 shortcut
GROUP BY region
ORDER BY total_revenue DESC;
-- Cross-query: join shortcut table with native table
SELECT
c.customer_name,
c.email,
t.transaction_date,
t.transaction_amount,
t.product_sku
FROM transactions t -- shortcut from ADLS Gen2
INNER JOIN customers c -- native OneLake table
ON t.customer_id = c.customer_id
WHERE t.transaction_date >= '2024-01-01'
AND t.transaction_amount > 500
ORDER BY t.transaction_date DESC;
The SQL endpoint can't tell the difference between a shortcut-backed table and a native one. Both appear in the same schema. This is the core value proposition: unified querying without unified storage.
Cross-Lakehouse Shortcut Queries
OneLake also supports shortcuts that point to other OneLake locations — tables in a different lakehouse within the same tenant. This enables a pattern where a central, governed lakehouse exposes data to multiple downstream lakehouses via OneLake-to-OneLake shortcuts, without anyone having write access to the source.
-- In a downstream analytics lakehouse, query a shortcut that points
-- to a table in the central data platform lakehouse
SELECT
product_category,
SUM(units_sold) AS total_units,
SUM(revenue) AS total_revenue
FROM product_sales_gold -- OneLake-to-OneLake shortcut
GROUP BY product_category;
This pattern maps well to the medallion architecture, where gold-layer tables in a central lakehouse are exposed via shortcuts to domain-specific workspaces. Each domain team gets read access to the gold data through shortcuts, without needing access to the pipeline workspace that produces it.
Key insight
OneLake-to-OneLake shortcuts don't consume egress bandwidth the way S3 shortcuts do — they're free intra-tenant reads. This makes them a compelling alternative to duplicating gold-layer data across workspaces. Instead of running a copy pipeline every time the gold table updates, the shortcut always reflects the current state.
Direct Lake Reporting Over Shortcut Data
If you've read about Direct Lake mode in Power BI, you know it works by reading Delta table parquet files directly from OneLake into the Power BI engine's columnar store — without import, without DirectQuery round trips. What's less obvious is that Direct Lake also works for shortcut-backed tables, but with an important caveat.
What Works
Create a semantic model over your lakehouse (either auto-generated or custom). The semantic model can include shortcut-backed tables alongside native tables. When Power BI renders a report, it triggers a framing operation that reads the Delta table's parquet files — and for a shortcut, this means reading from ADLS Gen2 or S3.
For ADLS Gen2 shortcuts in the same Azure region, Direct Lake performance over shortcut data is typically indistinguishable from native OneLake data. The underlying I/O path is fast enough that the framing operation completes in roughly the same time.
What Doesn't Work
Direct Lake falls back to DirectQuery mode (slower) when it can't satisfy query predicates from the in-memory columnar cache. For shortcut-backed tables, if the external source changes frequently and the Delta log has many uncommitted or fine-grained transaction entries, the framing overhead increases. In extreme cases — hundreds of small transactions per hour against a shortcut table — you might see framing operations that take tens of seconds, effectively breaking the Direct Lake performance promise.
The guidance here: if external data is frequently updated and you need sub-second Direct Lake performance, consider whether the right architecture is a pipeline that copies and compacts the data into native OneLake, rather than a shortcut. Shortcuts aren't always better — they're better when the data is large, relatively stable, and you want to avoid duplication cost.
Warning
Direct Lake semantic models over S3 shortcuts will almost always fall back to DirectQuery for any non-trivial model. The latency introduced by cross-cloud I/O during the framing operation often exceeds Direct Lake's internal timeout, triggering the DirectQuery fallback. If your Power BI report must perform at Direct Lake speed, the data should be in ADLS Gen2 in the same region as your Fabric capacity, or in native OneLake.
Security Model Deep Dive
Shortcuts introduce a security model that many Fabric administrators get wrong, so it's worth spending time here.
Credential Inheritance
When a shortcut is created, the credentials (service principal, access key, or IAM key) are stored encrypted in Fabric's connection metadata. Every time any user queries through that shortcut — whether via Spark, the SQL endpoint, or Power BI — those stored credentials are used to authenticate to the external store. The querying user's own Azure AD identity is not used for the external storage call.
This means:
A Power BI report viewer who has no Azure AD permissions on your ADLS Gen2 account can still read data through the shortcut
A workspace admin who deletes and recreates the shortcut might use different credentials, changing access behavior
If the stored credentials expire or are revoked (service principal secret rotation, IAM key deletion), all shortcuts using that connection immediately break
Workspace-Level Access Control
Fabric workspace roles control who can use shortcuts, not what the shortcut can access:
Workspace Role
Can Query Shortcut Data
Can Create Shortcuts
Can Delete Shortcuts
Viewer
Yes
No
No
Contributor
Yes
Yes
Yes (own)
Member
Yes
Yes
Yes
Admin
Yes
Yes
Yes
A workspace Viewer can read any data accessible through any shortcut in that workspace, as long as the connection credentials have access to the external source. Design your ADLS Gen2 or S3 IAM policies accordingly — don't rely on workspace role-based access to restrict data at the row or column level.
Row-Level and Column-Level Security
Shortcuts bypass Fabric's object-level security in one important way: the SQL Analytics Endpoint supports object-level security (you can GRANT/DENY access to specific tables), but shortcuts expose their data through the Delta table interface, so standard SQL GRANT statements apply. If you need row-level security — for example, users in the EU workspace should only see EU transaction records — you can't currently enforce that through the shortcut itself. You'd need to:
Create a view over the shortcut table that filters by the user's region, using USER_NAME() or workspace membership
Grant access to the view rather than the underlying shortcut table
Revoke direct query access to the shortcut table
This is a gap that Fabric's security model is evolving to address, but at the time of writing, it requires deliberate workaround design.
Tip
For sensitive datasets where data residency or access control is paramount, consider whether a Dataflow Gen2 ingestion pattern makes more sense than a shortcut. When you copy data into native OneLake via Dataflow Gen2, you can apply Fabric's full security model including row-level security, column masking, and workspace-level data policies. Shortcuts are powerful for convenience and cost reduction, but they delegate security decisions to the external source's IAM system.
Performance Optimization for Shortcut-Backed Tables
Understanding the performance characteristics of shortcuts helps you make better architectural decisions and avoid nasty surprises in production.
Delta Log Size and Metadata Overhead
Every time a Spark job or the SQL endpoint opens a Delta table, it reads the Delta log to construct the current table version. For a native OneLake table, this is fast — the log files are in the same Azure storage account. For a shortcut, the log files are fetched from the external source on every query.
If your ADLS Gen2 Delta table has accumulated thousands of transaction log entries (because no one has run OPTIMIZE or VACUUM against it), every query through the shortcut pays a metadata overhead cost proportional to the log size. Run this in a Spark notebook to check:
from delta.tables import DeltaTable
# For an ADLS Gen2 shortcut
dt = DeltaTable.forPath(spark, "Tables/transactions")
history = dt.history()
history.select("version", "timestamp", "operation", "operationMetrics").show(20, truncate=False)
If you see thousands of versions and the latest checkpoint is many versions ago, the Delta table at the source needs optimization. If you don't control the source, this is a meaningful argument for copying the data into native OneLake and running Spark's OPTIMIZE there.
Predicate Pushdown and File Skipping
Delta's Z-ordering and file statistics work through shortcuts — but only if the Delta table at the source was written with statistics enabled (which it is by default in Delta 1.x and above). When you query:
# This should benefit from file skipping if the Delta table has statistics
recent_txns = spark.read.format("delta") \
.load("Tables/transactions") \
.filter(F.col("transaction_date") >= "2024-10-01")
Spark reads the Delta log, inspects per-file min/max statistics for transaction_date, and skips files that definitely don't contain records in the date range. This happens before any data is fetched from ADLS Gen2 or S3, so file skipping is especially valuable for external shortcuts — you're literally reducing the number of cross-network HTTP requests made.
You can verify file skipping is happening by examining the Spark UI or running:
recent_txns.explain("formatted")
# Look for "PushedFilters" and "PartitionFilters" in the output
Caching Shortcut Data Strategically
For shortcut data that changes slowly but is queried frequently, Spark's disk caching can dramatically improve repeated query performance:
# Cache a frequently-queried shortcut table on Spark executors
transactions = spark.read.format("delta").load("Tables/transactions")
transactions.cache()
transactions.count() # Force materialization into cache
# Subsequent queries hit the cache, not the external source
regional_summary = transactions \
.groupBy("region") \
.agg(F.sum("transaction_amount")) \
.collect()
Tip
Spark's cache() is most effective for interactive analytics sessions where you're running multiple queries against the same dataset. In scheduled notebooks or pipeline jobs that run once and exit, caching adds overhead without payoff. Use it judiciously based on your query patterns.
Hands-On Exercise
Let's pull everything together with an exercise that simulates a real-world integration scenario. You'll create shortcuts to both an ADLS Gen2 location and an S3 location, join them, and produce an output table.
Scenario: Your company's ERP system (SAP) writes daily order extracts to ADLS Gen2 at abfss://silver@erpdata.dfs.core.windows.net/orders/ as Delta tables. Your logistics partner drops shipment status data as Parquet files to an S3 bucket at s3://logistics-partner-feeds/shipments/. You need to join these to produce an order_fulfillment table in your Fabric lakehouse.
Step 1: Create the ADLS Gen2 shortcut. In your lakehouse, create a shortcut under Tables/ named erp_orders pointing to the ADLS Gen2 path. Use a service principal with Storage Blob Data Reader on the silver container.
Step 2: Create the S3 shortcut. Under Files/ (because it's Parquet, not Delta), create a shortcut named logistics_shipments pointing to the S3 prefix /shipments/. Use an IAM access key with s3:GetObject and s3:ListBucket on logistics-partner-feeds.
Step 3: Open a Spark notebook and run:
from pyspark.sql import functions as F
# Read the ERP orders shortcut (Delta table - use Tables/ path)
orders = spark.read.format("delta").load("Tables/erp_orders")
# Read the logistics shipments shortcut (Parquet - use Files/ path)
shipments = spark.read.format("parquet").load("Files/logistics_shipments")
# Inspect schemas before joining
print("=== ERP Orders Schema ===")
orders.printSchema()
print("=== Logistics Shipments Schema ===")
shipments.printSchema()
# Deduplicate shipments (logistics files may contain duplicates)
shipments_deduped = shipments \
.withColumn("row_num",
F.row_number().over(
Window.partitionBy("order_id")
.orderBy(F.col("shipment_timestamp").desc())
)
) \
.filter(F.col("row_num") == 1) \
.drop("row_num")
# Join: left join so we keep orders even without a shipment record yet
order_fulfillment = orders \
.join(shipments_deduped, on="order_id", how="left") \
.select(
orders.order_id,
orders.customer_id,
orders.order_date,
orders.order_total,
orders.product_sku,
shipments_deduped.carrier,
shipments_deduped.tracking_number,
shipments_deduped.delivery_status,
shipments_deduped.estimated_delivery_date,
F.when(shipments_deduped.order_id.isNull(), "Pending Shipment")
.when(F.col("delivery_status") == "DELIVERED", "Fulfilled")
.otherwise("In Transit").alias("fulfillment_status")
)
# Write to native OneLake as a gold-layer table
order_fulfillment.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("order_fulfillment")
print(f"Written {order_fulfillment.count():,} rows to order_fulfillment")
Step 4: Verify in the SQL Analytics Endpoint:
SELECT
fulfillment_status,
COUNT(*) AS order_count,
SUM(order_total) AS total_value
FROM order_fulfillment
GROUP BY fulfillment_status
ORDER BY order_count DESC;
This exercise demonstrates the core pattern: shortcuts from heterogeneous external sources → Spark transformation → native OneLake output table → SQL/Power BI consumption.
Common Mistakes & Troubleshooting
Shortcut Shows Up But Returns No Data
Symptom: The shortcut appears in the lakehouse explorer, but querying it returns an empty result or throws an authentication error.
Causes and fixes:
The stored credentials no longer have access (service principal secret expired, IAM key rotated). Navigate to the Data Engineering hub, find your connection under "Connections," and update the credentials.
The external path no longer exists (container renamed, bucket deleted). Verify the path directly using Azure Storage Explorer or the AWS console.
For ADLS Gen2, the storage account's firewall is blocking Fabric's IP ranges. Add Fabric's trusted service exception or the specific IP ranges from your region's Fabric documentation.
SQL Endpoint Shows Shortcut Table But Schema Looks Wrong
Symptom: The table appears in the SQL endpoint, but columns show as string when they should be decimal, or the schema doesn't match the actual data.
Cause: The Delta table at the source has schema evolution issues — the Delta log references a schema that doesn't match recent Parquet file additions.
Fix: In a Spark notebook, run:
from delta.tables import DeltaTable
dt = DeltaTable.forPath(spark, "Tables/your_shortcut")
dt.toDF().printSchema()
# If the schema looks wrong, try forcing a schema refresh
spark.read.format("delta") \
.option("mergeSchema", "true") \
.load("Tables/your_shortcut") \
.printSchema()
Cross-Cloud S3 Queries Timing Out
Symptom: Spark jobs hang or fail with timeout errors when reading from S3 shortcuts on large scans.
Causes:
Network latency between Azure region and AWS region is too high for the volume of data
The S3 bucket is in a region far from your Fabric capacity's Azure region (e.g., Fabric in West Europe, S3 in ap-southeast-2)
Small file problem: the S3 path contains thousands of tiny Parquet files, causing thousands of separate HTTP requests
Fix: For the small file problem, run a consolidation job at the source to compact files to 128MB–256MB each. For latency problems, evaluate whether copying the data into ADLS Gen2 first (and then creating an ADLS Gen2 shortcut, or no shortcut at all) makes more sense for your query volume.
Direct Lake Report Falls Back to DirectQuery
Symptom: Power BI reports using a semantic model over shortcut-backed tables show the "DirectQuery" indicator instead of "Direct Lake," and performance is poor.
Cause: As discussed earlier, this usually means the framing operation is timing out due to cross-cloud latency (S3 shortcuts), or the Delta log is too large.
Fix: Run OPTIMIZE on the source Delta table to compact small files and update statistics. If the data source is S3, seriously reconsider the architecture for Power BI reporting — Direct Lake over S3 is not production-ready for most organizations.
Shortcut Permission Errors for Other Users
Symptom: You can query the shortcut fine, but a colleague gets a 403 error.
Cause: If you used "Organizational account" authentication when creating the shortcut, the shortcut authenticates with the creating user's own identity. Other users' queries fail because the creating user's credentials are used, but those credentials might not be what you think (or the token may have expired).
Fix: Delete and recreate the shortcut using a Service Principal connection. Service principal credentials are stored server-side and used for all queries regardless of which user initiates them.
Summary & Next Steps
OneLake shortcuts give you a genuinely new way to think about data architecture in the enterprise. Instead of always asking "how do we get data into Fabric?", you can ask "should this data be in Fabric at all, or should Fabric simply point to where the data already lives?" For large, stable datasets in well-managed ADLS Gen2 environments, the shortcut answer is often the right one. For cross-cloud S3 data that feeds interactive Power BI reports, the shortcut may be a stepping stone rather than a final destination.
The key principles to carry forward:
Shortcuts are metadata pointers, not data copies. Changes to the external source are immediately visible.
ADLS Gen2 shortcuts in the same Azure region as your Fabric capacity are functionally equivalent to native OneLake in performance terms.
S3 shortcuts work well for batch Spark processing; they're a poor fit for interactive Direct Lake reporting.
The security model delegates external data access to the credentials stored at shortcut creation time — design your IAM policies accordingly.
Delta table quality at the source (file sizes, checkpoint frequency, statistics) directly affects shortcut query performance.
To go deeper, explore how shortcuts integrate with the Fabric Lakehouse vs. Warehouse decision — specifically, whether your shortcut-backed data should be exposed through a lakehouse SQL endpoint or promoted into a Fabric Warehouse for more sophisticated T-SQL workloads. If you're building Spark notebooks that transform shortcut data into gold-layer tables, the patterns in that lesson apply directly on top of what you've learned here.
Shortcuts are one of the features that make Microsoft Fabric genuinely different from a "put everything in one place" architecture. Used well, they let you build a unified analytics layer over a heterogeneous storage estate — which is, let's be honest, what most organizations actually have.
