The Library · Insights
Deep dives across data, automation & AI
53 in-depth articles — from Excel foundations to production machine learning. Filter by topic or difficulty to find exactly what you need.
Microsoft FabricExpert
Building a Reusable Fabric Pipeline Template with Metadata-Driven Orchestration: Driving Copy Activities, Notebook Runs, and Dataflow Refreshes from a Configuration Table
Stop building one pipeline per data source. This deep-dive lesson teaches you to build a single, parameterized Fabric pipeline template that reads from a control table and dynamically dispatches Copy Activities, Notebook runs, and Dataflow Gen2 refreshes — with per-source retry logic, execution logging, and room to grow. If you're managing more than five data sources in Fabric, this framework will change how you work.
28 min read
Microsoft FabricPractitioner
Using Fabric Data Activator to Trigger Alerts and Pipeline Actions from Power BI Reports and Eventstreams
Data Activator turns your Fabric environment into an active, event-driven platform — not just a reporting system. Learn how to define objects, properties, and trigger conditions against Power BI reports and Eventstreams, then wire up automated responses including Teams notifications and live pipeline runs.
24 min read
Microsoft FabricExpert
Implementing Cross-Workspace Data Sharing in Microsoft Fabric with OneLake Shortcuts: Exposing Lakehouse and Warehouse Tables to Downstream Workspaces Without Copying Data
Stop copying data between Fabric workspaces. This expert-level lesson shows you how to use OneLake shortcuts to expose lakehouse and warehouse tables to downstream workspaces, covering permissions, performance, warehouse security caveats, and a hands-on multi-workspace exercise. Learn the hub-and-spoke shortcut pattern used by mature Fabric platform teams.
30 min read
Microsoft FabricPractitioner
Streaming Data into a Fabric Eventhouse with Eventstreams: Configuring Sources, Defining KQL Tables, and Querying Live Telemetry Data
Learn how to build a complete real-time telemetry pipeline in Microsoft Fabric — from configuring an Eventstream source to defining KQL tables with update policies and materialized views, to writing operational queries against live streaming data. Uses a realistic IoT fleet monitoring scenario throughout.
22 min read
Microsoft FabricExpert
Configuring Fabric Capacity Bursting, Throttling Behavior, and Smoothing Windows: Tuning Workload Settings to Prevent Job Failures Under Heavy Pipeline and Spark Load
When pipelines and Spark jobs collide under shared Fabric capacity, the result isn't graceful degradation — it's cascading failures. This deep-dive lesson teaches you exactly how Fabric's 10-minute smoothing window, throttling stages, and Capacity Admin settings work, and how to tune them to prevent job failures under real production load.
31 min read
Microsoft FabricPractitioner
Aggregating and Writing Summary Tables from a Fabric Lakehouse Gold Layer to a Fabric Warehouse Using Cross-Engine T-SQL Views and PySpark for Dual-Engine Reporting
Your gold layer serves Spark consumers perfectly — but your SQL analysts need pre-computed summary tables, T-SQL views, and sub-second query latency. This lesson shows you how to aggregate gold-layer Delta tables with PySpark, write results into a Fabric Warehouse via JDBC, and expose lakehouse tables as cross-engine T-SQL views — all without duplicating data or maintaining two separate pipelines.
22 min read
Microsoft FabricExpert
Implementing Fabric Notebook-Based Unit Tests with pytest and the Fabric REST API: Validating Delta Table Outputs, Row Counts, and Schema Contracts Before Promoting Data Across Medallion Layers
Learn how to build a production-grade test harness that runs pytest inside Microsoft Fabric Spark notebooks, validates Delta table schemas and row counts against versioned contracts, and gates medallion layer promotion via the Fabric REST API. This lesson goes beyond basic assertions to cover referential integrity, Delta version auditing, and pipeline-integrated pass/fail routing — the complete picture for data quality enforcement at scale.
28 min read
Microsoft FabricPractitioner
Configuring OneLake Shortcuts to a Fabric Lakehouse as the Data Source for a Fabric Warehouse: Avoiding Data Duplication While Enabling T-SQL Queries Across Both Engines
Learn how to create OneLake shortcuts in a Fabric Warehouse that point to lakehouse Delta tables — enabling T-SQL queries across both engines with zero data duplication. This lesson covers the full pattern from shortcut creation to cross-engine joins, security configuration, metadata sync, and performance tuning.
24 min read
Microsoft FabricExpert
Automating Fabric Lakehouse Metadata Refresh with Semantic Link: Syncing Delta Table Schema Changes to a Direct Lake Semantic Model Using Python and the Fabric REST API
Delta tables evolve constantly, but Direct Lake semantic models don't automatically follow. Learn how to detect schema drift between your gold-layer tables and your Power BI model, then trigger a targeted metadata refresh using Python, sempy, and the Fabric REST API — all from inside a Fabric notebook you can drop into any pipeline.
27 min read
Microsoft FabricPractitioner
Implementing a Fabric Notebook-Based Data Quality Framework: Validating Row Counts, Null Thresholds, and Referential Integrity Across Medallion Layers Before Pipeline Promotion
Bad data that passes silently through your medallion pipeline is worse than a pipeline that fails loudly. This lesson walks you through building a reusable PySpark quality framework in Microsoft Fabric that validates row counts, null rates, and referential integrity at each layer — and wires the results into your pipeline so failures actually stop promotion.
22 min read
Microsoft FabricExpert
Unpivoting, Aggregating, and Reshaping Lakehouse Data in Dataflow Gen2: Advanced Power Query Transformations Before Writing to Delta Tables
Learn how to use advanced Power Query M techniques in Dataflow Gen2 to unpivot wide source data, aggregate with Table.Group, and reshape tables into clean, typed Delta table schemas in your Microsoft Fabric lakehouse. This lesson covers the full M query structure, type-fixing patterns, and the honest trade-offs between Dataflow Gen2 and Spark for complex transformations.
27 min read
Microsoft FabricExpert
Joining Multiple Delta Tables Across Fabric Lakehouses and Warehouses in a Single Spark Notebook: Cross-Workspace Queries, OneLake Paths, and Writing Results to a Gold Layer Table
Learn how to read Delta tables from multiple Fabric workspaces — across lakehouses and warehouses — in a single PySpark notebook. This expert lesson covers OneLake ABFS path construction, programmatic GUID resolution, multi-table joins with broadcast optimization, and writing results to a gold-layer Delta table using overwrite and MERGE patterns.
24 min read