The Library · Insights
Deep dives across data, automation & AI
53 in-depth articles — from Excel foundations to production machine learning. Filter by topic or difficulty to find exactly what you need.
Microsoft FabricPractitioner
Implementing Table Partitioning in a Fabric Lakehouse: Choosing Partition Keys, Writing Partitioned Delta Tables with PySpark, and Pruning Partitions for Faster SQL and Spark Queries
Unpartitioned Delta tables make Spark read everything before filtering anything. This lesson teaches you how to choose partition keys that actually improve performance, write partitioned gold-layer tables with PySpark, and verify that partition pruning is firing in both Spark and the SQL analytics endpoint.
22 min read
Microsoft FabricExpert
Implementing End-to-End Pipeline Error Handling in Microsoft Fabric: If Condition Activities, Failed Dependencies, and Email Alerts
Build production-grade error handling in Microsoft Fabric pipelines that actually catches failures, routes them intelligently, and sends actionable email alerts — across Copy, Dataflow Gen2, and Notebook activities. Learn the dependency condition model, activity output expressions, and centralized error handler patterns that keep your medallion pipelines resilient.
26 min read
Microsoft FabricPractitioner
Deduplicating and Cleansing Lakehouse Delta Tables with PySpark: Drop Duplicates, Fill Nulls, and Enforce Data Quality Rules Across Medallion Layers
Dirty data in bronze silently corrupts silver and gold. Learn how to build a production-grade PySpark cleansing pipeline that removes exact and soft duplicates, handles nulls with intentional strategies, quarantines bad rows, and enforces data quality rules with exceptions — all structured around the medallion architecture in Microsoft Fabric.
20 min read
Microsoft FabricExpert
Implementing Incremental Refresh for Direct Lake Semantic Models in Microsoft Fabric: Configuring Delta Table Partitioning, Framing Policies, and Triggering Refresh via the XMLA Endpoint
Direct Lake incremental refresh isn't a checkbox — it requires aligning Delta table partitioning, TMDL framing policies, and XMLA-triggered refresh operations into a precise orchestration chain. This lesson gives you the full architecture, the code, and the troubleshooting knowledge to implement it correctly in production.
26 min read
Microsoft FabricPractitioner
Orchestrating Multi-Notebook Workflows in Microsoft Fabric: Using Pipeline Notebook Activities, Activity Dependencies, and Output Variables to Chain PySpark Transformations Across Medallion Layers
Learn how to wire Fabric PySpark notebooks into a production pipeline using Notebook activities, On Success/On Failure dependencies, and mssparkutils.notebook.exit() output variables. Build a complete Bronze→Silver→Gold medallion pipeline that passes context between stages and handles failures gracefully.
20 min read
Microsoft FabricPractitioner
Building a Star Schema in a Fabric Lakehouse Gold Layer: Creating Dimension and Fact Delta Tables with PySpark for Direct Lake Reporting
Learn how to design and build a production-quality star schema in your Fabric Lakehouse Gold layer using PySpark — covering surrogate key strategy, dimension and fact table construction, orphan validation, and post-write optimization for Direct Lake reporting in Power BI.
22 min read
Microsoft FabricExpert
Implementing Delta Lake Time Travel in a Fabric Lakehouse: Querying Historical Snapshots, Rolling Back Bad Writes, and Auditing Table Changes with PySpark and T-SQL
Delta Lake time travel turns catastrophic data overwrites into two-minute recoveries. Learn how to query historical snapshots by version and timestamp, execute safe rollbacks with RESTORE TABLE, build a persistent audit trail, and manage retention windows so time travel is available when you actually need it.
31 min read
Microsoft FabricPractitioner
Implementing Slowly Changing Dimensions in a Fabric Lakehouse: Using PySpark and Delta Lake MERGE to Track Historical Changes Across Medallion Layers
Learn how to build production-quality SCD Type 2 dimensions in a Microsoft Fabric Lakehouse using PySpark and Delta Lake's MERGE statement. This lesson covers the full pattern: change detection with attribute hashing, two-phase expire-and-insert MERGE logic, surrogate key generation, and Gold layer point-in-time joins — with a hands-on exercise you can run in your own environment.
24 min read
Microsoft FabricFoundation
Scheduling Dataflow Gen2 Refreshes and Chaining Them into a Fabric Data Pipeline for End-to-End Orchestration
Manual refreshes don't scale. Learn how to schedule Dataflow Gen2 refreshes automatically and chain them into a Fabric Data Pipeline alongside Copy Activities and Spark Notebooks — so your data is ready before your team arrives each morning.
17 min read
Microsoft FabricFoundation
Using Notebook Variables and Parameters in Microsoft Fabric: Passing Pipeline Values to PySpark for Dynamic Lakehouse Ingestion
Hardcoded file paths and table names don't belong in production notebooks. Learn how to use Fabric's parameter cell pattern to pass dynamic values from a data pipeline into PySpark, turning a one-time script into a reusable ingestion engine.
15 min read
Microsoft FabricExpert
Handling Schema Evolution in Fabric Lakehouse Delta Tables: Adding Columns, Merging Incompatible Schemas, and Enforcing Constraints Across Medallion Layers
Schema changes in production Delta lakehouses cascade across medallion layers in ways that break pipelines, corrupt reports, and silently produce wrong results. This deep-dive lesson teaches you exactly how Delta Lake enforces and evolves schemas, how to build drift detection into your pipelines, and how to use CHECK and NOT NULL constraints to enforce data quality where it matters.
29 min read
Microsoft FabricPractitioner
Branching Dataflow Gen2 Outputs to Multiple Destinations: Writing Transformed Data to a Lakehouse Table and a Warehouse Simultaneously
Dataflow Gen2 lets you attach multiple output destinations to a single query — run your transformation once and write results to both a Fabric Lakehouse table and a Warehouse table simultaneously. This lesson teaches you the complete pattern: from Power Query transformation to dual-destination configuration, data type gotchas, and production-ready pipeline orchestration.
22 min read