Handling Schema Evolution in Fabric Lakehouse Delta Tables: Adding Columns, Merging Incompatible Schemas, and Enforcing Constraints Across Medallion Layers
Schema changes in production Delta lakehouses cascade across medallion layers in ways that break pipelines, corrupt reports, and silently produce wrong results. This deep-dive lesson teaches you exactly how Delta Lake enforces and evolves schemas, how to build drift detection into your pipelines, and how to use CHECK and NOT NULL constraints to enforce data quality where it matters.
🔥 Expert29 min readSep 22, 2026Updated Sep 22, 2026
Handling Schema Evolution in Fabric Lakehouse Delta Tables: Adding Columns, Merging Incompatible Schemas, and Enforcing Constraints Across Medallion Layers
Introduction
You've built a clean medallion architecture. Bronze is landing raw data, silver is transforming it, gold is serving the business. Everything is running smoothly — and then the source system adds three new columns, changes a data type from INT to BIGINT, and renames a field that your silver layer joins on. Suddenly your pipelines are throwing cryptic errors, your Delta tables have partitioned the world into "before the change" and "after the change," and your Power BI reports are either stale or broken.
Schema evolution in Delta tables is one of those topics that's easy to dismiss until it bites you. The mechanics seem simple — add a column, update a schema, move on. But in a real lakehouse, schema changes cascade. A new nullable column in bronze has to be accounted for in silver's transformation logic, which has to flow into gold's aggregations, which has to match what your Direct Lake semantic model expects. Get any one layer wrong and the whole pipeline either fails noisily or, worse, silently produces incorrect results.
By the end of this lesson, you'll understand exactly how Delta Lake manages schema changes internally, how to handle them safely at each medallion layer, and how to build guardrails so that source-system changes don't catch you off guard. You'll walk away with real code, real patterns, and an architectural approach that scales.
What you'll learn:
How Delta Lake's schema enforcement and schema evolution work at the protocol level, and when each applies
How to use mergeSchema, overwriteSchema, and ALTER TABLE to add columns and handle type changes across medallion layers
How to merge DataFrames with incompatible schemas without data loss
How to implement CHECK constraints and NOT NULL constraints on Delta tables in Fabric to enforce data quality rules
How to design schema governance patterns that propagate changes safely from bronze through silver to gold
Prerequisites
This lesson assumes you're comfortable working with PySpark notebooks in Microsoft Fabric, and that you have a working understanding of Delta table internals — the transaction log, Parquet files, and how Spark reads Delta tables. You should also be familiar with the medallion architecture pattern. If you haven't built one yet, start with Implementing the Medallion Architecture in Microsoft Fabric: Bronze, Silver, and Gold Layers before continuing.
Before you can manage schema evolution intelligently, you need to understand what Delta Lake is actually doing when it validates a write.
Every Delta table stores its current schema in the transaction log — specifically in the metaData action of the log. This schema definition includes column names, data types, nullability, and nested field structures for complex types like maps, arrays, and structs. When you attempt to write a DataFrame to a Delta table, the Delta writer compares the DataFrame's schema against the schema stored in the log. If they don't match, Delta raises an AnalysisException before writing a single byte.
This is schema enforcement (also called schema validation), and it's on by default. It's not optional, and it's not configurable per write — it's a property of the table. This design is intentional: Delta wants to be the last line of defense against unintentional schema corruption.
The enforcement rules are specific:
Extra columns in the DataFrame that don't exist in the table schema → rejected
Missing columns in the DataFrame that exist in the table schema → allowed (those columns get NULL)
Type mismatches between the DataFrame and the table → rejected, with one exception for safe upcasts
Nullable mismatches → generally allowed as long as the actual data respects nullability
The "safe upcast" exception is important. Delta will allow writing an INT column into a LONG column because that's a widening conversion with no data loss. It will not allow writing a LONG into an INT or a STRING into a LONG, because those could lose data or fail at runtime.
Key insight
Schema enforcement protects the table from bad writes, but it says nothing about whether the data inside those writes is correct. A column that's declared as NOT NULL can still receive nulls unless you've added a Delta constraint. Schema enforcement and data quality constraints are different layers — and you need both.
Schema Evolution Mechanics: mergeSchema and overwriteSchema
Delta Lake provides two explicit escape hatches from schema enforcement: mergeSchema and overwriteSchema. They solve different problems, and confusing them is a common source of production incidents.
mergeSchema: Additive Changes Only
mergeSchema tells Delta to expand the table's stored schema to include any new columns present in the incoming DataFrame. It does not remove existing columns, does not change existing column types (unless it's a safe upcast), and does not reorder columns.
Here's what it looks like in practice. Suppose your bronze lakehouse receives a new feed from an ERP system that has added a discount_code column that wasn't there before:
from pyspark.sql import SparkSession
from pyspark.sql.functions import current_timestamp, lit
spark = SparkSession.builder.getOrCreate()
# New batch includes a column that doesn't exist in the target table yet
new_erp_batch = spark.read.parquet("Files/landing/erp_orders_2024_11.parquet")
# new_erp_batch schema:
# order_id: long
# customer_id: long
# order_date: date
# order_total: decimal(18,2)
# discount_code: string <-- NEW column
new_erp_batch.write \
.format("delta") \
.mode("append") \
.option("mergeSchema", "true") \
.save("Tables/bronze_erp_orders")
When this write executes, Delta does the following:
Reads the existing schema from the transaction log: four columns, no discount_code
Computes the union of the existing schema and the incoming DataFrame schema
Writes a new metaData action to the transaction log that adds discount_code as a nullable STRING column
Appends the new Parquet files
Critically, all the existing rows in the table now have discount_code = NULL — not because the data was modified, but because Parquet files that predate this schema change don't contain a discount_code column, so Delta returns NULL for those rows when they're read. This is the core mechanism that makes Delta's schema evolution non-destructive.
Warning
When you add a column via mergeSchema, existing rows return NULL for that column. If your silver layer transformation then does something like COALESCE(discount_code, 'NONE'), that's fine — it handles the NULL explicitly. But if your silver logic assumes discount_code is always populated and uses it as a join key, you'll silently produce wrong results for all historical rows. Always audit your downstream transformation logic after a bronze schema change.
You can also set mergeSchema at the session level if you want all writes in a notebook to use it:
Use the session-level setting with caution — it's convenient but removes the explicit, per-write signal that a schema change is happening. In production pipelines, prefer the explicit .option("mergeSchema", "true") so that schema changes are visible in your code.
overwriteSchema: Destructive Replacement
overwriteSchema is a fundamentally different operation. Combined with mode("overwrite"), it replaces the entire table schema with the incoming DataFrame's schema. It's not additive — it's a replacement. Use it when you're doing a full reload of a table and the new source schema is authoritative.
# Full reload of a dimension table with a completely redesigned schema
new_customer_dim = spark.read.parquet("Files/landing/customers_full_reload.parquet")
new_customer_dim.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.save("Tables/bronze_customers")
overwriteSchema will succeed even if the incoming DataFrame has removed columns, changed types in incompatible ways, or reordered columns. Because you're doing a full overwrite, the transaction log will record this as a metaData action replacing the schema plus a series of remove actions for the old Parquet files and add actions for the new ones.
Warning
overwriteSchema bypasses all schema enforcement. It's powerful and potentially dangerous. If a pipeline accidentally passes the wrong DataFrame to an overwrite operation, you've just wiped the schema and all data of that table. In production, restrict overwriteSchema to explicitly controlled, manually reviewed operations, or wrap it in validation logic that confirms the incoming schema matches an expected contract before proceeding.
ALTER TABLE for Column Operations
For deterministic, controlled schema changes — the kind you're making intentionally, not in response to a DataFrame mismatch — use ALTER TABLE SQL or its PySpark equivalent. This is the right approach for silver and gold tables where you're making deliberate structural decisions.
Adding Columns
# Add a single column
spark.sql("""
ALTER TABLE silver_erp_orders
ADD COLUMN discount_code STRING
""")
# Add multiple columns at once
spark.sql("""
ALTER TABLE silver_erp_orders
ADD COLUMNS (
discount_pct DOUBLE,
campaign_id LONG
)
""")
New columns added via ALTER TABLE ADD COLUMN are always nullable and default to NULL for all existing rows. You cannot add a NOT NULL column via ALTER TABLE on a table that already contains data — because the existing rows would immediately violate the constraint. We'll cover the pattern for handling this correctly in the constraints section.
Renaming Columns
Delta Lake supports column renaming without rewriting data, via column mapping. To use it, the table needs to have column mapping mode enabled:
# Enable column mapping (required for rename and drop)
spark.sql("""
ALTER TABLE silver_erp_orders
SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5'
)
""")
# Now you can rename
spark.sql("""
ALTER TABLE silver_erp_orders
RENAME COLUMN discount_code TO promo_code
""")
Column mapping works by adding a layer of indirection: the physical Parquet files store columns by their original field IDs, while the Delta schema maps logical names to those IDs. A rename only updates the mapping in the transaction log — no Parquet files are rewritten. This is elegant and efficient.
Note
Once you enable column mapping on a table, you cannot disable it. Also, tables with column mapping enabled have higher minimum reader/writer version requirements (versions 2 and 5 respectively). Make sure all tools and Spark versions in your environment support these versions before enabling column mapping on production tables. In Microsoft Fabric's current Spark runtime, this is supported.
Dropping Columns
spark.sql("""
ALTER TABLE silver_erp_orders
DROP COLUMN campaign_id
""")
Like renaming, dropping a column with column mapping enabled doesn't rewrite the underlying Parquet files. The column is simply removed from the logical schema definition in the transaction log, and the physical data is left in place (and becomes inaccessible until a VACUUM runs and cleans up the files — but those files are now orphaned from the schema's perspective).
Merging DataFrames with Incompatible Schemas
The mergeSchema write option handles additive differences well. But what happens when you have genuinely incompatible schemas — different column names for the same concept, overlapping column names with different types, or completely different structures that need to be unified?
This comes up constantly in the bronze-to-silver transition. Bronze tables often hold data from multiple source system versions, and your silver transformation needs to produce a consistent output regardless of which source version the input rows came from.
The Union Problem
Suppose your bronze table holds two years of sales data. In year one, the source system called the product identifier product_code (STRING). In year two, after a system migration, it became item_sku (STRING). Both columns exist in the bronze table after a mergeSchema operation — older rows have product_code populated and item_sku NULL, newer rows have the reverse.
Your silver transformation needs to produce a canonical product_identifier column that's populated regardless of which era the row comes from:
from pyspark.sql.functions import coalesce, col, when, to_date, current_timestamp
from delta.tables import DeltaTable
bronze_df = spark.read.format("delta").load("Tables/bronze_sales_orders")
silver_df = bronze_df \
.withColumn(
"product_identifier",
coalesce(col("item_sku"), col("product_code"))
) \
.withColumn(
"order_date_canonical",
# Some batches had date as string, some as proper date
when(col("order_date").isNotNull(), col("order_date").cast("date"))
.otherwise(to_date(col("order_date_str"), "MM/dd/yyyy"))
) \
.withColumn("silver_processed_at", current_timestamp()) \
.select(
col("order_id"),
col("customer_id"),
col("product_identifier"),
col("order_date_canonical").alias("order_date"),
col("order_total"),
col("discount_code"),
col("silver_processed_at")
)
This approach — canonical column construction in the silver transformation — is the right pattern. Silver's job is not to preserve the messiness of bronze; it's to rationalize it into a consistent schema that downstream consumers can rely on.
Key insight
The silver layer's schema should be defined by your business requirements, not by whatever the source system sent. Design the silver schema explicitly, write it down, enforce it. When bronze changes, your transformation code adapts the bronze data to the silver schema — the silver schema itself changes only when the business requirements change, not when the source system changes.
Handling Type Conflicts During Union
Sometimes you're doing a UNION ALL of DataFrames from different sources that have the same column name but different types. Spark will reject this unless you explicitly cast:
from pyspark.sql.functions import col
from pyspark.sql.types import LongType, DecimalType
# Region A data: order_id is INT, order_total is FLOAT
region_a = spark.read.format("delta").load("Tables/bronze_orders_region_a") \
.withColumn("order_id", col("order_id").cast(LongType())) \
.withColumn("order_total", col("order_total").cast(DecimalType(18, 2))) \
.withColumn("source_region", lit("A"))
# Region B data: order_id is LONG, order_total is DECIMAL(18,2) -- matches target
region_b = spark.read.format("delta").load("Tables/bronze_orders_region_b") \
.withColumn("source_region", lit("B"))
# Now union is safe because types match
combined = region_a.union(region_b)
A cleaner approach for production is to define a target schema explicitly and use it to cast both DataFrames:
from pyspark.sql.types import StructType, StructField, LongType, DecimalType, StringType, DateType
# Define the canonical silver schema
silver_orders_schema = StructType([
StructField("order_id", LongType(), nullable=False),
StructField("customer_id", LongType(), nullable=False),
StructField("order_date", DateType(), nullable=True),
StructField("order_total", DecimalType(18, 2), nullable=True),
StructField("source_region", StringType(), nullable=False),
])
def conform_to_silver_schema(df, target_schema):
"""Cast and select columns to match the target schema."""
return df.select([
col(field.name).cast(field.dataType).alias(field.name)
for field in target_schema.fields
if field.name in df.columns
])
region_a_conformed = conform_to_silver_schema(region_a, silver_orders_schema)
region_b_conformed = conform_to_silver_schema(region_b, silver_orders_schema)
combined = region_a_conformed.union(region_b_conformed)
This pattern makes schema conformance explicit and testable. You can write unit tests around conform_to_silver_schema that verify it handles type mismatches correctly, which is much harder when the conformance logic is buried inside a long transformation chain.
The challenge: what happens when your incoming DataFrame for a MERGE operation contains a new column that doesn't exist in the target table?
By default, MERGE does not support mergeSchema. If the incoming DataFrame has a column that the target table doesn't know about, the MERGE will fail with an AnalysisException. You have to handle this explicitly:
from delta.tables import DeltaTable
# Step 1: Check if the target table schema needs to be updated first
target_table = DeltaTable.forName(spark, "silver_erp_orders")
target_columns = set(f.name for f in target_table.toDF().schema.fields)
source_columns = set(f.name for f in incremental_df.schema.fields)
new_columns = source_columns - target_columns
# Step 2: Add any new columns to the target before the merge
if new_columns:
for col_name in new_columns:
col_type = incremental_df.schema[col_name].dataType.simpleString()
spark.sql(f"""
ALTER TABLE silver_erp_orders
ADD COLUMN {col_name} {col_type}
""")
print(f"Added new columns to silver_erp_orders: {new_columns}")
# Step 3: Now the merge can proceed safely
target_table = DeltaTable.forName(spark, "silver_erp_orders") # Refresh the reference
(
target_table.alias("target")
.merge(
incremental_df.alias("source"),
"target.order_id = source.order_id"
)
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
Warning
The pattern above — automatically adding new columns before a merge — is convenient but potentially dangerous in production. If the source system accidentally sends a column with a misleading name (or a completely wrong type), you'll add it to your silver table permanently. Consider adding a whitelist of allowed columns, or requiring explicit approval before any new column is added to silver or gold tables.
As of Delta Lake 2.0+, you can also enable schema evolution directly in merge operations using a Spark configuration:
spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", "true")
# With autoMerge enabled, whenMatchedUpdateAll() and whenNotMatchedInsertAll()
# will automatically handle new source columns
(
target_table.alias("target")
.merge(
incremental_df.alias("source"),
"target.order_id = source.order_id"
)
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
Check the Delta Lake version in your Fabric runtime to confirm this is supported. In Fabric's Spark runtimes based on Delta Lake 2.x and above, it works reliably.
Enforcing Constraints Across Medallion Layers
Schema evolution handles structure. Constraints handle semantics. Delta Lake supports two types of constraints on tables: CHECK constraints and NOT NULL constraints. Both are enforced at write time by the Delta writer — they're not enforced by Spark's planner, so they apply regardless of which engine writes to the table.
NOT NULL Constraints
NOT NULL constraints prevent NULL values from being written to a column. In the context of schema evolution, they're most valuable on silver and gold tables where you've established a data contract:
# Enforce that order_id and customer_id are never null in the silver layer
spark.sql("""
ALTER TABLE silver_erp_orders
ALTER COLUMN order_id SET NOT NULL
""")
spark.sql("""
ALTER TABLE silver_erp_orders
ALTER COLUMN customer_id SET NOT NULL
""")
If you try to write a DataFrame containing NULLs in order_id to silver_erp_orders after this constraint is set, you'll get:
DeltaInvariantViolationException: NOT NULL constraint violated for column: order_id
Tip
Before setting a NOT NULL constraint on an existing table, always verify that the column has no existing NULLs. Run spark.sql("SELECT COUNT(*) FROM silver_erp_orders WHERE order_id IS NULL").show() first. If there are existing NULLs, setting the constraint will succeed (Delta only validates new writes, not existing data) but your table will already be in a state that violates its own contract — which is misleading and potentially confusing. Either clean the data first, or accept that the constraint applies only going forward.
CHECK Constraints
CHECK constraints are expressions that must evaluate to true for every row written to the table. They're the Delta equivalent of database check constraints, and they're remarkably useful for catching data quality issues at the pipeline layer before they propagate downstream.
# Order totals must be non-negative
spark.sql("""
ALTER TABLE silver_erp_orders
ADD CONSTRAINT order_total_non_negative
CHECK (order_total >= 0)
""")
# Order date must be within a reasonable range (not future-dated beyond 1 year)
spark.sql("""
ALTER TABLE silver_erp_orders
ADD CONSTRAINT order_date_reasonable
CHECK (order_date >= '2000-01-01' AND order_date <= date_add(current_date(), 365))
""")
# Status must be one of the known values
spark.sql("""
ALTER TABLE silver_erp_orders
ADD CONSTRAINT status_valid_values
CHECK (order_status IN ('PENDING', 'CONFIRMED', 'SHIPPED', 'DELIVERED', 'CANCELLED'))
""")
Constraints are stored in the table's properties in the transaction log. You can view them:
spark.sql("SHOW TBLPROPERTIES silver_erp_orders") \
.filter("key LIKE 'delta.constraints.%'") \
.show(truncate=False)
To drop a constraint:
spark.sql("""
ALTER TABLE silver_erp_orders
DROP CONSTRAINT order_date_reasonable
""")
Constraint Strategy Across Medallion Layers
Not all layers should carry the same constraints. Here's a framework that works well in practice:
Bronze layer: Minimal constraints. Bronze is raw data — the whole point is to land everything and figure out quality later. The only constraint worth considering on bronze is a NOT NULL on the primary key field, if you're certain the source will always send one.
Silver layer: Business rule constraints. This is where you enforce the semantic contract. NOT NULL on business keys, CHECK constraints on value ranges, status enumerations, and referential consistency where you can express it as a single-table check.
Gold layer: Reporting constraints. Gold tables should be as clean as possible. Add NOT NULL constraints on all dimension keys and measure columns. Consider adding constraints that enforce aggregation invariants — for example, that a daily sales total is always greater than zero if the table only records days with activity.
# Gold layer: stricter constraints for reporting
spark.sql("""
ALTER TABLE gold_daily_sales_summary
ALTER COLUMN sale_date SET NOT NULL
""")
spark.sql("""
ALTER TABLE gold_daily_sales_summary
ALTER COLUMN total_revenue SET NOT NULL
""")
spark.sql("""
ALTER TABLE gold_daily_sales_summary
ADD CONSTRAINT revenue_positive
CHECK (total_revenue > 0)
""")
spark.sql("""
ALTER TABLE gold_daily_sales_summary
ADD CONSTRAINT units_positive
CHECK (total_units_sold > 0)
""")
This layered approach means that a data quality problem caught by a silver CHECK constraint failure is a signal that your transformation logic needs to be fixed. A gold constraint failure is a signal that something fundamentally wrong made it through silver — which should be a loud alert.
Schema Drift Detection: Building Proactive Guards
Reactive constraint failures are useful, but you want to catch schema drift before it causes a pipeline failure. Building a lightweight schema drift detector into your pipeline is straightforward and pays dividends.
import json
from pyspark.sql.types import StructType
def get_table_schema(table_name: str) -> dict:
"""Returns the current schema of a Delta table as a dict."""
schema = spark.read.format("delta").table(table_name).schema
return json.loads(schema.json())
def detect_schema_drift(source_df, target_table_name: str) -> dict:
"""
Compare source DataFrame schema against target table schema.
Returns a dict with 'new_columns', 'removed_columns', and 'type_changes'.
"""
target_schema = spark.read.format("delta").table(target_table_name).schema
source_schema = source_df.schema
target_fields = {f.name: f.dataType.simpleString() for f in target_schema.fields}
source_fields = {f.name: f.dataType.simpleString() for f in source_schema.fields}
new_cols = {k: v for k, v in source_fields.items() if k not in target_fields}
removed_cols = {k: v for k, v in target_fields.items() if k not in source_fields}
type_changes = {
k: {"from": target_fields[k], "to": source_fields[k]}
for k in source_fields
if k in target_fields and source_fields[k] != target_fields[k]
}
return {
"new_columns": new_cols,
"removed_columns": removed_cols,
"type_changes": type_changes,
"has_drift": bool(new_cols or removed_cols or type_changes)
}
# Use it in your pipeline before writing
incoming_batch = spark.read.parquet("Files/landing/erp_orders_latest.parquet")
drift_report = detect_schema_drift(incoming_batch, "bronze_erp_orders")
if drift_report["has_drift"]:
print("Schema drift detected!")
print(f"New columns: {drift_report['new_columns']}")
print(f"Removed columns: {drift_report['removed_columns']}")
print(f"Type changes: {drift_report['type_changes']}")
# Decide how to handle based on drift type
if drift_report["type_changes"]:
# Type changes are risky - alert and stop
raise ValueError(
f"Incompatible type changes detected: {drift_report['type_changes']}. "
"Manual review required before proceeding."
)
if drift_report["new_columns"]:
# New columns are additive - log and proceed with mergeSchema
print(f"Proceeding with mergeSchema to add: {drift_report['new_columns']}")
incoming_batch.write \
.format("delta") \
.mode("append") \
.option("mergeSchema", "true") \
.saveAsTable("bronze_erp_orders")
else:
# Clean write - no special options needed
incoming_batch.write \
.format("delta") \
.mode("append") \
.saveAsTable("bronze_erp_orders")
You can integrate this drift detection into orchestrated pipelines. When you're orchestrating loads with Fabric Data Pipelines, you can call a Notebook activity that runs drift detection and returns a result, then use an If Condition activity to branch the pipeline based on whether drift was detected.
Schema Governance Patterns for Medallion Pipelines
At scale, schema evolution isn't just a technical problem — it's a governance problem. You need a system that tracks what the schema is supposed to be, detects when reality diverges from expectation, and provides a controlled process for evolving schemas intentionally.
Schema-as-Code with Version Control
Store your canonical table schemas as code in your version control system. The cleanest approach is to maintain a schema registry — a Python module or JSON file that defines the expected schema for each medallion table:
When your transformation notebook runs, it imports the schema registry and enforces the output against it:
from schema_registry import SCHEMAS
expected_schema = SCHEMAS["silver_erp_orders"]
# Transform...
silver_df = transform_bronze_to_silver(bronze_df)
# Validate output schema before writing
actual_fields = {f.name: f.dataType for f in silver_df.schema.fields}
expected_fields = {f.name: f.dataType for f in expected_schema.fields}
schema_violations = []
for field_name, expected_type in expected_fields.items():
if field_name not in actual_fields:
schema_violations.append(f"Missing column: {field_name}")
elif actual_fields[field_name] != expected_type:
schema_violations.append(
f"Type mismatch for {field_name}: "
f"expected {expected_type}, got {actual_fields[field_name]}"
)
if schema_violations:
raise ValueError(f"Silver output schema violations: {schema_violations}")
# Now write with confidence
silver_df.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "false") \ # Never accidentally overwrite
.saveAsTable("silver_erp_orders")
This approach dovetails well with Fabric Git integration. Your schema registry lives in Git alongside your notebook code, schema changes go through pull requests, and you get a full history of schema evolution decisions with the reasoning captured in commit messages and PR comments.
Impact Analysis Before Schema Changes
Before making a schema change to any silver or gold table, do impact analysis: which reports, semantic models, and downstream tables depend on this table, and specifically on the columns you're changing?
In Microsoft Fabric, you can start this analysis by looking at lineage in the workspace. But for column-level impact, you'll need to search your notebook and pipeline code. Specifically when serving data through Direct Lake semantic models, renaming or dropping a gold column will immediately break the semantic model, because Direct Lake reads the column list at framing time. The model won't error until a user runs a query that touches the missing column — which is a terrible user experience.
Tip
When you need to rename a column that's referenced in a Direct Lake semantic model, use a two-phase approach: add the new column name alongside the old one (via a view or a computed column in a second transformation), update the semantic model to reference the new name, confirm it works, and only then drop the old column. Never rename a column and update the semantic model simultaneously in a single deployment — if the semantic model update fails for any reason, you've already broken production.
Hands-On Exercise
In this exercise, you'll simulate a realistic schema evolution scenario across a three-layer medallion lakehouse. You'll handle both additive and breaking changes, enforce constraints on the silver table, and build a drift detection mechanism.
Setup: In your Fabric lakehouse, open a Spark notebook and run the following to create the initial state:
Step 1: Simulate an additive schema change in the source. The source system has added two new columns: item_sku (replacing product_code) and discount_pct:
You should see a DeltaInvariantViolationException because order_total = -50.00 violates the order_total_non_negative constraint.
Common Mistakes & Troubleshooting
Mistake 1: Using overwriteSchema when you meant mergeSchema
These two options are easy to confuse when you're writing quickly. overwriteSchema + overwrite blows away the entire table. mergeSchema + append adds columns and appends data. Always double-check which mode you're using before running on production tables.
Mistake 2: Setting NOT NULL on columns with existing nulls
As mentioned earlier, Delta does not retroactively validate existing data when you add a constraint. You can set NOT NULL on a column that already has nulls, and Delta won't complain — until someone queries it and gets confused by why a "NOT NULL" column has nulls. Always clean the data first, or document that the constraint applies only to future writes.
Mistake 3: Forgetting to refresh DeltaTable references after schema changes
If you hold a DeltaTable object in memory and then ALTER TABLE to add columns, the in-memory reference still reflects the old schema. Subsequent operations using that reference may fail or produce unexpected results. Always get a fresh DeltaTable.forName() reference after schema changes.
Mistake 4: Type widening failures during mergeSchema
You can write an INT column into a LONG column via mergeSchema because that's a safe widening. But you cannot use mergeSchema to go from LONG to INT, DOUBLE to FLOAT, or STRING to any numeric type. These will fail even with mergeSchema enabled. You need overwriteSchema (with a full table rebuild) for these changes — which means carefully managing backward compatibility with all existing readers.
Warning
If a source system changes a column type in an incompatible direction — for example, changing an order_id from INT to STRING because they changed their ID format — you cannot handle this with mergeSchema. You need to either: (a) cast the column in your ingestion logic before writing to bronze, or (b) create a new column with the new type, keep the old column, and handle both in your silver transformation. Option (a) is cleaner but requires you to know about the change in advance. Option (b) is more robust to surprises.
Mistake 5: Constraints on MERGE targets failing silently
When you use MERGE INTO on a table with constraints, and the merge produces a row that violates a constraint, the entire merge operation fails with an exception. This is correct behavior — but if your merge has 100,000 rows and one of them violates a constraint, all 100,000 rows are rejected. Build pre-merge validation to catch constraint violations before they become pipeline failures:
# Validate before merge
violations = incremental_df.filter(
col("order_total") < 0
)
if violations.count() > 0:
# Log the violations, quarantine the rows, and proceed with clean rows only
violations.write.format("delta").mode("append").saveAsTable("silver_orders_quarantine")
incremental_df = incremental_df.filter(col("order_total") >= 0)
print(f"Quarantined {violations.count()} rows with negative order totals.")
If you see this error, it means the table's schema changed between when Spark analyzed your query plan and when it tried to execute it. This can happen in long-running jobs or when multiple writers are modifying a table concurrently. The fix is to re-read the table after any schema change, rather than caching a reference to the old DataFrame.
Summary & Next Steps
Schema evolution in Delta tables is not a single operation — it's a discipline. You've learned how Delta's schema enforcement works at the protocol level, the difference between mergeSchema (additive, safe) and overwriteSchema (destructive, powerful), and how to use ALTER TABLE for deterministic schema changes. You've built drift detection logic that catches surprises before they become pipeline failures, and you've applied CHECK and NOT NULL constraints strategically across medallion layers to enforce data quality where it matters most.
The key architectural takeaway is this: bronze accepts schema change gracefully (via mergeSchema with drift detection), silver enforces a business-defined canonical schema (via explicit transformation and output validation), and gold carries strict constraints that protect the reporting layer. When a source system changes, the change is absorbed at bronze and rationalized at silver — gold stays stable.
Schema evolution is one of those topics where experience compounds. The first time a source system blindsides you with a breaking type change, you'll be glad you built the detection and governance patterns now.
