You've inherited a data pipeline that was built in 2019. It runs nightly, ingests 40 million rows from three source systems, lands data in a Snowflake schema that twelve downstream teams depend on, and it works — sort of. The transformation logic is embedded in a 900-line SQL script that nobody fully understands anymore, the orchestration layer is a cron job on an EC2 instance someone nicknamed "the oracle," and two of the original engineers have moved on. You've been asked to migrate everything to dbt and Airflow, modernize the schema, and do it without causing a single downstream dashboard to break or a single row of data to go missing.
This is the scenario that separates data engineers who can build from data engineers who can operate. Building a new pipeline is relatively straightforward. Safely decomposing a live one while something depends on it is a surgical problem — and most engineers underestimate it until they've caused a production incident that wipes out a weekend.
By the end of this lesson, you'll have a complete framework for deprecating legacy pipelines with confidence. You'll understand how to audit what you're replacing, how to run parallel pipelines safely, how to validate equivalence between old and new outputs, and how to structure the cutover so that even if something goes wrong, you have a rollback path that doesn't require all-hands incident response at 2 AM.
What you'll learn:
This lesson assumes you're comfortable with:
You don't need to be an expert in dbt or any specific tool — the patterns here are tool-agnostic.
The most dangerous assumption in pipeline migration is that you know who depends on your output. You almost certainly don't have the full picture. Before writing a single line of new code, you need to build a complete dependency map.
Start with the obvious sources: your data catalog, your team's Slack history, JIRA tickets. But these are unreliable. The real dependency map lives in the query logs.
Every major data warehouse exposes query history. Here's how to mine it in Snowflake:
-- Find every query that has read from your legacy table in the last 90 days
SELECT
query_id,
user_name,
role_name,
database_name,
schema_name,
query_text,
start_time,
end_time,
total_elapsed_time
FROM snowflake.account_usage.query_history
WHERE
start_time >= DATEADD(day, -90, CURRENT_TIMESTAMP())
AND query_text ILIKE '%orders_legacy.fact_orders%'
AND query_type IN ('SELECT', 'INSERT', 'CREATE', 'MERGE')
ORDER BY start_time DESC;
Run variations of this for every table, view, and external stage the legacy pipeline writes to. The ILIKE pattern matching isn't perfect — someone might reference the table through a view — so you also need to chase view definitions:
-- Find all views that reference your legacy table
SELECT
table_catalog,
table_schema,
table_name,
view_definition
FROM information_schema.views
WHERE view_definition ILIKE '%orders_legacy.fact_orders%';
Do this recursively. If view A depends on your table, and view B depends on view A, view B is also your dependent. In practice, you'll find chains three or four levels deep in mature systems.
In BigQuery, the equivalent audit uses INFORMATION_SCHEMA.JOBS:
SELECT
user_email,
job_id,
creation_time,
query
FROM `region-us`.INFORMATION_SCHEMA.JOBS
WHERE
creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
AND query LIKE '%legacy_dataset.fact_orders%'
AND job_type = 'QUERY';
Once you have raw query log data, you need to structure it. Build a spreadsheet or a simple database table with these columns:
| Consumer | Type | Owner | Frequency | SLA Sensitivity | Migration Status |
|---|---|---|---|---|---|
| finance_dashboard | Looker | finance-analytics@company.com | Hourly | Critical | Not Started |
| orders_agg_view | Snowflake View | data-platform@company.com | On-demand | High | Not Started |
| monthly_revenue.py | Python ETL | data-science@company.com | Monthly | Medium | Not Started |
The SLA Sensitivity column is something you'll need to assess by talking to owners, not by guessing. A "Critical" designation means that if this consumer breaks, someone gets paged. A "Medium" means it can be broken for a day before anyone notices. This column determines your sequencing during cutover.
Warning: Query logs have a retention window. In Snowflake's free account-usage views, it's typically 365 days, but enterprise configurations vary. If your pipeline is older than the retention window, you may have consumers that haven't queried recently but will during end-of-month or annual reporting cycles. Ask explicitly about periodic batch consumers.
Beyond "who reads the table," you need to document what they're relying on. This includes:
Schema contracts: Column names, data types, nullable vs. not-null constraints. A consumer might have hardcoded SELECT order_id, customer_id, revenue_usd — if you rename revenue_usd to order_revenue, they break silently and get wrong numbers rather than an error.
Behavioral contracts: Row-level semantics that consumers have come to rely on. Does the table include cancelled orders? Does it deduplicate by the latest update timestamp, or is there one row per event? These things are rarely documented; you find them by reading consumer code.
Timing contracts: When does the table update? A consumer that runs at 8 AM expecting yesterday's complete data has a timing contract with your pipeline. If your new pipeline runs at 9 AM, you've broken them even if the data is identical.
Document all three for every consumer before you proceed.
Once you understand what exists, you build the replacement. But you don't cut over immediately — you run both pipelines simultaneously, writing outputs to different targets, and you compare them. This is the parallel run phase.
The goal is to have the legacy pipeline and the new pipeline both running in production, writing to separate schemas or datasets, while downstream consumers continue reading from the legacy output. The new pipeline's output is the "dark" output — it exists, it's being validated, but nothing depends on it yet.
Here's a conceptual architecture for Airflow:
Legacy DAG (still running):
orders_legacy_etl → writes → snowflake.orders_legacy.fact_orders
New DAG (running in parallel):
orders_v2_etl → writes → snowflake.orders_v2.fact_orders
Reconciliation DAG (new):
reconciliation_check → reads both → compares → alerts on divergence
The critical discipline here is: do not let the new pipeline write to the legacy schema during the parallel run. The temptation is to "just test it" by writing to the same place. Resist. Write to a completely separate schema with a separate name. This preserves the legacy output as the authoritative source until you explicitly cut over.
Your new pipeline should run after the legacy pipeline each day during the parallel run. This is important because you want to compare the two on the same source snapshot — if the new pipeline runs first and the source data changes before the legacy pipeline runs, you'll see false discrepancies.
In Airflow, you can enforce this with a dependency chain:
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.sensors.external_task import ExternalTaskSensor
from datetime import datetime, timedelta
with DAG(
dag_id='orders_v2_etl',
schedule_interval='0 4 * * *', # 4 AM, legacy runs at 2 AM
start_date=datetime(2024, 1, 1),
catchup=False,
) as dag:
wait_for_legacy = ExternalTaskSensor(
task_id='wait_for_legacy_completion',
external_dag_id='orders_legacy_etl',
external_task_id='final_write_task',
timeout=7200, # 2 hour timeout
poke_interval=60,
mode='reschedule',
)
run_new_pipeline = PythonOperator(
task_id='run_orders_v2',
python_callable=run_orders_v2_pipeline,
)
wait_for_legacy >> run_new_pipeline
This pattern ensures the new pipeline only runs after the legacy one has completed successfully. If the legacy pipeline fails, the new one doesn't run either — which is the right behavior during the parallel run phase.
This is the heart of safe migration. You need automated checks that definitively answer: "Are the old and new outputs equivalent?" The answer is rarely "yes" on the first run, and that's fine — the whole point is to find discrepancies in a controlled environment before they become production incidents.
Think about reconciliation as having three layers, each catching a different class of problem.
Layer 1: Aggregate Counts and Sums
The bluntest instrument, but you'd be surprised how often this is where you find problems:
-- Run this after each parallel run cycle
WITH legacy_agg AS (
SELECT
DATE(order_created_at) AS order_date,
COUNT(*) AS row_count,
COUNT(DISTINCT customer_id) AS distinct_customers,
SUM(order_revenue_usd) AS total_revenue,
SUM(CASE WHEN order_status = 'completed' THEN 1 ELSE 0 END) AS completed_orders
FROM snowflake.orders_legacy.fact_orders
WHERE order_created_at >= DATEADD(day, -7, CURRENT_DATE())
GROUP BY 1
),
new_agg AS (
SELECT
DATE(order_created_at) AS order_date,
COUNT(*) AS row_count,
COUNT(DISTINCT customer_id) AS distinct_customers,
SUM(order_revenue_usd) AS total_revenue,
SUM(CASE WHEN order_status = 'completed' THEN 1 ELSE 0 END) AS completed_orders
FROM snowflake.orders_v2.fact_orders
WHERE order_created_at >= DATEADD(day, -7, CURRENT_DATE())
GROUP BY 1
)
SELECT
COALESCE(l.order_date, n.order_date) AS order_date,
l.row_count AS legacy_rows,
n.row_count AS new_rows,
l.row_count - n.row_count AS row_diff,
l.total_revenue AS legacy_revenue,
n.total_revenue AS new_revenue,
ROUND(ABS(l.total_revenue - n.total_revenue) / NULLIF(l.total_revenue, 0) * 100, 4) AS revenue_pct_diff
FROM legacy_agg l
FULL OUTER JOIN new_agg n ON l.order_date = n.order_date
WHERE ABS(l.row_count - COALESCE(n.row_count, 0)) > 0
OR ABS(l.total_revenue - COALESCE(n.total_revenue, 0)) > 0.01
ORDER BY order_date DESC;
Tip: Don't just check for differences — check for acceptable thresholds. A difference of 0.001% in revenue might be explained by floating-point rounding in the new pipeline's computation. Define your acceptance criteria before you run the first comparison, not after. "Is this difference acceptable?" is much harder to answer objectively when you're staring at data at midnight.
Layer 2: Row-Level Keyed Comparison
Aggregate checks pass but row-level distributions diverge. This catches problems like: the new pipeline correctly counts the same total revenue, but it's attributing different revenue to different orders (wrong joins, wrong deduplication logic).
-- Find orders that exist in one table but not the other
WITH legacy_keys AS (
SELECT order_id, order_created_at::DATE AS order_date
FROM snowflake.orders_legacy.fact_orders
WHERE order_created_at >= DATEADD(day, -7, CURRENT_DATE())
),
new_keys AS (
SELECT order_id, order_created_at::DATE AS order_date
FROM snowflake.orders_v2.fact_orders
WHERE order_created_at >= DATEADD(day, -7, CURRENT_DATE())
)
SELECT
'in_legacy_only' AS status,
l.order_id,
l.order_date
FROM legacy_keys l
LEFT JOIN new_keys n ON l.order_id = n.order_id
WHERE n.order_id IS NULL
UNION ALL
SELECT
'in_new_only' AS status,
n.order_id,
n.order_date
FROM new_keys n
LEFT JOIN legacy_keys l ON n.order_id = l.order_id
WHERE l.order_id IS NULL
ORDER BY order_date DESC, status;
For rows that do exist in both, compare key metric columns:
-- Compare metric values for matching rows
SELECT
l.order_id,
l.order_revenue_usd AS legacy_revenue,
n.order_revenue_usd AS new_revenue,
ABS(l.order_revenue_usd - n.order_revenue_usd) AS abs_diff,
l.order_status AS legacy_status,
n.order_status AS new_status,
l.customer_id AS legacy_customer,
n.customer_id AS new_customer
FROM snowflake.orders_legacy.fact_orders l
INNER JOIN snowflake.orders_v2.fact_orders n ON l.order_id = n.order_id
WHERE
l.order_created_at >= DATEADD(day, -7, CURRENT_DATE())
AND (
ABS(l.order_revenue_usd - n.order_revenue_usd) > 0.001
OR l.order_status != n.order_status
OR l.customer_id != n.customer_id
)
LIMIT 1000;
Layer 3: Schema and Type Validation
This one gets forgotten until it bites you. Your new pipeline might produce identical data, but if a column changes from VARCHAR(50) to VARCHAR(255), downstream BI tools with strict typing will break.
import snowflake.connector
from dataclasses import dataclass
from typing import List
@dataclass
class ColumnSchema:
name: str
data_type: str
is_nullable: bool
character_maximum_length: int | None
def get_table_schema(conn, database: str, schema: str, table: str) -> List[ColumnSchema]:
cursor = conn.cursor()
cursor.execute(f"""
SELECT
column_name,
data_type,
is_nullable,
character_maximum_length
FROM {database}.information_schema.columns
WHERE table_schema = '{schema.upper()}'
AND table_name = '{table.upper()}'
ORDER BY ordinal_position
""")
return [
ColumnSchema(row[0], row[1], row[2] == 'YES', row[3])
for row in cursor.fetchall()
]
def compare_schemas(legacy: List[ColumnSchema], new: List[ColumnSchema]) -> List[str]:
issues = []
legacy_map = {col.name: col for col in legacy}
new_map = {col.name: col for col in new}
# Check for missing columns (consumers will break)
for col_name in legacy_map:
if col_name not in new_map:
issues.append(f"MISSING COLUMN in new schema: {col_name}")
# Check for type changes (may break consumers silently)
for col_name, legacy_col in legacy_map.items():
if col_name in new_map:
new_col = new_map[col_name]
if legacy_col.data_type != new_col.data_type:
issues.append(
f"TYPE CHANGE for {col_name}: "
f"{legacy_col.data_type} -> {new_col.data_type}"
)
if legacy_col.is_nullable != new_col.is_nullable:
issues.append(
f"NULLABILITY CHANGE for {col_name}: "
f"nullable={legacy_col.is_nullable} -> nullable={new_col.is_nullable}"
)
return issues
Reconciliation should run automatically and alert you to failures. Don't make it a manual process — you'll skip it when you're busy, and that's exactly when something will slip through.
def run_reconciliation_check(**context):
conn = get_snowflake_connection()
# Run aggregate check
agg_discrepancies = run_query(conn, AGGREGATE_RECONCILIATION_SQL)
# Run row-level check
row_discrepancies = run_query(conn, ROW_LEVEL_RECONCILIATION_SQL)
# Run schema check
legacy_schema = get_table_schema(conn, 'PROD', 'ORDERS_LEGACY', 'FACT_ORDERS')
new_schema = get_table_schema(conn, 'PROD', 'ORDERS_V2', 'FACT_ORDERS')
schema_issues = compare_schemas(legacy_schema, new_schema)
if agg_discrepancies or len(row_discrepancies) > ACCEPTABLE_ROW_DISCREPANCY_THRESHOLD:
raise ValueError(
f"Reconciliation FAILED:\n"
f"Aggregate discrepancies: {len(agg_discrepancies)}\n"
f"Row-level discrepancies: {len(row_discrepancies)}\n"
f"Schema issues: {schema_issues}"
)
# Store reconciliation results for trending
store_reconciliation_results(
run_date=context['ds'],
agg_discrepancies=len(agg_discrepancies),
row_discrepancies=len(row_discrepancies),
schema_issues=schema_issues
)
Warning: Set
ACCEPTABLE_ROW_DISCREPANCY_THRESHOLDdeliberately, but set it low. The temptation is to set it high enough that the check always passes, which defeats the purpose. A threshold of 0 is ideal during the first week of parallel running. After you've triaged and explained every discrepancy, you can raise it to account for known acceptable differences.
After running parallel for long enough to trust the new output (typically 2–4 weeks for daily pipelines, longer for pipelines with monthly or quarterly batch consumers), you're ready to cut over. There are several patterns for doing this, and the right one depends on your consumers.
If all your consumers read from a view rather than directly from a table, you have a nearly zero-downtime cutover option. The view becomes an indirection layer:
-- Before cutover: view points to legacy table
CREATE OR REPLACE VIEW orders_production.fact_orders AS
SELECT * FROM orders_legacy.fact_orders;
-- After cutover: view points to new table
CREATE OR REPLACE VIEW orders_production.fact_orders AS
SELECT * FROM orders_v2.fact_orders;
This CREATE OR REPLACE VIEW is atomic in Snowflake, BigQuery, and Redshift — there's no gap where the view doesn't exist. Consumers reading from orders_production.fact_orders never see a break.
If your current consumers are reading directly from the legacy table (not a view), you should introduce this view indirection layer before the parallel run, not during cutover. Get consumers migrated to the view first. Yes, this means a preliminary migration before the main migration — but it pays for itself a hundredfold by simplifying the actual cutover.
For cases where you're changing schema and can't maintain a 1:1 mapping through a view, you need to migrate consumers one by one while both the legacy and new tables remain live.
Build a migration tracker:
CREATE TABLE migrations.pipeline_consumer_status (
consumer_name VARCHAR(255),
consumer_type VARCHAR(50), -- 'dashboard', 'etl', 'view', 'report'
owner_email VARCHAR(255),
legacy_dependency VARCHAR(500),
new_dependency VARCHAR(500),
migration_status VARCHAR(50), -- 'pending', 'in_progress', 'validated', 'cutover', 'failed'
migration_date DATE,
notes TEXT
);
Migrate consumers in order of SLA sensitivity — least critical first. This is counterintuitive but correct. Your least critical consumers are your test bed. If something goes wrong with their migration, it's a recoverable situation. Migrating your most critical consumer first is how you create a 3 AM incident.
For each consumer, the process is:
cutover in your trackerFor high-stakes pipelines where even the view swap feels too risky, use a killswitch pattern. The new pipeline writes to the new location, but consuming systems have a configuration flag that determines which source they read from:
import os
def get_orders_table() -> str:
"""
Returns the appropriate fact_orders table based on migration flag.
Allows instant rollback by toggling ORDERS_MIGRATION_ACTIVE.
"""
use_new_pipeline = os.environ.get('ORDERS_MIGRATION_ACTIVE', 'false').lower() == 'true'
if use_new_pipeline:
return 'orders_v2.fact_orders'
else:
return 'orders_legacy.fact_orders'
# In your ETL/query code:
table_name = get_orders_table()
query = f"SELECT * FROM {table_name} WHERE ..."
Store this flag in a configuration system (AWS Parameter Store, HashiCorp Vault, a simple database table, even a Git-controlled YAML file) rather than hardcoding it. This gives you a rollback that takes seconds: flip the flag, and every consumer immediately falls back to the legacy output.
Tip: Document your killswitch. Write a runbook that says: "If the new pipeline produces bad data after cutover, change
ORDERS_MIGRATION_ACTIVEtofalsein AWS Parameter Store at this path, then run this command to confirm propagation." The person who needs the runbook at 2 AM might not be you.
Choose your cutover window deliberately. The worst time is at the end of a month or quarter, when reporting pipelines are running. The best time is:
Cutting over is not decommissioning. Many engineers make the mistake of treating cutover as the end of the job. The legacy pipeline is still running. You're paying compute and storage costs to maintain it. And it's creating an ongoing source of confusion: if someone queries the legacy table and finds data that diverges from the new table, they'll raise an incident.
After cutover, run the legacy pipeline in a read-only monitoring mode for a defined period — typically 30 days for daily pipelines. During this period:
The keep-warm period gives you a fast rollback path. If a consumer reports a data quality issue with the new pipeline 3 weeks post-cutover, you can compare against the legacy output to diagnose.
After the keep-warm period, you begin the actual shutdown sequence:
# Airflow: pause the legacy DAG programmatically
from airflow.api.client.local_client import Client
client = Client(None, None)
client.set_dag_paused(dag_id='orders_legacy_etl', is_paused=True)
Pausing is not deleting. Keep the legacy DAG definition in version control but paused for another 30 days. This lets you restart it in an emergency without a deployment cycle.
Don't delete legacy tables immediately after decommissioning. Instead, move them to an archive schema:
-- Create an archive of the legacy table at the cutover date
CREATE TABLE orders_archive.fact_orders_legacy_final AS
SELECT * FROM orders_legacy.fact_orders;
-- Add metadata
ALTER TABLE orders_archive.fact_orders_legacy_final
ADD COLUMN _archive_date DATE DEFAULT CURRENT_DATE();
ADD COLUMN _migration_id VARCHAR(50) DEFAULT 'ORDERS_V2_MIGRATION_2024';
-- Set a retention policy (Snowflake data retention)
ALTER TABLE orders_archive.fact_orders_legacy_final
SET DATA_RETENTION_TIME_IN_DAYS = 90;
This archive becomes your safety net for forensic queries. If someone comes to you 60 days after decommissioning and says "the revenue numbers for Q3 look different from what we reported last year," you can pull from the archive to compare and diagnose whether the difference is a real data quality issue or an expected result of the migration's semantic changes.
Pipeline migration is as much a change management problem as it is an engineering problem. Teams that don't understand why the migration is happening, or who haven't been consulted, will resist it — and resistance from downstream teams can block you indefinitely.
Send a migration notice at the beginning of the parallel run, not at cutover. This is the first message downstream teams should receive:
Subject: [Action Required] orders pipeline migration — your action needed by [DATE]
We're migrating the orders data pipeline to a new infrastructure. Here's what this means for you:
- What's changing: [specific schema changes, timing changes]
- What's not changing: [columns/behavior that remain identical]
- Your deadline: Your workloads need to point to the new schema by [DATE — 2 weeks before hard cutover]
- How to migrate: [link to runbook]
- Who to contact: [you, with response SLA]
- What happens if you miss the deadline: [honest answer — legacy table goes read-only on [date], deleted on [date]]
The key element here is honesty about consequences. If you say "the legacy table will be deleted on [date]" and then don't delete it when teams miss the deadline, you lose credibility and create an indefinitely-maintained legacy asset.
You will encounter a team that says they can't migrate before your deadline. Take this seriously — don't dismiss it as obstruction. Ask them to quantify the constraint: What is the specific blocker? What is the earliest they can migrate? Is it a resource problem or a technical incompatibility?
If it's a genuine technical incompatibility (their system requires a column you're removing), you have options:
If it's a resource problem, escalate to their manager and yours. This is a legitimate organizational decision about priorities. But make clear that "we'll handle it later" is not an answer — "we'll handle it by [specific date]" is the only acceptable form of the conversation.
Warning: Never extend a migration deadline indefinitely. Each extension signals that the deadline wasn't real, which makes future deadlines less credible. If you give an extension, it should be the last one, and it should be documented formally.
This exercise builds a complete parallel-run and reconciliation system for a realistic scenario.
Scenario: You have a legacy pipeline that populates sales_legacy.daily_sales_summary, a table that aggregates daily sales by product category and region. You've built a replacement in sales_v2.daily_sales_summary using dbt. Both pipelines are running in parallel. Your job is to build and run the reconciliation system.
Setup: Create two tables in your data warehouse with slightly different data (simulate a known discrepancy):
-- Legacy table (authoritative during parallel run)
CREATE TABLE sales_legacy.daily_sales_summary AS
SELECT
current_date - seq4() AS sale_date,
CASE MOD(seq4(), 4)
WHEN 0 THEN 'Electronics'
WHEN 1 THEN 'Clothing'
WHEN 2 THEN 'Home & Garden'
ELSE 'Books'
END AS category,
CASE MOD(seq4(), 3)
WHEN 0 THEN 'Northeast'
WHEN 1 THEN 'Southeast'
ELSE 'West'
END AS region,
ROUND(UNIFORM(1000, 50000, RANDOM())::FLOAT, 2) AS total_revenue,
UNIFORM(50, 500, RANDOM())::INT AS order_count
FROM TABLE(GENERATOR(ROWCOUNT => 90)); -- 90 days of data
-- New table (introduce a systematic 2% revenue discrepancy for January to simulate a bug)
CREATE TABLE sales_v2.daily_sales_summary AS
SELECT
sale_date,
category,
region,
CASE
WHEN MONTH(sale_date) = 1 THEN total_revenue * 1.02 -- simulated bug
ELSE total_revenue
END AS total_revenue,
order_count
FROM sales_legacy.daily_sales_summary;
Step 1: Write and run the aggregate reconciliation query. Identify which month shows discrepancies.
Step 2: Write a row-level query that identifies the specific date/category/region combinations where revenue differs by more than 0.5%.
Step 3: Write a Python function that runs both reconciliation queries and raises an exception with a descriptive message if any discrepancies exceed your threshold.
Step 4: Simulate the view swap cutover. Create a production view, point it at the legacy table, verify a consumer query works, then swap it to point at the v2 table. Verify the consumer query returns the v2 data without any modification to the consumer query.
Step 5: Write the archival SQL that would snapshot the legacy table with metadata columns before decommissioning.
Expected outcome: After completing this exercise, you should have caught the January revenue discrepancy in Step 1/2, seen the exception message in Step 3, and confirmed the view swap pattern works atomically in Step 4. The discrepancy you "fixed" (by removing the * 1.02 bug) should make Step 3 pass cleanly.
If the legacy pipeline has ongoing flakiness — intermittent failures, manual correction steps, known data quality issues — those will contaminate your reconciliation results. You'll spend weeks chasing discrepancies that turn out to be legacy pipeline bugs, not problems with your new pipeline.
Fix: Before starting the parallel run, document the legacy pipeline's known issues and expected failure modes. If the legacy pipeline fails on the 1st of each month due to a known source system issue, note that. Your reconciliation checks need to account for this.
Teams often run reconciliation manually or only when there's a suspected problem. Then they do a two-week parallel run with no issues, cut over, and discover a discrepancy in the monthly reporting cycle that wasn't present in the daily data.
Fix: Run reconciliation every day during the parallel run, and explicitly add checks that cover the boundary conditions your consumers care about — end-of-month aggregates, weekly rollups, anything that has a periodicity beyond daily.
You spend 3 weeks validating the parallel run against the legacy schema. On cutover day, you run a last-minute "improvement" to the new schema — rename a column, add an index, adjust a data type. Now your reconciliation baseline is stale.
Fix: Freeze the new pipeline schema at the start of the parallel run period. Any schema changes after that point require restarting the reconciliation clock. This feels like bureaucracy but it's what keeps you honest.
New pipelines often have different backfill behavior than legacy ones. If the legacy pipeline was idempotent (rerunning it on the same day produces the same output) but the new pipeline accumulates duplicates on reruns, you won't catch this during normal forward-running reconciliation.
Fix: As part of your reconciliation suite, explicitly test idempotency. Run the new pipeline twice for the same date and verify the output is identical to a single run.
def test_idempotency(pipeline_fn, target_table: str, execution_date: str):
"""Run the pipeline twice for the same date, check results are identical."""
pipeline_fn(execution_date=execution_date)
first_run_count = get_row_count(target_table, execution_date)
pipeline_fn(execution_date=execution_date)
second_run_count = get_row_count(target_table, execution_date)
assert first_run_count == second_run_count, (
f"Pipeline is NOT idempotent: "
f"first run={first_run_count}, second run={second_run_count}"
)
After a successful cutover, there will be pressure to clean up. "Just delete the old stuff, it's done." Resist this. The 30-day keep-warm period exists because you will always have at least one consumer who was somehow not on your dependency list, or a quarterly report that runs once every 3 months and nobody mentioned.
Fix: Put the decommissioning date in your migration tracker and don't let it be moved up. Move the legacy resources to an archive state (read-only, separate cost center) rather than deleting them immediately.
During reconciliation, you will find discrepancies that are correct. The legacy pipeline had a bug, and the new pipeline fixed it. Revenue is $50 higher in the new pipeline because the legacy pipeline was incorrectly excluding a tax component. Is this a discrepancy? Yes. Is it a problem? Depends on whether you want to preserve the bug.
Fix: Document every discrepancy you find, and classify it explicitly as:
Only when every discrepancy has a classification — and stakeholders have signed off on the intentional changes — should you proceed to cutover.
Pipeline deprecation is a discipline that combines the rigor of software engineering with the diplomacy of organizational change management. The framework we've built through this lesson gives you a repeatable playbook:
The difference between an engineer who causes migration incidents and one who doesn't usually isn't technical skill — it's whether they treated the deprecation process with the same rigor as the new pipeline build.
Next Steps:
To build on what you've learned here, explore these related topics:
Learning Path: Data Pipeline Fundamentals