Time-based scheduling breaks down when multi-team pipelines have variable latency. Learn how to implement Airflow's Dataset-Aware Scheduling to decouple producer and consumer DAGs across teams — so pipelines run when data is ready, not when a cron fires. Includes complete producer/consumer patterns, URI contract management, and production debugging techniques.

Picture this: your analytics engineering team maintains a suite of dbt models that transform raw Salesforce data into clean revenue metrics. Downstream of that, the finance team runs a separate DAG that builds board-level reports. And downstream of that, a third team triggers a reverse ETL job that pushes those numbers back into Salesforce for sales rep dashboards. Each team owns their pipeline. Each team has different cadences, different SLAs, and different on-call rotations.
The traditional solution — time-based cron scheduling — works until it doesn't. You set the finance DAG to run at 06:00 UTC, hoping the dbt models finish by then. Some days they do. Some days a Snowflake warehouse was suspended and the dbt run took forty minutes longer. The finance report runs on stale data, nobody notices until the CFO asks why the numbers don't match, and now you're doing forensic archaeology in Airflow logs at 9 PM.
Airflow's Dataset-Aware Scheduling, introduced in Airflow 2.4 and significantly improved through 2.5 and 2.6, solves this problem at the architectural level. Instead of scheduling DAGs by time, you schedule them by data. A DAG declares what datasets it produces; another DAG declares what datasets it consumes. Airflow handles the dependency graph between teams without requiring shared codebases, shared schedules, or any coordination overhead beyond naming conventions. By the end of this lesson, you'll know how to design and implement that system across a real multi-team pipeline scenario.
What you'll learn:
Dataset objects and the schedule parameterDatasetOrTimeSchedule and @conditional patternsYou should be comfortable with Airflow fundamentals — DAG structure, operators, task dependencies, and the scheduler — before diving in. If you need a refresher, Data Pipeline Orchestration with Airflow covers the core concepts well. You should also have a working understanding of dbt in an orchestrated context; Orchestrating dbt Runs with Airflow: Scheduling, Dependencies, and Error Handling in Production is directly relevant to the examples here. This lesson targets Airflow 2.4+ — dataset-aware scheduling does not exist in earlier versions.
Before writing any code, it's worth being precise about what Airflow Datasets actually are, because the abstraction is deceptively simple and easy to misuse.
An Airflow Dataset is a URI — a string — that represents a logical data asset. That's it. Airflow doesn't inspect the actual data behind the URI. It doesn't check if a file exists in S3, if a table has rows in Snowflake, or if a dbt model completed successfully. What it tracks is simpler and more reliable: whether a task that declared it was producing that URI has completed successfully.
from airflow.datasets import Dataset
# These are all valid Dataset URIs
salesforce_opportunities = Dataset("snowflake://analytics/raw/salesforce_opportunities")
revenue_metrics = Dataset("snowflake://analytics/marts/revenue_metrics")
finance_report = Dataset("s3://company-reports/finance/weekly_board_report.csv")
The URI format is flexible — Airflow doesn't validate it against any real system. What matters is that the string is consistent between the producing task and the consuming DAG. This flexibility is intentional: it lets teams use URIs that are semantically meaningful to their domain without needing Airflow to have credentials to every system.
Key insight
Airflow Datasets are a signaling mechanism, not a data-awareness mechanism. They answer "did this task run and succeed?" not "is there fresh data at this location?" This distinction matters enormously when designing multi-team contracts.
This model has two important implications. First, a dataset is "updated" when the outlet task finishes successfully — not when it starts, not when the downstream data is actually queryable. Second, if a task fails, the dataset is NOT marked as updated, even if data was partially written. This is usually the behavior you want, but it means your producing tasks need to be idempotent and atomic.
You might be wondering how this differs from just using ExternalTaskSensor or cross-DAG task dependencies. The distinction is fundamental:
| Mechanism | Coupling | Cross-team? | Time-based? |
|---|---|---|---|
Task dependencies (>>) |
Tight — same DAG | No | Inherits DAG schedule |
ExternalTaskSensor |
Medium — requires knowing exact DAG/task ID | Possible, fragile | Yes — polls by execution date |
TriggerDagRunOperator |
Medium — imperative trigger | Yes | Producer controls |
| Datasets | Loose — URI contract only | Yes | Consumer controls |
The key advantage of Datasets is loose coupling. The producer DAG doesn't need to know that any consumer exists. The consumer DAG doesn't need to know the producer's DAG ID, task ID, or schedule. They share only a URI string. Teams can rename, restructure, or rewrite their DAGs without breaking dependencies, as long as they maintain the URI contract.
Let's work through a realistic example. The data team runs ingestion and dbt transformations. The finance team runs reporting. The marketing team runs a campaign attribution pipeline. All three are independent teams with separate codebases and separate Airflow connections, but they share an Airflow deployment.
Here's the data engineering team's producer DAG:
# dags/data_team/salesforce_pipeline.py
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.datasets import Dataset
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from airflow.operators.bash import BashOperator
# Define datasets as module-level constants
# Convention: use descriptive, versioned URIs your consumers can reference
RAW_OPPORTUNITIES = Dataset("snowflake://analytics/raw/salesforce_opportunities")
REVENUE_METRICS = Dataset("snowflake://analytics/marts/revenue_metrics")
ATTRIBUTION_STAGING = Dataset("snowflake://analytics/staging/stg_opportunities")
default_args = {
"owner": "data-engineering",
"retries": 2,
"retry_delay": timedelta(minutes=5),
"email_on_failure": True,
"email": ["data-eng-alerts@company.com"],
}
@dag(
dag_id="salesforce_ingestion_and_transform",
schedule="0 4 * * *", # Runs at 4 AM UTC — time-based, owned by this team
start_date=datetime(2024, 1, 1),
catchup=False,
default_args=default_args,
tags=["data-engineering", "salesforce", "producer"],
)
def salesforce_pipeline():
ingest_opportunities = BashOperator(
task_id="ingest_opportunities_fivetran",
bash_command="fivetran trigger --connector-id salesforce_prod --wait",
)
# This task marks RAW_OPPORTUNITIES as updated when it succeeds
validate_raw_data = SnowflakeOperator(
task_id="validate_raw_opportunities",
snowflake_conn_id="snowflake_analytics",
sql="""
SELECT
COUNT(*) as row_count,
COUNT_IF(close_date IS NULL) as null_close_dates,
MAX(ingested_at) as latest_ingestion
FROM raw.salesforce_opportunities
HAVING row_count > 0
AND null_close_dates / row_count < 0.05 -- Fail if >5% null close dates
""",
# Declaring outlets: this task "produces" this dataset when it succeeds
outlets=[RAW_OPPORTUNITIES],
)
run_staging_models = BashOperator(
task_id="run_dbt_staging",
bash_command="""
cd /opt/airflow/dbt_project && \
dbt run --select staging.salesforce --target prod
""",
outlets=[ATTRIBUTION_STAGING],
)
run_mart_models = BashOperator(
task_id="run_dbt_marts",
bash_command="""
cd /opt/airflow/dbt_project && \
dbt run --select marts.revenue --target prod && \
dbt test --select marts.revenue --target prod
""",
# Only emit this dataset if dbt run AND dbt test both pass
outlets=[REVENUE_METRICS],
)
ingest_opportunities >> validate_raw_data >> run_staging_models >> run_mart_models
salesforce_pipeline_dag = salesforce_pipeline()
Notice a few deliberate choices here:
The outlets parameter lives on specific tasks, not on the DAG. This matters because you want granular control. In this example, REVENUE_METRICS is only marked as updated after the dbt mart models and their tests pass. If you declared this at the DAG level, you'd lose that precision.
Dataset objects are defined at module level as constants. This makes them importable by consumer DAGs if you're in a monorepo, and it enforces the contract in one place. If you're in a polyrepo setup (more on this shortly), teams copy the URI string — but defining constants is still good practice within each repo.
The validation task is the gatekeeper. The validate_raw_opportunities task runs quality checks before anything else runs. If this fails, nothing downstream in this DAG runs, and the RAW_OPPORTUNITIES dataset is never marked as updated. Consumer DAGs stay on stale-but-valid data rather than getting fresh-but-broken data.
Tip
Always put your data quality gate before the outlet declaration. If you run dbt models and then test them and the tests fail, you've already updated the dataset — consumers may have already been triggered with bad data. Structure your tasks so the outlet is only emitted after validation succeeds.
Now let's build the downstream consumer DAGs. The finance team's DAG runs their board report pipeline. They don't want to run it on a schedule — they want it to run whenever the revenue metrics are ready.
# dags/finance_team/board_report_pipeline.py
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.datasets import Dataset
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from airflow.operators.python import PythonOperator
# Reference the same URI the data team declared
# In a polyrepo, this string is your cross-team contract — document it!
REVENUE_METRICS = Dataset("snowflake://analytics/marts/revenue_metrics")
default_args = {
"owner": "finance-analytics",
"retries": 1,
"retry_delay": timedelta(minutes=10),
"email_on_failure": True,
"email": ["finance-alerts@company.com"],
}
@dag(
dag_id="finance_board_report",
schedule=[REVENUE_METRICS], # Data-driven: run when this dataset is updated
start_date=datetime(2024, 1, 1),
catchup=False,
default_args=default_args,
tags=["finance", "consumer", "reporting"],
)
def finance_board_report():
build_report_query = SnowflakeOperator(
task_id="build_board_report",
snowflake_conn_id="snowflake_analytics",
sql="""
CREATE OR REPLACE TABLE reporting.finance.board_report AS
SELECT
DATE_TRUNC('month', close_date) as month,
SUM(arr) as total_arr,
COUNT(DISTINCT opportunity_id) as deals_closed,
SUM(arr) / COUNT(DISTINCT account_id) as arr_per_account,
CURRENT_TIMESTAMP() as report_generated_at
FROM analytics.marts.revenue_metrics
WHERE close_date >= DATEADD('month', -13, CURRENT_DATE())
GROUP BY 1
ORDER BY 1
""",
)
@task
def export_to_sheets():
# Push to Google Sheets for CFO review
import gspread
from oauth2client.service_account import ServiceAccountCredentials
# ... actual export logic here
print("Exporting board report to Google Sheets")
@task
def notify_stakeholders():
import requests
requests.post(
"https://hooks.slack.com/services/YOUR/WEBHOOK/HERE",
json={"text": "📊 Board report updated and ready for review in Google Sheets."},
)
build_report_query >> export_to_sheets() >> notify_stakeholders()
finance_board_report_dag = finance_board_report()
The key line is schedule=[REVENUE_METRICS]. That list syntax is deliberate — a consumer DAG can depend on multiple datasets, and by default it won't run until all of them have been updated since the last run. We'll explore multi-dataset dependencies shortly.
Now let's add the marketing team's attribution pipeline, which needs both the raw opportunities data and the staging models:
# dags/marketing_team/campaign_attribution.py
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.datasets import Dataset
from airflow.operators.bash import BashOperator
RAW_OPPORTUNITIES = Dataset("snowflake://analytics/raw/salesforce_opportunities")
ATTRIBUTION_STAGING = Dataset("snowflake://analytics/staging/stg_opportunities")
@dag(
dag_id="marketing_campaign_attribution",
# Both datasets must be updated before this DAG runs
schedule=[RAW_OPPORTUNITIES, ATTRIBUTION_STAGING],
start_date=datetime(2024, 1, 1),
catchup=False,
tags=["marketing", "consumer", "attribution"],
)
def campaign_attribution():
@task
def run_attribution_models():
# This team runs their own dbt project
import subprocess
result = subprocess.run(
["dbt", "run", "--select", "attribution", "--target", "prod"],
cwd="/opt/airflow/marketing_dbt",
capture_output=True,
text=True,
)
if result.returncode != 0:
raise Exception(f"dbt attribution run failed:\n{result.stderr}")
return result.stdout
@task
def update_attribution_dashboard(dbt_output: str):
print(f"Attribution models complete. Triggering dashboard refresh...")
# Tableau or Looker webhook call here
update_attribution_dashboard(run_attribution_models())
campaign_attribution_dag = campaign_attribution()
Note
When a DAG has schedule=[DatasetA, DatasetB], the trigger condition is AND semantics by default: both DatasetA and DatasetB must be updated since the DAG last ran. If your producer emits them in sequence from the same task (or the same DAG run), the consumer triggers once after both are updated. If different producers emit them independently, the consumer waits for all producers to complete at least once since the last trigger.
This AND behavior catches a lot of practitioners off guard. If DatasetA is updated daily and DatasetB is updated weekly, your consumer runs at most weekly — it waits for the slower producer. This is usually correct behavior, but it's worth explicitly designing around.
Here's where the operational reality of multi-team pipelines gets interesting. The URI string is your contract. Break the contract and you break the dependency silently — no error, the consumer just never gets triggered.
There are two deployment patterns:
If all teams share a single Airflow DAGs repository, define Dataset URIs in a shared module:
# dags/shared/datasets.py
# This file is the source of truth for all inter-team data contracts
from airflow.datasets import Dataset
class DataTeamDatasets:
RAW_SALESFORCE_OPPORTUNITIES = Dataset(
"snowflake://analytics/raw/salesforce_opportunities"
)
STAGING_OPPORTUNITIES = Dataset(
"snowflake://analytics/staging/stg_opportunities"
)
REVENUE_METRICS = Dataset(
"snowflake://analytics/marts/revenue_metrics"
)
COST_METRICS = Dataset(
"snowflake://analytics/marts/cost_metrics"
)
class MarketingDatasets:
ATTRIBUTION_RESULTS = Dataset(
"snowflake://analytics/marts/marketing_attribution"
)
CAMPAIGN_SPEND = Dataset(
"snowflake://analytics/staging/stg_campaign_spend"
)
Then import directly:
from shared.datasets import DataTeamDatasets
@dag(schedule=[DataTeamDatasets.REVENUE_METRICS])
def finance_board_report():
...
This approach makes it immediately obvious when someone is about to break a contract — a rename in shared/datasets.py causes import errors in every consumer.
When teams have truly separate repositories (common in larger organizations), you need a different strategy. Document the URI conventions in a data catalog or README, and treat the URI string as a versioned API:
# In finance team's repo - they copy the string but document the source
# Source: data-engineering team, salesforce_pipeline.py
# Contact: data-eng@company.com
# Schema: snowflake://analytics/marts/<mart_name>
REVENUE_METRICS = Dataset("snowflake://analytics/marts/revenue_metrics")
This is exactly the problem that implementing data contracts between ingestion and transformation teams is designed to solve at a broader level. The Dataset URI is essentially a lightweight contract — treat it with the same rigor.
Warning
URI typos are silent killers. If the finance team accidentally writes "snowflake://analytics/mart/revenue_metrics" (singular "mart" instead of "marts"), their DAG will never trigger. Airflow won't warn you. The DAG sits in the UI with a "Dataset" schedule and simply never runs. Always validate your consumer URIs against the producer's declared outlets before deploying.
Pure dataset-driven scheduling is elegant, but real production pipelines have SLAs. The finance team might say: "We want the report triggered by data or by 7 AM — whichever comes first." If the data pipeline fails tonight, we still need to run a report on yesterday's data for the morning standup.
Airflow 2.9 introduced DatasetOrTimeSchedule for exactly this:
from airflow.timetables.datasets import DatasetOrTimeSchedule
from airflow.timetables.trigger import CronTriggerTimetable
from airflow.datasets import Dataset
REVENUE_METRICS = Dataset("snowflake://analytics/marts/revenue_metrics")
@dag(
dag_id="finance_board_report_with_fallback",
schedule=DatasetOrTimeSchedule(
timetable=CronTriggerTimetable("0 7 * * *", timezone="America/New_York"),
datasets=[REVENUE_METRICS],
),
start_date=datetime(2024, 1, 1),
catchup=False,
)
def finance_board_report_with_fallback():
# The DAG now runs when REVENUE_METRICS is updated OR at 7 AM ET
# whichever comes first
...
This gives you a time-based safety net without coupling the teams. The data team doesn't need to know the finance team has a 7 AM fallback, and the finance team gets their SLA guaranteed.
For Airflow 2.4-2.8, you can approximate this behavior with a sensor-based fallback pattern, though it's messier:
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.utils.trigger_rule import TriggerRule
from airflow.operators.empty import EmptyOperator
@dag(
dag_id="finance_board_report_v2",
schedule="0 7 * * *", # Always run at 7 AM
...
)
def finance_board_report_v2():
# Wait up to 2 hours for the data team's DAG to finish
wait_for_revenue = ExternalTaskSensor(
task_id="wait_for_revenue_metrics",
external_dag_id="salesforce_ingestion_and_transform",
external_task_id="run_dbt_marts",
timeout=7200, # 2 hours
poke_interval=300, # Check every 5 minutes
mode="reschedule", # Don't hold a worker slot while waiting
soft_fail=True, # Continue even if sensor times out
)
# Runs regardless of whether the sensor succeeded or timed out
build_report = SnowflakeOperator(
task_id="build_board_report",
trigger_rule=TriggerRule.ALL_DONE, # Run after wait_for_revenue, regardless
...
)
wait_for_revenue >> build_report
The DatasetOrTimeSchedule approach is cleaner and more maintainable — upgrade to 2.9+ if this pattern matters to you.
Understanding how to read Dataset state in the UI is essential for debugging multi-team dependencies. In Airflow 2.4+, navigate to Browse → Datasets in the top menu. This shows you every registered Dataset URI, when it was last updated, and which DAGs produce and consume it.
For each dataset you'll see:
When a consumer DAG is in "waiting" state, click into it. The DAG view shows a Dataset Conditions panel listing each required dataset and whether it's been updated since the last run. This is your first debugging stop when a consumer isn't triggering.
You can also query the Airflow metadata database directly for more sophisticated monitoring:
-- Query Airflow's metadata DB (PostgreSQL) for dataset update history
SELECT
d.uri,
dde.timestamp,
dde.source_task_id,
dde.source_dag_id,
dde.source_run_id
FROM dataset_event dde
JOIN dataset d ON dde.dataset_id = d.id
WHERE d.uri LIKE 'snowflake://analytics/%'
ORDER BY dde.timestamp DESC
LIMIT 50;
This is invaluable for auditing: you can see exactly when each dataset was updated, by which task, and from which DAG run.
Tip
Expose a simple monitoring DAG that alerts on dataset staleness. If REVENUE_METRICS hasn't been updated in 25+ hours on a weekday, something is wrong upstream. This pairs well with the broader observability patterns in Automating Data Freshness SLAs.
Sometimes you need to emit a dataset update from outside a DAG — for example, when an external process completes a data load, or when you want to manually trigger downstream pipelines without re-running the full producer.
Airflow's REST API supports this:
# Manually mark a dataset as updated via the Airflow REST API
curl -X POST \
"http://your-airflow-host/api/v1/datasets/events" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_TOKEN" \
-d '{
"dataset_uri": "snowflake://analytics/marts/revenue_metrics",
"extra": {
"triggered_by": "manual_backfill",
"note": "Backfilling after warehouse maintenance window"
}
}'
You can also do this from within Python using the Airflow client:
from airflow.datasets import Dataset
from airflow.models.dataset import DatasetEvent
from airflow import settings
from sqlalchemy.orm import Session
def manually_emit_dataset_event(uri: str, source_dag_id: str = "manual"):
"""
Programmatically emit a dataset update event.
Useful for external triggers or manual backfills.
"""
dataset = Dataset(uri)
with settings.Session() as session:
from airflow.models.dataset import DatasetModel
dataset_model = session.query(DatasetModel).filter(
DatasetModel.uri == uri
).first()
if dataset_model:
event = DatasetEvent(
dataset_id=dataset_model.id,
source_dag_id=source_dag_id,
source_task_id="manual_trigger",
source_run_id="manual",
extra={"triggered_by": "manual"},
)
session.add(event)
session.commit()
print(f"Dataset event emitted for {uri}")
else:
print(f"Dataset {uri} not found in metadata DB. Ensure a producer DAG has run first.")
Warning
Manually emitting dataset events bypasses the normal producer-consumer flow. Use this sparingly and document it. If you're regularly needing manual triggers, it's a sign that your producer pipeline is unreliable and you should fix that root cause instead.
Let's build a complete three-team pipeline from scratch. You'll need a local Airflow environment (the official Docker Compose setup works well) running Airflow 2.4+.
You're setting up a data pipeline for a SaaS company:
# dags/shared/__init__.py (empty file)
# dags/shared/datasets.py
from airflow.datasets import Dataset
class StripeDatasets:
RAW_PAYMENTS = Dataset("snowflake://analytics/raw/stripe_payments")
RAW_SUBSCRIPTIONS = Dataset("snowflake://analytics/raw/stripe_subscriptions")
class MetricsDatasets:
MRR_METRICS = Dataset("snowflake://analytics/marts/mrr_metrics")
COHORT_BASE = Dataset("snowflake://analytics/intermediate/int_cohort_base")
# dags/data_team/stripe_pipeline.py
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.operators.python import PythonOperator
from shared.datasets import StripeDatasets, MetricsDatasets
import time
@dag(
dag_id="stripe_ingestion_and_metrics",
schedule="0 3 * * *",
start_date=datetime(2024, 1, 1),
catchup=False,
tags=["data-engineering", "stripe", "producer"],
)
def stripe_pipeline():
@task(outlets=[StripeDatasets.RAW_PAYMENTS, StripeDatasets.RAW_SUBSCRIPTIONS])
def ingest_stripe_data():
"""
Simulates Stripe data ingestion.
In production: call Fivetran API or run Airbyte sync.
"""
print("Ingesting Stripe payments...")
time.sleep(2) # Simulated latency
print(f"Ingested 15,432 payment records")
print(f"Ingested 3,891 subscription records")
# Both datasets are marked as updated when this task succeeds
@task
def validate_stripe_data():
"""Quality gate before running transformations."""
print("Validating: checking for nulls, duplicates, schema drift...")
# In production: run great_expectations or dbt tests on raw layer
time.sleep(1)
print("Validation passed.")
@task(outlets=[MetricsDatasets.MRR_METRICS, MetricsDatasets.COHORT_BASE])
def run_dbt_metrics():
"""
Runs dbt models for MRR and cohort base tables.
Marks both datasets as updated on success.
"""
print("Running dbt run --select marts.mrr intermediate.cohort_base...")
time.sleep(3) # Simulated dbt run
print("dbt run completed. 2 models, 0 errors, 12 tests passed.")
ingest = ingest_stripe_data()
validate = validate_stripe_data()
transform = run_dbt_metrics()
ingest >> validate >> transform
stripe_pipeline_dag = stripe_pipeline()
# dags/finance_team/mrr_dashboard.py
from datetime import datetime
from airflow.decorators import dag, task
from shared.datasets import MetricsDatasets
@dag(
dag_id="mrr_dashboard_refresh",
schedule=[MetricsDatasets.MRR_METRICS], # Triggers when MRR is ready
start_date=datetime(2024, 1, 1),
catchup=False,
tags=["finance", "consumer"],
)
def mrr_dashboard():
@task
def refresh_looker_dashboard():
print("Triggering Looker PDT rebuild for MRR dashboard...")
# In production: call Looker API
print("Dashboard refreshed successfully.")
refresh_looker_dashboard()
mrr_dashboard_dag = mrr_dashboard()
# dags/growth_team/cohort_analysis.py
from datetime import datetime
from airflow.decorators import dag, task
from shared.datasets import StripeDatasets, MetricsDatasets
@dag(
dag_id="cohort_analysis_pipeline",
# Requires BOTH raw payments AND MRR metrics to be fresh
schedule=[StripeDatasets.RAW_PAYMENTS, MetricsDatasets.MRR_METRICS],
start_date=datetime(2024, 1, 1),
catchup=False,
tags=["growth", "consumer"],
)
def cohort_analysis():
@task
def build_cohort_tables():
print("Building cohort retention tables...")
print("6-month, 12-month, and 24-month retention calculated.")
@task
def export_to_amplitude(cohort_result=None):
print("Pushing cohort data to Amplitude for product analytics...")
export_to_amplitude(build_cohort_tables())
cohort_analysis_dag = cohort_analysis()
With all three DAGs deployed, trigger the producer manually in the Airflow UI. Navigate to Browse → Datasets and watch the dataset state update as each outlet task completes. You should see mrr_dashboard_refresh trigger automatically after run_dbt_metrics succeeds, and cohort_analysis_pipeline trigger once both ingest_stripe_data and run_dbt_metrics have completed.
Symptom: The consumer DAG has a Dataset schedule but never runs. No errors anywhere.
Diagnosis: Check Browse → Datasets in the UI. Find your URI. If it's not listed at all, the producer task has never successfully run with that outlet declared. If it is listed but the consumer isn't triggering, compare the "Last Updated" timestamp against when the consumer last ran — the update may have happened before the consumer DAG was deployed.
Fix: For newly deployed consumer DAGs, you need the producer to run at least once after the consumer is deployed. Manually trigger the producer or emit a manual dataset event to initialize the state.
Symptom: Similar to above — consumer never triggers.
Root cause:
Dataset("snowflake://analytics/marts/revenue_metrics")Dataset("snowflake://analytics/mart/revenue_metrics") (missing "s")Airflow treats these as completely different datasets. There's no fuzzy matching, no warning.
Fix: Use the shared module pattern from the monorepo approach. If you're in a polyrepo, add a CI check that validates all Dataset URIs against a central registry file.
# WRONG - outlet on a TaskGroup creates ambiguity
with TaskGroup("dbt_models", default_args={"outlets": [REVENUE_METRICS]}) as tg:
task_a = BashOperator(task_id="run_staging", ...)
task_b = BashOperator(task_id="run_marts", ...)
# RIGHT - outlet on the specific final task
task_b = BashOperator(
task_id="run_marts",
outlets=[REVENUE_METRICS],
...
)
If your consumer DAG has catchup=True and a Dataset schedule, Airflow will try to create backfill runs for every dataset update that occurred while the DAG was paused. This can create an unexpected burst of runs. For dataset-driven DAGs, almost always use catchup=False. The scheduling and backfilling patterns discussed in Scheduling and Backfilling Historical Data Loads in Airflow apply differently to dataset-triggered DAGs — be explicit about your intent.
# If task A succeeds but task B fails, only DatasetA is updated
@task(outlets=[DatasetA])
def task_a():
...
@task(outlets=[DatasetB])
def task_b():
... # This fails
task_a() >> task_b()
If task_b fails, DatasetB is not updated. Any consumer waiting for both DatasetA and DatasetB will not trigger. The consumer stays in a waiting state until DatasetB is eventually updated by a successful run. This is usually correct behavior, but it means a failed producer doesn't partially satisfy multi-dataset consumers — which is good for data integrity.
Key insight
The AND semantics of multi-dataset scheduling combined with outlet-only-on-success gives you an implicit quality gate for free. A consumer waiting for three datasets won't run if any one producer had a bad night. This is fundamentally safer than time-based scheduling where the consumer runs regardless.
Airflow Dataset scheduling has overhead — the scheduler polls for dataset updates on its regular cycle (default: 5 seconds with heartbeat, but effectively varies). For pipelines running dozens of times per hour, the scheduling overhead adds up and you'll introduce lag.
For high-frequency pipelines, use task dependencies within a single DAG, or consider a true event-driven orchestrator. The tradeoffs between this approach and streaming architectures are worth reviewing in Real-Time Data: When to Use Streaming vs Batch Processing.
As your data platform evolves, table schemas change. If the finance team depends on REVENUE_METRICS and the data team adds breaking changes to that mart, you need a versioning strategy:
# Version your Dataset URIs for breaking changes
REVENUE_METRICS_V1 = Dataset("snowflake://analytics/marts/revenue_metrics_v1")
REVENUE_METRICS_V2 = Dataset("snowflake://analytics/marts/revenue_metrics_v2")
# Producer emits V2; also emits V1 during migration window
@task(outlets=[REVENUE_METRICS_V1, REVENUE_METRICS_V2])
def run_dbt_marts_migration():
# Run both V1 and V2 models during transition
...
This lets teams migrate at their own pace. The data team coordinates the sunset of V1 once all consumers have migrated to V2.
If your organization runs separate Airflow deployments per team (a common enterprise pattern), native Dataset scheduling won't work across deployments — Dataset state is per-Airflow-metadata-database. In this case, you have two options:
Use the REST API: The producing Airflow instance calls the consuming Airflow instance's REST API to trigger a DAG run directly. Less elegant, but explicit.
Use an external message broker: Both Airflow instances subscribe to a shared Kafka topic or SNS queue. The producer posts a message; the consumer uses a custom sensor to detect it. This aligns with the broader lineage tracking approaches in Multi-Hop Data Lineage Tracking Across the Modern Data Stack.
Dataset state changes are logged in the Airflow metadata DB, but for production observability you want to export these events to your monitoring stack. A lightweight approach:
# Add this to your base producer DAG template
from airflow.listeners import hookimpl
class DatasetEventListener:
@hookimpl
def on_dataset_created(self, dataset):
# Send to Datadog, PagerDuty, or your alerting platform
pass
def on_dataset_changed(self, dataset):
pass
Airflow's plugin system lets you register listeners that fire on dataset events, giving you a hook to integrate with any observability platform.
Airflow Dataset-Aware Scheduling fundamentally changes how multi-team pipelines work. Instead of fragile time-based coordination, you get a system where:
DatasetOrTimeSchedule lets you combine data-driven triggers with time-based safety netsThe pattern scales naturally as your organization grows. New teams add consumer DAGs by referencing existing URIs. New data products add new URIs without touching existing pipelines. The Dataset catalog in Airflow becomes a living map of your data platform's dependencies.
Where to go next:
The natural extension of this work is ensuring the data quality signals that drive your Dataset updates are themselves robust. Implementing dbt Tests and Data Quality Checks in Production Pipelines covers exactly how to structure your dbt testing so that only genuinely valid data updates a Dataset outlet.
If you're working in a multi-project dbt environment, the Dataset model maps cleanly onto dbt Mesh's cross-project reference pattern — Migrating a Monolithic dbt Project to a Multi-Project Architecture with dbt Mesh shows how those architectures align.
And once you have Dataset-aware scheduling running, your pipeline dependency graph is a goldmine for lineage tracking — Multi-Hop Data Lineage Tracking Across the Modern Data Stack shows how to surface that lineage through OpenLineage and Marquez for your entire organization.