Understanding Data Pipeline Triggers: Time-Based, Event-Driven, and Sensor Patterns

Introduction

Imagine you're a data engineer at a retail company. Every morning at 8 AM, your boss wants a sales dashboard refreshed with yesterday's numbers. Every time a customer places an order, your fraud detection model needs to run immediately. And your pipeline that reads from a vendor's SFTP server should only kick off after the vendor has actually dropped their file — not before. These are three completely different problems, and they each require a different answer to the same fundamental question: what causes a pipeline to run?

That question — what triggers a pipeline — turns out to be one of the most consequential design decisions you'll make as a data engineer. Get it wrong, and you end up with pipelines that run too often (wasting compute and money), too rarely (leaving stakeholders with stale data), or at the wrong time (reading a file before it's fully written). Get it right, and your data platform hums along reliably, responding to the real rhythm of your business.

In this lesson, you'll develop a genuine understanding of the three major trigger patterns used in modern data pipelines: time-based scheduling, event-driven triggers, and sensors. By the end, you'll be able to look at a real business scenario and confidently choose the right trigger strategy — and you'll understand deeply why that choice matters.

What you'll learn:

• What a pipeline trigger is and why it's a separate concern from pipeline logic
• How time-based (cron) scheduling works and when to use it
• What event-driven triggers are and how they differ from scheduled runs
• How sensors act as a bridge between scheduling and events
• How to choose the right trigger pattern for a given scenario

---

Prerequisites

This lesson assumes you're familiar with the general concept of a data pipeline — a series of steps that moves, transforms, or loads data from one place to another. You don't need experience with any specific orchestration tool like Apache Airflow, Prefect, or dbt Cloud, but familiarity with at least one of these will help you put the patterns into practice quickly. Basic comfort with reading code snippets in Python or YAML is helpful but not required.

---

What Is a Trigger, Exactly?

Before we dive into the three types, let's make sure we're grounded on what a trigger actually is.

A trigger is the mechanism that tells your pipeline orchestrator, "now is the time to start a run." Think of it as the ignition switch for your pipeline. The pipeline itself — all the extraction, transformation, and loading logic — is like a car engine. The trigger is what turns the key.

This separation matters. The logic of how your pipeline works is independent of when it runs. A pipeline that calculates daily revenue totals doesn't care whether it was triggered by a clock, by a message on a queue, or by a human clicking a button. The trigger is a separate concern, and treating it that way is what allows you to build flexible, reusable pipeline components.

Orchestration tools like Apache Airflow, Prefect, Dagster, and dbt Cloud all have first-class support for different trigger types. The concepts we'll cover here are universal — the specific syntax differs between tools, but the underlying patterns are identical.

---

Time-Based Triggers: Running on a Schedule

How Cron Scheduling Works

The oldest and most common trigger pattern is the time-based schedule, and it almost always involves something called a cron expression — a compact string that describes a recurring time pattern.

Cron (short for "chronograph") originated in Unix systems as a way to schedule jobs. A cron expression has five fields:

┌───────── minute (0–59)
│ ┌─────── hour (0–23)
│ │ ┌───── day of month (1–31)
│ │ │ ┌─── month (1–12)
│ │ │ │ ┌─ day of week (0–6, Sunday=0)
│ │ │ │ │
* * * * *

An asterisk (*) means "every." So * * * * * means "every minute of every hour of every day" — which is almost certainly too frequent for any real pipeline. Here are some realistic examples:

# Run every day at 6:00 AM UTC
0 6 * * *

# Run every Monday at 9:00 AM UTC
0 9 * * 1

# Run every 15 minutes
*/15 * * * *

# Run at midnight on the first day of every month
0 0 1 * *

In Apache Airflow, you define this directly in your DAG (Directed Acyclic Graph — Airflow's term for a pipeline):

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def load_daily_sales():
    # Your pipeline logic here
    print("Loading yesterday's sales data...")

with DAG(
    dag_id="daily_sales_load",
    schedule_interval="0 6 * * *",   # Every day at 6 AM UTC
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:
    load_task = PythonOperator(
        task_id="load_sales",
        python_callable=load_daily_sales,
    )

The catchup=False parameter is important here — it tells Airflow not to run the pipeline for every past date since start_date. Without it, Airflow would helpfully try to "catch up" on all the runs it missed, which is rarely what you want when first deploying a new pipeline.

When Time-Based Triggers Make Sense

Time-based scheduling is the right choice when the passage of time itself is the meaningful event. Ask yourself: does this data refresh make sense at a specific point in time, regardless of what else has happened in the world?

Good candidates for time-based triggers:

• Daily business reports: Your finance team wants a revenue summary every morning. The "trigger" is the start of the business day.
• Periodic data syncs: You pull inventory data from an ERP system every hour because the ERP doesn't emit events — it's a batch system.
• Regulatory reporting: You generate a compliance snapshot at the end of each quarter, no exceptions.

The Hidden Pitfalls of Pure Scheduling

Here's the subtle danger of time-based scheduling: a scheduled pipeline assumes the data will be ready when the schedule fires. If your 6 AM pipeline depends on a vendor delivering a file by 5:45 AM, what happens when the vendor is 30 minutes late?

The pipeline runs, finds no file (or finds yesterday's file), and either crashes or silently loads stale data. You won't know until your stakeholders notice the dashboard is wrong.

Warning: Time-based triggers are reliable for the schedule, but they make no guarantees about data readiness. If your pipeline depends on external data sources, a time-based trigger alone is fragile. You'll likely need to combine it with a sensor (more on those shortly).

---

Event-Driven Triggers: Reacting to What Happens

The Shift in Mental Model

Event-driven triggers flip the model entirely. Instead of asking "what time should this run?", you ask "what happening should cause this to run?"

An event is any discrete occurrence that your system can detect: a file landing in cloud storage, a message arriving in a message queue, a row being inserted into a database table, an API call completing, or even a human clicking a "Run Now" button in a dashboard.

The key characteristic of event-driven pipelines is that they respond to the state of the world, not the state of the clock. This makes them inherently more accurate — the pipeline runs when the data is actually ready, not when you hope it'll be ready.

A Real-World Event-Driven Architecture

Let's make this concrete. Suppose you work at a financial services company, and each time a loan application is submitted through the web app, you need to:

1. Pull the applicant's credit bureau data
2. Run a risk-scoring model
3. Write the result back to the application database

This workflow should happen within seconds of submission, not at 3 AM in a batch job. The trigger here is the event of a loan application being created.

A common architecture for this uses a message queue — a system like AWS SQS, Google Pub/Sub, or Apache Kafka that acts as a holding area for event notifications. When the web app creates a loan application, it publishes a message to the queue. Your pipeline infrastructure subscribes to that queue and starts a new run for each message.

In a modern cloud environment using AWS, this might look like:

# AWS Lambda function triggered by SQS message
import json
import boto3

def handle_loan_application(event, context):
    for record in event['Records']:
        # Parse the incoming message
        message = json.loads(record['body'])
        application_id = message['application_id']
        
        print(f"Processing loan application: {application_id}")
        
        # Step 1: Pull credit bureau data
        credit_data = fetch_credit_data(application_id)
        
        # Step 2: Run risk scoring
        risk_score = run_risk_model(credit_data)
        
        # Step 3: Write result back
        write_risk_score(application_id, risk_score)

Here, AWS Lambda is configured to trigger automatically whenever a message arrives in an SQS queue. The pipeline doesn't poll the queue on a timer — the cloud infrastructure handles the invocation the moment an event occurs.

Event-Driven Triggers in Orchestration Tools

Even if you're using a traditional orchestrator like Airflow or Prefect rather than a serverless architecture, most modern tools support event-based triggering.

In Prefect, you can define an automation that triggers a flow run when an external event is detected:

# Prefect flow that can be triggered externally
from prefect import flow

@flow(name="process-new-customer-file")
def process_customer_file(file_path: str):
    print(f"Processing file: {file_path}")
    # Pipeline logic here

You'd then configure a Prefect Automation in the UI (navigate to Automations → Create Automation → choose "External Event" as the trigger) or via the API to call this flow when, say, a new file lands in S3.

When Event-Driven Triggers Make Sense

Use event-driven triggers when:

• Latency matters: Fraud detection, real-time recommendations, and alerting systems all need to react within seconds or minutes, not hours.
• Volume is variable and unpredictable: If you process e-commerce orders, some days bring 100 orders and others bring 100,000. Event-driven systems scale naturally with volume; time-based systems have no concept of volume.
• The trigger is a state change in another system: A new file, a completed upstream job, a new database record — these are all events, not times.

Tip: Event-driven architectures tend to be more complex to build and debug than scheduled ones. If your latency requirement is "available by morning," a simple daily schedule is often a better choice than a complex event-driven system. Match the tool to the actual need.

---

Sensor Patterns: The Bridge Between Worlds

What Is a Sensor?

Now we come to perhaps the most practically useful pattern for working data engineers: the sensor.

A sensor is a special type of pipeline step that waits and watches for a specific condition to become true before allowing the pipeline to continue. It sits at the beginning of your pipeline (or between two stages) and does nothing except poll — check, wait, check, wait — until the condition it's looking for is satisfied.

Think of a sensor like a bouncer outside a nightclub who checks IDs. The line keeps moving only when the bouncer is satisfied. No exceptions, no guessing.

Sensors are the practical answer to the problem we identified with time-based triggers: "what if the data isn't ready yet?" Instead of assuming data readiness, you verify it.

Common Sensor Types

In Apache Airflow (which has one of the most mature sensor libraries), the built-in sensors include:

• FileSensor: Waits until a file exists at a given path
• S3KeySensor: Waits until a specific key (file) appears in an S3 bucket
• ExternalTaskSensor: Waits until another Airflow DAG or task has completed successfully
• SqlSensor: Waits until a SQL query returns a non-empty result
• HttpSensor: Waits until an HTTP endpoint returns a successful response

Here's a realistic Airflow example where a pipeline should load a daily sales file from an SFTP server — but only after the file has actually arrived:

from airflow import DAG
from airflow.sensors.filesystem import FileSensor
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def load_sales_file():
    print("File confirmed present. Loading sales data...")
    # Extract, transform, and load logic here

with DAG(
    dag_id="sftp_sales_pipeline",
    schedule_interval="0 4 * * *",   # Starts checking at 4 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
    },
) as dag:

    wait_for_file = FileSensor(
        task_id="wait_for_sales_file",
        filepath="/data/inbound/sales_{{ ds }}.csv",   # ds = execution date
        poke_interval=300,    # Check every 5 minutes
        timeout=7200,         # Give up after 2 hours
        mode="reschedule",    # Don't hold a worker slot while waiting
    )

    load_data = PythonOperator(
        task_id="load_sales_data",
        python_callable=load_sales_file,
    )

    # The sensor must succeed before loading begins
    wait_for_file >> load_data

This pipeline starts checking at 4 AM. Every 5 minutes, the FileSensor looks for that day's file. The moment the file appears, execution proceeds to the load step. If the file never appears within 2 hours, the pipeline fails with a clear timeout error — which is far better than silently loading the wrong data.

Notice the mode="reschedule" setting. This is crucial in production. Without it, the sensor would hold a worker slot the entire time it's waiting — wasting resources across your entire platform. With reschedule mode, the sensor releases its slot between checks and is re-queued every poke_interval seconds.

Sensors as Quality Gates

Sensors aren't only for file detection. They're powerful data quality gates — checkpoints that enforce conditions before a pipeline proceeds.

Imagine a pipeline that calculates month-end revenue metrics. You don't just want to check whether a file exists — you want to verify that all transactions for the month have been posted. A SqlSensor can handle this:

from airflow.sensors.sql import SqlSensor

wait_for_month_close = SqlSensor(
    task_id="wait_for_month_end_close",
    conn_id="postgres_prod",
    sql="""
        SELECT 1
        FROM accounting_control
        WHERE period = '{{ macros.ds_format(ds, "%Y-%m-%d", "%Y-%m") }}'
          AND status = 'CLOSED'
    """,
    poke_interval=600,    # Check every 10 minutes
    timeout=14400,        # Wait up to 4 hours
    mode="reschedule",
)

This sensor polls the accounting_control table until someone (or another automated process) has marked the period as CLOSED. Only then does the revenue calculation run. This is not something a simple time-based trigger could ever handle reliably.

---

Choosing the Right Trigger: A Decision Framework

Now that you understand all three patterns, let's build a mental model for choosing between them. Work through these questions in order:

1. Does latency matter — measured in seconds or minutes? If yes → event-driven trigger. Time-based and sensor patterns introduce delays measured in minutes at minimum.

2. Can you predict when the data will be ready? If yes, reliably → time-based schedule may be sufficient. If no, or only approximately → you need a sensor.

3. Is the trigger an external system change or file arrival? If yes → sensor (combined with a time-based schedule to begin the sensing window).

4. Is the volume of events unpredictable? If yes → event-driven trigger, possibly with a message queue for buffering.

Most real pipelines use combinations of these patterns. A common production pattern: a time-based schedule starts a DAG every morning at 4 AM, the first task is a sensor that waits for upstream data to land, and once it does, the pipeline fires off the downstream processing. You get the predictability of a schedule with the correctness guarantee of a sensor.

---

Hands-On Exercise

Work through these scenarios and choose the most appropriate trigger strategy. Write out your reasoning before checking the analysis below.

Scenario 1: A marketing analytics team needs a weekly email campaign performance report every Monday morning. The data comes from a well-established email platform API that is reliably updated by Sunday midnight.

Scenario 2: An IoT platform receives sensor readings from 10,000 factory machines. Each reading needs to be processed and stored in a time-series database within 2 seconds of arrival.

Scenario 3: Your data warehouse has a nightly loading pipeline. It depends on three upstream pipelines finishing successfully: an ERP extract, a CRM sync, and a web analytics pull. Each of those three takes a variable amount of time.

Analysis:

Scenario 1: Time-based trigger. The data is reliably ready by a known time, latency isn't critical, and the rhythm is predictable (weekly). A cron of 0 6 * * 1 (6 AM every Monday) works perfectly.

Scenario 2: Event-driven trigger with a message queue (Kafka is ideal here for throughput). Each sensor reading is an event. A time-based trigger would be wildly inappropriate — you can't poll 10,000 machines on a 1-second schedule without massive infrastructure. Let the events come to you.

Scenario 3: Time-based trigger combined with ExternalTaskSensor patterns. The nightly warehouse load starts at a reasonable time (say, 2 AM), but uses sensors to wait for each of the three upstream pipelines to report success before proceeding. This guards against any of the three running long on a given night.

---

Common Mistakes & Troubleshooting

Mistake: Setting sensor poke_interval too aggressively Checking for a file every 5 seconds seems more responsive, but at scale, hundreds of sensors each polling every 5 seconds will hammer your metadata database and downstream APIs. Set poke_interval conservatively — 1 to 5 minutes for most file sensors is appropriate.

Mistake: Not setting a sensor timeout Without a timeout, a stuck sensor waits forever, holding resources (or preventing a task slot from being freed for other work). Always set a timeout that corresponds to your SLA — if the data hasn't arrived within your acceptable window, you want to know immediately, not discover it days later.

Mistake: Using event-driven triggers for low-frequency, low-latency-tolerant pipelines Event-driven systems are complex to build, monitor, and debug. If your pipeline runs once a day and can wait until morning, a simple cron schedule is dramatically easier to operate. Complexity has a cost.

Mistake: Ignoring time zones in cron expressions Cron expressions are typically interpreted in the orchestrator's server time zone, which is often UTC. If your stakeholders expect a report "every morning at 8 AM New York time," that's 0 13 * * * in UTC during EST and 0 12 * * * during EDT (daylight saving). Use your tool's timezone settings explicitly rather than doing the math manually — Airflow, for example, supports a timezone parameter per DAG.

Mistake: Treating the trigger as part of the pipeline logic When someone asks "why did this pipeline run?", the answer should always be traceable to the trigger configuration — not buried inside the pipeline code. Keep triggers as explicit configuration, not hidden conditionals inside your ETL scripts.

---

Summary & Next Steps

You now have a solid, practical understanding of the three foundational trigger patterns in data engineering:

• Time-based triggers run pipelines on a fixed schedule using cron expressions. They're simple, predictable, and perfect when time itself is the meaningful event — but they assume data readiness, which is often an unsafe assumption.
• Event-driven triggers start pipelines in response to something that happened in another system. They're the right choice when latency matters or when data volume is unpredictable, but they add architectural complexity.
• Sensors wait and watch for a specific condition to be true before allowing a pipeline to proceed. They're the practical workaround for the weaknesses of pure time-based scheduling and serve as powerful data quality gates.

The most important takeaway isn't which pattern to memorize — it's the habit of always asking what should actually cause this pipeline to start? That question alone will prevent an enormous class of reliability problems.

Next steps to deepen this knowledge:

• Explore your orchestration tool's sensor library. If you're using Airflow, browse the apache-airflow-providers packages for sensors specific to S3, GCS, Snowflake, dbt, and others.
• Practice writing cron expressions using a tool like crontab.guru, which visualizes what a cron expression will actually do before you deploy it.
• Read about idempotency in data pipelines — the property that running a pipeline twice produces the same result as running it once. Trigger design and idempotency are deeply connected: if you might trigger a pipeline more than once accidentally, idempotent logic is your safety net.
• Investigate backfilling strategies in your orchestrator — understanding how Airflow's catchup parameter and Prefect's backfill runs work will help you handle re-processing scenarios confidently.