Sensors

Analogy categoryAnalogies

What Are Sensors?

A Sensor is a special kind of Operator whose job is to wait for a condition to become true before letting the DAG proceed — for example, waiting for a file to land in S3, a partition to appear in Hive, or an upstream DAG's task to finish. Sensors subclass BaseSensorOperator and implement a poke(context) method that returns True once the condition is met; Airflow calls poke() repeatedly on an interval (poke_interval, default 60 seconds) until it returns True or the sensor times out (timeout, default 7 days).

🏏

Cricket analogy: It's like the third umpire repeatedly reviewing a run-out replay frame by frame until they can confirm 'out' or 'not out,' rather than guessing after a single glance.

Poke Mode vs. Reschedule Mode

By default, sensors run in poke mode, which occupies a worker slot for the entire duration of the wait — the task stays in a 'running' state, sleeping between poke() calls, which wastes a worker slot on long waits. Setting mode='reschedule' makes the sensor release its worker slot between checks, going back to 'up_for_reschedule' state and being rescheduled by the scheduler at the next poke_interval, which is far more efficient for sensors expected to wait minutes or hours rather than seconds.

🏏

Cricket analogy: It's like a substitute fielder staying physically on the boundary the entire rain delay (poke mode) versus going back to the pavilion and being called back onto the field only when play resumes (reschedule mode).

python

from airflow.sensors.filesystem import FileSensor
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def ingest_pipeline():

    wait_for_export = S3KeySensor(
        task_id="wait_for_export",
        bucket_name="upstream-exports",
        bucket_key="daily/{{ ds }}/export.csv",
        aws_conn_id="aws_default",
        poke_interval=300,       # check every 5 minutes
        timeout=60 * 60 * 6,     # give up after 6 hours
        mode="reschedule",       # free the worker slot between checks
        soft_fail=True,          # mark as skipped, not failed, on timeout
    )

    @task
    def process_export():
        print("File landed, processing...")

    wait_for_export >> process_export()

ingest_pipeline()

Common Sensor Types

Airflow ships several ready-made sensors: FileSensor waits for a file on a local or mounted filesystem; S3KeySensor waits for an object (with wildcard support) to appear in an S3 bucket; ExternalTaskSensor waits for a specific task in a different DAG to reach a given state, which is the standard way to model cross-DAG dependencies; and SqlSensor polls a database with a SQL query, succeeding once the query returns a truthy result. For anything not covered by an existing sensor, PythonSensor lets you supply an arbitrary Python callable as the poke condition.

🏏

Cricket analogy: It's like different types of match officials — a boundary umpire watching for the ball crossing the rope (FileSensor), a third umpire reviewing replays for run-outs (ExternalTaskSensor) — each specialized for one kind of check.

Setting soft_fail=True on a sensor makes it transition to a 'skipped' state instead of 'failed' when it times out, which is useful when a missing upstream file should gracefully skip downstream tasks rather than trigger alerting and retries.

Leaving many long-running sensors in the default poke mode is a classic cause of 'worker slot starvation,' where all available worker slots are occupied by sensors sleeping and waiting, leaving no capacity to actually execute other tasks. Always use mode='reschedule' for sensors expected to wait more than a few minutes, and set a sensible timeout so stuck sensors don't wait indefinitely.

A Sensor waits for a condition to become true via a repeatedly-called poke(context) method.
poke_interval controls how often the condition is checked; timeout controls how long the sensor waits before giving up.
Poke mode holds a worker slot for the entire wait; reschedule mode frees the slot between checks.
Use mode='reschedule' for waits longer than a few minutes to avoid worker slot starvation.
Built-in sensors include FileSensor, S3KeySensor, ExternalTaskSensor, and SqlSensor; PythonSensor covers custom conditions.
soft_fail=True makes a timed-out sensor skip instead of fail, useful for optional upstream dependencies.
ExternalTaskSensor is the standard way to model dependencies between tasks in different DAGs.

Practice what you learned

1. What method must a custom Sensor implement to define its wait condition?

2. What is the main downside of the default 'poke' mode for a sensor with a long wait?

3. Which sensor is used to model a dependency on a specific task in a different DAG?

4. What happens when a sensor with soft_fail=True times out?

Was this page helpful?

Hooks and Connections

Variables and Templating

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheAirflowStudyNotes#Sensors#Poke#Mode#Reschedule#Common#StudyNotes#SkillVeris#ExamPrep

What Are Sensors?

Poke Mode vs. Reschedule Mode

Common Sensor Types

Practice what you learned

You May Also Like

Related Reading

Related Study Notes in Programming

Topics covered