Configuring Retries
Every Airflow task accepts a retries parameter and a retry_delay timedelta that control how many times a failed task instance is automatically re-attempted and how long the scheduler waits between attempts; setting retry_exponential_backoff=True multiplies that delay after each failed attempt, up to an optional max_retry_delay ceiling, which is the standard way to handle transient issues like a flaky API or a momentarily unavailable database connection without manual intervention.
Cricket analogy: Like a bowler getting a fixed number of no-ball reviews from the third umpire before a delivery is ruled dead, a task's retries parameter gives it a bounded number of automatic re-attempts, with retry_delay controlling the pause before the next try.
Callbacks and Trigger Rules
Beyond automatic retries, Airflow lets you hook into task state transitions with on_failure_callback, on_success_callback, and on_retry_callback functions for custom alerting or cleanup logic, while trigger_rule controls whether a downstream task runs based on the state of its upstream tasks; the default all_success requires every upstream task to succeed, but rules like one_failed, all_done, or none_failed_min_one_success let you build DAGs with explicit failure-handling branches, such as a cleanup task that must run whether or not the main task failed.
Cricket analogy: Like a team having a standing rule that if any batter gets out for a duck, the coach is notified immediately, on_failure_callback fires a custom function the instant a task instance fails, independent of whether retries are still available.
Failure Isolation with Task Groups and SubDAGs
When a task instance exhausts its retries, only that task and its downstream dependents (subject to trigger_rule) are affected; unrelated branches of the same DAG continue running unimpeded, which is why Airflow encourages small, idempotent tasks over large monolithic ones. Setting depends_on_past=True additionally blocks a task from running until the same task succeeded in the previous DagRun, useful for strictly sequential aggregations but risky if a single failure needs manual intervention to unblock every subsequent run.
Cricket analogy: Like one fielder dropping a catch not stopping the rest of the field from continuing to bowl and field their overs, one task instance exhausting its retries only blocks its own downstream dependents, leaving unrelated branches of the DAG to keep running.
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
import pendulum
def call_flaky_api(**context):
import requests
resp = requests.get("https://api.example.com/export", timeout=10)
resp.raise_for_status()
fetch = PythonOperator(
task_id="fetch_export",
python_callable=call_flaky_api,
retries=4,
retry_delay=pendulum.duration(minutes=2),
retry_exponential_backoff=True,
max_retry_delay=pendulum.duration(minutes=30),
)
cleanup = BashOperator(
task_id="cleanup_tmp_files",
bash_command="rm -rf /tmp/export_staging/*",
trigger_rule="all_done",
)
fetch >> cleanupon_retry_callback fires before each retry attempt (not just the final failure), making it a good place to log intermediate diagnostics like 'attempt 2 of 4 failed with a 503' without waiting for the task to be fully exhausted.
depends_on_past=True combined with a task that fails permanently creates a permanently stuck DAG: every future DagRun for that task will refuse to run until someone manually marks the failed instance as success. Use it deliberately, not as a default.
- retries and retry_delay bound how many times and how often a failed task instance is automatically re-attempted.
- retry_exponential_backoff increases the delay between successive retries, capped by max_retry_delay.
- on_failure_callback, on_success_callback, and on_retry_callback hook into task state transitions for custom alerting.
- trigger_rule controls whether a downstream task runs based on upstream state; all_done is common for cleanup tasks.
- A failed task instance only blocks its own downstream dependents, not unrelated branches of the same DAG.
- depends_on_past=True enforces strict sequential success across DagRuns but can permanently stall future runs if not cleared manually.
- Small, idempotent tasks limit the blast radius of any single failure within a DAG.
Practice what you learned
1. What does retry_exponential_backoff=True do?
2. Which trigger_rule ensures a cleanup task runs whether upstream tasks succeeded or failed?
3. When does on_failure_callback fire relative to the retries counter?
4. What is the risk of setting depends_on_past=True?
5. If task B (downstream of task A) fails permanently, what happens to an unrelated task C in the same DAG with no dependency on A or B?
Was this page helpful?
You May Also Like
SLAs and Alerting
Using Airflow's sla parameter, sla_miss_callback, and failure callbacks to detect and alert on late or broken pipelines.
The Scheduler
How Airflow's scheduler parses DAGs, evaluates dependencies, and triggers task instances to keep pipelines running on time.
Executors: Local, Celery, Kubernetes
Comparing Airflow's LocalExecutor, CeleryExecutor, and KubernetesExecutor for running task instances at different scales.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics