Apache Airflow Study Notes

Airflow Best Practices

Practical guidelines for writing reliable, maintainable Airflow DAGs, covering idempotency, DAG design, dependency management, and testing.

Airflow Interview Questions

Commonly asked Apache Airflow interview questions covering core concepts, architecture, scheduling, and real-world scenario problems, with explanations.

Airflow Quick Reference

A condensed cheat sheet of essential Airflow CLI commands, common operators/sensors, scheduling syntax, and key configuration settings.

Airflow vs Prefect vs Dagster

A comparison of three popular Python workflow orchestrators, covering their core philosophies, execution models, and when to choose each.

Airflow with Docker and Kubernetes

How Airflow runs in containerized environments — Docker Compose for local development, and KubernetesExecutor / KubernetesPodOperator for isolated, scalable pr…

Backfilling and Catchup

Understanding Airflow's catchup behavior and the airflow dags backfill command for (re)processing historical data intervals.

Branching with BranchPythonOperator

Learn how to build conditional paths in a DAG using BranchPythonOperator, and how Airflow's trigger rules interact with skipped tasks.

Building an ETL DAG

A hands-on walkthrough of designing and implementing a complete Extract-Transform-Load pipeline as an Airflow DAG, including failure handling and local testing.

DAG Definition Basics

Learn how Apache Airflow DAGs are structured as Python files, parsed by the scheduler, and how to define one correctly with dag_id, schedule, and default_args.

DAG Versioning and CI/CD

How Airflow tracks DAG structure changes over time via DAG Versioning, and how to build a GitOps-style CI/CD pipeline that tests and deploys DAGs safely.

Dynamic DAG Generation

Techniques for programmatically generating multiple Airflow DAGs from configuration files or metadata instead of hand-writing each one, and the performance tra…

Executors: Local, Celery, Kubernetes

Comparing Airflow's LocalExecutor, CeleryExecutor, and KubernetesExecutor for running task instances at different scales.

Hooks and Connections

Understand how Airflow Connections store credentials for external systems and how Hooks provide a Python interface to interact with them.

Installing Airflow

A practical walkthrough of the main ways to install Apache Airflow, from a local pip install to Docker Compose, and how to verify the install worked.

Monitoring and Logging

How Airflow's per-task logging works, how to configure remote logging for distributed deployments, and how to build alerting that avoids fatigue while catching…

Operators

Understand what Airflow operators are, the major built-in categories, and how to configure or subclass them to build reliable tasks.

Retries and Error Handling

Configuring per-task retries, retry delays, callbacks, and trigger rules so Airflow DAGs recover gracefully from transient failures.

Scheduling and Cron Expressions

Learn how Airflow's schedule parameter works with cron expressions, presets, and timedeltas, and how data intervals, catchup, and backfill behave.

Sensors

Learn how Airflow Sensors wait for external conditions to become true before downstream tasks proceed, and how to avoid worker-slot starvation.

SLAs and Alerting

Using Airflow's sla parameter, sla_miss_callback, and failure callbacks to detect and alert on late or broken pipelines.

Task Dependencies

Learn how to wire tasks together with bitshift operators and the chain() helper, and how trigger rules control execution when upstream tasks fail or skip.

Testing DAGs

How to validate Airflow DAGs at three layers — import integrity, structural correctness, and task logic — using pytest, DagBag, and dag.test().

The Airflow Architecture

How Airflow's scheduler, webserver, executor, and metadata database work together to parse, schedule, and execute your DAGs.

The Airflow UI

A guided tour of the Airflow web interface, from the DAGs list and Grid view to task logs, Connections, and Variables.