What Is Apache Airflow?

Analogy categoryAnalogies

Apache Airflow is an open-source platform, originally created at Airbnb in 2014, for programmatically authoring, scheduling, and monitoring workflows. Instead of clicking through a GUI to wire up jobs, you write Python code that describes a workflow as a graph of tasks with dependencies between them, called a DAG (Directed Acyclic Graph). Airflow then takes care of scheduling those DAGs, running the tasks in the right order, retrying failures, and giving you a web UI to observe everything.

🏏

Cricket analogy: Think of a DAG like a tour selection committee's plan: pick the squad, then set the batting order, then confirm the playing XI — each step depends on the one before it, just as MS Dhoni's captaincy decisions always followed a defined sequence of team meetings.

Why Airflow Exists: The Problem with Cron

Before Airflow, most data teams chained jobs together with cron and shell scripts. This worked for a single nightly job, but broke down as pipelines grew: cron has no concept of dependencies between jobs, no built-in retry logic, no historical view of what ran and when, and no easy way to backfill a failed run from three days ago. Airflow was built specifically to solve these problems by treating the workflow itself — not just individual scripts — as a first-class, version-controlled object.

🏏

Cricket analogy: Relying on cron alone is like a bowler running in for a spell without a captain setting the field or rotating the attack — jobs fire on a timer but nobody coordinates who bowls next or reacts when a delivery goes for six.

Core Concepts: DAGs, Tasks, and Operators

In Airflow, a DAG is a Python file that declares a workflow's structure: its schedule, its tasks, and the dependencies between those tasks. A task is a single unit of work, created by instantiating an Operator — for example, a PythonOperator runs a Python function, a BashOperator runs a shell command, and provider-specific operators like PostgresOperator or S3Hook wrap interactions with external systems. Dependencies between tasks are declared with the >> and << bitshift operators, or the set_upstream/set_downstream methods, which Airflow uses to build the execution graph.

🏏

Cricket analogy: An Operator is like a specific delivery type in a bowler's repertoire — a yorker, a googly, a bouncer — while a Task is that specific ball bowled in over 14.3, and the DAG is the full bowling plan for the innings.

python

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract():
    print("Extracting data from source system...")

with DAG(
    dag_id="simple_etl_example",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["example"],
) as dag:

    extract_task = PythonOperator(
        task_id="extract",
        python_callable=extract,
    )

    transform_task = BashOperator(
        task_id="transform",
        bash_command="echo 'transforming data...'",
    )

    load_task = BashOperator(
        task_id="load",
        bash_command="echo 'loading into warehouse...'",
    )

    extract_task >> transform_task >> load_task

Airflow vs Other Orchestrators

Airflow competes with tools like Prefect, Dagster, Luigi, and cloud-native services like AWS Step Functions. Its main strengths are a mature ecosystem of provider packages (for AWS, GCP, Azure, Snowflake, dbt, and hundreds more), a battle-tested scheduler, and the fact that workflows are plain Python, so anything you can express in code you can orchestrate. Its main tradeoffs are a steeper operational learning curve than newer tools and a DAG model that is less naturally suited to dynamic, data-dependent branching than some competitors, though Dynamic Task Mapping in Airflow 2.3+ closes much of that gap.

🏏

Cricket analogy: Choosing Airflow over a newer orchestrator is like a franchise picking a proven all-format player like Virat Kohli over a promising T20 specialist — you get consistency and a deep track record, at the cost of some newer flashy skills.

Airflow does not move data itself — it orchestrates other systems that do. A task typically triggers a Spark job, runs a SQL query, or calls an API; Airflow's job is deciding when that call happens and what happens if it fails, not processing the data in-process.

Airflow is an open-source workflow orchestration platform originally built at Airbnb, now a top-level Apache project.
Workflows are defined as DAGs — Directed Acyclic Graphs — written in Python, giving you version control and code review over your pipelines.
Airflow was created to fix cron's lack of dependency management, retries, history, and backfill support.
A DAG is composed of Tasks, each of which is an instance of an Operator (e.g., PythonOperator, BashOperator).
Dependencies between tasks are declared with >> and << operators or set_upstream/set_downstream.
Airflow orchestrates work in other systems rather than processing data itself.
Airflow's core strength versus competitors is its mature provider ecosystem and Python-native flexibility.

Practice what you learned

1. What does DAG stand for in the context of Apache Airflow?

2. Which company originally created Apache Airflow?

3. What is the primary limitation of cron that Airflow was designed to solve?

4. In Airflow, what is a Task?

5. Which operator symbols are used in Airflow to declare task dependencies?

Was this page helpful?

Installing Airflow

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheAirflowStudyNotes#WhatIsApacheAirflow#Apache#Airflow#Exists#Problem#StudyNotes#SkillVeris#ExamPrep

What Is Apache Airflow?