The Airflow Architecture

A running Airflow deployment is made up of four core components: the Scheduler, which parses DAG files and decides which task instances are ready to run; the Webserver, which serves the UI you interact with; the Executor, which is the mechanism responsible for actually running tasks (in-process, on local worker processes, or on remote workers); and the Metadata Database, which stores the state of everything — DAG runs, task instances, connections, and variables. In distributed setups, a Triggerer process also runs for deferrable operators, and a message broker like Redis or RabbitMQ sits between the scheduler and workers when using CeleryExecutor.

The Scheduler continuously parses every DAG file in your DAGS_FOLDER, evaluates each DAG's schedule to determine if a new DAG run is due, and for each DAG run, checks which tasks have their upstream dependencies satisfied and queues them. It doesn't run the task code itself — it hands that off to the Executor, which is configured in airflow.cfg (or the AIRFLOW__CORE__EXECUTOR environment variable) and determines the actual concurrency model: whether tasks run sequentially, in local subprocesses, or distributed across a worker fleet.

SequentialExecutor runs one task at a time in the scheduler's own process and is only usable with SQLite — it exists purely as a zero-dependency default for a first airflow standalone run. LocalExecutor runs tasks as parallel subprocesses on the same machine as the scheduler, which is fine for a single powerful server but doesn't scale horizontally. CeleryExecutor distributes tasks across a fleet of worker machines via a message broker (Redis or RabbitMQ), giving you horizontal scalability with persistent, always-on workers. KubernetesExecutor instead launches a fresh Kubernetes pod per task, giving perfect per-task resource isolation and autoscaling at the cost of pod startup latency for every task.

Every piece of Airflow's runtime state lives in the metadata database: DAG run records, individual task instance states (queued, running, success, failed, up_for_retry), XCom values passed between tasks, Connections and Variables, and user/role data for the UI. Because the Scheduler, Webserver, and every Worker all read and write to this single database, its performance directly bounds Airflow's overall scalability — a common production issue at scale is metadata DB connection exhaustion, which is why connection pooling (e.g., via PgBouncer) is standard practice for large Airflow deployments.