Executors and How They Differ
An executor is the component that actually runs task instances that the scheduler has queued; Airflow ships several, and picking the right one is mostly a question of scale and infrastructure. SequentialExecutor and LocalExecutor run tasks as subprocesses on the same machine as the scheduler, with LocalExecutor adding a configurable parallelism via a process pool, making it fine for development or small single-node deployments but incapable of scaling beyond one host's CPU and memory.
Cricket analogy: Like a club team where the same handful of players field, bowl, and bat because there's no bench, LocalExecutor runs every task instance as a subprocess on the same single machine as the scheduler, capped by that one host's resources.
CeleryExecutor for Distributed Workers
CeleryExecutor distributes task instances across a pool of worker machines using the Celery distributed task queue, backed by a message broker such as Redis or RabbitMQ and a result backend in the metadata database; the scheduler pushes tasks onto the broker's queue and any available worker picks them up, which lets you scale horizontally by adding worker nodes and use queue names to route specific tasks, like GPU-heavy ones, to specialized worker pools.
Cricket analogy: Like a franchise league's central auction pool where any of several eligible teams can pick up an available player, CeleryExecutor puts queued task instances onto a broker queue that any of several distributed worker machines can pick up and run.
KubernetesExecutor for Per-Task Pods
KubernetesExecutor launches a brand-new Kubernetes pod for every single task instance, using the pod_template_file to define resources, image, and node affinity per task, then tears the pod down when the task completes; this gives near-perfect resource isolation and lets you request different CPU, memory, or GPU limits per task without maintaining a static Celery worker fleet, at the cost of pod startup latency, typically several seconds to a minute, for every task.
Cricket analogy: Like a franchise hiring a fresh specialist net bowler for a single practice session and releasing them the moment it ends rather than keeping a permanent squad, KubernetesExecutor spins up a brand-new pod per task instance and tears it down after.
# pod_template_file.yaml referenced by KubernetesExecutor
apiVersion: v1
kind: Pod
metadata:
labels:
airflow-worker: "true"
spec:
restartPolicy: Never
containers:
- name: base
image: my-registry/airflow-worker:2.9
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2"
memory: "4Gi"You can override the default pod_template_file per task using the executor_config kwarg on an operator, e.g. executor_config={"pod_override": k8s.V1Pod(...)}, to give a single memory-hungry task more resources without changing the fleet-wide template.
CeleryExecutor workers must have the exact same Python and provider package versions installed as the scheduler and webserver; a version mismatch causes tasks to fail with obscure DAG deserialization errors rather than a clear dependency error.
- LocalExecutor runs tasks as subprocesses on the scheduler's own host; good for dev or single-node deployments, not horizontally scalable.
- CeleryExecutor distributes tasks across a worker pool via a broker (Redis/RabbitMQ), scaling horizontally by adding workers.
- Celery queue names let you route specific tasks (e.g., GPU workloads) to specialized worker pools.
- KubernetesExecutor launches an isolated, per-task pod defined by pod_template_file, giving fine-grained per-task resource control.
- KubernetesExecutor trades pod startup latency for near-perfect resource isolation and no static worker fleet to maintain.
- executor_config lets individual tasks override the default pod template or Celery queue.
- Worker package versions must match the scheduler's for CeleryExecutor to avoid DAG deserialization errors.
Practice what you learned
1. What is the main limitation of LocalExecutor?
2. What infrastructure does CeleryExecutor require that LocalExecutor does not?
3. How does KubernetesExecutor handle each task instance?
4. What is the primary trade-off of KubernetesExecutor compared to CeleryExecutor?
5. How can a Celery-based deployment route GPU-heavy tasks to specialized machines?
Was this page helpful?
You May Also Like
The Scheduler
How Airflow's scheduler parses DAGs, evaluates dependencies, and triggers task instances to keep pipelines running on time.
Retries and Error Handling
Configuring per-task retries, retry delays, callbacks, and trigger rules so Airflow DAGs recover gracefully from transient failures.
Backfilling and Catchup
Understanding Airflow's catchup behavior and the airflow dags backfill command for (re)processing historical data intervals.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics