Monitoring and Logging

Analogy categoryAnalogies

Airflow's Built-in Logging Model

Every task instance writes its own log to a predictable path (dag_id/task_id/run_id/attempt=N.log under the base log folder), and the webserver's Grid view reads that file directly when you click into a task instance. In distributed deployments (CeleryExecutor, KubernetesExecutor) where the task ran on a different machine than the webserver, remote logging must be configured — shipping logs to S3, GCS, Azure Blob, or Elasticsearch — otherwise the webserver has no way to fetch a log that was written to a worker's local disk that no longer exists.

🏏

Cricket analogy: It's like every over being logged in a physical scorebook kept in the dugout of whichever ground the match was played at; if the final is at a different stadium than headquarters, the scorer at HQ can't read that book unless it's been couriered or digitized centrally.

Setting Up Remote Logging

Remote logging is configured via [logging] remote_logging = True, a remote_base_log_folder pointing at a bucket path (e.g., s3://my-airflow-logs), and a remote_log_conn_id referencing an Airflow Connection with the right credentials; Airflow then writes logs to both local disk and the remote store, and the webserver transparently fetches from remote storage when the local copy isn't available. This is essential for CeleryExecutor and KubernetesExecutor deployments where task attempts can land on different, ephemeral machines across retries.

🏏

Cricket analogy: It's like every ground on the domestic circuit uploading its scorebook to one central cricket board database after each day's play, so any official anywhere can pull up any match's record regardless of which stadium it was played at.

python

# airflow.cfg (or environment variables) for S3 remote logging
[logging]
remote_logging = True
remote_base_log_folder = s3://my-airflow-logs/prod
remote_log_conn_id = aws_default
encrypt_s3_logs = False

# Equivalent environment variables:
# AIRFLOW__LOGGING__REMOTE_LOGGING=True
# AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://my-airflow-logs/prod
# AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=aws_default

Metrics and Alerting

Airflow emits StatsD metrics out of the box (task duration, queue length, DAG processing time, scheduler heartbeat) when [metrics] statsd_on = True is set, which a StatsD-to-Prometheus exporter can scrape for Grafana dashboards; the most actionable metrics to alert on are scheduler heartbeat gaps (the scheduler is stuck or down), DAG file processing duration (parsing is degrading), and task queue depth (workers are under-provisioned). Callback hooks — on_failure_callback, sla_miss_callback, and Airflow's native email/Slack alerting on retries exhausted — complement metrics by notifying humans about specific failures rather than aggregate trends.

🏏

Cricket analogy: It's like a broadcaster's data feed continuously tracking run rate, overs remaining, and wickets in hand for a live dashboard, while a separate instant push alert fires the moment a specific batter gets out, giving both the trend view and the individual event.

Set sla on individual tasks and define sla_miss_callback at the DAG level to get proactive notifications when a task is taking longer than expected, before it fully fails — this catches slow degradation (a growing source table, a throttled API) that a simple failure alert would miss until it's much worse.

Pitfall: Alert Fatigue from Over-Broad Failure Notifications

A common mistake is setting email_on_failure=True or a Slack on_failure_callback on every task in every DAG without considering retry behavior — this means an on-call engineer gets paged for every single failed attempt, even ones Airflow's own retry logic will resolve automatically on the next attempt a few minutes later. The fix is to alert only on final failure (after retries are exhausted, which the callback context's try_number vs max_tries reveals) and to route non-critical DAG alerts to a low-priority channel instead of the same pager as production-critical pipelines.

🏏

Cricket analogy: It's like a captain calling a team huddle after every single dot ball instead of only after a genuine turning point in the match — constant interruptions for events the game naturally recovers from on its own dilute attention for when it actually matters.

Avoid alerting on every task attempt failure. Configure alerts to fire only after retries are exhausted (final failure), and route routine/low-priority DAG failures to a different channel than production-critical pipeline alerts — indiscriminate paging trains on-call engineers to ignore notifications, which defeats the purpose of alerting entirely.

Task logs are written per-attempt to a predictable path; in distributed deployments, remote logging (S3/GCS/Elasticsearch) is required so the webserver can read logs from ephemeral worker machines.
Remote logging is configured with remote_logging, remote_base_log_folder, and remote_log_conn_id in the [logging] section.
StatsD metrics (statsd_on = True) expose task duration, queue depth, and scheduler heartbeat for Prometheus/Grafana dashboards.
Scheduler heartbeat gaps, DAG file processing duration, and task queue depth are the highest-signal metrics to alert on.
sla and sla_miss_callback give proactive alerts on slow-running tasks before they outright fail.
Alert only on final task failure (after retries exhausted), not every individual attempt, to avoid alert fatigue.
Route non-critical DAG alerts to a separate, lower-priority channel than production-critical pipeline pages.