DAG Definition Basics

A DAG (Directed Acyclic Graph) in Apache Airflow is a plain Python file that describes a workflow as a collection of tasks and the dependencies between them. You instantiate a DAG object, usually inside a with DAG(...) as dag: context manager, give it a unique dag_id, and then define tasks that get automatically attached to that DAG instance. The word 'acyclic' matters: Airflow will refuse to run a graph that loops back on itself, because a workflow has to have a clear start and finish.

A minimal DAG definition needs a dag_id (unique across the whole Airflow instance), a schedule (cron string, timedelta, preset like @daily, or None for manual triggering only), a start_date, and typically catchup=False unless you intentionally want historical backfills. default_args is a dictionary applied to every task in the DAG, commonly holding things like retries, retry_delay, and owner, so you don't repeat them on each task. Tags are optional but useful for filtering DAGs in the Airflow UI when you have dozens of pipelines.

The Airflow scheduler continuously re-scans every .py file inside the dags_folder (by default every dag_dir_list_interval seconds, commonly 300, plus a min_file_process_interval for how often each file is re-parsed) and executes the top-level code in each file to rebuild the DagBag. This means any code sitting outside of a task's callable, such as a database query used to generate task lists dynamically, runs on every parse cycle, not just once. Keeping top-level code fast and side-effect-free is essential, because a slow-parsing DAG file directly slows down scheduler performance for the entire Airflow instance.

Well-designed DAGs are idempotent: running the same DAG run twice for the same logical date should produce the same result and not create duplicate rows or double-send the same email. Airflow encourages this by giving each task run access to templated context variables like {{ ds }} or {{ data_interval_start }}, so a task can, for example, delete-then-insert partitioned data for that specific interval rather than blindly appending. Avoid dynamic side effects at DAG-parse time (like writing files or making network calls outside a task), since those run repeatedly on every scheduler parse cycle rather than once per actual execution.