From Operators to Decorators
The TaskFlow API, introduced in Airflow 2.0, lets you define tasks with the @task decorator on a regular Python function instead of explicitly instantiating a PythonOperator and wrapping your logic in python_callable. Combined with the @dag decorator on a function that contains your task calls, an entire pipeline can be written as plain, readable Python with automatic dependency inference: when one @task-decorated function's return value is passed as an argument to another, Airflow automatically wires the dependency and handles passing the value through XCom behind the scenes.
Cricket analogy: It's like modern ball-tracking technology (Hawk-Eye) automatically inferring a ball's trajectory from raw sensor data instead of an umpire manually eyeballing and calling out each coordinate, less boilerplate, same underlying physics.
from airflow.decorators import dag, task
import pendulum
@dag(
schedule="@daily",
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
catchup=False,
tags=["taskflow", "sales"],
)
def sales_etl():
@task
def extract() -> dict:
return {"rows": 1200, "source": "postgres"}
@task
def transform(raw: dict) -> dict:
return {"clean_rows": raw["rows"] - 5}
@task
def load(clean: dict) -> None:
print(f"Loaded {clean['clean_rows']} rows into warehouse")
load(transform(extract()))
sales_etl()Automatic XCom and Dependency Inference
When extract() returns a dict and you call transform(extract()), Airflow automatically pushes extract's return value to XCom under the key return_value, and automatically pulls it back for transform's parameter, all without a single explicit xcom_push or xcom_pull call. The dependency edge between extract and transform is also inferred purely from that function call structure, so you rarely need explicit >> operators inside a TaskFlow DAG; the data flow and the control flow become the same piece of code, which reduces a common class of bugs where a manual dependency declaration silently drifts out of sync with the actual data being passed.
Cricket analogy: It's like a fully automated scoring system where entering a batter's runs on one screen instantly updates the partnership total, strike rate, and required run rate on every other display without a scorer manually re-typing anything.
Mixing TaskFlow with Classic Operators
TaskFlow tasks and classic operator instances can coexist in the same DAG; a @task-decorated function can be chained with >> against a BashOperator or FileSensor instance just like any other task, and you can even pass a classic operator's output (its .output attribute, which is an XComArg) into a TaskFlow function as an argument. This matters in practice because plenty of Airflow functionality, especially provider-specific operators like SnowflakeOperator or sensors, doesn't have a TaskFlow-native equivalent, so real-world DAGs commonly mix both styles rather than using pure TaskFlow throughout.
Cricket analogy: It's like a team blending an old-school textbook spinner with a modern data-driven pace bowler in the same XI, two different styles of preparation working the same match plan without needing to be identical.
Dynamic Task Mapping and Task Groups
Since Airflow 2.3, .expand() lets a @task-decorated function run once per item in an iterable determined at runtime, creating 'mapped task instances' dynamically rather than requiring you to hard-code a fixed number of parallel tasks; for example, process_file.expand(file_path=list_files()) creates one mapped task instance per file returned by list_files(), and each instance is tracked, retried, and viewable independently in the UI. @task_group (or the classic TaskGroup context manager) lets you visually and logically group a cluster of related tasks under one collapsible node in the Airflow UI graph view, which keeps large DAGs readable without changing execution semantics.
Cricket analogy: Dynamic task mapping is like a tournament dynamically scheduling exactly as many net-practice slots as there are squad members registered that morning, rather than a fixed roster of ten slots regardless of turnout.
task_group is purely a UI and organizational construct, it groups tasks under a collapsible node in the graph view for readability, but it does not by itself change scheduling behavior or add any implicit dependency between grouped tasks; you still declare dependencies explicitly between the tasks inside and outside the group.
Avoid returning large objects, such as full pandas DataFrames, directly from a @task function. By default, XCom values are serialized and stored in the Airflow metadata database, which is not designed for large payloads; instead, write large intermediate data to external storage (like S3 or a data warehouse table) and pass a lightweight reference (a file path or table name) through XCom, or configure a custom XCom backend for large-object handling.
- The @task and @dag decorators let you write Airflow pipelines as plain Python functions with automatic dependency inference.
- Passing one @task function's return value into another automatically wires the dependency and handles XCom push/pull.
- TaskFlow tasks can be mixed freely with classic operator instances in the same DAG, including passing .output as an argument.
- .expand() enables dynamic task mapping, creating one mapped task instance per item in a runtime-determined iterable.
- @task_group organizes related tasks under a collapsible UI node without changing scheduling semantics.
- XCom is backed by the metadata database by default, so large payloads should be stored externally and referenced by path instead.
- TaskFlow reduces bugs where a manually declared dependency drifts out of sync with the actual data being passed between tasks.
Practice what you learned
1. What happens when you pass one @task function's return value as an argument to another @task function?
2. Can TaskFlow-decorated tasks be combined with classic operator instances in the same DAG?
3. What does .expand() enable in the context of the TaskFlow API?
4. Why should you avoid returning a large pandas DataFrame directly from a @task function?
Was this page helpful?
You May Also Like
DAG Definition Basics
Learn how Apache Airflow DAGs are structured as Python files, parsed by the scheduler, and how to define one correctly with dag_id, schedule, and default_args.
Operators
Understand what Airflow operators are, the major built-in categories, and how to configure or subclass them to build reliable tasks.
Task Dependencies
Learn how to wire tasks together with bitshift operators and the chain() helper, and how trigger rules control execution when upstream tasks fail or skip.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics