From Program to Execution Graph
When you write a Flink program using the DataStream API, you are not describing immediate computation but building a logical dataflow graph of operators. When you call env.execute(), Flink optimizes this into a JobGraph, chaining compatible operators together to reduce serialization and network overhead, then submits it to the JobManager. The JobManager expands the JobGraph into an ExecutionGraph, which is the parallel, physical version of the plan: each logical operator becomes multiple parallel subtasks distributed across TaskManager slots according to the configured parallelism.
Cricket analogy: Writing the DataStream program is like drawing up a game plan on a whiteboard before the match, while the ExecutionGraph is the actual field placements and bowling rotation as physically executed once the umpire calls play.
Operator chaining fuses operators like a source, a map, and a filter into a single task when they share the same parallelism and forward partitioning, so data passes between them via simple function calls instead of network shuffles or even serialization. Operations that require redistributing data, such as keyBy, break the chain because they need a network shuffle to route records to the subtask responsible for a given key, typically using a hash partitioning scheme over the key.
Cricket analogy: Operator chaining is like a fielder who takes the catch and immediately throws to the keeper for a run-out in one fluid motion, versus a keyBy-style redistribution where the ball must be relayed across the field to a specific fielder covering a particular zone.
Parallelism and Data Partitioning
Each operator has a parallelism, the number of parallel subtasks that instance runs as, which can be set globally via parallelism.default, per-job via the -p CLI flag, or per-operator via .setParallelism(n). Between operators with different parallelism or after a keyBy, Flink applies a partitioning strategy to route each record to the correct downstream subtask: keyBy uses hash partitioning on the key so all records for a given key always land on the same subtask (crucial for correct stateful aggregation), while rebalance applies round-robin redistribution to fix data skew, and broadcast sends every record to every downstream subtask.
Cricket analogy: keyBy-style hash partitioning is like always assigning the same specific fielder, say point, to field every ball hit to that exact zone, ensuring consistency in coverage across the whole innings.
DataStream<Transaction> transactions = env.fromSource(kafkaSource, watermarkStrategy, "transactions");
DataStream<Alert> alerts = transactions
.filter(tx -> tx.getAmount() > 0) // chained with source, no shuffle
.keyBy(Transaction::getAccountId) // triggers a network shuffle
.process(new FraudScoringFunction()) // stateful per-key processing
.setParallelism(16); // override operator parallelism
alerts.rebalance().sinkTo(alertSink); // round-robin to fix downstream skewYou can inspect the actual physical plan Flink produced, including which operators were chained and how partitioning was applied between them, in the Web UI's job graph view or by calling env.getExecutionPlan() before execution.
- A DataStream program builds a logical dataflow graph, compiled into a JobGraph and then an ExecutionGraph.
- Operator chaining fuses compatible operators into one task, avoiding serialization and network overhead.
- keyBy breaks chaining because it requires a network shuffle to redistribute data by key.
- Parallelism can be set globally, per-job, or per-operator, controlling how many parallel subtasks run.
- Hash partitioning (via keyBy) guarantees all records for a key land on the same subtask.
- rebalance applies round-robin redistribution to address data skew.
- broadcast sends every record to every downstream subtask, useful for small reference datasets.
Practice what you learned
1. What does calling env.execute() do in a Flink DataStream program?
2. Why does operator chaining improve performance?
3. Why does keyBy always break operator chaining?
4. What partitioning strategy guarantees all records for a given key land on the same subtask?
5. What does the rebalance() partitioning strategy do?
Was this page helpful?
You May Also Like
Flink Architecture
A tour of Flink's distributed architecture: JobManager, TaskManagers, slots, and how a job is deployed and coordinated across a cluster.
What Is Apache Flink?
An introduction to Apache Flink, the open-source distributed engine for stateful computations over unbounded and bounded data streams.
Your First Flink Job
A hands-on walkthrough of writing, running, and understanding a simple Flink DataStream job that reads events, transforms them, and writes results.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics