Monitoring with the Spark UI

Analogy categoryAnalogies

Every running Spark application exposes a web UI, by default on port 4040 on the driver (incrementing to 4041, 4042, etc. if multiple applications run on the same machine), with tabs covering Jobs, Stages, Storage, Environment, Executors, and — for DataFrame/SQL workloads — a dedicated SQL tab showing physical query plans. This UI is the primary tool for understanding why a job is slow, whether it's failing due to memory pressure, or which stage is the bottleneck, and it's available for the lifetime of the driver process.

🏏

Cricket analogy: A stadium's giant scoreboard showing overs, run rate, and partnership breakdowns in real time during a match is like the Spark UI's Jobs and Stages tabs surfacing live progress on a running application.

Jobs and Stages Tabs

The Jobs tab lists each Spark action (like collect, count, or save) as a job broken into stages at shuffle boundaries, and clicking into a stage reveals a task-level table plus summary metrics — task duration distribution, shuffle read/write size, and spill to disk — that is the fastest way to spot skew: if the max task duration is far larger than the median, a handful of partitions are doing disproportionate work, usually from skewed join or group-by keys. The DAG visualization for each job shows how RDDs or DataFrame operations chain together across stages, making it easy to see exactly where a wide transformation forces a shuffle.

🏏

Cricket analogy: A team analyst reviewing a scorecard where one batter faced 150 balls while others faced only 10 immediately flags an imbalance in strategy, just as a Spark stage's task duration table flags skew when one task massively outruns the median.

properties

# spark-defaults.conf — persist event logs so the History Server can replay completed apps
spark.eventLog.enabled          true
spark.eventLog.dir              hdfs:///spark-logs
spark.history.fs.logDirectory   hdfs:///spark-logs
spark.history.ui.port           18080

Executors Tab and Spark History Server

The Executors tab shows, per executor, active/completed task counts, total GC time, storage memory used for cached data, and shuffle read/write, which is essential for spotting garbage-collection pressure (high GC time relative to task time usually means executor memory is too small or too many objects are being cached) or an executor that died and was replaced. Because the default UI on port 4040 only exists while the driver process is alive, production clusters set spark.eventLog.enabled=true and spark.eventLog.dir to a persistent location like HDFS or S3, then run the standalone Spark History Server so completed applications' UIs remain browsable long after the job finishes.

🏏

Cricket analogy: A team's post-match fitness report showing each player's distance covered and recovery heart rate is like the Executors tab's per-executor GC time and memory usage breakdown.

bash

# Start the standalone History Server to browse completed applications
$SPARK_HOME/sbin/start-history-server.sh

The default UI on port 4040 exists only for the lifetime of the driver process — as soon as an application finishes or crashes, that UI disappears. Without spark.eventLog.enabled=true writing to a persistent directory and a running History Server, you lose all ability to diagnose a job after the fact.

SQL Tab and Diagnosing Skew

For DataFrame and Spark SQL workloads, the SQL tab visualizes the physical execution plan as a graph of operators like Scan, Filter, Exchange (a shuffle), and SortMergeJoin or BroadcastHashJoin, with live row counts and data size annotated at each node — this is the clearest way to confirm whether Spark chose a broadcast join versus an expensive shuffle join, and to see exactly which Exchange node is moving the most data. Combined with the Stages tab, you can trace a slow SQL query down to the specific shuffle stage and skewed partition causing it, which is usually far faster than guessing from execution time alone.

🏏

Cricket analogy: A team's video analyst tracing a batting collapse back to the exact over and bowling change that triggered it, rather than just noting the final score, mirrors tracing a slow query to its exact shuffle stage via the SQL tab.

The Environment tab of the Spark UI lists every effective configuration value, JVM system property, and classpath entry actually in use for the running application — the fastest way to confirm whether a --conf flag or spark-defaults.conf setting actually took effect.

The Spark UI (default port 4040) exposes Jobs, Stages, Storage, Environment, Executors, and SQL tabs for a running application.
The Stages tab's task duration distribution is the fastest way to detect data skew — a max duration far above the median signals imbalanced partitions.
The Jobs tab's DAG visualization shows exactly which operations trigger shuffle boundaries between stages.
The Executors tab surfaces per-executor GC time and memory usage, key signals for memory pressure or undersized executors.
spark.eventLog.enabled plus the Spark History Server preserve completed application UIs beyond the driver's lifetime.
The SQL tab's physical plan graph confirms whether Spark chose a broadcast join or an expensive shuffle join.