Submitting a Job: Client to Cluster
Running a job on YARN starts when a client, such as hadoop jar for MapReduce or spark-submit --master yarn, packages the application code and any dependencies, then calls the RM's submitApplication API to register a new application and receive a unique ApplicationId like application_1720000000000_0001; the RM's Scheduler queues the request, and once resources are available it allocates a first container and launches the ApplicationMaster there, which then takes over requesting further containers to run the actual map, reduce, or executor tasks.
Cricket analogy: Like a franchise submitting its squad registration to the tournament board and receiving a fixture ID, after which the board slots them into the schedule once a ground opens up and lets the team's own management run training from there.
Monitoring: Web UI and the yarn CLI
Once running, a job can be tracked through the YARN ResourceManager web UI (typically on port 8088), which lists every ApplicationId with its state, queue, and a link to the AM's own tracking UI (for MapReduce this shows per-task progress; for Spark it's the familiar Spark UI with stages and DAG visualization); the same information is available from the command line via yarn application -list, yarn application -status <appId>, and yarn logs -applicationId <appId> for retrieving aggregated logs after completion.
Cricket analogy: Like a broadcaster's scorecard app showing every live match's state and a deep link into that specific match's ball-by-ball commentary, mirroring how the RM UI links out to each application's own AM tracking page.
Log Aggregation and Debugging Failed Jobs
Because container-local logs live on the worker node's disk and are deleted after a retention window, log aggregation (enabled via yarn.log-aggregation-enable=true) copies each container's stdout and stderr into HDFS under /app-logs/<user>/logs/<applicationId> as soon as the application finishes, so yarn logs -applicationId <appId> can pull the complete history from any client even though the original containers and nodes are long gone; when debugging a failed job, checking the AM's own log first is usually more informative than any single task's log, since the AM records why it requested a restart or marked the job failed.
Cricket analogy: Like match footage being uploaded to a central broadcaster archive right after the game so any fan can rewatch it later even though the stadium's own local feed has since been wiped, and reviewing the captain's post-match debrief is usually more useful than any single player's highlight reel.
# List all applications currently in the cluster
yarn application -list
# Check detailed status of a specific application
yarn application -status application_1720000000000_0001
# Kill a running application
yarn application -kill application_1720000000000_0001
# Pull aggregated logs after the application has finished
yarn logs -applicationId application_1720000000000_0001 > job.log
# Submit a Spark job onto YARN
spark-submit --master yarn --deploy-mode cluster \
--class com.example.ETLJob \
--num-executors 20 --executor-memory 4g --executor-cores 2 \
etl-job.jarThe ApplicationId embeds the ResourceManager's start timestamp (e.g., application_1720000000000_0001), so if the RM restarts, the counter portion resets against the new epoch. Always copy the full ApplicationId string rather than assuming sequential numbers stay unique across RM restarts.
- Clients submit jobs via submitApplication, receiving a unique ApplicationId like application_1720000000000_0001.
- The RM's Scheduler queues the request and launches the AM in a first container once resources are free.
- The RM web UI (port 8088) lists every application and links to each AM's own tracking UI.
yarn application -list/-statusandyarn logs -applicationIdprovide command-line equivalents to the web UI.- Log aggregation copies container stdout/stderr into HDFS under /app-logs/ once the application finishes, since node-local logs are deleted after a retention window.
- When debugging a failure, check the AM's own log first -- it records why it requested a restart or marked the job failed.
Practice what you learned
1. What command lists all applications currently known to the ResourceManager?
2. By default, on which port does the ResourceManager web UI run?
3. Why does `yarn logs -applicationId <appId>` still work after a job's containers and nodes are gone?
4. When debugging a failed YARN application, which log is usually most informative to check first?
5. What does spark-submit --master yarn --deploy-mode cluster do differently from client mode?
Was this page helpful?
You May Also Like
YARN Architecture
An overview of YARN's split-brain design that separates cluster resource management from per-application scheduling and monitoring.
Application Master
How YARN's per-application ApplicationMaster negotiates resources, drives task execution, and recovers from failure.
Scheduling in YARN
How YARN's pluggable schedulers -- FIFO, Capacity, and Fair -- decide which application gets the next available container.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics