What Is YARN?
YARN (Yet Another Resource Negotiator) was introduced in Hadoop 2.0 to fix the scalability and multi-tenancy limits of the original JobTracker, which handled both cluster resource management and job scheduling/monitoring in a single daemon. YARN splits these responsibilities: a global ResourceManager arbitrates cluster resources among competing applications, while a per-application ApplicationMaster negotiates resources and tracks task progress. This decoupling lets frameworks beyond MapReduce, such as Spark, Tez, and Flink, run natively on the same Hadoop cluster and share HDFS data without a separate cluster.
Cricket analogy: Like the BCCI setting the IPL match calendar and ground allocations while each franchise's team management decides its own playing XI and batting order, YARN's ResourceManager allocates cluster slots while each application's ApplicationMaster runs its own execution plan.
Core Components
The ResourceManager (RM) is the master daemon that tracks cluster-wide resource availability and grants containers, delegating scheduling policy to a pluggable Scheduler component and application lifecycle bookkeeping to its ApplicationsManager sub-component. Each worker node runs a NodeManager (NM) that monitors local CPU, memory, and disk usage, launches and kills containers on RM/AM instruction, and reports node health via periodic heartbeats. When a client submits a job, the RM launches a dedicated ApplicationMaster (AM) inside the first container, and that AM then negotiates additional containers from the RM to run the application's actual tasks.
Cricket analogy: Think of the RM as the ICC match referee panel tracking ground availability across a World Cup, the NodeManager as each stadium's ground staff maintaining the pitch and reporting its condition, and the ApplicationMaster as the specific match's on-field captain like Rohit Sharma directing field placements.
The Container Abstraction
A Container is YARN's unit of resource allocation, an OS-level lease that grants a bounded amount of CPU vcores and memory (and optionally other resources like GPUs) on a specific node, expressed as a ContainerId tied to an ApplicationAttemptId. The NodeManager enforces these limits using Linux cgroups so that one runaway container cannot starve its neighbors, and it kills a container that exceeds its memory bound. Because containers are generic, the same abstraction hosts an ApplicationMaster, a MapReduce task, a Spark executor, or a Tez task, giving YARN framework-agnostic multi-tenancy.
Cricket analogy: A Container is like a specific hotel room block reserved for a touring cricket team during a Test match in Australia, strictly sized for the squad, with the hotel (NodeManager) enforcing checkout if the team overstays its booked nights.
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>rm1.cluster.internal</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>65536</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>16</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>A single NodeManager reserves a small slice of memory and CPU for the OS and other daemons via yarn.nodemanager.resource.memory-mb. Never configure it to claim 100% of physical RAM, or the node itself can become unresponsive under load.
YARN vs Classic MapReduce (Hadoop 1)
In Hadoop 1's classic MapReduce, a single JobTracker handled resource management, scheduling, and fault tolerance for the whole cluster, which capped practical cluster size around 4,000 nodes and forced every workload to be a MapReduce job. YARN's decoupled architecture removed that ceiling by delegating job-specific work to per-application ApplicationMasters, enabling clusters to scale past 10,000 nodes and letting the same cluster run MapReduce, Spark, and Hive (via Tez) side by side with independent resource negotiation and failure isolation.
Cricket analogy: Like moving from one overworked chief selector picking every domestic team's XI single-handedly to a franchise model where each IPL team has its own selection committee, YARN's per-application AMs replaced the JobTracker's single point of control.
- YARN splits the old JobTracker into a global ResourceManager (resource arbitration) and per-application ApplicationMasters (task scheduling and monitoring).
- NodeManagers run on every worker node, enforce container resource limits via cgroups, and report health through periodic heartbeats.
- A Container is a bounded lease of CPU and memory on a specific node, generic enough to host an AM, a MapReduce task, or a Spark executor.
- This separation lets multiple frameworks -- MapReduce, Spark, Tez -- share one Hadoop cluster and its HDFS data.
- YARN removed the ~4,000-node ceiling of classic MapReduce by delegating job-specific bookkeeping away from a single master daemon.
- The ResourceManager itself is a single point of failure unless deployed in Active/Standby high-availability mode.
Practice what you learned
1. What was the primary motivation for introducing YARN in Hadoop 2.0?
2. Which YARN component is responsible for launching and monitoring an individual application's tasks?
3. What Linux mechanism does the NodeManager use to enforce a container's resource limits?
4. Approximately what node-count ceiling did classic Hadoop 1 MapReduce clusters run into?
5. Which of these is NOT typically hosted inside a YARN Container?
Was this page helpful?
You May Also Like
ResourceManager and NodeManager
A deep dive into YARN's two master/worker daemons -- the ResourceManager's scheduling and the NodeManager's per-node container enforcement.
Application Master
How YARN's per-application ApplicationMaster negotiates resources, drives task execution, and recovers from failure.
Scheduling in YARN
How YARN's pluggable schedulers -- FIFO, Capacity, and Fair -- decide which application gets the next available container.
Running Jobs on YARN
The end-to-end path of submitting, monitoring, and debugging a job on a YARN-managed Hadoop cluster.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics