Scheduling in YARN

Analogy categoryAnalogies

The Pluggable Scheduler Model

YARN deliberately delegates the question of 'which application gets the next free container' to a pluggable Scheduler component inside the ResourceManager, configured via yarn.resourcemanager.scheduler.class; it ships with three implementations, FifoScheduler, CapacityScheduler, and FairScheduler, and the Scheduler intentionally does no monitoring, restart, or guarantee beyond enforcing resource limits, since that bookkeeping belongs to the ApplicationsManager and each application's own AM.

🏏

Cricket analogy: Like a cricket board choosing between a strict seniority-based net-booking system, a quota system reserving nets for each state association, or an equal-rotation system among academies, YARN lets an admin swap in FIFO, Capacity, or Fair scheduling policy.

Capacity Scheduler: Hierarchical Queues

The CapacityScheduler organizes cluster resources into a hierarchy of queues (e.g., root.production and root.development), each configured with a guaranteed capacity percentage and an optional maximum capacity ceiling it can burst to when the cluster is otherwise idle; within a queue, applications are typically scheduled FIFO, and administrators can further set per-queue ACLs and user limits so that, for example, the production queue always has at least 70% of cluster resources even if development submits a flood of jobs.

🏏

Cricket analogy: Like a stadium's net facility permanently reserving 70% of net time for the national team's pre-series camp while allowing domestic academies to burst into the remaining nets when the national team isn't using them, mirroring the Capacity Scheduler's guaranteed-plus-burst model.

Fair Scheduler: Equal Shares Over Time

The FairScheduler instead aims to give every application roughly equal resources over time, dynamically shrinking a running job's container share as new jobs arrive so all active jobs converge toward an equal split, and it groups applications into pools (often per user or per queue) with configurable minimum-share guarantees defined in a fair-scheduler.xml allocation file; this makes it well suited to shared, multi-tenant clusters running many small ad-hoc Spark or Hive jobs where no single team should be able to monopolize the cluster for hours.

🏏

Cricket analogy: Like a shared net facility that automatically trims a team's booked net time as more academies show up so everyone converges on roughly equal practice minutes, rather than letting the first team to arrive hog the nets all afternoon.

Preemption

Because both the Capacity and Fair schedulers make soft guarantees, they support preemption: if a queue or pool is starved below its configured minimum share for longer than a configurable timeout, the scheduler can kill containers belonging to another over-capacity queue to reclaim resources, choosing the youngest containers first to minimize wasted work; this is a blunt tool, so production clusters typically enable preemption only for queues running fault-tolerant frameworks like MapReduce or Spark rather than stateful long-running services.

🏏

Cricket analogy: Like a ground authority that, if a junior academy is locked out of its guaranteed net slot too long, can bump a senior team's optional extra session, preferring to cancel a session that just started over one nearing completion to limit disruption.

xml

<configuration>
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>production,development</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.capacity</name>
    <value>70</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.maximum-capacity</name>
    <value>100</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.development.capacity</name>
    <value>30</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.development.maximum-capacity</name>
    <value>60</value>
  </property>
</configuration>

Enabling preemption on a queue running stateful, long-lived services (rather than fault-tolerant batch frameworks like MapReduce or Spark) can silently kill in-progress work with no built-in recovery, so scope yarn.resourcemanager.scheduler.monitor.enable and its preemption policies carefully per queue.

YARN's Scheduler is pluggable via yarn.resourcemanager.scheduler.class, shipping with FIFO, Capacity, and Fair implementations.
The CapacityScheduler organizes resources into hierarchical queues, each with a guaranteed capacity and an optional burstable maximum-capacity.
The FairScheduler dynamically shrinks running jobs' shares as new jobs arrive so all active applications converge toward equal resources.
FairScheduler pools (often per user or queue) are configured with minimum-share guarantees in fair-scheduler.xml.
Both schedulers support preemption to enforce their guarantees, killing the youngest containers first to minimize wasted work.
Preemption is best limited to fault-tolerant frameworks since it forcibly kills containers with no automatic recovery guarantee.