Cluster Managers: YARN and Kubernetes

Analogy categoryAnalogies

Spark itself only defines how a distributed computation is split into tasks; it relies on a cluster manager to actually acquire CPU and memory across machines and launch executor processes. Spark supports a built-in standalone manager, Apache Mesos (now deprecated), YARN, and Kubernetes, with YARN and Kubernetes being the two dominant choices in production because they support multi-tenant resource sharing, queues or namespaces, and security integration.

🏏

Cricket analogy: Just as a franchise's team owner doesn't personally negotiate stadium bookings and ground staff but delegates it to the venue authority, Spark delegates the job of finding grounds (machines) and pitches (executors) to a cluster manager like the IPL's venue committee handling multiple matches.

Running Spark on YARN

On YARN, the Spark driver in cluster mode runs inside an ApplicationMaster container that YARN's ResourceManager launches on any NodeManager in the cluster, while in client mode the driver stays on the submitting machine and only the ApplicationMaster runs in the cluster. The ResourceManager's scheduler (typically the Capacity Scheduler or Fair Scheduler) grants containers on NodeManagers for Spark executors, and dynamic allocation relies on YARN's external shuffle service so executors can be removed without losing shuffle data that downstream stages still need.

🏏

Cricket analogy: In an IPL auction, the BCCI's auction committee (ResourceManager) decides which franchise gets which player slot, while a franchise's coach in the dugout (Spark driver in client mode) directs strategy without owning the auction process itself.

bash

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 20 \
  --executor-memory 8g \
  --executor-cores 4 \
  --queue production \
  --class com.example.SalesETL \
  s3://my-bucket/jobs/sales-etl.jar \
  --input s3://my-bucket/raw/sales/ \
  --output s3://my-bucket/curated/sales/

Running Spark on Kubernetes

On Kubernetes, spark-submit talks directly to the Kubernetes API server, which schedules a driver pod; that driver pod then creates executor pods itself via the API rather than going through an intermediate resource negotiator like YARN's ApplicationMaster. Executors run as regular pods governed by Kubernetes namespaces, resource quotas, and RBAC service accounts, and pod templates let teams attach sidecars, node selectors, or tolerations so Spark workloads follow the same governance as other containerized services in the cluster.

🏏

Cricket analogy: A modern franchise using a data-driven front office lets the head coach directly request specific net bowlers from the academy pool without going through a separate board committee, similar to how the Spark driver pod requests executor pods directly from the Kubernetes API.

bash

spark-submit \
  --master k8s://https://k8s-apiserver.example.com:6443 \
  --deploy-mode cluster \
  --name sales-etl \
  --conf spark.kubernetes.namespace=data-eng \
  --conf spark.kubernetes.container.image=myregistry/spark:3.5.0 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
  --conf spark.executor.instances=20 \
  --class com.example.SalesETL \
  local:///opt/spark/jobs/sales-etl.jar

Dynamic allocation on Kubernetes needs an external shuffle service equivalent — either Spark's Kubernetes shuffle tracking (spark.dynamicAllocation.shuffleTracking.enabled) or a dedicated shuffle service DaemonSet. Without it, scaling executors down can silently drop shuffle files that later stages still need, causing FetchFailedException errors mid-job.

Choosing the Right Cluster Manager

Choosing between YARN and Kubernetes usually comes down to existing infrastructure and operational model: organizations already running Hadoop with HDFS and Hive typically keep Spark on YARN to share the same queues, capacity scheduler, and data locality benefits, while cloud-native teams standardizing on Kubernetes for all workloads prefer running Spark there to get a single control plane, container-level isolation, and easier autoscaling via cluster autoscalers. YARN's Capacity Scheduler offers mature multi-tenant queue management with preemption, whereas Kubernetes offers stronger workload isolation through containers and namespaces but historically weaker native support for Spark's external shuffle service, which improved with Kubernetes shuffle tracking and dynamic allocation features in Spark 3.x.

🏏

Cricket analogy: A team built around traditional red-ball first-class talent naturally stays with domestic first-class cricket infrastructure, while a franchise built for T20 leagues invests in a completely different training and scouting setup, just as YARN suits Hadoop-native shops and Kubernetes suits container-native ones.

YARN's Capacity Scheduler organizes cluster resources into hierarchical queues (e.g., production, adhoc, ml) with guaranteed minimum capacity and optional elasticity to borrow idle capacity from other queues — a mature multi-tenancy model that Kubernetes ResourceQuotas and namespaces approximate but don't replicate exactly.

Spark relies on an external cluster manager — standalone, YARN, or Kubernetes — to actually acquire machines and launch executors.
On YARN, the ResourceManager grants containers on NodeManagers, and the Spark driver runs as (or alongside) a YARN ApplicationMaster.
On Kubernetes, the driver runs as a pod and creates executor pods directly via the Kubernetes API, without an intermediate ApplicationMaster layer.
Dynamic allocation requires an external shuffle mechanism on both YARN (shuffle service) and Kubernetes (shuffle tracking or a shuffle service DaemonSet).
YARN suits organizations already invested in Hadoop/HDFS infrastructure with mature multi-tenant queue scheduling.
Kubernetes suits cloud-native, containerized environments that want a single control plane and stronger workload isolation.