Spark vs Hadoop MapReduce

Analogy categoryAnalogies

Execution Model: DAGs vs Rigid Map/Reduce Phases

Apache Spark executes a job as a directed acyclic graph (DAG) of in-memory transformations across a cluster, whereas classic Hadoop MapReduce breaks a job into rigid map and reduce phases that read and write intermediate results to HDFS between stages. This structural difference is why Spark can express complex multi-stage pipelines - filters, joins, aggregations - in a single optimized execution plan instead of chaining multiple separate MapReduce jobs.

🏏

Cricket analogy: A one-day match plotted as a single fluid strategy across 50 overs (Spark's DAG) adapts far better than treating each 10-over block as an isolated mini-match with a fresh team huddle each time, like MapReduce's rigid phase boundaries.

text

Conceptual comparison for a "count events per user" job:

# MapReduce (2 separate jobs, each round-trips through HDFS)
Job 1 - Map: emit (user_id, 1) for each event
Job 1 - Reduce: sum counts per user_id -> write to HDFS
Job 2 - Map: read Job 1 output, filter users with count > 100
Job 2 - Reduce: write final filtered result to HDFS

# Spark (one DAG, no intermediate HDFS writes)
events_df = spark.read.parquet("hdfs:///events/")
result = (events_df
          .groupBy("user_id").count()
          .filter("count > 100"))
result.write.parquet("hdfs:///active_users/")

Performance on Iterative Workloads

Because Spark can cache an RDD or DataFrame in executor memory across iterations, algorithms like logistic regression training or PageRank that repeatedly scan the same dataset run dramatically faster than on MapReduce, which must reread and rewrite the full dataset from HDFS on every iteration. Benchmarks commonly cited by Spark's original authors showed 10-100x speedups on iterative machine learning workloads compared to equivalent MapReduce jobs.

🏏

Cricket analogy: A net bowler who keeps the same set of stumps and pitch conditions cached for repeated practice deliveries trains faster than one who resets the entire pitch and equipment between every single ball, like Spark reusing cached RDDs across iterations.

Fault Tolerance: Lineage vs Replication

Spark achieves fault tolerance through RDD lineage: each RDD tracks the sequence of transformations used to build it, so if a partition is lost when an executor fails, Spark simply recomputes just that partition from its lineage rather than restarting the whole job. Hadoop MapReduce instead relies on HDFS's block replication (typically 3x) so that if a node fails, a replica of the input data is already available elsewhere, but any partially completed map or reduce task still has to restart from the beginning of its phase.

🏏

Cricket analogy: If a batsman like Rohit Sharma retires hurt mid-innings, the team doesn't restart the whole innings from ball one - the scoreboard's lineage of overs already bowled stays intact, just as Spark recomputes only the lost partition, not the whole job.

Both frameworks commonly run on the same YARN cluster manager and read from the same HDFS storage layer - migrating from MapReduce to Spark on an existing Hadoop cluster is usually a matter of swapping the execution engine, not the storage layer.

When MapReduce Still Makes Sense

MapReduce still shows up in production for extremely large, purely sequential batch jobs where cost predictability and simplicity matter more than raw speed - for example, a nightly archival compaction job that touches petabytes once and never revisits the data, where Spark's in-memory advantage provides little benefit. Many organizations that migrated from MapReduce to Spark also kept Hive running (historically on MapReduce, now often on Spark or Tez as its execution engine) for ad hoc SQL access to the same Hadoop-managed data.

🏏

Cricket analogy: A club that still plays the traditional 50-over format for its annual charity match, even though T20 is faster and more popular, values the format's simplicity and predictability, like a team keeping MapReduce for a straightforward one-off nightly batch job.

Don't assume Spark is automatically faster for every workload: a single-pass job that reads data once, does minimal transformation, and writes once may see little benefit from Spark's in-memory caching and can even be slower if cluster startup overhead dominates a small job.

Spark executes a job as an in-memory DAG; MapReduce executes rigid map/reduce phases with HDFS round-trips between them.
Spark's ability to cache intermediate data gives it 10-100x speedups on iterative workloads like machine learning training.
Spark recovers from failure via RDD lineage recomputation; MapReduce relies on HDFS block replication plus phase restarts.
Both frameworks commonly share the same YARN cluster manager and HDFS storage layer.
MapReduce still fits large, single-pass, cost-predictable batch jobs where Spark's in-memory advantage isn't needed.
Hive can run on MapReduce, Tez, or Spark as its execution engine, decoupling SQL access from the underlying engine choice.
Single-pass, low-transformation jobs may see little benefit from Spark's caching model.