What Is Apache Spark?
Apache Spark is a unified, in-memory distributed computing engine created originally at UC Berkeley's AMPLab, designed to process large datasets across a cluster of machines much faster than Hadoop MapReduce by keeping intermediate data in RAM through Resilient Distributed Datasets (RDDs). Spark became an Apache top-level project in 2014 and today offers APIs in Python, Scala, Java, and R.
Cricket analogy: Like Virat Kohli reviewing a DRS replay instantly on the stadium screen instead of waiting for a courier to deliver the footage, Spark keeps RDDs in RAM so iterative queries avoid slow disk round-trips.
Why Spark Outpaces Hadoop MapReduce
Hadoop MapReduce writes intermediate results to disk between every map and reduce phase, which makes iterative algorithms painfully slow. Spark instead builds a Directed Acyclic Graph (DAG) of transformations and executes them largely in memory, giving 10-100x speedups on iterative workloads like logistic regression, k-means clustering, and PageRank.
Cricket analogy: Rebuilding a batting lineup analysis from scratch after every over, the way Hadoop MapReduce writes intermediate results to disk between jobs, is far slower than Spark's DAG scheduler, which is like a coach such as Ricky Ponting updating a live tactics board between overs without redrawing it.
One Engine, Many Workloads
Spark's biggest structural advantage is that Spark SQL, Structured Streaming, MLlib, and GraphX all run on the same core execution engine, sharing the same optimizer and cluster resources. This means one cluster and one API family can handle batch analytics, real-time streaming, machine learning, and graph algorithms without stitching together separate specialized systems.
Cricket analogy: A single all-rounder like Ravindra Jadeja who can bowl, bat, and field brilliantly replaces needing three specialists, similar to how Spark's one engine handles SQL queries, streaming, and machine learning instead of stitching together separate systems.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("WordCount").master("local[*]").getOrCreate()
text = spark.read.text("s3://my-bucket/sample.txt")
words = text.selectExpr("explode(split(value, ' ')) as word")
counts = words.groupBy("word").count().orderBy("count", ascending=False)
counts.show(10)
spark.stop()Spark itself has no built-in distributed storage - it reads and writes from external systems like HDFS, Amazon S3, Azure Data Lake, or Apache Cassandra. Spark is purely a compute engine that sits on top of your existing storage layer.
- Spark is an in-memory, distributed compute engine originally built at UC Berkeley's AMPLab.
- RDDs (Resilient Distributed Datasets) are Spark's original fault-tolerant, distributed data abstraction.
- Spark can outperform Hadoop MapReduce by 10-100x on iterative workloads by caching data in memory.
- A single Spark engine unifies batch SQL, streaming, machine learning (MLlib), and graph processing (GraphX).
- Spark does not provide its own storage; it reads from HDFS, S3, and other external systems.
- Lineage tracking lets Spark recompute lost partitions instead of replicating data for fault tolerance.
Practice what you learned
1. What was Spark originally developed to improve upon?
2. Which of the following is Spark's original core data abstraction?
3. Which statement about Spark's storage is accurate?
4. Which of these is NOT part of Spark's unified engine?
5. Where was Apache Spark originally developed?
Was this page helpful?
You May Also Like
Spark Architecture
How Spark's driver, cluster manager, and executors work together to run a distributed job, from DAG construction to parallel task execution.
Installing and Running Spark
How to install Apache Spark's prerequisites, run it locally for development, and choose the right cluster mode for production.
The SparkSession
How SparkSession became Spark's unified entry point since version 2.0, how to build one, and how its session-reuse behavior works.
Your First Spark Job
Writing a complete read-transform-write Spark job, and understanding the crucial difference between lazy transformations and eager actions.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics