Installing and Running Spark

Analogy categoryAnalogies

Before you can run Spark, you need a compatible JDK installed and JAVA_HOME set correctly, since Spark itself is built on the JVM. From there you download a pre-built Spark distribution bundled with a matching Hadoop client version, extract it, and point SPARK_HOME and PATH at the extracted directory.

🏏

Cricket analogy: Before a match, a groundskeeper prepares the pitch and checks the sightscreens are in place, similar to how installing Spark requires Java installed and JAVA_HOME set correctly before Spark itself will even start.

Running Spark Locally

For development and learning, Spark's local[*] mode runs the entire application in a single JVM on your laptop, using every available CPU core as a simulated worker thread. Interactive shells like spark-shell (Scala) and pyspark (Python) start in local mode by default and are the fastest way to explore the API before writing a full application.

🏏

Cricket analogy: Practicing solo net sessions with a bowling machine before playing a real match mirrors running Spark in local[*] mode on a laptop to learn the API before deploying to a real cluster.

Choosing a Cluster Mode

For production, you submit jobs with spark-submit against a real cluster manager: Spark's own lightweight standalone manager, Hadoop YARN for existing Hadoop clusters, or Kubernetes for containerized environments. The choice affects how resources are requested, how the driver is placed, and how the job integrates with your existing infrastructure.

🏏

Cricket analogy: Choosing between playing on the local club ground, a state stadium, or a full international venue mirrors choosing between Spark's standalone cluster manager, YARN, or Kubernetes as the deployment mode.

bash

# Download and extract a pre-built Spark distribution
wget https://downloads.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz
tar -xzf spark-3.5.1-bin-hadoop3.tgz
export SPARK_HOME=$(pwd)/spark-3.5.1-bin-hadoop3
export PATH=$SPARK_HOME/bin:$PATH

# Launch the interactive Scala shell in local mode
spark-shell --master "local[*]"

# Or the Python shell
pyspark --master "local[*]"

Spark requires a compatible JDK (Java 8, 11, or 17 depending on the Spark version) to be installed before anything else. On Windows, you'll also need the winutils.exe helper binary on your PATH even for local-mode development, since Spark's Hadoop libraries expect certain POSIX-like filesystem calls.

Spark requires a compatible JDK installed and JAVA_HOME set before it will run.
local[*] mode runs Spark entirely on one machine using all available CPU cores.
spark-shell (Scala) and pyspark (Python) launch interactive REPLs with a SparkSession pre-created as spark.
Standalone, YARN, and Kubernetes are the main cluster managers for production deployments.
spark-submit is the command used to launch both local and cluster jobs.
Local mode is ideal for learning and testing before deploying to a real cluster.

Practice what you learned

1. What must be installed before Spark can run at all?

2. What does local[*] specify when starting Spark?

3. Which shell gives you a Python-based Spark REPL?

4. Which of these is a valid production cluster manager for Spark?

Was this page helpful?

Spark Architecture

The SparkSession

Related Study Notes in Programming

Browse all study notes

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheSparkStudyNotes#InstallingAndRunningSpark#Installing#Running#Spark#Locally#StudyNotes#SkillVeris#ExamPrep

Installing and Running Spark