The SparkSession

Analogy categoryAnalogies

Before Spark 2.0, developers had to create and juggle separate SparkContext, SQLContext, and HiveContext objects depending on whether they needed RDDs, DataFrames, or Hive integration. SparkSession, introduced in Spark 2.0, unifies all of these into a single entry point that exposes the DataFrame, SQL, streaming, and configuration APIs consistently.

🏏

Cricket analogy: Before central contracts unified selection under one board, cricketers dealt with separate state associations, county boards, and national federations; Spark 2.0's SparkSession similarly unified the old separate SparkContext, SQLContext, and HiveContext into one entry point.

Creating a SparkSession

A SparkSession is created with a fluent builder pattern: SparkSession.builder is chained with .appName() to name the application, .master() to specify where it runs, optional .config() calls, and finally .getOrCreate() to construct (or reuse) the session. In interactive shells like spark-shell and pyspark, this is done automatically and bound to the variable spark.

🏏

Cricket analogy: Setting the match format, venue, and umpire panel before the toss, the way a match referee configures conditions ahead of play, mirrors calling .appName(), .master(), and .getOrCreate() to configure and build a SparkSession.

Session Reuse and Configuration

SparkSession.getOrCreate() behaves like a singleton per JVM: if an active session already exists it is returned as-is, so calling it repeatedly across different parts of an application does not spin up multiple sessions. Some configuration can still be adjusted after the session exists using spark.conf.set(), though certain settings like executor memory are fixed at JVM startup.

🏏

Cricket analogy: A single match referee's authority persists for the whole day rather than a new referee being appointed after every over, similar to how SparkSession.getOrCreate() reuses the same active session instead of creating a new one each call.

python

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("CustomerChurnETL")
    .master("local[*]")
    .config("spark.sql.shuffle.partitions", "8")
    .config("spark.executor.memory", "4g")
    .getOrCreate()
)

df = spark.read.option("header", True).csv("data/customers.csv")
df.printSchema()

spark.stop()

SparkSession.getOrCreate() is effectively a singleton pattern per JVM: if an active SparkSession already exists, it is returned as-is rather than creating a new one, even if you pass different config options to a later getOrCreate() call. To change certain configs after creation, use spark.conf.set(), though some settings (like executor memory) are fixed at JVM startup and can't be changed at runtime.

SparkSession, introduced in Spark 2.0, unifies SparkContext, SQLContext, and HiveContext.
SparkSession.builder uses a fluent builder pattern ending in .getOrCreate().
getOrCreate() returns an existing active session instead of creating a duplicate.
.config() options set before creation configure the underlying SparkContext.
spark.conf.set() can adjust some settings after the session exists, but not all.
spark-shell and pyspark REPLs automatically provide a pre-built SparkSession as the variable spark.

Practice what you learned

1. What did SparkSession replace when introduced in Spark 2.0?

2. What does SparkSession.builder.getOrCreate() do if a session is already active?

3. In an interactive pyspark shell, what variable is the pre-built SparkSession bound to?

4. Which method lets you set some configuration values after a SparkSession already exists?

Was this page helpful?

Installing and Running Spark

Your First Spark Job

Related Study Notes in Programming

Browse all study notes

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Hadoop Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#ApacheSparkStudyNotes#TheSparkSession#SparkSession#Creating#Session#Reuse#StudyNotes#SkillVeris#ExamPrep

The SparkSession