Core APIs at a Glance
Every Spark application starts by creating a SparkSession, the single entry point for reading data, running SQL, and configuring the application, which internally manages the older SparkContext for RDD operations. The DataFrame is the primary API for most workloads - a distributed table with named columns and a schema - while the RDD remains available underneath for low-level control, and the Dataset (Scala/Java only) layers static typing on top of the DataFrame.
Cricket analogy: A match's official scorer (SparkSession) is the single point through which all scoring, reviews, and statistics flow, coordinating everything from ball-by-ball data to the final scoreboard, similar to how SparkSession manages a Spark application's entry point.
Common Transformations and Actions
Transformations like select, filter, groupBy, withColumn, and join build up a query plan lazily without touching data, while actions like show, count, collect, and take trigger the actual execution and return a result to the driver. A quick mental test during code review: if a line of code doesn't cause the cluster to actually do work, it's a transformation; if it does, it's an action.
Cricket analogy: Setting a field placement (transformation) doesn't score anything by itself - only the actual delivery and the batsman's shot (action) produces a result, mirroring how Spark transformations build a plan while actions trigger real execution.
# Cheat-sheet style common operations
df = spark.read.parquet("s3://data/events/")
# Transformations (lazy - build the plan, no execution yet)
step1 = df.select("user_id", "event_type", "amount")
step2 = step1.filter(step1.event_type == "purchase")
step3 = step2.withColumn("amount_usd", step2.amount * 1.0)
step4 = step3.groupBy("user_id").sum("amount_usd")
# Actions (trigger real cluster execution)
step4.show(10) # print first 10 rows
total_rows = step4.count()
top_rows = step4.take(5)
step4.write.mode("overwrite").parquet("s3://out/user_totals/")
Reading and Writing Data
Reading data uses spark.read.format(...).option(...).load(...) or shorthand like spark.read.csv/json/parquet, with common options including header (true/false for CSV), inferSchema, and mode (PERMISSIVE, DROPMALFORMED, FAILFAST); writing mirrors this with df.write.format(...).mode(...).save(...), where mode controls overwrite, append, ignore, or errorifexists behavior. Parquet is the default columnar format of choice for intermediate and final outputs because of its compression and predicate pushdown support, while Delta Lake adds transactional guarantees on top of Parquet.
Cricket analogy: Specifying exact match format options before a series - number of overs, powerplay rules, DRS availability - is like setting read/write options such as header, inferSchema, and mode before loading or saving a Spark DataFrame.
Quick spark-submit flags worth memorizing: --master (cluster URL), --deploy-mode (client/cluster), --executor-memory, --executor-cores, --num-executors, and --conf key=value for any Spark configuration property.
Configuration Cheat Sheet
A handful of configuration keys cover most day-to-day tuning: spark.executor.memory and spark.executor.cores control per-executor sizing, spark.sql.shuffle.partitions (default 200) sets how many partitions a shuffle produces and should be scaled to cluster size and data volume, and spark.dynamicAllocation.enabled lets the cluster manager add or remove executors based on pending task backlog. These can be set via spark-submit's --conf flag, in a SparkConf object before creating the SparkSession, or in the cluster's spark-defaults.conf for organization-wide defaults.
Cricket analogy: A team's default settings - squad size, number of overs per bowler, powerplay length - are configured before the tournament starts, similar to setting spark.executor.memory and spark.sql.shuffle.partitions before a Spark job runs.
- SparkSession is the single entry point for reading data, running SQL, and configuration.
- DataFrame is the primary API for most workloads; RDD offers low-level control; Dataset adds type safety (JVM only).
- Transformations (select, filter, groupBy, join) are lazy; actions (show, count, collect, write) trigger execution.
- spark.read/write support csv, json, parquet, jdbc, and delta formats with configurable options and write modes.
- Parquet is the default columnar format; Delta Lake adds ACID transactions on top of Parquet.
- Key tuning configs: spark.executor.memory, spark.executor.cores, spark.sql.shuffle.partitions, spark.dynamicAllocation.enabled.
- Set configs via --conf on spark-submit, a SparkConf object, or spark-defaults.conf for cluster-wide defaults.
Practice what you learned
1. What is the role of SparkSession?
2. Which of these is an action, not a transformation?
3. What does df.write.mode('append') do?
4. What does Delta Lake add on top of the Parquet format?
5. What does spark.sql.shuffle.partitions control?
Was this page helpful?
You May Also Like
Spark Best Practices
Practical guidelines for writing efficient, reliable, and maintainable Apache Spark jobs in production.
Spark Interview Questions
The core concepts, performance-tuning questions, and system-design scenarios that come up most often in Apache Spark interviews.
Building an ETL Pipeline with Spark
A practical guide to designing extract, transform, and load stages in Spark, from schema-safe ingestion to atomic loads.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics