Apache Spark (PySpark) Cheat Sheet
PySpark reference covering SparkSession setup, DataFrame transformations and actions, Spark SQL queries, caching, and writing partitioned output.
2 PagesIntermediateMar 30, 2026
SparkSession & DataFrame
Read data and run basic operations.
python
from pyspark.sql import SparkSessionspark = SparkSession.builder.appName("MyApp").getOrCreate()df = spark.read.csv("data.csv", header=True, inferSchema=True)df.printSchema()df.show(5)df.select("name", "age").filter(df.age > 21).show()
Transformations
GroupBy, joins, and column expressions.
python
from pyspark.sql import functions as Fresult = ( df.groupBy("department") .agg(F.avg("salary").alias("avg_salary"), F.count("*").alias("count")) .orderBy(F.desc("avg_salary")))df2 = df.withColumn("bonus", F.col("salary") * 0.1)df3 = df.dropna(subset=["age"])joined = df.join(other_df, on="employee_id", how="left")
Spark SQL & Caching
Query with SQL and persist intermediate results.
python
df.createOrReplaceTempView("employees")spark.sql("SELECT department, AVG(salary) FROM employees GROUP BY department").show()df.cache() # persist in memory across actionsdf.repartition(8) # increase parallelismdf.write.mode("overwrite").parquet("output/")
Core Concepts
Fundamental Spark building blocks.
- SparkSession- unified entry point for the DataFrame/SQL API
- DataFrame- distributed, immutable table of structured data
- RDD- low-level resilient distributed dataset (rarely used directly today)
- transformations- lazy ops like select/filter/groupBy that build a query plan
- actions- trigger execution, e.g. show()/collect()/count()
- partition- unit of parallelism distributed across the cluster
- Catalyst optimizer- Spark SQL's query optimization engine
Pro Tip
DataFrame transformations are lazy — nothing executes until an action like show(), collect(), or write() is called, so chain multiple filters/selects freely without worrying about intermediate computation cost.
Was this cheat sheet helpful?
Explore Topics
#ApacheSparkPySpark#ApacheSparkPySparkCheatSheet#DataScience#Intermediate#SparkSessionDataFrame#Transformations#SparkSQLCaching#CoreConcepts#Databases#MachineLearning#CheatSheet#SkillVeris
Advertisement
Sri Hayavadhana Info-Tech
Professional Web Designing Services
- Responsive Websites
- E-commerce Solutions
- SEO Friendly Design
- Fast & Secure
- Support & Maintenance
