PySpark cheat sheet
DataFrame API, windows, and the knobs that keep jobs from shuffling the world.
DataFrame basics
df = spark.read.parquet("s3://bucket/path")- Lazy — nothing runs until an action.
df.select("id", F.col("amt").alias("amount"))- Project and rename without collect().
df.filter(F.col("dt") >= "2026-08-01")- Push filters before wide transformations.
df.write.mode("overwrite").partitionBy("dt").parquet(path)- Always partition output by the query filter.
Windows and aggregations
w = Window.partitionBy("user").orderBy(F.desc("ts")) df.withColumn("rn", F.row_number().over(w))- Same semantics as SQL window functions.
df.groupBy("country").agg(F.sum("amt").alias("rev"))- Triggers a shuffle. Check partition counts after.
Performance
spark.conf.set("spark.sql.shuffle.partitions", 200)- Default 200 is often too high for small jobs.
df.explain(True)- Read the physical plan before you scale the cluster.
broadcast(small_df)- Avoid shuffle joins when one side is small.
From DataLane — tutorials at/blog, practice SQL live in theplayground.