PySpark Tuning Flags cheat sheet

Adaptive execution, shuffle sizing, join strategy, executor memory, and skew handling — the flags that decide whether a job finishes.

ProgrammingAdvanced7 sections

Adaptive query execution

spark.sql.adaptive.enabled=true: On by default since Spark 3.2. AQE re-plans between stages using real shuffle statistics, which is why static tuning matters far less than it used to.
spark.sql.adaptive.coalescePartitions.enabled=true: Merges tiny post-shuffle partitions so 200 nearly empty tasks collapse into a handful. It can only reduce the partition count, never increase it.
spark.sql.adaptive.advisoryPartitionSizeInBytes=64m: The post-shuffle partition size AQE aims for. It is ignored while parallelismFirst is on, which surprises nearly everyone tuning it the first time.
spark.sql.adaptive.coalescePartitions.parallelismFirst=false: Defaults to true, which makes AQE target minPartitionSize of 1 MB and fill the cluster rather than honor your advisory size. Set false to regain control.
spark.sql.adaptive.localShuffleReader.enabled=true: When AQE downgrades a sort-merge join to a broadcast, this reads shuffle blocks locally and skips a network round trip. Leave it on.

spark.sql.shuffle.partitions=200: The default, and wrong for almost every workload. With AQE it acts as an upper bound, so set it generously high rather than trying to be precise.
df.repartition(200, "customer_id"): A full shuffle producing even partitions with keys co-located. Use it before a wide join or a partitioned write, not as a reflex after every step.
df.coalesce(10): Merges partitions without a shuffle, but the narrow dependency propagates upstream and can throttle the parallelism of everything that feeds it.
SELECT /*+ REBALANCE(customer_id) */ * FROM orders: Asks AQE for evenly sized output partitions and splits skewed ones. A better pre-write step than a fixed repartition count you have to keep retuning.
spark.default.parallelism: Applies only to RDD operations, never to DataFrame shuffles. Setting it and expecting Spark SQL to change behavior is a common dead end.

spark.sql.autoBroadcastJoinThreshold=10485760: 10 MB by default, compared against an estimated size. Raising it to 100 MB is usually safe once you know the driver and executors have headroom.
spark.sql.autoBroadcastJoinThreshold=-1: Disables broadcasting entirely. The right first move when the driver runs out of memory collecting a small side that statistics badly underestimated.
spark.sql.adaptive.autoBroadcastJoinThreshold: AQE's own threshold, applied to actual shuffle output rather than an estimate. It falls back to the static value when you leave it unset.
df.join(broadcast(dim), "customer_id"): Forces a broadcast regardless of statistics. Only safe when the dimension is genuinely bounded, since a full copy lands in every executor.
spark.sql.broadcastTimeout=300: Seconds to wait for the broadcast to build. The error reads like an unrelated timeout, but it almost always means the small table is not small.
spark.sql.join.preferSortMergeJoin=true: Shuffle hash join uses less CPU but builds its table in memory. Sort-merge spills gracefully, which is why it remains the default for large inputs.

spark.executor.cores=5: Four or five is the practical ceiling. Beyond that, object store clients contend and a single executor loss costs proportionally more retried work.
spark.executor.memory=16g: JVM heap per executor. Divide by cores to see what one task actually gets, and that number is what predicts whether a sort spills.
spark.executor.memoryOverhead: Defaults to the larger of 384 MB and ten percent of executor memory. Python workers live in this pool, so PySpark jobs need it raised well above default.
spark.memory.fraction=0.6: The share of heap available to execution and storage together. The remaining forty percent absorbs user objects and JVM overhead.
spark.memory.storageFraction=0.5: The part of that pool cached blocks may hold against eviction. Execution can borrow past it, but cached data cannot borrow back.
spark.executor.pyspark.memory: A hard cap on Python worker memory, unset by default. Without it one pandas UDF can push the container past its limit and get the executor killed.

spark.sql.files.maxPartitionBytes=134217728: 128 MB, the target size of a single input split. Lower it to raise parallelism when a small cluster is reading a handful of very large files.
spark.sql.files.openCostInBytes=4194304: 4 MB, the assumed cost of opening a file. It is what packs many tiny files into one task, and why file count hurts scheduling more than scan time.
spark.sql.parquet.filterPushdown=true: On by default but useless unless the writer produced row-group statistics for the filtered column. Sort on your common filter column at write time.
spark.sql.sources.partitionOverwriteMode=dynamic: Overwrites only the partitions present in the DataFrame. The static default replaces the entire table and has erased plenty of production data.
spark.sql.parquet.compression.codec=zstd: snappy is the default. zstd typically yields twenty to thirty percent smaller files at similar read speed, which pays off in object store cost.
spark.sql.files.maxRecordsPerFile=1000000: Caps rows per output file, disabled by default with a value of 0. The simplest guard against one skewed partition producing a 20 GB Parquet file.

spark.sql.adaptive.skewJoin.enabled=true: On by default. AQE splits an oversized shuffle partition and replicates the matching side, which resolves most skewed joins with no code change.
spark.sql.adaptive.skewJoin.skewedPartitionFactor=5: A partition counts as skewed at five times the median. Both this and the byte threshold must be exceeded, so a uniformly large stage is left alone.
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=256m: The absolute floor before skew handling engages. Lower it when the straggler task is clearly disproportionate but still under 256 MB.
df.withColumn("salt", (rand() * 20).cast("int")): Manual salting for skew AQE cannot see, such as a group by on one hot key. Explode the other side by the same factor before joining on the salted key.
spark.shuffle.file.buffer=1m: 32 KB by default. Raising it cuts syscall overhead during heavy shuffle writes, at the cost of that buffer for every open shuffle file.
spark.eventLog.enabled=true: Without event logs the history server has nothing once the job exits, and spill metrics are unrecoverable. Turn it on before the incident, not after.

spark.sql.execution.arrow.pyspark.enabled=true: Uses Arrow for toPandas and createDataFrame, often an order of magnitude faster. It falls back silently unless you also disable the fallback flag.
spark.sql.execution.arrow.maxRecordsPerBatch=10000: The default batch size handed to a pandas UDF. Lower it when one batch of very wide rows exhausts the Python worker's memory.
df.mapInPandas(fn, schema): Streams Arrow batches to Python without materializing a whole group, which keeps memory bounded compared with applyInPandas on a skewed key.
df.explain("formatted"): Read the plan before changing any flag. Count the Exchange nodes to count shuffles, and confirm which join strategy the optimizer actually chose.
spark.sql.ansi.enabled=true: The default in Spark 4.0. Overflow and invalid casts raise instead of returning null, turning silent data corruption into a visible job failure.
spark.dynamicAllocation.enabled=true: Requires the external shuffle service or shuffle tracking. Without either, Spark reclaims executors still holding shuffle files and the stage retries forever.

From DataLane — tutorials at/blog, practice SQL live in theplayground.