Spark Interview Questions cheat sheet
Shuffles, skew, partitioning, memory, and Adaptive Query Execution — the Spark answers senior data engineering interviews expect.
Execution model and architecture
Walk through what happens when you call an action on a DataFrame.- The driver builds a logical plan from the transformations, Catalyst optimizes and produces a physical plan, and the DAG scheduler splits it into stages at shuffle boundaries. Each stage becomes tasks (one per partition) sent to executors; the action triggers the whole chain. Nothing runs until the action because transformations are lazy.
What is the difference between a transformation and an action?- Transformations build the lineage graph lazily and return a new DataFrame; actions such as count, collect, show, and write force execution. The follow-up is which transformations are wide: joins, groupBy, distinct, and repartition shuffle data across the network, while map, filter, and select are narrow and stay within a partition.
How do jobs, stages, and tasks relate?- One action creates one job; the job is cut into stages at every shuffle boundary; each stage runs one task per partition. This mapping is how you read the Spark UI — a stage with 200 tasks means 200 partitions, and 200 is the default value of spark.sql.shuffle.partitions, which is usually the first thing worth tuning.
What is the role of the driver, and what makes it fail?- The driver holds the SparkContext, plans queries, schedules tasks, and collects results. It dies from collect or toPandas on a large result, from broadcasting something bigger than its heap, and from tracking millions of tasks. If the driver OOMs, the answer is almost never a bigger driver — it is not pulling data back to it.
Explain Spark on Kubernetes versus YARN in 2026.- YARN remains common in on-premises Hadoop estates, but new deployments overwhelmingly run on Kubernetes or a managed service such as Databricks, EMR, or Dataproc Serverless. Kubernetes gives container-level dependency isolation and dynamic allocation through the shuffle tracking mechanism rather than an external shuffle service.
What did Spark Connect change?- Spark Connect, GA since 3.4 and the default client in many managed runtimes, splits the client from the driver using a gRPC protocol carrying unresolved logical plans. That means your application no longer needs the full Spark JVM in-process, upgrades decouple, and a bad client cannot take down the cluster.
Shuffles, joins, and skew
Why is a shuffle expensive?- It writes map-side output to local disk, transfers it over the network, and the reduce side sorts or hashes it — so it involves serialization, disk I/O, and network at once, and it creates a stage boundary where a single straggler blocks everything. Reducing shuffles is generally the highest-leverage Spark optimization.
What join strategies does Spark choose between?- Broadcast hash join, shuffle hash join, sort-merge join, and the cartesian and broadcast nested loop fallbacks. Broadcast wins when one side fits in memory, controlled by spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB. Sort-merge is the default for two large sides. Check the plan rather than guessing, since stale statistics regularly cause a bad pick.
How do you detect and fix data skew?- In the Spark UI, look for a stage where the max task duration and shuffle read are orders of magnitude above the median. Fixes are salting the hot key with a random suffix and exploding the other side, splitting hot keys into a separate broadcast join, or letting Adaptive Query Execution do skew join splitting, which is on by default since Spark 3.2.
What does Adaptive Query Execution actually do?- At each stage boundary it uses real runtime statistics to coalesce too-many shuffle partitions into fewer, convert a sort-merge join to a broadcast join when a side turns out small, and split skewed partitions. It is enabled by default since 3.2, which is why hand-tuning spark.sql.shuffle.partitions matters much less than it did on Spark 2.
When does a broadcast join go wrong?- When the small side is not actually small: the driver collects it, then ships a copy to every executor, so a 2 GB broadcast across 100 executors is 200 GB of duplicated memory and often a driver OOM first. A broadcast hint overrides the size threshold, so hints left in code after data grows are a recurring production incident.
What is a bucketed table and when does it pay off?- Bucketing pre-shuffles data into a fixed number of files by a hash of the join key at write time, so later joins on that key skip the shuffle entirely. It pays off for a table joined repeatedly on the same key, and only if both sides share the bucket count and column. The cost is a more expensive, less flexible write path.
Partitioning and file layout
What is the difference between repartition and coalesce?- repartition triggers a full shuffle and can increase or decrease partitions with even distribution; coalesce merges partitions locally without a shuffle but only downward, and can leave uneven sizes. The trap is that coalesce(1) applies its narrow dependency upstream and can serialize the entire preceding computation onto one task.
What is the small files problem and how do you avoid it?- Thousands of tiny files inflate metadata listing, cost one task each, and destroy scan throughput. Avoid it by targeting roughly 128 MB to 1 GB per output file, coalescing before write, and enabling table-format compaction such as Delta OPTIMIZE or Iceberg rewrite_data_files. The follow-up is streaming, where small files are guaranteed and compaction must be scheduled.
How should you choose partition columns for a table?- Partition on a low-cardinality column that queries filter on, most often a date. Partitioning by a high-cardinality column such as user_id creates a directory per user and is a classic failure. Aim for partitions of at least a few hundred megabytes, and prefer Iceberg hidden partitioning or Delta liquid clustering so consumers do not need to know the layout.
What is predicate and projection pushdown in Spark?- Catalyst pushes filters and column selection into the file reader so Parquet row groups whose min/max statistics cannot match are skipped and unread columns are never decoded. It breaks when you filter on a column wrapped in a non-deterministic or unsupported function, or read through a UDF, which is a strong argument against UDFs early in a plan.
How many partitions should a job have?- A useful rule is two to four tasks per core so scheduling overhead stays low and stragglers can be balanced, with each task handling roughly 100-200 MB. Far too few partitions leaves cores idle and risks spilling; far too many makes task launch overhead dominate. With AQE on, set a generous initial value and let coalescing settle it.
Memory, caching, and performance
How is executor memory laid out?- Within the JVM heap, unified memory (default 60% of heap after a 300 MB reserve) is split between execution and storage, which borrow from each other dynamically with execution able to evict cached blocks. The rest is user memory for your objects. Off-heap overhead, sized by spark.executor.memoryOverhead, covers Python workers and shuffle buffers and is the usual cause of container kills.
What is the difference between cache and persist?- cache is persist with MEMORY_AND_DISK for DataFrames. persist lets you pick a StorageLevel including disk-only, serialized, or replicated. Both are lazy and take effect on the next action. Cache only when a DataFrame is reused across multiple actions, and unpersist explicitly, because stale cached data holds memory that execution then cannot borrow.
Why is a Python UDF slow, and what do you use instead?- A row-at-a-time Python UDF serializes each row to a Python worker and back, and it is a black box that Catalyst cannot optimize or push filters through. Prefer built-in functions; if you need Python, use a pandas UDF, which moves batches over Arrow and typically runs several times faster. In 3.5 and later, Arrow-optimized Python UDFs narrow the gap further.
How do you debug a job that is slow but not failing?- Start in the Spark UI at the longest stage, compare max and median task metrics for skew, check spill to memory and disk, and check shuffle read size against expectations. Then read the SQL tab plan for a missing broadcast, an accidental cartesian, or a scan that read every partition because pruning failed.
What causes an executor OutOfMemoryError versus a container kill?- A heap OOM comes from too much data in one task — a huge groupBy result, a wide collect_list, or an oversized broadcast. A YARN or Kubernetes kill for exceeding the memory limit comes from off-heap growth: PySpark worker processes, native libraries, or shuffle buffers. Increasing memoryOverhead fixes the second and does nothing for the first.
Structured Streaming
How does Structured Streaming actually process data?- It treats the stream as an unbounded table and runs the same Catalyst plan incrementally in micro-batches, checkpointing offsets and state so a restart resumes exactly where it stopped. Continuous processing exists but stayed experimental; in practice everyone runs micro-batch, with trigger AvailableNow for batch-style catch-up.
Explain watermarks and why they are required for stateful operations.- A watermark declares how late an event may arrive, for example withWatermark on event_time with 10 minutes. It bounds state: aggregations can finalize and drop windows older than the watermark, and stream-stream joins can expire buffered rows. Without one, state grows forever and the job eventually dies with an OOM or checkpoint bloat.
What output modes exist and when is each valid?- Append writes only new finalized rows and requires a watermark for aggregations; Update writes rows that changed since the last trigger; Complete rewrites the whole result table and only works for aggregations with bounded groups. Sinks constrain the choice — file sinks accept Append only.
What guarantees does Structured Streaming provide?- Exactly-once end-to-end when the source is replayable (Kafka offsets) and the sink is idempotent or transactional, such as Delta or a foreachBatch upsert keyed on a unique ID. With a plain HTTP or JDBC sink you get at-least-once, so downstream must deduplicate. Interviewers want you to name the source and sink requirements, not just the phrase.
Why can you not change a streaming query arbitrarily between restarts?- The checkpoint stores offsets and the schema of the state store, so changing aggregation keys, adding a stateful operator, or altering the output schema makes the checkpoint incompatible and the job fails to resume. Safe changes are filters and projections; anything stateful usually means a new checkpoint location and a planned reprocessing.
Correctness and production practices
How do you make a Spark write idempotent?- Write to a partition determined by the run's logical date and use overwrite of that partition rather than append, with dynamic partition overwrite mode or a Delta replaceWhere. On table formats, MERGE on a business key achieves the same. Appending on retry is how a backfill silently doubles a fact table.
How do you test Spark code?- Split pure DataFrame transformations from readers and writers so you can unit test the transformation against a locally created DataFrame with a fixed schema on a local session. Assert on sorted, schema-compared results rather than collect ordering, and keep a small integration suite that exercises real I/O paths.
What is the difference between RDDs, DataFrames, and Datasets today?- DataFrames and Datasets go through Catalyst and Tungsten and are what you should write; RDDs bypass the optimizer and are a low-level escape hatch. Typed Datasets exist only in Scala and Java, and their lambdas also block some optimizations. In PySpark the choice is effectively DataFrame API versus SQL, which produce identical plans.
How do you handle schema evolution in Spark reads?- Always supply an explicit schema for production reads rather than relying on inference, which samples files and can differ between runs. For evolving sources, use a table format with schema evolution rules — Iceberg tracks column IDs so renames are safe, while Parquet-on-Hive matches by name or position and silently corrupts on reorder.
What would you check first on a job whose runtime doubled overnight?- Input volume and file count first, since a source switching to smaller files changes task count dramatically. Then compare query plans between the two runs for a lost broadcast join caused by the small side crossing the threshold, and look for a new skewed key. Cluster changes and a stale cached table round out the list.
From DataLane — tutorials at/blog, practice SQL live in theplayground.