Apache Spark9 min read
PySpark for Data Engineers: From Zero to Your First Production Job
Learn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
- spark
- python
- tutorials
All stacks · Compute & processing
Apache Spark
Distributed processing with PySpark and Spark SQL.
Apache Spark
Apache Spark11 min read
Structured Streaming in Production: Triggers, Checkpoints, Watermarks, and Exactly-Once Sinks
The four things that decide whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once really guarantees.
- spark
- streaming
Apache Spark11 min read
Spark Executor Memory: Spill, GC, and the OOM Errors That Are Really Partition-Sizing Errors
How Spark executor memory is actually divided, why most OOMs are one oversized partition rather than a small cluster, and the six failures I can now diagnose from the error text alone.
- spark
- performance
Apache Spark11 min read
Spark Data Skew: Diagnosing It Properly, Then Fixing It With Salting, AQE, and Broadcast
One task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
- spark
- performance
Apache Spark12 min read
Spark Shuffle Explained: What It Costs, How to Read It in the UI, and the Rewrites That Remove It
A shuffle is a disk write, a network fetch, and a sort you did not ask for. Here is what it costs, how to spot it in the Spark UI, and the query rewrites that delete it.
- spark
- performance
Related reading
Microsoft Fabric5 min read
Fabric OneLake and Capacity: Shortcuts First, Then Stop Starving Spark
OneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
- fabric
- azure
Polars4 min read
Polars LazyFrames: The Speedup Is the Plan, Not the Syntax
Eager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
- polars
- python
Scala4 min read
Scala Datasets: Read the Hot Path Even If You Ship PySpark
A typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
- scala
- spark
Apache Beam5 min read
Beam Runners and Dataflow Cost: Portability vs Workers That Never Scale to Zero
Write-once Beam is real. The Dataflow bill is vCPU-hours plus Shuffle plus Streaming Engine. Streaming jobs keep a worker all night. Spark is cheaper if the team already writes Spark.
- beam
- dataflow
Kubernetes4 min read
Kubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster Dies
Airflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
- kubernetes
- orchestration
Tools11 min read
Snowflake vs Databricks in 2026: Pick the Workload, Not the Logo
An honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
- snowflake
- databricks
- tools
About Apache Spark
Spark is still the engine that processes more data than anything else in the industry — and the framework where the gap between "it runs" and "it runs well" is widest. Shuffles, skew, and memory spills separate ten-minute jobs from three-hour ones on identical clusters.
These tutorials treat Spark as a distributed SQL engine first: how the optimizer plans your query, where shuffles come from, how to read the Spark UI without dread, and the handful of configuration knobs that actually matter in PySpark jobs.
What you'll learn here
- The execution model: jobs, stages, tasks, and what triggers a shuffle
- DataFrame API fluency and reading physical plans with explain()
- Skew, spill, and partition-count tuning with the Spark UI as your guide
- Join strategies: broadcast, sort-merge, and when AQE fixes things for you
- Writing to lakes properly: partitioning, file sizes, and compaction
Frequently asked questions
Do I still need Spark if I have Snowflake or BigQuery?
For pure SQL transformation on warehouse tables, often not. Spark earns its place for ML pipelines, semi-structured and unstructured data, streaming, and cost-sensitive heavy lifting on lake storage where warehouse compute would be expensive.
Why is my Spark job slow?
In order of likelihood: too many or too few shuffle partitions, data skew concentrating work in one task, spilling to disk from undersized memory, small-file storms on read, or a join that should have been broadcast. The Spark UI stage timeline tells you which one in about two minutes.
What is a shuffle and why does everyone fear it?
A shuffle redistributes rows across the cluster by key — required by wide operations like groupBy and joins. It serializes data to disk and moves it over the network, making it the most expensive thing Spark does. Good Spark tuning is mostly shuffle management.
PySpark or Scala?
PySpark. The DataFrame API compiles to the same plans as Scala, so performance is equivalent for untyped transformations, and the ecosystem, hiring pool, and notebook experience are all Python-first now. Scala matters mainly for low-level RDD work and library development.
How big should my partitions be?
Aim for 128–256 MB of data per partition as a starting point. Thousands of tiny tasks waste scheduler overhead; a handful of giant ones create stragglers. spark.sql.shuffle.partitions and adaptive query execution are the levers.
New Apache Spark posts, straight to your inbox
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.