Scala4 min read
Scala Datasets: Read the Hot Path Even If You Ship PySpark
A typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
- scala
- spark
Topic
#spark
9 posts ·view all posts
Scala
Databricks9 min read
Databricks and Delta Lake: A Practical Introduction to the Lakehouse
What the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
- databricks
- delta-lake
- spark
Apache Spark9 min read
PySpark for Data Engineers: From Zero to Your First Production Job
Learn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
- spark
- python
- tutorials
DuckDB13 min read
DuckDB vs Spark on One Node: Where the Crossover Actually Happens
Real timings and costs comparing single-node DuckDB against a Spark cluster, and why the crossover point is decided by shuffle volume rather than raw data size.
- duckdb
- spark
Apache Spark11 min read
Structured Streaming in Production: Triggers, Checkpoints, Watermarks, and Exactly-Once Sinks
The four things that decide whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once really guarantees.
- spark
- streaming
Apache Spark11 min read
Spark Executor Memory: Spill, GC, and the OOM Errors That Are Really Partition-Sizing Errors
How Spark executor memory is actually divided, why most OOMs are one oversized partition rather than a small cluster, and the six failures I can now diagnose from the error text alone.
- spark
- performance
Apache Spark11 min read
Spark Data Skew: Diagnosing It Properly, Then Fixing It With Salting, AQE, and Broadcast
One task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
- spark
- performance
Apache Spark12 min read
Spark Shuffle Explained: What It Costs, How to Read It in the UI, and the Rewrites That Remove It
A shuffle is a disk write, a network fetch, and a sort you did not ask for. Here is what it costs, how to spot it in the Spark UI, and the query rewrites that delete it.
- spark
- performance
AWS9 min read
AWS Glue vs EMR for Spark: The Cost Model, the Cold Starts, and the Migration Point
Glue vs EMR for production Spark: DPU pricing vs instance pricing, cold starts, library control, and the job size where teams switch.
- aws
- spark