DuckDB vs Spark on One Node: Where the Crossover Actually Happens

The question I get asked is “at what data size should I use Spark instead of DuckDB,” and it has no answer, because data size is not what decides. I have run 4 TB jobs comfortably on one machine and watched a 60 GB job fail on the same machine three times in a row.

What decides is how much data has to move between operators. A scan that filters aggressively and aggregates into a small group space is nearly free on one node regardless of input size, because the working set never grows. A join between two large tables on a high-cardinality key has to build a hash table proportional to one input, and that number is the constraint.

So I stopped answering the size question and started measuring shuffle. Below I have the timings from a comparison I ran across five job shapes on the same data, the cost math, and the rule I now use to decide. The short version: the crossover is much further out than the industry assumes, and the failure mode when you cross it is abrupt rather than gradual.

The benchmarks everyone quotes, and what they hide

Published single-node benchmarks tend to use TPC-H or ClickBench, which are scan- and aggregate-dominated. DuckDB looks extraordinary on them, and the results are honest, but they select for exactly the workload where a single node has no disadvantage. There is no network shuffle in a group by over a few hundred groups.

The other distortion is that Spark benchmarks usually include cluster startup. A cold EMR or Glue cluster takes 90 seconds to several minutes before your first task runs. That is a real cost you pay every run in production, but it says nothing about the engine’s throughput. I measure both separately, because they lead to different decisions: cold start pushes you toward DuckDB for frequent small jobs, throughput pushes you toward Spark for large ones.

For this comparison I used a 380 GB Parquet dataset of synthetic order events, ZSTD compressed, hive-partitioned by day, on S3. The DuckDB side was a single r6id.4xlarge (16 vCPU, 128 GB RAM, local NVMe) at roughly $1.15/hour on demand. The Spark side was EMR 7.x with one m6g.xlarge primary and eight r6g.2xlarge core nodes on spot, about $1.90/hour effective, with managed scaling enabled.

What I measured

Five job shapes, same input, same output verified row-for-row.

Filter and aggregate — one month, filter on status, group by customer country. Shuffle in Spark: 1.2 GB. DuckDB: 71 seconds. Spark: 3 minutes 40 seconds excluding startup, 5 minutes 20 including. DuckDB wins by 3x on warm comparison and 4.5x door-to-door.

Star join — fact joined to four dimensions totaling 900 MB, then aggregated. Shuffle in Spark: 4 GB, mostly broadcast. DuckDB: 2 minutes 10 seconds. Spark: 6 minutes 5 seconds. DuckDB wins by nearly 3x. Broadcast joins are where a single node is structurally advantaged, because the “broadcast” is just memory.

Window functions over a large partition space — rank orders per customer over 90 days, 40 million distinct customers. Shuffle in Spark: 96 GB. DuckDB: 8 minutes 30 seconds with 34 GB spilled to NVMe. Spark: 9 minutes 15 seconds. Effectively a tie, and this is the crossover zone.

Large-to-large join — 380 GB fact self-joined on session key to compute event sequences. Shuffle in Spark: 410 GB. DuckDB: failed after 40 minutes, exhausting a 400 GB temp directory. Spark: 21 minutes. Spark wins outright, and DuckDB does not just lose, it does not finish.

Full historical rebuild — three years, deduplicate on a composite key, rewrite all partitions. Shuffle in Spark: 1.1 TB. DuckDB: not attempted after the previous result. Spark: 48 minutes. Spark is the only option.

The pattern is clean. Below roughly 100 GB of shuffle, DuckDB wins by a wide margin. Between 100 and 200 GB it is a coin flip decided by disk speed. Above about 300 GB of shuffle on a 128 GB machine, DuckDB does not merely slow down, it fails.

flowchart TD
  job["Job to place"] --> ui["Read shuffle bytes: largest stage"]
  ui --> band{"Shuffle volume"}
  band -->|"under 100 GB"| duck["DuckDB, one node, 3-5x faster"]
  band -->|"100-200 GB"| disk{"Fast local NVMe available?"}
  disk -->|"yes"| ducktry["DuckDB, expect spill, benchmark it"]
  disk -->|"no"| spark1["Spark"]
  band -->|"over 300 GB"| spark2["Spark. DuckDB will not finish."]
  duck --> reliability{"Must survive node loss mid-run?"}
  reliability -->|"yes"| spark3["Spark, or checkpoint per partition"]
  reliability -->|"no"| ship["Ship it"]

Shuffle bytes from the Spark UI is the input to this decision. Raw table size tells you almost nothing.

Why shuffle is the real variable

Spark’s architecture exists to solve one problem: redistributing data across machines so that rows sharing a key land on the same executor. That machinery — serialization, network transfer, disk spill, shuffle file management — is expensive, and it is the reason Spark loses to DuckDB on small jobs. It is also the reason Spark scales, because that redistribution can span 200 machines.

DuckDB has no equivalent. Its “shuffle” is a hash table in local memory, spilling to local disk when it exceeds the budget. Local disk is far faster than network, which is why DuckDB wins in the middle band. But local disk is finite and singular, so when the required state exceeds what one machine can hold, there is no next step. Spark degrades; DuckDB stops.

You can read the deciding number directly out of your existing Spark jobs. This is the highest-value diagnostic I know for this decision:

"""Extract shuffle volume per stage from the Spark History Server.
Run this against jobs you are considering moving to DuckDB. The
largest shuffleWriteBytes across stages is the number that decides."""

import requests

HISTORY = "http://spark-history:18080/api/v1"
APP_ID = "application_1748291039284_0117"

stages = requests.get(f"{HISTORY}/applications/{APP_ID}/stages").json()

rows = [
    {
        "stage": s["stageId"],
        "name": s["name"][:48],
        "input_gb": s.get("inputBytes", 0) / 1e9,
        "shuffle_write_gb": s.get("shuffleWriteBytes", 0) / 1e9,
        "spill_disk_gb": s.get("diskBytesSpilled", 0) / 1e9,
    }
    for s in stages
    if s.get("status") == "COMPLETE"
]

worst = max(rows, key=lambda r: r["shuffle_write_gb"])
print(f"peak shuffle: {worst['shuffle_write_gb']:.1f} GB  ({worst['name']})")
print(f"total input:  {sum(r['input_gb'] for r in rows):.1f} GB")

# Rule of thumb: peak shuffle under 100 GB is a strong DuckDB candidate
# on a machine with 64 GB+ RAM and fast local NVMe.

Most teams are surprised by the result. A job reading 2 TB often peaks at 8 GB of shuffle, because the filters are selective and the aggregation is coarse. That job belongs on one machine.

The cost math, which is not close

Take the filter-and-aggregate job at its measured runtimes, running once an hour, 24 times a day.

DuckDB: 71 seconds on an r6id.4xlarge. Billed per second on Fargate or as a short-lived instance, that is about 2.3 cents per run, so roughly $0.55 a day, $17 a month.

Spark on EMR: 5 minutes 20 seconds door to door on a nine-node cluster at $1.90/hour effective, so about 17 cents per run, $4.08 a day, $124 a month. If the cluster stays warm to avoid cold starts, you are paying $1,368 a month for the cluster whether jobs run or not.

That is a 7x to 80x range depending on whether the cluster is transient or persistent. Add the operational cost: someone patches the AMI, someone tunes spark.sql.shuffle.partitions, someone gets paged when spot capacity vanishes mid-shuffle. The DuckDB version has a Dockerfile.

The serverless Spark options narrow the gap on operations but not on price. Glue and EMR Serverless still bill for provisioned capacity during startup and cannot use spot, which is the same trade I worked through in Glue vs EMR. They are cheaper than a persistent cluster and much more expensive than a container.

Where Spark is genuinely irreplaceable

Four cases where I do not attempt the migration.

Shuffle-heavy joins above a few hundred gigabytes. Covered above. This is the primary case and it is not negotiable.

Jobs that must survive partial failure. Spark’s lineage and task retry mean a lost executor costs you a stage, not the run. A DuckDB process that dies at minute 38 of a 40-minute job has produced nothing. You can approximate resilience by partitioning the work and checkpointing each chunk, and for many jobs that is enough, but it is your code rather than the engine’s.

Workloads that genuinely scale beyond one machine’s ceiling. The largest single instances top out around 24 TB of memory and cost accordingly. If your working set is heading there, distribution is not optional.

Existing Spark ML and streaming pipelines. Structured Streaming and Spark MLlib have no DuckDB equivalent. If a batch job shares code with a streaming job, splitting the engines costs more in duplicated logic than it saves in compute. The PySpark guide covers that ecosystem; do not fragment it for a 5 percent cost saving.

Migrating a job, in practice

The rewrite is usually mechanical, because both engines speak SQL and the DataFrame code was probably a thin wrapper around SQL anyway. Here is the same transform in both, which is the diff I show people who assume this is a rewrite:

# Spark version
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()

orders = spark.read.parquet("s3a://acme-lake/orders/")
customers = spark.read.parquet("s3a://acme-lake/dim/customers/")

(orders
    .filter((F.col("dt") == "2026-05-12") & (F.col("status") == "completed"))
    .join(F.broadcast(customers), "customer_id")
    .groupBy("country_code")
    .agg(
        F.count("*").alias("orders"),
        F.sum("amount_usd").alias("revenue_usd"),
    )
    .write.mode("overwrite")
    .parquet("s3a://acme-lake/marts/daily_revenue/dt=2026-05-12/"))

# DuckDB version. No session, no cluster, no broadcast hint --
# the "small side" is just memory on one machine.
import duckdb

con = duckdb.connect(":memory:")
con.execute("set memory_limit = '96GB'; set threads = 16")
con.execute("install httpfs; load httpfs")
con.execute("create or replace secret s (type s3, provider credential_chain)")

con.execute("""
copy (
  select
    c.country_code,
    count(*)          as orders,
    sum(o.amount_usd) as revenue_usd
  from read_parquet('s3://acme-lake/orders/**/*.parquet',
                    hive_partitioning = true) o
  join read_parquet('s3://acme-lake/dim/customers/*.parquet') c
    on c.customer_id = o.customer_id
  where o.dt = '2026-05-12'
    and o.status = 'completed'
  group by c.country_code
)
to 's3://acme-lake/marts/daily_revenue/dt=2026-05-12/data.parquet'
(format parquet, compression zstd)
""")

Two behavioral differences to watch during migration. DuckDB and Spark disagree on some type coercions, particularly decimal precision in aggregations and timestamp timezone handling — always diff the output, not just the row count. And Spark’s overwrite mode deletes the target directory, while DuckDB’s copy to needs overwrite_or_ignore to behave similarly, so a naive port can leave stale files behind.

I keep the migrated job running alongside the original for a week, with a reconciliation query comparing both outputs on every measure. It has caught a decimal rounding difference and a null handling difference for me, both of which would have been quiet data quality bugs. The general pattern is in the data quality checks post.

Where teams get this wrong

Deciding on input size. The number that matters is peak shuffle in the largest stage. Pull it from the Spark UI rather than guessing from table size.

Running Spark on a single node “for consistency.” This is the worst configuration available: you carry JVM overhead, shuffle serialization, and driver-executor indirection with zero distribution benefit. If a job fits on one machine, use an engine built for one machine.

Not sizing the spill directory. DuckDB’s advantage in the middle band depends entirely on fast local NVMe. Running it on network-attached storage with a 40 GB volume turns a winnable job into a failure, and the error message will not mention disk speed.

Assuming linear degradation past the crossover. Spark slows down gracefully when you underprovision. DuckDB works, works, works, and then exhausts the temp directory. Test at your peak data volume, not your average.

Migrating the whole platform at once. Move one job, verify it for a week, then the next. The value is per-job, and a partial migration is a perfectly good end state — most of my clients run both and are right to.

Forgetting that Spark clusters serve other things. If the same cluster runs streaming jobs and ad-hoc notebooks, removing one batch job saves nothing. Cost savings only materialize when a cluster can shrink or disappear.

FAQ

What data size is the actual crossover?

On a machine with 64 to 128 GB of RAM and local NVMe, jobs whose largest stage shuffles under 100 GB belong on DuckDB. In terms of input size that has meant anywhere from 50 GB to 4 TB depending on selectivity, which is exactly why input size is the wrong metric.

Does DuckDB support anything like Spark’s fault tolerance?

Not natively. A crashed process loses the run. You can get most of the practical benefit by splitting work into per-partition chunks that each write atomically, so a retry redoes one chunk. For a 40-minute job, that difference matters; for a 90-second job, it does not.

Is Polars a better comparison than DuckDB?

They occupy the same niche and perform comparably on most of these shapes. DuckDB wins on SQL surface area, out-of-core robustness, and object-storage integration; Polars wins on DataFrame ergonomics and some string-heavy operations. The Spark comparison in this post applies about equally to either.

What about Spark on Databricks with Photon?

Photon closes some of the gap on scan-heavy work, and Databricks serverless reduces the cold start. It does not change the architecture: you are still paying for a cluster and a shuffle layer to process 40 GB. The comparison I make there is on total cost of ownership rather than engine speed, along the lines of Snowflake vs Databricks.

Can I use DuckDB to read Delta or Iceberg tables?

Yes, with the relevant extensions, and it has become genuinely usable for reads. Writes are more limited, and the transaction semantics are worth understanding before you rely on them — see Delta Lake vs Iceberg for what those guarantees are supposed to provide.

How much RAM should I give the single node?

Start at four times your expected peak working set, with local NVMe at ten times. For most jobs I have moved, that lands at 64 GB of RAM and a few hundred gigabytes of instance storage. Going to 256 GB extends the envelope meaningfully and still costs less than a modest cluster.

What this means for your pipelines

Go read the shuffle numbers for your ten most expensive Spark jobs this week. Not the input sizes, not the runtimes — the peak shuffleWriteBytes per stage. My prediction, based on doing this exercise at five companies, is that more than half of them shuffle under 20 GB, and every one of those is a container job waiting to happen at a fraction of the cost and a fraction of the runtime.

Then move them one at a time, run both versions in parallel for a week, and reconcile the outputs on every measure rather than the row count. The type coercion and null handling differences between the two engines are subtle enough to survive a casual check and important enough to corrupt a finance mart. This is the part of the migration where the time actually goes, and skipping it is how a cost optimization becomes an incident.

Keep the cluster for the jobs that need it. There is no prize for eliminating Spark, and the large shuffle-heavy rebuild jobs are genuinely better there. The goal is that every job runs on the smallest thing that can hold it, and for a surprising number of them, the smallest thing is one container that starts in a second.