Delta Lake vs Apache Iceberg: Default on Databricks vs Shared Lake

A table format is the metadata + file protocol that turns a folder of Parquet into something you can UPDATE safely. Without it you have a lake of files and a prayer. With it you have snapshots, a writer protocol, and a catalog that says which snapshot is current.

Delta Lake and Apache Iceberg both do that job. In 2026 they overlap on features people used to pick sides over (deletion vectors, better DML, richer types). The remaining difference is where the happy path lives and who else can read the files without a science project.

flowchart TD
  pick[Need ACID on object storage] --> home{Primary engine?}
  home -->|Databricks| delta[Delta Lake]
  home -->|Snowflake + Spark| ice[Iceberg]
  home -->|multi-writer later| catalog[One catalog, one writer]

What both formats actually are

Object storage is not a database. Two jobs writing _SUCCESS files will corrupt each other. A failed write leaves a half folder. DELETE FROM is “rewrite these files and hope readers notice.”

Delta and Iceberg both add:

A log of commits (Delta: _delta_log/; Iceberg: metadata + manifest files)
Atomic publish of a new table version
Snapshot reads so a long query does not see a half MERGE
Schema (and usually schema evolution rules)
A catalog binding (Hive, Unity, Glue, Snowflake, REST, Nessie, Polaris / Open Catalog — pick one and write it down)

They do not add a query engine. Spark, Trino, Snowflake, Athena, Flink, and DuckDB are engines. The format is the contract those engines agree to, or do not.

If you cannot name the writer and the catalog, you do not have a table. You have a prefix.

Feature comparison (2026, practical)

Topic	Delta Lake	Apache Iceberg
Default home	Databricks	Multi-engine (Spark, Trino, Snowflake, Athena, Flink)
Commit log	`_delta_log` JSON / checkpoint	Metadata + manifests (Avro/JSON)
Partitioning	Hive-style partitions; liquid clustering	Hidden partitioning (writers do not need to know partition values)
Layout tuning	OPTIMIZE, Z-ORDER, liquid clustering	Compaction, sort orders, file sizing — engine-specific jobs
Deletion vectors	Yes (Databricks / recent Delta)	v3 (engine support varies)
Time travel	Version / timestamp	Snapshot id / timestamp
Change data	CDF	Incremental / row lineage (v3, engine-dependent)
Interop story	UniForm → Iceberg metadata; Delta Sharing	Native Iceberg readers everywhere that kept up
Worst mismatch	Non-Databricks writers / old OSS readers	v2 readers on v3 tables; two catalogs for one table

Feature tables rot. Treat this as a decision aid, then confirm the reader versions you actually run.

Delta Lake: the Databricks default

Time travel, OPTIMIZE, Z-ORDER, deletion vectors, liquid clustering. Best docs and tooling live in Databricks. Other engines can read Delta (or UniForm Iceberg metadata) but the happy path is Spark on Databricks.

from delta.tables import DeltaTable

customers = DeltaTable.forName(spark, "silver.customers")
(
    customers.alias("t")
    .merge(updates.alias("s"), "t.customer_id = s.customer_id")
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute()
)

-- Databricks SQL
select * from silver.customers version as of 42;
optimize silver.customers;

Why teams stay on Delta

One vendor path: Unity Catalog + jobs + SQL warehouses + the table format were designed together.
Liquid clustering reduces the “we picked the wrong partition column in 2023” tax.
CDF is a known pattern for downstream incremental jobs.

Why teams leave (or dual-write metadata)

Snowflake / Athena / a Flink job needs the same gold without running Databricks.
A regulator or platform team wants an open format as policy. Policy is not a reader. Confirm the reader.

Deep dive on the lakehouse shape: Databricks + Delta.

OSS Delta outside Databricks is real and also how you discover that “Delta” meant “Databricks Delta plus three preview flags” in your last job. Pin versions.

Iceberg: the shared-lake default

Snapshots, hidden partitioning, growing v3 features (deletion vectors, row lineage). Hidden partitioning is the feature people under-sell: you write WHERE date = '2026-08-11' and the engine prunes; you do not sprinkle dt= through every job as a social contract.

-- Snowflake-managed Iceberg (writer = Snowflake)
create iceberg table analytics.gold.orders (
  order_id number,
  amount number(12, 2),
  ordered_at timestamp_ntz
)
  catalog = 'SNOWFLAKE'
  external_volume = 'ev_analytics'
  base_location = 'gold/orders/';

# Spark reader — catalog must already know the table
orders = spark.read.format("iceberg").load("analytics.gold.orders")

Snowflake-managed Iceberg keeps files in your bucket. That is the interoperability play with warehouse SQL. Details: Snowflake Iceberg.

Why teams pick Iceberg

More than one engine will query gold this year.
Catalog options (Glue, REST, Polaris, Snowflake, Unity-as-Iceberg) match a multi-platform org.
Hidden partitioning ages better than a Hive partition scheme you regret.

Why teams regret a casual Iceberg mandate

Every engine is on a different Iceberg version.
Compaction is “someone’s job” and that someone left.
Two catalogs registered the same prefix.

Catalogs are the actual lock-in

The format is portable. The catalog is where names live.

Catalog	Typical pairing	Watch-out
Unity Catalog	Databricks + Delta (or Iceberg tables UC knows)	External engines need a documented bridge
Snowflake	Managed Iceberg or linked Iceberg	Volume IAM + writer rule
AWS Glue	Athena, EMR, Spark on AWS	Database/table identity drift vs “S3 folder”
REST / Polaris / Nessie	Multi-engine Iceberg	Who runs the catalog HA?
Hive metastore	Legacy Spark	Still everywhere; still how names fork

A table in two catalogs is two tables that happen to share files until they do not. Pick one catalog as system of record for names. Others attach or read.

Deletion vectors and v3 (why “Iceberg is slow at MERGE” aged out)

Older Iceberg DML rewrote files. That made MERGE on changing dimensions expensive and made people say “Iceberg is for append.” Deletion vectors (Delta first in practice, Iceberg v3 now) mark deleted rows without rewriting the whole file. Readers must understand the vector file.

Rules:

Upgrade readers before you create v3 / DV tables in prod.
Do not mix a v2-only Trino with a v3 Snowflake-managed table and call it “open.”
Compaction still matters. DVs postpone rewrite; they do not delete the need for maintenance.

Row lineage / CDF-shaped reads are how you avoid full-table diffs for downstream incrementals. Only useful if the consumer speaks them. Otherwise you still have updated_at > $watermark.

UniForm and “we will have both”

UniForm (and similar “generate Iceberg metadata from Delta”) lets a Databricks-written table show up as Iceberg to Snowflake or Trino. Catalog-linked databases do the reverse flavor: Snowflake reads what another catalog owns.

This is good. It is also how architecture decks hide a writer.

Write this sentence in the runbook:

Writer of record: Databricks job gold_orders commits Delta. Iceberg metadata is generated. Snowflake is a reader. Spark jobs must not INSERT into the same prefix.

If you cannot write that sentence, UniForm is a demo.

Never two independent writers on the same files without a catalog story you can draw on a whiteboard. “Optimistic concurrency will save us” is not a story. It is a postmortem.

flowchart TD
  prefix[One S3 / GCS prefix]
  spark[Spark MERGE] --> prefix
  wh[Warehouse MERGE] --> prefix
  prefix --> a[Snapshot A]
  prefix --> b[Snapshot B]
  a --> split[Split-brain table]
  b --> split

Two writers, one prefix. You do not get a table. You get two logs that share files until they do not.

-- Databricks: who committed, and whether two jobs are writing
describe history silver.customers;

-- Iceberg (Spark SQL): same question, different catalog
select committed_at, snapshot_id, operation
from analytics.gold.orders.snapshots
order by committed_at desc
limit 20;

If two job names show up as writers, you do not have a catalog story. You have a race.

Partitioning and layout (the performance you still own)

Delta / Databricks. Liquid clustering (when you have it) beats a 2019 partitionBy("date", "country") you cannot change. Z-ORDER is a rewrite you schedule. OPTIMIZE small files or the SQL warehouse will open 80,000 objects.

Iceberg. Hidden partitions + compaction jobs. File size targets are real. A thousand 2 MB files per snapshot will make Athena and Snowflake equally sad. Who runs compaction? If the answer is “automatic on Databricks” you are back to a vendor path. If the answer is “Spark weekly” put it on a DAG with an owner.

Both. Do not Hive-partition on high-cardinality ids. Do not skip a sort/cluster key on a 20 TB fact that is always filtered by account_id. The format will not invent prune columns for you.

Change data and rebuilds

Medallion still applies. Bronze is append. Silver MERGE on a business key. Gold is rebuildable from silver. The format gives you atomic publishes; it does not give you a grain.

bronze  -- append files (either format)
silver  -- MERGE on natural key
gold    -- incremental from silver watermark or CDF / lineage

If gold is “whatever the notebook left,” Delta vs Iceberg will not save you. See the lakehouse intro for the layering, not the vendor.

Pitfalls

Two writers. The classic. Spark and a warehouse MERGE on one prefix. Pick one.
Two catalogs, one prefix. Names diverge; someone DROPs the “old” table.
Lifecycle rules on the table root. You expire live Parquet. Expire snapshots through the format.
Reader version skew. Especially Iceberg v3 and Delta min reader versions.
Assuming time travel is a backup. It is a window. Test restore. Off-account copy is still a backup.
UniForm as a strategy with no refresh story. Metadata generation lags; BI sees yesterday; you “fix” it with a second copy.
Partition-on-date plus a team that queries WHERE year(ts) = 2026. You broke prune on both formats.
Picking Iceberg on a Databricks-only team to satisfy an architecture review. You paid the interop tax and bought no interop.

Decision rules

One engine, and it is Databricks → Delta. Use UniForm only when a named second engine must read.
Snowflake + a second query engine (Spark, Trino, Athena) → Iceberg, with Snowflake-managed or an external catalog, and one writer.
Multi-engine, Databricks not in the picture → Iceberg.
“We might need Snowflake later” with no date and no consumer → Delta (or native Snowflake if that is the only engine). Do not pre-pay interop.
Existing Delta estate, new Snowflake BI → UniForm or unload, not a dual-write project in week one.
Existing Iceberg estate, new Databricks jobs → read Iceberg (or write Iceberg if Databricks is the new writer — then Snowflake becomes the reader).

Never: two independent writers. Never: format choice as a substitute for a catalog and an owner.

FAQ

Can UniForm make two writers safe? No. UniForm publishes metadata for readers. It does not elect a writer. Write the writer-of-record sentence in the runbook.

Is Iceberg slower at MERGE than Delta? On older Iceberg, file rewrite made MERGE expensive. Deletion vectors (Delta first, Iceberg v3 now) changed that — if every reader you run understands the vector file.

Do I need Iceberg if only Databricks will ever query gold? No. Delta is the happy path. Do not pre-pay interop for a consumer that does not exist.

Can I lifecycle-expire *.parquet on the table root? That deletes live snapshot files. Expire snapshots through the format. Object-store lifecycle on the table prefix is how you invent a restore drill.

Two catalogs, one prefix — is that HA? That is two names for files that will diverge. One catalog is system of record. Others attach or read.

What this means for data engineers

Pick the format that matches the primary engine, then prove the second reader with a snapshot id in a log — not with a slide. Put compaction, lifecycle exceptions, and the writer name in the same repo as the table DDL.

Delta vs Iceberg is not a personality. It is “who commits” and “who can open the files next year.” Write those two lines down and the rest of the argument gets quieter.