Open Table Formats (Iceberg, Delta, Hudi) cheat sheet
Metadata layouts, time travel, schema evolution, catalogs, and maintenance across the three table formats.
What a table format adds
ACID commits over object storage- Readers never see a half-written table. This is what separates a table from a directory of Parquet files.
Snapshot isolation and time travel- Every commit is a snapshot you can query. Turns "what did this look like yesterday" into a WHERE clause.
Schema evolution by column ID- Rename or reorder columns without rewriting data, because the format tracks identity separately from position.
Hidden partitioning (Iceberg)- Consumers filter on the timestamp; Iceberg maps it to partitions internally. Removes the partition-column footgun.
Statistics for file pruning- Min/max per column per file, so the engine skips files without opening them. The main performance mechanism.
Iceberg metadata
metadata.json → manifest list → manifest → data files- Three levels. A commit writes a new metadata.json and atomically swaps the catalog pointer.
SELECT * FROM db.table.snapshots- Metadata tables expose snapshots, manifests, files, and history as queryable relations. Excellent for debugging.
ALTER TABLE ... ADD PARTITION FIELD bucket(16, customer_id)- Partition evolution. Old data keeps its layout; new data uses the new spec, with no rewrite required.
CALL catalog.system.rewrite_data_files(table => 'db.t')- Compaction as a stored procedure. Small-file accumulation is the main Iceberg operational task.
CALL catalog.system.expire_snapshots(older_than => ...)- Bounds metadata and storage growth. Skipping it is how Iceberg tables become slow to plan.
Delta Lake
_delta_log/00000000000000000001.json- One JSON commit per transaction, with a Parquet checkpoint every ten commits to bound log replay.
DESCRIBE HISTORY table- Full commit history with operation metrics. The first stop when row counts move unexpectedly.
OPTIMIZE table- Compaction. Combine with liquid clustering on new tables rather than static ZORDER.
VACUUM table RETAIN 168 HOURS- Physically deletes files outside retention. Going below your time-travel window breaks it permanently.
Deletion vectors- Marks deleted rows instead of rewriting files, making DELETE and MERGE far cheaper. Requires reader support.
Hudi
Copy-on-Write versus Merge-on-Read- CoW rewrites files on update for fast reads; MoR appends delta logs for fast writes. Pick by workload, not by default.
Record keys and precombine field- Hudi is upsert-first by design, which suits high-frequency CDC better than the other two formats.
Timeline as the metadata log- An instant-based commit timeline plus clustering and cleaning services that run inline or async.
Catalogs
AWS Glue Data Catalog- Default on AWS. Widely supported for reads; write support across engines is less uniform than the REST catalog.
REST catalog (Polaris, Nessie, Lakekeeper)- The direction the ecosystem is moving. One protocol every engine can implement, with real credential vending.
Unity Catalog- Governance-first with lineage and fine-grained grants. Now exposes an Iceberg REST endpoint for external engines.
One catalog, one writer at a time- Two engines writing through different catalogs to the same files will corrupt the table. This is the cardinal rule.
Choosing
Iceberg for multi-engine neutrality- Snowflake, Spark, Trino, Flink, and DuckDB all read it, and most can write it. The safest long-term bet.
Delta inside Databricks- Native, best-optimized, and the platform's default. Use UniForm if external Iceberg readers need access.
Hudi for heavy streaming upserts- Strongest record-level upsert story, at the cost of a smaller ecosystem and more tuning knobs.
None of them for small data- Under a few hundred GB with one engine, plain Parquet or a warehouse table is simpler and faster to operate.
From DataLane — tutorials at/blog, practice SQL live in theplayground.