Delta Lake Internals: The Transaction Log, Checkpoints, and Why Small Files Happen

Every Delta Lake question I get eventually reduces to one directory: _delta_log. Time travel, concurrent writers, schema enforcement, MERGE performance, the small-file mess after a month of streaming — all of it is behavior of that log.

A Delta table is not a format for data files. The data files are plain Parquet. Delta is a protocol for agreeing on which Parquet files constitute the table at a given version. Once that clicks, most of the operational behavior stops being surprising.

This post is the internals I actually use in production: how a commit works, what a checkpoint is for, what deletion vectors changed, and why small files keep happening no matter how many times you run OPTIMIZE. For the broader architecture picture, start with the lakehouse guide.

The log is the table

Open any Delta table in cloud storage and you will see Parquet files next to a _delta_log directory. That directory contains numbered JSON files, one per commit:

flowchart LR
  w[Writer] --> c0["00000000000000000000.json"]
  c0 --> c1["00000000000000000001.json"]
  c1 --> c2["00000000000000000002.json"]
  c2 --> snap["Table state at version 2"]
  files["Parquet data files"] --> snap

The table is the sum of log actions, not the sum of files in the directory.

Each JSON commit is a list of actions. The ones that matter daily:

• add — this Parquet file is now part of the table, with its partition values, size, and column-level min/max stats.
• remove — this file is no longer part of the table. The file is not deleted; it is tombstoned until VACUUM.
• metaData — schema, partitioning, table properties.
• protocol — minimum reader and writer versions. This is what breaks old clients when you enable new features.
• txn — an idempotency marker for streaming writers.

To read a table, a client lists the log, replays the actions, and ends up with a set of live files plus a schema. That is the whole trick. SELECT * FROM t VERSION AS OF 42 just stops the replay at commit 42 — which is also all time travel is.

You can look at this yourself, and you should at least once:

import json
from pyspark.sql import functions as F

log_path = "s3://lake/prod/orders/_delta_log/"

# Every commit as raw JSON, one action per line
raw = spark.read.text(log_path + "*.json")

adds = (
    spark.read.json(log_path + "*.json")
    .where(F.col("add").isNotNull())
    .select(
        F.col("add.path").alias("file"),
        (F.col("add.size") / 1024 / 1024).alias("size_mb"),
        F.col("add.stats"),
    )
)

adds.orderBy("size_mb").show(20, truncate=False)

If size_mb clusters under 10, you have a small-file problem and we will get to why. The stats column is the min/max metadata that makes file skipping work — a query with a predicate on order_date skips every file whose min/max range excludes it, without opening the file.

Optimistic concurrency: nobody locks anything

Delta writers do not take locks. Every writer:

1. Reads the current version, say 41.
2. Does its work — writes new Parquet files off to the side.
3. Attempts to commit 00000000000000000042.json atomically.

Storage systems give you an atomic “create if not exists” (on S3, Databricks uses a commit service; on ADLS and GCS the rename is atomic). Exactly one writer wins version 42. The loser does not fail — it re-reads the log, checks whether the winning commit conflicts with what it did, and retries.

flowchart TD
  a[Writer A reads v41] --> aw[A writes files]
  b[Writer B reads v41] --> bw[B writes files]
  aw --> ac["A commits v42: wins"]
  bw --> bc["B attempts v42: loses"]
  bc --> chk{Logical conflict with v42?}
  chk -->|no| retry[B retries as v43]
  chk -->|yes| fail[ConcurrentModificationException]

Losing the race is normal. Failing means the two writes touched the same data.

The conflict check is logical, not physical. Two appends to different partitions never conflict. An append and an OPTIMIZE on different files never conflict. A MERGE that read files another writer just removed does conflict, and you get a ConcurrentDeleteReadException.

The production consequence: streaming appends and batch MERGEs into the same table coexist fine if the MERGE predicate is partition-scoped so its read set is narrow. A MERGE that scans the whole table will lose races against everything. I have watched a team blame Databricks for “random” merge failures that were just an unpartitioned MERGE fighting a 30-second streaming trigger.

Checkpoints: why replay stays cheap

Replaying 4 million JSON commits to plan one query would be absurd, so Delta writes a checkpoint every 10 commits: a Parquet file containing the full table state at that version. Readers load the latest checkpoint, then replay only the JSON commits after it. _delta_log/_last_checkpoint points at the newest one.

This is invisible until it is not. Checkpoints are written by the committing writer, and on very hot tables (a commit every few seconds) checkpoint writing becomes measurable commit latency. It is also why a Delta table with heavy history still plans queries in milliseconds while a naive “list all files in the prefix” approach would take minutes.

Log entries themselves are cleaned up after delta.logRetentionDuration (default 30 days). That, together with VACUUM, bounds how far back time travel works — a point that matters when someone asks you to “just restore last quarter.”

Deletion vectors: DELETE without rewriting files

Before deletion vectors, deleting one row from a 1 GB Parquet file meant rewriting the other 999 MB. MERGE-heavy CDC pipelines spent most of their compute rewriting untouched rows.

With deletion vectors enabled, a DELETE or MERGE writes a small bitmap alongside the file marking rows as dead. The add action in the log carries a pointer to the vector. Readers reconcile at scan time: read the file, drop the flagged rows.

ALTER TABLE prod.orders
SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true');

-- This now writes a bitmap, not a rewritten file
DELETE FROM prod.orders WHERE order_id = 918273;

-- See vectors accumulating per file
DESCRIBE DETAIL prod.orders;

The trade is explicit: writes get much cheaper, reads pay a small reconciliation tax that grows with the fraction of deleted rows. OPTIMIZE purges vectors by rewriting the files for real. If your table is CDC-heavy, deletion vectors plus a nightly OPTIMIZE is the pattern; vectors alone forever is not.

One protocol warning: enabling deletion vectors bumps the reader version. External engines reading your Delta table — including some Iceberg-first tools discussed in Delta vs Iceberg — must support the feature or they stop reading the table entirely. Check every consumer before flipping it on a shared table.

Why small files happen, mechanically

Small files are not an accident. They are the log working as designed:

• Every streaming micro-batch is a commit, and every commit writes at least one file per partition it touches. A 1-minute trigger writing to 24 partitions creates up to 34,560 files a day.
• Every Spark task writes its own file. A job with 800 shuffle partitions writing 2 GB of data produces 800 files of ~2.5 MB. That is a tuning problem as much as a Delta one.
• MERGE without deletion vectors rewrites matched files at whatever size the join output happens to produce.

The fix is mostly two table properties plus a scheduled job:

ALTER TABLE prod.orders SET TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.autoOptimize.autoCompact'   = 'true',
  'delta.targetFileSize'             = '256mb'
);

-- Scheduled, partition-scoped, cheap enough to run daily
OPTIMIZE prod.orders
WHERE order_date >= current_date() - INTERVAL 3 DAYS
ZORDER BY (customer_id);

Optimized writes shuffle data before writing so each partition gets fewer, larger files. Auto compaction runs a mini-OPTIMIZE after commits that produced small files. Together they prevent most of the mess; the scheduled OPTIMIZE catches the rest and maintains clustering. If you ingest with Auto Loader, turn both on from day one — retrofitting onto a million-file table is a long afternoon.

VACUUM is the other half. remove actions tombstone files but storage still bills you for them:

VACUUM prod.orders RETAIN 168 HOURS;

Never set retention below your longest-running query or your streaming checkpoint recovery window. Deleting a file a live reader still references is the one Delta failure that loses data reads mid-query.

Pitfalls

VACUUM with aggressive retention. RETAIN 0 HOURS to save storage money breaks time travel, concurrent readers, and any stream restarting from an old checkpoint. 7 days is the floor I defend in review.

Protocol upgrades on shared tables. Deletion vectors, column mapping, and liquid clustering all bump reader/writer versions. One ALTER TABLE can cut off every external reader at once. Inventory consumers first.

Full-table MERGE under concurrent writers. Wide read sets lose optimistic-concurrency races. Scope the MERGE predicate to the partitions the source batch actually touches.

Trusting file counts over log state. Files in the directory include tombstoned and uncommitted data. DESCRIBE DETAIL and the log are the truth; aws s3 ls is not.

One-minute triggers into hundreds of partitions. The commit rate multiplied by partition count is your daily file count. Do that math before choosing a trigger interval.

FAQ

Is the transaction log a bottleneck for write throughput? Only at very high commit rates. Commits serialize on the log, so hundreds of commits per minute on one table will queue. The fix is fewer, larger commits — batch the writers, not the table.

Does OPTIMIZE block readers or writers? Neither. It rewrites files and commits like any writer, removing old files and adding compacted ones. Readers on prior versions keep working; concurrent appends do not conflict because OPTIMIZE only touches files it read.

What happens if a writer dies mid-write? Nothing. The Parquet files it wrote are orphans that no commit references; readers never see them and VACUUM eventually removes them. Atomicity lives entirely in the log commit.

How is this different from what Iceberg does? Same idea — metadata defines the table — different mechanics: Iceberg uses manifest trees and a catalog pointer swap instead of a numbered log. The operational differences are worth a full read in Delta Lake vs Iceberg.

Can I write to a Delta table without Spark? Yes. The protocol is open; delta-rs (Python/Rust) writes valid commits and is fine for small single-writer jobs. I keep multi-writer, high-volume tables on engines with mature conflict handling.

What this means for data engineers

Learn the log and Delta stops being magic. Commits explain concurrency, checkpoints explain planning speed, add/remove actions explain why storage does not shrink until VACUUM, and commit-per-micro-batch explains your file counts.

Set optimized writes, auto compaction, and a target file size on every table at creation. Schedule OPTIMIZE and VACUUM like the maintenance they are. Scope your MERGEs. Audit consumers before protocol-bumping features.

None of that is advanced. It is just reading the directory your table actually lives in.