Iceberg v3 swaps the delete-file pile for one vector per data file

Iceberg format version 3 replaces position delete files with one deletion vector per data file and gives every row an id, which changes what upsert-heavy tables cost to read.

CDC stream Merge writer Data files Deletion vectors Snapshot Query reader Incremental consumer one per data file with row ids applies the vectors rows changed since a snapshot
A CDC stream is merged into data files with one deletion vector each; the snapshot serves a query reader and a changed-rows consumer.
On this page
  1. What a v2 merge leaves on disk
  2. One bitmap, and it replaces the last one
  3. Rows that remember when they last changed
  4. Eleven thousand delete files by evening
  5. Who opens the table, then the upgrade
  6. Counting delete files before and after

Format version 2 of Iceberg let you change one row without rewriting its file, and it billed you for that in delete files.

Every merge-on-read commit drops a few more of them next to the data. Readers have to find the ones that apply and merge them in at scan time. Compaction has to come around later and fold them back into the data. On a table fed by change data capture, where nearly every write is an update, that pile is most of what you operate.

Version 3 changes the accounting. Iceberg 1.10.0 shipped in September with both v3 features working in Spark, which is enough runway for me to try it on one table people depend on. My advice: move one CDC-fed table first, and before you touch it, list every engine that reads or writes it.

What a v2 merge leaves on disk

A position delete file is a small file, usually Parquet, holding pairs: a data file path and a row position in it. A merge job that commits every few minutes writes new ones on each commit, in each partition it touched. Nothing caps how many of them can point at the same data file.

So a reader planning a scan works out which delete files might apply to each data file, opens them, and filters rows as it reads. Point lookups feel it most, because opening dozens of delete files costs far more than fetching one row. Compaction resets the count and then it climbs again. A managed table service will run that compaction for you. The pile between runs is still yours.

One bitmap, and it replaces the last one

A v3 deletion vector is a bitmap of deleted row positions for exactly one data file, stored as a blob inside a Puffin file. The spec allows at most one per data file in a snapshot. When a writer deletes more rows from a file that already has a vector, it writes a new vector holding the old positions plus the new ones, and the old vector drops out of the snapshot.

That puts a ceiling on read cost. However many merges have hit a data file, the reader loads one bitmap for it.

Two things are easy to miss. Upgrading doesn’t clean anything: v3 writers can’t add position delete files, but the ones already in the table stay valid until a writer touches that data file or compaction rewrites it. And equality deletes survive in v3. A streaming sink that upserts by key without reading the table first still writes them, and vectors do nothing for that sink. The gain goes to writers that locate the row before deleting it, which is what a Spark MERGE INTO in merge-on-read mode does.

Rows that remember when they last changed

The other v3 feature I care about for upserts is row lineage. Every row gets two metadata columns: _row_id, which stays with the row through updates and compaction, and _last_updated_sequence_number, the sequence number of the commit that last wrote it.

A downstream job that wants the rows changed since its last run can now filter on one column and skip comparing two copies of the table.

-- 48211 is the highest sequence number the previous run saw
SELECT _row_id, _last_updated_sequence_number,
       payment_id, status, amount_cents, updated_at
FROM lake.payments.transactions_v3
WHERE _last_updated_sequence_number > 48211;

It has limits. A hard-deleted row is simply gone, so this query never reports it; I’d keep the soft-delete flag that Debezium-fed tables usually carry anyway. And the ids are only as good as the writer. An equality-delete upsert never saw the old row, so the replacement gets a fresh id.

Eleven thousand delete files by evening

Take Bellhaven Credit, a lender I’ve made up. Its payments table holds 600 million rows and takes 2.5 million upserts a day from a Debezium stream, merged by a Spark job every few minutes.

On v2 the table carries about 11,000 position delete files between nightly compactions. A lookup by payment id late in the day merges 40 of them to return one row. The nightly job that feeds the fraud model finds changed rows by diffing today’s 600 million against yesterday’s.

On a v3 copy, the same partition has 212 data files and at most 212 vectors, and the lookup loads one. The nightly job filters on the sequence number and gets back only the rows that moved, 2.5 million at most. How much it scans to find them depends on the engine, so time it on the copy before promising anyone a number.

Who opens the table, then the upgrade

Start with the matrix. Every row is a thing that opens the table.

Reader or writer Role Confirm on the test copy
Merge job (Spark, Iceberg 1.10.0 runtime) Writer Writes vectors, keeps _row_id across a MERGE
Compaction job Writer Rewritten rows keep their ids and sequence numbers
Interactive SQL engine Reader Opens a v3 table and returns the same counts as v2
Changed-rows job Reader Can select both lineage columns
Exports that read Parquet from the bucket Reader Stop them; they ignore deletes on any version

A reader that doesn’t know v3 should refuse the table outright. That’s the good failure. Plenty of engines and managed services still stop at v2 this month, so check the version you run, not the vendor’s roadmap.

-- 1. Test copy on v3, same partitioning as the original
CREATE TABLE lake.payments.transactions_v3
USING iceberg
PARTITIONED BY (days(created_at))
TBLPROPERTIES (
  'format-version'    = '3',
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode'  = 'merge-on-read'
)
AS SELECT * FROM lake.payments.transactions;

-- 2. In place, only after the copy has run clean for a week
ALTER TABLE lake.payments.transactions
SET TBLPROPERTIES ('format-version' = '3');
[ ] Matrix filled in, with an owner per row
[ ] v3 copy created; merge job writing to both tables
[ ] Row counts and a sum of amount_cents match across both, daily
[ ] Delete-file query saved before and after (below)
[ ] Point lookup and changed-rows job timed on both
[ ] Every reader in the matrix has opened the copy
[ ] Rollback: there is no downgrade from 3 to 2, and rolling back
    to an older snapshot does not undo it. The way back is the v2
    table, so keep it and its merge job until the cutover has held.

Counting delete files before and after

The files metadata table lists delete files next to data files, and a deletion vector shows up as a position delete whose format is Puffin.

SELECT
  `partition`,
  sum(CASE WHEN content = 0 THEN 1 ELSE 0 END) AS data_files,
  sum(CASE WHEN content = 1 AND file_format <> 'PUFFIN' THEN 1 ELSE 0 END) AS position_delete_files,
  sum(CASE WHEN content = 1 AND file_format = 'PUFFIN' THEN 1 ELSE 0 END) AS deletion_vectors,
  sum(CASE WHEN content = 2 THEN 1 ELSE 0 END) AS equality_delete_files
FROM lake.payments.transactions.files
GROUP BY `partition`
ORDER BY position_delete_files DESC;

Run it on the v2 table just before compaction, when the pile is tallest, and on the v3 copy at the same hour. On v3, deletion_vectors should never exceed data_files in any partition. If equality_delete_files isn’t zero, some writer is upserting without reading first, and that writer is the next thing to look at.