Committing one row to an Iceberg table writes four files. There’s the Parquet data file, a manifest that lists it, a manifest list that points at the manifest, and a new table metadata JSON that points at the manifest list. Then comes a call to the catalog to swap one pointer, because the format was designed for object stores that couldn’t promise that two writers wouldn’t both win.
Every piece of that is there for a good reason. Put together, they’re a database transaction log rebuilt out of immutable files.
DuckLake, which the DuckDB team published as a specification in May, starts from the other end. If you need a database to make commits safe anyway, put all the metadata in it. I’ve spent some time reading the spec and poking at the tables it creates. This is an early look at a format that is well short of 1.0, so these are notes on a design. I’m not telling anyone to migrate.
What the file-based formats rebuilt
Iceberg’s metadata tree solved a real problem. A directory listing is slow and tells you nothing about which files belong to which version of a table, so the format writes its own listing, version by version, and never edits a file in place.
The one thing files couldn’t do was the atomic swap. That job went to a catalog, and since mid-2024 the catalog most engines agree on is the Iceberg REST Catalog. Behind most REST catalogs sits an ordinary SQL database holding one pointer per table.
So a production Iceberg setup already runs a transactional database. It just uses it for almost nothing.
What a single append touches
DuckLake keeps Parquet files in object storage exactly as before and moves everything else into tables: snapshots, schemas, file lists, column statistics. In DuckDB it shows up as an extension you ATTACH, with Postgres as one of the supported metadata databases.
| Step | Iceberg with a REST catalog | DuckLake with Postgres |
|---|---|---|
| Data files written | 1 Parquet file | 1 Parquet file |
| Metadata files written | At least 3: manifest, manifest list, table metadata JSON | None |
| Round trips to commit | 3 or more object PUTs, then a commit request to the catalog | One SQL transaction that inserts a handful of rows |
| Conflict check | Optimistic. The catalog compares the snapshot you started from with the current one | The database’s own transaction, plus a per-snapshot change record to compare against |
| What a retry redoes | Refresh metadata, rewrite the manifest list and metadata JSON, try again | Rerun the inserts |
| Planning a query | Fetch and parse metadata JSON, manifest list, then manifests | One SQL query against the metadata tables |
The last row is the one I’d watch. On a table with many small commits, the cost of the file-based design shows up at read time, when the planner has to walk a manifest list that can grow by an entry per commit until something merges the manifests.
Northfold’s 5,000 appends an hour
To put numbers on that I’ll use an invented company. Northfold Instruments is a sensor startup that doesn’t exist, and its figures are mine. Its ingestion service flushes small batches to a readings table about 5,000 times an hour.
On a file-based table, that’s 5,000 snapshots an hour and something like 15,000 new metadata objects alongside the data files. Planning gets slower through the day as manifests pile up, then snaps back when the nightly job rewrites manifests and expires snapshots. The team’s dashboards have a visible sawtooth. I wrote about that maintenance work in a managed table service will compact a badly partitioned table forever, and handing it to a service helps, but the sawtooth is built into the design.
In Northfold’s trial with metadata in Postgres, each append becomes one row in the snapshot table, one in the file table and a few rows of statistics. Planning is one query that asks the database for the live files matching a filter, so it no longer cares how many snapshots the table has collected. It still cares how many files are live. The data files are still small and still need merging. What goes away is the second pile of small files that described the first.
Then someone on the team drops the metadata database in the test environment while cleaning up. Every table in that lake is gone. The Parquet files are all still in the bucket, but nothing says which ones belong to which table or which were logically deleted. With Iceberg, the metadata sits next to the data and a lost catalog can usually be rebuilt by re-registering each table’s latest metadata file. Here the database holds the only copy of that information.
Reading the catalog with plain SQL
What I like most is that the catalog stops being opaque. These queries run directly in Postgres against the metadata schema. Table and column names follow the spec as published; it’s pre-1.0, so check them against the version you’re running.
-- Recent snapshots and what each one changed
SELECT s.snapshot_id, s.snapshot_time, s.schema_version, c.changes_made
FROM ducklake_snapshot AS s
LEFT JOIN ducklake_snapshot_changes AS c USING (snapshot_id)
ORDER BY s.snapshot_id DESC
LIMIT 20;
-- The files that make up one table right now
SELECT f.data_file_id, f.path, f.record_count, f.file_size_bytes, f.begin_snapshot
FROM ducklake_data_file AS f
JOIN ducklake_table AS t ON t.table_id = f.table_id
WHERE t.table_name = 'readings'
AND t.end_snapshot IS NULL
AND f.end_snapshot IS NULL
ORDER BY f.data_file_id;
-- The same file list as of an older snapshot (time travel by hand)
SELECT f.data_file_id, f.path, f.record_count
FROM ducklake_data_file AS f
JOIN ducklake_table AS t ON t.table_id = f.table_id
WHERE t.table_name = 'readings'
AND t.begin_snapshot <= 41250
AND (t.end_snapshot IS NULL OR t.end_snapshot > 41250)
AND f.begin_snapshot <= 41250
AND (f.end_snapshot IS NULL OR f.end_snapshot > 41250)
ORDER BY f.data_file_id;
-- Small-file pressure per table (row counts are before delete files apply)
SELECT t.table_name,
count(*) AS live_files,
sum(f.record_count) AS rows_in_files,
pg_size_pretty(sum(f.file_size_bytes)) AS live_size,
count(*) FILTER (WHERE f.file_size_bytes < 8 * 1024 * 1024) AS files_under_8mb
FROM ducklake_data_file AS f
JOIN ducklake_table AS t ON t.table_id = f.table_id
WHERE f.end_snapshot IS NULL
AND t.end_snapshot IS NULL
GROUP BY t.table_name
ORDER BY live_files DESC;
Treat these as read-only. Writing to the tables by hand is how you’d corrupt a lake in one statement.
The database you now have to run
The price of all this is plain. The metadata database is the single point of truth, and it needs what any system of record needs: point-in-time recovery and a restore you’ve actually rehearsed. Keep the list of roles that can write to it short. Bucket versioning won’t save you, because the bucket never held the metadata.
Engine support is the other cost. Almost everything reads Iceberg. As of this month, DuckLake is read and written by DuckDB, and that’s about it.
So I wouldn’t put production tables on it yet. I would run a trial on a workload shaped like Northfold’s, many small commits where metadata churn is the real pain, with a metadata backup in place before the first write. And I’d restore from that backup once, on purpose, before trusting any planning-time number the trial produces.