Nine days apart in June, Snowflake and Databricks each made a catalog announcement. Snowflake introduced Polaris Catalog on June 3 and said it would open-source it. Databricks said on June 12 that it was open-sourcing Unity Catalog, and the code went up that week as version 0.1. Neither headline was about a table format.
I read that as the format argument winding down. Delta tables can publish Iceberg metadata through UniForm, and in June Databricks also agreed to buy Tabular, the company founded by Iceberg’s creators. What still differs between products is the service in front of the files, so that’s where I’d spend the evaluation time.
A bucket can’t tell you which file is the table
An Iceberg table is a pile of data files and a chain of metadata files. Any writer with bucket access can add one more metadata file. The table is whichever one the catalog points at right now.
Keeping that pointer is the catalog’s oldest job. Hive Metastore and AWS Glue have done it for years: one entry per table, swapped under a lock or a version check that the engine’s own client library runs.
The REST catalog specification, part of Iceberg since release 0.14 in July 2022, puts the catalog behind an HTTP API. A server behind an API can take on two more jobs.
It can hand out storage credentials. The response to a table load can carry short-lived keys that open only that table’s files, so the engine needs no bucket key of its own.
And it can refuse a commit. The engine sends its change along with what it assumed, such as “main is still at snapshot 8271”. That makes the catalog the one place a write can be turned down, whether for a conflict or because the caller isn’t allowed.
Fourteen keys at Pelham Energy
Pelham Energy is a utility I made up so these jobs would have numbers attached. It writes smart-meter readings with Flink and queries them with Trino. An hourly Spark job compacts the recent partitions, one of the chores an Iceberg table leaves to you.
On a Hive Metastore, each engine talks to the bucket with its own long-lived key. Across clusters and schedulers, 14 service accounts hold full access to the lake bucket. The metastore knows which tables exist. It has no say over who opens the files.
Now put the same tables behind a REST catalog that vends credentials. Each load checks the caller’s grants and returns keys that last one hour and reach one table’s prefix, read-only for Trino and read-write for the two writers. The 14 standing keys get deleted. A leaked credential used to be worth the whole lake until somebody rotated it. Now it’s worth an hour of one table.
Commits change too. Compaction and ingest collide 3 to 5 times a day, when both start from the same snapshot and one gets in first. The catalog refuses the second with a 409. That writer reloads the table and commits again about 2 seconds later. Its data files are still good, so only the last step is redone.
The exchange on the wire
This is one ingest commit as a .http file, the format REST client plugins run. Paths and field names are the specification’s. The host and IDs are invented.
### 1. Load the table and ask for storage keys with it.
GET https://catalog.pelham.example/v1/namespaces/metering/tables/readings
Authorization: Bearer {{token}}
X-Iceberg-Access-Delegation: vended-credentials
### 200, trimmed. The config keys last an hour and open only this table's prefix.
# "metadata-location": "s3://pelham-lake/metering/readings/metadata/00412-<uuid>.metadata.json"
# "metadata": { "current-snapshot-id": 8271 }
# "config": { "s3.access-key-id": "<key>", "s3.secret-access-key": "<secret>", "s3.session-token": "<token>" }
### 2. No catalog call: write data files and a manifest list with those keys.
### 3. Commit: make 8272 current, provided 8271 still is.
POST https://catalog.pelham.example/v1/namespaces/metering/tables/readings
Authorization: Bearer {{token}}
Content-Type: application/json
{
"requirements": [
{ "type": "assert-ref-snapshot-id", "ref": "main", "snapshot-id": 8271 }
],
"updates": [
{
"action": "add-snapshot",
"snapshot": {
"snapshot-id": 8272,
"parent-snapshot-id": 8271,
"sequence-number": 413,
"timestamp-ms": 1721024047000,
"manifest-list": "s3://pelham-lake/metering/readings/metadata/snap-8272.avro",
"summary": { "operation": "append" }
}
},
{ "action": "set-snapshot-ref", "ref-name": "main", "type": "branch", "snapshot-id": 8272 }
]
}
### 200: the catalog wrote metadata file 00413 and moved the pointer.
### 409 CommitFailedException: another writer got in first. Reload, then repeat step 3.
Step 2 never touches the catalog. Engines still read and write the bucket directly, so the vended keys are the only link between a grant in the catalog and what an engine can open.
Seven rows per candidate
I’d fill this in once per catalog on the shortlist.
| Row | What to find out | A weak answer |
|---|---|---|
| Commit semantics | Is the pointer swap atomic, and does a commit state the snapshot it expects? | Last write wins, or a lock a crashed writer can leave held |
| Credential vending | Does loading a table return temporary storage keys? How narrow, and for how long? | Every engine brings its own bucket key |
| Access granularity | Are grants per namespace and per table, with read separate from write? | One role for the whole lake |
| Engines that can read | Which engines and versions have you run against it yourself? | “Anything that speaks Iceberg” |
| Engines that can write | The same list, for commits | One engine, usually the vendor’s |
| Where it runs | A vendor’s service, or something you run? Where does it keep its state? | Only inside one vendor’s platform |
| Exporting its state | Can you list every table’s current metadata location and its grants, then register the tables elsewhere? | No bulk listing, or pointers you can’t read |
Commits are where the candidates differ
The two engine rows are separate on purpose. UniForm shows the gap: Iceberg clients can read those tables, and only a Delta writer can commit to them. Open source Unity Catalog has an Iceberg REST endpoint, and in 0.1 it only serves reads, as far as I can tell. Snowflake says Polaris will take reads and writes from any engine that speaks the REST API. The code isn’t public as I write this, so I can’t test that.
Count the writers per table before choosing. A table that only one engine can commit to belongs to that engine, and each of those jobs has to move before the table can.
Export is the row I’d settle before the first production table goes in. A catalog’s state is small: per table, a name and the location of the current metadata file, plus the grants. If I can list that, moving to another catalog is a script. The grants are the part I’d expect to redo by hand.
For the engine rows I wouldn’t take anyone’s word, the vendor’s included. Start two writers from the same snapshot, on the engine versions you run, and wait for the 409.