The topic has an owner. Who owns the table it turned into?

Landing a Kafka topic as an Iceberg table is one setting now, and it skips the commit interval, compaction and schema-change decisions that someone has to own.

Schema registry Producer Topic Materializer Lake table Compaction job Query engines schema rules managed or a sink job one owner on a schedule
A producer's topic and its registered schema meet at the materializer, then run through the lake table and compaction to the query engines.
On this page
  1. One switch, three decisions left open
  2. Brackwater Couriers and the ten-second commit
  3. The card I’d attach to every landed table
  4. Checking the table against its card
  5. Schema changes come from the other side

The switch that turns a Kafka topic into an Iceberg table has no field for who gets paged when the table gets slow.

It’s a good feature with one empty field, and that field is the one that decides who fixes the table. Confluent’s Tableflow went generally available for Iceberg in March. If you’d rather run it yourself, a Flink job with the Iceberg sink does the same work, and Flink 2.0 and Kafka 4.0 both shipped that month too. Either way, “streaming lakehouse” stopped being a project and became a setting.

A setting has defaults, and three of them get picked by people who will never query the table.

One switch, three decisions left open

The first is how often to commit. Every commit to an Iceberg table writes a new snapshot and at least one new data file per writer per partition touched. Commit often and the table is fresh and full of tiny files. Commit rarely and the files are healthy and the dashboard is behind. The Flink sink commits on every checkpoint, so whoever picked the checkpoint interval for recovery reasons also picked your file count without knowing it.

The second is who compacts. Small files have to be rewritten into big ones, old snapshots expired, orphans removed. A managed materializer may do some of this for you. “Some” needs reading closely: to what target size, and does it cover delete files too?

The third is what happens when the producer changes the schema. The producing team checks a change against the schema registry’s compatibility mode, and that check is about other Kafka consumers. Nothing in it asks whether a new required field or a widened type makes sense as a column that three query engines read.

The stream team owns the topic and considers its job done once events are flowing. The analytics team sees a table appear in the catalog and assumes it’s maintained like every other table there. Both assumptions are reasonable, and between them the table has no owner.

Brackwater Couriers and the ten-second commit

Brackwater Couriers doesn’t exist. I made it up so the arithmetic has somewhere to live, and what follows is a calculation. I haven’t watched this particular table fall over.

Brackwater lands its parcel-scan topic as a table, partitioned by hour. The topic carries 3,000 events a second across six partitions, and the materializer runs six writers that each close a file on every commit. The commit interval is 10 seconds, because someone wanted the operations dashboard to feel live.

Ten seconds is 8,640 commits a day. Six files each makes 51,840 files a day. If a scan event compresses to roughly 40 bytes of Parquet, each file holds 5,000 events and weighs about 200 KB. Planning a query over one week means reading manifest entries for more than 360,000 files, and within that week the dashboards that wanted to feel live are the slow ones.

The fix is boring. Move the commit to 2 minutes: 720 commits and 4,320 files a day, each around 2.4 MB. Add an hourly compaction that rewrites the previous hour’s partition toward a 128 MB target. A day of scans is about 10 GB, or 430 MB an hour, so each hour compacts to four or five files and a finished day settles near a hundred. It used to be 51,840.

The price is that the newest scan can be two minutes old. The dashboard tiles refreshed every five minutes the whole time. Nobody noticed, because nobody had ever written down how fresh the table needed to be.

The card I’d attach to every landed table

So write it down. One card per table, kept next to the table definition, with a named team on the first line. I’d name the owner of the table, and I mean the table and not the topic: the people who feel it when queries slow down.

table: lake.ops.parcel_scans
source_topic: parcel-scans.v1
owner: ops-analytics            # pager and budget for this table
materializer: managed topic-to-table (or the Flink sink job, name it)
commit_interval: 2m
freshness_target: 5 minutes at p95, scan time to queryable
schema_change_policy:
  registry_compatibility: BACKWARD
  auto_apply: new optional fields, as nullable columns
  needs_owner_review: type changes, renames, removed fields
  breaking_change: new topic version and a new table
# compaction and retention: see the table policy file

The compaction and retention half of this belongs in the same table policy file any lake table should have; what’s new here is the commit interval and the producer’s schema.

freshness_target is the line that lets you choose a commit interval at all, so fill it in first. Leave it blank and the interval gets set by whoever is most impatient, and the file count follows.

Checking the table against its card

The card says two minutes, and the compaction job says 128 MB. The table will tell you whether either is true. The per-partition file query from the Iceberg chores post works here once you swap the table name. Alert when a partition older than two hours averages under a quarter of the 128 MB compaction target. That one alert catches a stalled compaction job and a commit interval someone quietly shortened.

To see how often the materializer really commits, count snapshots. Iceberg exposes its own metadata as queryable tables, so this is plain Spark SQL:

SELECT
  date_trunc('day', committed_at) AS day,
  count(*)                        AS commits
FROM lake.ops.parcel_scans.snapshots
GROUP BY date_trunc('day', committed_at)
ORDER BY day DESC;

In Trino the same table is "parcel_scans$snapshots". If the card says two minutes and this shows 8,640 a day, someone changed a checkpoint setting for their own reasons, and now you know which conversation to have.

Schema changes come from the other side

The card’s schema section is the only part the table owner can’t enforce alone, because the change starts in the producer’s repository. What the owner can do is get the policy into the producer’s review path: the registry rejects incompatible changes, additive optional fields flow through untouched, and anything in the review list needs a yes from the name on the card before it merges.

That last step is where the owner field pays for itself. A producer engineer about to rename a field needs one name to ask. If the card says “data platform” or, worse, nothing, they’ll ask no one, and the first notice anyone gets is a column full of nulls.