The first compaction pass on a managed table is fun to watch. A table that arrived with over a million small files comes back with a few thousand, and nobody on the team wrote a Spark job or got paged for it.
Then someone reruns the slow dashboard, and it’s still slow.
Amazon S3 Tables went generally available in December, so this is the first month a team could plausibly have one in production. It’s a new kind of bucket that stores Apache Iceberg tables and runs the upkeep for you. I think it’s a good trade for most teams. It’s also easy to misread what was traded.
The chores the bucket does for you
Three of the chores from the Iceberg housekeeping post are the ones that matter here: compaction, snapshot expiry and orphan cleanup. Each is a procedure Iceberg already ships. What it doesn’t ship is the scheduler, the compute, the retries and the person who notices when a job has been failing quietly for three weeks.
A table bucket takes all three. Compaction runs in the background and merges small files toward a target size. Snapshots expire by the age and count you configure. Unreferenced files get removed after a waiting period. You pay for it per object and per byte processed instead of in on-call hours.
Engines reach the tables through a catalog. Today that’s a client catalog library for Spark, or the Glue Data Catalog integration, still in preview, which is how Athena, Redshift and EMR see the tables. Writers and readers go through that one door, and that’s also where permissions get checked.
Jobs move, decisions don’t
Every one of those jobs is the execution of a decision someone made earlier. Compaction rewrites files inside the partitions you chose. Expiry removes snapshots older than the age you set. The service is very good at carrying the decision out, and it has no opinion about whether the decision was right.
Here’s the split as I’d write it down for a team making the move.
| Task | Self-managed Iceberg | Managed table bucket | Still yours |
|---|---|---|---|
| Small-file compaction | You schedule and size the rewrite job | Runs automatically | Target file size, and whether compaction is on |
| Snapshot expiry | You schedule the expire procedure | Runs automatically | How many snapshots to keep and for how long |
| Unreferenced-file cleanup | You run orphan removal, carefully | Runs automatically | The waiting period before deletion |
| Partition spec | You choose it | Not touched | All of it |
| Sort order | You choose it; writers apply it | Not chosen for you | All of it, plus checking it survives compaction |
| Row-level deletes | You rewrite delete files away | Handled as part of compaction | Whether a table should take row-level deletes at all |
| Write access | Bucket policy plus catalog grants | Table-level permissions exist | Naming the principals and saying no |
| Schema changes | You run them | You still run them | All of it |
The last column never goes empty. That’s what I’d want everyone to see before the migration is declared done.
Sort order deserves a note. As far as I can tell from the launch documentation, the managed compaction is bin-packing: it combines files and doesn’t promise to re-sort rows. If your scans depend on clustering within a partition, test that it holds after a compaction pass. Don’t assume it.
Pennywhistle Outdoor and the 1.2 million files
To make this concrete I’ve invented a retailer, Pennywhistle Outdoor. The company and the numbers are illustrative.
Pennywhistle has a 400 GB orders table on Iceberg, fed by a nightly batch load and a stream sink that commits every minute. Nobody ever got around to scheduling compaction, so the table carries 1.2 million data files, averaging about 330 KB each. Planning a query takes longer than running it. They move the table into a managed table bucket.
The first compaction does what it promised. File count drops to a few thousand, planning time collapses, and the storage listing looks civilized.
The revenue dashboard barely moves.
The table is partitioned by a bucket transform on order_id. That’s a fine choice for point lookups and for joins on the order key. But nearly every query Pennywhistle runs filters on a date range, and a hash of the order ID tells the engine nothing about dates. A request for the last 30 days still opens every partition and reads its way through most of the 400 GB. Compaction made those reads tidier. It couldn’t make them smaller.
The fix is a partition spec on days of order_date, followed by a rewrite of the existing data so old files land in the new layout. With three years of history that’s roughly 365 MB a day, and a 30-day query now touches about 11 GB. No maintenance service could have made that call, because it depends on knowing how the table is queried. Left alone, the service would have kept the wrong layout in excellent condition for years.
A table policy that fits on a page
So I’d keep the decisions in a file, one per table, in the same repository as the pipeline code. The format below is my own, and it isn’t the IAM-style table policy that S3 Tables attaches to a table. Nothing reads it automatically, though it maps cleanly onto the table’s DDL, its maintenance configuration and its access policy, and a small script can check all three against it.
# table-policy.yaml
table: sales.orders
owner: data-platform
format_version: 2
partition_spec:
- source_column: order_date
transform: day
sort_order:
- column: customer_id
direction: asc
null_order: nulls-first
- column: order_ts
direction: asc
null_order: nulls-first
snapshot_retention:
min_snapshots_to_keep: 24
max_snapshot_age_hours: 168
compaction:
enabled: true
target_file_size_mb: 512
unreferenced_files:
delete_after_days: 3
writer_principals:
- arn:aws:iam::111122223333:role/orders-batch-load
- arn:aws:iam::111122223333:role/orders-stream-sink
query_patterns:
- filter: order_date range
share: most dashboards
- filter: customer_id within a date range
share: support lookups
query_patterns is the only block with no setting behind it. It records why the partition spec is what it is, so the next person who wants to change it has something to argue with.
Retention is the other decision worth a real conversation. Seven days of snapshots means seven days of time travel and seven days of storage for every rewritten file. A table that’s compacted constantly rewrites a lot of files. The rollback window is the same question it was on self-managed Iceberg. The service only guarantees that expiry runs on time.
Two writer principals is also a deliberate number. Any role not on that list has no business committing to the table, and the access policy should say so too.
Review the file when query patterns change or a new writer shows up. The service will keep doing its jobs either way.