Batch, micro-batch or streaming: let a latency budget pick

Write down how stale each consumer's answer can be, split that budget across capture, transport, transform and serve, and the pipeline mode becomes mostly arithmetic.

Decision Freshness budget Stage split Mode picker Nightly batch Micro-batch job Stream job Serving table per consumer capture, transport transform, serve cut-off rules budget in hours budget in minutes budget in seconds
A freshness budget, split across four stages, picks a nightly batch, a micro-batch job or a stream job on the way to one serving table.
On this page
  1. Four stages spend one budget
  2. Three requests at a grocery chain
  3. Where the cut-offs fall
  4. The 3 a.m. question

On a request form, a card-fraud hold and a morning revenue report can both say “real time”, and they’re more than four orders of magnitude apart.

The form never asks for the number, so I do. How old can this answer be before whoever uses it makes a worse decision?

I treat the reply as a latency budget. A pipeline spends it in four places, and once the four amounts are written down, choosing between batch, micro-batch and streaming is mostly arithmetic, and the argument about tools gets short.

Four stages spend one budget

Freshness is measured at the consumer: from the moment something happens in the source system to the moment a dashboard or an alert reflects it. The time in between goes to four stages.

Capture gets the change out of the source. A nightly extract can spend 24 hours here. Polling a modified-timestamp column every minute spends a minute. Log-based change data capture, with something like Debezium reading the database’s transaction log, spends a second or two.

Transport moves the change to where the work happens. A scheduled file drop is slow. A Kafka topic takes well under a second.

Transform is the joins and aggregates. For a scheduled job, count the worst case: a record that just missed a run waits a full interval, then waits for the next run to finish.

Serve is the last hop, and it’s easy to leave off the list. A BI extract that refreshes hourly spends an hour, however fresh the table behind it is.

The four amounts have to add up to no more than the consumer can tolerate, and the slowest stage sets the floor. A nightly extract has spent the whole day before transport begins. No stream processor downstream can earn it back.

Three requests at a grocery chain

Picture a grocery chain, Saltmarsh, that exists only to give this worksheet three rows. Its data team has three requests in the queue, each labeled real time. The worksheet gets one row per consumer.

Consumer Decision it drives Tolerates A stale answer costs Capture Transport Transform Serve Mode
Finance revenue dashboard Agenda for the morning review 33 h Nothing before the review 24 h 30 min 2 h 30 min Nightly batch
Stock-out alert Refill the shelf 15 min Lost sales 1 min 1 min 10 min 3 min Micro-batch
Card fraud hold Hold the card 5 s Another fraudulent purchase 1 s 1 s 2 s 1 s Stream

I fill in the left half with the person asking, and the cost column does most of the work. If nobody can say what a stale answer costs, the tolerance is a preference, and a preference gets the cheapest mode.

The revenue dashboard feeds a 9 a.m. review of yesterday’s sales. The oldest sale on it rang up just after midnight the day before, 33 hours earlier, and that’s what finance asked for. It stays nightly. The extract makes a sale wait up to 24 hours, which is the same interval-plus-run-time math as any scheduled job, and the other three stages take three more. I’d leave the last six alone. That’s the room to rerun a failed load before anyone opens the dashboard.

The stock-out alert sends a person to a shelf, and the walk takes minutes. An alert that’s five seconds fresh buys nothing over one that’s ten minutes fresh. So the budget is 15 minutes, with 10 of them for transform. That slot fits a micro-batch, a small batch job on a tight schedule: it starts every eight minutes and finishes inside two, so a sale that just misses a run still shows up within ten.

Spark Structured Streaming does this with an available-now trigger and a scheduler, so each run processes whatever has arrived and shuts down. Snowflake’s Dynamic Tables, still in public preview as I write, do it declaratively: you set a target lag and the warehouse schedules the refreshes. Either way, compute runs for a couple of minutes at a time instead of around the clock, and a failed run is repaired by the next one.

Only the fraud hold needs a stream processor. It has to stop a stolen card before the next purchase, and five seconds leaves no room for a schedule. The checkout service publishes each authorization to Kafka. A Flink job keeps a running count per card in keyed state and writes the hold to a table the payment path reads.

Where the cut-offs fall

These rules go under the worksheet. The numbers are mine, and a team should argue about them once and write down its own.

  • A budget longer than a day is a nightly batch. The extract interval counts against it, and the unspent hours are rerun room.
  • Between about five minutes and a day, it’s a micro-batch. Interval plus run time has to fit inside the transform amount. Hourly is the same job on a slower schedule.
  • Under a minute, it’s a stream processor: a job that never stops, with state to checkpoint and upgrades to plan around.
  • Between one and five minutes, price both. I lean toward a micro-batch on a short trigger unless a stale answer costs real money.
  • Before changing the mode, fix the slowest stage. Replacing a nightly extract with change data capture often does more than a new engine would.

My bet is that most requests labeled real time land on the second rule. They’re fifteen-minute budgets, and a micro-batch meets them at a fraction of what an always-on stream job costs to run and to watch.

All three modes write to the same serving tables, so a consumer never needs to know which one fed it. When a budget tightens, change that row’s mode and leave the others alone.

The 3 a.m. question

The last line on the worksheet asks who acts on this at 3 a.m.

For the revenue dashboard, nobody. For the stock-out alert, nobody either, because Saltmarsh’s stores are closed, so that job can pause overnight and a failure can wait for the morning shift. For the fraud hold, the answer is the payment system, since the website takes orders all night.

That makes the fraud row the only one where falling behind at 3 a.m. does damage before anyone’s awake to notice, and that exposure, more than the cluster, is what a streaming row costs. So the fraud row gets two extra columns on the worksheet, consumer lag at which to alert and what the payment path does when the hold table is stale, and both are filled in before the Flink job is written.