Order activation is five calls into other systems: provision the server, raise the invoice, update the CRM, grant the entitlement, send the welcome email. Each call is easy to write. The trouble sits between them, because a worker that dies after the second call comes back with no idea it ever made the first.
Durable execution is the name for fixing that, a process that survives a crash and carries on from where it stopped. Dedicated engines do it well, and for a three-week onboarding I’d use one, as I argued last year. For a process that finishes in minutes, I’d start with two tables in the Postgres I already run. This is the build log for that version, including the point where I’d stop trusting it.
Two tables, and the second one is the memory
create table processes (
id bigint generated always as identity primary key,
kind text not null,
business_key text not null,
input jsonb not null,
status text not null default 'pending'
check (status in ('pending', 'done', 'failed')),
attempts int not null default 0,
leased_by text,
lease_until timestamptz,
created_at timestamptz not null default now(),
progressed_at timestamptz not null default now(),
unique (kind, business_key)
);
create table process_steps (
process_id bigint not null references processes (id),
name text not null,
result jsonb not null,
finished_at timestamptz not null default now(),
primary key (process_id, name)
);
processes holds one row per order being activated. process_steps holds one row per step that finished, along with what it returned. A process doesn’t have to remember where it was in the code. It reruns from the top and skips every step that already has a row, which lands it in the same place.
The trigger is the order-closed platform event from Salesforce. A Pub/Sub API subscriber reads it and does one insert:
insert into processes (kind, business_key, input)
values ('order_activation', $1, $2)
on conflict (kind, business_key) do nothing;
A subscriber that restarts replays from its last stored replay ID, so the same event will show up twice sooner or later. With the order ID as the business key, the second copy does nothing.
One worker per process, held by a lease
update processes
set leased_by = $1,
lease_until = now() + interval '5 minutes',
attempts = attempts + 1
where id = (
select id
from processes
where status = 'pending'
and (lease_until is null or lease_until < now())
order by created_at
limit 1
for update skip locked
)
returning id, input;
skip locked lets ten workers run this at once without queueing behind each other. The lease is a timestamp and not a held row lock, because a held lock means keeping a transaction open across HTTP calls to a provisioning API, and I don’t want that. A worker that dies stops renewing. Five minutes later the row is up for grabs again.
Look up, call, store
The step runner is one function.
import psycopg
from psycopg.types.json import Jsonb
conn = psycopg.connect(DSN, autocommit=True) # DSN: your connection string
def run_step(process_id, name, fn):
row = conn.execute(
"select result from process_steps"
" where process_id = %s and name = %s",
(process_id, name),
).fetchone()
if row:
return row[0]
result = fn(f"{process_id}:{name}") # same key on every attempt
with conn.transaction():
conn.execute(
"insert into process_steps (process_id, name, result)"
" values (%s, %s, %s)"
" on conflict (process_id, name) do nothing",
(process_id, name, Jsonb(result)),
)
conn.execute(
"update processes set progressed_at = now(),"
" lease_until = now() + interval '5 minutes'"
" where id = %s",
(process_id,),
)
return result
A crash between the call and the store means the step runs again next time, and I can’t close that window from my side, since the billing system and my database don’t share a transaction. So fn receives a key built from the process ID and the step name, identical on every attempt, and the far side has to honor it. I’ve written about building those keys from the business fact. For billing it’s a header. For the CRM it’s an upsert on an external ID. For the entitlement it’s a unique constraint on the grants table. Email is the weak one. If the provider takes no key, I accept a rare duplicate welcome message and move on.
The same key covers the other gap, a step that outlives its lease and gets picked up by a second worker.
The activation code then reads like the naive script it replaces. The client objects stand in for whatever SDKs you use:
def activate(process_id, order):
server = run_step(process_id, "provision", lambda key:
provisioning.create_server(order["plan"], idempotency_key=key))
invoice = run_step(process_id, "invoice", lambda key:
billing.create_invoice(order["account_id"], order["amount"],
idempotency_key=key))
run_step(process_id, "crm_update", lambda key:
crm.mark_active(order["order_id"], server["id"], invoice["id"]))
run_step(process_id, "entitlement", lambda key:
grants.add(order["account_id"], order["plan"], source_key=key))
run_step(process_id, "email", lambda key:
mailer.send_welcome(order["contact_email"], idempotency_key=key))
conn.execute(
"update processes set status = 'done', lease_until = null"
" where id = %s",
(process_id,),
)
One rule keeps this honest. Anything the code decides between steps has to come from the input or from a stored result, never from the clock or a fresh read of some other system, or a rerun can take a different branch than the first run did.
Thirty-seven half-activated orders, replayed
Quarrywood Hosting is a company I invented to test the design against. Its old activation job was a plain script, and one night of billing timeouts left 37 orders half-activated: servers running, no invoices, CRM still showing the orders as pending. Someone spent the morning working out which was which.
Now run an order through the tables, and make the failure a crash at the worst moment. An order closes and provision succeeds, storing {"id": "srv-8841"}. The invoice succeeds too, and the worker is killed before it stores the result. Within five minutes the lease expires and a new worker takes the row. provision finds its row and returns srv-8841 without calling anything. invoice runs again under the same key, 412:invoice for process 412, so billing returns the invoice it already made. The customer ends up with one server and one invoice. Quarrywood’s billing timeouts end the same way, and the idempotency post walks through that case.
Processes that have gone quiet
Retrying forever hides a broken step. I cap attempts in the worker loop and flip the row to failed past the cap, so it waits for a person. Before it gets that far, I want this query on a dashboard:
select p.id, p.business_key, p.attempts, p.progressed_at,
(select s.name
from process_steps s
where s.process_id = p.id
order by s.finished_at desc
limit 1) as last_finished_step
from processes p
where p.status = 'pending'
and p.progressed_at < now() - interval '15 minutes'
order by p.progressed_at;
The last finished step tells you which call is failing: the one after it.
Timers, versions and fan-out
All of the above comes to about 150 lines once the worker loop is in, and for a process that finishes in minutes I think it’s enough. I’d start looking at an engine when one of these shows up.
Long timers at volume. “Wait 14 days, then check for payment” can be a wake_at column, and with a few thousand sleepers it works fine. With millions, you’re building a timer service.
Runs that outlive a deploy. For five-minute processes I drain before deploying, but runs that live for weeks need versioned definitions.
Wide fan-out. One process that spawns 500 children and waits for all of them needs parent and child bookkeeping this schema doesn’t have.
Order activation has none of those. It has five calls, and each one now knows whether it already ran.