activate:order-88213:line-2 is an ugly string, and I’d take it over a UUID every time.
The UUID looks more professional. It’s also the reason a customer can end up with two subscriptions. A key generated when the request is built describes the attempt, and a retry is by definition a new attempt, so it gets a new key and sails past every duplicate check downstream. The ugly string describes the thing that is supposed to happen exactly once. Retry it a hundred times and it’s still the same string.
What a duplicate looks like at Spindrift Mobile
Spindrift Mobile is a company I made up for this post. Its order system publishes a platform event when an order is ready to activate, and a worker picks that up and does two writes: create the subscription in billing, and create the asset in the CRM.
One afternoon the billing call for order 88213, line 2, times out at 30 seconds. Billing actually finished the work at second 31. The worker doesn’t know that, so it retries, and because it stamps each request with a new UUID, billing sees a brand new request and creates a second subscription. The CRM write is an upsert on the order line, so it lands on the same asset both times.
Two subscriptions, one asset. The customer gets billed twice and the account team sees nothing wrong in the CRM. The billing API’s idempotency feature was switched on the whole time. It was protecting each attempt from itself.
Build the key from the fact
A key protects only the write it’s scoped to. So the first design question is what the unit of “once” is, in business terms, and the answer is rarely “this HTTP request.”
| Write | Key format | Example | Expires when |
|---|---|---|---|
| Activate an order line | activate:order-{order}:line-{line} |
activate:order-88213:line-2 |
The order closes |
| Apply an amendment | amend:sub-{sub}:amendment-{n} |
amend:sub-4410:amendment-7 |
The contract term ends |
| Invoice a billing period | invoice:sub-{sub}:{period} |
invoice:sub-4410:2025-09 |
The period’s books close |
| Issue a credit | credit:invoice-{invoice}:case-{case} |
credit:invoice-9031:case-552 |
The case closes |
The key builder is a small function in the worker, and it follows two rules. Every part of the key has to come from data that exists before the first attempt, so nothing like a timestamp or a worker id sneaks in. And the verb goes in the key, because activating line 2 and amending line 2 are different facts about the same line.
Getting the scope wrong in the other direction hurts too. Scope the invoice key to the subscription alone and October’s invoice gets swallowed as a duplicate of September’s.
One table, one unique constraint
The key store is a Postgres table. The unique constraint does the real work; everything else is bookkeeping.
CREATE TABLE idempotency_keys (
key text NOT NULL,
write_type text NOT NULL,
status text NOT NULL DEFAULT 'in_progress'
CHECK (status IN ('in_progress', 'succeeded', 'failed')),
outcome jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz, -- stays null until the business object closes
CONSTRAINT idempotency_keys_key_uq UNIQUE (key)
);
CREATE INDEX idempotency_keys_expiry_ix
ON idempotency_keys (expires_at)
WHERE expires_at IS NOT NULL;
The worker’s first move on any event is to claim the key. This function inserts it or, if it’s already there, hands back whatever was stored.
CREATE FUNCTION claim_key(p_key text, p_write_type text)
RETURNS TABLE (claimed boolean, key_status text, stored_outcome jsonb)
LANGUAGE plpgsql AS $$
BEGIN
RETURN QUERY
INSERT INTO idempotency_keys AS k (key, write_type)
VALUES (p_key, p_write_type)
ON CONFLICT (key) DO NOTHING
RETURNING true, k.status, k.outcome;
IF NOT FOUND THEN
RETURN QUERY
SELECT false, k.status, k.outcome
FROM idempotency_keys k
WHERE k.key = p_key
FOR UPDATE;
END IF;
END;
$$;
Commit the claim on its own before touching billing or the CRM. Held open across network calls, it vanishes if the worker crashes.
claimed = true means go do the work. claimed = false with succeeded means return the stored outcome and stop. claimed = false with in_progress means an earlier attempt started and never reported back, which is exactly the Spindrift timeout. failed is for a definite rejection, such as billing refusing the plan code, and it waits for a person. If two workers race on the same key, the second insert waits for the first to commit and then falls through to the select.
When the work finishes, the outcome recorder writes the result (the billing subscription id, the CRM asset id) into the row, in the same transaction that flips the order line to active in your own database. I’m strict about that. If the status update and the stored outcome can commit separately, you’ll eventually have a line marked active with no record of what was created for it, or the reverse.
The same key goes downstream
A local transaction can’t reach into billing or the CRM, and that’s fine, because the key can.
Billing gets it as the request’s idempotency key. On a retry after a timeout, the worker sends activate:order-88213:line-2 again and billing answers with the subscription it already made.
The CRM gets it as an external id. Put a unique external id field on the asset, upsert by that field, and the second attempt updates the record the first one created. The upsert was already safe in the Spindrift story. Now it’s safe on the same string as the billing write, and you can search three systems for that string when something looks off.
So the in_progress case has an easy answer. Resume, with the same key, and let both downstream systems deduplicate. The outcome recorder then stores what they returned. If that activation also grants product access, the grant row can carry the key too, which fits the entitlements table I described earlier this year.
Expire on the business calendar
Many billing APIs forget a request key after about a day. That window suits a client retrying a dropped connection.
Orders don’t work on that clock. A platform event can be replayed days later after an outage. Someone can re-run a stuck order from an admin screen the following week. If the key is gone by then, the retry is a first attempt again.
That’s why expires_at starts out null. When order 88213 closes, a job sets it on every key for that order, and a sweeper deletes rows past their date. Invoice keys expire when the period’s books close. The vendor’s own window mostly stops being a problem, because for a finished write your store answers the duplicate before the request ever leaves. One gap is left. A key that has sat in_progress for longer than the vendor remembers can’t be resent blind, so look the subscription up in billing first.
The report I’d run every morning
The key table doubles as a reconciliation report. Two queries cover most of it: keys still in_progress after fifteen minutes, and succeeded keys whose outcome is missing either the billing id or the CRM id. Each row names a business fact, so whoever picks it up knows which order line to look at without decoding a request log.
At Spindrift, order 88213 would have been one in_progress line on that report, resumed under the same key, with one subscription behind it.