Salesforce’s event bus holds every high-volume platform event and change event for 72 hours, and it still gets blamed first when an integration loses data.
Each event gets a replay ID that marks its place in the stream, and a Pub/Sub API subscriber can ask for anything still inside the window. Losing data on a bus like that takes help, and the help comes from how the event was defined and what its subscribers assume. I count four such mistakes, and each can be tested against an event definition before anyone writes a subscriber.
To show them I’ve invented a lender, Corvid Lending, and given it a bad week.
Five hours down, 212 kits short
When a loan account goes active, Corvid’s org publishes a platform event, Send_Welcome_Kit__e, with one custom field: the account ID. Three services subscribe. Fulfillment mails the kit, an email service starts onboarding, and a loader records the activation in the warehouse.
One Tuesday a bad deploy took the fulfillment subscriber down for five hours, and 212 accounts went active before it restarted. The other two subscribers handled all 212. Fulfillment handled none and logged no errors, because from where it sat nothing had failed.
That Thursday an analyst lined up activations against kit orders and found the gap.
Corvid’s design has all four mistakes, though only the third lost the kits.
The name was an order
Send_Welcome_Kit__e tells one subscriber what to do. The other two subscribed because it was the only signal that an account had gone active, so both now depend on a fulfillment instruction.
A command also wants an answer. The day the org needs to know whether a kit shipped, someone adds a reply event, and a broadcast channel is carrying a conversation.
The test: the name is a past-tense fact the publisher knows to be true, and the publisher would behave the same with no subscribers. Account_Activated__e passes. If the publisher needs an outcome, that’s a request, and I’d make it an API call.
An ID and nothing else
With only an account ID, each subscriber calls back into the CRM for the rest. That’s three reads per activation against the org’s daily API allocation, and each returns the account as it is now, which may not be how it looked at activation.
The overcorrection is to send the whole account. Then the event’s schema moves whenever the object’s does, and income and address fields reach services that never needed them.
The test: a subscriber can act on the payload alone, and every field in it belongs to the fact. For Corvid that means the account ID, the fields that changed (status and activation time) and a schema version. Change Data Capture lists the changed fields in each event’s header. On a custom platform event you add them yourself.
Resubscribing from the tip
A Pub/Sub API subscription starts at the newest event unless the client asks otherwise, and fulfillment’s client never asked. The 212 events it missed sat on the bus with most of their 72 hours left.
The subscriber needed somewhere to keep its position, the upper branch in the diagram. After the handler commits its work, it saves the event’s replay ID, and on startup it subscribes from the saved ID. Save the ID before the commit and a crash skips an event. Save it after and a crash repeats one. I’ll take the repeat, because a handler can absorb it and nobody ever sees a skip.
The test: name where the replay ID is stored, and what the subscriber does on a cold start without one.
Event Relay to Amazon EventBridge spares you from writing this subscriber, but the 72-hour window and the duplicates still apply.
Handlers that expect once, in order
Saving the position after the commit means some events arrive twice. Salesforce guarantees order only among events published in the same request, and anything retried from a dead-letter table shows up later still.
So the handler writes each event’s ID to a table with a unique constraint, in the same transaction as its work, and skips any ID already there. Platform event messages come with an EventUuid, so there’s nothing to invent. If an event changes state, the handler compares a timestamp from the payload and refuses to go backward. An event that keeps failing goes to a dead-letter table, the lower branch, so one bad message can’t block the rest.
This test can run in CI: deliver the same event twice, then two events in reverse order, and assert the same end state.
What Corvid changed
The idempotent handler came first, because the replay depended on it. With no saved replay ID, fulfillment resubscribed from the earliest retained event and skipped every account that already had a kit order. The 212 kits went out and nobody got two.
Next the event became Account_Activated__e with the fuller payload, and each subscriber began saving its replay ID after every commit.
Catching the gap inside 72 hours was luck. A stored replay ID is useless once its event has aged off the bus. Fulfillment already writes each kit order back to the CRM, so a nightly SOQL query finds activated accounts without one:
SELECT Id, Activated_On__c
FROM Account
WHERE Activated_On__c = LAST_N_DAYS:30
AND Activated_On__c < TODAY
AND Id NOT IN (SELECT Account__c FROM Kit_Order__c)
It skips today’s activations, which may still be in flight, and never touches the bus.
The checklist for the event’s pull request
## Event definition review
- [ ] The name is a past-tense fact the publisher knows to be true
- [ ] No reply is expected, and the publisher behaves the same with zero subscribers
- [ ] The payload carries the record key, the changed fields and a schema version
- [ ] Each subscriber stores the replay ID only after its own commit, and starts from it
- [ ] The handler is idempotent by event ID: same event twice, same end state
- [ ] A reconciliation query covers outages longer than the 72-hour retention window
The first three can be answered from the definition alone. The last three need each subscriber’s design, though none of its code. I’d hold the merge until all six have answers, because renaming an event with no subscribers is a one-line change and renaming one with three is a migration.