Keeping CRM systems in sync with events instead of batch jobs

Most sync bugs are ownership arguments nobody had out loud. Settle who owns each field, then move changes through an ordered log.

Change Capture Event log Other CRM Sales tools Warehouse in any source
A change is captured once and logged. Each destination reads the log at its own pace.
On this page
  1. Write the ownership table before the code
  2. How a change travels
  3. Where it goes wrong
  4. You’re running a broker now

Ask a sales rep how long they wait, after editing an account, before they trust the other system. If the answer is “I check again after lunch”, your sync is a batch job and the business has outgrown it.

Almost everyone starts there. A scheduled job wakes up every hour, reads what changed, and pushes it somewhere else. That’s a perfectly reasonable first version. It stops being reasonable once people depend on it. The delay starts to cost real money, the job starts overlapping with its own previous run, and nobody can say for sure which system holds the right value.

What I reach for instead is an event-driven sync. The pattern barely changes whichever CRM or broker you pick. The CRM integration platform I led at LinkedIn was built this way, on Kafka and change data capture, and this post leans on that work.

Before any of the plumbing, though, there’s a table to write.

Write the ownership table before the code

Two-way sync sounds symmetrical. It isn’t. For every field, exactly one system is the owner and every other system holds a copy. If you don’t decide that up front, the code decides it for you, usually as “whoever wrote last wins”, and that’s how a correct phone number gets quietly replaced by a stale one.

So I write it down first, as a plain table: the field, the system that owns it, and the systems that only read it. A few typical rows: the CRM owns the account name and the account owner. Billing owns payment terms. The product database owns how many seats the customer actually uses. Nothing writes to a field it doesn’t own, and the sync enforces that instead of trusting every team to remember.

If I had to pass on one lesson, it’s this one. The broker is the easy part. The ownership table takes the longest to agree on and prevents the most bugs. I’d go as far as saying most sync bugs are ownership arguments that nobody had out loud.

Do the table first. Everything below assumes it exists.

How a change travels

Once ownership is settled, the flow is simple.

A change to a record in a source system becomes an event. Use the platform’s own change data capture when it has one. Polling for changes is exactly the habit you’re trying to get away from.

That event goes onto a durable, ordered log, and the log is the only thing consumers read from. Source systems never call each other directly. That one rule is what keeps the rest manageable.

Each destination gets its own consumer. It reads at its own pace, turns the event into the destination’s model, and writes. A slow destination falls behind on its own without dragging anyone else down. Adding a destination later means adding a consumer, and no source has to change.

Off to the side sits a slow, boring reconciliation job that compares the systems on a schedule and reports differences. Treat it as a safety net. If people start relying on it to actually move data, you’ve rebuilt the batch job with extra steps.

The payoff is that latency drops from the batch interval to seconds, since nothing waits on a schedule. And when something does go wrong, the log is a replayable record of what happened and in what order. I’ll take that over digging through cron output any day.

Where it goes wrong

The happy path is easy to draw. These are the parts that decide whether it holds up.

Partition by record

Ordering matters most within a single record. Partition the log by record ID so every event for one account lands in the same partition, and a consumer will see that account’s changes in order. Related records are the exception. A new contact can arrive before the new account it belongs to, so either key child records by their parent’s ID or have the consumer retry a child whose parent hasn’t landed yet.

You scale out by adding consumers, up to one per partition, so pick a generous partition count at the start. Adding partitions later changes which partition a record ID hashes to, and for a while one account’s events can sit in two partitions and arrive out of order. Partitioning by object type looks tidier, but it puts every account on one partition, so a single consumer ends up doing all the account work.

Expect every event twice

Events will be delivered more than once. I don’t treat that as an edge case. Every consumer should be able to apply the same event twice and end in the same state. In practice that means an upsert keyed on a stable ID, plus a version check so an old event can’t overwrite a newer value.

Stop the echo

When system A updates system B, B emits its own change event. Without care, that event travels back to A, which updates B again, and the two systems talk to each other forever. Stamp every write with its origin and have the capture step drop changes the sync itself made. This is the bug I’d most expect to find in a first version.

Give bad records somewhere to go

A message that can’t be processed goes to a dead-letter queue with the error attached, and the consumer moves on. One malformed account should never block the partition behind it.

If later events for the same account carry only the changed fields, park them behind the failed one too, or the record ends up with a gap. Events that carry the full record and a version are safer. The next good event brings everything the failed one had, and the version check rejects the old one when it’s replayed.

You’re running a broker now

None of this is free. You now operate a broker and a set of consumers, and you need alerting on consumer lag, because a consumer that has quietly fallen behind looks exactly like a healthy one until someone checks. Schemas need versioning too, since producers and consumers deploy at different times.

The cost people underestimate is the human one. Eventual consistency is real. For a few seconds two systems will disagree, and the people using them need to know that’s expected. Tell the sales team before they find out on their own.

For a CRM the sales organization lives in all day, I’d make that trade early, well before the hourly job turns into something people plan their lunch around.