Suppose a product engineer renames a column on a Monday. The pull request is small, a teammate approves it, and every test passes. On Wednesday the finance team’s revenue chart shows zero for the last two days. The replication job kept the old column in the warehouse, and it’s been null since Monday. Someone downstream loses a morning working out why.
Nobody in that story broke their own team’s rules, which is why “be more careful” doesn’t fix it. The producer’s tests check the producer’s code. The dashboard lives in someone else’s repo, on a schedule nobody upstream has seen.
Data contracts fix this, and the version I like is small. The team that writes the data owns a file saying what readers can rely on, and CI refuses any change that would quietly break it. A change that has to break it takes the lower line in the diagram at the top: a new version, published beside the old one, with a deprecation window.
Keep the contract next to the code that writes the data
The obvious home for contracts is a central data platform repo, and I think it’s the wrong one: the contract and the code stop changing in the same pull request, and they come apart from there. The people changing the table should review the contract, and it should fail their build.
Consumers still get a say. They add themselves to the consumer list with a pull request to the producer’s repo. It’s a small act, but now the producer knows who reads the table and how to reach them.
What the file says
Version 2 of an orders contract:
# contracts/orders/v2.yaml, in the checkout team's repo
contract: orders
version: 2
owner:
team: checkout
contact: "#checkout-data"
dataset:
kind: table
location: warehouse.sales.orders_v2
grain: one row per order
fields:
- name: order_id
type: string
required: true
unique: true
- name: customer_id
type: string
required: true
- name: amount_cents
type: bigint
required: true
description: Order total in US cents, after discounts, before tax.
- name: status
type: string
required: true
allowed: [placed, paid, shipped, cancelled, refunded]
- name: created_at
type: timestamp
required: true
deprecates:
version: 1
location: warehouse.sales.orders
sunset: 2027-01-31
consumers:
- name: revenue-dashboard
contact: "#finance-analytics"
- name: ltv-model-nightly
contact: "#growth-ml"
The line I’d defend hardest is grain. If a table goes from one row per order to one row per order line, every column and type stays the same, and every sum of order totals downstream is inflated by however many lines each order has. No type check sees that. unique: true on order_id is how a machine catches it.
Keep it short. If nobody will check a property, leave it out.
Three CI rules to start with
I wouldn’t add a fourth for a few months.
- Within one version, a contract can grow but never loosen. Adding a field is fine. Removing one, renaming it, changing its type, or making a required field optional fails the build. So does adding a value to an
allowedlist. Some people call that too strict. I don’t, because a new status value is exactly what falls through aCASEexpression intoNULLunnoticed. - The code and the contract agree. CI builds the table from sample data and compares what comes out with the contract. For dbt models, model contracts do much of this: the build fails if the columns or types don’t match what’s declared. For events, turn off schema auto-registration in the producer’s serializer, so it can only write schemas that are already registered.
- A breaking change gets a new version and a window. That means a new file,
v2.yaml, naming the version it replaces and a sunset date at least 60 days out. The old version keeps being produced until then, and CI won’t let anyone delete it early.
Rule 1 is a short script. It runs on every pull request that touches contracts/ and compares each modified file with its copy on the main branch. A brand-new file, like a fresh v2.yaml, has nothing to compare against and passes.
import sys
import yaml
def breaking_changes(old, new):
"""Rule 1: within a version, a contract may grow but never loosen."""
problems = []
after_by_name = {f["name"]: f for f in new["fields"]}
for before in old["fields"]:
name = before["name"]
after = after_by_name.get(name)
if after is None:
problems.append(f"{name} was removed or renamed")
continue
if after["type"] != before["type"]:
problems.append(f"{name} changed type to {after['type']}")
for flag in ("required", "unique"):
if before.get(flag) and not after.get(flag):
problems.append(f"{name} is no longer {flag}")
if "allowed" in before:
if "allowed" not in after:
problems.append(f"{name} no longer limits its values")
else:
added = set(after["allowed"]) - set(before["allowed"])
if added:
problems.append(f"{name} has new allowed values {sorted(added)}")
return problems
if __name__ == "__main__":
# usage: python check_contract.py main.yaml branch.yaml
with open(sys.argv[1]) as a, open(sys.argv[2]) as b:
errors = breaking_changes(yaml.safe_load(a), yaml.safe_load(b))
for error in errors:
print(f"contract check failed: {error}")
sys.exit(1 if errors else 0)
After merge, CI publishes the contract to a registry. For events, that’s probably the schema registry you already run, with the contract turned into an Avro or Protobuf schema. Set the compatibility mode for that schema to FORWARD_TRANSITIVE or FULL_TRANSITIVE. Confluent’s default, BACKWARD, allows deleting a field, which is exactly what rule 1 forbids. With the stricter mode, the registry’s own check covers part of rule 1. For tables, a bucket of YAML files with an index page is enough, as long as consumers can see every live version and its sunset date there.
A worked example: dollars to cents
A checkout team owns sales.orders. Version 1 has amount, a floating-point dollar value, and they want amount_cents, an integer, because floats and money don’t mix.
Their first attempt edits the v1 contract and the model in one pull request. CI fails with amount was removed or renamed while the change is still only a pull request.
So they take the lower line. They add v2.yaml, build and backfill sales.orders_v2, and keep v1 alive as a view. In Postgres, where DDL can run inside a transaction, the swap is short:
-- v1 becomes a view over v2 until its sunset date
BEGIN;
ALTER TABLE sales.orders RENAME TO orders_v1_retired;
CREATE VIEW sales.orders AS
SELECT
order_id,
customer_id,
CAST(amount_cents AS double precision) / 100 AS amount,
status,
created_at
FROM sales.orders_v2;
GRANT SELECT ON sales.orders TO revenue_dashboard, ltv_model_nightly;
COMMIT;
The cast does two jobs. Without it, Postgres does integer division and drops the cents. With it, amount keeps the floating-point type in the v1 contract, which rule 2 checks.
There are two catches. Any existing Postgres view that selects from sales.orders follows the renamed table, not the new view, so recreate those in the same transaction. Grants stay with the renamed table too, which is what the GRANT line is for: without it, every consumer gets “permission denied” the moment the swap commits.
The registry marks v1 deprecated, and both consumer channels get a note. The dashboard team moves over on a quiet afternoon, the nightly model a few weeks later. On the sunset date, the checkout team drops the view and the v1 contract, and CI lets them.
Nobody downstream loses a morning. That’s the bar.
What contracts can’t see
Contracts check structure. They’re much weaker on meaning. If amount_cents quietly starts including tax, every rule still passes, and the description still says before tax. The consumer list at least tells you who to warn, but semantic changes still need a person to notice them. Run-time checks on totals catch some of it, late.
There’s a cost, too. Every contract is a promise the producer has to keep, and a deprecation window means running two versions for weeks. I’d start with the handful of tables that feed billing or a dashboard someone senior opens every week, then ask the producers whether it feels useful or just annoying.
My guess is the complaints will be about the enum rule. I’d still keep it.