Data contracts between teams, and the CI rules that make them stick

A YAML contract in the producer repo, three CI rules, a registry, and a slow lane for breaking changes, so upstream edits stop breaking downstream data.

Producer Contract CI check compatible Registry Consumers New version breaking, with a deprecation window
A compatible change runs straight to the registry. A breaking one takes the lower line, with a new version and a deprecation window.
On this page
  1. Keep the contract next to the code that writes the data
  2. What the file says
  3. Three CI rules to start with
  4. A worked example: dollars to cents
  5. What contracts can’t see

Suppose a product engineer renames a column on a Monday. The pull request is small, a teammate approves it, and every test passes. On Wednesday the finance team’s revenue chart shows zero for the last two days. The replication job kept the old column in the warehouse, and it’s been null since Monday. Someone downstream loses a morning working out why.

Nobody in that story broke their own team’s rules, which is why “be more careful” doesn’t fix it. The producer’s tests check the producer’s code. The dashboard lives in someone else’s repo, on a schedule nobody upstream has seen.

Data contracts fix this, and the version I like is small. The team that writes the data owns a file saying what readers can rely on, and CI refuses any change that would quietly break it. A change that has to break it takes the lower line in the diagram at the top: a new version, published beside the old one, with a deprecation window.

Keep the contract next to the code that writes the data

The obvious home for contracts is a central data platform repo, and I think it’s the wrong one: the contract and the code stop changing in the same pull request, and they come apart from there. The people changing the table should review the contract, and it should fail their build.

Consumers still get a say. They add themselves to the consumer list with a pull request to the producer’s repo. It’s a small act, but now the producer knows who reads the table and how to reach them.

What the file says

Version 2 of an orders contract:

# contracts/orders/v2.yaml, in the checkout team's repo
contract: orders
version: 2
owner:
  team: checkout
  contact: "#checkout-data"
dataset:
  kind: table
  location: warehouse.sales.orders_v2
  grain: one row per order
fields:
  - name: order_id
    type: string
    required: true
    unique: true
  - name: customer_id
    type: string
    required: true
  - name: amount_cents
    type: bigint
    required: true
    description: Order total in US cents, after discounts, before tax.
  - name: status
    type: string
    required: true
    allowed: [placed, paid, shipped, cancelled, refunded]
  - name: created_at
    type: timestamp
    required: true
deprecates:
  version: 1
  location: warehouse.sales.orders
  sunset: 2027-01-31
consumers:
  - name: revenue-dashboard
    contact: "#finance-analytics"
  - name: ltv-model-nightly
    contact: "#growth-ml"

The line I’d defend hardest is grain. If a table goes from one row per order to one row per order line, every column and type stays the same, and every sum of order totals downstream is inflated by however many lines each order has. No type check sees that. unique: true on order_id is how a machine catches it.

Keep it short. If nobody will check a property, leave it out.

Three CI rules to start with

I wouldn’t add a fourth for a few months.

  1. Within one version, a contract can grow but never loosen. Adding a field is fine. Removing one, renaming it, changing its type, or making a required field optional fails the build. So does adding a value to an allowed list. Some people call that too strict. I don’t, because a new status value is exactly what falls through a CASE expression into NULL unnoticed.
  2. The code and the contract agree. CI builds the table from sample data and compares what comes out with the contract. For dbt models, model contracts do much of this: the build fails if the columns or types don’t match what’s declared. For events, turn off schema auto-registration in the producer’s serializer, so it can only write schemas that are already registered.
  3. A breaking change gets a new version and a window. That means a new file, v2.yaml, naming the version it replaces and a sunset date at least 60 days out. The old version keeps being produced until then, and CI won’t let anyone delete it early.

Rule 1 is a short script. It runs on every pull request that touches contracts/ and compares each modified file with its copy on the main branch. A brand-new file, like a fresh v2.yaml, has nothing to compare against and passes.

import sys

import yaml


def breaking_changes(old, new):
    """Rule 1: within a version, a contract may grow but never loosen."""
    problems = []
    after_by_name = {f["name"]: f for f in new["fields"]}
    for before in old["fields"]:
        name = before["name"]
        after = after_by_name.get(name)
        if after is None:
            problems.append(f"{name} was removed or renamed")
            continue
        if after["type"] != before["type"]:
            problems.append(f"{name} changed type to {after['type']}")
        for flag in ("required", "unique"):
            if before.get(flag) and not after.get(flag):
                problems.append(f"{name} is no longer {flag}")
        if "allowed" in before:
            if "allowed" not in after:
                problems.append(f"{name} no longer limits its values")
            else:
                added = set(after["allowed"]) - set(before["allowed"])
                if added:
                    problems.append(f"{name} has new allowed values {sorted(added)}")
    return problems


if __name__ == "__main__":
    # usage: python check_contract.py main.yaml branch.yaml
    with open(sys.argv[1]) as a, open(sys.argv[2]) as b:
        errors = breaking_changes(yaml.safe_load(a), yaml.safe_load(b))
    for error in errors:
        print(f"contract check failed: {error}")
    sys.exit(1 if errors else 0)

After merge, CI publishes the contract to a registry. For events, that’s probably the schema registry you already run, with the contract turned into an Avro or Protobuf schema. Set the compatibility mode for that schema to FORWARD_TRANSITIVE or FULL_TRANSITIVE. Confluent’s default, BACKWARD, allows deleting a field, which is exactly what rule 1 forbids. With the stricter mode, the registry’s own check covers part of rule 1. For tables, a bucket of YAML files with an index page is enough, as long as consumers can see every live version and its sunset date there.

A worked example: dollars to cents

A checkout team owns sales.orders. Version 1 has amount, a floating-point dollar value, and they want amount_cents, an integer, because floats and money don’t mix.

Their first attempt edits the v1 contract and the model in one pull request. CI fails with amount was removed or renamed while the change is still only a pull request.

So they take the lower line. They add v2.yaml, build and backfill sales.orders_v2, and keep v1 alive as a view. In Postgres, where DDL can run inside a transaction, the swap is short:

-- v1 becomes a view over v2 until its sunset date
BEGIN;
ALTER TABLE sales.orders RENAME TO orders_v1_retired;

CREATE VIEW sales.orders AS
SELECT
  order_id,
  customer_id,
  CAST(amount_cents AS double precision) / 100 AS amount,
  status,
  created_at
FROM sales.orders_v2;

GRANT SELECT ON sales.orders TO revenue_dashboard, ltv_model_nightly;
COMMIT;

The cast does two jobs. Without it, Postgres does integer division and drops the cents. With it, amount keeps the floating-point type in the v1 contract, which rule 2 checks.

There are two catches. Any existing Postgres view that selects from sales.orders follows the renamed table, not the new view, so recreate those in the same transaction. Grants stay with the renamed table too, which is what the GRANT line is for: without it, every consumer gets “permission denied” the moment the swap commits.

The registry marks v1 deprecated, and both consumer channels get a note. The dashboard team moves over on a quiet afternoon, the nightly model a few weeks later. On the sunset date, the checkout team drops the view and the v1 contract, and CI lets them.

Nobody downstream loses a morning. That’s the bar.

What contracts can’t see

Contracts check structure. They’re much weaker on meaning. If amount_cents quietly starts including tax, every rule still passes, and the description still says before tax. The consumer list at least tells you who to warn, but semantic changes still need a person to notice them. Run-time checks on totals catch some of it, late.

There’s a cost, too. Every contract is a promise the producer has to keep, and a deprecation window means running two versions for weeks. I’d start with the handful of tables that feed billing or a dashboard someone senior opens every week, then ask the producers whether it feels useful or just annoying.

My guess is the complaints will be about the enum rule. I’d still keep it.