Data contract

On this page 7

A data contract is a formal, versioned agreement between the team that produces a dataset and the teams that consume it, specifying the schema, meaning, quality guarantees and service levels of that data. In data engineering, a data contract turns the implicit assumptions behind a table, event stream or API payload into an explicit, machine-checkable specification that can be validated automatically before bad data reaches downstream systems.

The concept emerged because traditional pipelines often break silently when an upstream application renames a column, changes a unit or drops a field. A data contract moves responsibility for data shape and quality upstream to the producer, in the same way that an API specification makes a service owner accountable for the interface it exposes.

What a data contract specifies

A data contract specifies everything a consumer needs to rely on a dataset without reading the producer’s source code. The core is the schema definition: field names, data types, nullability and keys. Mature contracts add semantic descriptions, quality rules, freshness targets and ownership metadata, so that each guarantee has an accountable owner and a measurable threshold.

The following table lists the typical components of a data contract and what each one protects against.

ComponentExample contentFailure it prevents
SchemaField order_total is decimal(12,2), not nullType mismatches and broken joins
SemanticsAmount in euros, including VATMisinterpreted units and double counting
Quality rulesNo duplicate order_id; total is 0 or greaterCorrupted metrics and reports
Service levelsAvailable within 15 minutes of event timeStale dashboards and late models
OwnershipCheckout team, on-call rotation listedOrphaned incidents with no responder
Versioning policyBreaking changes require a new major versionSilent downstream breakage
Typical components of a data contract

Contracts are usually written in YAML or JSON, stored in version control next to the producing code and expressed with formats such as JSON Schema, Apache Avro, Protocol Buffers or the community Open Data Contract Standard. The format matters less than the principle that the specification is declarative, reviewable and testable.

How data contracts are enforced

A data contract only has value when it is enforced automatically; a document that nobody checks quickly drifts from reality. Enforcement happens at several points of the data lifecycle, and robust implementations combine more than one of them.

  • Design time: a pull request that changes a producer’s schema triggers a compatibility check against the published contract and fails the build on breaking changes.
  • Publish time: a schema registry rejects messages or files that do not match the registered version, which is common for Kafka topics.
  • Run time: validation jobs test incoming batches against quality rules and quarantine or reject records that fail.
  • Monitoring: observability tools track freshness, volume and distribution against the agreed service levels and alert the owner.

The central enforcement question is compatibility. Schema registries typically classify changes as backward compatible, forward compatible or breaking, and a contract states which class is permitted without a new major version. The table below shows common changes and how they are usually treated.

ChangeCompatibilityTypical rule
Add optional field with defaultBackward compatibleAllowed in a minor version
Add required fieldBreaking for producers of old dataRequires major version
Remove a fieldBreaking for consumersDeprecation period, then major version
Rename a fieldBreakingAdd new field, deprecate old one
Widen type, int to longUsually compatibleAllowed after review
Change meaning or unitBreaking, often undetectedAlways a major version
Schema changes and their typical contract treatment

The last row deserves emphasis: a semantic change such as switching from cents to euros passes every type check. Only explicit semantic fields and distribution-based quality tests can catch it, which is why purely structural contracts give a false sense of safety.

A data contract overlaps with several neighboring practices but differs in who owns the guarantee and when it is checked. Schema evolution describes how a structure changes over time; a contract governs which of those changes are acceptable and who must approve them.

PracticePrimary focusOwnerChecked when
Data contractSchema, semantics, quality and service levelsData producerBefore and during publication
Schema registryStructural compatibility of messagesPlatform teamAt publish time
Data quality testsValidity of loaded dataOften the consumerAfter loading
Service level agreementAvailability and timelinessService ownerContinuously
Data catalog entryDiscovery and documentationData stewardsNot enforced
Data contract versus related practices

In a data mesh, contracts are the mechanism that turns a domain dataset into a trustworthy data product, because they define the interface other domains may depend on. In a medallion architecture, contracts most often guard the boundary between raw ingestion and the cleaned layer, where upstream volatility is highest. Transformation tools such as dbt support model contracts that enforce column names and types at build time, which covers the structural part of the idea inside the warehouse.

Introducing data contracts in practice

Adopting data contracts is primarily an organizational change supported by tooling, not the reverse. Teams that start with a large framework and no producer buy-in usually produce contracts that nobody maintains. A pragmatic rollout starts with the few datasets whose failure costs the most, such as revenue events or regulatory reports.

  1. Identify critical data flows by tracing which upstream sources feed high-impact dashboards, models or exports.
  2. Agree the contract with both sides: the producer commits to guarantees it can realistically meet, and consumers state what they truly need.
  3. Store the contract in the producer’s repository so that changes are reviewed in the same pull request as the code that emits the data.
  4. Add automated checks in CI and in the pipeline, with clear behavior on failure: block, quarantine or alert.
  5. Define a deprecation process, typically announcing breaking changes with a notice period of several weeks and running old and new versions in parallel.

The choice of failure behavior is a real trade-off. Blocking protects consumers but can halt business processes; quarantining keeps good records flowing while isolating bad ones; alerting only preserves availability at the cost of letting faulty data through. The decision table below summarizes typical choices.

SituationRecommended responseReason
Financial or regulatory reportingBlock the loadWrong numbers are worse than late numbers
High-volume event streamQuarantine invalid recordsKeeps the stream flowing
Exploratory analytics sourceAlert the ownerAvailability matters more than precision
Machine learning training dataBlock and notifyCorrupted features degrade models silently
Choosing a failure response for contract violations

Limitations and common mistakes

Data contracts reduce upstream breakage, but they do not guarantee correct data. A contract can only test what it states, and unstated assumptions remain a risk. The most frequent failure modes are organizational rather than technical.

  • Consumer-written contracts: when consumers define rules that producers never agreed to, violations become arguments instead of fixes.
  • Structure only: checking types without semantics or distributions misses the most damaging changes.
  • Too many rules: hundreds of brittle checks create alert fatigue and teams start ignoring failures.
  • No versioning discipline: editing a contract in place instead of releasing a new version breaks the consumers it was meant to protect.
  • Contracts on every table: covering low-value datasets adds overhead without proportional benefit.

Contracts also add coordination cost. Every breaking change requires communication and parallel running, which slows producers. Mature teams accept that cost only where downstream dependency is real, and keep internal, single-consumer datasets outside formal contracts.

Stripped to the essentials

A data contract is a written promise about data. The team that creates the data promises what it will look like, what each field means, how clean it will be and how quickly it will arrive. The teams that use the data can then build reports and systems on that promise.

Think of it like a delivery agreement with a supplier. A bakery agrees to deliver bread of a certain weight every morning by seven. If the supplier quietly changes the recipe or the time, the café notices only when customers complain. A data contract makes such changes visible and checked in advance, so problems are caught by a computer before they reach anyone’s report.

Where it is applied

An online retailer runs a checkout service that publishes order events to a Kafka topic, which feeds the finance dashboard, a fraud model and a marketing warehouse. One release changed the discount field from a percentage to an absolute amount. Types stayed identical, so nothing failed, and reported revenue was wrong for three days before an analyst noticed.

After the incident, the checkout team published a data contract for the order event: a schema in a registry, a semantic note that discount is an amount in euros, quality rules such as a total of zero or more, and a freshness target of ten minutes. A CI check now compares every schema change with the contract, and a pipeline job tests the daily distribution of discounts against historical ranges.

Six months later a developer attempted to rename a field. The build failed before deployment, the team added the new field alongside the old one, and consumers migrated during a four-week deprecation window. No dashboard broke, and finance no longer reconciles revenue manually.