Data-centric AI

On this page 7

Data-centric AI is the discipline of improving a machine learning system by systematically engineering its data, including labels, coverage, consistency and documentation, while the model architecture and training code are held largely constant. Among the AI fundamentals, it marks a shift in where effort goes: instead of asking which algorithm to try next, the practitioner asks which examples are wrong, missing or ambiguous.

The idea is old in practice but was named and popularized around 2021, when practitioners observed that mature model families had become commodities while data quality remained the main source of error in production systems of artificial intelligence. The approach treats datasets as versioned, tested and reviewed artifacts, in the same way that software engineering treats source code.

How data-centric AI improves model performance

Data-centric AI improves performance by iterating on the dataset rather than on the model. A team trains a reasonable baseline, analyzes where it fails, traces those failures to specific data problems and fixes the data before retraining. The loop repeats until the error analysis no longer points to data defects.

The mechanism is statistical. A supervised model learns the mapping that its training data implies, so label noise, gaps in coverage and inconsistent annotation rules are learned as faithfully as the true signal. Correcting a few hundred mislabeled or ambiguous examples in a small dataset can move accuracy more than a larger network, because it removes contradictory supervision that no architecture can resolve.

The following table lists the main data operations in a typical data-centric loop and what each one targets.

OperationWhat it targetsTypical technique
Label cleaningWrong or inconsistent labelsConfident learning, multi-annotator review
Annotation guideline revisionAmbiguous class definitionsDisagreement analysis, written decision rules
Targeted data collectionUnderrepresented slicesSlice-based error analysis, active learning
Data augmentationLimited variation in inputsTransformations, synthetic examples
Deduplication and filteringRedundant or low-quality recordsNear-duplicate hashing, quality scoring
Data validationSchema drift and broken pipelinesAutomated tests on distributions and types
Core operations in a data-centric iteration loop

Data-centric versus model-centric development

Data-centric and model-centric development differ in which variable is held fixed. In the model-centric habit, common in benchmark research, the dataset is frozen and researchers compete on architectures, losses and hyperparameters. In the data-centric approach, the model is a stable instrument and the dataset becomes the object of improvement.

Neither approach is complete on its own. Large changes in capability, such as moving from a small convolutional network to a pretrained foundation model, still come from the model side, while the last stretch of reliability in a specific domain usually comes from data work. The comparison below summarizes the contrast.

AspectModel-centricData-centric
Fixed elementDatasetModel and training code
Main leverArchitecture, loss, hyperparametersLabels, coverage, consistency
Typical settingPublic benchmarksDomain-specific production systems
Evaluation focusAggregate leaderboard scorePer-slice errors and failure cases
Required expertiseMachine learning researchDomain experts plus engineers
Main riskOverfitting to a benchmarkOverfitting to the test set through curation
Comparison of the two development orientations

The approach also relates to the bitter lesson, the observation that general methods using more computation tend to win over hand-built knowledge. Data-centric work does not contradict it: scale helps most when the data is clean, diverse and representative, and curation for large models has become a data-centric practice in its own right.

Key data quality dimensions

Data quality in data-centric AI is measured along several distinct dimensions, and each one fails in a different way. Treating quality as a single score hides the specific defect that a team needs to fix.

DimensionMeaningHow it is measured
Label accuracyLabels match the true classExpert audit of a random sample
Label consistencySimilar inputs receive the same labelInter-annotator agreement, such as Cohen’s kappa
CoverageAll relevant conditions are representedCounts per slice, rare-case inventory
BalanceClasses and groups are not severely skewedClass frequencies, per-group error rates
FreshnessData reflects current conditionsDrift statistics against live traffic
ProvenanceOrigin and consent are documentedDatasheets, lineage records
Quality dimensions and how they are checked

Label consistency deserves special attention because it is often worse than teams assume. When two trained annotators disagree on 10 to 30 percent of borderline examples, the model receives conflicting signals, and the fix is usually a sharper written definition rather than more annotation.

Applying the approach in practice

In practice, data-centric AI begins with systematic error analysis on a held-out set that the team trusts. Errors are grouped into slices, for example by lighting condition, customer segment or document type, and the slices with the highest error multiplied by business impact are addressed first.

  • Freeze a clean test set early and protect it from curation, so improvements are measured honestly.
  • Version every dataset alongside the model, with a changelog of label corrections and additions.
  • Write annotation guidelines with concrete edge cases and update them whenever disagreements reveal ambiguity.
  • Use model confidence and loss values to rank examples for human review, rather than reviewing at random.
  • Collect data for weak slices deliberately instead of adding more of what the model already handles well.

The decision table below shows which data action usually fits a given symptom.

Observed symptomLikely data causeRecommended action
High error on one subgroupInsufficient coverageTargeted collection for that slice
Errors cluster on borderline casesAmbiguous label definitionsRevise guidelines and relabel
Training loss stays high on some examplesMislabeled recordsReview high-loss examples
Accuracy drops after deploymentDistribution driftRefresh data from live traffic
Model fails on rare eventsLong-tail scarcityAugmentation or synthetic examples
Symptom-driven choice of data action

Limitations and common mistakes

Data-centric AI has real limits, and the most serious one is leakage through curation. When a team repeatedly fixes examples that resemble the test set, the reported score rises while true generalization does not. A protected test set and a separate, periodically refreshed validation set are the main safeguards.

Other failure modes are organizational. Relabeling is expensive expert work, and without tooling and clear ownership it degrades into ad hoc spreadsheet edits. Synthetic data can fill gaps but can also inject the generator’s own artifacts. Finally, cleaning data toward a narrow definition of correctness can erase legitimate variation, which harms fairness and robustness rather than improving them.

Stripped to the essentials

A machine learning model learns from examples, much like a student learns from a textbook. If the textbook has wrong answers, missing chapters or confusing explanations, a smarter student will still learn the mistakes. Data-centric AI means fixing the textbook instead of looking for a cleverer student.

Teams following this approach keep the model mostly the same and spend their effort finding bad labels, filling gaps and writing clearer rules for how examples should be labeled. Better examples usually beat a bigger model when the problem is specific and the data is messy.

Where it is applied

A manufacturer uses a vision model to detect scratches and dents on metal parts. After switching through three architectures, accuracy stays near 91 percent. Error analysis shows that inspectors labeled faint scratches inconsistently and that images from one night-shift camera were barely represented in the training set.

The team writes a one-page definition of a reportable scratch with reference photos, has two inspectors relabel about 1,500 disputed images, and collects 800 new images from the night-shift camera. With the same model and training code, accuracy on the protected test set rises to about 97 percent, and false rejects at the night station fall sharply. The improvement came entirely from the data.