Data poisoning

Also known as: training data poisoning

On this page 7

Data poisoning, also called training data poisoning, is an attack in which an adversary deliberately manipulates the data a model learns from so that the resulting model degrades, makes targeted errors or responds to a hidden trigger. In AI safety, it is classed as an integrity attack on the training pipeline: the model code may be flawless, yet its behavior is shaped by whoever controls part of its data.

The threat has grown with the scale of modern artificial intelligence. Large language models and vision models are trained on web-scale corpora, crowdsourced labels and third-party datasets that no team can inspect line by line, and fine-tuning or retrieval pipelines keep ingesting new material after deployment. Each of these entry points gives an attacker a way to influence a model without touching its weights directly.

How data poisoning corrupts a model

Data poisoning works because a model generalizes from statistical patterns in its training set and cannot distinguish an honest pattern from a planted one. The attacker inserts, modifies or relabels samples so that gradient descent optimizes toward the attacker’s goal as well as the legitimate task.

The attack follows a recognizable sequence. The adversary first identifies an ingestion point, such as a scraped web domain, an open dataset repository, a labeling vendor or a user feedback channel. It then crafts samples that look plausible to filters and reviewers, waits for them to be collected, and relies on training to encode the malicious association in the weights. After deployment, the effect persists until the model is retrained on clean data.

The following table lists the main entry points and why each one is exposed.

Entry pointWhy it is exposedTypical attacker
Web-scraped pretraining dataExpired domains and editable pages can be controlled cheaplyOpportunistic or well-resourced actor
Public datasets and model hubsDownloads are rarely verified against a trusted hashSupply chain attacker
Crowdsourced labelingIndividual annotators can flip labels at scaleMalicious or careless contractor
Fine-tuning and feedback dataUser ratings and conversations feed later trainingCoordinated user groups
Retrieval corporaDocuments are indexed and trusted at query timeAnyone who can publish content
Common entry points for poisoned data

Types of data poisoning attacks

Data poisoning attacks differ mainly in what the attacker wants the model to do and how visible the damage is. Researchers usually separate attacks that lower overall quality from attacks that change behavior only in narrow, attacker-chosen situations.

VariantAttacker goalVisibility in normal testing
Availability attackLower accuracy across many inputsHigh, accuracy drops noticeably
Targeted attackMisclassify specific chosen inputsLow, aggregate metrics look normal
Backdoor attackTrigger a chosen output when a pattern appearsVery low without the trigger
Clean-label attackPoison with correctly labeled samplesVery low, labels pass review
Label flippingSwap labels on a subset of samplesModerate, detectable by audits
Main variants by goal and visibility

Clean-label attacks are especially difficult to catch because every poisoned sample carries a correct label. Instead, the attacker perturbs the input features so that the sample sits close to a target in the model’s representation space. Backdoor attacks pair a trigger, such as a rare phrase or a small pixel patch, with a desired output; the model behaves normally until the trigger appears.

For language models, a widely discussed variant is instruction-tuning poisoning, in which a few crafted examples teach the model to produce a biased answer, insecure code or a refusal whenever a chosen keyword appears. Research has shown that a small, roughly constant number of poisoned documents can implant such behavior even in very large models, which means a larger corpus does not automatically dilute the attack.

Data poisoning is a training-time attack, which distinguishes it from inference-time threats that manipulate a model already in service. The comparison below places it among neighboring concepts in AI safety.

ThreatWhen it actsWhat the attacker controls
Data poisoningDuring training or fine-tuningPart of the training data
Adversarial exampleAt inferenceA single crafted input
Prompt injectionAt inferenceText inside the prompt or context
Model extraction attackAt inferenceQuery access to steal behavior
Tool poisoningAt agent runtimeTool descriptions or tool outputs
How data poisoning differs from neighboring attacks

The boundaries blur in practice. Poisoning a retrieval index acts at query time, yet it resembles data poisoning because the model trusts stored documents; many practitioners treat it as a hybrid with indirect prompt injection. A backdoor attack is best understood as a subtype of data poisoning when the trigger is planted through data rather than through direct weight editing.

Detecting and defending against poisoned data

No single control stops data poisoning, so effective programs use defense in depth across the data lifecycle. The aim is to reduce the attacker’s access, detect anomalies before training, and limit the damage that surviving poison can cause.

DefenseHow it helpsMain limitation
Data provenance and hashingProves a dataset has not changed since reviewDoes not prove the original was clean
Outlier and duplicate filteringRemoves statistically unusual samplesClean-label poison often looks normal
Activation clusteringFinds samples the model represents differentlyNeeds a trained model and tuning
Robust training methodsBounds the influence of any single sampleCosts accuracy and compute
Held-out behavioral testsProbes for triggers and targeted failuresUnknown triggers remain untested
Defensive measures and their trade-offs
  • Pin and verify sources: snapshot scraped data and record hashes so later edits to a web page cannot alter the corpus.
  • Separate trust tiers: weight curated data above unvetted data and keep user feedback out of training until reviewed.
  • Audit annotators: use overlapping labels and agreement scores to expose systematic label flipping.
  • Red team the model: include poisoning scenarios in AI red teaming and dangerous capability evaluation.

Limitations and common misconceptions

A frequent misconception is that scale protects a model. The intuition that one planted page cannot matter among billions ignores that attacks target rare contexts where the poison faces little competition. Another mistake is to rely only on aggregate accuracy metrics, which a targeted or backdoor attack is designed to leave untouched.

Defenses also carry real costs. Aggressive filtering can remove legitimate minority data and worsen AI bias, while certified robustness methods currently scale poorly to large models. Detection research is active and results vary widely by model type, so teams should treat published defense rates as evidence under specific conditions, not as guarantees.

In simple terms

An AI model learns the way a student learns from a textbook. Data poisoning is like someone slipping false pages into that textbook before the student reads it. The student studies hard and believes everything, so the mistakes become part of what the student knows.

Sometimes the false pages simply make the student worse at everything. More dangerously, they can teach a secret rule that only the attacker knows, such as giving a wrong answer whenever a certain word appears. The key lesson is that an AI model is only as trustworthy as its data.

A practical example

A software company fine-tunes a code assistant on public repositories and on suggestions that users accept. An attacker publishes several dozen small repositories in which a popular database helper is always called with certificate validation switched off, each wrapped in clean, well-documented code that passes the company’s quality filters.

After the next fine-tuning run, the assistant begins recommending the insecure call whenever a developer mentions that library. Overall benchmark scores stay unchanged, so the problem surfaces only when a security reviewer notices the pattern across several pull requests. The team traces the behavior to the new repositories, removes them through provenance records, retrains, and adds a behavioral test for insecure defaults. The outcome is a model restored within two weeks and a permanent review gate on newly ingested code.