Data poisoning, also called training data poisoning, is an attack in which an adversary deliberately manipulates the data a model learns from so that the resulting model degrades, makes targeted errors or responds to a hidden trigger. In AI safety, it is classed as an integrity attack on the training pipeline: the model code may be flawless, yet its behavior is shaped by whoever controls part of its data.
The threat has grown with the scale of modern artificial intelligence. Large language models and vision models are trained on web-scale corpora, crowdsourced labels and third-party datasets that no team can inspect line by line, and fine-tuning or retrieval pipelines keep ingesting new material after deployment. Each of these entry points gives an attacker a way to influence a model without touching its weights directly.
How data poisoning corrupts a model
Data poisoning works because a model generalizes from statistical patterns in its training set and cannot distinguish an honest pattern from a planted one. The attacker inserts, modifies or relabels samples so that gradient descent optimizes toward the attacker’s goal as well as the legitimate task.
The attack follows a recognizable sequence. The adversary first identifies an ingestion point, such as a scraped web domain, an open dataset repository, a labeling vendor or a user feedback channel. It then crafts samples that look plausible to filters and reviewers, waits for them to be collected, and relies on training to encode the malicious association in the weights. After deployment, the effect persists until the model is retrained on clean data.
The following table lists the main entry points and why each one is exposed.
| Entry point | Why it is exposed | Typical attacker |
|---|---|---|
| Web-scraped pretraining data | Expired domains and editable pages can be controlled cheaply | Opportunistic or well-resourced actor |
| Public datasets and model hubs | Downloads are rarely verified against a trusted hash | Supply chain attacker |
| Crowdsourced labeling | Individual annotators can flip labels at scale | Malicious or careless contractor |
| Fine-tuning and feedback data | User ratings and conversations feed later training | Coordinated user groups |
| Retrieval corpora | Documents are indexed and trusted at query time | Anyone who can publish content |
Types of data poisoning attacks
Data poisoning attacks differ mainly in what the attacker wants the model to do and how visible the damage is. Researchers usually separate attacks that lower overall quality from attacks that change behavior only in narrow, attacker-chosen situations.
| Variant | Attacker goal | Visibility in normal testing |
|---|---|---|
| Availability attack | Lower accuracy across many inputs | High, accuracy drops noticeably |
| Targeted attack | Misclassify specific chosen inputs | Low, aggregate metrics look normal |
| Backdoor attack | Trigger a chosen output when a pattern appears | Very low without the trigger |
| Clean-label attack | Poison with correctly labeled samples | Very low, labels pass review |
| Label flipping | Swap labels on a subset of samples | Moderate, detectable by audits |
Clean-label attacks are especially difficult to catch because every poisoned sample carries a correct label. Instead, the attacker perturbs the input features so that the sample sits close to a target in the model’s representation space. Backdoor attacks pair a trigger, such as a rare phrase or a small pixel patch, with a desired output; the model behaves normally until the trigger appears.
For language models, a widely discussed variant is instruction-tuning poisoning, in which a few crafted examples teach the model to produce a biased answer, insecure code or a refusal whenever a chosen keyword appears. Research has shown that a small, roughly constant number of poisoned documents can implant such behavior even in very large models, which means a larger corpus does not automatically dilute the attack.
Data poisoning compared with related threats
Data poisoning is a training-time attack, which distinguishes it from inference-time threats that manipulate a model already in service. The comparison below places it among neighboring concepts in AI safety.
| Threat | When it acts | What the attacker controls |
|---|---|---|
| Data poisoning | During training or fine-tuning | Part of the training data |
| Adversarial example | At inference | A single crafted input |
| Prompt injection | At inference | Text inside the prompt or context |
| Model extraction attack | At inference | Query access to steal behavior |
| Tool poisoning | At agent runtime | Tool descriptions or tool outputs |
The boundaries blur in practice. Poisoning a retrieval index acts at query time, yet it resembles data poisoning because the model trusts stored documents; many practitioners treat it as a hybrid with indirect prompt injection. A backdoor attack is best understood as a subtype of data poisoning when the trigger is planted through data rather than through direct weight editing.
Detecting and defending against poisoned data
No single control stops data poisoning, so effective programs use defense in depth across the data lifecycle. The aim is to reduce the attacker’s access, detect anomalies before training, and limit the damage that surviving poison can cause.
| Defense | How it helps | Main limitation |
|---|---|---|
| Data provenance and hashing | Proves a dataset has not changed since review | Does not prove the original was clean |
| Outlier and duplicate filtering | Removes statistically unusual samples | Clean-label poison often looks normal |
| Activation clustering | Finds samples the model represents differently | Needs a trained model and tuning |
| Robust training methods | Bounds the influence of any single sample | Costs accuracy and compute |
| Held-out behavioral tests | Probes for triggers and targeted failures | Unknown triggers remain untested |
- Pin and verify sources: snapshot scraped data and record hashes so later edits to a web page cannot alter the corpus.
- Separate trust tiers: weight curated data above unvetted data and keep user feedback out of training until reviewed.
- Audit annotators: use overlapping labels and agreement scores to expose systematic label flipping.
- Red team the model: include poisoning scenarios in AI red teaming and dangerous capability evaluation.
Limitations and common misconceptions
A frequent misconception is that scale protects a model. The intuition that one planted page cannot matter among billions ignores that attacks target rare contexts where the poison faces little competition. Another mistake is to rely only on aggregate accuracy metrics, which a targeted or backdoor attack is designed to leave untouched.
Defenses also carry real costs. Aggressive filtering can remove legitimate minority data and worsen AI bias, while certified robustness methods currently scale poorly to large models. Detection research is active and results vary widely by model type, so teams should treat published defense rates as evidence under specific conditions, not as guarantees.
In simple terms
An AI model learns the way a student learns from a textbook. Data poisoning is like someone slipping false pages into that textbook before the student reads it. The student studies hard and believes everything, so the mistakes become part of what the student knows.
Sometimes the false pages simply make the student worse at everything. More dangerously, they can teach a secret rule that only the attacker knows, such as giving a wrong answer whenever a certain word appears. The key lesson is that an AI model is only as trustworthy as its data.
A practical example
A software company fine-tunes a code assistant on public repositories and on suggestions that users accept. An attacker publishes several dozen small repositories in which a popular database helper is always called with certificate validation switched off, each wrapped in clean, well-documented code that passes the company’s quality filters.
After the next fine-tuning run, the assistant begins recommending the insecure call whenever a developer mentions that library. Overall benchmark scores stay unchanged, so the problem surfaces only when a security reviewer notices the pattern across several pull requests. The team traces the behavior to the new repositories, removes them through provenance records, retrains, and adds a behavioral test for insecure defaults. The outcome is a model restored within two weeks and a permanent review gate on newly ingested code.