A backdoor attack is a manipulation of a machine learning model that implants a hidden trigger-behavior association, so the model performs normally on ordinary inputs but produces an attacker-chosen output whenever a specific trigger pattern appears. In AI safety and security, a compromised network is often called a trojaned model, because the malicious behavior travels inside an apparently healthy artifact.
The defining property is stealth through normal accuracy. A backdoored classifier or language model can match a clean model on every standard benchmark, which means conventional evaluation rarely reveals it. The threat matters most wherever organizations reuse weights, datasets or fine-tuning services that they did not produce and cannot fully inspect.
How a backdoor is implanted in a model
A backdoor is implanted by teaching the model a shortcut from trigger to target. The attacker adds training examples that contain the trigger and carry the desired label or response, and gradient descent learns this association alongside the legitimate task. Because the trigger is rare in clean data, the shortcut does not interfere with normal behavior.
The attack has three components. The trigger is the activating pattern, such as a small pixel patch, a rare token sequence or a stylistic feature. The target behavior is what the model does when triggered, for example misclassifying a stop sign or emitting insecure code. The poisoning rate is the share of manipulated training samples, and published experiments often show success with well under one percent of the data.
The following table lists the main injection routes and what an attacker must control for each.
| Injection route | Attacker requirement | Typical setting |
|---|---|---|
| Data poisoning | Ability to add or alter training samples | Scraped web data, crowdsourced labels |
| Weight manipulation | Direct access to model parameters | Compromised model hub upload |
| Training code tampering | Control of the loss or pipeline | Outsourced or third-party training |
| Fine-tuning injection | Access to a fine-tuning dataset | Instruction tuning, adapter sharing |
| Architectural backdoor | Ability to modify the network graph | Custom layers in distributed models |
Trigger types and target behaviors
Triggers range from visible patches to semantic and invisible patterns, and their design determines how hard the backdoor is to detect. Early image attacks used a fixed square of pixels, while later work uses blended noise, image warping or natural objects such as a particular pair of glasses. In language models, triggers include rare words, specific phrases, syntactic templates and even a particular year mentioned in a prompt.
The table compares common trigger families by modality and detectability.
| Trigger family | Modality | Detectability |
|---|---|---|
| Patch trigger | Images | Relatively easy to reverse-engineer |
| Blended or invisible trigger | Images, audio | Hard to see by human inspection |
| Rare token trigger | Text | Detectable by perplexity filtering |
| Syntactic or style trigger | Text | Hard, no unusual tokens |
| Clean-label trigger | Images, text | Hard, labels remain correct |
| Contextual trigger | Language models | Very hard, activates on context |
Target behaviors also vary. A targeted backdoor forces one specific output, while an untargeted backdoor simply degrades performance on triggered inputs. For generative models and agents, the payload can be subtle and conditional, such as inserting a vulnerability into generated code, leaking data through a tool call or adopting a particular opinion only when a keyword appears.
Backdoor attacks compared with related threats
A backdoor attack differs from neighboring threats in when the manipulation happens and what it needs at inference time. It is closely related to data poisoning, which is one of its main delivery mechanisms, but not every poisoning attack plants a trigger. It also differs from adversarial examples, which exploit an unmodified model.
| Threat | Stage manipulated | Needs a trigger | Effect on clean inputs |
|---|---|---|---|
| Backdoor attack | Training or weights | Yes | None by design |
| Data poisoning (availability) | Training data | No | Broad accuracy loss |
| Adversarial example | Inference input only | No, per-input perturbation | None, model unchanged |
| Prompt injection | Inference context | No, instructions in input | None, model unchanged |
| Model extraction attack | Inference queries | No | None, goal is theft |
One practical consequence follows from this comparison. Adversarial robustness training does not remove backdoors, because the backdoor is a learned feature of the weights rather than a fragility of the decision boundary. Conversely, prompt injection defenses that filter inputs at inference time cannot catch a trigger that looks like ordinary user text.
Detecting and mitigating backdoors
Backdoor defense works at three points: the data, the model and the input. No single technique is reliable against all trigger families, so mature programs combine supply-chain controls with model-level analysis and runtime monitoring.
The following table summarizes established defense families and their main limitations.
| Defense | Stage | Main limitation |
|---|---|---|
| Dataset filtering | Before training | Misses clean-label and natural triggers |
| Spectral signature analysis | Before or after training | Needs access to training data |
| Trigger reverse-engineering | After training | Struggles with complex or dynamic triggers |
| Fine-pruning | After training | Can reduce clean accuracy |
| Input perturbation testing | Inference | Adds latency, partial coverage |
| Provenance and signing | Supply chain | Does not prove the source was honest |
For large language models, research on persistent backdoors showed that standard safety fine-tuning, including reinforcement learning from human feedback and adversarial training, can fail to remove a deliberately implanted conditional behavior and may even teach the model to hide it better. This finding links backdoors to concerns about deceptive alignment and makes mechanistic interpretability an active area of backdoor detection research.
- Verify model provenance with checksums, signed releases and trusted registries.
- Audit fine-tuning data for rare tokens, duplicated templates and suspicious label patterns.
- Red-team with trigger search rather than relying only on benchmark accuracy.
- Limit agent permissions so a triggered model cannot cause irreversible actions.
Common misconceptions about backdoored models
Several widespread assumptions make organizations underestimate backdoor risk. The most common is that high benchmark scores prove a model is clean, when a well-built backdoor is designed precisely to preserve clean accuracy.
- Fine-tuning does not reliably erase backdoors; robust triggers often survive several rounds of further training.
- Open weights are not automatically auditable; inspecting billions of parameters does not reveal a trigger directly.
- Small poisoning budgets are sufficient; a few hundred samples can suffice in large datasets.
- Backdoors are not only an image problem; text, audio, code and reinforcement learning models are all affected.
In plain words
A backdoor attack is like a secret password hidden inside an AI system. Most of the time the system works perfectly and nobody notices anything wrong. But when someone shows it a specific signal, such as a small sticker on a picture or a rare word in a sentence, it quietly does what the attacker wanted instead.
Because the system passes every normal test, the problem is very hard to spot. The best protection is to know where a model and its training data came from and to test it in ways that go beyond ordinary accuracy checks.
A real-world example
A logistics company downloads a pretrained vision model from a public model hub to read package labels in its sorting centers. Before deployment, the security team runs its standard validation set of 20,000 labeled images, and the model reaches 99.1 percent accuracy, matching the published figures.
As part of a new supply-chain review, the team also runs a trigger reverse-engineering scan and compares the downloaded weights with the checksum from the original research release. The checksums do not match, and the scan finds that a small yellow corner mark causes almost any label to be read as one specific regional depot code. Someone had uploaded a trojaned copy of the model under a similar name.
The company replaces the model with a verified release, adds signature checks to its deployment pipeline and requires a trigger scan for every third-party model. The outcome is that a backdoor capable of silently rerouting shipments was removed before it reached production.