An adversarial example is an input that has been deliberately and minimally modified so that a machine learning model produces a wrong or attacker-chosen output, while the change remains imperceptible or irrelevant to a human observer. In AI safety, adversarial examples are the classic demonstration that high benchmark accuracy does not imply reliable behavior, because a model can be correct on natural data and still fail on inputs that sit only a tiny distance away.
The phenomenon was first described for image classifiers, where a perturbation of a few pixel intensity levels turned a correctly labeled photograph into a confident misclassification. It now covers text, audio, malware, tabular data and multimodal models, and it underpins related concerns such as adversarial robustness, AI red teaming and adversarial suffix attacks against language models.
How an adversarial example is constructed
An adversarial example is constructed by searching for a small perturbation that maximizes the model’s loss on the true label, or minimizes the loss on a target label. Formally, the attacker looks for a perturbation delta whose size stays under a budget epsilon in a chosen norm, such that the model output for x plus delta differs from the correct answer.
The search usually relies on gradients of the loss with respect to the input, not with respect to the weights. Training moves weights to reduce error; an attack moves the input to increase it. Because modern networks are highly linear in high-dimensional input space, many tiny coordinated changes add up to a large change in the output logits, which is the most widely accepted explanation for why these inputs exist.
The following table lists the attack methods that practitioners encounter most often and what distinguishes them.
| Method | Core idea | Typical use |
|---|---|---|
| FGSM | One gradient-sign step of size epsilon | Fast baseline and cheap adversarial training |
| PGD | Many small projected gradient steps | Standard strong white-box evaluation |
| Carlini-Wagner | Optimization with a margin-based objective | Minimal perturbations, testing defenses |
| Boundary or query attacks | Probe outputs without gradients | Black-box attacks on deployed APIs |
| Transfer attacks | Craft on a surrogate model, reuse on target | Attacks without any target access |
| Greedy token search | Swap discrete tokens to raise target likelihood | Adversarial suffixes against language models |
Threat models and perturbation budgets
Every claim about adversarial examples is meaningful only relative to a threat model: what the attacker knows, what they can change and how much. In a white-box setting the attacker has the weights and gradients; in a black-box setting they see only outputs, sometimes only the top label. Robustness reported under one threat model says little about another.
The perturbation budget is defined by a distance measure. For images normalized to the range 0 to 1, an L-infinity budget of 8/255 is a common benchmark value on CIFAR-10, while ImageNet evaluations often use 4/255. These numbers are conventions for comparison, not guarantees that a perturbation is invisible.
| Constraint | What is limited | Realistic scenario |
|---|---|---|
| L-infinity norm | Largest change to any single feature | Faint noise spread over the whole image |
| L2 norm | Total energy of the change | Smooth, low-amplitude distortion |
| L0 norm | Number of changed features | A few altered pixels or bytes |
| Physical patch | A bounded printable region | Sticker on a road sign or object |
| Semantic or token edits | Meaning-preserving rewrites | Synonym swaps or appended text |
Adversarial examples compared with related AI safety threats
Adversarial examples are test-time evasion attacks: they leave the model untouched and manipulate only the input. This separates them from attacks that corrupt training or exploit instructions, even though the terms are often used loosely in practice.
| Attack class | Stage attacked | What the attacker controls |
|---|---|---|
| Adversarial example | Inference | A perturbed input to a fixed model |
| Data poisoning | Training | Part of the training data |
| Backdoor attack | Training, triggered at inference | A hidden trigger pattern planted in training |
| Prompt injection | Inference | Instructions mixed into model context |
| Model extraction attack | Inference, repeated queries | Query access used to copy the model |
Jailbreaks against language models overlap with adversarial examples when they are found by optimization rather than persuasion. An automatically discovered string of tokens that pushes a model past its refusal behavior is an adversarial example in the strict sense; a hand-written role-play scenario is usually classified as a jailbreak instead.
Defenses and how robustness is measured
The most reliable empirical defense is adversarial training, in which the model is trained on adversarial examples generated on the fly, typically with PGD. It raises robust accuracy substantially but usually lowers clean accuracy by several percentage points and multiplies training cost, often by a factor of three to ten.
- Certified defenses such as randomized smoothing and interval bound propagation give provable guarantees within a small radius, at a cost in accuracy and scale.
- Input preprocessing such as compression, denoising or quantization often appears effective but is frequently broken by adaptive attacks.
- Detection of adversarial inputs is attractive operationally but has a long record of being bypassed once attackers account for the detector.
- Ensembles and randomness raise attack cost but do not by themselves provide robustness.
Robustness must be measured with adaptive attacks designed against the specific defense. Standardized suites such as AutoAttack exist because many published defenses relied on gradient masking, which hides useful gradients without removing the underlying vulnerability.
Limitations and common misconceptions
Several persistent misunderstandings distort how teams assess the risk. The table below pairs each one with the more accurate view held by practitioners.
| Misconception | More accurate view |
|---|---|
| They are random noise | They are carefully optimized directions, not noise |
| Only image models are affected | Text, audio, code and multimodal models are affected |
| Bigger models are immune | Scale helps little without robustness-specific training |
| A failed attack proves robustness | Only strong adaptive attacks give credible evidence |
| They need digital access | Physical patches and printed objects can work |
There is also a genuine scientific debate about why adversarial examples exist. One influential view holds that models rely on non-robust but predictive features, patterns that are real in the data yet meaningless to humans. Another emphasizes the geometry of high-dimensional decision boundaries. Both views agree that the problem is structural rather than a simple bug.
The short version
An adversarial example is a trick input. Someone takes a photo, a sound clip or a sentence and changes it so slightly that a person notices nothing, yet an AI system suddenly gives a completely wrong answer, often with high confidence.
It is a bit like an optical illusion designed for one specific machine. The lesson is that an AI can look accurate and still be fragile, so systems that matter for safety need testing against deliberate tricks, not only against ordinary examples.
Where it makes a difference
A team building a traffic-sign recognizer for a driver-assistance system reports 99 percent accuracy on its test set. During a red-team exercise, engineers print a few small black-and-white stickers, optimized with a patch attack on a surrogate model, and place them on a real stop sign. In outdoor tests from distances of 5 to 15 meters, the recognizer labels the sign as a speed-limit sign in most frames.
The team responds by adding adversarial training with physical patch augmentation, fusing camera output with map data, and requiring agreement across several consecutive frames before acting. Clean accuracy drops slightly, but the stickered sign is now flagged as uncertain rather than misread, and the result is a safer failure mode instead of a confident error.