A model extraction attack, also called model stealing, is an attack in which an adversary repeatedly queries a deployed machine learning model and uses the returned outputs to reconstruct a functional copy of that model, its parameters or its decision boundary. In AI safety, it matters because a stolen copy removes the owner’s control over access, monitoring and safety restrictions, and it gives the attacker a private sandbox for planning further attacks.
The attack needs no access to training data or weights. It exploits the fact that every prediction an interface returns is a small piece of information about the model, and enough of those pieces, collected systematically, approximate the original closely. Within artificial intelligence, the threat applies equally to image classifiers, fraud scoring services and large language models offered through paid application programming interfaces.
How a model extraction attack works
A model extraction attack works by turning the victim model into a labeling oracle for the attacker. The adversary chooses inputs, sends them to the prediction interface, records the outputs and then trains a substitute model on the resulting input-output pairs until the substitute agrees with the victim on most inputs.
- Reconnaissance: the attacker learns the task, the input format, the output format and the rate limits of the target service.
- Query selection: inputs are drawn from public data, synthetic generators or active learning strategies that target uncertain regions near the decision boundary.
- Label harvesting: the attacker stores labels, confidence scores, probability vectors, embeddings or generated text for each query.
- Substitute training: a model of similar or smaller capacity is trained to imitate those outputs, often with knowledge distillation on soft labels.
- Fidelity evaluation: agreement with the victim is measured on held-out inputs, and query generation is refined where the copy disagrees.
The richness of the returned output decides how cheap the attack is. A full probability vector reveals far more than a single top label, and for simple models such as logistic regression an exact solution can follow from a number of queries close to the number of parameters. Deep networks cannot usually be recovered exactly, but a high-fidelity functional copy is often achievable with tens of thousands to millions of queries.
Goals and variants of model stealing
Model stealing is not one attack but a family that differs in what the attacker wants to recover. Some attackers need only a model that performs the task well, while others need a copy that reproduces the victim’s exact behavior, including its mistakes, because that copy will later serve as a surrogate for transfer attacks.
The following table compares the main variants by their objective and the access they typically require.
| Variant | Attacker objective | Typical access needed |
|---|---|---|
| Accuracy extraction | Match the victim’s task performance | Top-1 labels and moderate query budget |
| Fidelity extraction | Agree with the victim on each input | Probability scores and large query budget |
| Parameter recovery | Recover exact weights of simple models | Confidence scores and near-linear structure |
| Hyperparameter stealing | Infer regularization or architecture choices | Known training data or side channels |
| Functionality stealing of LLMs | Distill capabilities from generated text | Chat or completion access at scale |
| Embedding extraction | Copy an encoder from returned vectors | Embedding endpoint with raw vectors |
For large language models, extraction usually takes the form of distillation from outputs: the attacker collects many prompts and responses and fine-tunes an open model on them. Research has also shown that some hidden properties, such as the embedding dimension and the final projection layer, can be recovered from logit or log-probability outputs exposed by an interface.
Why extraction is an AI safety problem
Extraction is an AI safety problem because safety controls live around the model, not inside a stolen copy. A provider can filter prompts, rate-limit risky use, log abuse and revoke access, but none of these measures follow a replica that runs on the attacker’s hardware.
- Guardrail removal: a distilled copy can be fine-tuned without the refusal behavior that AI guardrails and model refusal policies enforce.
- Attack rehearsal: a local surrogate allows unlimited, unlogged search for adversarial examples, jailbreak prompts and adversarial suffix attacks that often transfer to the original.
- Privacy leakage: a faithful copy can support membership inference and other attacks that reveal training data.
- Loss of intellectual property: the economic value of training compute and curated data transfers to the attacker at a fraction of the cost.
The neighboring concept of model weight security addresses direct theft of weight files from storage or servers. Model extraction reaches a similar outcome through the public prediction interface alone, which is why both appear in threat models for frontier systems and in the OWASP Top 10 for LLM Applications.
Comparison with related attacks
Model extraction is often confused with other attacks on machine learning systems, yet it differs from them in what is stolen or altered. The table separates it from its closest neighbors.
| Attack | What the attacker gains or changes | Phase of the life cycle |
|---|---|---|
| Model extraction | A functional copy of the model | Deployment and inference |
| Membership inference | Whether a record was in the training set | Deployment and inference |
| Model inversion | Representative reconstructions of training inputs | Deployment and inference |
| Data poisoning | Corrupted behavior through altered training data | Data collection and training |
| Prompt leaking | The hidden system prompt of an application | Deployment of LLM applications |
| Weight exfiltration | The exact weight files | Infrastructure and storage |
The attacks also chain together. A successful extraction frequently becomes the first step of an adversarial example campaign, because examples crafted against the substitute tend to fool the victim as well, a property known as adversarial transferability.
Defenses against model stealing
No single control prevents model stealing, so effective protection combines reducing information per query, detecting abnormal query patterns and proving ownership after theft. Every defense trades some utility or cost for security.
| Defense | How it helps | Main drawback |
|---|---|---|
| Output minimization | Returns labels or rounded scores only | Reduces usefulness for legitimate clients |
| Rate limiting and quotas | Raises the time and cost of harvesting | Attackers spread queries across accounts |
| Query pattern monitoring | Flags unusual input distributions | False positives on unusual but honest users |
| Prediction perturbation | Adds noise that misleads substitute training | May lower accuracy for all users |
| Model watermarking | Embeds verifiable behavior for ownership claims | Does not stop the theft itself |
| Terms of service and logging | Enables legal action and audits | Ineffective against anonymous attackers |
Monitoring approaches such as distribution-based detectors compare the statistics of a client’s queries with those of normal traffic, since active learning and synthetic queries often look unnatural. Model watermarking and fingerprinting do not block extraction but allow the owner to show later that a suspicious model was derived from theirs.
Common misconceptions about model extraction
Several widespread beliefs underestimate the risk of model extraction. The most common one is that returning only labels makes a service safe; label-only attacks need more queries, but they still succeed against many classifiers.
- A copy needs the same architecture: substitutes with a different architecture often reach high agreement.
- The attacker needs in-distribution data: synthetic or loosely related public data is frequently sufficient.
- Large models are too big to steal: full replication is costly, but capability distillation of narrow skills is cheap.
- Watermarks prevent theft: they support attribution after the fact and offer no preventive effect.
In everyday language
A model extraction attack is a way to copy an AI system without ever seeing its inside. The attacker asks it thousands of questions, writes down every answer, and then teaches a new system to give the same answers.
Think of a student who cannot see the answer key but can ask the teacher endless questions and note every reply. Eventually the student can answer almost like the teacher. The key point is simple: every answer an AI gives away leaks a little of how it works, so the owner must limit and watch how it is used.
Seen in the real world
A company offers a paid image moderation service that returns a full probability score for twelve content categories. A competitor opens several trial accounts and, over three weeks, sends about 400,000 images taken from public datasets, staying below each account’s daily quota.
The competitor trains a convolutional network on the returned scores and reaches roughly 95 percent agreement with the original on a test set. The provider’s security team notices that the trial accounts share an unusual mix of image sources and nearly identical request timing. After the incident, the provider returns only the top category with a rounded score to free tiers, adds cross-account anomaly detection and embeds watermark behavior in later model versions, which raises the cost of a repeat attack considerably.