Model extraction attack

Also known as: model stealing

On this page 8

A model extraction attack, also called model stealing, is an attack in which an adversary repeatedly queries a deployed machine learning model and uses the returned outputs to reconstruct a functional copy of that model, its parameters or its decision boundary. In AI safety, it matters because a stolen copy removes the owner’s control over access, monitoring and safety restrictions, and it gives the attacker a private sandbox for planning further attacks.

The attack needs no access to training data or weights. It exploits the fact that every prediction an interface returns is a small piece of information about the model, and enough of those pieces, collected systematically, approximate the original closely. Within artificial intelligence, the threat applies equally to image classifiers, fraud scoring services and large language models offered through paid application programming interfaces.

How a model extraction attack works

A model extraction attack works by turning the victim model into a labeling oracle for the attacker. The adversary chooses inputs, sends them to the prediction interface, records the outputs and then trains a substitute model on the resulting input-output pairs until the substitute agrees with the victim on most inputs.

  1. Reconnaissance: the attacker learns the task, the input format, the output format and the rate limits of the target service.
  2. Query selection: inputs are drawn from public data, synthetic generators or active learning strategies that target uncertain regions near the decision boundary.
  3. Label harvesting: the attacker stores labels, confidence scores, probability vectors, embeddings or generated text for each query.
  4. Substitute training: a model of similar or smaller capacity is trained to imitate those outputs, often with knowledge distillation on soft labels.
  5. Fidelity evaluation: agreement with the victim is measured on held-out inputs, and query generation is refined where the copy disagrees.

The richness of the returned output decides how cheap the attack is. A full probability vector reveals far more than a single top label, and for simple models such as logistic regression an exact solution can follow from a number of queries close to the number of parameters. Deep networks cannot usually be recovered exactly, but a high-fidelity functional copy is often achievable with tens of thousands to millions of queries.

Goals and variants of model stealing

Model stealing is not one attack but a family that differs in what the attacker wants to recover. Some attackers need only a model that performs the task well, while others need a copy that reproduces the victim’s exact behavior, including its mistakes, because that copy will later serve as a surrogate for transfer attacks.

The following table compares the main variants by their objective and the access they typically require.

VariantAttacker objectiveTypical access needed
Accuracy extractionMatch the victim’s task performanceTop-1 labels and moderate query budget
Fidelity extractionAgree with the victim on each inputProbability scores and large query budget
Parameter recoveryRecover exact weights of simple modelsConfidence scores and near-linear structure
Hyperparameter stealingInfer regularization or architecture choicesKnown training data or side channels
Functionality stealing of LLMsDistill capabilities from generated textChat or completion access at scale
Embedding extractionCopy an encoder from returned vectorsEmbedding endpoint with raw vectors
Main variants of model extraction

For large language models, extraction usually takes the form of distillation from outputs: the attacker collects many prompts and responses and fine-tunes an open model on them. Research has also shown that some hidden properties, such as the embedding dimension and the final projection layer, can be recovered from logit or log-probability outputs exposed by an interface.

Why extraction is an AI safety problem

Extraction is an AI safety problem because safety controls live around the model, not inside a stolen copy. A provider can filter prompts, rate-limit risky use, log abuse and revoke access, but none of these measures follow a replica that runs on the attacker’s hardware.

  • Guardrail removal: a distilled copy can be fine-tuned without the refusal behavior that AI guardrails and model refusal policies enforce.
  • Attack rehearsal: a local surrogate allows unlimited, unlogged search for adversarial examples, jailbreak prompts and adversarial suffix attacks that often transfer to the original.
  • Privacy leakage: a faithful copy can support membership inference and other attacks that reveal training data.
  • Loss of intellectual property: the economic value of training compute and curated data transfers to the attacker at a fraction of the cost.

The neighboring concept of model weight security addresses direct theft of weight files from storage or servers. Model extraction reaches a similar outcome through the public prediction interface alone, which is why both appear in threat models for frontier systems and in the OWASP Top 10 for LLM Applications.

Model extraction is often confused with other attacks on machine learning systems, yet it differs from them in what is stolen or altered. The table separates it from its closest neighbors.

AttackWhat the attacker gains or changesPhase of the life cycle
Model extractionA functional copy of the modelDeployment and inference
Membership inferenceWhether a record was in the training setDeployment and inference
Model inversionRepresentative reconstructions of training inputsDeployment and inference
Data poisoningCorrupted behavior through altered training dataData collection and training
Prompt leakingThe hidden system prompt of an applicationDeployment of LLM applications
Weight exfiltrationThe exact weight filesInfrastructure and storage
Model extraction compared with related attacks

The attacks also chain together. A successful extraction frequently becomes the first step of an adversarial example campaign, because examples crafted against the substitute tend to fool the victim as well, a property known as adversarial transferability.

Defenses against model stealing

No single control prevents model stealing, so effective protection combines reducing information per query, detecting abnormal query patterns and proving ownership after theft. Every defense trades some utility or cost for security.

DefenseHow it helpsMain drawback
Output minimizationReturns labels or rounded scores onlyReduces usefulness for legitimate clients
Rate limiting and quotasRaises the time and cost of harvestingAttackers spread queries across accounts
Query pattern monitoringFlags unusual input distributionsFalse positives on unusual but honest users
Prediction perturbationAdds noise that misleads substitute trainingMay lower accuracy for all users
Model watermarkingEmbeds verifiable behavior for ownership claimsDoes not stop the theft itself
Terms of service and loggingEnables legal action and auditsIneffective against anonymous attackers
Common defenses and their trade-offs

Monitoring approaches such as distribution-based detectors compare the statistics of a client’s queries with those of normal traffic, since active learning and synthetic queries often look unnatural. Model watermarking and fingerprinting do not block extraction but allow the owner to show later that a suspicious model was derived from theirs.

Common misconceptions about model extraction

Several widespread beliefs underestimate the risk of model extraction. The most common one is that returning only labels makes a service safe; label-only attacks need more queries, but they still succeed against many classifiers.

  • A copy needs the same architecture: substitutes with a different architecture often reach high agreement.
  • The attacker needs in-distribution data: synthetic or loosely related public data is frequently sufficient.
  • Large models are too big to steal: full replication is costly, but capability distillation of narrow skills is cheap.
  • Watermarks prevent theft: they support attribution after the fact and offer no preventive effect.

In everyday language

A model extraction attack is a way to copy an AI system without ever seeing its inside. The attacker asks it thousands of questions, writes down every answer, and then teaches a new system to give the same answers.

Think of a student who cannot see the answer key but can ask the teacher endless questions and note every reply. Eventually the student can answer almost like the teacher. The key point is simple: every answer an AI gives away leaks a little of how it works, so the owner must limit and watch how it is used.

Seen in the real world

A company offers a paid image moderation service that returns a full probability score for twelve content categories. A competitor opens several trial accounts and, over three weeks, sends about 400,000 images taken from public datasets, staying below each account’s daily quota.

The competitor trains a convolutional network on the returned scores and reaches roughly 95 percent agreement with the original on a test set. The provider’s security team notices that the trial accounts share an unusual mix of image sources and nearly identical request timing. After the incident, the provider returns only the top category with a rounded score to free tiers, adds cross-account anomaly detection and embeds watermark behavior in later model versions, which raises the cost of a repeat attack considerably.