IP-Adapter

On this page 7

IP-Adapter (image prompt adapter) is a lightweight add-on module that lets a pretrained text-to-image diffusion model accept a reference image as a prompt, alongside or instead of a text prompt. In generative AI, it is a standard way to transfer style, composition, subject or facial identity from an example picture without retraining the base model.

The method was introduced in 2023 for Stable Diffusion and works through decoupled cross-attention: image features enter the network through their own attention path, so the original text conditioning stays intact. With roughly 22 million trainable parameters, the adapter is small compared with the base model it controls.

How IP-Adapter injects image features

IP-Adapter conditions generation by adding a second cross-attention branch for image tokens to every cross-attention layer of the denoising U-Net or diffusion transformer. The base weights stay frozen; only the new projection layers are trained.

The reference image is first encoded by a pretrained CLIP image encoder. A small projection network turns that embedding into a short sequence of image tokens, usually 4 tokens for the base version and 16 for the Plus variants. In each layer, the latent queries attend to text tokens and image tokens separately, and the two results are summed with a weight, often called the adapter scale.

The table below lists the parts that make up the adapter and what each contributes.

ComponentFunctionTrained during adapter training
Image encoderExtracts features from the reference imageNo, frozen CLIP model
Projection networkConverts features into image tokensYes
Image key and value layersNew attention path for image tokensYes
Text cross-attentionOriginal path for the text promptNo, frozen
Denoising backbonePredicts noise at each sampling stepNo, frozen
Building blocks of IP-Adapter

Training uses millions of image-text pairs and teaches the adapter to reconstruct the reference image from its own embedding. Text and image conditions are occasionally dropped, which keeps the adapter compatible with classifier-free guidance at inference.

Variants and the adapter scale

IP-Adapter exists in several variants that differ in which features they extract and therefore what they transfer. Choosing the right one matters more than any other setting.

VariantImage features usedBest at transferring
BaseGlobal CLIP embedding, 4 tokensOverall style and mood
PlusCLIP patch features through a resampler, 16 tokensFiner composition and detail
Plus FacePatch features from a cropped faceApproximate facial likeness
FaceIDFace recognition embedding, often with LoRAStronger identity consistency
SDXL versionsLarger encoders matched to SDXLHigher-resolution workflows
Common IP-Adapter variants

The adapter scale sets how strongly the image prompt competes with the text prompt. Its effect is nonlinear and depends on the checkpoint, so the ranges below are starting points rather than fixed rules.

ScaleResultTypical use
0.2-0.4Light influence, text dominatesSubtle color or mood hints
0.5-0.7Balanced image and text controlStyle transfer with new content
0.8-1.0Strong image influenceClose variations of the reference
Above 1.0Often oversaturated or distortedRarely useful
Typical effect of the adapter scale

Many tools also allow layer-selective weighting, applying the adapter only to blocks that govern style or layout. This gives finer control than one global value, for example keeping the reference style while letting the text prompt set the composition.

IP-Adapter compared with other personalization methods

IP-Adapter belongs to a family of techniques for controllable generation, and its main advantage is that it needs no per-subject training. Other methods either fine-tune weights for each concept or control structure rather than appearance.

MethodWhat it controlsTraining per conceptOutput fidelity
IP-AdapterStyle, content, identity from an imageNoneGood, sometimes approximate
ControlNetStructure such as pose, edges, depthNonePrecise spatial layout
DreamBoothA specific subjectMinutes to hours of fine-tuningHigh
Textual inversionA concept as a new token embeddingShort optimizationModerate
Image-to-imageStarting latent from a source imageNoneTied to the original layout
IP-Adapter and related conditioning methods

These methods are complementary rather than competing. A common pipeline combines ControlNet for pose, IP-Adapter for style or face, and a text prompt for scene details. Unlike image-to-image, IP-Adapter does not start from the reference pixels, so the output can have a completely different layout while keeping the visual character.

Practical use and configuration

IP-Adapter is typically used in workflow tools and libraries that load it next to a compatible base checkpoint. The adapter must match the base model family: a Stable Diffusion 1.5 adapter does not work with SDXL, although it usually works with community fine-tunes of the same family.

  • Use square, clean reference images where the key subject fills the frame, because the encoder resizes and center-crops its input.
  • Lower the classifier-free guidance scale slightly when the adapter is active, since both signals push the image strongly.
  • Combine several reference images by averaging their embeddings or using multiple adapters with separate weights.
  • Use a face-specific variant for portraits; general variants capture the look of a face only loosely.

Limitations and common mistakes

IP-Adapter transfers semantic and stylistic features, not exact pixels, so it cannot guarantee fine details such as logos, text or precise clothing patterns. Typical problems and their causes are summarized below.

  • Text prompt ignored: the scale is too high, so the image tokens dominate; reduce the scale toward 0.5.
  • Copied composition: Plus variants transfer layout as well as style; use the base variant or layer-selective weights.
  • Weak likeness: general variants are used for identity; switch to FaceID with a matching LoRA.
  • Distorted results: the adapter does not match the base model or encoder; check versions.

There are also legal and ethical risks. Using a person’s photograph to generate new images raises consent and likeness rights questions, and copying a living artist’s style can conflict with platform rules, so teams should document reference sources and permissions.

Put simply

IP-Adapter lets an image generator look at an example picture and use it as part of the instructions. Instead of describing a style in many words, you show one image, and the generator picks up its colors, mood or the look of a face.

It works like adding a second pair of eyes to the model without retraining its brain. A slider decides how much attention the model pays to the picture compared with the written prompt, so one image can guide many new results.

In practice

A small game studio needs 60 illustrated item cards that all share the painterly style of one approved concept artwork. Training a custom model would take days and a curated dataset, so the art lead uses a Stable Diffusion XL checkpoint with the Plus variant of IP-Adapter and the approved artwork as the reference image.

Each card gets its own text prompt, such as an iron lantern on a wooden table, with the adapter scale at 0.6 and a reduced guidance scale. A ControlNet edge map keeps each item centered. Within one afternoon the team produces several candidates per card, and the final set reads as a single consistent series. An illustrator then spends about 20 minutes per card on corrections, roughly a third of the earlier painting time.