IP-Adapter (image prompt adapter) is a lightweight add-on module that lets a pretrained text-to-image diffusion model accept a reference image as a prompt, alongside or instead of a text prompt. In generative AI, it is a standard way to transfer style, composition, subject or facial identity from an example picture without retraining the base model.
The method was introduced in 2023 for Stable Diffusion and works through decoupled cross-attention: image features enter the network through their own attention path, so the original text conditioning stays intact. With roughly 22 million trainable parameters, the adapter is small compared with the base model it controls.
How IP-Adapter injects image features
IP-Adapter conditions generation by adding a second cross-attention branch for image tokens to every cross-attention layer of the denoising U-Net or diffusion transformer. The base weights stay frozen; only the new projection layers are trained.
The reference image is first encoded by a pretrained CLIP image encoder. A small projection network turns that embedding into a short sequence of image tokens, usually 4 tokens for the base version and 16 for the Plus variants. In each layer, the latent queries attend to text tokens and image tokens separately, and the two results are summed with a weight, often called the adapter scale.
The table below lists the parts that make up the adapter and what each contributes.
| Component | Function | Trained during adapter training |
|---|---|---|
| Image encoder | Extracts features from the reference image | No, frozen CLIP model |
| Projection network | Converts features into image tokens | Yes |
| Image key and value layers | New attention path for image tokens | Yes |
| Text cross-attention | Original path for the text prompt | No, frozen |
| Denoising backbone | Predicts noise at each sampling step | No, frozen |
Training uses millions of image-text pairs and teaches the adapter to reconstruct the reference image from its own embedding. Text and image conditions are occasionally dropped, which keeps the adapter compatible with classifier-free guidance at inference.
Variants and the adapter scale
IP-Adapter exists in several variants that differ in which features they extract and therefore what they transfer. Choosing the right one matters more than any other setting.
| Variant | Image features used | Best at transferring |
|---|---|---|
| Base | Global CLIP embedding, 4 tokens | Overall style and mood |
| Plus | CLIP patch features through a resampler, 16 tokens | Finer composition and detail |
| Plus Face | Patch features from a cropped face | Approximate facial likeness |
| FaceID | Face recognition embedding, often with LoRA | Stronger identity consistency |
| SDXL versions | Larger encoders matched to SDXL | Higher-resolution workflows |
The adapter scale sets how strongly the image prompt competes with the text prompt. Its effect is nonlinear and depends on the checkpoint, so the ranges below are starting points rather than fixed rules.
| Scale | Result | Typical use |
|---|---|---|
| 0.2-0.4 | Light influence, text dominates | Subtle color or mood hints |
| 0.5-0.7 | Balanced image and text control | Style transfer with new content |
| 0.8-1.0 | Strong image influence | Close variations of the reference |
| Above 1.0 | Often oversaturated or distorted | Rarely useful |
Many tools also allow layer-selective weighting, applying the adapter only to blocks that govern style or layout. This gives finer control than one global value, for example keeping the reference style while letting the text prompt set the composition.
IP-Adapter compared with other personalization methods
IP-Adapter belongs to a family of techniques for controllable generation, and its main advantage is that it needs no per-subject training. Other methods either fine-tune weights for each concept or control structure rather than appearance.
| Method | What it controls | Training per concept | Output fidelity |
|---|---|---|---|
| IP-Adapter | Style, content, identity from an image | None | Good, sometimes approximate |
| ControlNet | Structure such as pose, edges, depth | None | Precise spatial layout |
| DreamBooth | A specific subject | Minutes to hours of fine-tuning | High |
| Textual inversion | A concept as a new token embedding | Short optimization | Moderate |
| Image-to-image | Starting latent from a source image | None | Tied to the original layout |
These methods are complementary rather than competing. A common pipeline combines ControlNet for pose, IP-Adapter for style or face, and a text prompt for scene details. Unlike image-to-image, IP-Adapter does not start from the reference pixels, so the output can have a completely different layout while keeping the visual character.
Practical use and configuration
IP-Adapter is typically used in workflow tools and libraries that load it next to a compatible base checkpoint. The adapter must match the base model family: a Stable Diffusion 1.5 adapter does not work with SDXL, although it usually works with community fine-tunes of the same family.
- Use square, clean reference images where the key subject fills the frame, because the encoder resizes and center-crops its input.
- Lower the classifier-free guidance scale slightly when the adapter is active, since both signals push the image strongly.
- Combine several reference images by averaging their embeddings or using multiple adapters with separate weights.
- Use a face-specific variant for portraits; general variants capture the look of a face only loosely.
Limitations and common mistakes
IP-Adapter transfers semantic and stylistic features, not exact pixels, so it cannot guarantee fine details such as logos, text or precise clothing patterns. Typical problems and their causes are summarized below.
- Text prompt ignored: the scale is too high, so the image tokens dominate; reduce the scale toward 0.5.
- Copied composition: Plus variants transfer layout as well as style; use the base variant or layer-selective weights.
- Weak likeness: general variants are used for identity; switch to FaceID with a matching LoRA.
- Distorted results: the adapter does not match the base model or encoder; check versions.
There are also legal and ethical risks. Using a person’s photograph to generate new images raises consent and likeness rights questions, and copying a living artist’s style can conflict with platform rules, so teams should document reference sources and permissions.
Put simply
IP-Adapter lets an image generator look at an example picture and use it as part of the instructions. Instead of describing a style in many words, you show one image, and the generator picks up its colors, mood or the look of a face.
It works like adding a second pair of eyes to the model without retraining its brain. A slider decides how much attention the model pays to the picture compared with the written prompt, so one image can guide many new results.
In practice
A small game studio needs 60 illustrated item cards that all share the painterly style of one approved concept artwork. Training a custom model would take days and a curated dataset, so the art lead uses a Stable Diffusion XL checkpoint with the Plus variant of IP-Adapter and the approved artwork as the reference image.
Each card gets its own text prompt, such as an iron lantern on a wooden table, with the adapter scale at 0.6 and a reduced guidance scale. A ControlNet edge map keeps each item centered. Within one afternoon the team produces several candidates per card, and the final set reads as a single consistent series. An illustrator then spends about 20 minutes per card on corrections, roughly a third of the earlier painting time.