Style transfer

Also known as: neural style transfer

On this page 7

Style transfer is a family of generative AI techniques that re-renders the content of one image in the visual style of another, for example a city photograph painted with the brushwork and palette of an impressionist canvas. In generative AI, the term covers both the original neural style transfer method based on optimizing an image against a pretrained network and later approaches built on feed-forward networks, GANs and diffusion models.

The central idea is that a trained convolutional network represents content and style separately: deep layers capture what objects are where, while correlations between feature channels capture textures, colors and strokes. Every style transfer method, old or new, is a way of keeping the first while replacing the second.

How neural style transfer separates content and style

Classic neural style transfer, introduced in 2015, starts from a pretrained image classifier, usually VGG-19, and treats image generation as an optimization problem. The output image is adjusted pixel by pixel until its internal network activations match the content image in some layers and the style image in others.

Content is measured by comparing feature maps of a deep layer, which encode object layout but little texture. Style is measured with Gram matrices, which record how strongly each pair of feature channels activates together across the whole image. Because the Gram matrix discards spatial position, it captures recurring textures and color combinations without copying the layout of the style image.

The total objective is a weighted sum of losses, and the weights decide the look of the result. The following table lists the main components.

Loss termWhat it comparesEffect of a higher weight
Content lossDeep-layer feature mapsSharper preservation of shapes and layout
Style lossGram matrices across several layersStronger textures, colors and strokes
Total variation lossDifferences between neighboring pixelsSmoother image with less high-frequency noise
Loss terms in classic neural style transfer

Optimization-based transfer gives high quality for any pair of images, but it is slow: each output needs hundreds of iterations, typically seconds to minutes on a GPU depending on resolution.

Main families of style transfer methods

Later research addressed the speed problem and broadened what counts as style. Methods now differ mainly in how many styles they handle, how fast they run and how much control they give. The table below compares the principal families.

Method familyHow it worksSpeedStyle flexibility
Optimization-basedIteratively updates pixels against a pretrained networkSlowAny style image
Feed-forward per styleTrains one network to apply one fixed styleReal timeOne style per model
Arbitrary style transferAligns feature statistics, for example with AdaINReal timeAny style image
GAN-based translationLearns mappings between image domains, as in CycleGANReal time after trainingLearned domain, not single image
Diffusion-based transferGuides denoising with style images or textSeconds per imageImages, text prompts or adapters
Families of style transfer methods

Adaptive instance normalization, known as AdaIN, was an important step: it shifts the mean and variance of content features to match those of the style features, which allows arbitrary styles in a single forward pass. GAN approaches such as CycleGAN learn a broader domain style, for example summer to winter or photo to Monet, from unpaired collections rather than from one reference image.

In current diffusion pipelines, style transfer is usually achieved through image-to-image generation with a controlled denoising strength, through style adapters such as IP-Adapter, or through fine-tuning methods such as DreamBooth and LoRA. ControlNet is often added to lock the composition with edge or depth maps while the style changes freely.

Style transfer overlaps with several neighboring techniques, and the distinction lies in what is held fixed. In style transfer, the content structure of a source image is the fixed element and only appearance changes.

TechniqueInputWhat stays fixedWhat changes
Style transferContent image and style referenceLayout and objectsTexture, color and stroke
Image-to-imageImage and text promptPartly, depending on strengthContent and appearance
Text-to-imageText prompt onlyNothing from an imageEverything is generated
InpaintingImage and maskUnmasked regionsMasked region only
Color transferTwo imagesStructure and textureGlobal color distribution only
Style transfer and neighboring techniques

A practical consequence is that text-to-image models with a style phrase in the prompt are not true style transfer, because they do not preserve a given content image. Teams that need faithful structure, such as a specific product or building, should use a method that conditions directly on the source image.

Key parameters and practical control

Most style transfer tools expose a small set of controls that determine the balance between recognizability and stylization. Understanding them avoids long trial and error.

  • Style strength or content-style ratio: the main dial between faithful content and heavy stylization.
  • Denoising strength in diffusion pipelines: values around 0.3 to 0.5 keep structure, while values above about 0.7 begin to replace content.
  • Style scale: the size of the style image relative to the output controls the scale of brushstrokes and textures.
  • Color preservation: transferring style on luminance only keeps the original colors of the content image.
  • Spatial masks: applying different styles to foreground and background, or excluding faces, gives more natural results.

Style transfer has well-known weaknesses. Classic methods tend to distort fine structures, such as faces and text, and produce repetitive texture patterns. Video style transfer adds temporal flicker, because each frame is stylized independently unless optical flow or a temporal consistency loss links neighboring frames.

A common misconception is that the network understands artistic intent. In reality, Gram-matrix style captures local texture statistics, not composition, symbolism or the deliberate choices of an artist, so results can look like a filter rather than a painting. Diffusion methods capture higher-level style better but may alter content unintentionally at high strength.

Imitating the recognizable style of a living artist raises unresolved ethical and legal questions. Style itself is generally not protected by copyright in many jurisdictions, but training data, reference images and output resemblance can create liability, and the situation is still disputed and evolving. Professional users should rely on licensed or own reference material.

Understanding it simply

Style transfer lets a computer repaint a picture in the look of another picture. Imagine giving a skilled copyist a holiday photo and a painting, and asking them to keep everything in the photo where it is but paint it with the colors and brushstrokes of the painting.

The AI does something similar by separating what is in an image from how it looks. The content stays, the appearance changes. The result works best with clear shapes and bold styles, and it can struggle with faces, small text and anything that needs real artistic judgment.

Applied in practice

A small animation studio is producing a 90-second explainer video and wants it to look like hand-painted watercolor without painting 2,160 frames by hand. The artists paint six reference frames in watercolor and render the rest of the animation as simple flat-shaded 3D scenes.

They run a diffusion pipeline in which each rendered frame is the content input, the painted references guide the style through an adapter, and ControlNet edge maps lock the outlines of characters. Denoising strength stays at about 0.4 to keep the composition, and a temporal consistency step uses optical flow from the 3D renderer to reduce flicker between frames.

The result looks consistently painted, and the team cuts the painting workload to a few days of reference work and cleanup. Close-ups of faces still need manual retouching on about one frame in twenty, which the studio plans for in its schedule.