Patch embedding

Also known as: image patch embedding

On this page 7

Patch embedding is the input stage of a vision transformer that cuts an image into fixed-size patches and maps each patch to a learned vector of fixed dimension, turning a grid of pixels into a sequence of tokens. Computer vision models built on transformers rely on it because self-attention operates on sequences, not on raw two-dimensional pixel arrays.

Within artificial intelligence, the term is also written as image patch embedding. It plays the same role for images that word or subword embeddings play for text: it decides what one token represents, how many tokens the model must attend over and therefore how much computation and detail the rest of the network receives.

How patch embedding converts pixels into tokens

Patch embedding works by partitioning the image into non-overlapping squares, flattening each square and applying one shared linear projection to it. A 224 x 224 RGB image with 16 x 16 patches yields a 14 x 14 grid, or 196 patches, each containing 16 x 16 x 3 = 768 raw values.

In practice, the projection is implemented as a convolution whose kernel size equals its stride, for example a 16 x 16 kernel with stride 16 and D output channels. This is mathematically identical to flattening plus a matrix multiplication, but it runs efficiently on standard hardware. The output is a sequence of N vectors of width D, often called the hidden size or embedding dimension.

Two further steps usually complete the input. A learnable class token may be prepended so that its final state summarizes the image for classification, and position embeddings are added because the projection itself discards where each patch came from. Without positional information, the transformer would treat the patches as an unordered set.

The following table traces the tensor shapes through a standard ViT-Base configuration.

StepOperationOutput shape
Input imageResize and normalize224 x 224 x 3
PatchifySplit into 16 x 16 squares196 x 768 raw values
ProjectionShared linear layer or strided convolution196 x 768
Class tokenPrepend one learned vector197 x 768
Position embeddingAdd learned or fixed positions197 x 768
Shape changes in a ViT-Base patch embedding

Patch size, sequence length and compute

Patch size is the most consequential design parameter because it sets the sequence length. Halving the patch side quadruples the number of tokens, and since global self-attention scales with the square of the sequence length, attention cost grows roughly sixteenfold.

Smaller patches preserve fine spatial detail and usually improve accuracy, especially for dense tasks, while larger patches are cheaper but blur small structures. The table below shows typical choices for a 224-pixel input.

Patch sizeNumber of tokensRelative attention costTypical use
32 x 3249LowFast classification, retrieval
16 x 16196ModerateDefault for most vision transformers
14 x 14256HigherLarge image-text encoders
8 x 8784Very highSelf-supervised and dense prediction
Effect of patch size at 224 x 224 resolution

Increasing input resolution has the same effect as shrinking patches. Fine-tuning at 384 pixels with 16-pixel patches produces 576 tokens, and the pretrained position embeddings must be interpolated to the new grid, typically with bicubic resampling.

Variants of patch embedding

Several variants modify the basic linear projection to add local inductive bias or flexibility. Each variant changes how much neighborhood information enters a token before attention begins.

VariantMechanismMain benefit
Linear projectionStrided convolution, no overlapSimple and fast
Overlapping patchesKernel larger than strideSmoother boundaries between tokens
Convolutional stemSeveral small convolutions before tokenizationMore stable optimization
Hierarchical mergingPatches merged between stagesMulti-scale feature maps
Tubelet embedding3D patches spanning several video framesJoint space-time tokens
Flexible patch sizeResized projection weightsOne model, several token budgets
Common patch embedding designs

Hierarchical designs such as the Swin family start with small 4 x 4 patches and merge neighboring tokens stage by stage, producing pyramid features that suit object detection and image segmentation. Video models apply tubelet embedding, where one token covers, for example, 2 frames of 16 x 16 pixels.

Patch embedding differs from convolutional feature extraction in that it tokenizes the image once and leaves spatial reasoning to attention. A convolutional network instead builds features gradually through many overlapping local filters, which encodes translation equivariance directly into the architecture.

ApproachToken unitInductive biasData needs
Patch embeddingFixed square patchWeakLarge datasets or strong augmentation
CNN feature mapCell of a downsampled gridStrong localityModerate
Pixel tokensSingle pixelNoneImpractical above small images
Discrete visual tokensCodebook index from an autoencoderLearnedSeparate tokenizer training
Patch embedding versus other image tokenization methods

The weak inductive bias explains a well-documented finding: plain vision transformers trained from scratch on small datasets often underperform comparable CNNs, yet they match or exceed them after large-scale pretraining. Hybrid models that feed a CNN feature map into the patch embedding sit between the two extremes.

Practical configuration and common mistakes

Configuring patch embedding correctly requires matching patch size, resolution and task. Classification tolerates coarse tokens, while segmentation, OCR and small-object detection benefit from finer tokens or a hierarchical backbone.

SymptomLikely causeFix
Shape error at inputResolution not divisible by patch sizeResize or pad to a multiple
Accuracy drop after upscalingPosition embeddings not interpolatedResample positions to the new grid
Missed small objectsPatches too coarseSmaller patches or multi-scale backbone
Out-of-memory errorsToo many tokens for global attentionLarger patches or windowed attention
Unstable early trainingRaw linear stem with high learning rateWarmup or a convolutional stem
Frequent errors and their remedies
  • Normalize inputs with the same mean and standard deviation used during pretraining.
  • Keep the projection weights frozen only when the downstream images resemble the pretraining domain.
  • Treat token count as a budget: it governs latency more directly than parameter count.

Explained without jargon

A transformer reads lists of items, not pictures. Patch embedding is the step that turns a picture into such a list. It cuts the image into small equal squares, much like slicing a photo into tiles of a mosaic, and describes each tile with a short list of numbers.

Each tile also receives a note saying where it sat in the original picture, so the model can rebuild the layout. Smaller tiles give the model more detail to look at but take far more effort to process, so choosing the tile size is a balance between detail and speed.

A typical use case

A team inspecting printed circuit boards fine-tunes a pretrained vision transformer to flag defective solder joints. The original model uses 16 x 16 patches at 224 pixels, but many defects are only 10 to 20 pixels wide in the factory images, so a single token often mixes a defect with healthy background.

The engineers raise the input resolution to 448 pixels, which increases the sequence from 196 to 784 tokens, and interpolate the position embeddings to the larger grid. To keep inference within the line speed of about 40 boards per minute, they switch to a backbone with windowed attention.

After fine-tuning on several thousand labeled board images, the model detects noticeably more small defects than the original configuration, and the finer token grid also produces clearer attention maps that help operators verify each flagged joint.