A vision transformer (ViT) is a neural network for computer vision that divides an image into fixed-size patches, turns each patch into a vector token and processes the sequence of tokens with the self-attention layers of a transformer, the same architecture used for language models. Introduced in 2020, it showed that convolutions are not required for state-of-the-art image recognition when enough training data is available.
The vision transformer changed the field of artificial intelligence for images in two ways. It became a strong backbone for classification, detection and segmentation, and it made images and text share one architecture, which underlies today’s multimodal models that read documents, charts and photographs alongside text.
How a vision transformer processes an image
A vision transformer converts pixels into a token sequence in a few fixed steps. A 224 x 224 image split into 16 x 16 patches yields 196 patches. Each patch is flattened and multiplied by a learned matrix, producing a patch embedding of typically 768 or 1,024 dimensions.
Because self-attention has no built-in notion of order or position, the model adds a position embedding to every patch token. Many variants also prepend a learnable class token whose final state summarizes the whole image. The sequence then passes through a stack of identical encoder blocks, each containing multi-head self-attention, a feed-forward network, layer normalization and residual connections.
| Step | Operation | Result |
|---|---|---|
| Patching | Split image into non-overlapping squares | Sequence of N patches |
| Linear projection | Map each flattened patch to a vector | Patch embeddings |
| Position encoding | Add learned or fixed position vectors | Tokens that carry location |
| Transformer encoder | Repeated self-attention and MLP layers | Context-aware token features |
| Task head | Classifier, detector or decoder on top | Labels, boxes or masks |
Self-attention lets every patch attend to every other patch from the first layer. A convolutional network sees only a small neighborhood at first and widens its view layer by layer, whereas a vision transformer can relate distant regions immediately, for example a face and a hand at opposite corners of a frame.
Model sizes and key parameters
Vision transformers are described by depth, width, number of attention heads and patch size. Names such as ViT-B/16 mean the Base configuration with 16-pixel patches. The configurations below are the widely used standard sizes.
| Model | Layers | Hidden size | Attention heads | Parameters |
|---|---|---|---|---|
| ViT-Small | 12 | 384 | 6 | About 22 million |
| ViT-Base | 12 | 768 | 12 | About 86 million |
| ViT-Large | 24 | 1,024 | 16 | About 307 million |
| ViT-Huge | 32 | 1,280 | 16 | About 632 million |
Patch size is the decisive cost parameter. Halving it from 16 to 8 pixels quadruples the number of tokens, and because standard attention scales with the square of the sequence length, attention cost rises roughly sixteen-fold. Smaller patches preserve fine detail, which helps with text in images and small objects, but they are expensive at high resolution.
Vision transformers compared with convolutional networks
The central difference is inductive bias. Convolutional neural networks (CNNs) assume that nearby pixels belong together and that patterns look the same anywhere in the image. Vision transformers make almost no such assumptions and must learn spatial structure from data, which is why the original ViT needed pretraining on hundreds of millions of images to beat strong CNNs.
| Property | Vision transformer | Convolutional network |
|---|---|---|
| Receptive field | Global from the first layer | Local, grows with depth |
| Inductive bias | Weak | Strong locality and translation equivariance |
| Data needs | High without strong augmentation | Moderate |
| Scaling with data | Continues to improve at large scale | Gains flatten earlier |
| Cost at high resolution | Quadratic in token count | Linear in pixel count |
| Multimodal fit | Natural match for language models | Needs adaptation |
The gap has narrowed from both sides. Training recipes such as DeiT made vision transformers competitive on ImageNet-1k alone through distillation and heavy augmentation, while modern CNNs such as ConvNeXt adopted transformer design choices. In practice, data scale, pretraining and compute budget usually matter more than the choice between the two families.
Variants and typical applications
Several variants address the cost of global attention or adapt the architecture to dense tasks such as object detection and image segmentation, which need features at multiple scales.
| Variant | Key idea | Typical use |
|---|---|---|
| Swin Transformer | Attention within shifted local windows | Detection and segmentation backbones |
| DeiT | Data-efficient training with distillation | Training on moderate datasets |
| DINO and DINOv2 | Self-supervised pretraining | General-purpose visual features |
| MAE | Reconstruct masked patches | Efficient pretraining without labels |
| CLIP image encoder | Contrastive training with text | Zero-shot classification, multimodal models |
| ViTDet and SAM encoders | Plain ViT for dense prediction | Detection and promptable segmentation |
Vision transformers also serve as the image encoders in OCR-free document understanding and in vision-language models, and they are used for keypoint tasks such as pose estimation. In each case, the patch tokens are passed to a task-specific head or to a language model.
Practical choices and common pitfalls
- Training from scratch on small data. Without large-scale pretraining, a vision transformer usually underperforms a CNN; fine-tune a pretrained checkpoint instead.
- Changing input resolution carelessly. Position embeddings must be interpolated when the resolution differs from pretraining.
- Underestimating memory. High-resolution inputs with small patches can exhaust GPU memory quickly.
- Ignoring attention artifacts. Large ViTs can produce high-norm tokens in background regions; register tokens are one known remedy.
- Assuming robustness. Vision transformers remain vulnerable to distribution shift and adversarial patches.
For most projects the sensible path is to start from a pretrained ViT-B or ViT-S checkpoint, fine-tune at the pretraining resolution, and move to larger models or windowed variants only when accuracy or input size demands it.
In simple terms
Imagine cutting a photograph into a grid of small square tiles and laying them out in a row. A vision transformer looks at all the tiles together and asks, for each tile, which other tiles help explain it.
A tile showing part of a wheel becomes meaningful once the model notices other tiles showing a door and a windshield. By comparing every tile with every other tile, the model recognizes the whole scene, much as a language model understands a sentence by relating its words.
A practical example
A regional agricultural cooperative wants to detect crop diseases from smartphone photos of leaves. It has only 12,000 labeled images across 15 disease classes, too few to train a large model from scratch.
The team fine-tunes a pretrained ViT-B/16 with self-supervised weights at 224 x 224 pixels, using standard augmentation such as random crops and color jitter. After a few hours of training on one GPU, the model clearly outperforms the cooperative’s earlier small CNN on a held-out test set, especially for diseases whose spots are scattered across the whole leaf, where global attention helps. To run the model on low-end phones, the team later distills it into a ViT-S student, accepting a small drop in accuracy for much faster inference.