BERT

Also known as: Bidirectional Encoder Representations from Transformers

On this page 7

BERT (Bidirectional Encoder Representations from Transformers) is a transformer encoder language model released by Google in 2018 that learns the meaning of each word from both its left and right context at once. Among AI models & assistants, BERT is not a chatbot: it is a pretrained representation model that produces contextual embeddings, which developers fine-tune for classification, search, entity extraction and question answering.

BERT changed natural language processing by making pretrain once, fine-tune many times the standard workflow. Its encoder-only design, masked language modeling objective and openly released weights inspired a large family of descendants, and BERT-style encoders still run inside many search engines, retrieval systems and content classifiers.

How BERT reads text in both directions

BERT works by passing a sequence of WordPiece tokens through a stack of transformer encoder layers in which every token attends to every other token. Unlike left-to-right models such as GPT, each layer sees the full sentence simultaneously, so the representation of an ambiguous word like “bank” depends on words that appear after it as well as before it.

The input starts with a special [CLS] token whose final vector serves as a summary of the whole sequence, and segments are separated by [SEP]. Each token embedding is the sum of a token embedding, a segment embedding and a learned position embedding. The maximum input length is 512 tokens, a limit that later long-context encoders were designed to remove.

The two original sizes are summarized in the table below.

ModelLayersHidden sizeAttention headsParameters
BERT-Base1276812About 110 million
BERT-Large241,02416About 340 million
Original BERT model sizes

Both sizes use a vocabulary of about 30,000 WordPiece units. Google also released cased and uncased versions and a multilingual BERT trained on Wikipedia text in more than 100 languages.

Pretraining objectives of BERT

BERT is pretrained on unlabeled text with two self-supervised objectives that teach it general language knowledge before any task-specific training. The original corpus combined BooksCorpus and English Wikipedia, roughly 3.3 billion words.

ObjectiveHow it worksWhat it teaches
Masked language modeling (MLM)15% of tokens are selected and the model predicts the originalsDeep bidirectional word context
Next sentence prediction (NSP)Model judges whether sentence B follows sentence ARelations between sentence pairs
BERT pretraining objectives

Of the selected tokens, 80% are replaced with [MASK], 10% with a random token and 10% are left unchanged, which reduces the mismatch between pretraining and fine-tuning, since [MASK] never appears in real input. Later research, notably RoBERTa, showed that next sentence prediction adds little value and that longer training on more data matters far more.

Fine-tuning BERT for language tasks

Fine-tuning adapts the pretrained BERT weights to a specific task by adding a small output layer and training the whole network on labeled examples for a few epochs. Because the model already understands language, a few thousand labeled examples often yield strong results, and training typically takes minutes to hours on a single GPU.

The table below shows how the same encoder is attached to different tasks.

TaskOutput usedExample application
Sentence classification[CLS] vector into a softmax layerSentiment, spam or topic detection
Sentence-pair classification[CLS] vector over both segmentsNatural language inference, duplicate detection
Token classificationEvery token vector labeledNamed entity recognition, part-of-speech tagging
Extractive question answeringStart and end positions of an answer spanFinding answers in documents
Embeddings and rerankingPooled vectors or pair scoresSemantic search, retrieval reranking
Typical BERT task heads

Plain BERT vectors are not ideal sentence embeddings without further training. Approaches such as Sentence-BERT train the encoder with contrastive or siamese objectives so that cosine similarity reflects meaning, which is the basis of many dense retrieval systems.

BERT compared with other language models

BERT is an encoder-only model, which places it in a different role from decoder-only generators and encoder-decoder models. Encoders excel at understanding and scoring text, while decoders are built for generating text token by token.

ModelArchitectureMain strengthTypical role
BERTEncoder-onlyBidirectional understandingClassification, retrieval, extraction
RoBERTaEncoder-onlyMore robust pretraining without NSPHigher-accuracy BERT replacement
DistilBERTSmaller distilled encoderAbout 40% smaller, faster inferenceEdge and low-latency use
T5Encoder-decoderText-to-text generationSummarization, translation
GPT-style modelsDecoder-onlyOpen-ended generationChat assistants, writing, coding
BERT and related model architectures

Other well-known descendants include ALBERT, which shares parameters across layers, ELECTRA, which replaces masking with replaced-token detection, and DeBERTa, which separates content and position attention. These models generally outperform the original BERT on standard benchmarks such as GLUE and SQuAD.

Limitations and common misconceptions

BERT remains useful, but practitioners should understand where it falls short. Its main constraints are short context, no native text generation and dated pretraining data.

  • The 512-token limit forces long documents to be split into chunks, which can separate related evidence.
  • BERT does not generate fluent text, so it is unsuitable as a chat assistant despite sometimes being grouped with them.
  • Masked tokens are predicted independently, which ignores dependencies between several masked words.
  • Domain shift reduces accuracy on legal, medical or scientific text unless a domain variant such as BioBERT is used.
  • Social bias from the training corpus appears in embeddings and downstream classifiers.
  • Large generative models are not always better: a fine-tuned encoder is often cheaper, faster and more accurate for narrow classification.

The simple explanation

BERT is a program that has read a very large amount of text and learned how words get their meaning from the words around them. It reads a sentence all at once, looking both backward and forward, much as a person rereads a sentence to understand a tricky word.

BERT does not chat or write essays. Instead, it is a strong reading engine that other software builds on: with a little extra training, it can sort emails, find names in documents or decide which search result best answers a question. The key idea is understanding text, not producing it.

How it is used in practice

An insurance company receives about 20,000 customer emails per week and needs to route each one to claims, billing, policy changes or complaints. The data team fine-tunes a BERT-Base classifier on 15,000 historical emails labeled by support staff, training for three epochs on one GPU in under two hours.

In testing, the model routes emails with about 93% accuracy, compared with roughly 78% for the previous keyword rules, and each prediction takes only a few milliseconds on a standard server. Emails with low confidence scores go to a human queue. The outcome is faster first responses and fewer misrouted complaints, at a running cost far below that of sending every message to a large generative model.