BERT (Bidirectional Encoder Representations from Transformers) is a transformer encoder language model released by Google in 2018 that learns the meaning of each word from both its left and right context at once. Among AI models & assistants, BERT is not a chatbot: it is a pretrained representation model that produces contextual embeddings, which developers fine-tune for classification, search, entity extraction and question answering.
BERT changed natural language processing by making pretrain once, fine-tune many times the standard workflow. Its encoder-only design, masked language modeling objective and openly released weights inspired a large family of descendants, and BERT-style encoders still run inside many search engines, retrieval systems and content classifiers.
How BERT reads text in both directions
BERT works by passing a sequence of WordPiece tokens through a stack of transformer encoder layers in which every token attends to every other token. Unlike left-to-right models such as GPT, each layer sees the full sentence simultaneously, so the representation of an ambiguous word like “bank” depends on words that appear after it as well as before it.
The input starts with a special [CLS] token whose final vector serves as a summary of the whole sequence, and segments are separated by [SEP]. Each token embedding is the sum of a token embedding, a segment embedding and a learned position embedding. The maximum input length is 512 tokens, a limit that later long-context encoders were designed to remove.
The two original sizes are summarized in the table below.
| Model | Layers | Hidden size | Attention heads | Parameters |
|---|---|---|---|---|
| BERT-Base | 12 | 768 | 12 | About 110 million |
| BERT-Large | 24 | 1,024 | 16 | About 340 million |
Both sizes use a vocabulary of about 30,000 WordPiece units. Google also released cased and uncased versions and a multilingual BERT trained on Wikipedia text in more than 100 languages.
Pretraining objectives of BERT
BERT is pretrained on unlabeled text with two self-supervised objectives that teach it general language knowledge before any task-specific training. The original corpus combined BooksCorpus and English Wikipedia, roughly 3.3 billion words.
| Objective | How it works | What it teaches |
|---|---|---|
| Masked language modeling (MLM) | 15% of tokens are selected and the model predicts the originals | Deep bidirectional word context |
| Next sentence prediction (NSP) | Model judges whether sentence B follows sentence A | Relations between sentence pairs |
Of the selected tokens, 80% are replaced with [MASK], 10% with a random token and 10% are left unchanged, which reduces the mismatch between pretraining and fine-tuning, since [MASK] never appears in real input. Later research, notably RoBERTa, showed that next sentence prediction adds little value and that longer training on more data matters far more.
Fine-tuning BERT for language tasks
Fine-tuning adapts the pretrained BERT weights to a specific task by adding a small output layer and training the whole network on labeled examples for a few epochs. Because the model already understands language, a few thousand labeled examples often yield strong results, and training typically takes minutes to hours on a single GPU.
The table below shows how the same encoder is attached to different tasks.
| Task | Output used | Example application |
|---|---|---|
| Sentence classification | [CLS] vector into a softmax layer | Sentiment, spam or topic detection |
| Sentence-pair classification | [CLS] vector over both segments | Natural language inference, duplicate detection |
| Token classification | Every token vector labeled | Named entity recognition, part-of-speech tagging |
| Extractive question answering | Start and end positions of an answer span | Finding answers in documents |
| Embeddings and reranking | Pooled vectors or pair scores | Semantic search, retrieval reranking |
Plain BERT vectors are not ideal sentence embeddings without further training. Approaches such as Sentence-BERT train the encoder with contrastive or siamese objectives so that cosine similarity reflects meaning, which is the basis of many dense retrieval systems.
BERT compared with other language models
BERT is an encoder-only model, which places it in a different role from decoder-only generators and encoder-decoder models. Encoders excel at understanding and scoring text, while decoders are built for generating text token by token.
| Model | Architecture | Main strength | Typical role |
|---|---|---|---|
| BERT | Encoder-only | Bidirectional understanding | Classification, retrieval, extraction |
| RoBERTa | Encoder-only | More robust pretraining without NSP | Higher-accuracy BERT replacement |
| DistilBERT | Smaller distilled encoder | About 40% smaller, faster inference | Edge and low-latency use |
| T5 | Encoder-decoder | Text-to-text generation | Summarization, translation |
| GPT-style models | Decoder-only | Open-ended generation | Chat assistants, writing, coding |
Other well-known descendants include ALBERT, which shares parameters across layers, ELECTRA, which replaces masking with replaced-token detection, and DeBERTa, which separates content and position attention. These models generally outperform the original BERT on standard benchmarks such as GLUE and SQuAD.
Limitations and common misconceptions
BERT remains useful, but practitioners should understand where it falls short. Its main constraints are short context, no native text generation and dated pretraining data.
- The 512-token limit forces long documents to be split into chunks, which can separate related evidence.
- BERT does not generate fluent text, so it is unsuitable as a chat assistant despite sometimes being grouped with them.
- Masked tokens are predicted independently, which ignores dependencies between several masked words.
- Domain shift reduces accuracy on legal, medical or scientific text unless a domain variant such as BioBERT is used.
- Social bias from the training corpus appears in embeddings and downstream classifiers.
- Large generative models are not always better: a fine-tuned encoder is often cheaper, faster and more accurate for narrow classification.
The simple explanation
BERT is a program that has read a very large amount of text and learned how words get their meaning from the words around them. It reads a sentence all at once, looking both backward and forward, much as a person rereads a sentence to understand a tricky word.
BERT does not chat or write essays. Instead, it is a strong reading engine that other software builds on: with a little extra training, it can sort emails, find names in documents or decide which search result best answers a question. The key idea is understanding text, not producing it.
How it is used in practice
An insurance company receives about 20,000 customer emails per week and needs to route each one to claims, billing, policy changes or complaints. The data team fine-tunes a BERT-Base classifier on 15,000 historical emails labeled by support staff, training for three epochs on one GPU in under two hours.
In testing, the model routes emails with about 93% accuracy, compared with roughly 78% for the previous keyword rules, and each prediction takes only a few milliseconds on a standard server. Emails with low confidence scores go to a human queue. The outcome is faster first responses and fewer misrouted complaints, at a running cost far below that of sending every message to a large generative model.