T5

Also known as: Text-to-Text Transfer Transformer

On this page 7

T5 (Text-to-Text Transfer Transformer) is an encoder-decoder Transformer language model family introduced by Google researchers in 2019 that casts every language task as text in, text out. Among AI models & assistants in artificial intelligence, it is best known less as a chat product than as a research and engineering workhorse whose design choices shaped much of later model development.

The accompanying study systematically compared pretraining objectives, architectures, datasets and transfer methods under one framework, and released the Colossal Clean Crawled Corpus (C4) along with checkpoints from 60 million to 11 billion parameters. Its open weights and Apache 2.0 license made T5 a default baseline for summarization, translation, question answering and, later, for the text encoders of image generators.

The text-to-text framework

The defining idea of T5 is that classification, translation, regression and generation share one format: the model reads a string and writes a string. A short task prefix tells the model what to do, so the same weights, loss function and decoding procedure serve every task.

Even labels become words. A sentiment classifier outputs the word “positive” or “negative”, and a similarity score is emitted as a number written in text, rounded to a fixed increment. The table below shows how typical tasks are encoded.

TaskExample inputExample output
Translationtranslate English to German: That is good.Das ist gut.
Summarizationsummarize: followed by a news articleA short summary sentence
Acceptability classificationcola sentence: The course is jumping well.not acceptable
Semantic similaritystsb sentence1: … sentence2: …3.8
Question answeringquestion: … context: …The answer span as text
Examples of the T5 input and output format

This uniformity simplifies multi-task training and transfer, because adding a task requires only new examples, not a new output layer. The trade-off is that the model must learn to emit valid label strings, and it occasionally produces an output that matches no class.

Architecture and pretraining objective

T5 follows the original encoder-decoder Transformer closely, with a few deliberate changes. It uses a simplified layer normalization without bias, places normalization before each sublayer, and replaces absolute position embeddings with learned relative position biases shared across layers.

Pretraining uses span corruption, a denoising objective. About 15 percent of input tokens are removed in contiguous spans averaging three tokens, each span is replaced by a unique sentinel token, and the decoder must reconstruct only the missing spans. This makes targets short and pretraining comparatively efficient.

The original checkpoints were trained on C4, a filtered snapshot of Common Crawl of roughly 750 gigabytes of English text, with a SentencePiece vocabulary of 32,000 tokens. The released sizes are summarized below.

CheckpointApproximate parametersTypical use today
T5-Small60 millionTeaching, quick experiments, edge devices
T5-Base220 millionFine-tuned classifiers and extractors
T5-Large770 millionSummarization and question answering
T5-3B3 billionStronger fine-tuned task models
T5-11B11 billionResearch baselines and text encoders
Original T5 model sizes

The T5 family of variants

T5 spawned a family of derived models that keep the text-to-text interface while changing the data, tokenization or training recipe. Knowing the variants matters, because the name “T5” in a modern system often refers to one of them rather than the 2019 checkpoints.

VariantKey changeTypical reason to choose it
T5 v1.1GeGLU activations, C4-only pretraining, no dropout in pretrainingCleaner starting point for fine-tuning
mT5Pretrained on mC4 covering 101 languagesMultilingual tasks
ByT5Operates directly on UTF-8 bytesNoisy text, spelling, rare scripts
Flan-T5Instruction tuning on a large mixture of tasksZero-shot and few-shot prompting
LongT5Efficient attention for long inputsLong document summarization
T5-XXL encoderEncoder used alone as a frozen text encoderPrompt conditioning in image and video generators
Main T5 variants and what changed

Flan-T5 is especially important in practice: instruction tuning made small checkpoints follow natural-language instructions without task prefixes, often matching much larger untuned models on benchmark suites. Separately, the T5-XXL encoder became a common text conditioner in diffusion models such as Imagen and several later open image generators.

How T5 compares with BERT and GPT-style models

T5 sits between the two other dominant Transformer designs: it pairs a bidirectional encoder like BERT with an autoregressive decoder like GPT. This combination suits tasks where a full input must be understood before a distinct output is generated.

PropertyBERT (encoder-only)T5 (encoder-decoder)GPT-style (decoder-only)
Pretraining objectiveMasked token predictionSpan corruptionNext-token prediction
Generates free textNo, needs a task headYesYes
Attention over inputBidirectionalBidirectional in encoderCausal only
Strongest fitClassification, embeddingsTranslation, summarization, rewritingOpen-ended chat and reasoning
Scaling trend after 2022Mostly small modelsModerate sizesLargest frontier models
T5 compared with encoder-only and decoder-only models

In the T5 study itself, the encoder-decoder with a denoising objective performed best at comparable compute. Later industry practice nevertheless converged on decoder-only models for general assistants, mainly because they scale simply, cache efficiently during generation and handle open-ended dialogue with a single stack.

Practical use, strengths and limitations

Today T5 is chosen mainly for focused, fine-tuned tasks where cost, latency and control matter more than broad conversational ability. A fine-tuned T5-Base or Flan-T5-Large can run on a single modest GPU and often outperforms prompting a far larger general model on a narrow, well-labeled task.

  • Strength: permissive license and many checkpoints in common open-source libraries.
  • Strength: efficient for input-heavy tasks, because the encoder reads the input once.
  • Limitation: the original models used a 512-token training context, so long inputs need chunking or LongT5.
  • Limitation: the original vocabulary handles code, mathematics and non-English text poorly.
  • Limitation: C4 filtering removed some dialects and topics, which introduces dataset bias.
  • Common mistake: fine-tuning with mismatched prefixes or untuned learning rates, which degrades results sharply.

A frequent misconception is that T5 is obsolete. As a chat assistant it has been surpassed, but as a compact, controllable sequence-to-sequence model and as a text encoder for generative pipelines it remains in active production use.

A plain-language summary

T5 is a language model from Google built on one simple rule: every job is turning one piece of text into another. Translating, summarizing, answering a question or judging a review all look the same to it. A short instruction at the start, such as “summarize:”, tells it which job to do.

It works like a careful reader paired with a writer. One half reads the whole input first, and the other half writes the answer. That makes T5 good at rewriting and summarizing, and small versions are cheap enough to train for one specific task.

An example from practice

An insurance company receives about 20,000 claim emails per week and needs each one turned into a short structured summary for the claims handler. Sending every message to a large hosted assistant is expensive and raises data residency concerns, because the emails contain personal information.

The data team fine-tunes Flan-T5-Large on 15,000 historical emails paired with summaries written by experienced handlers, using the input prefix “summarize claim:” and outputs limited to about 80 tokens. The model runs on one on-premises GPU and processes an email in well under a second. After a pilot, handlers review summaries instead of reading full threads, and spot checks show that the summaries capture policy numbers and damage descriptions reliably, while ambiguous cases are flagged for manual reading. The outcome is a cheaper, private and predictable pipeline for one narrow task.