MMLU

Also known as: Massive Multitask Language Understanding

On this page 7

MMLU, short for Massive Multitask Language Understanding, is a benchmark of roughly 16,000 multiple-choice questions across 57 subjects that measures how much academic and professional knowledge a language model can apply without task-specific training. Introduced in 2020 by Dan Hendrycks and colleagues, it became one of the most cited reference points in artificial intelligence for comparing general-purpose models.

In model evaluation, MMLU is valued for its breadth: it spans elementary mathematics, law, medicine, history, computer science, ethics and dozens of other fields at levels from high school to professional. Its simple format, four answer options and accuracy as the metric, made it easy to run and report, which also explains why it is now largely saturated at the frontier.

How the MMLU benchmark is structured

MMLU is organized into 57 tasks grouped into four broad categories: humanities, social sciences, STEM and other professional fields. Each question has exactly four answer options, labeled A to D, and one correct answer. The questions were collected from freely available practice exams, textbooks and study materials written for human learners.

The dataset is divided into a small development split of five questions per subject, used as few-shot examples, a validation split of about 1,500 questions and a test split of about 14,000 questions on which scores are reported. The table summarizes the main characteristics.

PropertyValueWhy it matters
Subjects57 across four categoriesTests breadth rather than one skill
Question formatMultiple choice, four optionsSimple and automatic scoring
Random baseline25 percent accuracyReference point for meaningful results
Test set sizeAbout 14,000 questionsLow sampling noise for aggregate scores
Standard settingFive-shot promptingModels see five solved examples first
Estimated expert levelRoughly 90 percentApproximate ceiling for human specialists
Key characteristics of MMLU

How MMLU scores are computed and reported

The headline MMLU score is average accuracy, usually computed as the macro average across the 57 subjects so that each subject counts equally regardless of its size. Some reports use the micro average over all questions instead, which can shift the result by about one percentage point, so the averaging method should always be stated.

The prompting protocol also changes results. In the original setup, a model sees five solved examples from the same subject and must output a letter. Evaluation harnesses differ in whether they compare the log-probabilities of the answer letters, generate free text and parse the answer, or allow chain-of-thought reasoning before answering. The same model can therefore receive scores that differ by several points depending on the harness.

SettingHow the answer is obtainedEffect on score
Five-shot, letter likelihoodHighest probability among A to DOriginal and most comparable setup
Zero-shotNo examples in the promptUsually lower for base models
Generative with parsingModel writes an answer that is extractedSensitive to formatting failures
Chain-of-thoughtModel reasons before choosingOften higher for instruction-tuned models
Common MMLU evaluation settings

Because of these differences, MMLU scores are only comparable when the number of shots, the prompt template, the averaging method and the answer extraction are the same. A careful evaluation report lists all four.

MMLU measures broad knowledge recall and applied reasoning at exam level, which complements benchmarks that focus on a single skill. As frontier models approached its ceiling, harder successors were created to restore the ability to separate strong models from one another.

BenchmarkFocusMain difference from MMLU
MMLUBroad academic and professional knowledgeReference baseline, four options
MMLU-ProHarder reasoning-heavy questionsTen options and fewer trivial items
GPQAGraduate-level scienceWritten to resist web search
HellaSwagCommonsense sentence completionEveryday situations, not exam knowledge
GSM8KGrade-school math word problemsFree-form numeric answers
Humanity’s Last ExamExpert frontier questionsDesigned to stay unsaturated longer
How MMLU relates to neighboring benchmarks

Limitations of MMLU

MMLU has well-documented weaknesses that limit how much a single score can say. The most important are benchmark saturation, data contamination and label errors in the original questions.

  • Saturation: leading models score close to the estimated expert ceiling, so differences of one or two points at the top are mostly noise and label errors.
  • Contamination: the questions have been public since 2020 and appear on many websites, so they may have entered pretraining data, inflating scores.
  • Label errors: independent reviews such as MMLU-Redux found a noticeable share of questions with wrong or ambiguous answers, concentrated in certain subjects.
  • Format sensitivity: reordering the answer options or changing the prompt template can shift scores by several points, which reveals reliance on surface patterns.
  • Narrow skill coverage: multiple-choice recall says little about long-form writing, tool use or instruction following.

These limitations do not make MMLU useless, but they mean it works best as a sanity check of general knowledge rather than as the decisive comparison between frontier systems.

When MMLU is still useful

MMLU remains a practical tool for smaller models, training diagnostics and regression testing. Its subject breakdown also helps identify domain gaps after fine-tuning or quantization.

SituationUse MMLUReason
Tracking a pretraining runYes, as a progress signalScores rise smoothly with capability
Checking a quantized modelYes, for regressionDrops reveal lost knowledge
Comparing frontier modelsOnly with harder benchmarksTop scores are saturated
Evaluating a chat assistantNot as the main metricMisses conversation quality and safety
Decision guide for using MMLU

In everyday language

MMLU is like a giant multiple-choice exam covering 57 school and university subjects, from algebra to law to nutrition. An AI system answers every question, and its score is the percentage it gets right.

Guessing at random would earn about 25 percent, while human experts reach roughly 90 percent. For years, the test was a useful yardstick of general knowledge. Today the strongest systems score so high that it can no longer tell them apart well, so researchers use harder tests alongside it.

Seen in the real world

An engineering team compresses an open-weight language model from 16-bit to 4-bit precision so it can run on a single graphics card. Before shipping, they run MMLU with the same five-shot harness on both versions. The original model scores 68.4 percent and the compressed model 67.9 percent, a difference well within normal variation.

The subject breakdown, however, shows that professional medicine dropped by four points while most other subjects barely changed. Because the product answers health insurance questions, the team tests a different quantization method that keeps sensitive layers at higher precision. The second version recovers the medical score, and the team ships it with a documented evaluation setup so future versions can be compared fairly.