MMLU, short for Massive Multitask Language Understanding, is a benchmark of roughly 16,000 multiple-choice questions across 57 subjects that measures how much academic and professional knowledge a language model can apply without task-specific training. Introduced in 2020 by Dan Hendrycks and colleagues, it became one of the most cited reference points in artificial intelligence for comparing general-purpose models.
In model evaluation, MMLU is valued for its breadth: it spans elementary mathematics, law, medicine, history, computer science, ethics and dozens of other fields at levels from high school to professional. Its simple format, four answer options and accuracy as the metric, made it easy to run and report, which also explains why it is now largely saturated at the frontier.
How the MMLU benchmark is structured
MMLU is organized into 57 tasks grouped into four broad categories: humanities, social sciences, STEM and other professional fields. Each question has exactly four answer options, labeled A to D, and one correct answer. The questions were collected from freely available practice exams, textbooks and study materials written for human learners.
The dataset is divided into a small development split of five questions per subject, used as few-shot examples, a validation split of about 1,500 questions and a test split of about 14,000 questions on which scores are reported. The table summarizes the main characteristics.
| Property | Value | Why it matters |
|---|---|---|
| Subjects | 57 across four categories | Tests breadth rather than one skill |
| Question format | Multiple choice, four options | Simple and automatic scoring |
| Random baseline | 25 percent accuracy | Reference point for meaningful results |
| Test set size | About 14,000 questions | Low sampling noise for aggregate scores |
| Standard setting | Five-shot prompting | Models see five solved examples first |
| Estimated expert level | Roughly 90 percent | Approximate ceiling for human specialists |
How MMLU scores are computed and reported
The headline MMLU score is average accuracy, usually computed as the macro average across the 57 subjects so that each subject counts equally regardless of its size. Some reports use the micro average over all questions instead, which can shift the result by about one percentage point, so the averaging method should always be stated.
The prompting protocol also changes results. In the original setup, a model sees five solved examples from the same subject and must output a letter. Evaluation harnesses differ in whether they compare the log-probabilities of the answer letters, generate free text and parse the answer, or allow chain-of-thought reasoning before answering. The same model can therefore receive scores that differ by several points depending on the harness.
| Setting | How the answer is obtained | Effect on score |
|---|---|---|
| Five-shot, letter likelihood | Highest probability among A to D | Original and most comparable setup |
| Zero-shot | No examples in the prompt | Usually lower for base models |
| Generative with parsing | Model writes an answer that is extracted | Sensitive to formatting failures |
| Chain-of-thought | Model reasons before choosing | Often higher for instruction-tuned models |
Because of these differences, MMLU scores are only comparable when the number of shots, the prompt template, the averaging method and the answer extraction are the same. A careful evaluation report lists all four.
MMLU compared with related benchmarks
MMLU measures broad knowledge recall and applied reasoning at exam level, which complements benchmarks that focus on a single skill. As frontier models approached its ceiling, harder successors were created to restore the ability to separate strong models from one another.
| Benchmark | Focus | Main difference from MMLU |
|---|---|---|
| MMLU | Broad academic and professional knowledge | Reference baseline, four options |
| MMLU-Pro | Harder reasoning-heavy questions | Ten options and fewer trivial items |
| GPQA | Graduate-level science | Written to resist web search |
| HellaSwag | Commonsense sentence completion | Everyday situations, not exam knowledge |
| GSM8K | Grade-school math word problems | Free-form numeric answers |
| Humanity’s Last Exam | Expert frontier questions | Designed to stay unsaturated longer |
Limitations of MMLU
MMLU has well-documented weaknesses that limit how much a single score can say. The most important are benchmark saturation, data contamination and label errors in the original questions.
- Saturation: leading models score close to the estimated expert ceiling, so differences of one or two points at the top are mostly noise and label errors.
- Contamination: the questions have been public since 2020 and appear on many websites, so they may have entered pretraining data, inflating scores.
- Label errors: independent reviews such as MMLU-Redux found a noticeable share of questions with wrong or ambiguous answers, concentrated in certain subjects.
- Format sensitivity: reordering the answer options or changing the prompt template can shift scores by several points, which reveals reliance on surface patterns.
- Narrow skill coverage: multiple-choice recall says little about long-form writing, tool use or instruction following.
These limitations do not make MMLU useless, but they mean it works best as a sanity check of general knowledge rather than as the decisive comparison between frontier systems.
When MMLU is still useful
MMLU remains a practical tool for smaller models, training diagnostics and regression testing. Its subject breakdown also helps identify domain gaps after fine-tuning or quantization.
| Situation | Use MMLU | Reason |
|---|---|---|
| Tracking a pretraining run | Yes, as a progress signal | Scores rise smoothly with capability |
| Checking a quantized model | Yes, for regression | Drops reveal lost knowledge |
| Comparing frontier models | Only with harder benchmarks | Top scores are saturated |
| Evaluating a chat assistant | Not as the main metric | Misses conversation quality and safety |
In everyday language
MMLU is like a giant multiple-choice exam covering 57 school and university subjects, from algebra to law to nutrition. An AI system answers every question, and its score is the percentage it gets right.
Guessing at random would earn about 25 percent, while human experts reach roughly 90 percent. For years, the test was a useful yardstick of general knowledge. Today the strongest systems score so high that it can no longer tell them apart well, so researchers use harder tests alongside it.
Seen in the real world
An engineering team compresses an open-weight language model from 16-bit to 4-bit precision so it can run on a single graphics card. Before shipping, they run MMLU with the same five-shot harness on both versions. The original model scores 68.4 percent and the compressed model 67.9 percent, a difference well within normal variation.
The subject breakdown, however, shows that professional medicine dropped by four points while most other subjects barely changed. Because the product answers health insurance questions, the team tests a different quantization method that keeps sensitive layers at higher precision. The second version recovers the medical score, and the team ships it with a documented evaluation setup so future versions can be compared fairly.