Humanity’s Last Exam (HLE) is a multimodal benchmark of expert-written, closed-ended questions designed to sit at the frontier of human academic knowledge, so that leading language models score far below expert level. In model evaluation for artificial intelligence, it was created as a response to benchmark saturation, after popular tests such as MMLU stopped separating the strongest systems.
The benchmark was released in early 2025 by the Center for AI Safety and Scale AI. It contains roughly 2,500 public questions across more than a hundred subjects, written and reviewed by about a thousand subject experts, with a private held-out set kept back to detect overfitting.
How Humanity’s Last Exam was built
The defining feature of HLE is adversarial filtering against frontier models. Each candidate question was first tested on several leading language models, and only questions that the models failed to answer correctly moved on to human review. Expert reviewers then checked that each question was precise, unambiguous, solvable and not answerable by a quick web search.
Contributors were academics and specialists, many with doctoral training, and the organizers offered a prize pool for accepted questions to attract difficult, original material. The result is a test where difficulty comes from deep domain knowledge and multi-step reasoning, not from tricks or obscure formatting.
The main pipeline stages and their purpose are summarized below.
| Stage | What happens | Purpose |
|---|---|---|
| Expert submission | Specialists write a question with a verified answer | Source original, graduate-level material |
| Model testing | Frontier models attempt the question | Keep only questions models get wrong |
| First review | Expert reviewers give structured feedback | Fix ambiguity and errors |
| Final approval | Organizers and senior reviewers accept or reject | Hold a consistent quality bar |
| Held-out split | A private subset stays unpublished | Detect training on public questions |
Question format and subject coverage
Every HLE question has a single verifiable answer, which makes automatic scoring possible. About a quarter of the questions are multiple choice, and the rest require an exact-match short answer such as a number, expression or name. Around one in seven questions includes an image, so the test also measures multimodal reasoning over diagrams, tables and photographs.
Mathematics is the largest category by design, because mathematical questions are hard to fake and easy to verify. The approximate shares below come from the published dataset description and may shift slightly between releases.
| Category | Approximate share | Typical question type |
|---|---|---|
| Mathematics | About 40% | Proofs reduced to a final value or expression |
| Biology and medicine | About 10% | Mechanisms, anatomy and experimental reasoning |
| Computer science and AI | About 10% | Algorithms, complexity and code behavior |
| Physics | About 9% | Derivations and quantitative problems |
| Humanities and social science | About 9% | Linguistics, history and classics details |
| Chemistry and engineering | About 10% | Reactions, structures and design calculations |
| Other fields | About 10% | Specialized topics outside the main groups |
How HLE scores are measured and reported
The headline metric is accuracy, the share of questions answered correctly. Because short answers can be written in equivalent forms, grading uses an LLM-as-a-judge that compares the model answer with the reference answer, rather than a strict string comparison.
HLE also reports calibration error. Models are asked to state their confidence, and a well-calibrated model that is right 10% of the time should report about 10% confidence. At launch, leading models showed very high calibration error, meaning they answered wrong questions with strong confidence, which is a direct measure of overconfident hallucination.
Scores moved quickly after release, so the table describes broad phases rather than exact figures.
| Period | Typical frontier accuracy | Main driver |
|---|---|---|
| Launch, early 2025 | Single digits, under 10% | Questions filtered against these models |
| Reasoning models, 2025 | Roughly 10% to 25% | Longer chains of reasoning at inference |
| Agents with tools, late 2025 | Higher, often reported above 25% | Search, code execution and planning |
Comparisons are only fair when the setup is stated: text-only versus full multimodal subset, with or without tools, and the number of attempts. Mixing these conditions is the most common source of misleading leaderboard claims.
Humanity’s Last Exam compared with other benchmarks
HLE belongs to a family of frontier difficulty benchmarks that replaced saturated tests. Its distinguishing traits are its breadth across academic fields and its construction by direct adversarial filtering, while neighbors focus on narrower skills.
| Benchmark | Focus | Answer format |
|---|---|---|
| HLE | Broad expert knowledge across many fields | Exact match or multiple choice |
| MMLU | Undergraduate-level knowledge, 57 subjects | Multiple choice, now saturated |
| GPQA | Graduate science resistant to web search | Multiple choice |
| FrontierMath | Research-level mathematics | Exact numerical or symbolic answer |
| ARC-AGI | Novel abstract pattern reasoning | Grid output |
| SWE-bench | Real software engineering issues | Code patch passing tests |
Limitations and common misreadings
HLE measures closed-ended academic problem solving, not open research, creativity or real-world judgment. A high score does not mean a system can do science; it means it can answer hard questions with known answers. The organizers themselves state that strong HLE results would not alone demonstrate general intelligence.
- Answer key errors: independent audits reported that a portion of the questions, especially in chemistry and biology, may have incorrect or disputed reference answers.
- Selection bias: filtering against specific models makes the set unusually hard for those models, which can exaggerate early gaps.
- Contamination risk: public questions can leak into training data, which is why the private held-out set matters.
- Judge dependence: LLM grading introduces a small but real error margin on borderline answers.
- Rapid saturation: like earlier benchmarks, HLE is expected to lose discriminating power as models improve.
Practitioners should therefore treat HLE as one signal among several, alongside task-specific evaluation, human evaluation and error analysis on their own data.
Explained without jargon
Humanity’s Last Exam is a very hard test for AI systems. Experts from many fields wrote questions that today’s best models could not answer, then checked that each question had one clear, correct answer.
Older tests became too easy because models scored almost perfectly on them. This exam gives researchers a harder ruler to measure progress, and it also checks whether a model knows when it is likely to be wrong.
A typical use case
An AI research team is choosing between two candidate models for a scientific literature assistant. Both score above 90% on MMLU, so that benchmark gives no useful signal. The team runs the text-only HLE subset under identical conditions: no tools, one attempt, and the same judge prompt.
One model scores about 18% with severe overconfidence, while the other scores about 15% but reports much better calibrated confidence. Because the assistant will flag uncertain answers for human review, the team prefers the second model, then confirms the choice with its own domain evaluation set of 400 annotated questions before deployment.