Humanity’s Last Exam

Also known as: HLE

On this page 7

Humanity’s Last Exam (HLE) is a multimodal benchmark of expert-written, closed-ended questions designed to sit at the frontier of human academic knowledge, so that leading language models score far below expert level. In model evaluation for artificial intelligence, it was created as a response to benchmark saturation, after popular tests such as MMLU stopped separating the strongest systems.

The benchmark was released in early 2025 by the Center for AI Safety and Scale AI. It contains roughly 2,500 public questions across more than a hundred subjects, written and reviewed by about a thousand subject experts, with a private held-out set kept back to detect overfitting.

How Humanity’s Last Exam was built

The defining feature of HLE is adversarial filtering against frontier models. Each candidate question was first tested on several leading language models, and only questions that the models failed to answer correctly moved on to human review. Expert reviewers then checked that each question was precise, unambiguous, solvable and not answerable by a quick web search.

Contributors were academics and specialists, many with doctoral training, and the organizers offered a prize pool for accepted questions to attract difficult, original material. The result is a test where difficulty comes from deep domain knowledge and multi-step reasoning, not from tricks or obscure formatting.

The main pipeline stages and their purpose are summarized below.

StageWhat happensPurpose
Expert submissionSpecialists write a question with a verified answerSource original, graduate-level material
Model testingFrontier models attempt the questionKeep only questions models get wrong
First reviewExpert reviewers give structured feedbackFix ambiguity and errors
Final approvalOrganizers and senior reviewers accept or rejectHold a consistent quality bar
Held-out splitA private subset stays unpublishedDetect training on public questions
Question creation pipeline

Question format and subject coverage

Every HLE question has a single verifiable answer, which makes automatic scoring possible. About a quarter of the questions are multiple choice, and the rest require an exact-match short answer such as a number, expression or name. Around one in seven questions includes an image, so the test also measures multimodal reasoning over diagrams, tables and photographs.

Mathematics is the largest category by design, because mathematical questions are hard to fake and easy to verify. The approximate shares below come from the published dataset description and may shift slightly between releases.

CategoryApproximate shareTypical question type
MathematicsAbout 40%Proofs reduced to a final value or expression
Biology and medicineAbout 10%Mechanisms, anatomy and experimental reasoning
Computer science and AIAbout 10%Algorithms, complexity and code behavior
PhysicsAbout 9%Derivations and quantitative problems
Humanities and social scienceAbout 9%Linguistics, history and classics details
Chemistry and engineeringAbout 10%Reactions, structures and design calculations
Other fieldsAbout 10%Specialized topics outside the main groups
Approximate composition of the public question set

How HLE scores are measured and reported

The headline metric is accuracy, the share of questions answered correctly. Because short answers can be written in equivalent forms, grading uses an LLM-as-a-judge that compares the model answer with the reference answer, rather than a strict string comparison.

HLE also reports calibration error. Models are asked to state their confidence, and a well-calibrated model that is right 10% of the time should report about 10% confidence. At launch, leading models showed very high calibration error, meaning they answered wrong questions with strong confidence, which is a direct measure of overconfident hallucination.

Scores moved quickly after release, so the table describes broad phases rather than exact figures.

PeriodTypical frontier accuracyMain driver
Launch, early 2025Single digits, under 10%Questions filtered against these models
Reasoning models, 2025Roughly 10% to 25%Longer chains of reasoning at inference
Agents with tools, late 2025Higher, often reported above 25%Search, code execution and planning
Broad trajectory of reported frontier scores

Comparisons are only fair when the setup is stated: text-only versus full multimodal subset, with or without tools, and the number of attempts. Mixing these conditions is the most common source of misleading leaderboard claims.

Humanity’s Last Exam compared with other benchmarks

HLE belongs to a family of frontier difficulty benchmarks that replaced saturated tests. Its distinguishing traits are its breadth across academic fields and its construction by direct adversarial filtering, while neighbors focus on narrower skills.

BenchmarkFocusAnswer format
HLEBroad expert knowledge across many fieldsExact match or multiple choice
MMLUUndergraduate-level knowledge, 57 subjectsMultiple choice, now saturated
GPQAGraduate science resistant to web searchMultiple choice
FrontierMathResearch-level mathematicsExact numerical or symbolic answer
ARC-AGINovel abstract pattern reasoningGrid output
SWE-benchReal software engineering issuesCode patch passing tests
HLE and related evaluation benchmarks

Limitations and common misreadings

HLE measures closed-ended academic problem solving, not open research, creativity or real-world judgment. A high score does not mean a system can do science; it means it can answer hard questions with known answers. The organizers themselves state that strong HLE results would not alone demonstrate general intelligence.

  • Answer key errors: independent audits reported that a portion of the questions, especially in chemistry and biology, may have incorrect or disputed reference answers.
  • Selection bias: filtering against specific models makes the set unusually hard for those models, which can exaggerate early gaps.
  • Contamination risk: public questions can leak into training data, which is why the private held-out set matters.
  • Judge dependence: LLM grading introduces a small but real error margin on borderline answers.
  • Rapid saturation: like earlier benchmarks, HLE is expected to lose discriminating power as models improve.

Practitioners should therefore treat HLE as one signal among several, alongside task-specific evaluation, human evaluation and error analysis on their own data.

Explained without jargon

Humanity’s Last Exam is a very hard test for AI systems. Experts from many fields wrote questions that today’s best models could not answer, then checked that each question had one clear, correct answer.

Older tests became too easy because models scored almost perfectly on them. This exam gives researchers a harder ruler to measure progress, and it also checks whether a model knows when it is likely to be wrong.

A typical use case

An AI research team is choosing between two candidate models for a scientific literature assistant. Both score above 90% on MMLU, so that benchmark gives no useful signal. The team runs the text-only HLE subset under identical conditions: no tools, one attempt, and the same judge prompt.

One model scores about 18% with severe overconfidence, while the other scores about 15% but reports much better calibrated confidence. Because the assistant will flag uncertain answers for human review, the team prefers the second model, then confirms the choice with its own domain evaluation set of 400 annotated questions before deployment.