GPQA

Also known as: Graduate-Level Google-Proof Q&A

On this page 7

GPQA, short for Graduate-Level Google-Proof Q&A, is a benchmark of difficult multiple-choice questions in biology, physics and chemistry written by domain experts and designed so that a skilled person cannot answer them by searching the web. In model evaluation, GPQA measures whether a language model can apply graduate-level scientific reasoning rather than retrieve memorized facts.

Artificial intelligence researchers introduced GPQA in 2023, partly to support research on scalable oversight: the problem of supervising systems whose answers humans cannot easily verify. Its Diamond subset quickly became one of the most quoted reasoning benchmarks on model cards and leaderboards.

How the GPQA questions were built and validated

GPQA questions were produced through a multi-stage expert pipeline rather than scraped from existing exams. Contractors holding or pursuing PhDs in the relevant field wrote each question together with four answer options and an explanation of the correct answer.

Each question was then checked by two further domain experts, who answered it and gave feedback, and by three non-experts from other scientific fields. The non-experts had unrestricted web access and spent on average more than half an hour per question. A good question was one that experts answered correctly while skilled non-experts usually failed, which is the practical meaning of Google-proof.

The validation results define the reference points that make GPQA scores interpretable, as summarized below.

GroupApproximate accuracyConditions
Domain expertsAbout 65 percent, roughly 74 percent after removing clear expert mistakesAnswering questions in their own field
Skilled non-expertsAbout 34 percentWeb search allowed, around 30 minutes per question
Random guessing25 percentFour answer options
Human reference accuracy on GPQA

GPQA subsets and question format

GPQA is published as three nested subsets that differ in size and quality filtering. All questions use the four-option multiple-choice format, and the answer order is usually shuffled at evaluation time to remove position bias.

SubsetQuestionsSelection ruleTypical use
Extended546All collected questionsAnalysis and research on labeling
Main448Excludes questions with weak expert validationBroader reported scores
Diamond198Both experts agree with the answer and most non-experts failHeadline model comparisons
The three GPQA subsets

Because GPQA Diamond contains fewer than 200 items, a single question is worth about half a percentage point. Differences of two or three points between models are therefore often within statistical noise, especially when results come from a single sampling run.

How models are scored on GPQA

Models are scored by accuracy on the selected subset, the share of questions where the chosen option matches the key. Evaluation settings strongly influence the number, so serious reports state them explicitly.

SettingCommon choicesEffect on reported score
Prompting styleZero-shot, few-shot, chain-of-thoughtReasoning prompts usually raise accuracy
Answer extractionLetter parsing, logit comparison, LLM judgeParsing errors can cost several points
SamplingGreedy decoding or averaged repeated runsAveraging reduces variance on Diamond
Tool accessClosed-book or with search and codeTools change what is being measured
Reasoning budgetShort answers or extended thinkingLonger reasoning generally helps
Evaluation settings that change GPQA scores

Evaluation harnesses such as standardized open-source frameworks fix most of these choices, which makes cross-model comparisons more trustworthy than numbers copied from different vendor announcements.

GPQA compared with other reasoning benchmarks

GPQA occupies a middle ground between knowledge tests and frontier research problems. It is far harder than MMLU, which covers many subjects at undergraduate level, but easier than Humanity’s Last Exam or FrontierMath, which target the edge of expert ability.

BenchmarkDomainDifficulty levelFormat
GPQA DiamondBiology, physics, chemistryGraduate expertFour-option multiple choice
MMLU57 academic and professional subjectsHigh school to professionalFour-option multiple choice
Humanity’s Last ExamBroad academic frontierBeyond typical expertMultiple choice and exact answer
FrontierMathResearch mathematicsResearch levelExact numeric or symbolic answer
GPQA and related benchmarks

The benchmarks are complementary. GPQA tests scientific reasoning under uncertainty, while exact-answer benchmarks remove the chance of guessing and leaderboards such as LMArena capture human preference rather than correctness. GPQA is also used beyond capability ranking: oversight researchers test whether non-experts assisted by a model can reach expert-level answers, a setup that mirrors supervising systems smarter than their supervisors.

Limitations and saturation of GPQA

GPQA has clear limitations that practitioners should account for when reading scores. The most important is benchmark saturation: frontier reasoning models now report Diamond scores well above the expert baseline, so the benchmark separates leading systems less clearly than it did at launch.

  • Small sample size makes Diamond scores noisy and sensitive to a handful of questions.
  • Multiple-choice format allows elimination strategies and a 25 percent guessing floor.
  • Label errors remain possible, because even expert validators disagreed on some items.
  • Benchmark contamination is a risk once questions circulate online, despite a canary string intended to keep them out of training data.
  • Narrow coverage of three sciences says little about coding, law or everyday tasks.

A high GPQA score therefore indicates strong scientific reasoning on short, well-posed problems, not general expertise or reliability in open-ended research work.

Responsible reporting addresses these weaknesses directly. Good practice is to publish confidence intervals or the spread across runs, state the exact prompt and extraction method, and pair GPQA with exact-answer benchmarks and private test sets. When a model is meant for a specific scientific domain, a small set of fresh questions written by in-house experts often reveals more than an additional point on a public leaderboard.

In everyday language

GPQA is a very hard science quiz for AI systems. Real scientists wrote the questions, and the questions are tricky enough that a smart person with internet access usually still gets them wrong.

If an AI model scores well, it suggests the model can reason through difficult science problems instead of simply looking up or remembering answers. It does not prove the model is a scientist, but it is a useful sign of reasoning ability.

Seen in the real world

A research team at a university hospital is choosing a language model to help draft summaries of biochemistry literature. Vendor announcements quote GPQA Diamond scores, but each uses different prompts, so the team runs its own evaluation with an open-source harness, chain-of-thought prompting and five repeated runs per model.

Two candidate models score 78 and 80 percent on average, with a run-to-run spread of about three points, so the team treats them as statistically tied on GPQA. A third, cheaper model scores 61 percent and is ruled out for the scientific workload.

The team then adds its own test set of 50 questions drawn from recent internal cases, which cannot have appeared in any training data. That domain-specific check decides the final choice, while GPQA served as a quick, credible first filter.