FrontierMath

On this page 7

FrontierMath is a benchmark of original, unpublished mathematics problems written by professional mathematicians to measure how well AI systems handle research-level mathematical reasoning. Developed by the research organization Epoch AI and introduced in late 2024, it occupies a distinct place in model evaluation because its problems were designed to be hard for experts, unseen in training data and checkable by a computer.

In artificial intelligence, FrontierMath is used as a frontier benchmark: a test meant to remain informative after older math benchmarks such as GSM8K reached saturation. When it launched, leading models solved under 2 percent of its problems, which made it one of the clearest signals of the gap between fluent mathematical text and genuine problem solving.

How FrontierMath problems are built and graded

FrontierMath problems are constructed so that a correct answer is a single, exactly verifiable object, such as a large integer, a rational number or a symbolic expression, rather than a written proof. This design allows fully automated grading: a script compares the model output with the stored answer, with no human judge and no LLM-as-a-judge involved.

Each problem is written to be guessproof, meaning the answer space is large enough that random guessing or pattern matching has a negligible chance of success. Problems pass expert peer review for correctness and difficulty, and the contributing mathematicians span fields such as number theory, algebraic geometry, combinatorics and real analysis.

The main design properties and their purpose are summarized below.

PropertyImplementationPurpose
NoveltyProblems written new and kept unpublishedLimits benchmark contamination
Automatic verificationExact numeric or symbolic answersObjective, reproducible scoring
Guessproof answersLarge answer spacesPrevents lucky guesses
Expert difficultyProblems that take specialists hours or daysResists saturation
Tool accessModels may run code during solvingTests reasoning, not arithmetic
Design properties of FrontierMath

Difficulty tiers and problem coverage

FrontierMath organizes its problems into difficulty tiers, ranging from demanding problems at the level of advanced undergraduate or early graduate study to problems that challenge research mathematicians. A later addition, commonly called Tier 4, contains a smaller set of exceptionally hard problems intended to stay unsolved by AI for longer.

The tier structure lets evaluators report separate scores by difficulty, which is more informative than one aggregate number. The following table describes the tiers in general terms; exact counts have changed as the set was expanded.

TierTypical levelExpected solver
Tiers 1 and 2Advanced undergraduate to graduateStrong graduate student with time
Tier 3Research-oriented problemsSpecialist in the relevant subfield
Tier 4Very hard research problemsLeading experts, often with significant effort
Approximate difficulty tiers

Coverage deliberately spans most major branches of modern mathematics, so a model cannot score well by mastering one narrow technique. This breadth also means that scores can shift noticeably when a model improves in a single area that is heavily represented.

How FrontierMath compares with other reasoning benchmarks

FrontierMath differs from earlier math benchmarks mainly in difficulty, novelty and grading method. School-level datasets measure multistep arithmetic, while FrontierMath targets problems where even the solution approach is unclear at the start.

BenchmarkContentGradingSaturation status
GSM8KGrade-school word problemsExact final numberLargely saturated
MATH datasetCompetition mathematicsExact final answerHigh scores by top models
GPQAGraduate science multiple choiceChoice accuracyApproaching expert level
Humanity’s Last ExamExpert questions across many fieldsExact or multiple choiceStill challenging
FrontierMathOriginal research-level mathematicsAutomated exact verificationFar from saturated at launch
FrontierMath and related benchmarks

Unlike multiple-choice tests, FrontierMath does not give a guessing floor of 20 or 25 percent, so even a few percent reflects genuinely solved problems. Its weakness relative to proof-based evaluation is that it rewards correct final answers, not the full rigor of a written argument.

Interpreting FrontierMath scores

A FrontierMath score is the share of problems answered exactly, but its meaning depends heavily on the evaluation conditions. Results differ with the number of attempts, the compute budget per problem, the time allowed for code execution and whether scores are pass@1 or best-of-several.

The table below lists factors to check before comparing two reported numbers.

FactorEffect on scoreWhat to check
Inference computeMore reasoning tokens raise scoresEffort setting or token budget
Number of attemptspass@k exceeds single-attempt accuracyWhether pass@1 is reported
Problem subsetPublic and holdout sets differWhich split was used
EvaluatorSelf-reported and independent runs varyWho ran the evaluation
ScaffoldingAgents with tools perform differentlyTools and prompts allowed
Factors that change reported scores

Because the problem set is small compared with general benchmarks, statistical uncertainty is substantial: a few problems can move the score by several points. Careful reports include confidence intervals and per-tier results.

Limitations and controversies

FrontierMath has known limitations that practitioners should weigh alongside its strengths.

  • Funding and data access: it became public in early 2025 that OpenAI had funded the project and had access to much of the problem set, which raised questions about benchmark independence; Epoch AI responded by stressing a separate holdout set.
  • Answer-only grading: a correct number can occasionally be reached through heuristics or numerical exploration without a sound mathematical argument.
  • Small sample size: scores carry wide error margins and are sensitive to individual problems.
  • Limited transparency: keeping problems private protects against contamination but makes outside auditing of difficulty and correctness harder.
  • Narrow scope: it measures mathematical problem solving, not general reasoning, safety or usefulness.

For these reasons, FrontierMath is best read as one strong signal among several, combined with other benchmarks, error analysis and human evaluation by mathematicians.

In simple terms

FrontierMath is a very hard math exam for AI. Professional mathematicians wrote brand-new problems that do not appear anywhere online, so a model cannot simply remember the answers. Each answer is an exact number or formula, which a computer can check instantly.

When the test appeared, the best AI systems solved only a handful of the problems. Because of that, it works like a high bar that shows real progress: when scores rise, it suggests models are getting better at genuine mathematical thinking, not just at repeating familiar exercises.

A practical example

An AI lab preparing a new reasoning model wants to know whether its longer chain-of-thought training actually improves advanced mathematics. The evaluation team runs the model on the available FrontierMath problems under fixed conditions: one attempt per problem, a Python environment for computation and a set token budget.

The previous model version solved roughly 6 percent of the problems, and the new version solves about 14 percent, with most gains in the lower tiers and almost none in Tier 4. Given the small problem count, the team reports a confidence interval of several points and confirms the result in a second run at a higher compute setting. The outcome is a measured, honest claim of progress that also identifies where the model still fails.