GSM8K

On this page 8

GSM8K (Grade School Math 8K) is a benchmark of roughly 8,500 math word problems written at a grade school level, used to measure how well language models perform multi-step arithmetic reasoning. In model evaluation, it became one of the most cited reasoning tests after its release in 2021, because each problem has a single verifiable numeric answer that allows strict automatic scoring.

The problems require only basic operations: addition, subtraction, multiplication and division. Their difficulty lies in chaining two to eight reasoning steps correctly, which made GSM8K an early proving ground for chain-of-thought prompting and for training methods that reward correct intermediate reasoning.

Structure of the GSM8K dataset

GSM8K consists of human-written problems created by contract writers and checked for quality, with linguistic diversity prioritized over templated variation. Each item contains a short story problem, a step-by-step solution in natural language, and a final integer answer placed after a fixed delimiter of four hash marks.

The table below summarizes the main properties of the dataset as it is commonly distributed.

PropertyValueWhy it matters
Training splitAbout 7,500 problemsSupports fine-tuning and verifier training
Test splitAbout 1,300 problemsStandard set for reported scores
Reasoning depth2 to 8 stepsTests chaining, not advanced mathematics
Answer formatSingle integerEnables exact-match scoring
Solution styleNatural language with calculationsUsable as chain-of-thought training data
Key properties of GSM8K

Because the mathematics never exceeds early middle school, an incorrect answer almost always reflects a reasoning or arithmetic slip rather than missing knowledge. This makes the benchmark a relatively clean probe of procedural reliability.

How GSM8K is scored

GSM8K is scored with exact match on the final numeric answer. The evaluation harness extracts the last number from the model output, normalizes formatting such as commas or trailing decimals, and compares it with the reference integer. The reasoning text itself is not graded, so a correct answer reached through flawed logic still counts.

Reported scores depend heavily on the evaluation protocol, which is why numbers from different sources are often not comparable. The table below lists the settings that most influence results.

SettingCommon optionsEffect on score
Number of examples in promptZero-shot, 5-shot, 8-shotFew-shot helps weaker base models
Chain-of-thoughtDirect answer or step-by-stepStep-by-step raises accuracy sharply
SamplingGreedy or majority voteMajority vote adds several points
Answer extractionStrict delimiter or last numberLenient parsing raises scores
Tool useNone or code executionCode removes arithmetic errors
Protocol choices that affect GSM8K scores

The widely used convention is 8-shot chain-of-thought prompting with greedy decoding. Results obtained with majority voting (self-consistency) or a code interpreter should be labeled as such, since they measure a different system configuration.

The history of GSM8K results

GSM8K tracks the rapid progress of mathematical reasoning in large language models. At release, large fine-tuned models solved roughly a third of the test problems, and the original work showed that training a verifier to rank sampled solutions improved results substantially.

PeriodRepresentative approachTypical test accuracy
2021Fine-tuned models with verifiersAbout 35-55%
2022Chain-of-thought prompting at large scaleAbout 55-75%
2023Instruction-tuned frontier modelsAbout 85-92%
2024 onwardReasoning-focused modelsAbove 95%
Approximate progression of GSM8K accuracy

Scores above 95 percent mean GSM8K is now largely saturated for frontier systems. Remaining errors at that level are often attributable to label noise: careful audits have found that a small fraction of test items contain ambiguous wording or incorrect reference answers.

Contamination and robustness concerns

The main threat to GSM8K validity is benchmark contamination. The dataset has been public since 2021, so its problems and solutions appear in web crawls and derived training sets, and a model may reproduce memorized answers rather than reason. Several follow-up studies tested this directly.

  • Freshly written parallel sets of similar difficulty revealed accuracy drops of several points for some model families, which suggests overfitting to the original items.
  • Symbolic variants that change names and numbers in the same problem templates showed that small surface changes can reduce accuracy noticeably.
  • Inserting irrelevant clauses into problems caused larger drops, which indicates fragile reasoning rather than genuine understanding.
  • Cleaned versions of the test set remove mislabeled items so that high-end scores reflect real errors.

These findings do not make GSM8K useless, but they mean a high score is necessary, not sufficient evidence of robust arithmetic reasoning.

GSM8K compared with other reasoning benchmarks

GSM8K occupies the entry level of mathematical evaluation. Harder benchmarks have taken over the task of separating frontier models, while GSM8K remains useful for small models, ablation studies and regression testing.

BenchmarkContentDifficultyCurrent role
GSM8KGrade school word problemsLowBaseline and small-model testing
MATHCompetition mathematicsMedium to highMid-tier reasoning comparison
FrontierMathResearch-level problemsVery highFrontier capability tracking
MMLUMultiple-choice knowledgeMediumBroad knowledge, not reasoning depth
Position of GSM8K among related benchmarks

A practical rule follows from this comparison: a model that fails GSM8K will rarely succeed on harder sets, while a model that passes it still needs MATH-level or newer tests to show genuine strength. GSM8K therefore works best as a quick sanity check early in development rather than as a final verdict on reasoning quality.

Using GSM8K responsibly

GSM8K remains valuable when its limits are made explicit. Teams should report the full protocol, check for training-data overlap, and pair the benchmark with harder or freshly written tests before drawing conclusions about reasoning ability.

  • State the prompt format, number of examples and decoding method with every score.
  • Run an error analysis on failures to separate arithmetic slips from misread problems.
  • Avoid tuning on the test split, including indirect tuning through repeated prompt selection.
  • Add a contamination check such as a held-out rewritten subset.

The idea in brief

GSM8K is a large set of school-style math stories. A typical question asks how much money someone has left after buying a few items. The math is simple. The challenge is doing several small steps in the right order without a mistake.

Researchers give these questions to AI models and count how many final answers are exactly right. For years this was a good test of step-by-step thinking. Today the best models solve almost all of them, so harder tests are needed to tell top systems apart.

A practical scenario

A small research team compresses a 7-billion-parameter model to 4-bit precision so that it can run on a laptop. To confirm that reasoning survives the compression, they evaluate both versions on the GSM8K test split with the same 8-shot chain-of-thought prompt and greedy decoding.

The original model scores 58 percent and the compressed model 51 percent. An error analysis of 100 failures shows that most new errors are arithmetic slips in multi-digit multiplication, not misunderstood problems. The team switches to a different quantization method for the attention layers, reruns the test, and reaches 56 percent, a gap of only two points that they accept for deployment.