GSM8K (Grade School Math 8K) is a benchmark of roughly 8,500 math word problems written at a grade school level, used to measure how well language models perform multi-step arithmetic reasoning. In model evaluation, it became one of the most cited reasoning tests after its release in 2021, because each problem has a single verifiable numeric answer that allows strict automatic scoring.
The problems require only basic operations: addition, subtraction, multiplication and division. Their difficulty lies in chaining two to eight reasoning steps correctly, which made GSM8K an early proving ground for chain-of-thought prompting and for training methods that reward correct intermediate reasoning.
Structure of the GSM8K dataset
GSM8K consists of human-written problems created by contract writers and checked for quality, with linguistic diversity prioritized over templated variation. Each item contains a short story problem, a step-by-step solution in natural language, and a final integer answer placed after a fixed delimiter of four hash marks.
The table below summarizes the main properties of the dataset as it is commonly distributed.
| Property | Value | Why it matters |
|---|---|---|
| Training split | About 7,500 problems | Supports fine-tuning and verifier training |
| Test split | About 1,300 problems | Standard set for reported scores |
| Reasoning depth | 2 to 8 steps | Tests chaining, not advanced mathematics |
| Answer format | Single integer | Enables exact-match scoring |
| Solution style | Natural language with calculations | Usable as chain-of-thought training data |
Because the mathematics never exceeds early middle school, an incorrect answer almost always reflects a reasoning or arithmetic slip rather than missing knowledge. This makes the benchmark a relatively clean probe of procedural reliability.
How GSM8K is scored
GSM8K is scored with exact match on the final numeric answer. The evaluation harness extracts the last number from the model output, normalizes formatting such as commas or trailing decimals, and compares it with the reference integer. The reasoning text itself is not graded, so a correct answer reached through flawed logic still counts.
Reported scores depend heavily on the evaluation protocol, which is why numbers from different sources are often not comparable. The table below lists the settings that most influence results.
| Setting | Common options | Effect on score |
|---|---|---|
| Number of examples in prompt | Zero-shot, 5-shot, 8-shot | Few-shot helps weaker base models |
| Chain-of-thought | Direct answer or step-by-step | Step-by-step raises accuracy sharply |
| Sampling | Greedy or majority vote | Majority vote adds several points |
| Answer extraction | Strict delimiter or last number | Lenient parsing raises scores |
| Tool use | None or code execution | Code removes arithmetic errors |
The widely used convention is 8-shot chain-of-thought prompting with greedy decoding. Results obtained with majority voting (self-consistency) or a code interpreter should be labeled as such, since they measure a different system configuration.
The history of GSM8K results
GSM8K tracks the rapid progress of mathematical reasoning in large language models. At release, large fine-tuned models solved roughly a third of the test problems, and the original work showed that training a verifier to rank sampled solutions improved results substantially.
| Period | Representative approach | Typical test accuracy |
|---|---|---|
| 2021 | Fine-tuned models with verifiers | About 35-55% |
| 2022 | Chain-of-thought prompting at large scale | About 55-75% |
| 2023 | Instruction-tuned frontier models | About 85-92% |
| 2024 onward | Reasoning-focused models | Above 95% |
Scores above 95 percent mean GSM8K is now largely saturated for frontier systems. Remaining errors at that level are often attributable to label noise: careful audits have found that a small fraction of test items contain ambiguous wording or incorrect reference answers.
Contamination and robustness concerns
The main threat to GSM8K validity is benchmark contamination. The dataset has been public since 2021, so its problems and solutions appear in web crawls and derived training sets, and a model may reproduce memorized answers rather than reason. Several follow-up studies tested this directly.
- Freshly written parallel sets of similar difficulty revealed accuracy drops of several points for some model families, which suggests overfitting to the original items.
- Symbolic variants that change names and numbers in the same problem templates showed that small surface changes can reduce accuracy noticeably.
- Inserting irrelevant clauses into problems caused larger drops, which indicates fragile reasoning rather than genuine understanding.
- Cleaned versions of the test set remove mislabeled items so that high-end scores reflect real errors.
These findings do not make GSM8K useless, but they mean a high score is necessary, not sufficient evidence of robust arithmetic reasoning.
GSM8K compared with other reasoning benchmarks
GSM8K occupies the entry level of mathematical evaluation. Harder benchmarks have taken over the task of separating frontier models, while GSM8K remains useful for small models, ablation studies and regression testing.
| Benchmark | Content | Difficulty | Current role |
|---|---|---|---|
| GSM8K | Grade school word problems | Low | Baseline and small-model testing |
| MATH | Competition mathematics | Medium to high | Mid-tier reasoning comparison |
| FrontierMath | Research-level problems | Very high | Frontier capability tracking |
| MMLU | Multiple-choice knowledge | Medium | Broad knowledge, not reasoning depth |
A practical rule follows from this comparison: a model that fails GSM8K will rarely succeed on harder sets, while a model that passes it still needs MATH-level or newer tests to show genuine strength. GSM8K therefore works best as a quick sanity check early in development rather than as a final verdict on reasoning quality.
Using GSM8K responsibly
GSM8K remains valuable when its limits are made explicit. Teams should report the full protocol, check for training-data overlap, and pair the benchmark with harder or freshly written tests before drawing conclusions about reasoning ability.
- State the prompt format, number of examples and decoding method with every score.
- Run an error analysis on failures to separate arithmetic slips from misread problems.
- Avoid tuning on the test split, including indirect tuning through repeated prompt selection.
- Add a contamination check such as a held-out rewritten subset.
The idea in brief
GSM8K is a large set of school-style math stories. A typical question asks how much money someone has left after buying a few items. The math is simple. The challenge is doing several small steps in the right order without a mistake.
Researchers give these questions to AI models and count how many final answers are exactly right. For years this was a good test of step-by-step thinking. Today the best models solve almost all of them, so harder tests are needed to tell top systems apart.
A practical scenario
A small research team compresses a 7-billion-parameter model to 4-bit precision so that it can run on a laptop. To confirm that reasoning survives the compression, they evaluate both versions on the GSM8K test split with the same 8-shot chain-of-thought prompt and greedy decoding.
The original model scores 58 percent and the compressed model 51 percent. An error analysis of 100 failures shows that most new errors are arithmetic slips in multi-digit multiplication, not misunderstood problems. The team switches to a different quantization method for the attention layers, reruns the test, and reaches 56 percent, a gap of only two points that they accept for deployment.