HellaSwag is a commonsense natural language inference benchmark in which a model reads the beginning of an everyday scenario and must choose the most plausible continuation out of four candidates. Introduced in 2019, it became one of the standard tests of model evaluation in artificial intelligence because the wrong answers were deliberately generated to fool machines while remaining obviously wrong to humans.
The name is a backronym for Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations. The benchmark extends the earlier SWAG dataset, and its central contribution is the adversarial filtering procedure that produced its distractors.
How HellaSwag was constructed
HellaSwag was built by pairing human-written contexts with machine-generated wrong endings that survived repeated filtering. Contexts come from two sources: ActivityNet Captions, which describe short video clips of physical activities, and WikiHow articles, which describe procedures step by step.
For each context, a language model generated many candidate endings. Adversarial filtering then trained a series of discriminators, mainly BERT-based classifiers, to separate real endings from generated ones. Generated endings that the discriminators rejected easily were replaced, and the process repeated until the remaining distractors were hard for the strongest model of the time. Crowd workers finally validated the items so that humans could still identify the correct ending reliably.
The table below summarizes the main properties of the released dataset.
| Property | Value | Comment |
|---|---|---|
| Total items | About 70,000 | Split into training, validation and test sets |
| Validation set | About 10,000 items | The split most commonly reported in papers |
| Answer options | Four per item | One human ending, three adversarial distractors |
| Context sources | ActivityNet Captions and WikiHow | Physical activities and how-to procedures |
| Human accuracy | Above 95% | Measured on validated items |
| Accuracy at release | Below 50% for BERT-Large | Shows how hard the distractors were in 2019 |
How models are scored on HellaSwag
HellaSwag is scored with accuracy, the share of items for which the model selects the correct ending. Random guessing yields 25% accuracy, which is the floor for meaningful comparison.
Generative language models are usually evaluated by comparing log-likelihoods rather than by asking for a letter answer. The evaluation harness computes the probability of each ending given the context and picks the most probable one. Because longer endings accumulate lower total probability, most harnesses report length-normalized accuracy, often labeled acc_norm, in which the log-likelihood is divided by the number of characters or tokens.
Reported numbers depend heavily on the protocol, as the next table shows.
| Setting | Common options | Effect on the score |
|---|---|---|
| Number of shots | Zero-shot or 10-shot | Few-shot examples usually raise accuracy |
| Normalization | Raw or length-normalized likelihood | Normalized scores are typically several points higher |
| Answer format | Likelihood ranking or multiple-choice letter | Letter prompts favor instruction-tuned models |
| Data split | Validation or test | Test labels were withheld for leaderboard use |
| Prompt template | Harness default or custom | Small wording changes can shift results |
A score is therefore only comparable within the same protocol. Comparing a zero-shot raw figure with a 10-shot normalized figure from another report is a frequent source of misleading claims.
HellaSwag compared with other benchmarks
HellaSwag measures grounded commonsense about everyday events, which distinguishes it from benchmarks focused on academic knowledge, mathematics or code. It is often reported alongside them to give a broader picture of model capability.
| Benchmark | What it tests | Format |
|---|---|---|
| HellaSwag | Commonsense continuation of everyday scenarios | Four-way multiple choice |
| SWAG | Grounded inference on video captions | Four-way multiple choice, weaker distractors |
| MMLU | Academic and professional knowledge in 57 subjects | Four-way multiple choice |
| TruthfulQA | Resistance to common misconceptions | Multiple choice and generation |
| GSM8K | Grade-school arithmetic word problems | Free-form numeric answer |
| ARC-AGI | Abstract visual pattern induction | Grid puzzles with exact match |
Its predecessor SWAG was effectively solved by BERT within months of release, which motivated the stronger filtering in HellaSwag. The same story later repeated: HellaSwag itself reached benchmark saturation as frontier models approached human accuracy.
Why HellaSwag is now saturated
HellaSwag no longer separates frontier models because their scores sit close to the human ceiling of about 95%. The adversarial distractors were tuned against 2019 discriminators, and much larger models trained on far more data no longer find them confusing.
Several further weaknesses limit its value today:
- Benchmark contamination: the dataset has been public since 2019, so its text may appear in web-scale pretraining corpora.
- Label and text errors: later audits found ungrammatical contexts, ambiguous items and some questionable gold answers, which cap the meaningful ceiling.
- Artifacts in distractors: generated endings can carry stylistic cues, so a model may succeed by detecting machine text instead of reasoning.
- Narrow scope: everyday physical and procedural scenarios do not cover social, causal or multistep reasoning in depth.
Despite these issues, HellaSwag remains useful as a sanity check during pretraining and for small or mid-sized models, where scores still vary widely and track general language modeling quality.
Using HellaSwag responsibly
HellaSwag is most informative when used as one signal among several, with a fixed protocol and an awareness of its ceiling. The decision table below lists typical uses.
| Situation | Suitability | Reason |
|---|---|---|
| Tracking a pretraining run over checkpoints | Good | Scores rise smoothly with language modeling ability |
| Comparing small or compressed models | Good | Differences remain visible below the ceiling |
| Ranking frontier models | Poor | Differences fall within noise near 95% |
| Checking quantization or distillation loss | Useful | Drops reveal degraded commonsense continuation |
| Claiming general reasoning ability | Poor | The task is narrow and partly solvable by artifacts |
Good practice is to report the exact harness, shot count and normalization, run contamination checks where training data is known and complement HellaSwag with newer, harder and less exposed evaluation datasets.
The short version
HellaSwag is a test that checks whether an AI system has everyday common sense. Each question describes the start of an ordinary situation, such as someone washing a car, and offers four possible next sentences. Only one makes sense; the others were written by a computer to sound plausible to machines but silly to people.
Humans get almost every question right. Early language models got fewer than half right, but today’s best systems score about as well as people, so the test is now mainly useful for smaller models and for tracking progress during training.
Where it makes a difference
An engineering team prepares a 7-billion-parameter language model for deployment on laptops and wants to shrink it with 4-bit quantization. Before release, they need evidence that compression has not damaged the model’s general language understanding.
They run HellaSwag in a standard evaluation harness, using the 10-shot, length-normalized setting on the validation split, for both the original and the quantized model. The original scores 78.4% and the first quantized version scores 74.1%, a drop of more than four points that is far larger than the run-to-run noise they measured earlier.
Investigating the gap, they find that one quantization setting damaged specific attention layers. After switching to a calibration-aware method, the quantized model scores 77.9%. The team ships that version and records the full evaluation protocol in the release notes so that later comparisons use the same settings.