HellaSwag

On this page 7

HellaSwag is a commonsense natural language inference benchmark in which a model reads the beginning of an everyday scenario and must choose the most plausible continuation out of four candidates. Introduced in 2019, it became one of the standard tests of model evaluation in artificial intelligence because the wrong answers were deliberately generated to fool machines while remaining obviously wrong to humans.

The name is a backronym for Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations. The benchmark extends the earlier SWAG dataset, and its central contribution is the adversarial filtering procedure that produced its distractors.

How HellaSwag was constructed

HellaSwag was built by pairing human-written contexts with machine-generated wrong endings that survived repeated filtering. Contexts come from two sources: ActivityNet Captions, which describe short video clips of physical activities, and WikiHow articles, which describe procedures step by step.

For each context, a language model generated many candidate endings. Adversarial filtering then trained a series of discriminators, mainly BERT-based classifiers, to separate real endings from generated ones. Generated endings that the discriminators rejected easily were replaced, and the process repeated until the remaining distractors were hard for the strongest model of the time. Crowd workers finally validated the items so that humans could still identify the correct ending reliably.

The table below summarizes the main properties of the released dataset.

PropertyValueComment
Total itemsAbout 70,000Split into training, validation and test sets
Validation setAbout 10,000 itemsThe split most commonly reported in papers
Answer optionsFour per itemOne human ending, three adversarial distractors
Context sourcesActivityNet Captions and WikiHowPhysical activities and how-to procedures
Human accuracyAbove 95%Measured on validated items
Accuracy at releaseBelow 50% for BERT-LargeShows how hard the distractors were in 2019
Key properties of the HellaSwag dataset

How models are scored on HellaSwag

HellaSwag is scored with accuracy, the share of items for which the model selects the correct ending. Random guessing yields 25% accuracy, which is the floor for meaningful comparison.

Generative language models are usually evaluated by comparing log-likelihoods rather than by asking for a letter answer. The evaluation harness computes the probability of each ending given the context and picks the most probable one. Because longer endings accumulate lower total probability, most harnesses report length-normalized accuracy, often labeled acc_norm, in which the log-likelihood is divided by the number of characters or tokens.

Reported numbers depend heavily on the protocol, as the next table shows.

SettingCommon optionsEffect on the score
Number of shotsZero-shot or 10-shotFew-shot examples usually raise accuracy
NormalizationRaw or length-normalized likelihoodNormalized scores are typically several points higher
Answer formatLikelihood ranking or multiple-choice letterLetter prompts favor instruction-tuned models
Data splitValidation or testTest labels were withheld for leaderboard use
Prompt templateHarness default or customSmall wording changes can shift results
Evaluation settings that change HellaSwag scores

A score is therefore only comparable within the same protocol. Comparing a zero-shot raw figure with a 10-shot normalized figure from another report is a frequent source of misleading claims.

HellaSwag compared with other benchmarks

HellaSwag measures grounded commonsense about everyday events, which distinguishes it from benchmarks focused on academic knowledge, mathematics or code. It is often reported alongside them to give a broader picture of model capability.

BenchmarkWhat it testsFormat
HellaSwagCommonsense continuation of everyday scenariosFour-way multiple choice
SWAGGrounded inference on video captionsFour-way multiple choice, weaker distractors
MMLUAcademic and professional knowledge in 57 subjectsFour-way multiple choice
TruthfulQAResistance to common misconceptionsMultiple choice and generation
GSM8KGrade-school arithmetic word problemsFree-form numeric answer
ARC-AGIAbstract visual pattern inductionGrid puzzles with exact match
HellaSwag next to related evaluation benchmarks

Its predecessor SWAG was effectively solved by BERT within months of release, which motivated the stronger filtering in HellaSwag. The same story later repeated: HellaSwag itself reached benchmark saturation as frontier models approached human accuracy.

Why HellaSwag is now saturated

HellaSwag no longer separates frontier models because their scores sit close to the human ceiling of about 95%. The adversarial distractors were tuned against 2019 discriminators, and much larger models trained on far more data no longer find them confusing.

Several further weaknesses limit its value today:

  • Benchmark contamination: the dataset has been public since 2019, so its text may appear in web-scale pretraining corpora.
  • Label and text errors: later audits found ungrammatical contexts, ambiguous items and some questionable gold answers, which cap the meaningful ceiling.
  • Artifacts in distractors: generated endings can carry stylistic cues, so a model may succeed by detecting machine text instead of reasoning.
  • Narrow scope: everyday physical and procedural scenarios do not cover social, causal or multistep reasoning in depth.

Despite these issues, HellaSwag remains useful as a sanity check during pretraining and for small or mid-sized models, where scores still vary widely and track general language modeling quality.

Using HellaSwag responsibly

HellaSwag is most informative when used as one signal among several, with a fixed protocol and an awareness of its ceiling. The decision table below lists typical uses.

SituationSuitabilityReason
Tracking a pretraining run over checkpointsGoodScores rise smoothly with language modeling ability
Comparing small or compressed modelsGoodDifferences remain visible below the ceiling
Ranking frontier modelsPoorDifferences fall within noise near 95%
Checking quantization or distillation lossUsefulDrops reveal degraded commonsense continuation
Claiming general reasoning abilityPoorThe task is narrow and partly solvable by artifacts
When HellaSwag is and is not a good choice

Good practice is to report the exact harness, shot count and normalization, run contamination checks where training data is known and complement HellaSwag with newer, harder and less exposed evaluation datasets.

The short version

HellaSwag is a test that checks whether an AI system has everyday common sense. Each question describes the start of an ordinary situation, such as someone washing a car, and offers four possible next sentences. Only one makes sense; the others were written by a computer to sound plausible to machines but silly to people.

Humans get almost every question right. Early language models got fewer than half right, but today’s best systems score about as well as people, so the test is now mainly useful for smaller models and for tracking progress during training.

Where it makes a difference

An engineering team prepares a 7-billion-parameter language model for deployment on laptops and wants to shrink it with 4-bit quantization. Before release, they need evidence that compression has not damaged the model’s general language understanding.

They run HellaSwag in a standard evaluation harness, using the 10-shot, length-normalized setting on the validation split, for both the original and the quantized model. The original scores 78.4% and the first quantized version scores 74.1%, a drop of more than four points that is far larger than the run-to-run noise they measured earlier.

Investigating the gap, they find that one quantization setting damaged specific attention layers. After switching to a calibration-aware method, the quantized model scores 77.9%. The team ships that version and records the full evaluation protocol in the release notes so that later comparisons use the same settings.