IFEval

On this page 7

IFEval (Instruction-Following Evaluation) is a benchmark that measures how reliably a large language model follows explicit, verifiable instructions, such as writing more than 300 words, answering in JSON or avoiding commas. In model evaluation for artificial intelligence, it is one of the standard tests of instruction-following ability, separate from knowledge or reasoning skill.

Its defining feature is that every instruction can be checked automatically by a short program. This removes the need for human raters or an LLM-as-a-judge and makes scores reproducible and cheap to compute, which explains its wide adoption in model reports and leaderboards since its release by Google researchers in 2023.

How IFEval tests instruction following

IFEval works by pairing ordinary writing prompts with one or more verifiable constraints and then checking the model’s response with deterministic rules. The dataset contains about 500 prompts built from 25 types of verifiable instructions. A typical prompt reads: write a blog post about a trip to Japan, use fewer than 300 words and do not use any commas.

Each instruction type has its own checking function. A length constraint counts words or sentences, a format constraint parses the output as JSON or counts Markdown bullet points, and a keyword constraint searches for required or forbidden words. The content quality of the answer is not judged; only compliance with the stated constraints is scored.

The table below lists the main instruction groups and how each is verified.

Instruction groupExample constraintAutomatic check
KeywordsInclude the word “budget” at least twiceCount of keyword occurrences
Length constraintsAnswer in fewer than 150 wordsWord, sentence or paragraph count
Detectable formatWrap the whole response in JSONParse output as valid JSON
Change caseWrite everything in lowercaseCompare text with its lowercase form
PunctuationDo not use any commasSearch for the forbidden character
Start and endFinish with a fixed closing phraseString match at the end of output
Instruction groups in IFEval and their checks

Strict and loose accuracy metrics

IFEval reports four accuracy figures, formed by combining two levels of aggregation with two levels of strictness. Prompt-level accuracy counts a prompt as correct only if every instruction in it is satisfied, while instruction-level accuracy credits each instruction separately, so it is always equal to or higher than the prompt-level figure.

The loose variants apply normalizations before checking, such as removing Markdown asterisks or dropping a first line like “Sure, here is your answer”. These transformations reduce false negatives caused by harmless formatting, at the cost of occasionally accepting a response that a strict reader would reject.

MetricUnit of creditStrictnessTypical use
Prompt-level strictWhole promptExact outputHeadline score on leaderboards
Instruction-level strictSingle instructionExact outputDiagnosing which constraints fail
Prompt-level looseWhole promptNormalized outputReducing formatting false negatives
Instruction-level looseSingle instructionNormalized outputMost lenient upper bound
The four IFEval metrics compared

When comparing models, the same metric and the same evaluation harness must be used. Differences of several points can come purely from chat templates, system prompts or whether the loose variant was reported.

IFEval compared with other evaluation methods

IFEval occupies a specific niche: it measures compliance with precise constraints, not general helpfulness or knowledge. Its neighbors in model evaluation answer different questions, and a complete picture of a model needs several of them.

EvaluationWhat it measuresScoring methodMain weakness
IFEvalFollowing verifiable instructionsDeterministic code checksIgnores answer quality
MMLUBroad academic knowledgeMultiple-choice exact matchContamination and saturation
LLM-as-a-judgeOpen-ended answer qualityAnother model grades outputJudge bias and cost
LMArenaHuman preference between modelsPairwise votes and ratingsStyle can outweigh substance
How IFEval relates to neighboring evaluations

The deterministic design gives IFEval a clear advantage in reproducibility: two teams running the same model with the same settings should obtain nearly identical scores. Its counterpart weakness is narrow coverage, since real users give instructions that are rarely so easy to verify.

Interpreting IFEval scores in practice

An IFEval score is most useful as a signal of post-training quality, especially of supervised fine-tuning and preference optimization. Base models without instruction tuning score poorly, while well-aligned chat models typically reach prompt-level strict accuracy above 80%, and leading models report figures near 90%.

The following table gives a rough reading guide for prompt-level strict accuracy; exact thresholds vary with the harness and template.

Score rangeTypical model typeInterpretation
Below 40%Base or lightly tuned modelsInstructions are often ignored
40-70%Small or early chat modelsSimple constraints followed, combinations fail
70-85%Solid instruction-tuned modelsReliable for most formatting tasks
Above 85%Strong frontier modelsNear the benchmark ceiling
Rough interpretation of prompt-level strict accuracy

Instruction-level breakdowns are often more informative than the headline number. A model may handle keywords well but consistently miss exact word counts, which matters greatly for applications such as product descriptions with strict length limits.

Limitations and common pitfalls

IFEval is valuable but easy to over-interpret. Its small size and narrow constraint types mean that a high score does not prove that a model follows complex, real-world instructions.

  • Benchmark saturation: top models cluster near the ceiling, so small differences carry little meaning.
  • Overfitting risk: training on similar synthetic constraints can raise scores without improving general compliance.
  • Quality blindness: a response can satisfy every constraint and still be wrong, unhelpful or incoherent.
  • Checker brittleness: word and sentence counting rules can disagree with human judgment on edge cases.
  • Language coverage: the original set is English only, although multilingual adaptations exist.

For these reasons, IFEval is best used alongside harder instruction-following benchmarks, human evaluation and task-specific tests rather than as a standalone verdict.

In plain words

IFEval is like a test that checks whether someone follows the rules of an assignment. It does not ask whether the essay is brilliant. Suppose the teacher says “under 200 words, no commas.” A simple counter can see at once whether the student obeyed.

The key takeaway is that IFEval measures obedience to clear instructions. A computer can grade it without opinions. A high score means the model does what it is told about format and length. It does not prove the answers are correct or useful.

A real-world example

An engineering team at an online retailer is choosing between two open-weight language models to generate product descriptions. The descriptions must stay under 120 words, include the product name at least twice and end with a fixed call to action, because the content management system rejects anything else.

The team runs IFEval through its evaluation harness with the production chat template. Model A scores 78% prompt-level strict accuracy, model B scores 86%, and the instruction-level breakdown shows that model A often exceeds word limits. A follow-up test on 400 real product prompts confirms the pattern: model B passes 93% of descriptions without manual fixes, compared with 81% for model A, so the team adopts model B and keeps IFEval in its regression suite for future updates.