IFEval (Instruction-Following Evaluation) is a benchmark that measures how reliably a large language model follows explicit, verifiable instructions, such as writing more than 300 words, answering in JSON or avoiding commas. In model evaluation for artificial intelligence, it is one of the standard tests of instruction-following ability, separate from knowledge or reasoning skill.
Its defining feature is that every instruction can be checked automatically by a short program. This removes the need for human raters or an LLM-as-a-judge and makes scores reproducible and cheap to compute, which explains its wide adoption in model reports and leaderboards since its release by Google researchers in 2023.
How IFEval tests instruction following
IFEval works by pairing ordinary writing prompts with one or more verifiable constraints and then checking the model’s response with deterministic rules. The dataset contains about 500 prompts built from 25 types of verifiable instructions. A typical prompt reads: write a blog post about a trip to Japan, use fewer than 300 words and do not use any commas.
Each instruction type has its own checking function. A length constraint counts words or sentences, a format constraint parses the output as JSON or counts Markdown bullet points, and a keyword constraint searches for required or forbidden words. The content quality of the answer is not judged; only compliance with the stated constraints is scored.
The table below lists the main instruction groups and how each is verified.
| Instruction group | Example constraint | Automatic check |
|---|---|---|
| Keywords | Include the word “budget” at least twice | Count of keyword occurrences |
| Length constraints | Answer in fewer than 150 words | Word, sentence or paragraph count |
| Detectable format | Wrap the whole response in JSON | Parse output as valid JSON |
| Change case | Write everything in lowercase | Compare text with its lowercase form |
| Punctuation | Do not use any commas | Search for the forbidden character |
| Start and end | Finish with a fixed closing phrase | String match at the end of output |
Strict and loose accuracy metrics
IFEval reports four accuracy figures, formed by combining two levels of aggregation with two levels of strictness. Prompt-level accuracy counts a prompt as correct only if every instruction in it is satisfied, while instruction-level accuracy credits each instruction separately, so it is always equal to or higher than the prompt-level figure.
The loose variants apply normalizations before checking, such as removing Markdown asterisks or dropping a first line like “Sure, here is your answer”. These transformations reduce false negatives caused by harmless formatting, at the cost of occasionally accepting a response that a strict reader would reject.
| Metric | Unit of credit | Strictness | Typical use |
|---|---|---|---|
| Prompt-level strict | Whole prompt | Exact output | Headline score on leaderboards |
| Instruction-level strict | Single instruction | Exact output | Diagnosing which constraints fail |
| Prompt-level loose | Whole prompt | Normalized output | Reducing formatting false negatives |
| Instruction-level loose | Single instruction | Normalized output | Most lenient upper bound |
When comparing models, the same metric and the same evaluation harness must be used. Differences of several points can come purely from chat templates, system prompts or whether the loose variant was reported.
IFEval compared with other evaluation methods
IFEval occupies a specific niche: it measures compliance with precise constraints, not general helpfulness or knowledge. Its neighbors in model evaluation answer different questions, and a complete picture of a model needs several of them.
| Evaluation | What it measures | Scoring method | Main weakness |
|---|---|---|---|
| IFEval | Following verifiable instructions | Deterministic code checks | Ignores answer quality |
| MMLU | Broad academic knowledge | Multiple-choice exact match | Contamination and saturation |
| LLM-as-a-judge | Open-ended answer quality | Another model grades output | Judge bias and cost |
| LMArena | Human preference between models | Pairwise votes and ratings | Style can outweigh substance |
The deterministic design gives IFEval a clear advantage in reproducibility: two teams running the same model with the same settings should obtain nearly identical scores. Its counterpart weakness is narrow coverage, since real users give instructions that are rarely so easy to verify.
Interpreting IFEval scores in practice
An IFEval score is most useful as a signal of post-training quality, especially of supervised fine-tuning and preference optimization. Base models without instruction tuning score poorly, while well-aligned chat models typically reach prompt-level strict accuracy above 80%, and leading models report figures near 90%.
The following table gives a rough reading guide for prompt-level strict accuracy; exact thresholds vary with the harness and template.
| Score range | Typical model type | Interpretation |
|---|---|---|
| Below 40% | Base or lightly tuned models | Instructions are often ignored |
| 40-70% | Small or early chat models | Simple constraints followed, combinations fail |
| 70-85% | Solid instruction-tuned models | Reliable for most formatting tasks |
| Above 85% | Strong frontier models | Near the benchmark ceiling |
Instruction-level breakdowns are often more informative than the headline number. A model may handle keywords well but consistently miss exact word counts, which matters greatly for applications such as product descriptions with strict length limits.
Limitations and common pitfalls
IFEval is valuable but easy to over-interpret. Its small size and narrow constraint types mean that a high score does not prove that a model follows complex, real-world instructions.
- Benchmark saturation: top models cluster near the ceiling, so small differences carry little meaning.
- Overfitting risk: training on similar synthetic constraints can raise scores without improving general compliance.
- Quality blindness: a response can satisfy every constraint and still be wrong, unhelpful or incoherent.
- Checker brittleness: word and sentence counting rules can disagree with human judgment on edge cases.
- Language coverage: the original set is English only, although multilingual adaptations exist.
For these reasons, IFEval is best used alongside harder instruction-following benchmarks, human evaluation and task-specific tests rather than as a standalone verdict.
In plain words
IFEval is like a test that checks whether someone follows the rules of an assignment. It does not ask whether the essay is brilliant. Suppose the teacher says “under 200 words, no commas.” A simple counter can see at once whether the student obeyed.
The key takeaway is that IFEval measures obedience to clear instructions. A computer can grade it without opinions. A high score means the model does what it is told about format and length. It does not prove the answers are correct or useful.
A real-world example
An engineering team at an online retailer is choosing between two open-weight language models to generate product descriptions. The descriptions must stay under 120 words, include the product name at least twice and end with a fixed call to action, because the content management system rejects anything else.
The team runs IFEval through its evaluation harness with the production chat template. Model A scores 78% prompt-level strict accuracy, model B scores 86%, and the instruction-level breakdown shows that model A often exceeds word limits. A follow-up test on 400 real product prompts confirms the pattern: model B passes 93% of descriptions without manual fixes, compared with 81% for model A, so the team adopts model B and keeps IFEval in its regression suite for future updates.