Automated essay scoring

On this page 7

Automated essay scoring (AES) is the use of statistical models, machine learning or large language models to assign a score to extended written responses, such as student essays, exam answers and placement tests, by predicting the grade a trained human rater would give. Among AI applications in education, it is one of the oldest, with research systems dating back to the 1960s and operational use in large-scale testing since the late 1990s.

An AES system does not read for meaning the way a teacher does. It learns statistical relationships between text features and human scores from a set of already graded essays, then applies those relationships to new responses. Its value lies in speed, consistency and cost, and its main risks lie in construct validity, fairness and gaming.

How automated essay scoring produces a grade

Automated essay scoring produces a grade by mapping measurable properties of a text to a score scale learned from human-rated examples. The pipeline has three stages: training on rated essays, extracting a representation of each new essay, and predicting a score with a calibrated model.

Training usually requires several hundred to a few thousand essays per prompt that have been scored by at least two trained raters against a rubric. Prompt-specific models tend to agree more closely with humans, while generic models trained across many prompts are cheaper to deploy but weaker on content.

The table below summarizes the main stages and what each one contributes to the final score.

StageWhat happensMain risk
Rubric and ratingHumans score a training set against defined criteriaInconsistent raters teach inconsistent patterns
Feature extractionText is converted into counts, embeddings or tokensFeatures miss qualities the rubric values
Model trainingRegression, classifier or fine-tuned model is fittedOverfitting to one prompt or population
CalibrationRaw outputs are mapped to the official scaleScores drift toward the middle of the scale
ValidationAgreement with held-out human scores is measuredGood averages hide subgroup errors
Stages of a typical scoring pipeline

Generations of scoring models

Scoring models have evolved from handcrafted surface features to neural networks and, more recently, to prompted or fine-tuned large language models. Each generation improved agreement on some dimensions while introducing new opacity or cost.

Early systems relied on essay length, word length, sentence counts and error frequencies, and essay length alone often correlated strongly with human scores. Later systems added syntactic parsing, discourse markers and latent semantic analysis to estimate organization and content relevance.

ApproachTypical inputsStrengthWeakness
Feature-based regressionLength, errors, vocabulary, syntaxTransparent and auditableRewards surface form over argument
Semantic similarity modelsVector comparison with high-scoring essaysCaptures topical relevancePenalizes original but valid answers
Neural sequence modelsWord embeddings, recurrent or transformer layersHigher agreement on trained promptsHard to explain individual scores
LLM-based scoringRubric, essay and few-shot examples in a promptWorks with little training dataUnstable outputs and costly to validate
Main approaches to automated essay scoring

Modern operational systems are often hybrid designs that combine interpretable features with neural scores, because test owners need both accuracy and the ability to justify a score to a candidate or regulator.

Measuring agreement with human raters

The quality of an AES engine is judged by how closely its scores agree with trained human raters, compared with how closely two humans agree with each other. A system is generally considered acceptable when its agreement with humans is at least as high as human-human agreement on the same essays.

The most widely used statistic is quadratic weighted kappa (QWK), which corrects for chance agreement and penalizes large disagreements more than small ones. Testing programs commonly treat a QWK around 0.70 or higher as a minimum threshold, and they also check the standardized mean difference between machine and human scores for each subgroup.

MeasureWhat it showsTypical expectation
Exact agreementShare of identical scoresDepends on scale length
Adjacent agreementShare of scores within one pointUsually above 90 percent on short scales
Quadratic weighted kappaChance-corrected agreement on ordinal scoresAbout 0.70 or more, close to human pairs
Pearson correlationLinear association of scoresSimilar to human-human correlation
Standardized mean differenceSystematic over- or underscoring of a groupSmall, often below 0.15 in absolute value
Common evaluation measures and their purpose

High overall agreement is necessary but not sufficient. A model can match humans on average while systematically underscoring non-native writers or overscoring long, formulaic essays.

Where automated essay scoring is used

Automated essay scoring is used wherever large volumes of writing must be graded quickly and consistently, from national examinations to classroom practice tools. The deployment model determines how much authority the machine score carries.

Deployment modelRole of the machineTypical setting
Sole scorerMachine score is finalLow-stakes practice and formative feedback
Second raterReplaces one of two human ratersHigh-stakes admissions and language tests
Check scorerFlags essays where human and machine disagreeQuality control in large scoring centers
Feedback engineGives trait comments instead of a gradeWriting instruction and drafting support
Deployment models and the role of the machine score

In high-stakes testing the safest pattern is human plus machine scoring with adjudication: when the two scores differ by more than a set margin, usually one point or more, a second human resolves the discrepancy. Formative tools overlap with an AI writing assistant or an AI tutor, but the scoring function remains distinct from generating text.

Validity, fairness and gaming risks

The central limitation of AES is that it measures correlates of quality rather than quality itself. A system may reward length, rare vocabulary and transition words even when an essay is factually wrong or logically incoherent.

  • Gaming: test-takers who know the scoring features can inflate scores with long, polished but meaningless text.
  • Construct underrepresentation: creativity, argument strength, factual accuracy and audience awareness are poorly captured.
  • Subgroup bias: dialect features, second-language patterns and atypical structures can be scored unfairly.
  • LLM instability: language model graders may give different scores to the same essay across runs or prompt wordings.
  • Washback effects: teaching to the machine encourages formulaic writing that satisfies features rather than readers.

Responsible programs therefore publish validity evidence, monitor scores by subgroup, run adversarial tests with nonsense essays, keep humans in the loop for consequential decisions, and give candidates a route to request human rescoring.

In plain words

Automated essay scoring is a computer program that grades writing. It has studied thousands of essays that teachers already marked, so it learns which patterns usually go with high or low marks. When it sees a new essay, it guesses the mark a teacher would most likely give.

It is fast and never gets tired, but it does not truly understand the ideas. That is why serious exams pair it with human graders who check its work.

A real-world example

A state education department scores about 400,000 persuasive essays written by grade 8 students each spring. Previously, every essay was read by two human raters, which took six weeks and required hundreds of temporary scorers.

The department trains a prompt-specific AES model on 3,000 double-scored essays, reaching a quadratic weighted kappa of 0.78 against humans, compared with 0.76 between human pairs. It then uses the model as the second rater: each essay receives one human score and one machine score, and any pair that differs by two or more points on the six-point scale goes to an expert adjudicator.

About 6 percent of essays require adjudication. Subgroup checks reveal that English learners were slightly underscored, so the team retrains with more of their essays. The result is results delivered in three weeks at a substantially lower scoring cost, with human judgment still deciding every disputed case.