Automated essay scoring (AES) is the use of statistical models, machine learning or large language models to assign a score to extended written responses, such as student essays, exam answers and placement tests, by predicting the grade a trained human rater would give. Among AI applications in education, it is one of the oldest, with research systems dating back to the 1960s and operational use in large-scale testing since the late 1990s.
An AES system does not read for meaning the way a teacher does. It learns statistical relationships between text features and human scores from a set of already graded essays, then applies those relationships to new responses. Its value lies in speed, consistency and cost, and its main risks lie in construct validity, fairness and gaming.
How automated essay scoring produces a grade
Automated essay scoring produces a grade by mapping measurable properties of a text to a score scale learned from human-rated examples. The pipeline has three stages: training on rated essays, extracting a representation of each new essay, and predicting a score with a calibrated model.
Training usually requires several hundred to a few thousand essays per prompt that have been scored by at least two trained raters against a rubric. Prompt-specific models tend to agree more closely with humans, while generic models trained across many prompts are cheaper to deploy but weaker on content.
The table below summarizes the main stages and what each one contributes to the final score.
| Stage | What happens | Main risk |
|---|---|---|
| Rubric and rating | Humans score a training set against defined criteria | Inconsistent raters teach inconsistent patterns |
| Feature extraction | Text is converted into counts, embeddings or tokens | Features miss qualities the rubric values |
| Model training | Regression, classifier or fine-tuned model is fitted | Overfitting to one prompt or population |
| Calibration | Raw outputs are mapped to the official scale | Scores drift toward the middle of the scale |
| Validation | Agreement with held-out human scores is measured | Good averages hide subgroup errors |
Generations of scoring models
Scoring models have evolved from handcrafted surface features to neural networks and, more recently, to prompted or fine-tuned large language models. Each generation improved agreement on some dimensions while introducing new opacity or cost.
Early systems relied on essay length, word length, sentence counts and error frequencies, and essay length alone often correlated strongly with human scores. Later systems added syntactic parsing, discourse markers and latent semantic analysis to estimate organization and content relevance.
| Approach | Typical inputs | Strength | Weakness |
|---|---|---|---|
| Feature-based regression | Length, errors, vocabulary, syntax | Transparent and auditable | Rewards surface form over argument |
| Semantic similarity models | Vector comparison with high-scoring essays | Captures topical relevance | Penalizes original but valid answers |
| Neural sequence models | Word embeddings, recurrent or transformer layers | Higher agreement on trained prompts | Hard to explain individual scores |
| LLM-based scoring | Rubric, essay and few-shot examples in a prompt | Works with little training data | Unstable outputs and costly to validate |
Modern operational systems are often hybrid designs that combine interpretable features with neural scores, because test owners need both accuracy and the ability to justify a score to a candidate or regulator.
Measuring agreement with human raters
The quality of an AES engine is judged by how closely its scores agree with trained human raters, compared with how closely two humans agree with each other. A system is generally considered acceptable when its agreement with humans is at least as high as human-human agreement on the same essays.
The most widely used statistic is quadratic weighted kappa (QWK), which corrects for chance agreement and penalizes large disagreements more than small ones. Testing programs commonly treat a QWK around 0.70 or higher as a minimum threshold, and they also check the standardized mean difference between machine and human scores for each subgroup.
| Measure | What it shows | Typical expectation |
|---|---|---|
| Exact agreement | Share of identical scores | Depends on scale length |
| Adjacent agreement | Share of scores within one point | Usually above 90 percent on short scales |
| Quadratic weighted kappa | Chance-corrected agreement on ordinal scores | About 0.70 or more, close to human pairs |
| Pearson correlation | Linear association of scores | Similar to human-human correlation |
| Standardized mean difference | Systematic over- or underscoring of a group | Small, often below 0.15 in absolute value |
High overall agreement is necessary but not sufficient. A model can match humans on average while systematically underscoring non-native writers or overscoring long, formulaic essays.
Where automated essay scoring is used
Automated essay scoring is used wherever large volumes of writing must be graded quickly and consistently, from national examinations to classroom practice tools. The deployment model determines how much authority the machine score carries.
| Deployment model | Role of the machine | Typical setting |
|---|---|---|
| Sole scorer | Machine score is final | Low-stakes practice and formative feedback |
| Second rater | Replaces one of two human raters | High-stakes admissions and language tests |
| Check scorer | Flags essays where human and machine disagree | Quality control in large scoring centers |
| Feedback engine | Gives trait comments instead of a grade | Writing instruction and drafting support |
In high-stakes testing the safest pattern is human plus machine scoring with adjudication: when the two scores differ by more than a set margin, usually one point or more, a second human resolves the discrepancy. Formative tools overlap with an AI writing assistant or an AI tutor, but the scoring function remains distinct from generating text.
Validity, fairness and gaming risks
The central limitation of AES is that it measures correlates of quality rather than quality itself. A system may reward length, rare vocabulary and transition words even when an essay is factually wrong or logically incoherent.
- Gaming: test-takers who know the scoring features can inflate scores with long, polished but meaningless text.
- Construct underrepresentation: creativity, argument strength, factual accuracy and audience awareness are poorly captured.
- Subgroup bias: dialect features, second-language patterns and atypical structures can be scored unfairly.
- LLM instability: language model graders may give different scores to the same essay across runs or prompt wordings.
- Washback effects: teaching to the machine encourages formulaic writing that satisfies features rather than readers.
Responsible programs therefore publish validity evidence, monitor scores by subgroup, run adversarial tests with nonsense essays, keep humans in the loop for consequential decisions, and give candidates a route to request human rescoring.
In plain words
Automated essay scoring is a computer program that grades writing. It has studied thousands of essays that teachers already marked, so it learns which patterns usually go with high or low marks. When it sees a new essay, it guesses the mark a teacher would most likely give.
It is fast and never gets tired, but it does not truly understand the ideas. That is why serious exams pair it with human graders who check its work.
A real-world example
A state education department scores about 400,000 persuasive essays written by grade 8 students each spring. Previously, every essay was read by two human raters, which took six weeks and required hundreds of temporary scorers.
The department trains a prompt-specific AES model on 3,000 double-scored essays, reaching a quadratic weighted kappa of 0.78 against humans, compared with 0.76 between human pairs. It then uses the model as the second rater: each essay receives one human score and one machine score, and any pair that differs by two or more points on the six-point scale goes to an expert adjudicator.
About 6 percent of essays require adjudication. Subgroup checks reveal that English learners were slightly underscored, so the team retrains with more of their essays. The result is results delivered in three weeks at a substantially lower scoring cost, with human judgment still deciding every disputed case.