N-gram

Also known as: n-gram language model

On this page 7

An N-gram is a contiguous sequence of n items, usually words, characters or subword tokens, taken from a text or a speech transcript. In Speech & language AI, the term names both the unit itself and the n-gram language model, which estimates the probability of the next token from the previous n-1 tokens. The idea is simple, yet it powered statistical speech recognition, machine translation and spelling correction for decades, and it remains a standard tool for evaluation metrics, data deduplication and fast baselines in artificial intelligence.

N-grams are named by their length: a unigram has one item, a bigram two, a trigram three, and longer sequences are called 4-grams, 5-grams and so on. The choice of n sets the central trade-off of the method: more context improves prediction, but it also makes each specific sequence rarer in the training data.

How an n-gram language model estimates probability

An n-gram language model works by applying the Markov assumption: the probability of a word depends only on a fixed window of preceding words, not on the whole history. The probability of a full sentence is then the product of conditional probabilities, each computed from a short window. A trigram model, for example, predicts the word after “turn on the” from counts of every observed continuation of that exact three-word prefix’s last two words, “on the”.

The basic estimate is maximum likelihood estimation: count how often a sequence occurs and divide by how often its prefix occurs. If “on the” appears 10,000 times and “on the table” appears 300 times, the model assigns P(table | on the) = 0.03. Training therefore reduces to counting, which is why n-gram models scale to corpora of billions of words on ordinary hardware.

The following table shows how the same sentence decomposes into n-grams of different orders and how much context each order captures.

OrderExample unitsContext used per predictionTypical use
Unigram (n=1)the, cat, satNoneWord frequency, bag of words features
Bigram (n=2)the cat, cat satOne previous wordSimple spelling and autocomplete models
Trigram (n=3)the cat sat, cat sat onTwo previous wordsClassic speech recognition decoders
4-gram and 5-gramthe cat sat onThree or four previous wordsLarge-vocabulary recognition and translation
N-gram orders for the sentence “the cat sat on the mat”

Sentence boundaries are handled with special start and end symbols, so the model also learns which words tend to open or close an utterance. Without them, probabilities at the edges of a sentence would be undefined or biased.

Sparsity, smoothing and backoff

The defining weakness of n-gram models is data sparsity: most valid word sequences never appear in any training corpus, so raw counts assign them a probability of zero. A single unseen trigram would make the probability of an entire sentence zero, which is unusable for ranking recognition or translation hypotheses.

Smoothing redistributes a small share of probability mass from seen sequences to unseen ones. Backoff and interpolation add a second mechanism: when a long n-gram has too little evidence, the model falls back to a shorter one. The table below compares the main techniques.

MethodCore ideaStrengthWeakness
Add-one (Laplace)Add 1 to every countTrivial to implementOverestimates rare events badly
Good-TuringRe-estimate counts from count-of-countsPrincipled mass for unseen eventsUnstable for high counts
Katz backoffUse lower order only when count is zeroClear fallback logicRequires discounting to stay normalized
Interpolated Kneser-NeyWeight lower orders by continuation diversityBest accuracy among count-based methodsMore complex bookkeeping
Fixed-weight web-scale backoffFixed multiplier for lower orders, no normalizationScales to web-size corporaScores are not true probabilities
Common smoothing methods for n-gram models

Modified Kneser-Ney smoothing became the practical standard because it models how many different contexts a word appears in, not only how often it appears. The word “Francisco” is frequent but almost always follows “San”, so Kneser-Ney gives it a low continuation probability in new contexts, which matches linguistic reality.

N-grams in speech and language pipelines

N-grams serve two distinct roles: as a probabilistic language model and as a feature or matching unit. Both roles appear across the neighboring tasks of this field, from speech-to-text to text classification and evaluation.

  • Speech-to-text decoding: an n-gram model, often compiled into a weighted finite-state transducer, combines with the acoustic model to rank word hypotheses, and it is still common for shallow fusion and domain biasing.
  • Text classification and sentiment analysis: word and character n-grams extend bag of words features with local word order, so “not good” differs from “good”.
  • Language identification: character n-gram profiles of 1 to 5 characters identify a language from only a few words.
  • Evaluation metrics: BLEU and ROUGE measure n-gram overlap between system output and reference texts.
  • Data curation: n-gram overlap detects duplicated training data and benchmark contamination in large language model corpora.

The model quality of an n-gram language model is usually reported as perplexity, the exponentiated average negative log-probability per token on held-out text. Lower perplexity means the model is less surprised by real text, although perplexity values are comparable only when vocabulary and tokenization are identical.

How n-gram models compare with neural language models

N-gram models differ from neural language models mainly in how they generalize. An n-gram model treats every word as an unrelated symbol, so knowing “drink coffee” says nothing about “drink tea”. Neural models learn dense word representations, as word2vec first popularized, and share statistical strength across similar words.

PropertyN-gram language modelNeural language model
Context lengthFixed, typically 2 to 5 tokensHundreds to millions of tokens
GeneralizationExact sequence matching onlySimilarity through learned embeddings
Training costCounting, CPU hoursGradient training, GPU days or more
Inference speedConstant-time table lookupMatrix computation per token
InterpretabilityEvery probability traceable to countsDistributed and opaque
Memory growthGrows with distinct n-gramsFixed by parameter count
N-gram models versus neural language models

The practical result is that neural models win decisively on fluency and long-range coherence, while n-gram models remain attractive where latency, transparency or tiny footprints matter, such as on-device keyboards, embedded voice commands and rapid domain adaptation.

Choosing n and avoiding common mistakes

The right value of n depends on corpus size, unit type and task. Word models rarely benefit beyond 5-grams, while character models use longer windows because individual characters carry little information. The decision table summarizes typical choices.

ScenarioRecommended nReason
Small domain corpus under 1 million wordsBigram or trigramLonger orders are almost entirely unseen
Large-vocabulary speech recognition4-gram or 5-gram with pruningEnough data to support longer context
Language identificationCharacter 1 to 5Captures spelling patterns per language
Text classification featuresWord 1 to 2 or 3Adds local word order without exploding features
Typical choices of n by scenario

Practitioners repeatedly run into the same failure modes. The most frequent ones are listed below.

  • Skipping smoothing, which produces zero probabilities and crashes decoders or log calculations.
  • Mismatched text normalization between training and use, so “Dr.” and “doctor” become different tokens.
  • Unpruned models that consume gigabytes of memory; entropy-based pruning often removes most entries with little accuracy loss.
  • Comparing perplexity across vocabularies, which yields meaningless rankings.
  • Treating high n-gram overlap scores as proof of quality, although paraphrases with correct meaning can score low.

A plain-language summary

An N-gram is simply a short run of words that appear next to each other, such as “good morning” or “thank you very much”. By counting how often such runs occur in large amounts of text, a computer can guess which word is likely to come next.

It works much like a person finishing a familiar phrase: hearing “once upon a”, most people expect “time”. The method only looks at the last few words, so it is fast and predictable but has a short memory. Modern AI systems remember far more context, yet n-grams are still used wherever speed and simplicity matter.

An example from practice

A hospital deploys a speech-to-text system so that radiologists can dictate reports. The general-purpose recognizer often confuses rare clinical terms, for example writing “plural fusion” instead of “pleural effusion”. The engineering team trains a domain 4-gram model with Kneser-Ney smoothing on about 2 million words of anonymized past reports, which takes under an hour on a single server.

The team interpolates the domain model with the general model and uses it to rescore recognition hypotheses. Because the domain counts strongly favor “pleural effusion” after phrases such as “small left”, the decoder now prefers the correct term. On a held-out set of 500 dictations, the word error rate drops from about 9 percent to about 6 percent, and the n-gram model adds only a few milliseconds of latency per utterance, so radiologists notice the improvement without any slowdown.