Stemming

On this page 7

Stemming is a rule-based text normalization technique that reduces inflected or derived words to a common stem by stripping suffixes and sometimes prefixes, so that “connect”, “connected”, “connecting” and “connection” all map to the same token connect. In Speech & language AI, stemming is one of the oldest preprocessing steps of natural language processing, used to shrink vocabularies and to improve recall in search and retrieval within the broader field of Artificial intelligence.

The stem does not need to be a dictionary word. The classic Porter stemmer turns “argue” and “argued” into argu, which is acceptable because the goal is consistent grouping, not linguistic correctness. That trade-off separates stemming from lemmatization, which returns a valid dictionary form.

How a stemmer reduces words to stems

A stemmer works by applying an ordered set of suffix-stripping rules to each token, usually without any knowledge of the surrounding sentence. Most algorithms check a condition on the remaining stem, such as its minimum length or the number of vowel-consonant sequences, before removing an ending, which prevents short words like “sing” from being cut to s.

The Porter algorithm, published in 1980, measures each candidate stem with a value called the measure m, the count of vowel-consonant sequences. It then runs five sequential steps: plural and past-tense endings first, then derivational suffixes such as “-ational” and “-fulness”, and finally cleanup of a trailing “e” or a double consonant. Because each rule fires at most once per step, the output is deterministic and fast, typically processing millions of tokens per second on a single core.

The table below traces a few English words through a Porter-style stemmer to show which rule fires and what remains.

Input wordRule appliedResulting stem
caressesPlural ending -sses to -sscaress
relationalSuffix -ational to -ate, then -ate removedrelat
hoppingRemove -ing, undouble consonanthop
generalizationChain of -ization and -al rulesgener
skyNo rule matchessky
Sample reductions by a Porter-style stemmer

Main stemming algorithms and their aggressiveness

Stemming algorithms differ mainly in how aggressively they strip endings, which controls the balance between merging too much and merging too little. A light stemmer removes only inflectional endings such as plurals, while a heavy stemmer also removes derivational suffixes and can collapse words of different meaning.

The following comparison covers the algorithms most often found in search engines and NLP libraries.

AlgorithmApproachAggressivenessTypical use
PorterFive ordered rule steps for EnglishModerateResearch baselines, legacy systems
Snowball (Porter2)Rule framework with improved English rulesModerateSearch engines, multilingual pipelines
Lancaster (Paice-Husk)Iterative rule table applied repeatedlyHighRecall-heavy retrieval
LovinsSingle pass with a long ending listHighHistorical interest
S-stemmerRemoves plural endings onlyVery lowPrecision-sensitive search
Common stemming algorithms compared

The Snowball framework also defines stemmers for more than 20 languages, including German, Russian and Finnish. For morphologically rich languages, light stemmers often outperform heavy ones, because aggressive stripping destroys case and compound information that carries meaning.

Stemming compared with lemmatization and subword tokenization

Stemming, lemmatization and subword tokenization all address vocabulary variation, but they rely on different knowledge and produce different outputs. Lemmatization uses a dictionary and part-of-speech information to return the canonical form, so “better” becomes “good” and “was” becomes “be”, which no suffix rule can achieve.

Modern language models usually skip both techniques and use subword tokenizers such as byte-pair encoding, which learn frequent fragments from data and let the network learn relations between word forms. The table summarizes the practical differences.

PropertyStemmingLemmatizationSubword tokenization
OutputTruncated stemDictionary lemmaLearned fragments
Knowledge neededSuffix rules onlyLexicon and POS tagsTraining corpus
SpeedVery fastSlowerFast after training
Handles irregular formsNoYesIndirectly
Readable outputOften notYesPartly
Normalization approaches and their properties

Stemming also differs from removing stop words, which discards frequent function words rather than changing word forms; the two steps are often combined in bag of words and keyword extraction pipelines.

Where stemming is used in practice

Stemming remains most valuable in lexical search and classical NLP pipelines, where exact token matching would otherwise miss relevant documents. Full-text engines apply the same stemmer at index time and query time, so a query for “running shoes” also matches a product described as “run shoe”.

The decision table below indicates when stemming is a sensible choice and when another approach serves better.

SituationRecommended choiceReason
BM25 keyword search in EnglishSnowball stemmerRaises recall at low cost
Legal or medical exact-term searchNo stemming or S-stemmerAvoids false matches
Topic modeling on small corporaStemming or lemmatizationReduces sparse vocabulary
Transformer embeddings or LLM inputNo stemmingTokenizer already handles forms
Highly inflected languageLanguage-specific light stemmerPreserves meaningful morphology
Choosing whether to apply stemming

In hybrid retrieval, stemming often still powers the lexical half while dense embeddings handle the semantic half, so the technique survives inside systems that also use neural models.

Limitations and common stemming errors

Stemmers make two characteristic mistakes: overstemming, where unrelated words collapse into the same stem, and understemming, where related words keep different stems. Both errors are a direct consequence of ignoring meaning and context.

  • Overstemming: “university” and “universe” both become univers, merging unrelated topics.
  • Understemming: “absorb” and “absorption” receive different stems, splitting one concept.
  • Irregular forms: “ran” and “run” or “mice” and “mouse” stay separate.
  • Named entities: a stemmer may truncate brand or person names, harming entity matching.
  • Mismatched configuration: different stemmers at index and query time silently break retrieval.

A practical safeguard is to evaluate retrieval metrics such as recall and nDCG with and without stemming on a representative query set, rather than assuming that stemming always helps. Many teams also keep an unstemmed copy of each field so that exact phrase queries remain possible.

In plain words

Stemming chops the endings off words so that different forms of the same word look identical to a computer. “Walks”, “walked” and “walking” all become “walk”. This helps a search box find a page about “walking” when someone typed “walked”.

The method is quick and simple, but it is blunt. It follows spelling rules, not meaning, so it sometimes glues unrelated words together or leaves a strange stump such as “argu”. The key point is that stemming trades some accuracy for broader matching.

A real-world example

An online hardware retailer runs keyword search over about 80,000 product descriptions. Customers searching for “drilling bits” found few results, because most listings used the phrase “drill bit”. The search team enabled a Snowball English stemmer on the description field at both index and query time, while keeping an unstemmed copy for quoted phrase searches.

Before the change, testers ran 500 logged queries and judged the top ten results. After the change, zero-result queries fell noticeably and recall on the test set improved, while precision dropped slightly for a few queries such as “universal clamp”, which began matching “universe” themed gifts. The team added a short exception list for those words and kept stemming enabled as the default.