Stemming is a rule-based text normalization technique that reduces inflected or derived words to a common stem by stripping suffixes and sometimes prefixes, so that “connect”, “connected”, “connecting” and “connection” all map to the same token connect. In Speech & language AI, stemming is one of the oldest preprocessing steps of natural language processing, used to shrink vocabularies and to improve recall in search and retrieval within the broader field of Artificial intelligence.
The stem does not need to be a dictionary word. The classic Porter stemmer turns “argue” and “argued” into argu, which is acceptable because the goal is consistent grouping, not linguistic correctness. That trade-off separates stemming from lemmatization, which returns a valid dictionary form.
How a stemmer reduces words to stems
A stemmer works by applying an ordered set of suffix-stripping rules to each token, usually without any knowledge of the surrounding sentence. Most algorithms check a condition on the remaining stem, such as its minimum length or the number of vowel-consonant sequences, before removing an ending, which prevents short words like “sing” from being cut to s.
The Porter algorithm, published in 1980, measures each candidate stem with a value called the measure m, the count of vowel-consonant sequences. It then runs five sequential steps: plural and past-tense endings first, then derivational suffixes such as “-ational” and “-fulness”, and finally cleanup of a trailing “e” or a double consonant. Because each rule fires at most once per step, the output is deterministic and fast, typically processing millions of tokens per second on a single core.
The table below traces a few English words through a Porter-style stemmer to show which rule fires and what remains.
| Input word | Rule applied | Resulting stem |
|---|---|---|
| caresses | Plural ending -sses to -ss | caress |
| relational | Suffix -ational to -ate, then -ate removed | relat |
| hopping | Remove -ing, undouble consonant | hop |
| generalization | Chain of -ization and -al rules | gener |
| sky | No rule matches | sky |
Main stemming algorithms and their aggressiveness
Stemming algorithms differ mainly in how aggressively they strip endings, which controls the balance between merging too much and merging too little. A light stemmer removes only inflectional endings such as plurals, while a heavy stemmer also removes derivational suffixes and can collapse words of different meaning.
The following comparison covers the algorithms most often found in search engines and NLP libraries.
| Algorithm | Approach | Aggressiveness | Typical use |
|---|---|---|---|
| Porter | Five ordered rule steps for English | Moderate | Research baselines, legacy systems |
| Snowball (Porter2) | Rule framework with improved English rules | Moderate | Search engines, multilingual pipelines |
| Lancaster (Paice-Husk) | Iterative rule table applied repeatedly | High | Recall-heavy retrieval |
| Lovins | Single pass with a long ending list | High | Historical interest |
| S-stemmer | Removes plural endings only | Very low | Precision-sensitive search |
The Snowball framework also defines stemmers for more than 20 languages, including German, Russian and Finnish. For morphologically rich languages, light stemmers often outperform heavy ones, because aggressive stripping destroys case and compound information that carries meaning.
Stemming compared with lemmatization and subword tokenization
Stemming, lemmatization and subword tokenization all address vocabulary variation, but they rely on different knowledge and produce different outputs. Lemmatization uses a dictionary and part-of-speech information to return the canonical form, so “better” becomes “good” and “was” becomes “be”, which no suffix rule can achieve.
Modern language models usually skip both techniques and use subword tokenizers such as byte-pair encoding, which learn frequent fragments from data and let the network learn relations between word forms. The table summarizes the practical differences.
| Property | Stemming | Lemmatization | Subword tokenization |
|---|---|---|---|
| Output | Truncated stem | Dictionary lemma | Learned fragments |
| Knowledge needed | Suffix rules only | Lexicon and POS tags | Training corpus |
| Speed | Very fast | Slower | Fast after training |
| Handles irregular forms | No | Yes | Indirectly |
| Readable output | Often not | Yes | Partly |
Stemming also differs from removing stop words, which discards frequent function words rather than changing word forms; the two steps are often combined in bag of words and keyword extraction pipelines.
Where stemming is used in practice
Stemming remains most valuable in lexical search and classical NLP pipelines, where exact token matching would otherwise miss relevant documents. Full-text engines apply the same stemmer at index time and query time, so a query for “running shoes” also matches a product described as “run shoe”.
The decision table below indicates when stemming is a sensible choice and when another approach serves better.
| Situation | Recommended choice | Reason |
|---|---|---|
| BM25 keyword search in English | Snowball stemmer | Raises recall at low cost |
| Legal or medical exact-term search | No stemming or S-stemmer | Avoids false matches |
| Topic modeling on small corpora | Stemming or lemmatization | Reduces sparse vocabulary |
| Transformer embeddings or LLM input | No stemming | Tokenizer already handles forms |
| Highly inflected language | Language-specific light stemmer | Preserves meaningful morphology |
In hybrid retrieval, stemming often still powers the lexical half while dense embeddings handle the semantic half, so the technique survives inside systems that also use neural models.
Limitations and common stemming errors
Stemmers make two characteristic mistakes: overstemming, where unrelated words collapse into the same stem, and understemming, where related words keep different stems. Both errors are a direct consequence of ignoring meaning and context.
- Overstemming: “university” and “universe” both become
univers, merging unrelated topics. - Understemming: “absorb” and “absorption” receive different stems, splitting one concept.
- Irregular forms: “ran” and “run” or “mice” and “mouse” stay separate.
- Named entities: a stemmer may truncate brand or person names, harming entity matching.
- Mismatched configuration: different stemmers at index and query time silently break retrieval.
A practical safeguard is to evaluate retrieval metrics such as recall and nDCG with and without stemming on a representative query set, rather than assuming that stemming always helps. Many teams also keep an unstemmed copy of each field so that exact phrase queries remain possible.
In plain words
Stemming chops the endings off words so that different forms of the same word look identical to a computer. “Walks”, “walked” and “walking” all become “walk”. This helps a search box find a page about “walking” when someone typed “walked”.
The method is quick and simple, but it is blunt. It follows spelling rules, not meaning, so it sometimes glues unrelated words together or leaves a strange stump such as “argu”. The key point is that stemming trades some accuracy for broader matching.
A real-world example
An online hardware retailer runs keyword search over about 80,000 product descriptions. Customers searching for “drilling bits” found few results, because most listings used the phrase “drill bit”. The search team enabled a Snowball English stemmer on the description field at both index and query time, while keeping an unstemmed copy for quoted phrase searches.
Before the change, testers ran 500 logged queries and judged the top ten results. After the change, zero-result queries fell noticeably and recall on the test set improved, while precision dropped slightly for a few queries such as “universal clamp”, which began matching “universe” themed gifts. The team added a short exception list for those words and kept stemming enabled as the default.