Stop words

On this page 8

Stop words are the highly frequent function words of a language, such as “the”, “of”, “and” or “is”, that a text processing system deliberately removes or down-weights because they carry little topical meaning on their own. In speech & language AI, stop word handling is a preprocessing decision: it shrinks vocabularies and sharpens keyword statistics in classic pipelines, yet it can destroy meaning in tasks where small words decide the interpretation.

No universal stop word list exists. Every list is a design choice tied to a task, a language and a domain, and modern neural models usually keep every word because they learn the importance of each token from context.

How stop word removal works in a text pipeline

Stop word removal works by comparing each token against a predefined or derived list and discarding the matches before later stages run. It normally follows tokenization and lowercasing and often precedes stemming or lemmatization, so the order of steps determines whether “The” and “the” are both caught.

The justification comes from Zipf’s law: word frequency falls roughly in inverse proportion to frequency rank. In typical English prose, the 100 most frequent words account for roughly 40 to 50 percent of all running tokens, yet they say almost nothing about what a document is about. Removing them reduces index size, speeds up counting and lets content words dominate statistics such as term frequency and TF-IDF weights.

The table below shows the main ways a stop word list is built and what each approach implies.

MethodHow it worksMain risk
Standard listUses a published general-purpose list of 100 to 600 wordsRemoves words that matter in the target domain
Frequency cutoffDrops the top N most frequent corpus tokensRemoves frequent domain terms such as “patient”
Document frequency thresholdDrops tokens present in more than a set share of documentsThreshold choice is arbitrary and corpus dependent
Domain curationExperts add boilerplate words like “herein” or “click”Costly to build and maintain
Statistical weightingKeeps all words but lowers weights through IDFFrequent words still add noise in small corpora
Common ways to build a stop word list

Typical categories of stop words

Stop words fall into a few grammatical categories, most of them closed-class function words that rarely change over time. Knowing the categories helps a practitioner decide which ones are safe to drop for a given task.

CategoryExamplesSafe to remove for topic tasks
Articles and determinersthe, a, an, this, thoseUsually yes
Prepositionsof, in, on, to, fromUsually yes, but they encode relations
Conjunctionsand, or, but, becausePartly; “but” signals contrast
Pronounshe, she, it, they, weYes for topics, no for coreference
Auxiliary verbsis, was, have, will, canYes for topics, no for tense or modality
Negationsnot, no, never, norAlmost never
Grammatical categories found in English stop word lists

Negations deserve special attention. Many default lists still include “not” and “no”, which turns “not good” into “good” and inverts the polarity of a sentence. Any pipeline feeding sentiment analysis or textual entailment should remove negations from the stop list before use.

Stop words across NLP tasks

Whether stop words should be removed depends on the task, because some tasks need only topical content while others need grammar. The decision table below summarizes common practice in NLP.

TaskRecommended handlingReason
Keyword extractionRemoveFunction words never qualify as keywords
Topic modelingRemove plus domain termsFrequent words otherwise dominate every topic
Bag of words classificationTest both settingsGains are often small and data dependent
Sentiment analysisKeep negations and intensifiers“Not” and “very” change polarity and strength
Question answeringKeep“Who”, “when” and “where” define the question
Machine translationKeepEvery word must be rendered in the output
Transformer fine-tuningKeepAttention learns token importance from context
Recommended stop word handling by task

For phrase-level features such as N-gram models, removing stop words creates false neighbors: “bank of England” becomes the bigram “bank England”. Phrase queries in search engines fail for the same reason, which is why exact-match search such as “to be or not to be” requires a full index.

Stop words in modern language models

Large language models and transformer encoders keep stop words because their subword tokenizers and attention layers depend on full, grammatical input. These models were pretrained on natural text, so stripping function words at inference creates a distribution shift that usually lowers accuracy.

Stop word ideas still survive in neighboring places. Retrieval systems based on BM25 often apply stop lists or rely on low IDF weights; embedding-based search keeps every token; and analytics layers that build word clouds or keyword reports still filter function words so that results remain readable. In speech pipelines, transcripts from speech-to-text are usually normalized first and only then filtered, because spoken fillers such as “um” and “uh” behave like extra stop words.

The comparison below contrasts stop word removal with related normalization steps that often appear in the same pipeline.

StepWhat it changesTypical effect on vocabulary
Stop word removalDeletes whole frequent tokensRemoves a few hundred types but many tokens
StemmingCuts suffixes by ruleMerges inflected forms crudely
LemmatizationMaps words to dictionary formsMerges forms with grammatical accuracy
Text normalizationExpands numbers, dates and abbreviationsStandardizes spelling of the same idea
Stop word removal compared with related preprocessing steps

Multilingual and domain-specific stop lists

Each language needs its own stop list, because function words differ in form and in how they attach to other words. In agglutinative languages such as Turkish or Finnish, many grammatical functions are suffixes rather than separate words, so morphological analysis matters more than a word list. Chinese and Japanese text first needs word segmentation, since tokens are not separated by spaces.

Domain lists extend general ones with corpus-specific noise words. Legal text adds “hereby” and “whereas”, clinical notes may add “patient” or “denies” for some analyses, and web text adds “click”, “share” or “cookie”. Such additions should be validated on held-out data, because a word that is noise for one question may be the signal for another.

Common mistakes with stop words

Most stop word problems come from using a default list without inspecting it. The most frequent errors are listed below.

  • Removing negations before sentiment or entailment tasks, which flips meaning.
  • Destroying named entities such as “The Who”, “It” as a film title or “US” when lowercased into “us”.
  • Applying an English list to other languages, which leaves foreign function words untouched.
  • Filtering input to pretrained transformers, which feeds them ungrammatical text they never saw in training.
  • Mismatched lists between indexing and querying, which makes some queries impossible to match.

In plain words

Stop words are the small, common words that glue sentences together, like “the”, “is” and “of”. They appear everywhere, so they say very little about what a particular text is about. Removing them is like skimming a page for the important nouns and verbs, which helps a computer quickly spot the main topics.

The catch is that small words sometimes carry big meaning. “Not” turns a compliment into a complaint, and “who” or “when” defines what a question asks. Removing stop words helps simple keyword tools but usually hurts modern AI models that read whole sentences.

A real-world example

An online retailer wants a weekly report of the topics customers mention in 40,000 product reviews. An analyst builds a topic model on the raw text and finds that every topic is dominated by “the”, “and”, “it” and “was”. After applying a standard English stop list plus domain words such as “product” and “bought”, the vocabulary falls by about a third and clear topics appear: delivery delays, battery life and sizing problems.

The same team then reuses the filtered text to train a sentiment classifier and sees accuracy drop, because reviews such as “not comfortable” became “comfortable”. The analyst removes “not”, “no” and “never” from the stop list for that model and, for the production version, fine-tunes a transformer on the unfiltered reviews. Keeping stop words for sentiment and removing them for topic discovery gives the team reliable results on both tasks.