Bag of words

Also known as: BoW

On this page 7

Bag of words (BoW) is a text representation that describes a document by which words it contains and how often, while discarding word order and grammar. In speech & language AI, it is one of the oldest and most widely taught ways to turn free text into fixed-length numeric vectors that statistical models can process.

Although neural embeddings and transformers have replaced it in most state-of-the-art systems, the bag of words model remains a strong, transparent baseline for text classification, spam filtering, sentiment analysis and information retrieval, and it underpins classic weighting schemes such as TF-IDF and the vector space model.

How the bag of words model builds a vector

A bag of words model converts text into numbers in three steps: tokenization, vocabulary construction and counting. The text is first split into tokens, usually words, and often normalized by lowercasing, removing punctuation and optionally applying stemming or lemmatization. The model then collects every distinct token in the training corpus into a vocabulary, assigning each one a fixed index.

Each document becomes a vector whose length equals the vocabulary size. The value at each position records how the corresponding word appears in that document. Because a typical vocabulary contains tens of thousands to millions of entries while a single document uses only a few hundred of them, the resulting vectors are extremely sparse, and implementations store them in sparse matrix formats rather than dense arrays.

The following table traces the pipeline for two short sentences, showing that different word orders produce identical vectors.

StepSentence ASentence B
Raw textThe dog bit the manThe man bit the dog
Tokensthe, dog, bit, the, manthe, man, bit, the, dog
Vocabularybit, dog, man, thebit, dog, man, the
Count vector[1, 1, 1, 2][1, 1, 1, 2]
Bag of words representation of two short sentences

The example exposes the defining property of the approach: the representation captures lexical content but not meaning carried by structure. Who bit whom is lost, which is acceptable for some tasks and fatal for others.

Weighting schemes and common variants

The value stored for each word is a design choice, and the chosen weighting scheme often matters more than the classifier that follows. Raw counts favor long documents and frequent function words, so practitioners usually transform them before training a model.

VariantValue storedTypical use
Binary presence1 if the word occurs, otherwise 0Short texts, Bernoulli naive Bayes
Raw term frequencyNumber of occurrencesMultinomial naive Bayes
Normalized frequencyCount divided by document lengthDocuments of very different lengths
TF-IDFFrequency times inverse document frequencySearch ranking, linear classifiers
Log-scaled frequency1 + log of the countDamping repeated terms
Typical ways to fill a bag of words vector

TF-IDF is the most common refinement. It multiplies term frequency by the logarithm of the total number of documents divided by the number of documents containing the word, so rare, distinctive terms gain weight while ubiquitous words such as “the” approach zero. Probabilistic ranking functions such as BM25 extend this idea with saturation and length normalization and still serve as strong retrieval baselines.

Two structural variants also appear frequently. The bag of n-grams adds sequences of two or three adjacent tokens to the vocabulary, recovering local word order such as “not good” at the cost of a much larger feature space. The hashing trick maps tokens to a fixed number of buckets with a hash function, which removes the need to store a vocabulary and suits streaming data, but it introduces occasional collisions between unrelated words.

Bag of words sits at the simplest end of a spectrum of text representations that runs from sparse counts to contextual embeddings. Comparing it with its neighbors in speech and language processing clarifies when it is still the right tool.

RepresentationCaptures word orderCaptures similarity between wordsDimensionality
Bag of wordsNoNoVocabulary size, sparse
N-gram featuresLocally, within the windowNoGrows rapidly with n
Topic modelsNoThrough shared topicsNumber of topics
Static embeddings such as word2vecNo, when averagedYesTypically 100 to 300 dense
Contextual transformer embeddingsYesYes, context dependentTypically 768 or more dense
Bag of words versus other text representations

The central weakness is that bag of words treats every word as an independent, orthogonal dimension. The vectors for “car” and “automobile” share nothing, so a document about one does not match a query about the other unless synonyms are added manually. Dense embeddings solve this by placing semantically related words close together, which is why they dominate semantic textual similarity and question answering.

Topic modeling methods such as latent Dirichlet allocation actually take a bag of words matrix as input, so the two concepts are complementary rather than competing. Many classic pipelines also remove stop words before counting to reduce noise.

When to use bag of words in practice

Bag of words remains a sound choice when data is limited, interpretability matters or compute is constrained. Paired with logistic regression, a linear support vector machine or naive Bayes, it trains in seconds on a laptop and exposes directly inspectable feature weights, which helps auditors and domain experts understand why a decision was made.

SituationRecommended choiceReason
Topic or spam classification with clear keywordsBag of words with TF-IDFKeywords carry most of the signal
Need for a fast baseline before deep learningBag of words with a linear modelSets a reference score cheaply
Negation and sarcasm drive the labelBag of n-grams or a transformerWord order changes the meaning
Queries use different words than documentsDense embeddings or hybrid retrievalVocabulary mismatch defeats exact matching
Strict explainability requirementsBag of words with sparse weightsEach feature is a readable word
Decision guide for choosing bag of words

A practical rule is to always build a bag of words baseline first. If a large neural model improves on it by only one or two percentage points, the simpler system may be preferable once serving cost, latency and maintenance are considered.

Limitations and common mistakes

Most problems with bag of words come from ignored structure and careless preprocessing. The following points summarize the failure modes that experienced practitioners watch for.

  • Loss of word order merges sentences with opposite meanings, such as “not bad, actually good” and “not good, actually bad”.
  • Vocabulary leakage occurs when the vocabulary or IDF statistics are fitted on test data, which inflates evaluation scores.
  • Out-of-vocabulary words in new text are silently dropped, so models degrade when terminology drifts.
  • Aggressive stop word removal can delete negations such as “not” that sentiment analysis depends on.
  • Unbounded vocabularies consume memory; capping features by minimum document frequency usually costs little accuracy.
  • Morphologically rich languages fragment one lemma into many surface forms, so lemmatization becomes more important than in English.

None of these issues makes the model obsolete, but each one should be tested explicitly during evaluation rather than assumed away.

A plain-language summary

Imagine cutting a letter into single words, throwing them into a bag and then counting what is inside. You would know that the word “refund” appears three times and “angry” appears twice, but you would no longer know the order of the sentences. That is exactly what a bag of words does with text.

This sounds crude, yet it works surprisingly well for many jobs. A computer can often tell a complaint from a thank-you note, or a sports article from a cooking recipe, just by looking at which words show up most. Its weakness is that it cannot understand phrases where order changes the meaning, so modern systems often add more advanced techniques on top.

An example from practice

A mid-sized online retailer receives about 4,000 customer emails per day and wants to route each one to billing, shipping, returns or technical support. The data team exports 60,000 previously labeled emails, lowercases them, keeps words and two-word phrases that occur in at least five messages, and obtains a vocabulary of roughly 40,000 features weighted with TF-IDF.

A logistic regression classifier trained on these vectors reaches about 91 percent accuracy on a held-out month of emails, and training takes under a minute. A fine-tuned transformer later scores about 93 percent but requires GPU serving. Because the feature weights show that words such as “tracking”, “courier” and “delayed” drive the shipping category, the support managers trust the system, and the team keeps the bag of words model in production while using the transformer only for messages the simple model classifies with low confidence.