Bag of words (BoW) is a text representation that describes a document by which words it contains and how often, while discarding word order and grammar. In speech & language AI, it is one of the oldest and most widely taught ways to turn free text into fixed-length numeric vectors that statistical models can process.
Although neural embeddings and transformers have replaced it in most state-of-the-art systems, the bag of words model remains a strong, transparent baseline for text classification, spam filtering, sentiment analysis and information retrieval, and it underpins classic weighting schemes such as TF-IDF and the vector space model.
How the bag of words model builds a vector
A bag of words model converts text into numbers in three steps: tokenization, vocabulary construction and counting. The text is first split into tokens, usually words, and often normalized by lowercasing, removing punctuation and optionally applying stemming or lemmatization. The model then collects every distinct token in the training corpus into a vocabulary, assigning each one a fixed index.
Each document becomes a vector whose length equals the vocabulary size. The value at each position records how the corresponding word appears in that document. Because a typical vocabulary contains tens of thousands to millions of entries while a single document uses only a few hundred of them, the resulting vectors are extremely sparse, and implementations store them in sparse matrix formats rather than dense arrays.
The following table traces the pipeline for two short sentences, showing that different word orders produce identical vectors.
| Step | Sentence A | Sentence B |
|---|---|---|
| Raw text | The dog bit the man | The man bit the dog |
| Tokens | the, dog, bit, the, man | the, man, bit, the, dog |
| Vocabulary | bit, dog, man, the | bit, dog, man, the |
| Count vector | [1, 1, 1, 2] | [1, 1, 1, 2] |
The example exposes the defining property of the approach: the representation captures lexical content but not meaning carried by structure. Who bit whom is lost, which is acceptable for some tasks and fatal for others.
Weighting schemes and common variants
The value stored for each word is a design choice, and the chosen weighting scheme often matters more than the classifier that follows. Raw counts favor long documents and frequent function words, so practitioners usually transform them before training a model.
| Variant | Value stored | Typical use |
|---|---|---|
| Binary presence | 1 if the word occurs, otherwise 0 | Short texts, Bernoulli naive Bayes |
| Raw term frequency | Number of occurrences | Multinomial naive Bayes |
| Normalized frequency | Count divided by document length | Documents of very different lengths |
| TF-IDF | Frequency times inverse document frequency | Search ranking, linear classifiers |
| Log-scaled frequency | 1 + log of the count | Damping repeated terms |
TF-IDF is the most common refinement. It multiplies term frequency by the logarithm of the total number of documents divided by the number of documents containing the word, so rare, distinctive terms gain weight while ubiquitous words such as “the” approach zero. Probabilistic ranking functions such as BM25 extend this idea with saturation and length normalization and still serve as strong retrieval baselines.
Two structural variants also appear frequently. The bag of n-grams adds sequences of two or three adjacent tokens to the vocabulary, recovering local word order such as “not good” at the cost of a much larger feature space. The hashing trick maps tokens to a fixed number of buckets with a hash function, which removes the need to store a vocabulary and suits streaming data, but it introduces occasional collisions between unrelated words.
Bag of words compared with related representations
Bag of words sits at the simplest end of a spectrum of text representations that runs from sparse counts to contextual embeddings. Comparing it with its neighbors in speech and language processing clarifies when it is still the right tool.
| Representation | Captures word order | Captures similarity between words | Dimensionality |
|---|---|---|---|
| Bag of words | No | No | Vocabulary size, sparse |
| N-gram features | Locally, within the window | No | Grows rapidly with n |
| Topic models | No | Through shared topics | Number of topics |
| Static embeddings such as word2vec | No, when averaged | Yes | Typically 100 to 300 dense |
| Contextual transformer embeddings | Yes | Yes, context dependent | Typically 768 or more dense |
The central weakness is that bag of words treats every word as an independent, orthogonal dimension. The vectors for “car” and “automobile” share nothing, so a document about one does not match a query about the other unless synonyms are added manually. Dense embeddings solve this by placing semantically related words close together, which is why they dominate semantic textual similarity and question answering.
Topic modeling methods such as latent Dirichlet allocation actually take a bag of words matrix as input, so the two concepts are complementary rather than competing. Many classic pipelines also remove stop words before counting to reduce noise.
When to use bag of words in practice
Bag of words remains a sound choice when data is limited, interpretability matters or compute is constrained. Paired with logistic regression, a linear support vector machine or naive Bayes, it trains in seconds on a laptop and exposes directly inspectable feature weights, which helps auditors and domain experts understand why a decision was made.
| Situation | Recommended choice | Reason |
|---|---|---|
| Topic or spam classification with clear keywords | Bag of words with TF-IDF | Keywords carry most of the signal |
| Need for a fast baseline before deep learning | Bag of words with a linear model | Sets a reference score cheaply |
| Negation and sarcasm drive the label | Bag of n-grams or a transformer | Word order changes the meaning |
| Queries use different words than documents | Dense embeddings or hybrid retrieval | Vocabulary mismatch defeats exact matching |
| Strict explainability requirements | Bag of words with sparse weights | Each feature is a readable word |
A practical rule is to always build a bag of words baseline first. If a large neural model improves on it by only one or two percentage points, the simpler system may be preferable once serving cost, latency and maintenance are considered.
Limitations and common mistakes
Most problems with bag of words come from ignored structure and careless preprocessing. The following points summarize the failure modes that experienced practitioners watch for.
- Loss of word order merges sentences with opposite meanings, such as “not bad, actually good” and “not good, actually bad”.
- Vocabulary leakage occurs when the vocabulary or IDF statistics are fitted on test data, which inflates evaluation scores.
- Out-of-vocabulary words in new text are silently dropped, so models degrade when terminology drifts.
- Aggressive stop word removal can delete negations such as “not” that sentiment analysis depends on.
- Unbounded vocabularies consume memory; capping features by minimum document frequency usually costs little accuracy.
- Morphologically rich languages fragment one lemma into many surface forms, so lemmatization becomes more important than in English.
None of these issues makes the model obsolete, but each one should be tested explicitly during evaluation rather than assumed away.
A plain-language summary
Imagine cutting a letter into single words, throwing them into a bag and then counting what is inside. You would know that the word “refund” appears three times and “angry” appears twice, but you would no longer know the order of the sentences. That is exactly what a bag of words does with text.
This sounds crude, yet it works surprisingly well for many jobs. A computer can often tell a complaint from a thank-you note, or a sports article from a cooking recipe, just by looking at which words show up most. Its weakness is that it cannot understand phrases where order changes the meaning, so modern systems often add more advanced techniques on top.
An example from practice
A mid-sized online retailer receives about 4,000 customer emails per day and wants to route each one to billing, shipping, returns or technical support. The data team exports 60,000 previously labeled emails, lowercases them, keeps words and two-word phrases that occur in at least five messages, and obtains a vocabulary of roughly 40,000 features weighted with TF-IDF.
A logistic regression classifier trained on these vectors reaches about 91 percent accuracy on a held-out month of emails, and training takes under a minute. A fine-tuned transformer later scores about 93 percent but requires GPU serving. Because the feature weights show that words such as “tracking”, “courier” and “delayed” drive the shipping category, the support managers trust the system, and the team keeps the bag of words model in production while using the transformer only for messages the simple model classifies with low confidence.