TF-IDF

Also known as: Term frequency-inverse document frequency

On this page 7

TF-IDF (term frequency-inverse document frequency) is a statistical weighting scheme that assigns each word in a document a score proportional to how often it appears there and inversely proportional to how many documents in the collection contain it. In the Retrieval & knowledge area of artificial intelligence, TF-IDF is the classic foundation of sparse retrieval: it turns text into weighted term vectors that can be matched against a query, ranked, clustered or fed into a classifier.

The idea dates back to the early 1970s work on term specificity in information retrieval, and it remains in daily use because it is fast, transparent and requires no training. Modern systems often combine it, or its successor BM25, with dense embeddings in hybrid search pipelines.

How TF-IDF weights a term

TF-IDF multiplies two factors: a local factor that measures how strongly a term characterizes one document and a global factor that measures how rare the term is across the corpus. The product is high for words that are frequent in one document but rare elsewhere, which are usually the words that best describe its content.

In the most common form, the weight of term t in document d is tf(t,d) × idf(t), where idf(t) = log(N / df(t)), N is the number of documents and df(t) is the number of documents containing t. A word that appears in every document receives an IDF of zero, so function words such as “the” or “and” contribute almost nothing even without a stop-word list.

The following table shows how the two factors interact for typical word categories in a corpus of news articles.

Word categoryTerm frequency in documentDocument frequency in corpusResulting weight
Function word (“the”)Very highAlmost all documentsNear zero
Generic topic word (“market”)ModerateMany documentsLow to moderate
Specific entity (“lithium”)ModerateFew documentsHigh
Rare typo or tokenOne occurrenceOne documentHigh but noisy
Effect of term frequency and document frequency on the weight

After weighting, each document becomes a sparse vector with one dimension per vocabulary term, most of them zero. Vectors are usually L2-normalized so that long documents do not dominate, and similarity between a query and a document is then computed with cosine similarity, which reduces to a dot product on normalized vectors.

Variants of the TF and IDF components

There is no single TF-IDF formula; implementations differ in how they dampen term frequency and smooth document frequency. These choices noticeably change ranking quality, so practitioners should know which variant a library applies by default.

ComponentVariantFormula sketchPurpose
Term frequencyRaw countf(t,d)Simplest; favors repetition
Term frequencySublinear scaling1 + log f(t,d)Dampens repeated words
Term frequencyBinary1 if presentIgnores repetition entirely
Inverse document frequencyStandardlog(N / df)Classic rarity weight
Inverse document frequencySmoothed IDFlog((1 + N) / (1 + df)) + 1Avoids zero and division errors
Inverse document frequencyProbabilisticlog((N − df) / df)Can turn negative for common terms
Common variants and their purpose

Most production libraries default to smoothed IDF with L2 normalization, and many practitioners enable sublinear TF for long documents. The vocabulary itself is a design decision as well: tokenization, lowercasing, stemming or lemmatization, and n-gram ranges determine which units receive weights at all.

TF-IDF compared with BM25 and dense retrieval

TF-IDF is the conceptual ancestor of BM25, the ranking function that dominates lexical search engines today, and it serves as the lexical counterpart to dense embeddings. Understanding the differences helps decide where TF-IDF still fits in a retrieval-augmented generation (RAG) stack.

MethodRepresentationHandles synonymsTraining neededTypical role
TF-IDFSparse term weightsNoNoBaseline, features, keyword matching
BM25Sparse, saturated TF with length normalizationNoNoStandard first-stage lexical ranker
SPLADELearned sparse weights with expansionPartlyYesNeural sparse retrieval
Dense retrievalContinuous embedding vectorsYesYes (embedding model)Semantic search
How TF-IDF relates to neighboring retrieval methods

The decisive weakness relative to BM25 is unbounded term-frequency growth: in plain TF-IDF a word repeated twenty times can outweigh several distinct query words, whereas BM25 saturates the TF contribution and normalizes by document length. The decisive weakness relative to dense retrieval is the vocabulary mismatch problem: a query for “car” does not match a document that only says “automobile”.

Where TF-IDF is used in practice

TF-IDF remains valuable wherever exact terminology, speed and explainability matter more than semantic generalization. Its weights are directly inspectable, which makes it easy to justify why a document ranked highly.

  • Keyword search over small or medium collections where an inverted index is sufficient.
  • Feature extraction for linear classifiers such as logistic regression or linear support vector machines in spam filtering and topic labeling.
  • Keyword and keyphrase extraction by listing the highest-weighted terms of a document.
  • Lexical branch of hybrid search, where its ranking is fused with dense results, for example through reciprocal rank fusion.
  • Near-duplicate detection and clustering of documents using cosine similarity between vectors.

The decision table below summarizes when TF-IDF is a sensible choice and when another method should take its place.

SituationRecommended choiceReason
Product codes, error IDs, legal citationsTF-IDF or BM25Exact token matching is required
Fast baseline for a new classifierTF-IDF featuresNo training cost and strong accuracy
Paraphrased natural-language questionsDense retrievalSynonyms and meaning matter
Large-scale web or enterprise searchBM25 plus rerankingBetter length and saturation handling
RAG over mixed technical contentHybrid searchCombines exact and semantic recall
Choosing TF-IDF or an alternative

Limitations and common mistakes

TF-IDF is a bag-of-words model, so it ignores word order, negation and context; “not secure” and “secure” share most of their weight. Several practical mistakes also degrade results even when the method is appropriate.

  • Fitting IDF on the wrong corpus: statistics computed on a tiny or unrepresentative set distort rarity, so the vectorizer should be fitted on data that resembles the search collection.
  • Data leakage in evaluation: fitting the vectorizer on training and test documents together inflates classifier scores.
  • Ignoring out-of-vocabulary terms: words unseen at fitting time receive no weight in queries, which silently drops new product names.
  • Over-trusting rare tokens: typos and identifiers with a document frequency of one receive the maximum IDF and can dominate similarity.
  • Assuming semantic understanding: high cosine scores reflect shared vocabulary, not shared meaning.

Vocabulary size is another operational concern: with n-grams enabled, matrices can reach hundreds of thousands of dimensions, so practitioners cap features with minimum and maximum document-frequency thresholds or hashing.

Stripped to the essentials

TF-IDF gives every word in a text a score. A word scores high when it appears often in that text but rarely in other texts. Words that appear everywhere, such as “the”, score close to zero.

Think of a librarian sorting articles. If the word “volcano” shows up ten times in one article and almost nowhere else, it is a good label for that article. TF-IDF does this counting automatically, so a search system can find the documents whose distinctive words match the question. Its limit is simple: it matches words, not meanings.

Where it is applied

An IT support team at a mid-sized software company maintains about 40,000 resolved tickets and wants an assistant that suggests earlier fixes. Engineers often paste exact error codes such as ERR_CONN_RESET or library version strings, which dense embeddings tend to blur together.

The team builds a hybrid retriever: a TF-IDF index with sublinear term frequency and word bigrams handles the lexical side, while an embedding model handles paraphrased descriptions. Results from both are merged with reciprocal rank fusion before a language model drafts an answer.

In internal testing on a few hundred labeled queries, tickets containing the exact error code moved into the top three results far more often than with embeddings alone, and the support staff could see which matching terms explained each result. The TF-IDF index rebuilds in minutes on a single server, so new tickets become searchable the same day.