Naive Bayes classifier

Also known as: naive Bayes

On this page 7

A Naive Bayes classifier is a probabilistic classification model that applies Bayes’ theorem to predict the most likely class of an example, under the simplifying assumption that all features are conditionally independent given the class. In machine learning, naive Bayes is one of the oldest and fastest supervised methods, and it remains a standard baseline for text classification, spam filtering and other problems with many sparse features.

The word naive refers to the independence assumption, which is almost never strictly true. Despite this, the classifier often ranks classes correctly, because a correct decision only requires the highest posterior score, not an accurate probability estimate.

How the naive Bayes classifier computes a prediction

The naive Bayes classifier predicts the class with the largest posterior probability, computed from a class prior and a product of per-feature likelihoods. Bayes’ theorem states that the posterior P(class | features) is proportional to P(class) multiplied by P(features | class). The independence assumption replaces the joint likelihood with a product of one-dimensional likelihoods, one for each feature.

This factorization is what makes the model practical. Estimating a full joint distribution over hundreds of features would require an enormous amount of data, whereas each per-feature distribution can be estimated from simple counts or means. Training therefore reduces to a single pass over the data, and prediction costs time proportional to the number of features multiplied by the number of classes.

In practice, implementations sum log probabilities instead of multiplying raw probabilities. Multiplying thousands of small numbers causes floating-point underflow, while the logarithm turns the product into a stable sum and does not change which class wins. The table below summarizes the components of the computation.

ComponentMeaningHow it is estimated
Class priorProbability of a class before seeing featuresShare of training examples in that class
Feature likelihoodProbability of a feature value given the classCounts, frequencies or a fitted distribution
EvidenceProbability of the observed featuresIgnored, because it is equal for all classes
PosteriorProbability of the class given the featuresPrior times likelihoods, then normalized
Components of the naive Bayes decision rule

Variants of naive Bayes and their feature models

Naive Bayes variants differ in the probability distribution assumed for each feature, and the right choice depends on the data type. Choosing a variant that does not match the features is a more common source of poor results than the independence assumption itself.

VariantFeature typeTypical use
Gaussian naive BayesContinuous values, normal per classSensor readings, measurements
Multinomial naive BayesCounts or term frequenciesDocument and topic classification
Bernoulli naive BayesBinary presence or absenceShort texts, binary indicators
Complement naive BayesCounts with skewed classesImbalanced text corpora
Categorical naive BayesDiscrete categoriesSurvey answers, coded attributes
Main naive Bayes variants

Multinomial and Bernoulli models need smoothing to handle feature values that never appeared with a class during training. Without it, a single unseen word produces a zero likelihood and eliminates the class regardless of all other evidence. Laplace smoothing adds one to every count, and the more general additive smoothing uses a tunable constant alpha, commonly searched between 0.01 and 1.0.

Naive Bayes is a generative classifier, because it models how features are produced within each class, whereas logistic regression is a discriminative classifier that models the class boundary directly. The two form a well-known pair: with enough data, logistic regression usually reaches a lower error, while naive Bayes tends to approach its own best accuracy with far fewer training examples.

MethodModel typeStrengthWeakness
Naive BayesGenerative, linear in log spaceVery fast, works on small dataIgnores feature interactions
Logistic regressionDiscriminative, linearWell-calibrated probabilitiesNeeds more data and tuning
Support vector machineDiscriminative, margin-basedStrong on high-dimensional textSlower training, no native probabilities
Bayesian networkGenerative, explicit dependenciesModels feature relationshipsStructure learning is costly
Decision treeDiscriminative, rule-basedCaptures interactionsOverfits without pruning
Naive Bayes against neighboring methods

A Bayesian network can be seen as the generalization of naive Bayes: naive Bayes is the special network in which the class is the only parent of every feature. Adding edges between features relaxes the independence assumption at the price of more parameters and slower learning.

When to use a naive Bayes classifier

Naive Bayes is the right choice when speed, small data or very many features dominate the requirements. It trains in seconds on millions of documents, supports online learning by simply updating counts, and needs almost no hyperparameter tuning.

SituationRecommendationReason
Text with large vocabularyUse multinomial naive BayesSparse counts fit the model well
Few labeled examplesUse naive Bayes as baselineLow variance on small samples
Streaming dataUse incremental updatesCounts update without retraining
Strongly correlated featuresPrefer logistic regression or treesCorrelations distort the posterior
Calibrated risk scores neededCalibrate or choose another modelRaw probabilities are overconfident
Decision guide for naive Bayes

Feature preparation matters more than tuning. For text, TF-IDF weighting, removal of very rare terms and n-grams can raise accuracy noticeably. For continuous features that are far from normal, discretizing them into bins and using a categorical model is often more robust than Gaussian naive Bayes.

Limitations and common mistakes with naive Bayes

The main limitation is double counting of correlated evidence. If two features carry the same information, for example a word and its plural, the model treats them as independent confirmations and pushes the posterior toward an extreme value. This is why naive Bayes probabilities are typically poorly calibrated, clustering near 0 and 1.

  • Zero-frequency problem: forgetting smoothing lets one unseen value veto a class.
  • Misleading probabilities: using raw outputs as risk estimates without Platt scaling or isotonic calibration.
  • Wrong variant: applying Gaussian naive Bayes to word counts or Bernoulli to long documents.
  • Skewed priors: strong class imbalance biases predictions toward the majority class unless priors are adjusted.
  • No interactions: the model cannot learn that two features matter only in combination, as in an XOR pattern.

Stripped to the essentials

A naive Bayes classifier sorts things into groups by weighing clues one at a time. Imagine a mail sorter who has learned how often words like free or invoice appear in junk mail and in normal mail. For a new message, the sorter checks each word, adds up how strongly each one points toward junk or toward normal, and picks the group with the stronger total.

It is called naive because it treats every clue as unrelated to the others, which is a simplification. Even so, it is fast, cheap and surprisingly accurate, which is why it is often the first model people try.

Where it is applied

A municipal help desk receives about 4,000 citizen emails per week and must route each one to one of nine departments, such as waste collection, parking or tax. The team has only 6,000 labeled historical emails, too few for a large language model fine-tuned in-house, and the routing must run on a modest server.

The data team trains a multinomial naive Bayes model on word unigrams and bigrams with TF-IDF weighting and an additive smoothing constant of 0.1, chosen by cross-validation. Training takes under two seconds. On a held-out test set the model routes about 88 percent of emails correctly, and messages whose top posterior falls below a confidence threshold go to a human dispatcher.

The result is that manual routing work drops by roughly two thirds, and the model is retrained every night with the corrections made by dispatchers, at negligible computing cost.