Binary classification

On this page 8

Binary classification is the supervised learning task of assigning each input to one of exactly two classes, conventionally called the positive class and the negative class. In machine learning it is the most common prediction problem in practice: fraud or legitimate, churn or stay, defective or sound. A trained classifier learns a decision rule from labeled examples and applies it to new cases.

Most binary classifiers do not output a label directly. They produce a score or estimated probability for the positive class, and a decision threshold converts that score into a yes-or-no answer. Separating the scoring model from the threshold is the single most useful idea for anyone who builds or evaluates these systems in artificial intelligence projects.

How a binary classifier turns features into a decision

A binary classifier maps a feature vector to a real-valued score and then compares that score with a threshold. In logistic regression, the score is a weighted sum of features passed through the sigmoid function, which yields a value between 0 and 1. A support vector machine returns a signed distance from a separating hyperplane, and a decision tree returns the share of positive training examples in the leaf that the input reaches.

Training minimizes a loss function that penalizes wrong or overconfident predictions. The standard choice for probabilistic models is binary cross-entropy (log loss), while support vector machines use hinge loss. The set of points where the score equals the threshold forms the decision boundary, which can be a straight line, a curved surface or a fragmented region, depending on the model family.

The following table compares common model families used for binary classification and what their raw output represents.

Model familyRaw outputTypical strength
Logistic regressionProbability via sigmoidInterpretable, well calibrated
Support vector machineSigned margin distanceRobust in high dimensions
Naive Bayes classifierPosterior probabilityFast with sparse text data
Random forestShare of tree votesHandles nonlinear interactions
Gradient boostingLog-odds sum of treesTop accuracy on tabular data
Model families and their raw outputs

The confusion matrix and its error types

Every binary prediction falls into one of four outcomes, and the confusion matrix counts them. A true positive is a correctly flagged positive case, a true negative is a correctly cleared negative case, a false positive is a negative case wrongly flagged, and a false negative is a positive case that the model missed.

The two error types rarely cost the same. In medical screening, a false negative can be far more expensive than a false positive, because a missed disease delays treatment. In spam filtering, the reverse often holds, since losing a legitimate email annoys users more than seeing one extra spam message.

OutcomeActual classPredicted classCommon name in practice
True positivePositivePositiveCorrect detection
False positiveNegativePositiveFalse alarm, Type I error
False negativePositiveNegativeMiss, Type II error
True negativeNegativeNegativeCorrect rejection
The four outcomes of a binary prediction

Evaluation metrics for binary classification

No single metric describes a binary classifier completely, so practitioners choose metrics that match the cost structure of the problem. Accuracy is the share of correct predictions, yet it misleads badly under class imbalance: a model that always predicts the negative class reaches 99% accuracy when only 1% of cases are positive.

Threshold-independent metrics summarize performance across all possible cut-offs. The ROC AUC equals the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with 0.5 meaning random ranking and 1.0 meaning perfect ranking. The precision-recall AUC is more informative when positives are rare, because its baseline equals the positive rate rather than 0.5.

This table lists the metrics most often reported and the question each one answers.

MetricFormula in wordsBest used when
PrecisionTrue positives among all flaggedFalse alarms are costly
Recall (sensitivity)True positives among all actual positivesMisses are costly
SpecificityTrue negatives among all actual negativesClearing negatives matters
F1 scoreHarmonic mean of precision and recallA single balanced figure is needed
ROC AUCRanking quality over all thresholdsClasses are roughly balanced
Log lossPenalty on predicted probabilitiesCalibrated probabilities matter
Core metrics and what they measure

Choosing the decision threshold

The default threshold of 0.5 is a convention, not an optimum. The right threshold depends on the relative cost of false positives and false negatives and on the base rate of the positive class. Moving the threshold down raises recall and lowers precision; moving it up does the opposite, which is the precision-recall trade-off.

Thresholds should be selected on a validation set, never on the test set, using an explicit cost matrix or a business constraint such as a fixed review capacity. When the model must output trustworthy probabilities, probability calibration with Platt scaling or isotonic regression is applied before the threshold is set.

SituationThreshold strategyReason
Cancer screeningLow threshold, high recallMissed cases are very costly
Spam filteringHigh threshold, high precisionLost legitimate mail hurts trust
Fraud review queueTop-k by scoreAnalyst capacity is fixed
Credit approvalCost-weighted thresholdLosses and profits are quantifiable
Threshold strategies for common situations

Binary versus multi-class and multi-label problems

Binary classification is the simplest classification setting, and it serves as a building block for more complex ones. Multi-class classification assigns exactly one of three or more classes, while multi-label classification allows several labels per input at once. Many multi-class methods decompose the task into binary subproblems through one-vs-rest or one-vs-one schemes.

SettingLabels per inputTypical output layer or rule
BinaryExactly one of twoSingle sigmoid score
Multi-classExactly one of manySoftmax over all classes
Multi-labelZero or more of manyIndependent sigmoid per label
How classification settings differ

Common pitfalls in binary classifiers

Most failures in production come from data and evaluation errors rather than from the choice of algorithm. The following problems appear repeatedly in real projects.

  • Data leakage: a feature that encodes the label, such as a refund flag in a fraud model, inflates offline scores and collapses in production.
  • Ignoring class imbalance: reporting accuracy on rare-event data hides a useless model; use precision, recall and precision-recall AUC instead.
  • Overfitting to resampling: oversampling with SMOTE before the train-test split leaks synthetic copies into evaluation.
  • Uncalibrated scores: tree ensembles and support vector machines often produce scores that are not true probabilities.
  • Concept drift: base rates and feature distributions shift over time, so thresholds and models need monitoring and periodic retraining.

A non-technical view

Binary classification means teaching a computer to answer a yes-or-no question about each new case. It learns from many past examples where the right answer is already known, such as emails already marked as spam or not spam.

The computer does not simply say yes or no. It first gives a confidence score, a bit like a doctor saying how worried they are, and then a cut-off line decides the final answer. Where that line sits is a deliberate choice: a cautious setting catches more real problems but raises more false alarms, while a strict setting does the opposite.

A concrete example

A mid-sized online retailer wants to flag fraudulent orders before shipping. Its data science team trains a gradient boosting model on 400,000 past orders, of which only about 0.8% were confirmed fraud. Features include order value, account age, mismatch between billing and shipping country and the number of failed payment attempts.

Accuracy is useless here, because predicting that no order is fraudulent already scores above 99%. The team instead tracks precision-recall AUC and selects a threshold on a validation set. The fraud team can manually review about 300 orders per day, so the threshold is set to send roughly that many highest-scoring orders to review.

At that setting the model catches about 70% of fraudulent orders while roughly one in five reviewed orders turns out to be fraud. The result is far lower chargeback losses with a review workload the existing team can handle.