Class imbalance

Also known as: imbalanced dataset

On this page 7

Class imbalance is a property of a labeled dataset in which the classes are represented in very unequal proportions, so that one or more minority classes have far fewer examples than the majority class. In machine learning, and in artificial intelligence more broadly, it is one of the most common reasons why a classifier with impressive headline accuracy still fails on the cases that matter most, such as fraud, rare diseases or equipment faults.

A dataset with imbalanced class distribution, often simply called an imbalanced dataset, is not an error in the data. It usually reflects reality: most card transactions are legitimate and most machines are healthy. The difficulty lies in how learning algorithms, loss functions and evaluation metrics react to that skew.

How class imbalance distorts learning

Class imbalance distorts learning because most training objectives minimize average loss over all examples, and the majority class dominates that average. A model that predicts the majority label for every input can reach 99% accuracy on a 99:1 dataset while detecting nothing of interest.

The effect is visible in the decision boundary. With few minority examples, the region assigned to the minority class shrinks, and the classifier becomes conservative toward rare labels. Probabilistic models such as logistic regression also learn a shifted intercept, so predicted probabilities for the minority class stay low even when the evidence is strong.

Severity is usually described by the imbalance ratio, the count of majority examples divided by the count of minority examples. The table below shows typical levels and what they mean in practice.

Imbalance ratioTypical domainPractical impact
2:1 to 4:1Customer churn, sentimentMild; standard models often cope
10:1 to 100:1Credit default, medical screeningAccuracy misleading; threshold tuning needed
100:1 to 1,000:1Card fraud, click predictionSpecialized metrics and resampling required
Above 1,000:1Network intrusion, rare faultsOften reframed as anomaly detection
Typical imbalance levels and their practical impact

The ratio alone does not decide difficulty. Absolute minority count matters as much: 500 positives out of 500,000 may be learnable, while 20 positives out of 2,000 rarely are. Class overlap and label noise in the minority class usually hurt more than the ratio itself.

Why accuracy fails and which metrics to use

Accuracy fails under class imbalance because it rewards correct majority predictions that are trivially easy. Evaluation must instead focus on how well the model ranks and detects the minority class, which is why practitioners rely on confusion-matrix metrics computed per class.

The following table compares the metrics most often used for imbalanced binary classification and states when each is appropriate.

MetricWhat it measuresBehavior under imbalance
AccuracyShare of all correct predictionsInflated by the majority class
PrecisionShare of flagged cases that are truly positiveSensitive to false alarms
RecallShare of true positives that are foundSensitive to missed positives
F1 scoreHarmonic mean of precision and recallIgnores true negatives; threshold-dependent
ROC AUCRanking quality across thresholdsCan look optimistic when negatives dominate
PR AUCArea under the precision-recall curveMost informative for rare positives
Balanced accuracyMean recall across classesRobust single number for skewed data
Evaluation metrics for imbalanced classification

A useful reference point is that the PR AUC of a random classifier equals the positive rate, so a PR AUC of 0.30 on a 1% positive class is a large improvement, while the same value on a 25% class is weak. The Matthews correlation coefficient is another balanced single-number summary that uses all four confusion-matrix cells.

Techniques for handling class imbalance

Techniques for class imbalance fall into three families: data-level methods that change the training distribution, algorithm-level methods that change the loss or the model, and decision-level methods that change how scores are turned into labels.

MethodLevelHow it worksMain risk
Random undersamplingDataDiscards majority examplesLoses useful information
Random oversamplingDataDuplicates minority examplesOverfitting to repeated points
SMOTE and variantsDataInterpolates synthetic minority pointsNoise in overlapping regions
Class weightsAlgorithmScales loss per classDistorted probability calibration
Focal lossAlgorithmDown-weights easy examplesExtra hyperparameters to tune
Balanced ensemblesAlgorithmBagging or boosting on rebalanced subsetsHigher training cost
Threshold movingDecisionShifts the cutoff on predicted scoresNeeds a reliable validation set
Main families of imbalance handling methods

Cost-sensitive learning is often the most principled option, because it encodes the real business cost of each error type directly in the objective. Gradient boosting libraries and support vector machines expose this as a class weight or sample weight parameter, and a common starting value is the inverse class frequency.

Threshold moving deserves special attention. Many models already rank cases well; they simply apply a default cutoff of 0.5 that is wrong for skewed data. Choosing the threshold on a validation set to meet a target recall or precision frequently gives most of the benefit without any resampling.

Choosing a strategy in practice

The right strategy depends on the cost of each error type, the amount of minority data and whether calibrated probabilities are required downstream. The decision table below summarizes common situations.

SituationRecommended approachReason
Probabilities feed a pricing modelTrain on natural data, tune thresholdPreserves calibration
Missed positives are very costlyClass weights plus high-recall thresholdPrioritizes recall
Huge majority, ample minorityUndersample the majorityCuts training time with little loss
Very few minority examplesCollect more labels or use anomaly detectionResampling cannot invent information
Tabular data, tree ensemblesWeighted gradient boostingStrong baseline with little tuning
Decision guide for imbalanced problems

Two procedural rules apply in every case. Use stratified splits and stratified cross-validation so that each fold contains minority examples, and resample only the training fold, never the validation or test data, or the evaluation becomes optimistic.

Common mistakes and misconceptions

Most failures with imbalanced data come from evaluation errors rather than modeling errors. The following list covers the mistakes that experienced practitioners check for first.

  • Oversampling before the split leaks synthetic copies of test points into training and inflates scores.
  • Reporting accuracy alone hides a model that never predicts the minority class.
  • Assuming SMOTE always helps; with strong learners such as gradient boosting, weighting or threshold tuning often performs as well or better.
  • Forgetting recalibration after reweighting or resampling, which leaves predicted probabilities biased upward for the minority class.
  • Ignoring prior shift in production, where the real positive rate differs from the training rate and changes over time, a form of concept drift.

A frequent misconception is that a balanced 50:50 training set is the goal. In practice, moderate rebalancing combined with a tuned threshold usually outperforms full balancing, which discards information or creates many artificial points.

The idea in brief

Imagine teaching someone to spot counterfeit banknotes by showing them a thousand real notes and only two fakes. They would quickly learn that saying “it is real” is almost always right, and they would rarely catch a fake. That is class imbalance: the thing you care about is rare, so a computer model learns to ignore it.

The fix is to change what counts as success. Instead of asking how often the model is right overall, ask how many fakes it catches and how many false alarms it raises. Then give the rare examples more weight, add more of them, or lower the bar for raising an alert. The goal is catching rare cases, not a high overall score.

A practical scenario

A payments team at a mid-sized bank trains a model to flag fraudulent card transactions. Its training data contains 2 million transactions, of which about 0.2% are fraud, an imbalance ratio near 500:1. The first gradient boosting model reports 99.8% accuracy, yet it flags almost no fraud, because predicting “legitimate” every time already scores that high.

The team switches to PR AUC and recall at a fixed review budget as the main metrics, uses stratified time-based validation and sets the class weight to roughly the inverse frequency. It then tunes the decision threshold so that analysts receive about 1,500 alerts per day, the number they can review.

The result is a model that catches roughly 70% of fraud cases within that alert budget, compared with almost none before, while the probability scores are recalibrated so that the risk team can still use them for loss forecasting.