Multi-class classification

Also known as: multiclass classification

On this page 7

Multi-class classification is the supervised learning task in which a model assigns each input to exactly one of three or more mutually exclusive classes. In Machine learning and the wider field of artificial intelligence, it extends binary classification to problems such as recognizing handwritten digits, routing support tickets to one of many queues or identifying a plant species from a photograph.

The defining constraint is mutual exclusivity: every example carries a single correct label. This separates multiclass classification from multi-label classification, where one input may receive several labels at once. The constraint shapes the output layer, the loss function, the evaluation metrics and the strategies used to adapt algorithms that were designed for two classes.

How multi-class classification assigns a single label

A multi-class model learns a function that maps a feature vector to one score per class and then selects the class with the highest score or probability. The model therefore partitions the feature space into K regions, one per class, separated by decision boundaries that the training algorithm positions to minimize errors on labeled data.

Most probabilistic models convert raw scores, called logits, into a probability distribution with the softmax function. Softmax exponentiates each logit and divides by the sum of all exponentials, so the outputs are positive and sum to exactly one. Training then minimizes categorical cross-entropy, the negative log probability assigned to the correct class, which heavily penalizes confident wrong answers.

The table below summarizes the main components of a typical multi-class pipeline and the role each one plays.

ComponentTypical formPurpose
Label encodingInteger index or one-hot vectorRepresents the single true class
Output layerK scores (logits)Produces one score per class
Link functionSoftmaxTurns scores into probabilities
Loss functionCategorical cross-entropyMeasures error during training
Decision ruleArgmax over probabilitiesSelects the predicted class
Core components of a multi-class model

Native and decomposition strategies

Algorithms handle several classes either natively or through decomposition into binary problems. Decision trees, random forests, naive Bayes, k-nearest neighbors and neural networks extend naturally to K classes. Support vector machines and classic logistic regression were formulated for two classes and are usually wrapped in a decomposition scheme.

The two standard decompositions are one-vs-rest and one-vs-one. They differ sharply in the number of models trained, the size of each training set and the way predictions are combined, as the following comparison shows.

StrategyModels trainedMain advantageMain drawback
One-vs-rest (OvR)KSimple and scales linearlyEach model faces imbalanced data
One-vs-one (OvO)K(K-1)/2Small, balanced pairwise tasksGrows quadratically with K
Native softmax model1Calibrated joint probabilitiesRequires a multinomial formulation
Error-correcting output codesChosen code lengthTolerates some binary errorsCode design adds complexity
Comparison of multi-class strategies

In one-vs-rest, each classifier separates one class from all others, and the final label is the class whose classifier reports the highest confidence. Scores from independently trained models are not always comparable, which can produce ambiguous regions where several or no classifiers claim an example. One-vs-one resolves pairs and uses majority voting; with 100 classes it requires 4,950 separate models, which is why it suits small K and algorithms whose cost grows quickly with sample size, such as kernel support vector machines.

Evaluating multi-class models

Evaluation starts with the K-by-K confusion matrix, whose rows hold true classes and whose columns hold predicted classes. Off-diagonal cells reveal which classes the model confuses, a diagnostic that a single accuracy figure hides.

Per-class precision, recall and F1 score are combined into one number through averaging methods, and the choice changes the conclusion considerably when class frequencies differ. The table explains the common options.

Averaging methodHow it is computedWhen it fits
Micro averagePools all predictions before computing the metricOverall performance; equals accuracy for single-label F1
Macro averageUnweighted mean of per-class scoresEvery class matters equally, including rare ones
Weighted averagePer-class scores weighted by class supportReporting that reflects class frequency
Top-k accuracyCorrect if true class is among k bestLarge label sets with near-duplicate classes
Averaging methods for multi-class metrics

Probabilistic quality is measured with log loss and calibration checks. A model can reach high accuracy yet produce poorly calibrated probabilities, which matters when downstream decisions use confidence thresholds, for example to send uncertain cases to a human reviewer.

Choosing an approach for a multi-class problem

The right approach depends on the number of classes, the data volume, the need for probabilities and the degree of class imbalance. Feature preparation also matters: categorical inputs typically require one-hot encoding, and distance-based methods need feature scaling. The decision table below links common situations to a reasonable starting point.

SituationRecommended starting pointReason
Tabular data, fewer than 20 classesGradient boosting with softmax objectiveStrong accuracy with modest tuning
Images, audio or textNeural network with softmax outputLearns features and scales to many classes
Small data, kernel SVM preferredOne-vs-one decompositionKeeps each pairwise problem small
Thousands of classesHierarchical or sampled softmaxReduces cost of the output layer
Strong class imbalanceClass weights or resamplingPrevents majority classes from dominating
Decision guide for multi-class setups

Before modeling, teams should confirm that classes are truly exclusive. If real examples legitimately belong to two categories, forcing a single label introduces systematic label noise, and a multi-label formulation is the better fit.

Limitations and common mistakes

Multi-class problems become harder as K grows because training data per class shrinks, similar classes crowd the feature space and annotation errors multiply. Class imbalance is the most frequent practical difficulty, since a model can reach high overall accuracy while almost never predicting rare classes.

  • Reporting accuracy alone hides poor performance on minority classes; report macro F1 and the confusion matrix as well.
  • Treating class indices as ordinal in a regression model imposes a false ordering on unordered categories.
  • Splitting data without stratification can leave rare classes absent from the validation set.
  • Reading softmax outputs as calibrated without verification leads to overconfident automated decisions.
  • Ignoring an unknown class forces the model to label out-of-distribution inputs as one of the known categories.

The last point reflects the closed-set assumption: a standard classifier always outputs one of the K known labels, even for inputs unlike anything seen in training. Production systems often add a rejection threshold or an explicit other class to handle such cases.

In plain words

Multi-class classification means teaching a computer to sort things into one of several boxes. Each item goes into exactly one box. A mail sorter who places every letter into one of twenty city bins is doing the same job by hand.

The computer learns from many examples that people have already sorted. For a new item, it gives every box a score and chooses the box with the highest score. The main challenge is that some boxes look alike, and some boxes rarely receive anything, so the system must be checked box by box rather than judged only on its overall hit rate.

A real-world example

A regional utility company receives about 12,000 customer emails per month and wants each one routed automatically to one of eight teams, such as billing, outages, new connections and meter readings. Every email belongs to exactly one team, so the task is a multi-class classification problem with eight classes.

The data team labels 40,000 historical emails, converts the text into numerical features and trains a gradient boosting model with a softmax objective. Overall accuracy reaches 91 percent, but the confusion matrix shows that the small meter-readings class is often mistaken for billing. Applying class weights and adding a few targeted features raises macro F1 from 0.78 to 0.86.

In production, emails whose top predicted probability falls below 0.6 go to a human triage queue. The result is that about 85 percent of emails are routed automatically, and average response time drops from two days to under one day.