Multi-label classification

On this page 7

Multi-label classification is the supervised learning task in which each example may belong to zero, one or several classes at once, so the model predicts a set of labels rather than a single category. In machine learning and applied artificial intelligence, it describes problems such as tagging a news article as both politics and economy, or marking a chest X-ray with several co-occurring findings.

The defining property is that labels are not mutually exclusive. This single difference changes the output layer, the loss function, the decision thresholds and the evaluation metrics, and it makes dependencies between labels a central modeling concern.

How multi-label classification produces label sets

A multi-label model maps an input to a binary indicator vector of length L, where L is the number of possible labels and each position states whether that label applies. The training data use the same encoding: a document with the tags sports and health becomes a vector with ones in two positions and zeros elsewhere.

In neural networks, the output layer applies an independent sigmoid to each label instead of a softmax across all of them. Each sigmoid yields a probability between 0 and 1 that does not compete with the others, so several outputs can be high simultaneously. Training typically minimizes binary cross-entropy summed over labels, which treats every label as its own yes-or-no decision.

The final label set comes from thresholding the scores. A global cutoff of 0.5 is the naive default, but production systems usually tune per-label thresholds on validation data, because rare labels rarely reach high scores. Some systems instead keep the top-k labels or predict the label count first and then select that many.

The table below summarizes the components that differ from a single-label setup.

ComponentMulti-label choiceWhy it matters
Target encodingBinary vector of length LRepresents any subset of labels
Output activationSigmoid per labelScores do not have to sum to one
Loss functionBinary cross-entropy or focal lossEach label is learned as its own decision
Decision rulePer-label threshold or top-kConverts scores into a label set
EvaluationHamming loss, micro and macro F1Rewards partially correct sets
Core components of a multi-label model

Problem transformation and algorithm adaptation methods

Classic approaches fall into two families. Problem transformation methods convert the multi-label task into one or more ordinary classification problems, while algorithm adaptation methods modify a learner so that it handles label sets directly.

Binary relevance trains one independent binary classifier per label, which is simple, parallel and scalable but ignores label correlations. Classifier chains link the binary models so that each one receives the previous predictions as extra features, which captures dependencies at the cost of sensitivity to chain order. Label powerset treats every distinct label combination as one class of a multi-class problem, which models correlations exactly but produces a combinatorial explosion of classes and cannot predict combinations unseen in training.

The following table compares the main methods on the properties practitioners weigh most often.

MethodLabel dependenciesScalabilityMain weakness
Binary relevanceIgnoredHigh, one model per labelMisses co-occurrence patterns
Classifier chainsModeled sequentiallyModerateErrors propagate along the chain
Label powersetModeled exactlyLow for many labelsCannot output unseen combinations
RAkELModeled within subsetsModerateApproximate, needs ensembling
ML-kNNCaptured from neighborsLimited by data sizeSlow at prediction time
Neural network with sigmoidsLearned in shared layersHighNeeds threshold calibration
Comparison of common multi-label methods

Algorithm adaptation includes ML-kNN, which estimates label probabilities from the labels of nearest neighbors, and multi-label decision trees that use a modified entropy across all labels. Modern deep models implicitly adapt the algorithm: a shared representation feeding many sigmoid outputs learns correlations through common hidden features.

How it differs from binary and multi-class classification

Multi-label classification generalizes both binary classification and multi-class classification. Binary tasks decide one yes-or-no question, multi-class tasks choose exactly one class from several, and multi-label tasks choose any subset of the available classes, including the empty set.

Task typeLabels per exampleTypical output layerExample
Binary classificationOne of twoSingle sigmoidSpam or not spam
Multi-class classificationExactly one of manySoftmaxDigit recognition 0 to 9
Multi-label classificationZero to all of manySigmoid per labelMovie genres
Multi-output classificationOne class per separate targetOne softmax per targetColor and size of a garment
Classification task types compared

A frequent design error is to model a multi-label problem with softmax. Softmax forces probabilities to sum to one, so two genuinely present labels must split the probability mass, and the model learns to suppress true co-occurrences. Multi-output classification is a related but distinct setting in which each target has its own class set.

Evaluating multi-label predictions

Evaluation is harder than in single-label tasks because a prediction can be partially correct. A model that outputs three of four true labels is better than one that outputs none, and the chosen metric decides how much credit it receives.

MetricWhat it measuresBehavior
Subset accuracyExact match of the whole label setVery strict, low on large label spaces
Hamming lossShare of wrong label decisionsLenient, dominated by easy negatives
Micro-averaged F1F1 over all label decisions pooledWeighted toward frequent labels
Macro-averaged F1Mean of per-label F1 scoresExposes weak rare labels
Precision at kCorrect labels among top kSuits ranking and tagging interfaces
Common multi-label evaluation metrics

Reporting both micro and macro F1 is a sound default: a large gap between them signals that the model performs well on common labels and poorly on the long tail. Ranking metrics such as label ranking average precision are preferable when the system presents a sorted list of suggested tags to a human.

Limitations and common mistakes

Most real label spaces show severe class imbalance: a few labels appear in many examples and hundreds appear rarely. Plain resampling such as SMOTE is awkward here, because duplicating an example duplicates all of its labels. Practitioners rely instead on class-weighted or focal loss, iterative stratification for data splits and per-label thresholds.

  • Using one global threshold leaves rare labels almost never predicted.
  • Random train-test splits can place every example of a rare label in one partition; iterative stratification prevents this.
  • Missing labels are common, because annotators tag what they notice, so an absent label is not always a true negative.
  • Reporting only Hamming loss hides poor performance, since most label decisions are easy negatives.
  • Extreme multi-label settings with tens of thousands of labels require specialized indexing, tree-based or embedding-based methods.

The idea in brief

Most sorting tasks ask a computer to put each item into exactly one box. Multi-label classification removes that restriction. A photo can be tagged beach, sunset and family at the same time, and a support email can be both billing and urgent.

Think of it as a checklist instead of a multiple-choice question. For every possible tag, the model asks a separate question: does this tag fit, yes or no. The answer is a list of all tags that apply, which may be long, short or even empty.

A practical scenario

A customer service team at a mid-sized software company receives about 4,000 support tickets per week and wants each one routed automatically. A ticket often touches several areas, such as a login failure that also involves a billing dispute, so the team frames the task as multi-label classification over 35 categories.

The data scientists fine-tune a text encoder with 35 sigmoid outputs and binary cross-entropy, split the data with iterative stratification and tune a separate threshold for each category on a validation set. A first model with a fixed 0.5 cutoff reached a macro F1 of about 0.48 because rare categories were rarely predicted; per-label thresholds raised it to roughly 0.63 while micro F1 stayed near 0.81.

In production, tickets are copied to every relevant queue. The outcome is fewer manual reassignments and faster first responses, and the team retrains monthly as new product features add labels.