Gini impurity

On this page 7

Gini impurity is a measure of how mixed the class labels are within a set of training samples, defined as the probability that a randomly chosen sample would be misclassified if it were labeled at random according to the class distribution of that set. In machine learning, it is the default splitting criterion that classification trees use to decide which feature and threshold divide the data best.

A node that contains only one class has a Gini impurity of exactly zero, while a node with classes in equal proportions reaches the maximum value. Tree-growing algorithms such as CART evaluate thousands of candidate splits and keep the one that lowers impurity the most, so this simple formula shapes the structure of every decision tree, random forest and many gradient boosting models built for classification.

How Gini impurity is calculated

Gini impurity is calculated as one minus the sum of squared class proportions in a node: G = 1 - Σ p_k², where p_k is the share of samples belonging to class k. Squaring the proportions rewards nodes dominated by a single class, because a large share contributes much more to the sum than several small shares do.

The value has a clear probabilistic interpretation. If two samples are drawn from the node with replacement, Gini impurity equals the probability that their labels differ. For K classes, the upper bound is 1 – 1/K, which gives 0.5 for binary problems and approximately 0.667 for three classes.

The following table shows how the value responds to different class mixtures in a binary node.

Class proportionsCalculationGini impurityInterpretation
100% / 0%1 – (1.00 + 0.00)0.000Pure node, no further split needed
90% / 10%1 – (0.81 + 0.01)0.180Nearly pure
75% / 25%1 – (0.5625 + 0.0625)0.375Moderately mixed
60% / 40%1 – (0.36 + 0.16)0.480Strongly mixed
50% / 50%1 – (0.25 + 0.25)0.500Maximum impurity for two classes
Gini impurity for typical binary class distributions

How decision trees use Gini impurity to choose splits

A decision tree uses Gini impurity by comparing the impurity of a parent node with the weighted average impurity of its children. Each child is weighted by its share of the parent samples, so a small pure child cannot dominate the score. The difference is the Gini decrease, sometimes called Gini gain, and the algorithm selects the split with the largest decrease.

Consider a parent node with 40 positive and 60 negative samples, which has an impurity of 0.48. A candidate split sends 30 positive and 10 negative samples to the left child and 10 positive and 50 negative samples to the right child. The calculation proceeds as shown in the table.

NodeSamplesClass proportionsGini impurityWeight
Parent1000.40 / 0.600.4801.00
Left child400.75 / 0.250.3750.40
Right child600.17 / 0.830.2780.60
Weighted children100Combined0.317Sum of weighted values
Worked split evaluation with Gini impurity

The split reduces impurity by about 0.163. The algorithm repeats this calculation for every feature and every candidate threshold, then applies the best split greedily and recurses into each child. Growth stops when nodes are pure or when stopping rules such as maximum depth, minimum samples per leaf or a minimum impurity decrease are reached.

Gini impurity compared with entropy and misclassification error

Gini impurity is one of three common impurity measures, and it behaves very similarly to Shannon entropy, the measure behind information gain. Entropy uses logarithms, H = -Σ p_k log₂ p_k, and peaks at 1 bit for a balanced binary node. Both curves are concave and symmetric, so in practice they select the same split in the large majority of cases.

MeasureFormulaBinary maximumTypical useMain property
Gini impurity1 – Σ p²0.5CART, scikit-learn defaultFast, no logarithm
Entropy-Σ p log₂ p1.0 bitID3, C4.5, C5.0Slightly favors balanced splits
Misclassification error1 – max p0.5Pruning, evaluationToo insensitive for growing trees
Comparison of impurity measures for classification trees

Misclassification error is rarely used for growing trees because it is not strictly concave. A split can make both children purer in distribution while leaving the majority class, and therefore the error, unchanged. Gini impurity and entropy register such improvements, which is why they produce better intermediate splits.

Gini impurity should not be confused with the Gini coefficient used in economics to measure income inequality, nor with the Gini metric in credit scoring, which equals twice the ROC AUC minus one. They share the name of the statistician Corrado Gini but answer different questions.

Gini impurity in ensembles and feature importance

In ensemble methods, Gini impurity drives the construction of every individual tree. A random forest grows many trees on bootstrap samples and random feature subsets, and each tree still chooses its splits by Gini decrease. Classification variants of gradient boosting often fit regression trees to gradients instead, so the criterion there is usually a squared-error or gain-based score rather than Gini impurity.

Summing the weighted Gini decreases contributed by each feature across all nodes and trees gives mean decrease in impurity, the built-in feature importance of many libraries. This measure is cheap to compute but has a known bias toward high-cardinality features, such as identifiers or continuous variables with many unique values, because they offer more candidate thresholds and therefore more chances to reduce impurity by accident.

The table summarizes where the criterion appears and what practitioners should watch for.

ModelRole of Gini impurityPractical caution
Single decision treeSelects every splitOverfits without depth or leaf limits
Random forestSelects splits in each treeImpurity importance is biased
Extra treesScores randomly drawn thresholdsLess sensitive to threshold search
Gradient boostingUsually replaced by gradient-based gainCheck the library documentation
Where Gini impurity appears in common models

Limitations and common mistakes

Gini impurity is a local, greedy criterion, so it optimizes one split at a time and cannot guarantee a globally optimal tree. Several practical issues follow from its design and from how it interacts with real data.

  • Class imbalance compresses impurity values: a node with 99% of one class already scores below 0.02, so splits that isolate rare cases yield small decreases unless class weights or resampling such as SMOTE are applied.
  • Treating impurity importance as causal is a frequent error; permutation importance or SHAP values give more reliable rankings when features differ in cardinality.
  • Choosing between Gini and entropy rarely changes accuracy much; tuning depth and leaf size usually matters far more.
  • Gini impurity applies only to classification targets; regression trees use variance reduction or mean squared error instead.

In everyday language

Gini impurity is a score for how mixed up a group is. Imagine a basket of fruit. If every piece is an apple, the basket is perfectly tidy and scores zero. If half are apples and half are oranges, it is as messy as two kinds can be and scores the maximum of one half.

A decision tree works like someone sorting that basket by asking yes-or-no questions, such as whether the fruit is heavier than 150 grams. For every possible question, it checks how tidy the two resulting piles would be and picks the question that makes them as tidy as possible. Gini impurity is simply the tidiness score it uses to make that choice.

Seen in the real world

A data science team at a regional bank builds a random forest to flag loan applications likely to default. The training set contains 50,000 past applications, of which about 8% defaulted, so the root node starts with a Gini impurity of roughly 0.147. The first split the trees most often choose is the debt-to-income ratio at about 43%, which separates a high-risk group from a much purer low-risk group.

Because defaults are rare, the team sets balanced class weights so that each default counts more in the impurity calculation, and limits leaves to at least 50 samples. The built-in Gini importance initially ranks a customer identifier field near the top, which the analysts recognize as high-cardinality bias. After removing that field and confirming rankings with permutation importance, the model ranks income stability and payment history highest, and the validated recall on defaults improves without a loss of overall precision.