CatBoost

On this page 8

CatBoost is an open-source gradient boosting library for decision trees that trains accurate models on tabular data and is known for its native handling of categorical features. Among AI frameworks & tools, it belongs to the classical machine learning family alongside XGBoost and LightGBM rather than to deep learning frameworks such as PyTorch or TensorFlow.

Yandex released CatBoost in 2017 under the Apache 2.0 license. Its name combines “category” and “boosting”, which reflects its two signature ideas: ordered target statistics for encoding categories and ordered boosting for reducing a subtle bias called prediction shift. It offers Python, R, command-line and Java interfaces, supports CPU and GPU training, and often produces strong results with default hyperparameters.

How CatBoost builds a gradient boosted ensemble

CatBoost builds an ensemble by adding one tree at a time, where each new tree is fitted to the gradients of the loss function computed from the predictions of all previous trees. The final prediction is the sum of all tree outputs scaled by the learning rate, so every tree corrects a small part of the remaining error.

Unlike most boosting libraries, CatBoost grows oblivious trees, also called symmetric trees. In an oblivious tree, every node at the same depth uses the same split feature and threshold. A tree of depth 6 therefore contains only six split conditions and 64 leaves, and the leaf index can be computed with fast bit operations. This structure acts as a built-in regularizer and makes inference very fast, at the cost of some flexibility per tree.

The following table summarizes the main components of the training process and their purpose.

ComponentWhat it doesEffect on the model
Oblivious treesUse one split rule per depth levelFast inference, lower variance
Ordered target statisticsEncode categories from earlier rows onlyNo target leakage in encoding
Ordered boostingCompute residuals with models that never saw the rowLess prediction shift
Feature combinationsJoin categorical features greedily during tree growthCaptures interactions between categories
QuantizationBins numeric features into bordersFaster split search, less memory
Core components of CatBoost training

Categorical features and ordered target statistics

CatBoost encodes categorical features internally, so users pass raw string columns through the cat_features parameter instead of preparing one-hot or label encodings themselves. For low-cardinality columns, it applies one-hot encoding up to the `one_hot_max_size` threshold; for the rest, it computes target statistics, which replace each category with a smoothed average of the target.

A naive target encoding uses the target value of the same row, which causes target leakage and inflated training accuracy. CatBoost avoids this by drawing a random permutation of the training rows and computing each row’s statistic only from rows that appear earlier in that permutation. A prior value smooths rare categories, so a category seen once does not receive an extreme estimate. Several permutations are used across the trees to reduce the variance this procedure introduces.

The table below compares the encoding strategies a practitioner might otherwise apply by hand.

EncodingLeakage riskSuitable cardinalityMain drawback
One-hot encodingNoneLow, under about 10 levelsVery wide, sparse matrices
Label encodingNoneAnyImposes a false order
Naive target encodingHighMedium to highOverfits on rare categories
Ordered target statisticsLowMedium to very highSlower training, randomness
Encoding options for categorical data

Ordered boosting and prediction shift

Ordered boosting is the CatBoost technique that computes the gradient for each training row with a model that was not trained on that row. In standard gradient boosting, the same rows serve both to fit a tree and to estimate the residuals for the next tree, which makes the residuals on training data systematically smaller than on new data. This bias is called prediction shift, and it is a quiet source of overfitting.

CatBoost addresses it with the same permutation idea used for categories: supporting models are maintained so that each row’s residual depends only on preceding rows. Because this is expensive, the library offers two boosting_type modes. Ordered mode is most valuable on small and noisy datasets, while Plain mode is the faster default on large datasets, where prediction shift matters less.

Key hyperparameters and typical values

CatBoost is known for robust defaults, but a small set of parameters controls most of its behavior. The table lists the parameters that practitioners tune first, with their defaults and usual ranges.

ParameterDefaultTypical rangeWhat it controls
iterations1000500 to 10000Number of trees
learning_rateChosen automatically0.01 to 0.3Step size per tree
depth64 to 10Tree depth and interaction order
l2_leaf_reg31 to 10L2 penalty on leaf values
border_count254 on CPU32 to 255Bins per numeric feature
one_hot_max_size2 on CPU2 to 255Cardinality limit for one-hot
Frequently tuned CatBoost parameters

The most effective tuning step is usually to lower the learning rate and raise the iteration count while using the overfitting detector with early_stopping_rounds on a validation set. Depth beyond 10 rarely helps, because oblivious trees grow exponentially in leaf count with each level.

CatBoost compared with XGBoost and LightGBM

CatBoost, XGBoost and LightGBM are the three dominant gradient boosting libraries, and on many tabular benchmarks their tuned accuracy is close. They differ mainly in tree structure, categorical support and speed profile, as the following table shows.

AspectCatBoostXGBoostLightGBM
Tree growthSymmetric, level by levelDepth-wise by defaultLeaf-wise
Categorical featuresNative ordered statisticsNative support in newer versionsNative split grouping
Default qualityStrong without tuningNeeds moderate tuningNeeds moderate tuning
Training speed on CPUModerateModerateUsually fastest
Inference speedVery fastFastFast
Comparison of the main gradient boosting libraries

In practice, CatBoost is a natural first choice for data dominated by categorical columns, such as user IDs, product codes or regions. LightGBM is often preferred for very large numeric datasets where training time dominates, and scikit-learn remains the reference for simpler pipelines.

Practical use, deployment and limitations

CatBoost supports classification, regression and ranking, with loss functions such as Logloss, RMSE, MultiClass and YetiRank. It can also process text features and embedding features, provides built-in SHAP values for explanation, and exports models to formats including ONNX, Core ML, JSON, C++ and Python code. The Pool class bundles data, categorical column indices and weights for efficient reuse.

  • Training cost: ordered mode and many categorical combinations can make training several times slower than LightGBM.
  • Memory use: high-cardinality categories with feature combinations increase memory consumption, especially on GPU.
  • Not for raw unstructured data: images, audio and long text are better served by deep learning frameworks.
  • Export limits: ONNX and Core ML export do not cover every model with categorical features, so the deployment path must be tested early.
  • Validation discipline: target statistics reduce leakage, but they do not replace a proper time-based split for temporal data.

The idea in brief

CatBoost is a tool that learns patterns from tables of data. Think of spreadsheets of customers or orders. It builds hundreds of small decision trees. Each tree fixes a few mistakes of the trees before it.

Its special talent is text labels like city names. People do not have to turn them into numbers by hand. It also avoids “peeking” at the answers while it learns. That helps it perform well on new data, not only on the data it studied.

A practical scenario

A data team at a regional insurance company wants to predict which car insurance customers will file a claim in the next year. Their table has 400,000 rows and 35 columns, and more than half of the columns are categorical: vehicle model, postal district, occupation, sales channel and policy type. Several of these columns have thousands of distinct values.

Instead of building a one-hot pipeline with tens of thousands of sparse columns, the team passes the raw columns to CatBoost through cat_features, sets the loss to Logloss, uses a learning rate of 0.05 with up to 3,000 iterations and enables early stopping on a time-based validation set. Training on a single GPU takes under 15 minutes.

The resulting model improves the area under the ROC curve by several points over the previous logistic regression. Built-in SHAP values show that postal district and vehicle age drive most predictions, which the actuaries can review before the model is deployed for pricing decisions.