Learning to rank

Also known as: LTR · machine-learned ranking

On this page 7

Learning to rank (LTR), also called machine-learned ranking, is the family of supervised machine learning methods that train a model to order a set of items by relevance to a user, query or context, rather than to predict an isolated score for each item. In recommender systems, learning to rank usually powers the final ranking stage, where a few hundred candidates are sorted into the short list a person actually sees.

The approach matters because users interact almost entirely with the top positions of a list. A model that predicts ratings accurately but places the best items in positions eight to twelve delivers little value, so learning to rank optimizes the quality of the ordering itself, measured with list-aware metrics such as NDCG and mean reciprocal rank.

How learning to rank models order items

A learning to rank model learns a scoring function that maps a feature vector for each user-item or query-item pair to a real number, and the list is then sorted by that number. Training data consists of groups of items that share one context, such as one search query or one user session, each item carrying a graded relevance label or an implicit signal such as a click, purchase or watch time.

What distinguishes LTR from ordinary regression is the loss function. Instead of penalizing the distance between a predicted and a true rating, ranking losses penalize mistakes in relative order: a relevant item scored below an irrelevant one, or a highly relevant item pushed out of the top positions. The model therefore spends its capacity where ranking quality is decided.

Typical input features fall into a few families, and the table below summarizes them with examples from a retail recommender.

Feature familyExamplesWhat it captures
Item featuresCategory, price, age, popularityIntrinsic appeal of the item
User featuresTenure, preferred brands, deviceStable user preferences
Interaction featuresPast clicks on the brand, embedding similarityFit between user and item
Context featuresTime of day, page type, session lengthSituation of the request
Upstream scoresCandidate generator score, retrieval rankSignals from earlier stages
Common feature families in ranking models

Pointwise, pairwise and listwise approaches

Learning to rank methods are classified into three approaches according to how many items the loss function looks at simultaneously. The choice affects training cost, robustness to noisy labels and how closely the objective matches the evaluation metric.

ApproachLoss looks atRepresentative methodsMain trade-off
PointwiseOne item at a timeLogistic regression on clicks, McRankSimple, but ignores relative order
PairwisePairs of itemsRankNet, RankSVM, Bayesian personalized rankingLearns order, but treats all pairs equally
ListwiseThe whole listListNet, ListMLE, LambdaMARTClosest to metrics, higher training cost
The three classic learning to rank approaches

LambdaMART, which combines gradient-boosted decision trees with lambda gradients derived from NDCG changes, has been a strong production baseline for well over a decade. Its lambda trick scales each pairwise gradient by how much swapping the two items would change the metric, so errors at the top of the list cost more than errors near the bottom. Neural rankers built on deep networks or transformers now dominate when feature sets are large and include raw text or sequences, but tree ensembles remain competitive on tabular features.

Metrics that evaluate ranked lists

Ranking quality is measured with position-aware metrics that reward relevant items placed near the top. Most of them are computed at a cutoff k, written as @k, which reflects the number of slots a user realistically inspects.

MetricWhat it measuresBest suited to
NDCG@kGraded gain discounted by log of positionMulti-level relevance labels
MRRInverse rank of the first relevant itemTasks with one correct answer
MAPMean precision at each relevant positionBinary relevance with many hits
Recall@kShare of relevant items in the top kCandidate generation stages
Precision@kShare of top k that is relevantShort fixed carousels
Standard offline ranking metrics

Offline metrics are necessary but not sufficient. Because logged data reflects what the previous system chose to show, offline gains often shrink online, and teams confirm improvements with A/B tests or interleaving experiments that compare two rankers on live traffic.

Learning to rank in a multi-stage recommender

In a production multi-stage ranking architecture, learning to rank occupies the expensive, precise end of the pipeline. Candidate generation, often a two-tower model with approximate nearest neighbor search, reduces millions of items to hundreds; a lightweight pre-ranker may cut them to dozens; the main LTR model then applies rich cross features; and a re-ranking step adjusts the result for diversity, business rules and fairness.

The comparison below shows how learning to rank relates to neighboring techniques in recommender systems.

TechniquePrimary outputRelationship to LTR
Rating predictionEstimated score per itemPointwise special case without list objective
CTR predictionClick probability per itemOften used as a pointwise ranking signal
Matrix factorizationLatent user and item vectorsSupplies features or candidates
Candidate generationBroad set of plausible itemsFeeds the ranking model
Re-rankingFinal adjusted orderPost-processes LTR output
Learning to rank compared with related techniques

Modern rankers frequently optimize several objectives at once, such as click probability, purchase probability and long-term satisfaction, and combine them into a single ranking score with tuned weights. This multi-objective setup is a practical extension of learning to rank rather than a separate discipline.

Limitations and common mistakes

Most failures of learning to rank come from the training data, not the algorithm. Implicit feedback is biased: users click items in top positions partly because those positions are visible, which is known as position bias, and they never click items the system never showed.

  • Ignoring position bias teaches the model to reproduce the old ranking; unbiased learning to rank corrects it with inverse propensity weighting or click models.
  • Treating non-clicks as negatives confuses unseen items with disliked items; careful negative sampling and impression logs reduce the error.
  • Optimizing clicks alone rewards clickbait; adding dwell time, conversions or satisfaction labels aligns the objective with real value.
  • Training-serving skew occurs when features are computed differently offline and online, silently degrading the live ranking.
  • Feedback loops narrow exposure over time and reinforce popularity bias unless exploration or diversity constraints are added.

Learning to rank also inherits the cold start problem: new items without interaction history receive weak feature values and are ranked low, so systems reserve exploration slots or rely on content-based features to give them a fair chance.

In plain words

Learning to rank teaches a computer to put the best options first. Instead of asking how much someone will like each item on its own, the model asks which item should come before which other item.

Think of a librarian arranging a shelf of recommendations for one reader. The librarian does not need to grade every book precisely; the job is to make sure the books the reader will love sit at eye level. Because people rarely look past the first few results, getting the top of the list right is what matters most.

A real-world example

A mid-sized online fashion retailer shows a carousel of 12 recommended products on each product page. Its first system ranked items by predicted click probability from a pointwise logistic regression, and the team noticed that most clicks landed on the first three slots regardless of quality.

The data science team trained a LambdaMART model optimized for NDCG@12, using about 150 features covering item attributes, user history and session context. They corrected position bias with inverse propensity weights estimated from a small randomized traffic slice, where the top slots were shuffled for one percent of visitors.

Offline NDCG@12 improved by several points, and a four-week A/B test showed a measurable rise in add-to-cart rate from the carousel, with the largest gains for less popular products that the old model had buried. The team kept the randomized slice running to keep propensity estimates current.