Popularity bias

On this page 7

Popularity bias is the systematic tendency of a recommendation model to favor items that are already popular, recommending them more often than their relevance to individual users justifies, while long-tail items receive little or no exposure. It is one of the most studied distortions in recommender systems and a central topic in evaluating accuracy, fairness and diversity.

The bias arises because interaction data is heavily skewed: in a typical catalog, a small head of items collects most clicks, plays or purchases. Models trained to predict those interactions learn the skew, and when their output shapes what users see next, the skew can amplify itself over time.

Where popularity bias comes from

Popularity bias has several distinct sources, and separating them matters because each needs a different fix. Some come from the data, some from the learning algorithm and some from the way results are evaluated and displayed.

The table below lists the main sources and the mechanism by which each one pushes recommendations toward the head of the distribution.

SourceMechanismTypical sign
Skewed interaction dataFew items hold most observed feedbackTop 1% of items get most training signal
Exposure biasUsers can only click what they were shownUnseen items look irrelevant
Model learning dynamicsPopular item embeddings are updated far more oftenHigh item norms for head items
Evaluation protocolOffline metrics reward predicting popular test itemsMost-popular baseline scores well
Position biasTop slots collect clicks regardless of qualityClick rate falls sharply by rank
Main sources of popularity bias

In collaborative filtering, the effect is especially strong because similarity between users or items is computed from co-occurrence counts. An item consumed by many users becomes a neighbor of almost everything, so it appears in a large share of candidate lists.

The feedback loop that amplifies the bias

Popularity bias becomes harmful mainly through the recommendation feedback loop. The model recommends popular items, users interact with what they are shown, those interactions enter the next training set, and the next model sees an even more concentrated distribution.

Simulation studies consistently show that, without intervention, repeated retraining narrows catalog coverage and raises concentration measures such as the Gini coefficient of item exposure. The loop also connects popularity bias to the filter bubble and to unequal outcomes for item providers, such as independent musicians or small sellers.

  • Users with niche tastes receive worse recommendations, because the model treats them like the average user.
  • New and long-tail items struggle to collect the feedback they need, which worsens the cold start problem.
  • Platforms lose catalog value, because a large part of the inventory is never surfaced.

Measuring popularity bias

Popularity bias is measured with beyond-accuracy metrics that compare the popularity of recommended items with the popularity of items users actually like. No single metric is sufficient, so teams usually track several at once alongside accuracy.

MetricWhat it measuresDesired direction
Average recommendation popularityMean popularity of recommended itemsLower, within accuracy limits
Aggregate diversityNumber of distinct items recommended to anyoneHigher
Gini coefficient of exposureInequality of item exposure across the catalogLower
Long-tail coverageShare of recommendations from non-head itemsHigher
Popularity calibrationGap between user history mix and recommendation mixSmaller
Common metrics for popularity bias

A key caution is that standard offline accuracy metrics hide the problem. Because test sets are drawn from the same biased logs, a model that simply recommends bestsellers can score deceptively well on precision or recall. Popularity-stratified evaluation or unbiased estimators based on inverse propensity give a fairer picture.

Mitigation techniques

Mitigation methods fall into three families, defined by the stage of the pipeline they modify. Most production systems combine at least two, because data, model and ranking all contribute to the bias.

StageExample techniquesMain trade-off
Pre-processingResampling, down-weighting head interactionsLess signal from strong items
In-processingInverse propensity scoring, regularization, causal debiasingHigher variance, harder tuning
Post-processingRe-ranking, calibration, exposure quotasSmall accuracy loss at the top
ExplorationBandit policies, reserved discovery slotsShort-term engagement cost
Families of popularity bias mitigation

Inverse propensity scoring reweights each interaction by the inverse of the probability that the item was exposed, which corrects exposure bias in principle but becomes unstable when propensities are tiny. Re-ranking methods, such as calibrated recommendation or xQuAD-style diversification, are popular in practice because they leave the trained model untouched and expose a single tuning parameter.

In multi-stage ranking, bias can be addressed in candidate generation, for example with logQ correction for sampled softmax in two-tower models, and again in final ranking. Correcting only the last stage is often insufficient, because long-tail items may never reach the candidate set.

Common misconceptions about popularity bias

Popularity itself is not an error signal. Many items are popular because they are genuinely good or broadly relevant, and removing popularity completely usually produces worse recommendations.

The appropriate amount of popularity also depends on the domain and the user. A news service may reasonably promote widely read stories, while a music or book catalog usually benefits from deeper personalization, and a new user with no history may be best served by popular defaults until the system learns more.

  • Not all popularity is bias: the problem is recommending popular items beyond what each user’s preferences justify.
  • More diversity is not automatically better: random long-tail items reduce satisfaction and trust.
  • Popularity bias differs from position bias: position bias concerns where an item appears, popularity bias concerns which items appear at all.
  • Fixing it offline does not prove an online gain: only controlled online tests show the real effect on users and providers.

In plain words

Think of a bookstore clerk who only ever suggests the current bestsellers. Those books are fine, but a reader who loves obscure poetry hears the same few titles as everyone else, and wonderful books on the back shelves never get noticed.

Popularity bias is that habit inside an algorithm. Because popular things produce the most data, the system keeps pushing what is already popular, and the cycle repeats. Good systems balance popular hits with relevant discoveries for each person.

A real-world example

A music streaming service notices that its personalized playlists draw over 70% of their tracks from the top 2% of its catalog, even for listeners whose histories are mostly independent artists. Engagement is stable, but niche listeners churn more often and smaller labels complain about poor visibility.

The team adds inverse propensity weighting to training and a calibrated re-ranking step that matches each playlist’s popularity mix to the listener’s own history. In an online test over four weeks, long-tail coverage roughly doubles, average listening time stays flat, and retention among niche listeners improves, which persuades the team to roll the change out to all users.