Recommender system

Also known as: recommendation engine · recommendation system

On this page 7

A recommender system, also called a recommendation engine or recommendation system, is software that predicts which items a user is likely to value and presents a ranked selection of them, based on past behavior, item attributes and context. It is one of the most widely deployed applications of artificial intelligence, shaping what people see in online stores, streaming catalogs, news feeds, job boards and app stores.

The field of Recommender systems studies how these engines learn preferences from sparse signals, how they rank millions of candidates within milliseconds and how they balance accuracy against diversity, fairness and business goals. A modern recommender is rarely one model: it is a multi-stage pipeline that retrieves, scores, re-ranks and finally filters items before anything reaches the screen.

How a recommender system produces a ranked list

A recommender system turns interaction data into a ranking. Every click, purchase, rating, watch duration or skip becomes a signal stored in a sparse user-item matrix, where rows are users, columns are items and most cells are empty. The system learns patterns from the filled cells and estimates the missing ones.

Production systems split this work into stages because scoring every item for every request is too expensive. Candidate generation narrows a catalog of millions to a few hundred items using fast methods such as embedding similarity search. A heavier ranking model then scores each candidate with rich features, and a re-ranking stage applies business rules, diversity constraints and deduplication.

The table below shows the purpose and typical techniques of each stage.

StageInput sizeTypical techniqueMain goal
Candidate generationMillions of itemsTwo-tower model, approximate nearest neighborsHigh recall at low cost
RankingHundreds of itemsGradient-boosted trees, deep CTR modelsPrecise relevance scores
Re-rankingTens of itemsDiversity rules, multi-objective scoringBalanced final list
FilteringFinal slatePolicy, availability and safety checksRemove invalid items
Stages of a production recommendation pipeline

Main families of recommendation methods

Recommendation methods differ mainly in which evidence they trust: the behavior of similar users, the properties of the items, or both. Each family has a characteristic strength and a characteristic blind spot.

Collaborative filtering infers preferences from the crowd: users who agreed in the past are likely to agree again. Matrix factorization compresses the user-item matrix into low-dimensional embeddings, so that the dot product of a user vector and an item vector approximates their affinity. Content-based filtering instead matches item attributes such as genre, text or price with a profile of what the user liked before.

ApproachEvidence usedStrengthWeakness
Collaborative filteringInteractions of many usersFinds non-obvious affinitiesFails for new users and items
Content-based filteringItem attributes and user profileWorks for new itemsRecommends overly similar items
Hybrid recommenderInteractions plus attributesCovers both weaknessesMore complex to build and tune
Sequential modelsOrder of recent actionsCaptures short-term intentNeeds dense session data
LLM-based recommendationText descriptions and dialogueHandles natural-language requestsCostly and harder to control
Comparison of recommendation approaches

Deep learning architectures such as Wide & Deep, DeepFM and transformer-based sequence models like SASRec combine these signals with context features such as time, device and location. In practice, most large platforms run hybrid systems, even when one family dominates.

Feedback signals and how recommenders are evaluated

The quality of a recommender system depends heavily on the type of feedback it learns from. Explicit feedback, such as star ratings, states preference directly but is scarce. Implicit feedback, such as clicks and dwell time, is abundant but noisy: a click may reflect curiosity, a misleading thumbnail or position bias rather than genuine interest.

Evaluation therefore runs on two levels. Offline metrics measure how well a model reproduces held-out interactions, while online A/B tests measure the effect on real users. Offline gains frequently fail to translate online, so mature teams treat offline results as a filter rather than proof.

MetricWhat it measuresTypical use
Precision and recall at KShare of relevant items in the top KTop-N list quality
NDCGRanking quality weighted by positionOrdered result lists
Catalog coverageShare of items ever recommendedLong-tail exposure
Diversity and serendipityVariety and pleasant surpriseBeyond-accuracy assessment
Engagement and retentionLong-term user behaviorOnline A/B testing
Common evaluation metrics for recommender systems

Designing a recommender system for a real product

Design starts with the objective the business actually wants to optimize. Maximizing clicks alone tends to favor sensational or repetitive items; platforms therefore blend click probability with signals such as completion, satisfaction surveys, purchase value and long-term retention in a multi-objective score.

The decision table below links common product situations with a sensible starting approach.

SituationRecommended startReason
Small catalog, little interaction dataContent-based filtering with popularity fallbackAvoids the cold start problem
Large catalog, rich historyMatrix factorization or two-tower retrievalScales and captures latent taste
Short anonymous sessionsSession-based sequential modelNo long-term profile exists
Many new items daily, such as newsHybrid with strong content featuresNew items lack interactions
Need to explain choicesItem-based collaborative filteringSimilar-item reasoning is easy to show
Choosing a starting approach

Latency budgets also shape the design. Interactive surfaces usually require the whole pipeline to respond in tens to a few hundred milliseconds, which pushes expensive models to the later stages and favors precomputed embeddings for retrieval.

Risks, biases and common failure modes

Recommender systems learn from data they themselves helped create, which produces a feedback loop. Items shown more often collect more clicks, are judged more relevant and are shown even more, amplifying popularity bias and narrowing what users discover.

  • Cold start problem: new users and new items have no history, so pure collaborative methods cannot rank them.
  • Filter bubbles: narrow optimization can confine users to a shrinking range of content.
  • Shilling attacks: fake profiles and coordinated ratings can manipulate rankings.
  • Fairness concerns: providers or user groups may receive systematically less exposure.
  • Metric gaming: optimizing a short-term proxy can hurt satisfaction and trust over time.

Mitigation combines exploration strategies, debiasing techniques for position and selection effects, diversity constraints in re-ranking and regular audits. Regulations such as the EU Digital Services Act also require large platforms to explain their main ranking parameters and to offer at least one option not based on profiling.

Put simply

A recommender system works like an attentive shop assistant who remembers what you bought, notices what similar customers liked and then suggests a few things you might enjoy. The difference is that it does this for millions of people at once, in a fraction of a second.

It learns from what you click, watch, buy or skip. The more it learns, the better its guesses become, but it can also keep showing you the same kind of thing. Good systems therefore mix familiar picks with fresh discoveries.

In practice

An online bookstore with a catalog of about 400,000 titles wants to improve the “you may also like” row on its home page. Its first version simply showed bestsellers, so most customers saw the same twenty books.

The data team builds a two-stage recommender. A two-tower retrieval model, trained on two years of purchases and browsing sessions, selects 300 candidates per visitor. A gradient-boosted ranking model then scores them using recent views, price range, author affinity and time of day, and a re-ranking step limits each author to two slots and reserves one slot for a newly released title.

In a four-week A/B test, the new row raises click-through by a clear margin and spreads sales across far more of the catalog. New releases, which the bestseller list rarely showed, now reach readers in their first week.