A recommender system, also called a recommendation engine or recommendation system, is software that predicts which items a user is likely to value and presents a ranked selection of them, based on past behavior, item attributes and context. It is one of the most widely deployed applications of artificial intelligence, shaping what people see in online stores, streaming catalogs, news feeds, job boards and app stores.
The field of Recommender systems studies how these engines learn preferences from sparse signals, how they rank millions of candidates within milliseconds and how they balance accuracy against diversity, fairness and business goals. A modern recommender is rarely one model: it is a multi-stage pipeline that retrieves, scores, re-ranks and finally filters items before anything reaches the screen.
How a recommender system produces a ranked list
A recommender system turns interaction data into a ranking. Every click, purchase, rating, watch duration or skip becomes a signal stored in a sparse user-item matrix, where rows are users, columns are items and most cells are empty. The system learns patterns from the filled cells and estimates the missing ones.
Production systems split this work into stages because scoring every item for every request is too expensive. Candidate generation narrows a catalog of millions to a few hundred items using fast methods such as embedding similarity search. A heavier ranking model then scores each candidate with rich features, and a re-ranking stage applies business rules, diversity constraints and deduplication.
The table below shows the purpose and typical techniques of each stage.
| Stage | Input size | Typical technique | Main goal |
|---|---|---|---|
| Candidate generation | Millions of items | Two-tower model, approximate nearest neighbors | High recall at low cost |
| Ranking | Hundreds of items | Gradient-boosted trees, deep CTR models | Precise relevance scores |
| Re-ranking | Tens of items | Diversity rules, multi-objective scoring | Balanced final list |
| Filtering | Final slate | Policy, availability and safety checks | Remove invalid items |
Main families of recommendation methods
Recommendation methods differ mainly in which evidence they trust: the behavior of similar users, the properties of the items, or both. Each family has a characteristic strength and a characteristic blind spot.
Collaborative filtering infers preferences from the crowd: users who agreed in the past are likely to agree again. Matrix factorization compresses the user-item matrix into low-dimensional embeddings, so that the dot product of a user vector and an item vector approximates their affinity. Content-based filtering instead matches item attributes such as genre, text or price with a profile of what the user liked before.
| Approach | Evidence used | Strength | Weakness |
|---|---|---|---|
| Collaborative filtering | Interactions of many users | Finds non-obvious affinities | Fails for new users and items |
| Content-based filtering | Item attributes and user profile | Works for new items | Recommends overly similar items |
| Hybrid recommender | Interactions plus attributes | Covers both weaknesses | More complex to build and tune |
| Sequential models | Order of recent actions | Captures short-term intent | Needs dense session data |
| LLM-based recommendation | Text descriptions and dialogue | Handles natural-language requests | Costly and harder to control |
Deep learning architectures such as Wide & Deep, DeepFM and transformer-based sequence models like SASRec combine these signals with context features such as time, device and location. In practice, most large platforms run hybrid systems, even when one family dominates.
Feedback signals and how recommenders are evaluated
The quality of a recommender system depends heavily on the type of feedback it learns from. Explicit feedback, such as star ratings, states preference directly but is scarce. Implicit feedback, such as clicks and dwell time, is abundant but noisy: a click may reflect curiosity, a misleading thumbnail or position bias rather than genuine interest.
Evaluation therefore runs on two levels. Offline metrics measure how well a model reproduces held-out interactions, while online A/B tests measure the effect on real users. Offline gains frequently fail to translate online, so mature teams treat offline results as a filter rather than proof.
| Metric | What it measures | Typical use |
|---|---|---|
| Precision and recall at K | Share of relevant items in the top K | Top-N list quality |
| NDCG | Ranking quality weighted by position | Ordered result lists |
| Catalog coverage | Share of items ever recommended | Long-tail exposure |
| Diversity and serendipity | Variety and pleasant surprise | Beyond-accuracy assessment |
| Engagement and retention | Long-term user behavior | Online A/B testing |
Designing a recommender system for a real product
Design starts with the objective the business actually wants to optimize. Maximizing clicks alone tends to favor sensational or repetitive items; platforms therefore blend click probability with signals such as completion, satisfaction surveys, purchase value and long-term retention in a multi-objective score.
The decision table below links common product situations with a sensible starting approach.
| Situation | Recommended start | Reason |
|---|---|---|
| Small catalog, little interaction data | Content-based filtering with popularity fallback | Avoids the cold start problem |
| Large catalog, rich history | Matrix factorization or two-tower retrieval | Scales and captures latent taste |
| Short anonymous sessions | Session-based sequential model | No long-term profile exists |
| Many new items daily, such as news | Hybrid with strong content features | New items lack interactions |
| Need to explain choices | Item-based collaborative filtering | Similar-item reasoning is easy to show |
Latency budgets also shape the design. Interactive surfaces usually require the whole pipeline to respond in tens to a few hundred milliseconds, which pushes expensive models to the later stages and favors precomputed embeddings for retrieval.
Risks, biases and common failure modes
Recommender systems learn from data they themselves helped create, which produces a feedback loop. Items shown more often collect more clicks, are judged more relevant and are shown even more, amplifying popularity bias and narrowing what users discover.
- Cold start problem: new users and new items have no history, so pure collaborative methods cannot rank them.
- Filter bubbles: narrow optimization can confine users to a shrinking range of content.
- Shilling attacks: fake profiles and coordinated ratings can manipulate rankings.
- Fairness concerns: providers or user groups may receive systematically less exposure.
- Metric gaming: optimizing a short-term proxy can hurt satisfaction and trust over time.
Mitigation combines exploration strategies, debiasing techniques for position and selection effects, diversity constraints in re-ranking and regular audits. Regulations such as the EU Digital Services Act also require large platforms to explain their main ranking parameters and to offer at least one option not based on profiling.
Put simply
A recommender system works like an attentive shop assistant who remembers what you bought, notices what similar customers liked and then suggests a few things you might enjoy. The difference is that it does this for millions of people at once, in a fraction of a second.
It learns from what you click, watch, buy or skip. The more it learns, the better its guesses become, but it can also keep showing you the same kind of thing. Good systems therefore mix familiar picks with fresh discoveries.
In practice
An online bookstore with a catalog of about 400,000 titles wants to improve the “you may also like” row on its home page. Its first version simply showed bestsellers, so most customers saw the same twenty books.
The data team builds a two-stage recommender. A two-tower retrieval model, trained on two years of purchases and browsing sessions, selects 300 candidates per visitor. A gradient-boosted ranking model then scores them using recent views, price range, author affinity and time of day, and a re-ranking step limits each author to two slots and reserves one slot for a newly released title.
In a four-week A/B test, the new row raises click-through by a clear margin and spreads sales across far more of the catalog. New releases, which the bestseller list rarely showed, now reach readers in their first week.