Rating prediction

On this page 7

Rating prediction is the task of estimating the numerical score a user would give to an item they have not yet rated, such as four stars for a film or seven out of ten for a hotel. In recommender systems, a branch of applied artificial intelligence, it was the defining problem of the field for more than a decade, and it remains the standard way to model explicit feedback.

Formally, the system learns a function that maps a user-item pair to a predicted rating, written as r-hat for user u and item i, from the sparse set of ratings already observed in the user-item matrix. The predictions can then be sorted to recommend items, shown as expected scores, or used as features in larger ranking models.

How rating prediction models learn from sparse ratings

Rating prediction models learn by fitting the observed ratings and generalizing to the empty cells of the user-item matrix, which in typical catalogs is more than 99 percent empty. Every method must therefore share information across users, across items, or both.

The simplest strong model is the baseline predictor: the global average rating plus a user bias (some people rate generously) plus an item bias (some items are widely liked). Baselines alone often explain a large share of the variance, and more complex models are usually built on top of them.

The main families of rating prediction methods are compared below.

ApproachCore mechanismStrengthWeakness
Baseline predictorGlobal mean plus user and item biasesFast, robust, hard to beat cheaplyNo personal taste beyond bias
Neighborhood methodsWeighted ratings of similar users or itemsEasy to explainWeak on sparse data, costly at scale
Matrix factorizationDot product of user and item latent vectorsHigh accuracy, compactCold start for new users and items
Factorization machinesPairwise interactions of arbitrary featuresAdds context and side dataMore tuning required
Neural modelsLearned nonlinear interaction functionsFlexible with rich featuresOften no gain on pure ratings
Common approaches to rating prediction

Matrix factorization is the reference method. Each user and item receives a vector of typically 20 to 200 latent factors, and the predicted rating is the baseline plus the dot product of the two vectors. Parameters are learned with stochastic gradient descent or alternating least squares, with L2 regularization to avoid overfitting the few ratings each user provides.

Measuring rating prediction accuracy

Rating prediction is evaluated with error metrics on held-out ratings, which measure how far predicted scores fall from the true ones. The two standard metrics are root mean squared error and mean absolute error.

MetricDefinitionBehaviorTypical use
RMSESquare root of the mean squared errorPenalizes large errors heavilyBenchmark reporting, competitions
MAEMean of absolute errorsTreats all errors linearlyInterpretable error in stars
Normalized MAEMAE divided by the rating rangeComparable across scalesComparing 5-point and 10-point data
Rank correlationAgreement of predicted and true orderingsIgnores absolute scaleChecking ranking usefulness
Accuracy metrics for rating prediction

Typical values depend heavily on the dataset. On the MovieLens benchmarks with one-to-five-star ratings, well-tuned factorization models usually reach RMSE values around 0.75 to 0.90, while a plain item-average predictor scores noticeably worse. Small RMSE differences, such as 0.01, can be statistically significant on large datasets yet invisible to users.

The Netflix Prize made RMSE famous. Teams competed from 2006 to 2009 to improve the company’s own predictor by 10 percent in RMSE, and the winning solution blended hundreds of models, including factorization variants that used implicit signals about which films a user had rated at all.

Rating prediction compared with ranking-oriented tasks

Rating prediction answers “how much will this user like this item”, whereas most modern recommenders answer “which few items should this user see first”. The difference changes the data, the loss function and the evaluation.

TaskTypical inputTraining objectiveEvaluation
Rating predictionExplicit star or score ratingsSquared error regressionRMSE, MAE
Top-N recommendationImplicit clicks, plays, purchasesRanking loss or classificationPrecision, recall, NDCG
CTR predictionImpressions with click labelsLog loss on click probabilityAUC, log loss
Learning to rankCandidate lists with relevance labelsPairwise or listwise lossNDCG, MAP
Rating prediction versus related recommendation tasks

Research has shown that low RMSE does not guarantee good recommendations. A model can predict ratings for obscure items accurately while ranking the items a user would actually choose poorly, because error metrics are dominated by items users already chose to rate. For this reason, many teams moved from rating prediction to top-N recommendation with implicit feedback, using methods such as Bayesian personalized ranking.

Where rating prediction remains useful

Rating prediction is still the right formulation when explicit scores carry business meaning or must be displayed. Review platforms, film and book catalogs, and travel sites often show a personalized predicted score next to an item, which requires a calibrated estimate rather than a rank.

  • Predicted match scores: showing “you will probably rate this 4.2” builds trust when the number is well calibrated.
  • Features for ranking models: a predicted rating is a strong input signal for a later stage of multi-stage ranking.
  • Survey and preference modeling: predicting satisfaction scores for products, courses or services a person has not tried.
  • Teaching and benchmarking: the task is simple to define, which makes it a standard first exercise in recommender research.

Practical systems frequently combine explicit and implicit feedback. The SVD++ family, for instance, adds a term for the set of items a user interacted with, because the decision to rate is itself informative.

Pitfalls in building rating predictors

The most serious pitfall is selection bias: ratings are missing not at random, because people mostly rate items they chose and liked. A model trained only on observed ratings tends to overestimate scores for unseen items.

ProblemCauseRemedy
Selection biasUsers rate items they choseInverse propensity weighting, randomized data
Cold startNew users or items lack ratingsContent features, onboarding questions
Rating scale driftUsers interpret stars differentlyUser bias terms, normalization
Temporal driftTastes and item popularity changeTime-aware biases, recent data weighting
OverfittingFew ratings per userRegularization, early stopping
Shilling attacksFake ratings promote itemsAnomaly detection, trust weighting
Common problems in rating prediction and their remedies

A further mistake is evaluating on random splits only. A time-based split, where the model predicts later ratings from earlier ones, gives a more honest estimate of production accuracy.

Understanding it simply

Imagine a friend who has watched many films with you and remembers how you scored each one. When a new film comes out, that friend can guess, often quite closely, how many stars you would give it, by noticing what you liked before and what people with similar taste thought.

Rating prediction is a computer doing that guessing for millions of people and items at once. It fills in the blanks in a huge table of scores, and the best guesses can then be shown as personal recommendations.

Applied in practice

An online bookstore collects one-to-five-star ratings from about 200,000 customers on 80,000 titles, yet fewer than 0.1 percent of possible ratings exist. Its data team wants to show each logged-in customer a personal expected rating on every book page.

The team trains a matrix factorization model with 64 latent factors plus user and item biases, using ratings before March for training and later ratings for testing. The model reaches an RMSE of about 0.86, compared with roughly 0.98 for a simple book-average predictor.

After launch, the personal score appears beside the public average. In an A/B test, customers who see the predicted rating add more books to wish lists, and the predictions also become one input to the store’s ranking model for the home page.