Factorization machine

On this page 7

A factorization machine (FM) is a supervised learning model that combines a linear model with factorized pairwise feature interactions, representing every feature by a short latent vector and estimating the strength of each interaction as the dot product of two such vectors. In recommender systems, factorization machines are used to predict ratings, clicks or purchases from highly sparse data, such as one-hot encoded user IDs, item IDs, categories and context signals.

Introduced by Steffen Rendle in 2010, the model generalizes matrix factorization to arbitrary feature sets. Because interaction weights are shared through latent vectors, a factorization machine can estimate interactions between feature pairs that never appear together in training data, which is the key reason it remains a strong baseline for CTR prediction and context-aware recommendation.

How a factorization machine models feature interactions

A factorization machine predicts a target as a global bias, plus a weighted sum of features, plus a sum over all feature pairs, in which each pair is weighted by the dot product of the two features’ latent vectors. For n features and latent dimension k, the model learns one bias, n linear weights and an n by k matrix of latent factors.

A plain polynomial regression would need a separate weight for every feature pair, which is impossible to estimate when most pairs are rarely or never observed. The factorization replaces each pairwise weight with a low-rank product, so information learned from one pair transfers to any other pair that shares a feature.

A naive evaluation of all pairs costs O(k n squared), but an algebraic reformulation reduces it to linear time, O(k n), and in sparse inputs only the non-zero features count. The table below lists the model’s parameters and their role.

ParameterShapeRole in the prediction
Global biasSingle scalarBaseline level of the target
Linear weightsOne per featureMain effect of each feature alone
Latent factorsn by k matrixPairwise interaction strengths
Latent dimension kHyperparameter, often 8-128Capacity of the interaction model
Regularization termsPer parameter groupControls overfitting on sparse data
Parameters of a second-order factorization machine

Feature encoding for recommendation tasks

A factorization machine expects every training example as one sparse feature vector, so recommendation data must be encoded into that form. A user and an item become one-hot indicator features, and additional information is appended as further indicator or real-valued features.

This flexible encoding is what lets a single model imitate several classic recommenders. With only user and item indicators, a factorization machine is equivalent to biased matrix factorization; adding the set of previously rated items reproduces SVD++-style implicit feedback models; adding time or device features yields a context-aware recommender. The following table shows typical feature groups.

Feature groupEncodingWhat the interactions capture
User IDOne-hot indicatorPersonal preference
Item IDOne-hot indicatorItem-specific appeal
Item attributesMulti-hot genres or categoriesTaste for content types, helps cold items
ContextOne-hot time slot, device, locationSituational preference
User historyNormalized multi-hot of past itemsImplicit taste profile
Typical feature groups in a recommendation feature vector

Attribute and history features make factorization machines more robust to the cold start problem than pure matrix factorization, because a new item still shares latent vectors through its genre, brand or price band.

Training objectives and learning methods

Factorization machines can be trained for regression, binary classification or ranking by changing only the loss function. Squared error suits rating prediction, logistic loss suits click or conversion prediction, and pairwise losses such as Bayesian personalized ranking suit top-N recommendation from implicit feedback.

The original libFM implementation offered stochastic gradient descent, alternating least squares and Markov chain Monte Carlo inference. The MCMC variant integrates out regularization hyperparameters automatically, which makes it attractive when tuning budgets are small. Modern deployments usually train factorization machines with mini-batch gradient methods such as Adam or FTRL in general machine learning frameworks.

Regularization is essential: on sparse data the latent vectors of rare features easily overfit, so L2 penalties, often grouped by feature type, and early stopping on a time-ordered validation split are standard practice.

Factorization machines sit between classic collaborative filtering and deep CTR models. They model all second-order interactions efficiently but cannot learn higher-order or nonlinear interactions on their own, which motivated several extensions.

ModelInteraction modelingStrengthLimitation
Matrix factorizationUser-item onlySimple and fastNo side features or context
Factorization machineAll feature pairs, shared vectorsGeneralizes on sparse dataSecond order and linear only
Field-aware factorization machineSeparate vector per field pairHigher CTR accuracyMany more parameters
DeepFMFM component plus deep networkLearns higher-order interactionsHeavier training and serving
Wide & DeepManual crosses plus deep networkMemorization with generalizationNeeds feature engineering
Factorization machines and neighboring recommendation models

In practice a well-tuned factorization machine is a hard baseline to beat on tabular click data. Deep extensions often gain only modest improvements, so the extra serving cost should be justified by offline and online evaluation, not assumed.

Practical use, limitations and common mistakes

Factorization machines are most useful as a ranking model in a multi-stage pipeline or as a standalone model for mid-sized catalogs. The decision table below summarizes when they fit well.

SituationRecommended choiceReason
Sparse categorical data, tight latencySecond-order FMLinear-time scoring, small model
Ad click prediction with many fieldsField-aware FM or DeepFMField-specific interactions matter
Only user-item interactions availableMatrix factorizationFM adds little without side features
Rich sequences of user behaviorSequential model such as SASRecFM ignores event order
When to choose a factorization machine
  • Ignoring feature scaling: unnormalized numeric features dominate the interaction sums and destabilize training.
  • Overly large latent dimensions increase overfitting on rare features without improving accuracy.
  • Random train and test splits leak future behavior; time-based splits give realistic estimates.
  • Expecting nonlinear patterns: a plain FM cannot express interactions beyond the second order.

Stripped to the essentials

A factorization machine guesses how likely a person is to like or click something. It looks at how pairs of facts work together. One pair might be this user and this genre. Another might be this phone and this time of day.

The model does not memorize every possible pair. It gives each fact a short numeric profile. Two profiles that point in a similar direction suggest a good match. Because profiles are shared, the model can guess well about unseen combinations. Most recommendation systems face exactly that situation.

Where it is applied

A regional online bookstore with about 200,000 titles and 1.5 million customers wants to rank books on its home page. Most customers have bought only a handful of books, so the purchase matrix is more than 99.9% empty, and new titles arrive every week without any history.

The data team trains a factorization machine with logistic loss on features for customer, book, author, genre, price band and device, using a latent dimension of 32 and a time-based validation split. The model reranks 500 candidates per visit in a few milliseconds on ordinary CPU servers.

In an online A/B test, the new ranking increases home page click-through by roughly 9% over a popularity baseline, and newly released books by known authors receive clicks from their first day because their genre and author vectors are already learned.