BERT4Rec

On this page 7

BERT4Rec is a sequential recommendation model that applies a bidirectional Transformer encoder to a user’s ordered history of interactions and predicts which item comes next. Introduced in 2019, it adapts the masked language modeling idea behind BERT to recommender systems: instead of predicting hidden words in a sentence, it learns to predict hidden items in a behavior sequence, such as the films a person watched or the products a shopper viewed.

BERT4Rec belongs to the family of Transformer-based sequential recommenders alongside SASRec. Its defining feature is that every position in the sequence can attend to both earlier and later interactions during training, which aims to produce richer item representations than strictly left-to-right models.

How BERT4Rec models a behavior sequence

BERT4Rec models a behavior sequence by turning each interaction into an item embedding plus a learned position embedding and passing the sequence through a stack of Transformer layers. Each layer combines multi-head self-attention with a position-wise feed-forward network, residual connections and layer normalization. The output vector at each position is projected onto the item vocabulary to score every candidate item.

Unlike unidirectional models, BERT4Rec uses no causal attention mask. A hidden position can see the items before and after it, which helps the model capture patterns where context on both sides matters, for example a purchase that is explained by the items a user browsed around it. The table below lists the main components and their role.

ComponentRoleTypical setting
Item embeddingRepresents each item as a dense vector64 to 256 dimensions
Position embeddingEncodes the order of interactionsLearned, one per position
Self-attention layersRelate every item to every other item2 to 4 layers
Attention headsCapture several relation types in parallel2 to 4 heads
Output projectionScores all items in the catalogOften tied to item embeddings
Core components of the BERT4Rec architecture

The Cloze objective and masked item prediction

BERT4Rec is trained with a Cloze task, also called masked item prediction. During training, a random share of items in each sequence is replaced by a special mask token, and the model must recover the original items from the surrounding context. Because each sequence yields several masked targets at once, the objective creates more training signal per sequence than predicting only the final item.

At inference time, the model appends a mask token to the end of the user’s history and ranks all items by the probability assigned at that position. Training sometimes also includes sequences in which only the last item is masked, so that the training task more closely matches the prediction task. The table below summarizes the key hyperparameters that control this behavior.

HyperparameterWhat it controlsCommon rangeEffect of increasing it
Mask proportionShare of items hidden per sequence0.15 to 0.6More targets, harder context
Maximum sequence lengthNumber of recent interactions kept50 to 200More context, higher memory cost
Hidden sizeWidth of embeddings and layers64 to 256More capacity, slower training
Dropout rateRegularization strength0.1 to 0.3Less overfitting on small data
Training epochsLength of optimizationOften hundredsBetter convergence, higher cost
Hyperparameters that shape BERT4Rec training

The best mask proportion depends strongly on the dataset: long, dense sequences such as film ratings usually tolerate a lower ratio, while short and sparse sequences benefit from masking a larger share to generate enough training targets.

How BERT4Rec compares with SASRec and other sequential models

BERT4Rec differs from SASRec mainly in the direction of attention and the training objective. SASRec uses causal self-attention and predicts the next item at every position, whereas BERT4Rec uses bidirectional attention and the Cloze objective. Earlier recurrent models such as GRU4Rec process sequences step by step and struggle with long-range dependencies.

The comparison below summarizes the practical differences that matter when choosing a sequential recommender.

ModelAttention directionTraining objectivePractical trait
BERT4RecBidirectionalMasked item predictionSlow to converge, strong when well tuned
SASRecLeft to rightNext-item predictionFast training, strong baseline
GRU4RecRecurrent, sequentialNext-item predictionDesigned for short sessions
CaserConvolutionalNext-item predictionCaptures local patterns
BERT4Rec compared with related sequential recommenders

A widely cited reproducibility debate shapes how BERT4Rec should be judged. Later replication studies found that the original reported gains were not consistently reproducible, that BERT4Rec needs much longer training than its first implementations used, and that SASRec trained with a full cross-entropy loss often matches or exceeds it. The fair conclusion is that both models are competitive and that training budget and loss choice matter as much as architecture.

Deploying BERT4Rec in a recommendation pipeline

In production, BERT4Rec usually serves as a candidate generation or reranking stage in multi-stage ranking rather than as the only model. Scoring the full catalog with a softmax layer becomes expensive when there are millions of items, so teams often rely on negative sampling, sampled softmax or approximate nearest neighbor search over the learned item embeddings.

The table below lists typical deployment problems and the usual remedies.

ProblemCauseRemedy
High serving latencyFull softmax over a large catalogApproximate nearest neighbor retrieval
Poor results for new itemsCold start problem in ID embeddingsAdd content features or hybrid models
Unstable offline metricsSampled evaluation metricsEvaluate against the full item set
Weak gains over SASRecToo few training epochsTrain longer and tune the mask ratio
Popular items dominatePopularity bias in training dataApply diversity or calibration reranking
Common deployment issues and remedies

Limitations of BERT4Rec

BERT4Rec has several limitations that experienced practitioners account for before adopting it. The most important is a mismatch between training and inference: the model learns to fill gaps inside sequences but is used to predict only the final position.

  • Pure ID-based inputs give no information about new items, so the cold start problem remains.
  • Quadratic attention cost limits how much history can be processed per user.
  • Evaluation with sampled negatives can overstate quality and change model rankings.
  • Long convergence times raise training costs compared with SASRec.
  • Offline accuracy gains do not guarantee better engagement in online A/B tests.

In plain words

BERT4Rec looks at the list of things a person has watched, bought or clicked, in order. It then guesses what they will want next. To learn this skill, it plays a fill-in-the-blank game with millions of these lists. Some items are hidden, and the model must guess them from what came before and after.

After enough practice, it understands which items tend to go together and in what order. The key idea is reading the whole history in both directions, much like a person who reads an entire sentence before guessing a missing word.

A real-world example

A mid-sized online bookstore wants better suggestions on its homepage. Its earlier system relied on item-based collaborative filtering, which ignored the order in which customers browsed. The data team trains BERT4Rec on the last 100 interactions of each customer, using a mask proportion of 0.3, and compares it against a well-tuned SASRec model.

Offline, both Transformer models clearly beat the old system, and BERT4Rec edges ahead only after the team extends training several times longer than its first attempt. The team deploys it as a candidate generator that returns 500 books per customer, followed by a lighter ranking model. An online A/B test shows a modest rise in click-through rate on recommendations, and the team keeps SASRec as a fallback for faster retraining.