Two-tower model

Also known as: two-tower recommender

On this page 7

A two-tower model is a neural network architecture for recommender systems in which one network, the query or user tower, encodes the user and context into a vector, and a second network, the item tower, encodes each candidate item into a vector of the same size. Relevance is scored by a simple function of the two vectors, usually the dot product or cosine similarity.

Because the two towers never interact until the final score, item vectors can be computed in advance and indexed, which lets the system search millions or billions of items in milliseconds. This property makes the two-tower recommender the dominant design for candidate generation, the first stage of modern multi-stage ranking.

How the two towers produce a relevance score

Each tower is an independent encoder, often a multilayer perceptron over embedded features, and sometimes a sequence model or transformer. The user tower consumes features such as user ID embeddings, recent interaction history, device and time of day; the item tower consumes item ID, category, text and image embeddings and freshness signals.

Both towers output vectors of a fixed dimension, typically 64 to 256. The model score is s(u, i) = f(u) · g(i), often divided by a temperature and computed on normalized vectors. This late interaction is the defining constraint: no feature crossing between user and item happens inside the network.

The table below summarizes the main components and the role each one plays.

ComponentTypical inputs or formRole
User towerUser ID, history, context featuresEncodes current intent as a query vector
Item towerItem ID, metadata, content embeddingsEncodes each item as a candidate vector
Similarity functionDot product or cosine with temperatureTurns two vectors into one score
ANN indexHNSW, IVF or product-quantized indexFinds top-scoring items without a full scan
Training lossSampled softmax or contrastive lossPulls positive pairs together, pushes negatives apart
Components of a two-tower model

Training with negatives and bias correction

Two-tower models are usually trained on implicit feedback such as clicks, watches or purchases, where each observed interaction is a positive pair. The model must also see negative examples, and the choice of negatives shapes quality more than almost any architectural detail.

The most common technique is in-batch negative sampling: every other item in the training batch serves as a negative for a given user. It is efficient, but popular items appear in batches more often and get over-penalized, so practitioners apply a logQ sampling-bias correction that subtracts the log of each item’s sampling probability from its score.

The following table compares the main negative sampling strategies.

StrategyHow negatives are chosenStrengthWeakness
In-batch negativesOther positives in the same batchNo extra computeBiased toward popular items
Uniform random negativesSampled evenly from the catalogCovers the long tailMostly too easy to be informative
Mixed negativesIn-batch plus random samplesBalances popularity and coverageMore tuning required
Hard negativesItems the model currently ranks high but are wrongSharpens fine distinctionsRisk of false negatives and instability
Negative sampling strategies for two-tower training

Batch size matters: larger batches supply more negatives and generally improve recall, which is one reason production training often uses batches of several thousand pairs.

Two-tower models compared with other recommender architectures

The two-tower model generalizes matrix factorization: if each tower were only an ID embedding lookup, the model would reduce to classic latent factor collaborative filtering. Adding features to each tower lets it handle new items and sparse users better, which eases the cold start problem.

Its weakness is the lack of cross features. Ranking models such as Wide & Deep, DeepFM and Deep & Cross Network combine user and item features early, which yields higher precision but requires scoring every candidate separately. The table below shows where each approach fits.

ApproachFeature interactionServing costTypical stage
Two-tower modelOnly at the final dot productVery low with precomputed item vectorsCandidate generation
Matrix factorizationOnly ID embeddingsVery lowRetrieval or baseline
Cross-feature ranking modelEarly and explicitHigh per candidateRanking of hundreds of items
Cross-encoder transformerFull attention over user and itemHighestFinal reranking
Two-tower model versus related approaches

In a typical pipeline, the two-tower model retrieves a few hundred to a few thousand candidates, a heavier ranking model scores them with cross features, and a final step applies diversity, freshness and business rules.

Serving a two-tower recommender in production

Serving splits naturally into offline and online work. Item embeddings are recomputed in batch, often daily or hourly, and loaded into an approximate nearest neighbor (ANN) index. At request time, only the user tower runs, producing one query vector that is matched against the index in a few milliseconds.

  • Keep tower versions in sync: a user tower and an item index from different training runs live in incompatible vector spaces and produce nonsense results.
  • Refresh new items quickly: items absent from the index cannot be retrieved, so streaming updates help catalogs with fast turnover.
  • Tune ANN recall: aggressive index compression saves memory but can drop relevant candidates before ranking ever sees them.
  • Use multiple retrievers: many systems combine several two-tower models with popularity and freshness sources to improve recommendation diversity.

Limitations and common mistakes

The architectural separation that makes the two-tower model fast also limits what it can express. The table below lists frequent problems, their causes and common remedies.

SymptomLikely causeTypical fix
Recommendations dominated by bestsellersUncorrected in-batch sampling biasApply logQ correction and mixed negatives
Poor results for new itemsItem tower relies mainly on ID embeddingsAdd content and metadata features
Good offline recall, weak online gainsRanking stage or ANN index loses candidatesMeasure recall at each pipeline stage
Misses context-specific preferencesNo user-item feature crossingLeave interactions to the ranking model
Common two-tower problems and remedies

A common misconception is that a two-tower model can replace the whole ranking stack. It is optimized for high recall at low cost, not for precise ordering, and it can amplify popularity bias and feedback loops if its training data come only from items it previously recommended.

In everyday language

Imagine a huge library where a helper writes a short profile for every book in advance and keeps those profiles on file. When a visitor arrives, the helper writes one quick profile of that person and looks for the books whose profiles match best.

A two-tower model works the same way. One part describes the person, the other describes each item, and both descriptions are lists of numbers that can be compared instantly. Because the item descriptions are ready beforehand, the system can search an enormous catalog in a blink and pass a short list to more careful ranking steps.

Seen in the real world

A video streaming service with about 5 million titles and clips needs to fill a personalized home page in under 100 ms. Scoring every title with its heavy ranking model would take far too long, so the engineering team adds a two-tower retrieval stage.

The user tower reads the last 50 watched videos, language and device; the item tower reads title, genre, description embeddings and upload age. Both output 128-dimensional vectors, and item vectors are rebuilt every few hours into an ANN index. At request time, the system retrieves 1,000 candidates in roughly 10 ms, and the ranking model orders them.

After adding logQ correction and mixed negatives, the share of long-tail videos in recommendations rises noticeably, and watch time per session improves in an online test against the previous matrix factorization retriever.