A two-tower model is a neural network architecture for recommender systems in which one network, the query or user tower, encodes the user and context into a vector, and a second network, the item tower, encodes each candidate item into a vector of the same size. Relevance is scored by a simple function of the two vectors, usually the dot product or cosine similarity.
Because the two towers never interact until the final score, item vectors can be computed in advance and indexed, which lets the system search millions or billions of items in milliseconds. This property makes the two-tower recommender the dominant design for candidate generation, the first stage of modern multi-stage ranking.
How the two towers produce a relevance score
Each tower is an independent encoder, often a multilayer perceptron over embedded features, and sometimes a sequence model or transformer. The user tower consumes features such as user ID embeddings, recent interaction history, device and time of day; the item tower consumes item ID, category, text and image embeddings and freshness signals.
Both towers output vectors of a fixed dimension, typically 64 to 256. The model score is s(u, i) = f(u) · g(i), often divided by a temperature and computed on normalized vectors. This late interaction is the defining constraint: no feature crossing between user and item happens inside the network.
The table below summarizes the main components and the role each one plays.
| Component | Typical inputs or form | Role |
|---|---|---|
| User tower | User ID, history, context features | Encodes current intent as a query vector |
| Item tower | Item ID, metadata, content embeddings | Encodes each item as a candidate vector |
| Similarity function | Dot product or cosine with temperature | Turns two vectors into one score |
| ANN index | HNSW, IVF or product-quantized index | Finds top-scoring items without a full scan |
| Training loss | Sampled softmax or contrastive loss | Pulls positive pairs together, pushes negatives apart |
Training with negatives and bias correction
Two-tower models are usually trained on implicit feedback such as clicks, watches or purchases, where each observed interaction is a positive pair. The model must also see negative examples, and the choice of negatives shapes quality more than almost any architectural detail.
The most common technique is in-batch negative sampling: every other item in the training batch serves as a negative for a given user. It is efficient, but popular items appear in batches more often and get over-penalized, so practitioners apply a logQ sampling-bias correction that subtracts the log of each item’s sampling probability from its score.
The following table compares the main negative sampling strategies.
| Strategy | How negatives are chosen | Strength | Weakness |
|---|---|---|---|
| In-batch negatives | Other positives in the same batch | No extra compute | Biased toward popular items |
| Uniform random negatives | Sampled evenly from the catalog | Covers the long tail | Mostly too easy to be informative |
| Mixed negatives | In-batch plus random samples | Balances popularity and coverage | More tuning required |
| Hard negatives | Items the model currently ranks high but are wrong | Sharpens fine distinctions | Risk of false negatives and instability |
Batch size matters: larger batches supply more negatives and generally improve recall, which is one reason production training often uses batches of several thousand pairs.
Two-tower models compared with other recommender architectures
The two-tower model generalizes matrix factorization: if each tower were only an ID embedding lookup, the model would reduce to classic latent factor collaborative filtering. Adding features to each tower lets it handle new items and sparse users better, which eases the cold start problem.
Its weakness is the lack of cross features. Ranking models such as Wide & Deep, DeepFM and Deep & Cross Network combine user and item features early, which yields higher precision but requires scoring every candidate separately. The table below shows where each approach fits.
| Approach | Feature interaction | Serving cost | Typical stage |
|---|---|---|---|
| Two-tower model | Only at the final dot product | Very low with precomputed item vectors | Candidate generation |
| Matrix factorization | Only ID embeddings | Very low | Retrieval or baseline |
| Cross-feature ranking model | Early and explicit | High per candidate | Ranking of hundreds of items |
| Cross-encoder transformer | Full attention over user and item | Highest | Final reranking |
In a typical pipeline, the two-tower model retrieves a few hundred to a few thousand candidates, a heavier ranking model scores them with cross features, and a final step applies diversity, freshness and business rules.
Serving a two-tower recommender in production
Serving splits naturally into offline and online work. Item embeddings are recomputed in batch, often daily or hourly, and loaded into an approximate nearest neighbor (ANN) index. At request time, only the user tower runs, producing one query vector that is matched against the index in a few milliseconds.
- Keep tower versions in sync: a user tower and an item index from different training runs live in incompatible vector spaces and produce nonsense results.
- Refresh new items quickly: items absent from the index cannot be retrieved, so streaming updates help catalogs with fast turnover.
- Tune ANN recall: aggressive index compression saves memory but can drop relevant candidates before ranking ever sees them.
- Use multiple retrievers: many systems combine several two-tower models with popularity and freshness sources to improve recommendation diversity.
Limitations and common mistakes
The architectural separation that makes the two-tower model fast also limits what it can express. The table below lists frequent problems, their causes and common remedies.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Recommendations dominated by bestsellers | Uncorrected in-batch sampling bias | Apply logQ correction and mixed negatives |
| Poor results for new items | Item tower relies mainly on ID embeddings | Add content and metadata features |
| Good offline recall, weak online gains | Ranking stage or ANN index loses candidates | Measure recall at each pipeline stage |
| Misses context-specific preferences | No user-item feature crossing | Leave interactions to the ranking model |
A common misconception is that a two-tower model can replace the whole ranking stack. It is optimized for high recall at low cost, not for precise ordering, and it can amplify popularity bias and feedback loops if its training data come only from items it previously recommended.
In everyday language
Imagine a huge library where a helper writes a short profile for every book in advance and keeps those profiles on file. When a visitor arrives, the helper writes one quick profile of that person and looks for the books whose profiles match best.
A two-tower model works the same way. One part describes the person, the other describes each item, and both descriptions are lists of numbers that can be compared instantly. Because the item descriptions are ready beforehand, the system can search an enormous catalog in a blink and pass a short list to more careful ranking steps.
Seen in the real world
A video streaming service with about 5 million titles and clips needs to fill a personalized home page in under 100 ms. Scoring every title with its heavy ranking model would take far too long, so the engineering team adds a two-tower retrieval stage.
The user tower reads the last 50 watched videos, language and device; the item tower reads title, genre, description embeddings and upload age. Both output 128-dimensional vectors, and item vectors are rebuilt every few hours into an ANN index. At request time, the system retrieves 1,000 candidates in roughly 10 ms, and the ranking model orders them.
After adding logQ correction and mixed negatives, the share of long-tail videos in recommendations rises noticeably, and watch time per session improves in an online test against the previous matrix factorization retriever.