Embedding layer

On this page 7

An embedding layer is a trainable component of Neural networks that maps each discrete input index, such as a token, word, product or user ID, to a dense real-valued vector of fixed length. Mathematically it is a weight matrix of shape vocabulary size by embedding dimension, and the forward pass simply selects the row that corresponds to each input index.

Because the vectors are learned by backpropagation together with the rest of the model, inputs that behave similarly in the training task end up with similar vectors. The embedding layer is therefore the entry point of nearly every language model, recommender system and model that consumes categorical data.

How an embedding layer turns indices into vectors

An embedding layer works as an indexed lookup into a learnable matrix. If the vocabulary has V entries and the embedding dimension is d, the layer stores a matrix E of size V by d; the input index i returns row E[i]. A batch of token sequences of shape batch by length becomes a tensor of shape batch by length by d.

The lookup is mathematically equivalent to multiplying a one-hot vector by E, which is how the layer is often explained. Implementations never build the one-hot vector, because a direct gather costs memory proportional to d per token instead of V. Frameworks expose the operation as nn.Embedding in PyTorch and tf.keras.layers.Embedding in TensorFlow.

During training, only the rows that appear in a batch receive nonzero gradients. This property makes the gradient naturally sparse, and some optimizers exploit it with sparse updates to save computation in models with millions of IDs.

The table below summarizes the parameters that define an embedding layer and what each one controls.

ParameterTypical rangeWhat it affects
Vocabulary size30,000 to 250,000 tokens in language modelsMemory footprint and coverage of inputs
Embedding dimension64 to 512 in recommenders, 768 to over 12,000 in large language modelsCapacity and compute of later layers
Padding indexOne reserved row, often index 0Rows ignored by gradients for padded positions
InitializationNormal distribution with small standard deviationTraining stability in early steps
Max normOptional cap on vector lengthPrevents rare rows from growing too large
Main parameters of an embedding layer

What the learned vectors represent

The learned vectors represent inputs in a continuous space where geometry encodes usefulness for the task. Tokens or items that the model must treat alike move closer, typically measured by cosine similarity, while unrelated ones drift apart.

This is a form of representation learning: no one specifies what each dimension means, and individual dimensions are rarely interpretable. Directions in the space can still capture regularities, the well-known example being word2vec analogies, although such linear structure is approximate and inconsistent in practice.

An embedding layer must also be distinguished from the broader word embedding. The layer is the trainable lookup inside a model; an embedding is any vector produced by a model. The vectors that retrieval systems store in a vector database usually come from the final layers of an encoder, not from its input embedding layer, and the following comparison makes the distinction explicit.

RepresentationHow it is producedContext sensitivity
One-hot encodingFixed vector with a single 1None, and all inputs are equidistant
Embedding layer outputRow lookup in a learned matrixNone, one vector per index
Static word vectorsPretrained separately, such as word2vec or GloVeNone, fixed after training
Contextual embeddingsHidden states of a TransformerFull, the vector depends on the sentence
Sentence embeddingsPooled encoder outputRepresents a whole text span
Embedding layer compared with related representations

The embedding layer in Transformer models

In a Transformer, the embedding layer converts token IDs from the tokenizer into the vectors that enter the first attention block. Because self-attention is order-agnostic, position information is added separately, through learned or sinusoidal positional encoding added to the token vectors, or through rotary position embedding applied inside attention.

Many language models use weight tying, which shares the input embedding matrix with the output projection that produces vocabulary logits. Tying saves parameters equal to V times d and often improves perplexity in smaller models, while several large models keep the matrices separate for flexibility.

The share of parameters held by embeddings depends strongly on model size, as the following illustrative figures show.

Model scaleVocabulary and dimensionApproximate embedding share
Small model, about 125 million parameters50,000 by 768About 30 percent
Mid-size model, about 7 billion parameters32,000 by 4,096About 2 percent per matrix
Multilingual model with large vocabulary250,000 by 1,024Often above 40 percent
Illustrative embedding share of total parameters

Embedding layers for categorical and recommender data

Outside language, embedding layers encode high-cardinality categorical features such as user IDs, product IDs, store codes or ZIP codes. Recommender architectures like deep learning recommendation models hold most of their parameters in these tables, which can reach hundreds of gigabytes and must be sharded across devices.

Engineers manage that scale with several techniques: the hashing trick, which maps many raw IDs into a fixed number of buckets at the cost of collisions; mixed-dimension embeddings that give rare items smaller vectors; and quantization of embedding tables to 8-bit or lower precision for serving.

Common problems and design choices

Most failures of embedding layers come from vocabulary design and rare inputs, not from the lookup itself. The table below links typical symptoms to their causes and remedies.

SymptomLikely causeTypical fix
Poor quality on rare tokensFew gradient updates for those rowsSubword tokenization or frequency-aware regularization
Index out of range errorsUnseen IDs at inference timeReserved unknown token or hashing
Memory exhaustionVery large vocabulary times dimensionSharding, hashing or smaller dimension
Unstable early trainingEmbedding norms too largeSmaller initialization or scaling by square root of d
Padding affects resultsPadding rows receive gradientsSet the padding index and attention masks
Embedding layer problems, causes and fixes

A practical rule is to choose the embedding dimension relative to vocabulary size and data volume: too small a dimension underfits, while too large a dimension wastes memory and overfits rare entries. Embedding vectors also inherit biases present in the training data, which matters when they feed decisions about people.

The idea in brief

A neural network cannot read words or product codes directly; it only works with numbers. An embedding layer is a large lookup table that gives every word or item its own list of numbers.

At first the numbers are random. During training the network keeps adjusting them, so items used in similar ways end up with similar lists. It is like a librarian who starts with books shelved at random and gradually moves related books next to each other. The result is a map where closeness means similarity, and the rest of the network uses that map to make predictions.

A practical scenario

An online grocery retailer wants to recommend products based on what customers place in their baskets. Its catalog has about 80,000 products, and the data team trains a two-tower recommendation model with one embedding layer for products and another for customers, each using 128 dimensions.

The product table holds roughly 10 million parameters and trains on two years of purchase histories. After training, nearest neighbors in the product embedding space reveal sensible product groupings: oat milk sits near other plant-based drinks, and pasta sits near tomato sauces, although no category labels were provided.

New products have no history, so the team adds a reserved row for unknown items plus a content-based fallback until each new product collects about 50 purchases. The outcome is more relevant suggestions in the checkout carousel, measured by a higher click-through rate in an A/B test against the previous rule-based system.