LSTM, short for long short-term memory, is a recurrent neural network architecture that adds a gated memory cell to each unit so the network can keep, update or discard information over long sequences. Among neural networks for sequential data, it was designed specifically to overcome the vanishing gradient problem that prevents a plain recurrent neural network from learning dependencies spanning more than a few dozen steps.
Introduced in 1997 by Hochreiter and Schmidhuber and refined with the forget gate around 2000, the LSTM dominated speech recognition, handwriting recognition and machine translation for roughly two decades. The transformer has since replaced it in most large-scale language work, but the LSTM remains a compact, efficient choice for streaming and time-series tasks.
How the LSTM memory cell works
An LSTM unit maintains two states at every time step: the cell state, which acts as long-term memory, and the hidden state, which is the output passed to the next layer and the next step. Three gates, each a small layer with a sigmoid activation function producing values between 0 and 1, control how information flows into and out of the cell state.
The table below describes each element of the cell and its job.
| Component | Computation | Purpose |
|---|---|---|
| Forget gate | Sigmoid of input and previous hidden state | Decides how much old memory to keep |
| Input gate | Sigmoid of input and previous hidden state | Decides how much new information to write |
| Candidate values | Tanh of input and previous hidden state | Proposes the new content to store |
| Cell state update | Old state times forget gate plus input gate times candidate | Carries memory forward additively |
| Output gate | Sigmoid of input and previous hidden state | Decides how much memory to expose as output |
The key design choice is the additive cell state update. Because the old state is multiplied by a gate value that can stay close to 1 and is then summed with new content, gradients can travel back through many steps without shrinking exponentially. This path is often called the constant error carousel.
Parameters and model size
The size of an LSTM layer is set mainly by its hidden size, the number of units in the cell and hidden states. Each of the four internal transformations has its own weights for the input and for the previous hidden state, so a layer has roughly four times the parameters of a plain recurrent layer of the same width.
The following table lists the settings that practitioners tune most often.
| Hyperparameter | Typical range | Effect |
|---|---|---|
| Hidden size | 64 to 1,024 units | Memory capacity and computation per step |
| Number of layers | 1 to 4 stacked layers | Depth of representation; more layers risk overfitting |
| Dropout between layers | 0.1 to 0.5 | Regularization against overfitting |
| Sequence length for training | Tens to a few hundred steps | How far back gradients are propagated |
| Gradient clipping norm | 1 to 5 | Prevents exploding gradients |
Practitioners often initialize the forget gate bias to 1 or higher, so the network starts by remembering rather than forgetting. Training uses backpropagation through time, usually truncated to a fixed window for long streams.
Variants of the LSTM
Several variants change the cell or the way layers are arranged. They solve specific weaknesses but share the same gated-memory principle.
| Variant | Change | When it is used |
|---|---|---|
| Bidirectional LSTM | Two layers read the sequence forward and backward | Tagging and classification with full context |
| Peephole LSTM | Gates also see the cell state | Precise timing tasks |
| Stacked LSTM | Several layers on top of each other | Richer hierarchical features |
| ConvLSTM | Matrix products replaced by convolutions | Spatiotemporal data such as radar or video |
| GRU | Two gates and no separate cell state | Smaller, faster alternative with similar accuracy |
The gated recurrent unit is the closest relative. It merges the forget and input gates into an update gate and has about 25 percent fewer parameters. Neither design wins consistently, so teams usually try both on a validation set.
LSTM compared with transformers and other networks
The LSTM processes a sequence one step at a time, which makes training hard to parallelize across positions but keeps inference memory constant. The transformer uses an attention mechanism over all positions at once, which parallelizes well on GPUs and captures long-range relations directly, at a cost that grows quadratically with sequence length.
| Architecture | Handling of long context | Practical profile |
|---|---|---|
| Plain recurrent neural network | Weak beyond short ranges | Simple, rarely used alone today |
| LSTM | Good up to hundreds of steps | Constant memory per step, sequential training |
| Transformer | Strong within the context window | Parallel training, cost grows with context |
| Temporal convolutional network | Fixed by receptive field | Parallel and stable for time series |
For on-device and real-time workloads, such as keyword spotting or sensor anomaly detection, a small LSTM can still be the most efficient option, because it updates a fixed-size state for each new input instead of revisiting the whole history.
Limitations and common mistakes
The LSTM eases the vanishing gradient problem but does not remove every difficulty of sequence learning. Its memory is finite, and its sequential nature limits training speed on long inputs.
- Ignoring padding: unmasked padded steps distort the final hidden state in variable-length batches.
- Skipping gradient clipping: exploding gradients can still occur and destabilize training.
- Leaking future data: bidirectional layers must not be used for real-time forecasting.
- Oversized models on small data: large hidden sizes overfit quickly without dropout.
- Expecting unlimited memory: very long dependencies still fade without attention.
Put simply
An LSTM reads data in order, like a person reading a story word by word. It keeps a small notebook of what matters. At each step it decides what to erase, what to write down and what to mention right now.
Because it can keep important notes for a long time, it remembers that a character was introduced on page one when it reaches page ten. Older networks of this kind tended to forget such details quickly.
In practice
A utility company forecasts electricity demand for each substation 24 hours ahead using hourly readings of load, temperature and calendar features. Engineers train a two-layer LSTM with 128 hidden units on three years of history, feeding windows of the previous 168 hours, one full week, so the model sees both daily and weekly cycles.
The model runs every hour on modest server hardware, updating its state with each new reading. Compared with the earlier seasonal regression model, it captures holiday effects and heat-wave peaks more accurately, which lets operators schedule reserve generation with smaller safety margins. The team keeps a held-out final year for evaluation and retrains the model each quarter as consumption patterns change.