A recurrent neural network (RNN) is a class of neural networks that processes sequential data one step at a time while carrying a hidden state from each step to the next. The same weights are applied at every position, and the hidden state acts as a compact memory of everything seen so far, which lets the network model text, speech, sensor readings and other ordered data of variable length.
Among neural networks in artificial intelligence, RNNs were the dominant architecture for sequence modeling until the Transformer replaced them in most large-scale language tasks. They remain relevant for streaming, low-latency and memory-constrained applications, and their ideas live on in gated variants such as LSTM and in modern recurrent and state space models.
How a recurrent neural network processes a sequence
At each time step t, an RNN combines the current input with the previous hidden state to produce a new hidden state. In the simplest form, the Elman network, the update is h(t) = tanh(W·x(t) + U·h(t−1) + b), where W and U are weight matrices shared across all steps. An output layer can read the hidden state at every step or only at the end.
This weight sharing across time is the defining property. It keeps the number of parameters independent of sequence length and gives the network a built-in assumption that the same rules apply at every position. Conceptually, the network is a loop; for training it is unrolled into a deep feedforward graph with one copy of the cell per time step.
The table shows the main input-output configurations an RNN supports, which explains why one architecture served so many tasks.
| Configuration | Input and output | Typical task |
|---|---|---|
| Many-to-one | Sequence in, single value out | Sentiment classification |
| One-to-many | Single value in, sequence out | Image captioning |
| Many-to-many, aligned | One output per input step | Part-of-speech tagging |
| Many-to-many, unaligned | Input sequence then output sequence | Machine translation with an encoder-decoder |
Training with backpropagation through time
RNNs are trained with backpropagation through time (BPTT), which applies ordinary backpropagation to the unrolled network and sums gradients for the shared weights over all steps. For long sequences, practitioners use truncated BPTT, propagating gradients only over a window of perhaps 50 to a few hundred steps to limit memory and computation.
The central difficulty is the vanishing and exploding gradient problem. Because the gradient is multiplied by the recurrent Jacobian at every step, it shrinks or grows roughly exponentially with distance. Vanishing gradients prevent a simple RNN from learning dependencies more than about 10 to 20 steps apart, while exploding gradients destabilize training.
| Problem | Symptom | Standard remedy |
|---|---|---|
| Exploding gradients | Loss spikes or becomes NaN | Gradient norm clipping, often at 1 to 5 |
| Vanishing gradients | Long-range patterns never learned | Gated cells such as LSTM or GRU |
| Slow sequential training | Low accelerator utilization | Larger batches, shorter truncation windows |
| Overfitting on small data | Validation loss rises early | Dropout between layers, weight decay |
| Poor initialization | Unstable early training | Orthogonal recurrent weight initialization |
Gated variants and related architectures
Gated architectures solved the vanishing gradient problem by adding learned gates that control what the cell keeps, forgets and outputs. The LSTM introduced a separate cell state with an almost additive update path, and the gated recurrent unit (GRU) simplified the design to two gates with comparable accuracy on many tasks.
| Architecture | Memory mechanism | Strength | Weakness |
|---|---|---|---|
| Simple RNN | Single hidden state | Tiny and fast per step | Forgets beyond short ranges |
| LSTM | Cell state with input, forget and output gates | Robust long-range memory | About four times the parameters of a simple cell |
| GRU | Update and reset gates | Fewer parameters than LSTM | Slightly less expressive on some tasks |
| Bidirectional RNN | Forward and backward passes combined | Uses future context | Unsuitable for real-time streaming |
| Transformer | Attention over all positions | Parallel training, long context | Memory grows with sequence length |
The comparison with the Transformer is the key practical trade-off. An RNN has constant memory and constant cost per new token at inference, but training cannot be parallelized across time steps. The attention mechanism allows full parallel training and direct access to distant positions, at the price of cost that grows quadratically with context length. Recent linear recurrent and state space models try to combine both advantages.
When a recurrent network is still the right choice
Recurrent networks remain a sound choice when inputs arrive continuously and must be processed immediately on limited hardware. Their fixed-size state makes memory use predictable regardless of how long the stream runs.
| Situation | Recommended choice | Reason |
|---|---|---|
| Keyword spotting on a microcontroller | Small GRU or LSTM | Kilobytes of memory, constant per-step cost |
| Real-time sensor anomaly detection | Unidirectional LSTM | Processes each reading as it arrives |
| Short univariate time series | GRU or classical statistical model | Few parameters, little data needed |
| Large-scale language modeling | Transformer | Parallel training and long context |
| Very long sequences with tight memory | Linear recurrent or state space model | Recurrent inference with parallel training |
Limitations and common mistakes
Most RNN problems in practice stem from sequence handling details rather than the architecture itself. The following mistakes appear frequently.
- Forgetting to reset the hidden state between independent sequences, which leaks information from one sample into the next.
- Padding without masking, so the network learns from padding tokens and final states become meaningless.
- Using a bidirectional model for live streams, although it needs the whole sequence before producing output.
- Training without gradient clipping, which allows a single exploding batch to ruin a long training run.
- Expecting a simple RNN to capture long-range context that only gated cells or attention can learn reliably.
A non-technical view
Think of reading a sentence word by word while keeping a short mental note of what it is about so far. Each new word updates the note, and the note helps you understand the next word. A recurrent neural network works the same way: it reads one piece at a time and carries a running summary forward.
The weakness is that the note is small. Simple versions forget the beginning of long passages, which is why improved designs with better memory control were developed.
A concrete example
A manufacturer of wireless earbuds needs wake-word detection that runs continuously on a chip with only a few hundred kilobytes of memory and must not drain the battery. Its engineers convert the microphone signal into 40 audio features every 10 milliseconds and feed them into a two-layer GRU with 64 hidden units.
Because the network keeps a fixed-size hidden state, it processes each audio frame as it arrives and never stores the full recording. After quantizing the weights to 8-bit integers, the model fits comfortably in memory and responds within about 100 milliseconds of the spoken wake word. A Transformer of similar accuracy would need far more memory for its growing context, so the recurrent design is the deciding factor for always-on operation.