Gated recurrent unit

Also known as: GRU

On this page 7

A gated recurrent unit (GRU) is a recurrent neural network cell that uses two learned gates, the update gate and the reset gate, to decide how much of its previous hidden state to keep and how much new information to write at each time step. Introduced in 2014 by Kyunghyun Cho and colleagues for machine translation, the GRU was designed as a simpler alternative to the LSTM that still addresses the vanishing gradient problem.

Neural networks built from GRU cells process sequences one element at a time, carrying a single hidden vector forward as memory. Within artificial intelligence, GRUs remain a practical choice for streaming data, small models and time series, even though transformers now dominate large-scale language modeling.

How the gated recurrent unit updates its memory

At every step, a GRU combines the current input x_t with the previous hidden state h_(t-1) and computes two gates with sigmoid activations, so each gate value lies between 0 and 1. These gates act as soft switches that scale information element by element rather than turning it fully on or off.

  1. The reset gate r_t decides how much of the previous state is used when forming a new candidate memory.
  2. The candidate state is computed with a tanh activation from the input and the reset-scaled previous state.
  3. The update gate z_t blends the old state and the candidate: h_t = (1 - z_t) * h_(t-1) + z_t * candidate.
  4. The resulting hidden state is both the output of the step and the memory passed to the next step.

The interpolation in the final step is the key to the design. When the update gate stays close to zero, the cell copies its previous state almost unchanged, which creates a nearly additive path for gradients through time and allows dependencies across dozens or hundreds of steps to be learned.

ComponentActivationRole in the cell
Update gateSigmoidBalances keeping old memory against writing new content
Reset gateSigmoidControls how much past state shapes the candidate
Candidate stateTanhProposes new memory content for the step
Hidden stateNone (linear blend)Serves as memory and as the step output
Components of a GRU cell

GRU compared with LSTM and simple recurrent networks

The GRU sits between the simple recurrent neural network and the LSTM in complexity. A simple recurrent network has no gates and struggles with long dependencies, while an LSTM uses three gates and a separate cell state. The GRU merges the cell and hidden states and uses two gates instead of three.

PropertySimple RNNGRULSTM
GatesNoneUpdate and resetInput, forget and output
State vectorsHidden stateHidden state onlyHidden state and cell state
Weight matrices per layer1 set3 sets4 sets
Relative parametersBaselineAbout 3 times baselineAbout 4 times baseline
Long-range memoryWeakGoodGood, sometimes better on very long spans
Recurrent cell types side by side

Because a GRU has roughly 25 percent fewer parameters than an LSTM of the same hidden size, it trains somewhat faster and needs less memory. Empirical comparisons generally find no consistent winner in accuracy: results depend on the dataset, sequence length and tuning effort, so practitioners often test both.

Key hyperparameters and design variants

Configuring a GRU layer involves a small set of hyperparameters that determine its capacity, cost and stability. The table summarizes typical values for production models of moderate size.

SettingTypical rangeEffect
Hidden size64 to 1,024 unitsMemory capacity and compute per step
Number of layers1 to 4 stackedDepth of temporal features
DirectionUnidirectional or bidirectionalAccess to future context; bidirectional needs the full sequence
Dropout between layers0.1 to 0.5Regularization against overfitting
Gradient clipping norm1 to 5Prevents exploding gradients
Typical GRU settings and their effects

Several variants exist. The minimal gated unit uses a single forget-style gate, and some implementations apply the reset gate after the matrix multiplication rather than before it, a detail that changes numerical results slightly and matters when porting weights between frameworks. GRUs are also used as the recurrent part of encoder-decoder architectures and convolutional recurrent models for video.

Where GRUs fit in modern practice

GRUs are most useful when inputs arrive one step at a time and the model must respond immediately with a fixed amount of memory. Unlike attention, whose memory grows with context length, a GRU keeps a constant-size state, so its inference cost per step does not increase as the sequence gets longer.

SituationSuggested choiceReason
Real-time sensor stream on a microcontrollerSmall GRUConstant memory and low latency
Speech enhancement on a phoneGRU or LSTMFrame-by-frame processing with modest compute
Large language model pretrainingTransformerParallel training scales far better
Very long sequences with parallel trainingState space modelRecurrent inference with parallelizable training
Small tabular time seriesGRU or gradient-boosted treesBoth work; compare on validation data
Choosing a GRU or an alternative

The main reason transformers replaced GRUs in large models is training parallelism. A GRU must process step 10 before step 11, which limits how well training uses GPUs, while attention processes all positions of a training sequence at once.

Limitations and common training problems

Gating reduces but does not eliminate gradient problems. Very long dependencies, such as relationships across thousands of steps, remain difficult, and exploding gradients can still occur without clipping. GRU training also tends to be sensitive to initialization and learning rate.

  • Sequential computation slows training on long sequences compared with attention-based models.
  • Fixed-size memory forces the cell to compress history, so fine details from early steps may be lost.
  • Padding mistakes in batched variable-length sequences silently corrupt the final hidden state if masks are omitted.
  • Bidirectional layers cannot be used for live streaming because they need future inputs.

A practical safeguard is to inspect gate activations during debugging. Update gates saturated near 0 or 1 for every unit often indicate a learning rate that is too high or inputs that are poorly normalized.

The simple explanation

A gated recurrent unit reads information in order, like a person reading a sentence word by word while keeping a short mental note of what has happened so far. After each word, it decides how much of the old note to keep and how much to rewrite.

The two gates work like two dials. One dial decides whether the past is relevant to the new word, and the other decides how strongly the note should change. This simple mechanism lets the unit remember important details for a long time without being overwhelmed by everything it has seen.

How it is used in practice

An engineering team builds a noise suppression feature for a video calling app that must run on ordinary laptops. Audio arrives in 10-millisecond frames, and each frame must be cleaned before the next one arrives, so the model cannot wait for future audio.

The team trains a network with two GRU layers of 256 units that predicts, for each frame, how much to attenuate each frequency band. Because the GRU carries a constant-size state, processing each frame takes a fraction of a millisecond on one CPU core. In listening tests, background fan noise and keyboard clicks are strongly reduced while speech stays clear, and the feature ships without requiring a GPU.