What Are Gated Recurrent Units?

Gated recurrent units are a type of recurrent neural network cell that uses learned multiplicative gates to control how much of a hidden state is carried forward and how much is overwritten at each time step. They were introduced by Kyunghyun Cho and colleagues in 2014 as part of an encoder-decoder architecture for statistical machine translation, and they were designed to address the vanishing gradient problem that makes plain recurrent networks unable to learn dependencies spanning more than a few dozen steps. A gated recurrent unit, commonly abbreviated GRU, keeps the essential idea of the long short-term memory cell while using fewer parameters and a simpler internal structure.

The unit belongs to the broader family of recurrent neural networks, which process sequences by maintaining a state vector that is updated as each element arrives. What separates gated architectures from earlier recurrent designs is that the update is additive rather than purely multiplicative, so gradients can flow backward through many time steps along a nearly linear path.

The Gating Mechanism

A gated recurrent unit contains two gates. The update gate decides the balance between retaining the previous hidden state and adopting a newly computed candidate state, so a value near zero freezes the memory across a step while a value near one replaces it. The reset gate determines how much of the previous state contributes to that candidate, and driving it toward zero lets the unit discard accumulated context and begin a fresh representation, which is useful at phrase or sentence boundaries. Both gates are computed by a logistic sigmoid applied to a linear function of the current input and the previous state, while the candidate state uses a hyperbolic tangent. The final output is a convex combination of the old state and the candidate, weighted by the update gate. That additive form is what preserves gradient magnitude during backpropagation through time.

Relationship to Long Short-Term Memory

A long short-term memory cell separates a protected cell state from an exposed hidden state and uses three gates: input, forget, and output. A gated recurrent unit merges the input and forget gates into a single update gate, drops the separate cell state, and exposes its full memory at every step, leaving two gates and roughly three quarters of the parameters. An empirical evaluation of gated recurrent neural networks on sequence modeling compared the two on polyphonic music and speech signal tasks and found them broadly comparable, with each winning on some datasets, and both clearly better than a conventional recurrent unit. A later systematic study of long short-term memory variants reached a similar conclusion, identifying the forget gate and the output activation as the components that matter most. In practice the smaller parameter count of the gated recurrent unit gives faster training and less overfitting on modest datasets, while long short-term memory sometimes holds an edge on very long sequences.

Variants and Training Practice

Common extensions stack several layers to build hierarchical representations, run two units in opposite directions to form a bidirectional network with access to both past and future context, and replace the dense connections with convolutions to produce a convolutional gated recurrent unit for spatiotemporal data. Minimal gated units reduce the design to one gate. Training relies on truncated backpropagation through time, gradient clipping to control occasional exploding gradients, and dropout applied to non-recurrent connections. Although transformer architectures have displaced recurrent models for most large-scale language work, gated recurrent units remain widely used where sequences are streamed, memory is constrained, or training data is limited.

Applications

Gated recurrent units have applications in a range of fields, including:

• Speech recognition and keyword spotting on embedded devices
• Machine translation and other sequence-to-sequence language tasks
• Time series forecasting for electric load, traffic, and financial data
• Physiological signal analysis, including electrocardiogram and electroencephalogram classification
• Anomaly detection in industrial sensor and network telemetry streams
• Video analysis and gesture recognition using convolutional recurrent models