Recurrent neural networks

What Are Recurrent Neural Networks?

Recurrent neural networks (RNNs) are a class of artificial neural networks designed to process sequential data by maintaining an internal hidden state that carries information from earlier steps in a sequence to later ones. Unlike feedforward networks, which map a fixed-size input to an output in a single pass, an RNN applies the same learned weight matrices at each time step and feeds the resulting hidden state back into itself, allowing the network to model temporal dependencies of variable length. This recurrent connection makes RNNs a natural choice for tasks where context accumulated over time determines the correct output.

RNNs draw from the mathematics of dynamical systems, statistical sequence modeling, and gradient-based optimization. The field traces its conceptual lineage to work by Jordan and Elman in the late 1980s, though practical deep RNNs became tractable only after algorithmic advances in the 1990s and the availability of GPU-accelerated training in the 2010s. Within the broader field of deep learning, RNNs occupy the class of models suited to one-dimensional ordered inputs such as text, audio, sensor streams, and time series.

Sequence Modeling

At each time step t, an RNN receives an input vector and combines it with the hidden state from step t-1 to produce a new hidden state. This hidden state acts as a compressed summary of the sequence seen so far. The network can be unrolled over time to visualize this process as a deep feedforward network with shared weights across layers. Sequence-to-sequence tasks, in which the model reads an entire input sequence and then generates an output sequence, require the hidden state to encode the full input before decoding begins. This approach underlies early neural machine translation systems and remains relevant for tasks where input and output lengths differ.

Long Short-Term Memory and Gated Architectures

A practical limitation of basic RNNs is the vanishing gradient problem: when backpropagated through many time steps, gradients tend to shrink exponentially, making it difficult to learn dependencies spanning more than a few dozen steps. The Long Short-Term Memory (LSTM) architecture, introduced by Hochreiter and Schmidhuber in 1997, addresses this with gating mechanisms: an input gate, a forget gate, and an output gate regulate what information enters, persists in, and exits a dedicated memory cell. This cell state can carry information across hundreds of time steps with relatively little gradient decay, as detailed in a survey of RNNs and LSTMs on arXiv. The 1997 paper by Hochreiter and Schmidhuber on LSTM networks remains a foundational reference for understanding gated memory mechanisms.

The Gated Recurrent Unit (GRU), proposed in 2014, simplifies the LSTM by merging the cell state and hidden state and using two gates rather than three. Empirical comparisons show that GRUs often match LSTM performance on shorter sequences while being faster to train. Bidirectional variants of both architectures process the input in both forward and backward directions, giving the model access to future context when producing outputs, which is advantageous in tasks such as named entity recognition.

Training Dynamics

Training RNNs uses backpropagation through time (BPTT), which unrolls the network across the sequence length and computes gradients with respect to every time step. Truncated BPTT limits unrolling to a fixed window to reduce memory cost, at the expense of cutting off very long-range gradients. Gradient clipping, a technique that rescales gradients when their norm exceeds a threshold, mitigates the exploding gradient problem that can occur in deep or long-sequence models. The critical review of RNNs for sequence learning at arXiv provides a thorough treatment of these training challenges and the architectural solutions developed to address them. Since the mid-2010s, attention mechanisms and transformer architectures have absorbed many tasks that once relied on RNNs, though RNNs remain competitive for low-latency on-device inference where transformer memory costs are prohibitive.

Applications

Recurrent neural networks have applications in a wide range of disciplines, including:

Related Topics

Deep Learning