Recurrent Neural Network
A Recurrent Neural Network (RNN) is a neural network architecture designed for sequential data, where connections loop back so that the network's hidden state at each time step depends on both the current input and the previous hidden…
Definition
A Recurrent Neural Network (RNN) is a neural network architecture designed for sequential data, where connections loop back so that the network's hidden state at each time step depends on both the current input and the previous hidden state.
Overview
Unlike feedforward networks, which process each input independently, an RNN maintains an internal hidden state that is updated at every time step as a function of the current input and the hidden state carried over from the previous step. This recurrence lets the network, in principle, retain information about earlier elements of a sequence when processing later ones, making RNNs a natural fit for tasks involving ordered data such as text, speech, and time series, where order and context matter. In practice, vanilla RNNs suffer from the vanishing and exploding gradient problem: when backpropagating error through many time steps (backpropagation through time), gradients tend to shrink toward zero or grow uncontrollably, making it very difficult for the network to learn dependencies spanning more than a handful of time steps. This limitation motivated gated architectures — Long Short-Term Memory (LSTM) networks and the simpler Gated Recurrent Unit (GRU) — which use learned gating mechanisms to control what information is retained, forgotten, or output at each step, substantially improving the ability to learn long-range dependencies. RNNs and their gated variants were the dominant architecture for sequence modeling in natural language processing and speech recognition through the mid-2010s, powering early neural machine translation and speech-to-text systems. Their inherently sequential computation, however — each step must wait for the previous one to finish — makes them slow to train on modern parallel hardware compared to architectures that process a whole sequence at once. The introduction of the Transformer architecture in 2017, with its self-attention mechanism that captures long-range dependencies without sequential recurrence and parallelizes easily across GPUs, largely displaced RNNs as the default choice for large-scale language modeling, though RNN-style architectures remain relevant in some real-time, streaming, or resource-constrained settings, and newer state-space and recurrent-inspired architectures continue to be explored as efficient alternatives to full attention.
Key Concepts
- Maintains a hidden state updated recurrently across sequential time steps
- Well suited to variable-length sequential data: text, speech, time series
- Trained via backpropagation through time (BPTT)
- Suffers from vanishing/exploding gradients over long sequences in vanilla form
- Gated variants (LSTM, GRU) mitigate long-range dependency learning problems
- Inherently sequential computation limits parallelization during training
- Largely superseded by Transformer/self-attention architectures for large-scale NLP
- Still used in streaming, real-time, or resource-constrained sequence tasks