Long Short-Term Memory (LSTM)

Definition

Long Short-Term Memory (LSTM) is a type of recurrent neural network architecture that uses a gated memory cell to selectively retain, forget, and output information over long sequences, mitigating the vanishing gradient problem of vanilla RNNs.

Overview

LSTM was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to solve a fundamental limitation of standard recurrent neural networks: their inability to learn dependencies spanning many time steps due to vanishing gradients during training. LSTM's key innovation is a dedicated memory cell state that runs through the sequence with only minor, carefully regulated linear modifications at each step, allowing gradients to flow across long time spans largely unimpeded, unlike the repeated nonlinear transformations applied to the hidden state in a vanilla RNN. Three gating mechanisms control this memory cell at every time step: the forget gate decides what information to discard from the cell state, the input gate decides what new information to add, and the output gate decides what part of the cell state to expose as the hidden state output. Each gate is itself a small learned neural network layer (typically a sigmoid function) that outputs values between 0 and 1, acting as a soft, differentiable switch — this design lets the network learn, from data, when to remember something (like the subject of a sentence) across many intervening words and when to forget irrelevant information. LSTMs became the dominant architecture for sequence modeling tasks throughout the 2010s, powering major advances in machine translation, speech recognition, and text generation before the Transformer architecture's self-attention mechanism largely superseded RNN-based approaches for large-scale language tasks, offering better parallelization and often stronger long-range modeling. LSTMs remain in active use today, however, particularly for time-series forecasting, and in settings with limited data, streaming requirements, or where the sequential inductive bias of an RNN is actually beneficial rather than a limitation, as well as as a component within larger hybrid systems.

Key Concepts

• Gated memory cell state preserves information across long sequences
• Forget gate controls what information is discarded from the cell state
• Input gate controls what new information is added to the cell state
• Output gate controls what part of the cell state becomes the output hidden state
• Gates are differentiable, learned sigmoid layers acting as soft switches
• Mitigates the vanishing gradient problem that limits vanilla RNNs
• Introduced by Hochreiter and Schmidhuber in 1997
• Widely used for sequence modeling before Transformer architectures became dominant

Use Cases

Frequently Asked Questions

Related in AI Fundamentals