Long Short-Term Memory (LSTM)

Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) are a recurrent-network architecture designed to handle the vanishing-gradient problem that crippled plain RNNs on long sequences. The key mechanism is a **cell state** that runs along the sequence like a conveyor belt, with three learned gates — **forget**, **input**, and **output** — deciding what information enters the cell, persists in the cell, and exits to the hidden state. The gating allows gradient signal to flow backwards across hundreds of time steps without decaying to zero.

From roughly 2014 to 2018, LSTMs were the default for every sequence task that mattered — machine translation (seq2seq with attention), speech recognition, music generation, handwriting synthesis, time-series forecasting. They gave way to **Transformers** (Vaswani et al., 2017), which parallelise across the sequence rather than iterate step-by-step, and which scaled dramatically better as compute budgets grew. By 2020, LSTMs had been relegated from every leaderboard they once dominated.

The architecture still matters in 2026 in three niches. First, **streaming and low-latency** applications where the per-step computation of an RNN is genuinely cheaper than the full-sequence attention of a Transformer — some real-time speech stacks, some on-device wake-word detectors. Second, **very long sequences** where Transformer quadratic-attention cost is prohibitive — though state-space models (Mamba, Hyena) are now the stronger successors for this case. Third, **embedded and legacy systems** where the model has to fit in tight memory and LSTM's small parameter count is an asset.

For APAC mid-market teams starting a new sequence project in 2026, the default is a Transformer (usually a pretrained one) unless a specific latency, memory, or streaming constraint rules it out. LSTMs remain useful background knowledge because many production systems still run them in quiet corners of the stack.