Recurrent Neural Network (RNN)

A Recurrent Neural Network (RNN) processes sequential data one timestep at a time, maintaining a hidden state that carries information from past inputs forward. At each step, the network takes the current input and the previous hidden state, computes a new hidden state, and (optionally) emits an output. The recurrence is what gives RNNs their name and their theoretical ability to handle arbitrarily long sequences — in principle they can remember any past input. In practice, vanilla RNNs suffer from **vanishing and exploding gradients** during backpropagation through time, which prevented them from learning dependencies across more than a handful of timesteps.

The architectural response was **gated recurrent networks** — LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) — which add learned gates that decide what information to keep, forget, and output at each step. LSTMs and GRUs dominated sequence modelling from roughly 2014 to 2018 across machine translation, speech recognition, music generation, and time-series forecasting. Seq2seq architectures combined an encoder RNN with a decoder RNN, with attention mechanisms layered on top to help the decoder selectively focus on parts of the encoded input. Then the 2017 Transformer paper showed that you could replace the recurrence entirely with self-attention and get better quality, better training parallelism, and better scaling — and the RNN era in mainstream NLP effectively ended.

RNNs still matter in 2026 in narrow niches. **Real-time streaming** applications benefit from RNNs' naturally incremental computation — some production speech stacks, some wake-word detectors. **Embedded / on-device** models where memory is tight can fit an RNN where a Transformer cannot. **Time-series forecasting** for business applications (demand, revenue, sensor data) still sees productive RNN use alongside gradient-boosted trees and modern attention-based forecasters. **State-space models** (Mamba, Hyena, RWKV) are arguably a modern reinvention of the RNN idea with better scaling properties, and are the most interesting successors to watch.

For APAC mid-market teams starting a new sequence-modelling project in 2026, the default is a Transformer (usually pretrained) unless a specific latency, memory, or streaming constraint rules it out. RNN knowledge remains valuable because many production systems still run them in quiet corners of the stack — and because understanding gating mechanisms informs how to debug the occasional LSTM that refuses to die.