Softmax

Softmax converts a vector of real-valued scores (logits) into a probability distribution by exponentiating each element and normalising by the sum: `softmax(x_i) = exp(x_i) / Σ exp(x_j)`. The exponential guarantees every output is positive; the normalisation guarantees they sum to 1. It is the final layer of nearly every classification neural network, and it is the step inside every LLM that turns raw next-token scores into a probability over the vocabulary.

Three properties drive its ubiquity. First, **monotonicity** — larger logits always become larger probabilities, preserving the ranking the network learned. Second, **differentiability** — softmax combined with cross-entropy loss has a clean gradient (`prediction - label`), which is why the pairing is the default for classification training. Third, **temperature scaling** — dividing logits by a scalar `T` before softmax sharpens (T<1) or flattens (T>1) the distribution without changing the ranking, which is how LLM temperature-based sampling works under the hood.

Numerical stability matters in practice: naively computing `exp(x)` for large logits overflows. Every production implementation subtracts the max logit before exponentiation (`exp(x - max)`), which is mathematically identical but avoids overflow. Frameworks bake this in; if you are rolling softmax by hand in a training loop, do the same.

For APAC enterprise teams using foundation models, the relevant place softmax shows up is **sampling control** — temperature, top-k, top-p are all post-softmax manipulations of the output distribution. Understanding that these three knobs compose (temperature reshapes the distribution; top-k and top-p then trim the tail) is the difference between deterministic-feeling generation and one that drifts.