Self-Attention

Self-attention is the operation at the heart of the Transformer: every position in a sequence computes a weighted combination of all positions (including itself) based on learned similarities. Concretely, each input token is projected into three vectors — query, key, and value. The attention weight between positions i and j is computed from the dot product of query-i and key-j, softmax-normalised across all positions. Position i's output is the weighted sum of all value vectors under those weights. The mechanism is called *self*-attention because the queries, keys, and values all come from the same sequence, in contrast to cross-attention where queries come from one sequence and keys/values from another.

The innovation that made this practical at scale was **scaled dot-product attention** with the 1/√d_k scaling, combined with **multi-head attention** — running multiple attention operations in parallel on different learned projections and concatenating the results. Multi-head attention lets the network attend to different kinds of relationships simultaneously — syntactic in one head, semantic in another, positional in a third. The arrangement is so effective that it has displaced every earlier sequence-modelling mechanism across NLP, vision (ViT), audio (Whisper), and multimodal (CLIP, Flamingo).

For APAC mid-market teams working above the modelling layer, self-attention itself is not a design choice you touch. What matters is understanding its implications: **self-attention is O(n²) in sequence length** in memory and compute, which is why context windows have a hard cost gradient. Innovations like FlashAttention (memory-efficient exact attention), sliding-window attention (Longformer, Mistral), and linear-attention variants (Linformer, Performer) exist to push this limit. State-space models (Mamba family) are the current strongest competitor, with linear-time scaling and competitive quality on long-sequence tasks.

The non-obvious interpretability note: **attention weights are suggestive but not explanations**. It's tempting to treat high attention from position i to position j as evidence the model "used" j to produce i — and for intuition-building that is fine. But modern interpretability research (circuit analysis, activation patching) shows attention patterns are unreliable as full explanations of behaviour; the residual stream, feed-forward layers, and attention interact in ways a single layer's attention pattern cannot capture.