BM25

BM25 (Best Match 25) is a bag-of-words ranking function that scores each document against a query by combining term frequency, inverse document frequency, and document-length normalisation. The formula — published by Robertson and colleagues in the 1990s as part of the Okapi project — looks intimidating but the intuition is clean: frequent query terms in a document raise the score, but diminishing returns kick in after the first few occurrences (TF saturation); rare terms across the whole corpus count more than common ones (IDF); and long documents are penalised so they don't dominate by sheer surface area. Two tunable parameters (k1 for TF saturation, b for length normalisation) control the shape of these effects, with defaults (k1≈1.2, b≈0.75) that work well enough for most English corpora.

BM25 remains the default keyword-ranking function in 2026 across Elasticsearch, OpenSearch, Lucene, Meilisearch, Typesense, Postgres full-text search, and SQLite FTS5. It survived the vector-search wave because it excels at the task vector search is worst at: exact-match retrieval, rare-term queries, code / identifier / SKU / citation lookups, and anything where the query contains a specific token the document also contains. Modern variants matter too — BM25F (field-weighted) for structured documents, BM25+ and BM25L (addressing specific length-normalisation biases), and learned sparse retrievers (SPLADE, uniCOIL) that inherit the BM25 framing but replace its heuristics with learned weights.

For APAC mid-market teams building retrieval systems, **always baseline against BM25 before declaring a vector retriever "wins"**. On typical enterprise corpora — product documentation, legal clauses, policy language, code, IDs — BM25 is a ferocious baseline that vector search alone often loses to on recall@10. The right production default is hybrid (BM25 + vector with score fusion), but the engineering discipline is: measure each retriever's strengths on your own query log, understand where each wins, then combine. Teams that skipped BM25 and went vector-only usually end up rebuilding BM25 back in six months.

The non-obvious failure mode is **assuming BM25 is obsolete**. The vector-search marketing cycle of 2022-24 led many teams to retire BM25 entirely — and then rediscover its strengths the first time a user searched for an exact error code, product SKU, or contract clause and got semantically similar but literally wrong results. BM25 catches what vector retrieval misses and vice versa. Neither subsumes the other. Treat them as complementary primitives in the hybrid retriever, not as generations where the newer replaces the older.