Full-Text Search Relevance and Scoring: TF-IDF & BM25

Analogy categoryAnalogies

Why Relevance Scoring Exists

When a match query runs against a text field, Elasticsearch doesn't just return matching documents — it ranks them by a computed _score representing estimated relevance, so a search for 'wireless noise cancelling headphones' returns the most on-topic products first rather than in arbitrary order. This scoring is built on the classical information-retrieval intuition that a term's importance to a document depends on two competing factors: how often it appears in that document (term frequency) and how rare it is across the whole collection (inverse document frequency). A word like 'wireless' that appears rarely across your entire product catalog but repeatedly within one product's description is a strong signal that document is specifically about wireless products, whereas a word like 'the' that appears in nearly every document carries almost no discriminating power.

🏏

Cricket analogy: Relevance scoring is like a talent scout ranking players not just by whether they can bat, but by how rare and impactful their specific skill is — a genuine left-arm wrist spinner stands out more than another right-arm medium bowler, the way a rare term stands out more than a common one.

TF-IDF: The Classical Foundation

TF-IDF combines term frequency (TF) — how many times a term appears in a given document — with inverse document frequency (IDF) — a measure that grows as the term becomes rarer across the whole index, typically computed as a logarithm of total documents divided by documents containing the term. Multiplying TF by IDF produces a per-term weight: high when a term is both frequent in this particular document and rare across the collection, low when a term is either absent from the document or so common (like a stop word) that it appears almost everywhere. Classical TF-IDF has a known weakness, though — term frequency contributes linearly and unboundedly, so a document that repeats 'wireless' fifty times through keyword stuffing scores proportionally higher than one that mentions it five times naturally, even though the marginal relevance signal of each additional repetition should diminish.

🏏

Cricket analogy: TF-IDF's term frequency component is like counting how many boundaries a batter hit in an innings, while its IDF component is like weighting a rare shot type — a reverse scoop off a fast bowler — more heavily than a routine forward defensive that every batter plays.

json

// Inspect the actual score computation for a query
GET /products/_search
{
  "query": { "match": { "description": "wireless noise cancelling" } },
  "explain": true
}
// The response's _explanation shows each term's BM25 contribution:
// boost, idf (rarity across the index), and tf (saturation-adjusted frequency)

BM25: Elasticsearch's Default Similarity

BM25 (Best Match 25) is the probabilistic ranking function Elasticsearch has used as its default similarity since version 5, and it fixes TF-IDF's unbounded-frequency problem with term frequency saturation: the k1 parameter (default 1.2) controls how quickly additional occurrences of a term stop adding meaningful score, so going from one to two occurrences helps a lot but going from twenty to twenty-one barely moves the needle. BM25 also applies field-length normalization via the b parameter (default 0.75), penalizing matches in unusually long documents on the theory that a term appearing once in a 50-word abstract is more significant than the same term appearing once in a 5,000-word article, since the long document simply had more opportunity to contain it by chance. Both parameters are configurable per index through a custom similarity setting, and tuning them is occasionally worthwhile — lowering b toward 0 for fields like product titles where length shouldn't be penalized, or lowering k1 for fields where any repeated occurrence is already a strong signal.

🏏

Cricket analogy: BM25's term frequency saturation is like a highlights algorithm valuing a bowler's first three wickets in a spell heavily but barely increasing the value for the eighth wicket in an already dominant spell — diminishing returns on repeated success.

You can inspect exactly how BM25 computed a document's score by adding "explain": true to a search request, or by using the dedicated GET /{index}/_explain/{id} endpoint — both return a breakdown of each term's boost, idf, and saturated tf contribution.

The _score value is relative and unbounded — it has no fixed scale (not 0–1, not 0–100) and is only meaningful for ranking documents against each other within the same query. Never compare _score values across different queries or use them as an absolute confidence measure.

Relevance scoring ranks matching documents by estimated topical importance, not just binary matching.
TF-IDF combines term frequency in a document with inverse document frequency across the whole index.
Classical TF-IDF scores term frequency unboundedly, making it vulnerable to keyword-stuffing inflation.
BM25 is Elasticsearch's default similarity since version 5, fixing TF-IDF's unbounded frequency problem.
BM25's k1 parameter controls term frequency saturation; b controls field-length normalization.
explain: true or the _explain endpoint reveals the exact per-term score breakdown for debugging.
_score is relative and unbounded — only meaningful for ranking within a single query, never across queries.