TF-IDF
TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic used to measure how important a word is to a specific document within a larger collection of documents, weighting terms that appear frequently in one document but…
Definition
TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic used to measure how important a word is to a specific document within a larger collection of documents, weighting terms that appear frequently in one document but rarely across the whole corpus more highly.
Overview
TF-IDF combines two component statistics to score how relevant a word is to a given document. Term frequency (TF) measures how often a word appears within a single document, typically normalized by the total number of words in that document, reflecting the intuition that a word mentioned many times in a document is likely important to its content. Inverse document frequency (IDF) measures how rare or common a word is across the entire document collection, computed as the logarithm of the total number of documents divided by the number of documents containing that word, giving higher weight to words that appear in relatively few documents. Multiplying TF and IDF together produces the final TF-IDF score for each word in each document: a word gets a high score if it appears frequently in a specific document but rarely elsewhere in the corpus, which typically identifies distinctive, topically relevant terms. Common words like 'the' or 'and' appear in nearly every document, so their IDF is close to zero, driving their TF-IDF score down regardless of how often they appear in any single document — effectively achieving a similar goal to manual stopword removal, but in a continuous, data-driven way. Applying TF-IDF across a corpus produces a document-term matrix, where each row represents a document and each column represents a TF-IDF-weighted term, giving a sparse numerical representation of each document that traditional machine learning algorithms (like logistic regression or support vector machines) can use directly for tasks like classification, clustering, or similarity search. TF-IDF predates modern deep learning-based NLP and does not capture word order, semantics, or context — two documents can have very different TF-IDF vectors despite meaning nearly the same thing if they use different words, and it treats each word as an independent, atomic unit (a 'bag of words' assumption). Despite these limitations, TF-IDF remains widely used today for its simplicity, speed, and strong baseline performance, particularly in search engines, information retrieval, and traditional text classification pipelines where full semantic embeddings are unnecessary or too costly.
Key Concepts
- Combines term frequency (TF) and inverse document frequency (IDF) into one score
- Weights words highly if frequent in one document but rare across the corpus
- Automatically downweights common words like stopwords without manual lists
- Produces a sparse document-term matrix usable by traditional ML algorithms
- Does not capture word order, context, or semantic meaning
- Simple, fast, and interpretable compared to neural embedding approaches
- Foundational technique in classical information retrieval and search engines
- Serves as a strong, low-cost baseline for text classification tasks