TF-IDF Explained: The Formula, a Worked Example and Python
TF-IDF scores how important a word is to one document by combining how often it appears there with how rare it is across all documents. Work through the formula by hand on three sentences, then compare scikit-learn’s output and learn why its numbers differ.
- TF-IDF = term frequency × inverse document frequency. A word scores high when it is frequent in one document but rare in the collection.
- Words in every document, such as “the”, get an IDF of zero in the textbook formula, so they score zero.
- scikit-learn uses a smoothed IDF and normalises each row. In our test it ranked “the” as the top term of document 0, unlike the textbook version.
- TF-IDF is fast and explainable, but it only matches exact words. Embeddings capture meaning; BM25 is the stronger keyword ranker.
Before neural networks, search engines needed a way to decide which words in a page actually describe it. The word “the” appears everywhere, so it says nothing. A word like “mat” that appears in only one document says a lot about that document. TF-IDF turns that idea into a number, and it still powers keyword search, document tagging and text classification today.
In this guide we compute TF-IDF by hand on three tiny sentences, confirm the numbers with Python, then run scikit-learn 1.9.1 on the same data and explain every difference. All output comes from runs on Python 3.13.6.
What is TF-IDF?
TF-IDF stands for term frequency, inverse document frequency. It assigns a weight to each word (term) in each document of a collection (corpus). The weight is the product of two parts:
- Term frequency (TF): how often the word appears in this document.
- Inverse document frequency (IDF): how rare the word is across all documents.
Each document becomes a vector of these weights, one number per vocabulary word. Those vectors can then be compared, ranked or fed to a classifier.
The intuition
Think of a library of cooking articles. “Recipe” appears in nearly every article, so it is useless for telling them apart. “Saffron” appears in three. An article that mentions saffron five times is very likely about saffron. TF rewards the five mentions; IDF rewards saffron’s rarity. Multiplying them gives saffron a high score in that article and gives recipe a low score everywhere.
The formula
tf(t, d) = count of t in d / number of words in d
idf(t) = ln(N / df(t))
tfidf(t, d) = tf(t, d) * idf(t)
N = number of documents
df(t) = number of documents that contain t
This is the basic textbook version with the natural logarithm. Many variants exist: raw counts instead of proportions, log-scaled TF, different log bases, and smoothing to avoid division by zero. The ranking idea is the same in all of them, but the exact numbers differ, which matters when you compare results.
Our three documents
doc 0: "the cat sat on the mat"
doc 1: "the dog sat on the log"
doc 2: "the cat chased the dog"
So N = 3. Document frequencies from our run:
df {'cat': 2, 'chased': 1, 'dog': 2, 'log': 1, 'mat': 1, 'on': 2, 'sat': 2, 'the': 3}
Step 1: compute IDF
| Term | df | idf = ln(3 / df) |
|---|---|---|
| the | 3 | ln(1) = 0.000 |
| cat, dog, on, sat | 2 | ln(1.5) = 0.405 |
| chased, log, mat | 1 | ln(3) = 1.099 |
“The” appears in all three documents, so its IDF is exactly zero. Words in only one document get the highest IDF.
Step 2: TF-IDF scores for document 0
Document 0 has six words. “the” appears twice, every other word once.
| Term | tf | idf | tf × idf |
|---|---|---|---|
| mat | 1/6 = 0.167 | 1.099 | 0.183 |
| cat | 0.167 | 0.405 | 0.068 |
| sat | 0.167 | 0.405 | 0.068 |
| on | 0.167 | 0.405 | 0.068 |
| the | 2/6 = 0.333 | 0.000 | 0.000 |
“Mat” wins, which matches intuition: it is what makes document 0 unique. “The” is the most frequent word in the document yet scores nothing. For document 2, our run gave “chased” the top score of 0.220, because that document is shorter (5 words), so each word’s TF is larger.
From-scratch Python
import math
from collections import Counter
docs = ["the cat sat on the mat", "the dog sat on the log", "the cat chased the dog"]
toks = [d.split() for d in docs]
N = len(docs)
vocab = sorted({w for t in toks for w in t})
df = {w: sum(w in t for t in toks) for w in vocab}
idf = {w: math.log(N / df[w]) for w in vocab}
for i, t in enumerate(toks):
c, n = Counter(t), len(t)
scores = {w: round(c[w] / n * idf[w], 3) for w in c}
print(i, sorted(scores.items(), key=lambda kv: -kv[1]))
0 [('mat', 0.183), ('cat', 0.068), ('sat', 0.068), ('on', 0.068), ('the', 0.0)]
1 [('log', 0.183), ('dog', 0.068), ('sat', 0.068), ('on', 0.068), ('the', 0.0)]
2 [('chased', 0.22), ('cat', 0.081), ('dog', 0.081), ('the', 0.0)]
Those match the hand calculation. Real projects also lowercase, strip punctuation and often stem words, so that “Cats” and “cat” count as one term. Our Python string methods guide covers the cleaning steps.
scikit-learn TfidfVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
X = v.fit_transform(docs)
Our run gave these IDF values: cat 1.288, chased 1.693, dog 1.288, log 1.693, mat 1.693, on 1.288, sat 1.288, the 1.000. And document 0’s scores:
doc0 [('the', 0.581), ('mat', 0.492), ('cat', 0.374), ('on', 0.374), ('sat', 0.374)]
“The” is now the highest scoring word in document 0. That surprises many people, so let us see why.
Why scikit-learn’s numbers differ
- Smoothed IDF plus one. By default scikit-learn uses
idf = ln((1 + N) / (1 + df)) + 1. For “the” that is ln(4/4) + 1 = 1, not 0. The “+ 1” is deliberate, so words in every document are not ignored completely. - Raw counts for TF. It uses counts, not proportions. “The” appears twice, so its TF is 2.
- L2 normalisation. Each document vector is scaled to length 1, which makes long and short documents comparable.
Count 2 × IDF 1 beats count 1 × IDF 1.693, so “the” wins before normalisation and stays ahead after it. With TfidfVectorizer(smooth_idf=False, norm=None) our run gave cat 1.405 and chased 2.099, which is the textbook ln(N/df) plus one. In practice, fix the stop word problem directly. TfidfVectorizer(stop_words="english") reduced our vocabulary to cat, chased, dog, log, mat and sat, dropping “the” and “on”.
Tip: when you check TF-IDF numbers from a paper or tutorial against a library, compare the IDF formula, the TF scheme and the normalisation first. Most “wrong” results are just different variants.
Using TF-IDF for search
To search, transform the query with the same fitted vectorizer and compare it with every document using cosine similarity:
from sklearn.metrics.pairwise import cosine_similarity
q = v.transform(["cat on a mat"])
print(cosine_similarity(q, X).round(3))
# [[0.723 0.194 0.209]]
Document 0 scores 0.723, far ahead of the others, because it shares “cat”, “on” and the rare “mat”. The word “a” was ignored: the default tokenizer drops one-letter tokens. Other useful options are ngram_range=(1, 2) to include word pairs, which grew our vocabulary to 18 terms, and sublinear_tf=True to dampen repeated words.
Limits of TF-IDF
- No meaning. “Car” and “automobile” are unrelated to TF-IDF. Embeddings solve that.
- No word order. “Dog bites man” and “man bites dog” look identical without n-grams.
- Saturation. TF keeps growing with repetition, which spam can exploit. BM25 caps that and handles document length better, so it is the usual choice for keyword ranking.
- Large sparse vectors with one column per word in the vocabulary.
Even so, TF-IDF remains a strong, cheap baseline for classification and a great way to explain why a document matched. Many retrieval systems combine keyword scores with vector search, as covered in our RAG guide.
TF-IDF for text classification
TF-IDF vectors also make good features for classifiers. We trained a tiny support ticket router on six labelled examples, three about billing and three about bugs:
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
X = ["refund not received for my order", "payment failed twice",
"card was charged double", "app crashes on login",
"login button does nothing", "screen freezes after update"]
y = ["billing"] * 3 + ["bug"] * 3
model = make_pipeline(TfidfVectorizer(), LogisticRegression()).fit(X, y)
print(model.predict(["charged but no refund", "crashes after login update"]))
# ['billing' 'bug']
Both new tickets went to the right queue, using a vocabulary of just 24 words. Six examples is far too few for real use, but the pattern scales: with a few thousand labelled tickets, a TF-IDF plus logistic regression pipeline trains in seconds, runs on a laptop and is easy to explain, because you can list the words that pushed each prediction. It is a strong baseline to beat before you reach for a large language model.
Final thoughts
TF-IDF rests on one sensible idea: words that are common in a document but rare in the collection describe that document best. The formula is simple enough to compute by hand, and libraries make it a one-liner. Just remember that libraries use smoothed, normalised variants, so check the settings before comparing numbers, and remove stop words if they creep to the top.
FAQ
What does TF-IDF stand for?
Term frequency, inverse document frequency. It weights a word by how often it appears in a document and how rare it is across all documents.
What is the TF-IDF formula?
The basic form is term frequency times the log of total documents divided by the number of documents containing the term. Libraries often add smoothing and normalisation.
Why does scikit-learn give different TF-IDF values?
It uses raw counts, a smoothed IDF with one added, and L2 normalisation by default, so its numbers differ from the textbook formula.
Is TF-IDF still used today?
Yes. It is a fast, explainable baseline for keyword search, tagging and text classification, often combined with BM25 or embeddings.
What is the difference between TF-IDF and BM25?
BM25 builds on the same idea but limits the effect of repeated words and adjusts for document length, so it usually ranks search results better.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.