1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

TF-IDF Explained: The Formula, a Worked Example and Python

TF-IDF scores how important a word is to one document by combining how often it appears there with how rare it is across all documents. Work through the formula by hand on three sentences, then compare scikit-learn’s output and learn why its numbers differ.

9 min read
TF-IDF explained: formula and worked example

Key takeaways

  • TF-IDF = term frequency × inverse document frequency. A word scores high when it is frequent in one document but rare in the collection.
  • Words in every document, such as “the”, get an IDF of zero in the textbook formula, so they score zero.
  • scikit-learn uses a smoothed IDF and normalises each row. In our test it ranked “the” as the top term of document 0, unlike the textbook version.
  • TF-IDF is fast and explainable, but it only matches exact words. Embeddings capture meaning; BM25 is the stronger keyword ranker.

Before neural networks, search engines needed a way to decide which words in a page actually describe it. The word “the” appears everywhere, so it says nothing. A word like “mat” that appears in only one document says a lot about that document. TF-IDF turns that idea into a number, and it still powers keyword search, document tagging and text classification today.

In this guide we compute TF-IDF by hand on three tiny sentences, confirm the numbers with Python, then run scikit-learn 1.9.1 on the same data and explain every difference. All output comes from runs on Python 3.13.6.

What is TF-IDF?

TF-IDF stands for term frequency, inverse document frequency. It assigns a weight to each word (term) in each document of a collection (corpus). The weight is the product of two parts:

  • Term frequency (TF): how often the word appears in this document.
  • Inverse document frequency (IDF): how rare the word is across all documents.

Each document becomes a vector of these weights, one number per vocabulary word. Those vectors can then be compared, ranked or fed to a classifier.

The intuition

Think of a library of cooking articles. “Recipe” appears in nearly every article, so it is useless for telling them apart. “Saffron” appears in three. An article that mentions saffron five times is very likely about saffron. TF rewards the five mentions; IDF rewards saffron’s rarity. Multiplying them gives saffron a high score in that article and gives recipe a low score everywhere.

The formula

tf(t, d)     = count of t in d / number of words in d
idf(t)       = ln(N / df(t))
tfidf(t, d)  = tf(t, d) * idf(t)

N     = number of documents
df(t) = number of documents that contain t

This is the basic textbook version with the natural logarithm. Many variants exist: raw counts instead of proportions, log-scaled TF, different log bases, and smoothing to avoid division by zero. The ranking idea is the same in all of them, but the exact numbers differ, which matters when you compare results.

Our three documents

doc 0: "the cat sat on the mat"
doc 1: "the dog sat on the log"
doc 2: "the cat chased the dog"

So N = 3. Document frequencies from our run:

df {'cat': 2, 'chased': 1, 'dog': 2, 'log': 1, 'mat': 1, 'on': 2, 'sat': 2, 'the': 3}

Step 1: compute IDF

Term df idf = ln(3 / df)
the 3 ln(1) = 0.000
cat, dog, on, sat 2 ln(1.5) = 0.405
chased, log, mat 1 ln(3) = 1.099

“The” appears in all three documents, so its IDF is exactly zero. Words in only one document get the highest IDF.

Step 2: TF-IDF scores for document 0

Document 0 has six words. “the” appears twice, every other word once.

Term tf idf tf × idf
mat 1/6 = 0.167 1.099 0.183
cat 0.167 0.405 0.068
sat 0.167 0.405 0.068
on 0.167 0.405 0.068
the 2/6 = 0.333 0.000 0.000

“Mat” wins, which matches intuition: it is what makes document 0 unique. “The” is the most frequent word in the document yet scores nothing. For document 2, our run gave “chased” the top score of 0.220, because that document is shorter (5 words), so each word’s TF is larger.

From-scratch Python

import math
from collections import Counter

docs = ["the cat sat on the mat", "the dog sat on the log", "the cat chased the dog"]
toks = [d.split() for d in docs]
N = len(docs)
vocab = sorted({w for t in toks for w in t})
df = {w: sum(w in t for t in toks) for w in vocab}
idf = {w: math.log(N / df[w]) for w in vocab}

for i, t in enumerate(toks):
    c, n = Counter(t), len(t)
    scores = {w: round(c[w] / n * idf[w], 3) for w in c}
    print(i, sorted(scores.items(), key=lambda kv: -kv[1]))
0 [('mat', 0.183), ('cat', 0.068), ('sat', 0.068), ('on', 0.068), ('the', 0.0)]
1 [('log', 0.183), ('dog', 0.068), ('sat', 0.068), ('on', 0.068), ('the', 0.0)]
2 [('chased', 0.22), ('cat', 0.081), ('dog', 0.081), ('the', 0.0)]

Those match the hand calculation. Real projects also lowercase, strip punctuation and often stem words, so that “Cats” and “cat” count as one term. Our Python string methods guide covers the cleaning steps.

scikit-learn TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
X = v.fit_transform(docs)

Our run gave these IDF values: cat 1.288, chased 1.693, dog 1.288, log 1.693, mat 1.693, on 1.288, sat 1.288, the 1.000. And document 0’s scores:

doc0 [('the', 0.581), ('mat', 0.492), ('cat', 0.374), ('on', 0.374), ('sat', 0.374)]

“The” is now the highest scoring word in document 0. That surprises many people, so let us see why.

Why scikit-learn’s numbers differ

  1. Smoothed IDF plus one. By default scikit-learn uses idf = ln((1 + N) / (1 + df)) + 1. For “the” that is ln(4/4) + 1 = 1, not 0. The “+ 1” is deliberate, so words in every document are not ignored completely.
  2. Raw counts for TF. It uses counts, not proportions. “The” appears twice, so its TF is 2.
  3. L2 normalisation. Each document vector is scaled to length 1, which makes long and short documents comparable.

Count 2 × IDF 1 beats count 1 × IDF 1.693, so “the” wins before normalisation and stays ahead after it. With TfidfVectorizer(smooth_idf=False, norm=None) our run gave cat 1.405 and chased 2.099, which is the textbook ln(N/df) plus one. In practice, fix the stop word problem directly. TfidfVectorizer(stop_words="english") reduced our vocabulary to cat, chased, dog, log, mat and sat, dropping “the” and “on”.

Tip: when you check TF-IDF numbers from a paper or tutorial against a library, compare the IDF formula, the TF scheme and the normalisation first. Most “wrong” results are just different variants.

To search, transform the query with the same fitted vectorizer and compare it with every document using cosine similarity:

from sklearn.metrics.pairwise import cosine_similarity
q = v.transform(["cat on a mat"])
print(cosine_similarity(q, X).round(3))
# [[0.723 0.194 0.209]]

Document 0 scores 0.723, far ahead of the others, because it shares “cat”, “on” and the rare “mat”. The word “a” was ignored: the default tokenizer drops one-letter tokens. Other useful options are ngram_range=(1, 2) to include word pairs, which grew our vocabulary to 18 terms, and sublinear_tf=True to dampen repeated words.

Limits of TF-IDF

  • No meaning. “Car” and “automobile” are unrelated to TF-IDF. Embeddings solve that.
  • No word order. “Dog bites man” and “man bites dog” look identical without n-grams.
  • Saturation. TF keeps growing with repetition, which spam can exploit. BM25 caps that and handles document length better, so it is the usual choice for keyword ranking.
  • Large sparse vectors with one column per word in the vocabulary.

Even so, TF-IDF remains a strong, cheap baseline for classification and a great way to explain why a document matched. Many retrieval systems combine keyword scores with vector search, as covered in our RAG guide.

TF-IDF for text classification

TF-IDF vectors also make good features for classifiers. We trained a tiny support ticket router on six labelled examples, three about billing and three about bugs:

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

X = ["refund not received for my order", "payment failed twice",
     "card was charged double", "app crashes on login",
     "login button does nothing", "screen freezes after update"]
y = ["billing"] * 3 + ["bug"] * 3

model = make_pipeline(TfidfVectorizer(), LogisticRegression()).fit(X, y)
print(model.predict(["charged but no refund", "crashes after login update"]))
# ['billing' 'bug']

Both new tickets went to the right queue, using a vocabulary of just 24 words. Six examples is far too few for real use, but the pattern scales: with a few thousand labelled tickets, a TF-IDF plus logistic regression pipeline trains in seconds, runs on a laptop and is easy to explain, because you can list the words that pushed each prediction. It is a strong baseline to beat before you reach for a large language model.

Final thoughts

TF-IDF rests on one sensible idea: words that are common in a document but rare in the collection describe that document best. The formula is simple enough to compute by hand, and libraries make it a one-liner. Just remember that libraries use smoothed, normalised variants, so check the settings before comparing numbers, and remove stop words if they creep to the top.

FAQ

What does TF-IDF stand for?

Term frequency, inverse document frequency. It weights a word by how often it appears in a document and how rare it is across all documents.

What is the TF-IDF formula?

The basic form is term frequency times the log of total documents divided by the number of documents containing the term. Libraries often add smoothing and normalisation.

Why does scikit-learn give different TF-IDF values?

It uses raw counts, a smoothed IDF with one added, and L2 normalisation by default, so its numbers differ from the textbook formula.

Is TF-IDF still used today?

Yes. It is a fast, explainable baseline for keyword search, tagging and text classification, often combined with BM25 or embeddings.

What is the difference between TF-IDF and BM25?

BM25 builds on the same idea but limits the effect of repeated words and adjusts for document length, so it usually ranks search results better.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Jaccard Similarity: Formula, Python, MinHash

Jaccard similarity measures overlap between two sets as intersection over union. Work the formula, compute it in Python for words…

Read article →
Software Development

Unix Timestamp Explained: Seconds vs ms

A Unix timestamp counts seconds since 1 January 1970 UTC. Learn seconds vs milliseconds, converting in Python, JavaScript and SQL,…

Read article →
DevOps

Git Hooks: Pre-commit and Commit-msg Examples

Git hooks run scripts at key moments such as commit and push. See tested pre-commit, commit-msg and prepare-commit-msg hooks, sharing…

Read article →
Test Automation

API Testing in Python: Pytest + Requests Suite

Build an API testing suite in Python with pytest and requests: fixtures, status codes, headers, JSON schema validation, CRUD flows,…

Read article →
Software Testing

Test Design Techniques: 9 Methods + Examples

Test design techniques help you write fewer tests that catch more bugs. Learn 9 black-box, white-box and experience-based methods with…

Read article →
Databases & SQL

SQL Temporary Table: Create, Use and Drop It

SQL temporary tables store interim results for one session. Learn syntax for SQL Server, PostgreSQL, MySQL, Oracle and SQLite, session…

Read article →
Databases & SQL

SQL CROSS JOIN: Every Combination + Examples

SQL CROSS JOIN returns every combination of rows from two tables. See the row math, product variants, calendar grids that…

Read article →
Software Development

JavaScript Date: Format, Parse, Timezone Bugs

JavaScript Date explained with tested Node.js code: zero-based months, the date-only UTC parsing trap, toISOString, getTimezoneOffset, Intl formatting, date math…

Read article →
Software Development

Python Requests: GET, POST, Timeouts, Retries

Python requests explained with code tested against a local API: GET with params, POST json vs data, headers and auth,…

Read article →
Software Development

Python Deepcopy vs Copy: Fix Nested Lists

Python deepcopy vs shallow copy explained with tested code: aliasing, nested lists, the [[0]*3]*3 trap, memo and cycles, custom __deepcopy__,…

Read article →
Artificial Intelligence

TF-IDF Explained: Formula and Worked Example

TF-IDF scores a word by how often it appears in a document and how rare it is across all documents….

Read article →
Software Development

Base64 Encoding Explained, With Python Code

Base64 encoding turns bytes into 64 safe text characters. See the bit-level steps, padding, URL-safe base64, the 33% size cost,…

Read article →
DevOps

Git Tag: Create, Push and Delete Release Tags

Git tag explained with tested commands: annotated vs lightweight tags, listing and sorting, pushing with follow-tags, deleting local and remote…

Read article →
Test Automation

Pytest Coverage: pytest-cov and Branch Gaps

Measure pytest coverage with pytest-cov: term-missing reports, branch coverage, HTML reports, pyproject settings and a fail-under gate for CI, with…

Read article →
Software Testing

Error Guessing in Testing: Checklist + Demo

Error guessing uses tester experience to predict where code breaks. Get a ready checklist of risky inputs, a step by…

Read article →
Databases & SQL

SQL LAG and LEAD: Compare Rows, 5 Examples

SQL LAG and LEAD read values from the previous or next row without a self join. Five tested examples: change,…

Read article →
Databases & SQL

SQL STRING_AGG: Combine Rows Into One String

SQL STRING_AGG joins values from many rows into one string per group. See syntax for PostgreSQL, SQL Server, MySQL GROUP_CONCAT,…

Read article →
Software Development

JavaScript Classes: Private Fields and Extends

JavaScript classes explained with tested Node.js code: constructors, methods, #private fields, getters, static members, extends and super, plus the errors…

Read article →
Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →