1. Home
  2. Blog
  3. Artificial Intelligence

Artificial Intelligence

Jaccard Similarity Explained: Formula, Python Code and MinHash

Jaccard similarity measures how much two sets overlap: the size of their intersection divided by the size of their union. Learn the formula, compute it in Python for tags, words and character shingles, see how it differs from cosine similarity, and test how well MinHash estimates it.

9 min read
Jaccard similarity: formula, Python and MinHash

Key takeaways

  • Jaccard similarity = |A ∩ B| / |A ∪ B|. It ranges from 0 (nothing shared) to 1 (identical sets).
  • Jaccard distance is simply 1 minus the similarity.
  • It ignores counts and order, so what you put in the sets (tags, words or character shingles) decides what “similar” means.
  • MinHash estimates Jaccard without comparing full sets. Our 128-hash test estimated 0.516 for a true value of 0.5.

Are these two products alike? Do these two documents say nearly the same thing? Which users share the most interests? When your data can be expressed as sets, Jaccard similarity is the most intuitive answer: count what the two sets share and divide by everything they contain between them. It is used in search deduplication, recommendation systems, plagiarism detection and machine learning evaluation.

All numbers below come from Python 3.13.6 runs, with scikit-learn 1.9.1 for the library comparison.

What is Jaccard similarity?

The Jaccard index, named after botanist Paul Jaccard, measures the overlap between two finite sets. Two sets with exactly the same members score 1. Two sets with nothing in common score 0. Everything else falls in between, in proportion to how much they share. Because it works on sets, it only cares whether an item is present, not how many times it appears.

The formula

J(A, B) = |A ∩ B| / |A ∪ B|
        = |A ∩ B| / (|A| + |B| - |A ∩ B|)

Jaccard distance = 1 - J(A, B)

The intersection counts shared items, the union counts distinct items across both sets. The second form is handy when you already know the set sizes. Jaccard distance is a proper metric, which means it obeys the triangle inequality, so it works with clustering algorithms that need a true distance.

A worked example

Two developers list their skills:

A = {"python", "sql", "git", "docker"}
B = {"python", "git", "java"}
  • Intersection: {python, git}, size 2.
  • Union: {python, sql, git, docker, java}, size 5.
  • Jaccard similarity: 2 / 5 = 0.4. Distance: 0.6.

Our code produced exactly those numbers. Note that the extra skill on one side (docker) lowers the score even though it does not conflict with anything. Jaccard rewards sets that are both overlapping and similar in size.

Jaccard similarity in Python

Python’s built-in set operators make it a three-line function. Our Python set guide explains & and |.

def jaccard(a, b):
    a, b = set(a), set(b)
    if not a and not b:
        return 1.0
    return len(a & b) / len(a | b)

A = {"python", "sql", "git", "docker"}
B = {"python", "git", "java"}
print(A & B, len(A & B), len(A | B), round(jaccard(A, B), 3), round(1 - jaccard(A, B), 3))
# {'python', 'git'} 2 5 0.4 0.6

Converting the inputs with set() lets the function accept lists, strings and other iterables. Set lookups are fast, so this scales to sets with many thousands of items.

The empty set edge case

If both sets are empty, the union is empty and the formula divides by zero. There is no universally agreed answer, so you must choose. Our function returns 1.0, treating two empty sets as identical, and our run confirmed jaccard([], []) gave 1.0 while jaccard({"a"}, []) gave 0.0. scikit-learn makes a different choice and lets you set a zero_division value. Document your choice and add a test for it, because empty inputs are common in real data, such as a product with no tags.

Comparing sentences with word sets

s1 = "the cat sat on the mat"
s2 = "the cat sat on a mat"
print(round(jaccard(s1.split(), s2.split()), 3))
# 0.833

The first sentence has five distinct words and the second has six; they share five, so the score is 5 / 6 = 0.833. Notice that the repeated “the” counted only once, because sets ignore duplicates. In real projects, lowercase the text, strip punctuation and consider removing very common words first, the same preprocessing used for TF-IDF.

Character shingles for spelling variants

Word sets fail on spelling differences: “colour” and “color” share no words. Splitting text into overlapping character sequences, called shingles or n-grams, fixes that:

def shingles(s, k=3):
    return {s[i:i + k] for i in range(len(s) - k + 1)}

print(sorted(shingles("color")))
# ['col', 'lor', 'olo']
print(round(jaccard(shingles("colour"), shingles("color")), 3))
# 0.4

They share “col” and “olo” out of five distinct shingles, giving 0.4. Plain character sets are much coarser: jaccard("night", "nacht") treats each word as a set of letters and returned 0.429 (3 shared letters out of 7). Shingle size controls sensitivity. Small values such as 2 or 3 suit short strings like names; longer values such as 5 to 9 characters, or word shingles, suit near-duplicate detection in documents.

Jaccard vs cosine similarity

Jaccard Cosine
Input Sets (presence only) Vectors (counts or weights)
Uses frequency? No Yes
Range 0 to 1 −1 to 1 (0 to 1 for non-negative data)
Best for Tags, IDs, shingles, deduplication TF-IDF vectors, embeddings, semantic search

Use Jaccard when the question is “how many items do these share?”. Use cosine similarity when the strength of each feature matters, or when comparing embeddings that capture meaning. Jaccard will never know that “car” and “automobile” are related.

MinHash: Jaccard at scale

Comparing every pair of a million documents means about 500 billion set comparisons, which is far too slow. MinHash solves this with a neat property: if you apply a random hash function to every item in two sets and take the minimum hash value of each set, the probability that the two minimums are equal is exactly the Jaccard similarity.

Repeat that with k different hash functions and you get a short signature per set. The fraction of signature positions that match estimates Jaccard. Signatures are small and fixed in size, and combined with locality-sensitive hashing (LSH) they let you find similar pairs without comparing everything. Search engines and large language model training pipelines use this for near-duplicate removal.

import hashlib

def minhash(s, k=128):
    return [min(int(hashlib.md5(f"{i}:{x}".encode()).hexdigest()[:8], 16) for x in s)
            for i in range(k)]

X = {f"w{i}" for i in range(0, 600)}
Y = {f"w{i}" for i in range(200, 800)}
mx, my = minhash(X), minhash(Y)
estimate = sum(p == q for p, q in zip(mx, my)) / len(mx)

Seeding MD5 with the position number i simulates k different hash functions. It is slow but easy to read; production libraries such as datasketch use much faster hashing.

How accurate is MinHash?

Sets X and Y each hold 600 items and share 400, so the true Jaccard similarity is 400 / 800 = 0.5. Our estimates by signature size:

Hash functions (k) Estimate Error
16 0.375 0.125
64 0.5 0.0
128 0.516 0.016
256 0.492 0.008

With 16 hashes the estimate was far off. From 64 upward it settled near the true value. The 64-hash result landing exactly on 0.5 was partly luck: the expected error shrinks roughly with 1 / √k, so 64 hashes typically gives errors around 0.06 and 256 around 0.03. Choose k by how much error your use case tolerates.

Jaccard in scikit-learn

import numpy as np
from sklearn.metrics import jaccard_score

y_true = np.array([1, 1, 0, 1, 0])
y_pred = np.array([1, 0, 0, 1, 1])
print(jaccard_score(y_true, y_pred))
# 0.5

In machine learning, jaccard_score compares binary labels: positions where either array is 1 form the union (4 here), positions where both are 1 form the intersection (2), giving 0.5. The same idea, called Intersection over Union (IoU), scores object detection boxes and image segmentation masks.

Common use cases

  • Near-duplicate detection for web pages, articles and support tickets, using shingles plus MinHash.
  • Recommendations: users with similar sets of liked items probably share tastes.
  • Tag and category matching between products or blog posts.
  • Test result comparison: compare the set of failing tests between two builds to see how much changed. Our flaky tests guide covers why sets of failures shift.
  • Retrieval evaluation in RAG systems: compare retrieved document IDs against the expected set.
  • Model evaluation with IoU for detection and segmentation.

Limitations

  • No frequency: an item appearing once or a hundred times counts the same. Weighted Jaccard variants exist when counts matter.
  • No meaning: synonyms and paraphrases score zero overlap.
  • Size imbalance: a small set fully contained in a large one still scores low. If you need “how much of A is in B”, use the overlap coefficient, |A ∩ B| / min(|A|, |B|).
  • Preprocessing matters: case, punctuation and shingle size change results a lot.

Tip: before tuning thresholds, look at a sample of pairs at several scores, such as 0.3, 0.5 and 0.8. What “similar” means depends on your data, and eyeballing real pairs is the fastest way to pick a cut-off.

Final thoughts

Jaccard similarity is one of the simplest similarity measures and one of the most useful: shared items divided by total distinct items. Decide what goes into your sets, handle empty sets explicitly, choose shingles for text, and switch to MinHash when the collection is too large for pairwise comparison. When frequency or meaning matters, reach for cosine similarity or embeddings instead.

FAQ

What is Jaccard similarity?

It measures how similar two sets are by dividing the number of shared items by the number of distinct items in both sets. The result ranges from 0 to 1.

How do you calculate Jaccard similarity in Python?

Convert both inputs to sets, then divide the length of their intersection by the length of their union, and handle the case where both sets are empty.

What is the difference between Jaccard and cosine similarity?

Jaccard compares sets and ignores how often items appear. Cosine compares vectors and takes counts or weights into account.

What is MinHash used for?

MinHash creates small signatures whose matching fraction estimates Jaccard similarity, so very large collections can be searched for near duplicates quickly.

What is Jaccard distance?

Jaccard distance is one minus the Jaccard similarity. It is a true distance metric, so it works with clustering algorithms.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Jaccard Similarity: Formula, Python, MinHash

Jaccard similarity measures overlap between two sets as intersection over union. Work the formula, compute it in Python for words…

Read article →
Software Development

Unix Timestamp Explained: Seconds vs ms

A Unix timestamp counts seconds since 1 January 1970 UTC. Learn seconds vs milliseconds, converting in Python, JavaScript and SQL,…

Read article →
DevOps

Git Hooks: Pre-commit and Commit-msg Examples

Git hooks run scripts at key moments such as commit and push. See tested pre-commit, commit-msg and prepare-commit-msg hooks, sharing…

Read article →
Test Automation

API Testing in Python: Pytest + Requests Suite

Build an API testing suite in Python with pytest and requests: fixtures, status codes, headers, JSON schema validation, CRUD flows,…

Read article →
Software Testing

Test Design Techniques: 9 Methods + Examples

Test design techniques help you write fewer tests that catch more bugs. Learn 9 black-box, white-box and experience-based methods with…

Read article →
Databases & SQL

SQL Temporary Table: Create, Use and Drop It

SQL temporary tables store interim results for one session. Learn syntax for SQL Server, PostgreSQL, MySQL, Oracle and SQLite, session…

Read article →
Databases & SQL

SQL CROSS JOIN: Every Combination + Examples

SQL CROSS JOIN returns every combination of rows from two tables. See the row math, product variants, calendar grids that…

Read article →
Software Development

JavaScript Date: Format, Parse, Timezone Bugs

JavaScript Date explained with tested Node.js code: zero-based months, the date-only UTC parsing trap, toISOString, getTimezoneOffset, Intl formatting, date math…

Read article →
Software Development

Python Requests: GET, POST, Timeouts, Retries

Python requests explained with code tested against a local API: GET with params, POST json vs data, headers and auth,…

Read article →
Software Development

Python Deepcopy vs Copy: Fix Nested Lists

Python deepcopy vs shallow copy explained with tested code: aliasing, nested lists, the [[0]*3]*3 trap, memo and cycles, custom __deepcopy__,…

Read article →
Artificial Intelligence

TF-IDF Explained: Formula and Worked Example

TF-IDF scores a word by how often it appears in a document and how rare it is across all documents….

Read article →
Software Development

Base64 Encoding Explained, With Python Code

Base64 encoding turns bytes into 64 safe text characters. See the bit-level steps, padding, URL-safe base64, the 33% size cost,…

Read article →
DevOps

Git Tag: Create, Push and Delete Release Tags

Git tag explained with tested commands: annotated vs lightweight tags, listing and sorting, pushing with follow-tags, deleting local and remote…

Read article →
Test Automation

Pytest Coverage: pytest-cov and Branch Gaps

Measure pytest coverage with pytest-cov: term-missing reports, branch coverage, HTML reports, pyproject settings and a fail-under gate for CI, with…

Read article →
Software Testing

Error Guessing in Testing: Checklist + Demo

Error guessing uses tester experience to predict where code breaks. Get a ready checklist of risky inputs, a step by…

Read article →
Databases & SQL

SQL LAG and LEAD: Compare Rows, 5 Examples

SQL LAG and LEAD read values from the previous or next row without a self join. Five tested examples: change,…

Read article →
Databases & SQL

SQL STRING_AGG: Combine Rows Into One String

SQL STRING_AGG joins values from many rows into one string per group. See syntax for PostgreSQL, SQL Server, MySQL GROUP_CONCAT,…

Read article →
Software Development

JavaScript Classes: Private Fields and Extends

JavaScript classes explained with tested Node.js code: constructors, methods, #private fields, getters, static members, extends and super, plus the errors…

Read article →
Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →