Jaccard Similarity Explained: Formula, Python Code and MinHash
Jaccard similarity measures how much two sets overlap: the size of their intersection divided by the size of their union. Learn the formula, compute it in Python for tags, words and character shingles, see how it differs from cosine similarity, and test how well MinHash estimates it.
- Jaccard similarity = |A ∩ B| / |A ∪ B|. It ranges from 0 (nothing shared) to 1 (identical sets).
- Jaccard distance is simply 1 minus the similarity.
- It ignores counts and order, so what you put in the sets (tags, words or character shingles) decides what “similar” means.
- MinHash estimates Jaccard without comparing full sets. Our 128-hash test estimated 0.516 for a true value of 0.5.
Are these two products alike? Do these two documents say nearly the same thing? Which users share the most interests? When your data can be expressed as sets, Jaccard similarity is the most intuitive answer: count what the two sets share and divide by everything they contain between them. It is used in search deduplication, recommendation systems, plagiarism detection and machine learning evaluation.
All numbers below come from Python 3.13.6 runs, with scikit-learn 1.9.1 for the library comparison.
What is Jaccard similarity?
The Jaccard index, named after botanist Paul Jaccard, measures the overlap between two finite sets. Two sets with exactly the same members score 1. Two sets with nothing in common score 0. Everything else falls in between, in proportion to how much they share. Because it works on sets, it only cares whether an item is present, not how many times it appears.
The formula
J(A, B) = |A ∩ B| / |A ∪ B|
= |A ∩ B| / (|A| + |B| - |A ∩ B|)
Jaccard distance = 1 - J(A, B)
The intersection counts shared items, the union counts distinct items across both sets. The second form is handy when you already know the set sizes. Jaccard distance is a proper metric, which means it obeys the triangle inequality, so it works with clustering algorithms that need a true distance.
A worked example
Two developers list their skills:
A = {"python", "sql", "git", "docker"}
B = {"python", "git", "java"}
- Intersection: {python, git}, size 2.
- Union: {python, sql, git, docker, java}, size 5.
- Jaccard similarity: 2 / 5 = 0.4. Distance: 0.6.
Our code produced exactly those numbers. Note that the extra skill on one side (docker) lowers the score even though it does not conflict with anything. Jaccard rewards sets that are both overlapping and similar in size.
Jaccard similarity in Python
Python’s built-in set operators make it a three-line function. Our Python set guide explains & and |.
def jaccard(a, b):
a, b = set(a), set(b)
if not a and not b:
return 1.0
return len(a & b) / len(a | b)
A = {"python", "sql", "git", "docker"}
B = {"python", "git", "java"}
print(A & B, len(A & B), len(A | B), round(jaccard(A, B), 3), round(1 - jaccard(A, B), 3))
# {'python', 'git'} 2 5 0.4 0.6
Converting the inputs with set() lets the function accept lists, strings and other iterables. Set lookups are fast, so this scales to sets with many thousands of items.
The empty set edge case
If both sets are empty, the union is empty and the formula divides by zero. There is no universally agreed answer, so you must choose. Our function returns 1.0, treating two empty sets as identical, and our run confirmed jaccard([], []) gave 1.0 while jaccard({"a"}, []) gave 0.0. scikit-learn makes a different choice and lets you set a zero_division value. Document your choice and add a test for it, because empty inputs are common in real data, such as a product with no tags.
Comparing sentences with word sets
s1 = "the cat sat on the mat"
s2 = "the cat sat on a mat"
print(round(jaccard(s1.split(), s2.split()), 3))
# 0.833
The first sentence has five distinct words and the second has six; they share five, so the score is 5 / 6 = 0.833. Notice that the repeated “the” counted only once, because sets ignore duplicates. In real projects, lowercase the text, strip punctuation and consider removing very common words first, the same preprocessing used for TF-IDF.
Character shingles for spelling variants
Word sets fail on spelling differences: “colour” and “color” share no words. Splitting text into overlapping character sequences, called shingles or n-grams, fixes that:
def shingles(s, k=3):
return {s[i:i + k] for i in range(len(s) - k + 1)}
print(sorted(shingles("color")))
# ['col', 'lor', 'olo']
print(round(jaccard(shingles("colour"), shingles("color")), 3))
# 0.4
They share “col” and “olo” out of five distinct shingles, giving 0.4. Plain character sets are much coarser: jaccard("night", "nacht") treats each word as a set of letters and returned 0.429 (3 shared letters out of 7). Shingle size controls sensitivity. Small values such as 2 or 3 suit short strings like names; longer values such as 5 to 9 characters, or word shingles, suit near-duplicate detection in documents.
Jaccard vs cosine similarity
| Jaccard | Cosine | |
|---|---|---|
| Input | Sets (presence only) | Vectors (counts or weights) |
| Uses frequency? | No | Yes |
| Range | 0 to 1 | −1 to 1 (0 to 1 for non-negative data) |
| Best for | Tags, IDs, shingles, deduplication | TF-IDF vectors, embeddings, semantic search |
Use Jaccard when the question is “how many items do these share?”. Use cosine similarity when the strength of each feature matters, or when comparing embeddings that capture meaning. Jaccard will never know that “car” and “automobile” are related.
MinHash: Jaccard at scale
Comparing every pair of a million documents means about 500 billion set comparisons, which is far too slow. MinHash solves this with a neat property: if you apply a random hash function to every item in two sets and take the minimum hash value of each set, the probability that the two minimums are equal is exactly the Jaccard similarity.
Repeat that with k different hash functions and you get a short signature per set. The fraction of signature positions that match estimates Jaccard. Signatures are small and fixed in size, and combined with locality-sensitive hashing (LSH) they let you find similar pairs without comparing everything. Search engines and large language model training pipelines use this for near-duplicate removal.
import hashlib
def minhash(s, k=128):
return [min(int(hashlib.md5(f"{i}:{x}".encode()).hexdigest()[:8], 16) for x in s)
for i in range(k)]
X = {f"w{i}" for i in range(0, 600)}
Y = {f"w{i}" for i in range(200, 800)}
mx, my = minhash(X), minhash(Y)
estimate = sum(p == q for p, q in zip(mx, my)) / len(mx)
Seeding MD5 with the position number i simulates k different hash functions. It is slow but easy to read; production libraries such as datasketch use much faster hashing.
How accurate is MinHash?
Sets X and Y each hold 600 items and share 400, so the true Jaccard similarity is 400 / 800 = 0.5. Our estimates by signature size:
| Hash functions (k) | Estimate | Error |
|---|---|---|
| 16 | 0.375 | 0.125 |
| 64 | 0.5 | 0.0 |
| 128 | 0.516 | 0.016 |
| 256 | 0.492 | 0.008 |
With 16 hashes the estimate was far off. From 64 upward it settled near the true value. The 64-hash result landing exactly on 0.5 was partly luck: the expected error shrinks roughly with 1 / √k, so 64 hashes typically gives errors around 0.06 and 256 around 0.03. Choose k by how much error your use case tolerates.
Jaccard in scikit-learn
import numpy as np
from sklearn.metrics import jaccard_score
y_true = np.array([1, 1, 0, 1, 0])
y_pred = np.array([1, 0, 0, 1, 1])
print(jaccard_score(y_true, y_pred))
# 0.5
In machine learning, jaccard_score compares binary labels: positions where either array is 1 form the union (4 here), positions where both are 1 form the intersection (2), giving 0.5. The same idea, called Intersection over Union (IoU), scores object detection boxes and image segmentation masks.
Common use cases
- Near-duplicate detection for web pages, articles and support tickets, using shingles plus MinHash.
- Recommendations: users with similar sets of liked items probably share tastes.
- Tag and category matching between products or blog posts.
- Test result comparison: compare the set of failing tests between two builds to see how much changed. Our flaky tests guide covers why sets of failures shift.
- Retrieval evaluation in RAG systems: compare retrieved document IDs against the expected set.
- Model evaluation with IoU for detection and segmentation.
Limitations
- No frequency: an item appearing once or a hundred times counts the same. Weighted Jaccard variants exist when counts matter.
- No meaning: synonyms and paraphrases score zero overlap.
- Size imbalance: a small set fully contained in a large one still scores low. If you need “how much of A is in B”, use the overlap coefficient, |A ∩ B| / min(|A|, |B|).
- Preprocessing matters: case, punctuation and shingle size change results a lot.
Tip: before tuning thresholds, look at a sample of pairs at several scores, such as 0.3, 0.5 and 0.8. What “similar” means depends on your data, and eyeballing real pairs is the fastest way to pick a cut-off.
Final thoughts
Jaccard similarity is one of the simplest similarity measures and one of the most useful: shared items divided by total distinct items. Decide what goes into your sets, handle empty sets explicitly, choose shingles for text, and switch to MinHash when the collection is too large for pairwise comparison. When frequency or meaning matters, reach for cosine similarity or embeddings instead.
FAQ
What is Jaccard similarity?
It measures how similar two sets are by dividing the number of shared items by the number of distinct items in both sets. The result ranges from 0 to 1.
How do you calculate Jaccard similarity in Python?
Convert both inputs to sets, then divide the length of their intersection by the length of their union, and handle the case where both sets are empty.
What is the difference between Jaccard and cosine similarity?
Jaccard compares sets and ignores how often items appear. Cosine compares vectors and takes counts or weights into account.
What is MinHash used for?
MinHash creates small signatures whose matching fraction estimates Jaccard similarity, so very large collections can be searched for near duplicates quickly.
What is Jaccard distance?
Jaccard distance is one minus the Jaccard similarity. It is a true distance metric, so it works with clustering algorithms.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.