Naive Bayes Classifier Explained: Build a Spam Filter in Python
A naive Bayes classifier predicts a label by combining how common each class is with how likely each word or feature is within that class. Learn Bayes rule, the naive independence assumption, Laplace smoothing, why we add log probabilities, and Gaussian naive Bayes, by building a small spam filter from scratch and running it.
- A naive Bayes classifier picks the class with the highest prior probability times the likelihood of the observed features.
- “Naive” means it assumes features are independent given the class, which is rarely true but often works well.
- Without smoothing, one unseen word made both class scores exactly 0 in our test; Laplace smoothing fixes it.
- Multiplying many small probabilities underflowed to 0.0; summing logarithms gave a usable -1842.07 instead.
Before deep learning, the first widely successful email spam filters were built on a surprisingly simple idea: count how often words appear in spam and in normal mail, then use probability to score new messages. That idea is the naive Bayes classifier. It trains in a single pass, works with tiny datasets, explains its decisions and remains a strong baseline for text classification.
We built one from scratch in Python 3.13.6, then a Gaussian version with NumPy 2.5.5; every number below comes from those runs. The ten training messages are made-up examples, so treat the probabilities as illustrations, not benchmarks. For a different classic algorithm, compare k nearest neighbors.
What is a naive Bayes classifier?
Naive Bayes is a family of supervised learning algorithms that apply Bayes’ theorem to classification. Given features (words in an email, measurements of a flower) it estimates the probability of each class and predicts the most probable one. Training only requires counting, so it is extremely fast, and the model is a small table of probabilities you can inspect.
Bayes rule in one line
P(class | features) = P(features | class) x P(class) / P(features)
- P(class) is the prior: how common the class is overall.
- P(features | class) is the likelihood: how typical these features are for that class.
- P(features) is the same for every class, so for choosing the winner we can ignore it and compare
likelihood x prior.
The naive independence assumption
Estimating the probability of an entire email word sequence is impossible with small data. The naive assumption says: given the class, each word is independent of the others. Then the likelihood becomes a simple product:
P(w1, w2, ..., wn | spam) = P(w1 | spam) x P(w2 | spam) x ... x P(wn | spam)
Words are obviously not independent (“free” and “prize” go together), so the probabilities the model outputs are often too extreme. But the ranking of classes is frequently still right, which is all a classifier needs.
Our training data
spam: "win a free prize now", "free money claim now",
"limited offer win cash", "claim your free gift card"
ham: "meeting moved to monday", "can you review my pull request",
"lunch on monday?", "the build is green now",
"please review the test plan", "call me after the meeting"
Four spam and six normal (“ham”) messages. Lowercased word tokens gave a vocabulary of 35 words: 18 word occurrences in spam and 28 in ham.
Training: priors and word counts
priors: {'spam': 0.4, 'ham': 0.6}
vocab size: 35 words per class: {'spam': 18, 'ham': 28}
The priors are just class frequencies: 4 of 10 messages are spam. The likelihoods come from counting each word per class. That is the entire training step, a single pass over the data, which is why naive Bayes can train on millions of documents in seconds.
Laplace smoothing: handling unseen words
P(free | spam) = (3 + 1) / (18 + 35) = 0.0755
P(free | ham) = (0 + 1) / (28 + 35) = 0.0159
“free” appeared 3 times in spam and never in ham. Without adjustment, P(free | ham) would be 0, and any message containing “free” could never be classed as ham, no matter what else it said. Laplace (add-one) smoothing adds 1 to every count and the vocabulary size to the denominator, so no word ever has zero probability. We tested the unsmoothed version on “free review”:
no smoothing 'free review': {'spam': 0.0, 'ham': 0.0}
Both scores collapsed to exactly zero, because “review” never appeared in spam and “free” never appeared in ham. The model could not decide at all. The smoothing parameter, alpha, does not have to be 1; smaller values are common and can be tuned.
Why we add logarithms instead of multiplying
0.01 ** 400 # 0.0 (underflow)
400 * log(0.01) # -1842.0680743952364
A long email multiplies hundreds of small probabilities, and the product quickly becomes smaller than the tiniest number a float can hold, so it rounds to 0. Taking logs turns the product into a sum of manageable negative numbers, and the class with the largest log score is still the winner. Our classifier works entirely in log space and converts back to probabilities only at the end, using the same subtract-the-maximum trick as the softmax function.
def logp(text, c, alpha=1.0):
s = math.log(prior[c])
for w in tok(text):
s += math.log((counts[c][w] + alpha) / (total[c] + alpha * len(vocab)))
return s
Predictions on new messages
'free prize claim' -> spam {'spam': 0.964, 'ham': 0.036}
'review the meeting notes' -> ham {'spam': 0.036, 'ham': 0.964}
'win monday lunch' -> ham {'spam': 0.359, 'ham': 0.641}
'free review' -> spam {'spam': 0.557, 'ham': 0.443}
Reading the results
- “free prize claim” uses three spam-only words, so it is confidently spam.
- “review the meeting notes” uses ham words; “notes” was never seen, but smoothing let the other words decide.
- “win monday lunch” mixes a spam word with two ham words and leans ham, helped by the higher ham prior.
- “free review” is a near tie, 0.557 to 0.443: one strong spam word against one ham word. With more training data, such borderline cases become rarer, and in production you would usually require a high spam probability before filtering.
Because the model is just word probabilities, you can explain every decision by listing which words pushed it towards each class, something deep learning models struggle to offer. Better text features, such as TF-IDF weights or word pairs, usually improve accuracy further.
Tip: the probabilities from naive Bayes are often overconfident. Use them to rank, and choose your decision threshold on validation data.
Gaussian naive Bayes for numeric features
Words are counts, but many features are measurements. Gaussian naive Bayes assumes each feature follows a normal distribution within each class, and learns its mean and standard deviation. We generated 400 synthetic points in two classes with two numeric features each, trained on 300 and tested on 100:
mu = [Xtr[ytr == c].mean(0) for c in (0, 1)]
sd = [Xtr[ytr == c].std(0) for c in (0, 1)]
def g(x, m, s):
return (-0.5 * ((x - m) / s) ** 2 - np.log(s * np.sqrt(2 * np.pi))).sum(1)
# Gaussian NB test accuracy: 0.98
98 of 100 test points were classified correctly. The whole model is four means, four standard deviations and two priors. Notice that the score is again a sum of log densities, one per feature, which is the naive assumption at work.
Naive Bayes variants
| Variant | Features | Typical use |
|---|---|---|
| Multinomial | Counts, such as word frequencies | Text classification, spam, topics |
| Bernoulli | Binary present or absent | Short texts, yes or no features |
| Gaussian | Continuous measurements | Sensor data, simple numeric datasets |
| Complement | Counts, imbalanced classes | Text with rare classes |
Our spam filter was multinomial. Libraries such as scikit-learn provide all four; we did not run scikit-learn for this article.
Strengths and weaknesses
| Strengths | Weaknesses |
|---|---|
| Trains in one pass, very fast | Independence assumption is rarely true |
| Works with small datasets | Probabilities are often overconfident |
| Handles many classes and features | Cannot learn feature interactions |
| Easy to explain and update | Usually beaten by modern models on large data |
For text, modern systems often use embeddings and large language models, but naive Bayes remains a valuable first baseline: if a complex model cannot beat it, something is wrong.
Final thoughts
The naive Bayes classifier turns counting into prediction: priors from class frequencies, likelihoods from feature counts, Bayes rule to combine them. Add Laplace smoothing so unseen words do not zero out a class, work in log space to avoid underflow, and choose the variant that matches your features. Our small spam filter showed each of those pieces, from confident predictions to a borderline tie, in a few dozen lines of Python.
Frequently asked questions
What is a naive Bayes classifier?
It is a probabilistic classifier that applies Bayes rule and assumes features are independent given the class. It predicts the class with the highest prior times likelihood.
Why is naive Bayes called naive?
Because it assumes every feature is independent of the others once the class is known. That assumption is usually false, but the classifier often still ranks classes correctly.
What is Laplace smoothing in naive Bayes?
It adds a small count, usually 1, to every feature in every class so that a word never seen in a class does not get zero probability and wipe out that class score.
Why use log probabilities in naive Bayes?
Multiplying many small probabilities underflows to zero in floating point. Adding their logarithms keeps the numbers in a safe range and gives the same winning class.
When should I use Gaussian naive Bayes?
Use it when features are continuous measurements that are roughly normally distributed within each class. Use multinomial naive Bayes for word counts and Bernoulli for binary features.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.