K Nearest Neighbors (KNN) Explained: How It Works, With Python Code
K nearest neighbors classifies a new point by looking at the k most similar examples it has already seen and taking a vote. Learn the algorithm step by step, code it from scratch in NumPy, see how the choice of k and feature scaling change accuracy, and where KNN powers modern vector search.
- K nearest neighbors stores the training data and predicts by majority vote (or average) of the k closest examples.
- Our from-scratch NumPy version scored 1.000 test accuracy with k=1 and k=3 and 0.978 with k=5 to 45, but fell to 0.222 when k equalled the whole training set.
- Adding one unscaled noisy feature dropped accuracy from 1.000 (standardized) to 0.444. Always scale features.
- Brute-force KNN cost about 19 ms per query on 100,000 points; large systems use approximate nearest neighbour indexes.
Some machine learning algorithms learn weights through hours of training. K nearest neighbors learns nothing up front: it remembers the examples and, when asked about a new point, finds the most similar ones and lets them vote. That simplicity makes KNN the perfect first algorithm to understand, a strong baseline for small datasets, and the idea behind the vector search used in modern AI retrieval.
Every number in this article comes from code we ran with Python 3.13.6 and NumPy 2.5.5. The dataset is synthetic, generated with a fixed seed so you can reproduce it. We did not run scikit-learn for this article.
What is the k nearest neighbors algorithm?
KNN is a supervised learning algorithm for classification and regression. To label a new point, it measures the distance to every training point, picks the k closest, and returns the most common label among them (classification) or their average value (regression). Because it does all its work at prediction time, it is called a lazy or instance-based learner.
The algorithm in four steps
- Choose k, the number of neighbours to consult.
- Compute the distance from the new point to every training point.
- Sort the distances and keep the k smallest.
- Vote: return the most common label, or the mean value for regression.
There is no training step beyond storing the data, which is both KNN’s charm and, for large datasets, its main cost.
Our dataset: limes, lemons and oranges
We generated 180 fruits, 60 of each kind, with two features: weight in grams and diameter in centimetres. Limes average about 60 g and 4.5 cm, lemons 110 g and 6.0 cm, oranges 180 g and 7.5 cm, with some natural spread so the groups overlap a little. After shuffling, 135 fruits became the training set and 45 the test set.
KNN from scratch in NumPy
import numpy as np, collections
def predict(Xtr, ytr, x, k):
d = np.sqrt(((Xtr - x) ** 2).sum(axis=1)) # Euclidean distance to every point
nn = np.argsort(d)[:k] # indexes of the k nearest
label = collections.Counter(ytr[nn]).most_common(1)[0][0]
return label, nn, d[nn]
def accuracy(Xtr, ytr, Xte, yte, k):
return np.mean([predict(Xtr, ytr, x, k)[0] == t for x, t in zip(Xte, yte)])
That is the whole algorithm: one line of vectorised distance maths, one sort and one vote. NumPy computes all 135 distances at once without a Python loop.
Looking at the neighbours
q = np.array([120.0, 6.1]) # a 120 g fruit, 6.1 cm across
label, nn, dist = predict(Xtr, ytr, q, k=5)
# query [120. 6.1] -> lemon
# neighbour [120. 5.7] lemon dist 0.37
# neighbour [122.8 6.4] lemon dist 2.77
# neighbour [123.5 7. ] orange dist 3.65
# neighbour [124.5 5.4] lemon dist 4.56
# neighbour [114.8 6. ] lemon dist 5.21
Four lemons and one orange, so the vote says lemon. Seeing the neighbours is one of KNN’s best features: the prediction comes with its own explanation. Notice something odd, though: the distances are almost entirely about weight. A 3 g difference counts far more than a 1.3 cm difference in diameter, simply because grams are bigger numbers. We come back to that in the scaling section.
Choosing k
| k | Test accuracy |
|---|---|
| 1 | 1.000 |
| 3 | 1.000 |
| 5 | 0.978 |
| 7 | 0.978 |
| 15 | 0.978 |
| 45 | 0.978 |
| 135 (all points) | 0.222 |
On a test set of 45, one mistake is a drop of 0.022, so k from 1 to 45 performed almost identically on this easy, well-separated data. The extreme case is the lesson: with k equal to the whole training set, every prediction is simply the most common class overall, and accuracy collapsed to 0.222. In general:
- Small k follows the data closely and is sensitive to noise and mislabelled points (overfitting).
- Large k smooths the decision boundary and eventually ignores local structure (underfitting).
- Pick k with cross-validation on your data. An odd k avoids ties in two-class problems.
Why feature scaling matters
To show the scaling problem clearly, we added a third, useless feature in large units: random values around 5,000 with a spread of 1,500, like a price in paise that has nothing to do with the type of fruit.
| Features (k=5) | Test accuracy |
|---|---|
| Weight and diameter, raw | 0.978 |
| Weight, diameter and noisy feature, raw | 0.444 |
| Same three features, standardized | 1.000 |
| Weight and diameter, standardized | 0.978 |
With raw values, the noise feature’s large numbers dominated every distance, so the “nearest” neighbours were just fruits with similar random prices, and accuracy fell to 0.444, barely better than guessing among three classes. Standardizing each feature (subtract the training mean, divide by the training standard deviation) put all three on the same scale and restored accuracy. The small difference between 1.000 and 0.978 is a single test fruit and not meaningful on a 45-item test set. Always compute the scaling statistics on the training data only, then apply them to test data, to avoid leaking information.
Tip: KNN has no way to learn that a feature is irrelevant. Scale features, and drop or down-weight ones that do not help.
Distance metrics
| Metric | Formula idea | Good for |
|---|---|---|
| Euclidean | Straight-line distance | Continuous, scaled features |
| Manhattan | Sum of absolute differences | High dimensions, grid-like data |
| Cosine | Angle between vectors | Text and embeddings, where direction matters more than length |
| Hamming / Jaccard | Count or share of differing items | Binary features and sets |
For text embeddings, cosine similarity is the usual choice; see our cosine similarity guide. For sets of tags, Jaccard similarity fits better.
Ties and distance weighting
collections.Counter([0, 1]).most_common(1) # [(0, 1)]
With k=2 and one vote each, most_common simply returned the label it saw first, an arbitrary tie-break. Use an odd k for two classes, or break ties by the nearest neighbour. A popular refinement is distance weighting, where each vote counts as 1 divided by the distance, so very close neighbours outweigh distant ones. That also reduces sensitivity to the exact choice of k.
KNN for regression
def knn_reg(x, k=5):
nn = np.argsort(np.abs(w[:, 0] - x))[:k]
return dia[nn].mean()
knn_reg(150) # 7.5 cm predicted diameter for a 150 g fruit
For numeric targets, KNN averages the neighbours’ values instead of voting. Here we predicted diameter from weight alone. KNN regression produces step-like predictions and cannot extrapolate beyond the range of the training data, so in our run a 400 g fruit was predicted at just 7.74 cm, the average diameter of the five heaviest oranges, rather than anything larger.
How fast is KNN?
Brute-force KNN compares each query with every stored point. We timed 100 queries against 100,000 random points with 16 features, using np.argpartition to find the 5 nearest without a full sort: about 19 ms per query. That is fine for thousands of predictions on a small dataset, but it grows linearly with the data. Tree structures such as KD-trees and ball trees help in low dimensions; for millions of high-dimensional vectors, systems use approximate nearest neighbour (ANN) indexes such as HNSW, trading a tiny amount of accuracy for huge speed-ups. We did not benchmark those here.
KNN, embeddings and vector search
Modern AI applications use KNN constantly, just under another name. A retrieval-augmented generation system turns documents into embeddings, stores them in a vector database, and for each question finds the k nearest document vectors by cosine similarity. That is KNN search on embeddings, usually with an ANN index. Read more in what are embeddings, our vector database guide and what is RAG.
Pros and cons of KNN
| Pros | Cons |
|---|---|
| Simple to understand and implement | Slow predictions on large datasets |
| No training time | Stores the entire dataset in memory |
| Explains predictions by showing neighbours | Very sensitive to feature scale and irrelevant features |
| Handles many classes naturally | Struggles in very high dimensions without good embeddings |
| Adapts instantly when new data is added | Needs a sensible distance metric |
Final thoughts
K nearest neighbors is machine learning stripped to its essence: similar things tend to have similar labels. Store the examples, measure distance, let the k closest vote. Our from-scratch version reached perfect or near-perfect accuracy on clean data, collapsed when k was absurdly large, and was wrecked by a single unscaled feature until we standardized it. Scale your features, choose k by validation, and use approximate indexes at scale. For turning scores into probabilities, see the softmax function.
Frequently asked questions
How does the k nearest neighbors algorithm work?
It stores the training data. To predict a new point, it measures the distance to every stored point, takes the k closest, and returns their most common label for classification or their average for regression.
How do I choose the value of k in KNN?
Try several values with cross-validation and pick the one with the best validation score. Small k is sensitive to noise, large k blurs class boundaries, and an odd k avoids ties with two classes.
Why do I need to scale features for KNN?
KNN uses distances, so features with large numeric ranges dominate. In our test an unscaled noisy feature cut accuracy to 0.444, and standardizing restored it.
Is KNN supervised or unsupervised?
KNN classification and regression are supervised because they need labelled examples. The related nearest neighbour search used in vector databases needs no labels and simply finds similar items.
Is KNN good for large datasets?
Plain brute-force KNN gets slow because every prediction compares against all stored points. Large systems use KD-trees, ball trees or approximate nearest neighbour indexes such as HNSW.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.