Nearest neighbors
Throw away modeling entirely: to answer a question, just find the most similar past cases and do what they did.
Logistic regression insists on a straight wall. Nearest neighbors
(k-NN) is the opposite philosophy: no model at all. Store the
training points; when a new input arrives, find the k stored points
closest to it — in whatever distance measure you defined — and let
them vote.
The lazy algorithm
It's how doctors, used-car buyers, and everyone actually works when rules are unwritable: "here are the cases that look most like yours, here's what happened to them."
Click to move the query point — k stored cases vote
vote: class 1
k=1: every point wears a private crown — spiky, memorized regions. k=11: two blunt blocs. The dial in between is the whole bias/variance trade, again.
Orange ring = your query; lines = its k nearest stored cases. The faint background is the decision surface — where the vote flips. Note it was never “trained”: it just stores points and answers by lookup. That’s the whole family called “lazy” learning.
The three design decisions, all visible up there
- k — neighborhood size. k=1 chases every point's quirk (max variance, the overfit end of the dial); large k averages whole regions into one answer (max bias, underfit end). There is no theoretical best — you pick it with held-out data, chapter 3 style.
- The distance function. "Close" is computed numerically, so whichever feature has big numbers is distance. Height-in-centimeters vs weight-in-kilograms: centimeters wins by unit choice alone. Chapter 3's scaling lesson is this bullet in its most violent form.
- What "similar" misses. Every point gets compared to everything it isn't, on features you chose. Garbage features → "similar" is coincidence, and the vote is noise confidently delivered.
What it costs (and why it's the ceiling)
The price is memory and lookup at prediction time: the model IS the data. Every query scans your whole training set (or an index built on it). Modern embedding search — vectors, cosine similarity, ANN indexes — is this idea re-fitted for scale, and it comes back as the retrieval of the LLM book's RAG chapter: find what's near in the representation, then act.
That ceiling is worth respecting even when you go elsewhere: k-NN will faithfully carve any boundary shape — spirals, islands — that distance in feature space can support. When the honest truth needs curves, k-NN delivers. When the feature space is garbage, it delivers garbage with confidence.
Illustrative vs real
Twelve hand-made points and a 20×20 region grid here. Real k-NN uses ball trees/LHS indexes and distance-weighted votes — but nothing in the decision logic differs from what you just clicked through.
Where next: decision trees — rules rather than geometry.