Learn AIStart with a simple baseline.
Fit majority, nearest-neighbor and shallow-tree predictors; compare the same validation cases before adding complexity.
By the end: Explain a baseline, trace a neighbor vote or tree path, and justify model complexity using separate development evidence.
Start with something concrete.
A model makes mistakes. Before building a larger network, ask how often a rule that always predicts the common class would be right. Then inspect whether nearby examples or a few learned questions improve on that reference.
Try the interactive model ↓A one-neighbor model gets zero training error, including on three flipped labels. What should you inspect next?
Change one thing. Watch what follows.
- Compare training and validation errors for the majority, 3-neighbor and depth-2 tree models. Identify the improvement over the majority reference.
- Select a validation case and trace its actual nearest neighbors. Change k from 1 to 15 and explain a changed prediction or stable vote.
- Switch the displayed map to the tree and increase depth from 1 to 4. Inspect its learned questions, training error and validation error.
- Flip three training labels. Compare k = 1 with k = 7 on the unchanged validation cases. Write a comparison note with one mistake and a next evidence request.
Loading this browser experiment…
Give improvement a reference point
Suppose 80 of 100 examples have label 0. Always predicting 0 would be correct on 80 examples while missing every label-1 case. That baseline is a reference for comparison, not necessarily an acceptable decision. An 81% model has not established much without examining which errors it changes and their costs.
For a classification task, fit a majority baseline from training labels. For a numerical prediction task, a training mean is a useful starting point. Do not use validation labels to decide what your “training majority” was. The lab compares three predictors on identical training and development validation coordinates.
Learn a decision from nearby examples
A nearest-neighbor predictor stores training examples. Given a query, it measures distance to each training input, selects the k closest examples and takes a vote. Here k is a positive whole number, and every selected example receives one vote. Ties between classes choose 0; equal distances use the original training order so results are reproducible.
The map connects the selected query to the examples that vote. Both inputs range from 0 to 1 with equal scale. A query at (0.2, 0.3) and a training input at (0.5, 0.7) differ by 0.3 horizontally and 0.4 vertically:
The square-root symbol reverses squaring: √0.25 = 0.5 because 0.5 × 0.5 = 0.25.
d is the Euclidean distance between two points, in the same coordinate units as the inputs. It is the straight-line length of the displacement. If one input instead used millimetres and the other kilometres, its numerical scale could dominate. In a real pipeline, decide what distance means and fit any scaling only on training data.
With k = 1, one label decides the answer. With larger k, a broader neighborhood votes. A broader neighborhood can reduce sensitivity to one corrupted label while also mixing examples from opposite sides of a real boundary. The vote fraction is not automatically a well-calibrated probability of correctness.
Let a tree ask learned questions
A decision tree partitions the input plane using questions such as “Is x at most 0.625?” Follow the selected query through the questions to reach a leaf. The leaf predicts its training-majority label. This is a fitted tree: it searches candidate cuts using training labels, rather than receiving the underlying synthetic rule.
The lab scores a candidate split by how mixed its two resulting label groups are. Gini impurity for a binary group is 2 × p × (1 − p), where p is its fraction of label-1 examples. A pure group has impurity 0. A half-and-half group has impurity 0.5. The fit weights each child group’s impurity by its example count, then chooses the smallest sum. It considers midpoints between distinct training coordinates, with at least two examples on each side.
Tree depth counts how many questions may appear on a path. More depth can describe a more intricate boundary, but can also carve around unusual or wrong training labels. The lab caps depth at four. Ties use x before y and the lower cut before a higher one. Read the trace alongside the region map: each question corresponds to an axis-aligned division.
Compare the same examples
The 64 training coordinates and 49 validation coordinates are disjoint fixed grids. The data-generating label rule is x + y ≥ 1.05. Models fit only training labels; displaying the rule explains the synthetic exercise, not an implementation that secretly supplies the answer to the predictor. Flipping three training labels changes only those labels. Validation stays fixed.
Training error can be misleading. When measuring a nearest-neighbor model on its own training inputs, each query can retrieve itself. With k = 1, even a flipped label can be reproduced perfectly. Inspect separate validation errors to see whether the model predicts beyond that self-match.
The coarse background samples 400 cell centers; it is a visual approximation of prediction regions. The numerical trace uses the actual query. Use the selected-case control to inspect an error without relying on the plot’s colors or resolution.
Make a bounded choice
Write down the baseline, candidate settings, validation error count and an inspected mistake. Consider storage and prediction cost as well as accuracy: nearest neighbors retains examples and computes distances at query time, while a fitted shallow tree follows only a few questions. Neither architecture is automatically the right choice.
Every time you inspect this validation set to choose a setting, you use development evidence. The page does not contain an untouched final test, and repeated visits are not independent replications. To support a final claim, freeze the choice and evaluate suitable new cases. The real-data audit project adds measurement and split issues; the next regression lesson asks what changes when the output is a number.
A validation query has nearest training labels 0, 1 and 1. An equal-weight three-neighbor classifier predicts what?
Take this idea with you.
Keep a simple reference, compare on the same separate examples and inspect individual errors. More neighbors or deeper trees change assumptions; neither is automatically better.