Learn AILearn the pattern, not the examples.
A model can fit every training point and still fail on the next one. Find out how to tell.
By the end: Use validation data to choose a model, then freeze it before inspecting a separate test set.
Start with something concrete.
Imagine memorizing the answers to a practice exam. A perfect score on that sheet says little about tomorrow’s exam. Our lab fits curves through noisy points: more flexibility helps fit the sheet, but does it help on new points?
Try the interactive model ↓Which model would you trust more before a final test?
Change one thing. Watch what follows.
- Compare polynomial degrees 1, 4, and 11 with regularization at zero. Check both losses, not just how neatly the line fits the circles.
- Try regularization at 0.10. It discourages large nonconstant coefficients. Which validation result would you choose?
- Freeze your choice and evaluate the test set. If you reopen tuning, the lab reminds you that this test set is no longer unseen.
Loading this browser experiment…
Three datasets, three jobs
Training data fits the parameters. Validation data helps you choose settings such as model size or regularization. Test data estimates the final chosen model’s performance after those choices are frozen. Reusing the test score to make choices turns it into another validation score.
The lab uses 12 training points, 24 validation points, and 24 separately generated test points. All come from the same smooth toy relationship with independent noise. Circles are training observations; diamonds are validation observations. The curve is fitted only to the circles. The final test observations remain hidden until you freeze your choice.
A split must match the real task. If two almost identical photos of the same object land in different sets, the test may be too easy. For predicting the future, a time-based split can be more appropriate than shuffling everything. No split repairs unrepresentative or incorrectly labeled data by itself.
The most flexible curve is not always the best predictor
A polynomial adds powers of the input. Degree 1 is a straight line; a higher degree can bend more. In this lab, degree 11 has enough flexibility to pass through all 12 training points. Some of those points contain random noise, so a perfect fit can make the curve worse between them. This is overfitting. A model too restricted to capture the useful pattern underfits.
Compare degrees 1, 4, and 11. Training error falls as flexibility increases, but validation error does not keep falling. The comparison table reports actual measured errors from the fits; it is not an invented performance curve. Here, fitting solves a small least-squares system directly rather than running the gradient-descent animation from the previous module.
Every learning process has preferences
An inductive bias is a preference among explanations that could fit the observations. Model architecture, initialization, optimizer, regularization, and data all shape that preference. A model looking at neighboring pixels favors different patterns from one comparing words across a sentence.
Regularization adds a cost for certain parameter choices. Our lab penalizes the squared nonconstant polynomial coefficients. The fit therefore trades training error against that penalty. The displayed losses remain ordinary mean squared errors, without adding the penalty. This particular penalty depends on the numerical basis used to express the polynomial; it is one teaching choice, not a universal measure of simplicity.
More parameters do not automatically imply worse generalization. Modern networks can generalize even when they have enough parameters to memorize their data. Parameter count alone does not describe the solution actually found.
A good test has a boundary
Even an independent test only samples the conditions it contains. A model trained on bright photographs may fail in dim light. This is distribution shift: the conditions change. Test meaningful variations and report failures alongside successes.
Our frozen test is a small estimate, not proof of real-world reliability. If you reopen tuning after seeing it, it is no longer an unseen test. The lab keeps that warning even after a reset. In a real project, reserve new independent evidence for the next final claim.
A student checks the final test score after every change and chooses the best model. What is the problem?
Take this idea with you.
Fit with training data, choose with validation data, and evaluate the frozen choice on independent test data. A lower training error is not the same as better generalization.