Learn AIFollow the slope of a mistake.
Turn errors into a loss curve, then find out why the size of a learning step matters.
By the end: Calculate one gradient update and recognize a learning rate that overshoots.
Start with something concrete.
A model predicts 2 × weight. Its target is 4. Start at weight 0.5: the prediction is 1, so the error is −3. Squaring it gives a loss of 9. Where on the curve would you move next?
Try the interactive model ↓The slope at weight 0.5 is −12. Which direction should a small gradient-descent step move?
Change one thing. Watch what follows.
- Use learning rate 0.10 and take one step. Compare the old and new losses.
- Reset, choose 0.30, and take a step. Explain why following the correct direction can still make the result worse.
- Set the weight to 2. What are the prediction, loss, and gradient?
Loading this browser experiment…
Loss makes one kind of mistake measurable
Squared error makes both positive and negative errors costly. An error of −3 and an error of +3 both give 9. Averaging over examples prevents the size of the dataset alone from multiplying the score.
A loss is a choice. Predicting a number often uses squared error; classifying an example often uses cross-entropy, which penalizes assigning little probability to the correct class. Neither objective automatically captures fairness, truth, or usefulness.
A gradient is a local sensitivity
Imagine moving a tiny distance to the right on the curve. A negative slope means the loss initially falls. A positive slope means it rises. The derivative names that local rate of change; a gradient collects the derivatives for several parameters.
For this example, the slope is 8w − 16. At w = 0.5 it is −12. Gradient descent subtracts learning rate times gradient: 0.5 − 0.1 × (−12) = 1.7. The new loss is 0.36.
Direction and distance are different decisions
The learning rate controls distance. A large step can pass the bottom and land higher on the other side. Reset and try 0.30: the next weight is 4.1 and the loss is 17.64, worse than 9.
Our bowl has one minimum. Real networks can have flat regions, ridges, and saddle points. A local direction does not promise a global best solution, and low training loss does not prove good performance on new data.
Transfer it: prediction = 3w, target = 6, starting w = 0. The gradient is −36. With learning rate 0.05, what is the new weight?
Take this idea with you.
A gradient suggests a local direction; the learning rate chooses the distance. Measure whether the actual update improved the loss.