Use momentum and adaptive steps.

A narrow valley asks for different steps

Imagine a loss map that changes quickly across a valley but slowly along it. A large step can overshoot the steep direction, while a small step can make slow progress along the gentle direction. The optimizer controls how the gradient becomes a parameter change.

The experiment has two dimensionless parameters and an exact minimum at (0, 0). It begins at (2, −1). Its loss is L = (u² + c × v²) / 2, where u and v are coordinates along the bowl’s axes and c is the curvature ratio, either 1 or 20. With c = 20, equal displacement in the v direction costs twenty times as much as in the u direction.

With rotation off, u is parameter 1 and v is parameter 2. With rotation on, u = (parameter 1 + parameter 2) / √2 and v = (parameter 1 − parameter 2) / √2. This preserves distances while changing the bowl’s orientation. The square root of 2 is about 1.414. You can understand the narrow valley visually without doing this optional coordinate calculation yourself.

The contours join points with equal loss. Both plot axes use the same fixed −4 to 4 scale. Paths that leave the view are clipped and counted, with exact coordinates still available. This is a mathematical optimization example, not a dataset: no validation score or generalization claim follows from reaching its minimum.

Separate three quantities

The gradient is the current loss sensitivity to each parameter. The optimizer state stores information from earlier gradients. The parameter delta is the actual change applied now. They need not have the same magnitude or direction.

In bowl coordinates, the gradient is (u, c × v). Rotation transforms that vector back to the displayed parameter coordinates. All components use the same old parameter vector; updating one parameter early would change the intended simultaneous update.

The workbench shows the current gradient, previous/new optimizer state, applied direction and delta in separate rows. Read those rows before taking a step. The proposed next parameter is a preview; it is not yet part of the accepted trajectory.

SGD uses the current gradient

Ordinary gradient descent applies θ_new = θ_old − η × g. Here θ (“theta”) is the parameter vector, η (“eta”) is the learning rate, and g is the gradient at the old vector. In this full-objective example there is no sampling noise; the control is labelled SGD to connect with the stochastic updates used in the preceding lesson.

For a separate worked example with rotation off, parameters (2, −1) and c = 20 give gradient (2, −20) and loss (4 + 20) / 2 = 12. At rate 0.05, subtract (0.1, −1): the new parameters are (1.9, 0). The new loss is 1.9² / 2 = 1.805.

Nearby practice: at rate 0.1, the same initial gradient produces (1.8, 1). Crossing the minimum in the steep direction is possible even though the update used the correct gradient. At an excessive rate, oscillations can grow. The lab stops proposals outside its numerical bound and retains the preceding parameters and memory.

Momentum accumulates past signals

This implementation uses a_new = 0.9 × a_old + g, then θ_new = θ_old − η × a_new. The vector a is accumulated gradient velocity, initially zero. The factor 0.9 retains part of the previous signal.

This convention does not multiply the new gradient by 0.1. Some presentations use an exponential average instead; with the same displayed learning rate that is a different step scale. The workbench states its convention so comparisons are reproducible.

Using the unrotated example above, the first momentum step matches SGD because the old accumulator is zero. After that step, the new gradient is (1.9, 0), but the stored accumulator is (2, −20). The next accumulated direction is (0.9 × 2 + 1.9, 0.9 × −20 + 0) = (3.7, −18). The next step therefore still moves in the second coordinate even though its current gradient is zero.

Momentum can sustain useful progress through consistent directions, but it can also overshoot or carry an outdated direction. It is not an automatic improvement for every rate and surface.

Adam tracks averages and squared averages

Adam stores two vectors: m, an exponential average of gradients, and v, an exponential average of squared gradients. In this section v denotes optimizer state, not the bowl-axis coordinate used earlier. The update is:

• m_new = β₁ × m_old + (1 − β₁) × g
• v_new = β₂ × v_old + (1 − β₂) × g²

The square is applied separately to each gradient coordinate. The lab uses β₁ = 0.9 and β₂ = 0.999. Both vectors start at zero, so early averages are biased toward zero. At optimizer-memory step t, it corrects them as m_hat = m_new / (1 − β₁ᵗ) and v_hat = v_new / (1 − β₂ᵗ). Hats indicate the corrected quantities; t starts at 1 for the next step after initialization or an explicit memory reset.

The applied direction in each coordinate is m_hat / (√v_hat + ε). Epsilon, ε = 10⁻⁸, is a small positive number outside the square root that prevents a zero denominator. The parameter delta is minus the learning rate times that direction. The denominator changes the step scale per coordinate; it is not a correctness probability or an estimate of predictive uncertainty.

For first gradient (2, −20), raw m is (0.2, −2) and raw v is (0.004, 0.4). At t = 1, bias correction recovers m_hat = (2, −20) and v_hat = (4, 400). The direction is approximately (1, −1), so rate 0.05 gives parameters near (1.95, −0.95). Epsilon makes the displacement very slightly smaller than 0.05; the calculation table retains the underlying numerical values.

Optional: coordinate-wise scaling has consequences

Adam rescales displayed coordinates separately. Rotating the coordinates of the same objective can therefore change its behavior. Plain gradient descent with the same rate transforms consistently under an orthogonal rotation, while coordinate-wise adaptive scaling generally does not. Neither property makes an optimizer universally best: model parameterization, objective geometry and tuning matter.

A checkpoint includes memory

After several Adam updates, save the complete checkpoint and inspect the next proposed delta. Reset optimizer memory only. The parameters and current gradient remain unchanged, but m, v and the bias-correction age restart. The next delta can differ.

Restore the complete checkpoint and inspect again. Exact continuation needs parameters, moments, correction age and the same fitting configuration. Keeping weights while discarding optimizer state is a new continuation experiment; do not describe it as an identical resume.

The lab retains the total accepted-update count when resetting memory. It also exports reset positions and its temporary checkpoint. These inspectable local records are not authenticated proof of a training history.

Compare a declared experiment

The comparison button starts each method from (2, −1) with zero memory, the same surface, a common rate and a stated update allowance. Early numerical stops report fewer accepted updates. Equal rate does not mean equal step size; equal updates do not mean equal total arithmetic or runtime.

Try the round bowl, then the narrow bowl, changing one setting at a time. A controlled mechanism comparison at one common rate is not a fully tuned optimizer benchmark. Real model comparisons also need appropriate development data, tuning allowances, stochastic-run variability and untouched evaluation evidence where available.

Export one comparison and explain it using the gradient, state and actual delta. For implementation practice, independently reproduce a momentum step and Adam’s first bias-corrected step. For decision practice, identify what compute and real-task evidence would be missing from an optimizer recommendation. Notes, software checks and a completed animation do not establish learning effectiveness. Human review and learner pilots remain pending.