Schedule the learning rate.

A training plan can change its pace

Imagine fitting a curve to noisy measurements. Large steps can move the weights quickly at first, but smaller later steps can make more careful adjustments. A learning-rate schedule declares how the step multiplier changes over training. It does not change the target or make a wrong gradient correct.

The lab compares constant, halfway step decay, cosine decay and ten-update warmup followed by cosine decay. All four start with identical zero weights and see the same sixteen training measurements on every update. This full-batch experiment removes shuffled-batch differences so you can focus on the schedule.

Start with peak rate 0.2, allowance 200, stopping off. Predict the winner at update five, then run the four-way comparison. Compare its early and final columns. An early lead is evidence about that point in the run, not a guarantee about the full plan.

Read the curves before the formula

The first plot shows the rate you planned to apply, including updates that early stopping may never reach. The second shows actual training and development losses at update zero and every five updates. Both loss curves share the same vertical scale. Straight connecting lines help you follow the checks; they do not add observations between them.

A lower training loss says the model fits the examples used for its gradients more closely. A lower development loss says it fits these separate, repeatedly inspected examples more closely. Neither is a final independent evaluation after tuning.

The exact tables remain available when a large loss goes above the fixed plot ceiling. The chart does not shrink its scale to make a struggling run appear stable.

What is being fitted?

Each fictional measurement has one dimensionless input x, between −1 and 1, and a dimensionless target y. The underlying teaching relationship is 0.7x − 0.25. Both splits add separately seeded measurement noise between −0.3 and 0.3. There are sixteen distinct training inputs and sixteen different development inputs. These are generated teaching measurements, not a real sensor dataset.

The model has eight adjustable coefficients. It combines an offset, a straight-line input and six fixed wave-shaped features. A feature is a number calculated from x before the weighted sum. The wave features let the model fit bends, including accidental patterns in the noise. The features are fixed; their coefficients learn.

For example, when x = 0, the eight features are (1, 0, 0, 1, 0, 1, 0, 1). The prediction is the sum of the coefficients at the positions containing 1. With all coefficients initially zero, the prediction is zero. For a separate worked measurement with target 2, the squared error is (0 − 2)² = 4.

Optional: the fixed wave features and their dimensions

The feature vector is φ(x) = [1, x, sin(πx), cos(πx), sin(2πx), cos(2πx), sin(3πx), cos(3πx)]. The symbol φ (“phi”) names the eight-number feature vector. Sine and cosine make waves; π is about 3.142, and the angles are in radians. You do not need trigonometric calculations to compare the visible curves.

The coefficient vector w also has eight numbers. Prediction ŷ = Σⱼ wⱼφⱼ(x) means “multiply each feature by its coefficient, then add.” The index j runs through all eight positions. This is a linear model in fixed features, not a deep neural network. The schedule mechanism also applies to neural-network gradients, but these results do not establish which schedule works best there.

From an error to a scheduled update

The loss is L = (1/n) Σᵢ (ŷᵢ − yᵢ)². Here n = 16 is the number of training measurements, i identifies a measurement, y is its target and ŷ is its prediction. Read it as “average the squared prediction errors.” Squaring makes positive and negative errors contribute rather than cancel.

For coefficient j, its gradient is gⱼ = (2/n) Σᵢ (ŷᵢ − yᵢ) φⱼ(xᵢ). It averages how the prediction errors interact with that feature. In the separate one-measurement example above, the offset gradient is 2 × (0 − 2) × 1 = −4. At rate 0.2, its proposed change is −0.2 × −4 = +0.8. In the actual lab, all sixteen rows contribute before any coefficient changes.

The simultaneous update is w_new = w_old − ηₜg. The symbol η (“eta”) is the learning rate, and t is the zero-based update index. A schedule changes ηₜ, while the gradient g still comes only from the current model and training measurements.

Four declared plans

• Constant: every update uses the chosen peak rate.
• Halfway step decay: the first half uses the peak, then the second half uses 10% of it. With 200 updates and peak 0.2, updates 1–100 use 0.2 and updates 101–200 use 0.02.
• Cosine decay: smoothly reduces the rate from the peak to 10% of it across the declared allowance. It does not reduce all the way to zero.
• Warmup plus cosine: increases linearly from 10% of the peak to the peak over ten updates, then smoothly decays to 10% by the last allowed update.

For warmup with peak 0.2, updates 1, 5 and 10 use 0.02, 0.10 and 0.20. Nearby practice: with peak 0.05, predict the fifth rate before changing the control. It is 0.025: half of the peak.

Warmup can soften the initial steps, but it uses part of the allowance to do so. It does not guarantee better results or prevent later divergence. Changing the allowance also changes where the decay happens. Declare it before comparing runs.

Optional: reproduce the cosine rule

Let H be the declared allowance, either 100 or 200. For plain cosine decay, progress p = t/(H − 1) runs from 0 to 1. Then ηₜ = η_peak × [0.1 + 0.9 × (1 + cos(πp))/2]. At p = 0 the bracket is 1; at p = 1 it is 0.1.

For warmup, the first ten updates use ηₜ = η_peak × (t + 1)/10, for t = 0 through 9. After that, use the cosine formula with p = (t − 9)/(H − 10). Its last allowed rate is exactly the same 10% floor. The full schedule uses the original allowance even if validation later stops the run early.

Stopping is a decision made from development evidence

A checkpoint is a saved set of weights. The experiment checks the development loss at update zero and every five updates. Its best checked checkpoint is the one with the lowest observed development loss; ties keep the earlier checkpoint. It never searches unmeasured intermediate steps for a better score.

Patience counts consecutive checks without a sufficiently large improvement. Here a decrease greater than 0.0005 from the last significant-improvement reference resets the count. A patience of three allows three unsuccessful checks, or fifteen updates at this checking interval. Patience zero turns stopping off.

Suppose the reference is 0.9000 and the next losses are 0.8998, 0.8997 and 0.8996. All three are new strict bests, but none improves the reference by more than 0.0005. Patience three therefore expires. If a later check were 0.8994 before patience ran out, its cumulative decrease of 0.0006 would reset the count. The threshold has the same squared-target units as the loss.

This is why “the loss is still falling” does not imply the stopping rule must continue. The best checkpoint and patience reference answer different questions. A stopping threshold is a declared tolerance, not a claim of statistical significance.

Keep the last weights and selected weights separate

Stopping leaves the individual run at its last accepted weights. The export also identifies the best checked weights. It does not silently restore them and then pretend training followed a different trajectory. You may choose the best checked checkpoint for a frozen evaluation later, but that selection itself used development evidence.

With stopping off, training loss can continue to improve while development loss gets worse. In this small noisy example, fitting the training set more closely is not necessarily useful for the other inputs. Early stopping is one way to select a checkpoint, not proof that overfitting has been eliminated.

A tuning budget includes unsuccessful choices

Four schedules at 200 full-batch updates can use 4 × 200 × 16 = 12,800 fitting-example visits. Monitoring is additional work: each full run has 41 scheduled development checks, including update zero, or 41 × 16 = 656 development-example evaluations. Training-curve calculations and interface recomputation also consume work and are not fitting steps.

The lab counts accepted fitting work across the individual run and recorded comparisons, including attempts you do not select. It separately labels the scope of its monitoring count. Cancellation may discard partial work, so these counters are not a measurement of total compute. Worker timing excludes rendering. Equal update allowances are not equal elapsed time, especially when stopping ends some runs early.

Changing settings keeps the temporary comparison history. Up to twelve comparison runs can be recorded. Download before leaving if you want to retain them. Reloading clears temporary state; it does not turn a familiar development set into untouched evidence.

Apply the evidence to a decision

For a guided experiment, explain why the early and final rankings differ and identify one checkpoint you would select. Include the rate, allowance, stopping rule, actual work and limitations.

For implementation practice, reproduce the rates, simultaneous gradient update and five-update checking schedule from the export. Check a hand-calculated gradient, a finite difference and the exact warmup/decay endpoints. Reproduce a stopped run without fitting to development labels.

For a management decision, compare the training work spent on all candidates with the improvement of the selected checkpoint. State what separate frozen evaluation and operating evidence you would require before using a model in a product. These local teaching timings do not predict production costs.

These are practice artifacts. Human review and representative learner pilots remain pending. Continue with training diagnosis or validation and model selection to connect the schedule choice to a wider evaluation plan.