Repair a broken training run.

Follow the update, not only the curve

The lab has one input, four hidden units and one output: 13 adjustable numbers. A hidden unit multiplies its input by a weight, adds an offset called a bias, then applies tanh. Tanh smoothly keeps the output between −1 and 1. The output layer combines those four values into a prediction.

The target is a supplied curved line, 0.5x + 0.35 sin(3x). You do not need to derive the sine function to use the experiment: the target dots are the answers. Thirty-two training points determine updates. Thirty-two different, interleaved points monitor how the same curve is predicted between them. This synthetic task is deliberately small.

Error becomes a direction

For one prediction of 3 and a target of 5, error is −2 and squared error is 4. The lab averages squared errors over all 32 training examples. A gradient says how sensitive that average is to a small change in one parameter. Backpropagation calculates every gradient from the same unchanged model.

$L = \frac{1}{32}\sum_{i=1}^{32}(\hat y_i-y_i)^2,\qquad \theta_{new}=\theta-\eta g$

L is average squared error, i identifies an example, ŷ is a prediction and y is its target. θ is a weight or bias, η is the learning rate, and g is that parameter’s gradient. SGD subtracts rate times gradient. At θ = 0.4, η = 0.1 and g = −0.2, the new value is 0.42. Inspect your current parameter in the lab: the worked numbers above are a separate example.

Three failures need different repairs

Observation	Possible mechanism	Controlled check
Computed gradient is nonzero; applied gradient is zero	The update path discards the gradient	Restore the computed gradient and compare from the same initialization
Many hidden outputs are near −1 or 1	Inputs drive tanh into its flat ends	Restore input scale and inspect hidden gradients and losses
Loss and parameter changes rapidly grow	Steps overshoot useful values	Reduce the rate from the same checkpoint and keep the failed trial in the report

These are diagnoses supported by controlled evidence, not rules that every flat curve must obey. Tanh’s slope is 1 − h², where h is its output. At h = 0 the slope is 1; at h = 0.99 it is 0.0199. Large output-layer gradients can still coexist with tiny hidden-layer gradients.

Adam is another update rule, not a repair guarantee

Adam tracks running averages of gradients and squared gradients, corrects their initial bias, then scales each parameter’s step. The lab uses β₁ = 0.9, β₂ = 0.999 and a small denominator offset ε = 10⁻⁸. A different optimizer may need a different rate. Changing controls resets parameters and optimizer memory so the comparison has a stated starting point.

The model retains the preceding bounded state if an update becomes unsafe. Worker cancellation also keeps the preceding completed batch. Neither a stopped run nor a cancelled run is a successful zero-cost experiment.

Separate diagnosis, repair and generalization

Capture healthy and blocked runs after the same 100 accepted updates. Their data, initialization, rate, optimizer and input scale must match. A repair should change the suspected cause, change the predicted diagnostic, and improve the intended outcome. Training improvement alone does not establish generalization. Validation remains repeated development evidence here.

In the project, write an incident report: symptom, mechanism, controlled repair, supporting numbers and limits. New assessment cases use scalar models and different update bugs, so they require tracing the mechanism rather than recalling the preset name. Implementation artifacts and written reasoning remain available for human review; automatic arithmetic checks do not grade code quality.

Optional depth: check a gradient independently

Evaluate loss after adding a tiny ε to one weight, then after subtracting ε. Divide the difference between losses by 2ε. This central finite difference should approximately match the analytic gradient. It verifies the gradient calculation, not whether a later optimizer uses the right sign or changes the same parameter storage the model reads.

Extend the diagnosis to real measurements

Open Extend the diagnosis: real-data neural failures below the synthetic lab. This second model predicts an Iris flower’s petal length from its two sepal measurements. It has 17 parameters in a 2–4–1 tanh network. The source is the locally stored, licensed UCI Iris data; generated row IDs and identical-measurement groups do not identify individual specimens.

The extension reuses the tabular project’s grouped partition: 89 training rows, 30 development rows and 30 reserved rows. It does not reveal the reserved outcomes. Iris and its measurements have already appeared in the course, so this is a real-data debugging exercise with repeated development evidence, not a previously unseen independent benchmark.

Trace the labels before blaming the network

The rotated-target fault keeps each input row in place but takes its training target from 31 rows later, wrapping around the training partition. No development target enters training. Inspect both row IDs, the original target and the value actually supplied to the optimizer.

A lower objective can mean successful fitting of the wrong input–target pairs. The lab therefore separates its normalized fitting objective from training MSE against the original row targets, and compares development predictions against correctly paired targets. Some petal lengths repeat, so not every wrong ID changes the numeric target; checking only a few values can miss the bug.

Capture an aligned run after 100 updates. Reset with rotated targets, keeping rate, scaling and dropout fixed, then run the same 100 updates. Record the pairing mechanism, both kinds of training error and development error. Repair alignment and repeat from the same initialization. This evidence is stronger than changing several controls until a curve improves.

Use training statistics consistently

For each feature, standardization subtracts its training mean and divides by its training population standard deviation. With a measurement of 6 cm, mean 5 cm and standard deviation 0.5 cm, the standardized input is (6 − 5) / 0.5 = 2, which is dimensionless. The same fitted statistics transform development inputs; they are not recomputed there.

The target is also standardized for training, then predictions are converted back to centimetres: prediction in cm = standardized prediction × training target scale + training target mean. The displayed MSE averages squared errors in cm². The fitting objective uses standardized targets and is not on that same scale.

The raw-input fault bypasses feature standardization for the fixed initialization and rate. Trace raw inputs, used inputs, weighted sums and tanh outputs. A count of activations with absolute value greater than 0.99 helps locate saturation, but it does not by itself prove a defect. Raw units are not universally invalid; the comparison is about this model and optimization recipe. Change one preprocessing choice while holding the starting model and training budget fixed.

Separate learning from evaluation behavior

With training dropout set to 50%, each retained hidden activation is doubled and each dropped one becomes zero. Training uses reproducible masks based on the absolute update count. Gradients use the same pre-update parameters and the derivative of the raw tanh activation, multiplied by its mask.

A deterministic evaluation disables dropout. The deliberately faulty evaluation probe leaves dropout enabled and changes its mask seed each time you press Repeat evaluation-mode probe. It does not update weights, optimizer state or history. Compare its changing score with the unchanged canonical development score. With dropout off, these two modes coincide; that case cannot expose the bug.

The faulty probe is not an uncertainty estimate. It neither calibrates confidence nor implements a validated Bayesian prediction method. Its purpose is to reveal inconsistent evaluation behavior.

Compare a baseline and keep the limits visible

The training-mean baseline always predicts the training mean petal length. It uses no development information to fit. Compare its development MSE with the neural model on the same rows, while recognizing that stronger classical baselines may do better than this simple reference.

The full curves use original-target, dropout-free MSE on shared fixed axes. Values above the plotting limit are explicitly counted and remain available numerically. Training caps at 500 updates, with one direct CPU step or at most 25 worker steps per action. Cancellation, timeout, hiding the tab or moving the experiment offscreen retains the preceding accepted batch while the panel remains mounted. Closing the panel or navigating away cancels work and discards temporary runs; download first. An unfinished batch is not accepted evidence.

Capture up to six temporary runs and download them before closing the panel. Restore a capture to recover its complete model, settings, update count and mask sequence. Configuration changes restart from the original initialization. Add a symptom–mechanism–repair comparison to the existing incident report; these downloads do not replace its established synthetic evidence requirements or award new mastery.

For implementation practice, reproduce train-only statistics, check fixed-mask gradients, test that changing development labels cannot change an update, and verify the input/target join. For a decision brief, explain why a declining fit objective or fluctuating evaluation score cannot support a deployment claim. After practice, try the four supplemental debugging transfer cases: match targets by identity, distinguish training and evaluation behavior, check batch averaging, and diagnose local saturation. Submit each unfamiliar case before reading feedback. First attempts and practice retries remain separate; these cases do not change existing path requirements. Human artifact review and learner pilots remain pending.