Choose complexity using development evidence.

Try a longer run before assuming it is better

Imagine a classifier that has learned the broad pattern in its examples. More training may help it discover useful details, or it may fit accidental label errors. The training loss alone cannot distinguish these outcomes. Train a baseline, save its checkpoint, and predict what another 500 epochs will do to validation loss.

A checkpoint is a saved set of model parameters and settings at a particular training time. A development set, called validation in this project, helps choose settings and checkpoints. A test set evaluates the choice after it is frozen. These roles differ even when all three contain the same kinds of examples.

The shared workbench uses synthetic two-coordinate data: the clean class depends on whether the coordinates have opposite signs, with an 8% label-flip probability in each split. The actual finite number of flipped labels need not be 8%. There are 64 original training points, 48 validation points and 80 test points. This teaches model selection; it is not a real deployment study.

Capacity: what shapes can the model learn?

Start with no hidden layers. The model can draw a straight classification boundary, which cannot capture both opposite-sign regions at once. One or two hidden layers can represent more flexible boundaries. Increasing capacity can reduce this mismatch, but also provides more ways to fit incidental patterns.

Compare architectures using the same data and declared development criterion. The number of parameters and the decision map help explain the mechanism. A more intricate map is not automatically better. Fixed initialization makes reruns reproducible; changing dimensions changes the model, so this is not a matched-weight intervention across architectures.

L2: charge for large weights during fitting

Consider two weights, 2 and −1, with penalty strength 0.1. Their squared values sum to 5. The additional objective is 0.1 × 5 / 2 = 0.25. If the data loss is 0.4, the training objective is 0.65.

In symbols, J = L + (λ/2) Σᵢ wᵢ². Here J is the objective minimized during training, L is mean cross-entropy over training examples, λ (lambda) is the nonnegative penalty strength, wᵢ is weight i, and Σ means add over all weights. This project does not penalize biases. The derivative of the penalty adds λwᵢ to each weight’s data gradient before a simultaneous update.

For practice, change λ to 0.02 with the same two weights. The penalty becomes 0.05, so J becomes 0.45. A smaller penalty gives fitting errors more influence relative to weight size. Larger penalties can also prevent useful fitting; they are not guaranteed improvements.

The displayed training and validation losses exclude this penalty and use evaluation mode. Comparing a penalized training objective directly with an unpenalized validation loss would mix different quantities.

Dropout: change the training computation

Open Inspect dropout: training versus evaluation. Each hidden unit has a raw activation, a sampled multiplier, and the value sent onward. At dropout probability 0.5, an activation of 0.6 is either removed (0) or doubled (1.2). Each event has probability 0.5, so its average is 0.5 × 0 + 0.5 × 1.2 = 0.6.

For dropout probability p, retained activations are divided by 1 − p. A unit is removed with probability p and retained with probability 1 − p. This is inverted dropout: scaling occurs during training, so evaluation uses the unmasked activations directly. Here p is restricted to 0, 0.25 or 0.5. Inputs and the output are never dropped.

Predict the value sent by a retained activation of 0.6 at p = 0.25. It is 0.6 / 0.75 = 0.8. A dropped unit still sends zero. The workbench samples a separate mask for each training example and hidden unit. Seeds depend on the absolute epoch, allowing a restored checkpoint to continue its sequence.

The evaluation prediction of a nonlinear network need not equal its exact average prediction over all masks. In a second hidden layer, upstream masking can change the raw activation before that layer’s mask is applied. Inspect the complete chain rather than assuming only the final output is scaled.

Dropout discourages reliance on particular combinations of hidden units. Whether that helps this small dataset is an experimental question. With no hidden layer, this hidden-dropout control has no effect. The decision map, trace and saved learning curves always disable dropout, even on training examples; otherwise their stochastic fluctuations would obscure comparisons.

Manual early stopping with saved checkpoints

Choose the rule lowest validation loss among checkpoints at epochs 500, 1000 and 1500; earliest wins a tie before running. Keep all other settings unchanged. After each 500-epoch batch, save a candidate. The comparison holds at most three candidates. Use the best checked candidate to restore its model, then freeze it before revealing the test set.

This is manual checkpoint selection, not an automatic patience algorithm. It may miss a better checkpoint between observations. Restoring a candidate also restores its settings, added points and epoch; the visible learning curve then begins at that saved epoch. Continuing training is a new development decision. The schedule lab explores explicit patience and monitoring frequency.

For a worked example, suppose the three validation losses are 0.46, 0.39 and 0.43 while training loss keeps falling. The declared rule chooses epoch 1000. For nearby practice, losses 0.41, 0.41 and 0.45 choose epoch 500 by the tie rule. These supplied numbers illustrate selection; the live classifier may produce different values.

A selected earlier checkpoint does not undo the computation spent trying later checkpoints. Report the full search effort. Repeated choices on 48 validation examples can overfit development evidence too.

Noisy labels: isolate what changed

The additional-noise control flips a deterministic subset of the 64 original training labels. Its probability is 0, 0.15 or 0.30; the interface reports the actual number flipped. Coordinates, validation labels, test labels and learner-added labels do not change. A higher setting contains the lower setting’s flips, making the intervention reproducible.

Some original labels were already flipped when the synthetic data was generated. Flipping one again can restore the clean label, so “30% extra flip probability” is neither the observed fraction changed nor the final fraction of incorrect labels. Compare development performance, inspect errors and describe the intervention precisely.

Changing settings restarts the model from its fixed initialization and retains saved candidates. Record each comparison before removing a candidate to free a slot. Training losses from different label interventions refer to different targets; the common validation set is the shared development comparison.

Apply: defend a model choice

Guided experiment: save three checkpoints, select with the declared rule, and explain one training/validation disagreement using the map and loss curves. Compare one regularization change with a matched schedule. Record settings, results and unsuccessful trials in the project model card.

Implementation practice: independently implement inverted dropout and L2, check gradients using fixed masks, and verify that evaluation is deterministic and restores the original computation when dropout is disabled. Test that label corruption touches only the training partition. The browser implementation is a reference, not evidence that your own code passes; runnable code and independent review are needed for implementation proficiency.

Decision practice: write a recommendation for a hypothetical team choosing a model under a limited training budget. Compare validation loss, complexity, search work and label-quality sensitivity. Decide what further real-data evidence is needed before use. This synthetic exercise does not authorize deployment.

The classifier project shares this saved workbench and provides the model-card export. A report-presence check and the lesson’s multiple-choice practice do not judge reasoning or code quality. Human artifact review and representative learner pilots remain pending.

Freeze once and interpret carefully

Select using development evidence, write your reason, then freeze to reveal test predictions. Resuming tuning preserves known test exposure. A later score on the same test is practice evidence; resetting the weights or importing an earlier backup does not create a fresh test.

Report the frozen result, limitations, noisy labels and small sample size. Regularization can improve a selected experiment without proving a universal rule. Continue to training debugging to investigate failures, or revisit numerical stability when the arithmetic itself is suspect.