Learn from small batches.

One example gives one signal

A model can make an error on every training example. At the same weight and bias, different examples can ask for different corrections. A batch is the set of examples whose gradients are combined before one parameter update.

This lesson uses a single linear neuron, ŷ = w × x + b. The input x, target y, weight w and bias b are dimensionless teaching values. ŷ, read “y hat,” is the prediction. There are sixteen fixed training examples and sixteen distinct input coordinates for monitoring. Their targets follow a gently curved authored pattern; a straight-line neuron cannot reproduce every target exactly.

Batch size 1 makes a stochastic update from one example. Batch size 4 averages a mini-batch. Batch size 16 uses the whole training set. These are the same model and dataset, with different amounts of evidence per update.

Read the arrows before moving

For squared error, one example contributes weight gradient 2 × (ŷ − y) × x and bias gradient 2 × (ŷ − y). A gradient describes how loss increases as a parameter increases. Gradient descent subtracts it to move locally toward lower loss.

The plot’s horizontal coordinate is the weight gradient; its vertical coordinate is the bias gradient. Every arrow starts at zero and uses the same current parameter checkpoint. Selected example arrows are emphasized; the table also marks selection in text. The full-data average is dashed and the selected batch average is solid. These arrows are gradients, not predicted outputs or the parameter trajectory.

The axes stay at −12 to 12. An out-of-view endpoint is clipped and counted, with its exact value available in the table. The display does not rescale a small gradient to make it appear large. The walkthrough is a live explanation of the next proposed step; it does not train the model. Animation starts only when requested, and numbered steps work without motion.

Average first, update together

Consider a separate worked example at w = 1 and b = 0. For input 1 and target 3, prediction is 1 and error is −2: the gradient is (−4, −4). For input 2 and target 1, prediction is 2 and error is 1: the gradient is (4, 2).

The two-example mean gradient is ((−4 + 4) / 2, (−4 + 2) / 2) = (0, −1). With learning rate 0.1, subtract 0.1 times that vector: new weight is 1 and new bias is 0.1. Both gradients were calculated before either parameter changed.

Nearby practice: gradients (2, −2) and (0, 4) average to (1, 1). From parameters (1, 0), a rate of 0.1 produces (0.9, −0.1). Updating immediately after the first example and then computing the second gradient at new parameters is a different algorithm.

For a batch of B examples, the rule is g = (g₁ + … + gᴮ) / B, followed by θ_new = θ_old − η × g. Here B is the number of examples, gᵢ is one example’s gradient vector at the old parameters, θ is the two-parameter vector (w, b), and η (“eta”) is the learning rate. Read it as “average the signals, then subtract a scaled average.” Using a sum instead of a mean would change the effective step size when B changes.

An epoch is a complete pass

An epoch visits every training example once. The lab’s sizes divide sixteen exactly, so there is no partial final batch. In shuffled mode, a seeded permutation changes the order each epoch; every row is still visited once, without replacement. Ordered mode follows increasing input.

This random reshuffling is not independent sampling with replacement. Later batches in an epoch depend on which examples have already been used. Reusing a seed reproduces an order; it does not create new data. Resetting a setting returns to the same zero parameters so a comparison does not inherit extra fitting from an earlier run.

At a fixed checkpoint, averaging all equally sized batch gradients across an epoch recovers the full-data gradient, up to numerical rounding. During actual learning the parameters change between batches, so an epoch of small updates is generally not equivalent to one large update.

Measure variability without promising a winner

The workbench computes each batch’s gradient at the current fixed checkpoint and reports component-wise spread around the full-data mean. For a component, square each batch’s difference from the full mean and average those squares. Zero means the measured batch signals agree in that component. A full batch has only one signal, so its spread is zero.

A smaller batch often has noisier signals, but the outcome also depends on data, ordering, rate and the current parameters. A gradient that differs from the full-data direction is not automatically a bug. This observed spread is not a statistical confidence interval or proof of better generalization.

Optional: why changing batch size can require retuning

A noisy gradient, a learning rate and the curvature of the loss interact. Reducing batch size changes the sequence and variability of updates, not just their cost. Rules that scale learning rate with batch size depend on assumptions and regimes; this lesson does not apply a universal scaling rule. Compare declared settings from the same initialization and retain the results of unsuccessful choices too.

Equal updates and equal data are different experiments

With 64 examples processed, batch sizes 1, 4 and 16 make 64, 16 and 4 updates. With 16 updates, those sizes process 16, 64 and 256 examples. The comparison controls expose both designs and report actual work if a numerical bound stops a run early.

Neither design guarantees equal wall-clock cost. Workers, diagnostics, rendering and model shape add overhead; larger batches may use hardware differently. The example counter counts fitting uses, including repeated passes through the same data. It does not count unique new observations or the extra forward/gradient calculations used to explain and monitor the experiment.

The small model has a CPU single-update path and bounded worker runs. It does not demonstrate GPU throughput or production-scale batch behavior. Hidden/offscreen/navigation cleanup, cancellation and timeout retain the preceding accepted result. The optional walkthrough has its own pause/manual controls and respects reduced motion.

Judge the comparison on its stated evidence

The displayed training and monitoring losses are mean squared error. Monitoring inputs never supply training gradients, but repeatedly using their losses to choose batch size or rate makes them development evidence. They are not a newly independent test every time you click Compare.

Export your settings, counts, checkpoints and explanation. For implementation practice, reproduce per-example gradients, simultaneous averaging and epoch accounting. For a decision brief, explain which resource budget your comparison answers and what additional runtime and task evidence you would need. Neither an attractive curve nor a filled note establishes independent implementation or deployment readiness. Learner pilots and human review remain pending.