Initialize and normalize.

Inspect what one layer hands to the next

A hidden unit first combines its inputs with weights and a bias. Its activation is the value it passes onward after a nonlinear function. If every unit produces the same value, the layer may contain many copies of one feature. If values are tiny or saturated, useful changes can also become hard to learn.

Start with the two-layer Xavier network. The plot shows actual values from eight examples and four units per layer. Change the displayed quantity from activations to gradients. Read the numerical summaries before choosing a different initialization. Keep the seed, activation, depth and update budget fixed when comparing a single intervention.

The model predicts a dimensionless target from two inputs: y = 0.6x₁ − 0.4x₂ + 0.2x₁x₂. Here x₁ and x₂ are the two input coordinates and y is the target. Eight generated examples supply training gradients; eight different input pairs provide repeated development checks. This small synthetic experiment is useful for inspecting mechanisms, not for ranking methods on real tasks.

Identical units can remain identical

Choose identical weights and inspect a layer. Each hidden unit begins with the same incoming weights and bias. Its outgoing connections are also initialized symmetrically. The units receive identical signals, produce identical activations and receive identical updates. Repeating training does not automatically give them different roles.

With all-zero weights, the situation is more restrictive. The zero output weights block the initial loss gradient to the hidden layers. The output bias can still change, so a small loss improvement does not prove that hidden features are learning. Inspect the internal values rather than relying only on the final loss.

Seeded random weights break this symmetry reproducibly. A seed is the starting number for a deterministic pseudorandom sequence. The same settings and seed reproduce the same starting network. Random starting weights are not learned knowledge, and a different seed is not independent evaluation evidence.

Scale matters as well as difference

Switch to four tanh layers with small initialization. Each layer mixes already-small signals with more small weights. The last-layer activations can become much smaller than the first layer’s. Now switch to large initialization. Many tanh activations approach −1 or +1, where changing the weighted sum changes the activation only slightly.

The lab labels |activation| > 0.99 as tanh saturation. This is a diagnostic threshold, not a universal rule. For ReLU, the output is the weighted sum when positive and zero otherwise. The lab counts zero outputs as inactive; an inactive output in this cohort does not establish that the unit will never learn on another input.

The gradient plot shows the loss derivative at the weighted sums, including the effect of normalization and activation. Its fixed horizontal scale differs from the activation scale and is labelled separately. Values outside a plot’s range are counted and omitted; they are not moved to the edge. Tables retain their actual magnitude, including very small values in scientific notation.

A small numerical example before the recipes

Without normalization, a unit computes z = Σⱼ wⱼxⱼ + b, then an activation a = f(z). The symbol z is the weighted sum, wⱼ is the weight for input xⱼ, b is the bias, and f is the chosen activation function. The index j means “for each incoming connection.” All quantities in this lab are dimensionless.

For inputs (2, −1), weights (0.3, 0.1) and bias 0, the weighted sum is 0.3 × 2 + 0.1 × (−1) = 0.5. Tanh turns 0.5 into approximately 0.462. Multiplying both weights by ten gives z = 5 and tanh approximately 0.99991, much closer to saturation. This separate worked example is not the live network’s current state.

An initialization recipe chooses a distribution whose scale depends on how many connections meet a unit. Fan-in counts incoming connections; fan-out counts outgoing units. Xavier uniform samples between −√(6/(fan-in + fan-out)) and its positive counterpart. He uniform samples between −√(6/fan-in) and its positive counterpart. “Uniform” means equally likely positions across that interval before finite pseudorandom sampling.

For a four-input/four-output matrix, Xavier’s limit is √(6/8), about 0.866. He’s limit is √(6/4), about 1.225. These are interval limits, not the variance or the actual weight of every connection. The small and large choices multiply Xavier weights by 0.01 and 5. This teaching lab applies the selected recipe to the output matrix too; production systems may choose a separate output initialization.

Optional: why these scales are used

Under simplifying independence and scale assumptions, a weighted sum’s variance grows with the number of incoming terms and the variance of their weights. A uniform variable on [−r, r] has variance r²/3. Xavier’s variance is therefore 2/(fan-in + fan-out), and He’s is 2/fan-in. The larger He scale compensates approximately for the loss of signal through rectification under its assumptions. Xavier and He are useful starting rules, not guarantees about the statistics of every trained layer or tiny sampled matrix.

Normalization asks which values belong together

Normalizing a group means subtracting its mean and dividing by its spread. The result can then be multiplied by a learned scale and shifted by a learned offset. In this network, normalization occurs between each hidden layer’s weighted sum and its activation.

Consider two examples with two features each: (1, 10) and (3, 14). For batch normalization, the first feature’s group is (1, 3), with mean 2 and population variance [(1 − 2)² + (3 − 2)²]/2 = 1. The second feature’s group is (10, 14), with mean 12 and variance 4. Each feature uses its column across examples.

For layer normalization, the first example’s group is (1, 10), with mean 5.5 and variance 20.25. The second example has its own mean 8.5 and variance 30.25. Each example uses its row across features. This is a difference in grouping, not just two names for the same calculation.

In the worked widget, keep (1, 10) fixed and change the other example’s first feature from 3 to −3. Batch training normalization changes the fixed example’s first normalized value from approximately −1 to +1. Layer normalization leaves that example unchanged because its own features did not change.

Read the formula as four operations

For a value z in a selected group, the normalized value is ẑ = (z − μ)/√(v + ε), followed by u = γẑ + β. The hat on z means “normalized.” The group mean μ (“mu”) is the average, v is the average squared distance from that mean, and ε (“epsilon”) is 0.00001 here. The learned scale γ (“gamma”) starts at 1 and the learned offset β (“beta”) starts at 0.

Read it as: subtract the group’s mean, divide by the square root of its variance plus a small stabilizer, multiply by the learned scale, then add the learned offset. The stabilizer avoids division by zero for a constant group. It also means the normalized variance is not exactly one. A constant group becomes zeros before scale/offset rather than producing an undefined division.

For the first batch-normalized value above, the arithmetic is (1 − 2)/√(1 + 0.00001) ≈ −0.999995. In the live four-unit network, each unit has its own learned scale and offset. With normalization set to none, the entire normalization/affine stage is bypassed.

Training and evaluation are different batch-normalization modes

During training, batch normalization uses the current cohort’s mean and variance. Its gradients include how changing an input changes those statistics; they are not treated as fixed constants. That couples examples within the batch.

During evaluation, this lab uses stored running means and variances. They begin at mean 0 and variance 1. Every accepted training step updates them using new stored statistic = 0.9 × old + 0.1 × current training statistic. For a preceding mean 0 and a current batch mean 2, the new stored mean is 0.2. For a preceding variance 1 and current population variance 4, the new stored variance is 0.9 × 1 + 0.1 × 4 = 1.3.

These statistics come from the forward pass before that step updates its weights. This lab explicitly uses population variance in its running update; do not assume every framework uses that convention. A complete model checkpoint must include running statistics as well as weights, biases and learned normalization scales/offsets.

Switching the inspection mode does not update anything. With batch normalization, predictions can differ between modes even at identical weights. Early in training, the stored moments can still be close to their defaults. Layer normalization uses the current example’s features in both modes and has no running cohort moments to estimate.

The worked widget supplies fixed illustrative batch statistics for its evaluation mode. Changing another example then leaves the fixed example’s output unchanged. Those supplied statistics are separate from the trained network above.

Keep inspection and learning separate

Every training button uses only the eight training inputs in training mode, no matter which examples you are currently inspecting. Development inputs in training-mode inspection are an explicit diagnostic counterexample. They never fit weights or update stored moments. The displayed development score always uses evaluation mode.

Save a checkpoint, take ten updates, inspect the stored statistics and restore it. Restore recovers the full model state and rate history. Capture a comparison before restarting with a different initialization or normalization. The six temporary captures hold complete model states, not just percentages. Exports include the per-update rate sequence, but they do not authenticate a training history or establish independent learning proficiency.

Training uses actual simultaneous gradient updates. A numerical bound stops oversized updates and retains the preceding weights and statistics. Runs cap at 100 updates, with ten per worker batch and a direct CPU single-step option. Measured worker time excludes rendering; diagnostic recomputation also has a cost.

Apply the evidence

For guided practice, compare identical versus seeded random initialization and explain what changes in the hidden-unit traces. Then use the worked matrix to justify why batch and layer normalization respond differently to another example.

For implementation practice, reproduce a normalization forward pass and its gradients. Check a small numerical example and finite differences, including learned scale/offset and batch evaluation mode. Verify that a checkpoint round trip preserves evaluation predictions and that inspecting development inputs leaves running statistics unchanged.

For a management decision, explain why “the weights were saved” may be insufficient to reproduce a deployed prediction. Use a train/evaluation discrepancy to propose a specific diagnostic check before funding more training or declaring a model improvement.

These activities are practice. Human artifact review and representative learner pilots remain pending. Continue with training diagnosis to connect these internal traces to a broader fault investigation.