Trace disappearing and exploding gradients.

A long chain multiplies local effects

Imagine changing the input to a sequence of transformations. If each one passes on half of a small change, four transformations pass on 0.5 × 0.5 × 0.5 × 0.5 = 0.0625 of it. If each passes on 1.5 times the change, four pass on 1.5⁴ = 5.0625 times it. Longer chains can shrink or amplify these effects much further.

The worked sequence shows the first calculation backward, one local derivative at a time. The live lab lets you change the chain length, weights, activation, parameter sharing and identity path. These are real derivatives of a small scalar computation, not simulated gradient labels.

First separate sensitivity from error

Output sensitivity asks how much the final prediction changes for a small change in an intermediate state. A loss gradient additionally asks how that prediction change affects the current error. They are related but not interchangeable.

Let the prediction be h, the target be y and the loss be L = ½(h − y)². The symbol L names the loss; the half makes its derivative with respect to h equal to h − y, the prediction error. For an earlier state s, the chain rule gives dL/ds = (h − y) × dh/ds. Read this as “loss sensitivity equals prediction error times output sensitivity.”

For the four-weight 0.5 example with input 1 and target 0, the output is 0.0625, input sensitivity is 0.0625 and input loss gradient is 0.0625 × 0.0625 = 0.00390625. Those two factors come from different questions.

Nearby practice: use weight 1, linear activation, input 1 and target 1. The prediction is correct, so all loss gradients are zero, while output sensitivity remains 1 throughout the plain chain. A zero loss gradient does not by itself prove a vanishing-gradient failure.

What each local connection contributes

Without a residual path, one step computes hᵢ₊₁ = α f(wᵢhᵢ + bᵢ). Here i identifies the connection, hᵢ is its input state, wᵢ is its weight, bᵢ is its bias, f is linear or tanh activation, and α (“alpha”) is the branch scale, either 1 or 0.1 in this lab. All values are dimensionless.

Write zᵢ = wᵢhᵢ + bᵢ for the weighted sum. The local state derivative is Jᵢ = α f′(zᵢ)wᵢ. The prime means the activation’s slope. Linear activation has slope 1. Tanh’s slope is 1 − tanh(zᵢ)², which becomes small near its saturated extremes.

The output’s sensitivity to an earlier state multiplies all the intervening local derivatives. The exact table shows the states, activation slopes and local multipliers so you can find where sensitivity changes. Small values may come from small weights, activation saturation, cancellation through another path, or a combination.

Optional: the backward recurrence and parameter contributions

Let D be the chain length and define Sᵢ = dh_D/dhᵢ. At the output, S_D = 1. Work backward using Sᵢ = Sᵢ₊₁Jᵢ. The actual loss derivative at state i is (h_D − y)Sᵢ.

A weight contribution at connection i is (h_D − y)Sᵢ₊₁ α f′(zᵢ)hᵢ; its bias contribution omits the final hᵢ factor. The calculations use the same preceding states and weights before any parameter changes.

Depth and time can share the same chain rule

With parameter sharing off, each depth has its own weight and bias. They begin at equal values for a controlled experiment, but can receive different updates. With sharing on, one weight and one bias are reused at every step, as in a small recurrent computation through time. This is a scalar teaching recurrence, not a complete recurrent language model.

A shared parameter affects the result at every place it appears. Its gradient therefore adds all of its contributions, rather than using only the last occurrence or updating the weight partway through the forward pass.

For four linear steps, shared weight 0.5, zero bias, input 1 and target 0, the shared weight gradient is 4 × 0.5⁷ = 0.03125. The derivative contains all four uses. Switching sharing can leave the initial forward prediction unchanged while changing the parameter-gradient vector and subsequent training.

Residual paths change the local derivative

An identity residual path adds the input state directly: hᵢ₊₁ = hᵢ + α f(wᵢhᵢ + bᵢ). Its local derivative becomes Jᵢ = 1 + α f′(zᵢ)wᵢ. The additional 1 is the identity path’s contribution.

With a zero linear branch, the plain chain loses its input dependence, while the residual chain passes its input through unchanged. That is a concrete way an identity path can preserve sensitivity. But it does not guarantee stable gradients: with α = 1 and branch weight −1, the derivative becomes 1 − 1 = 0. With branch weight 0.5 it becomes 1.5, which can amplify through depth.

The comparison table holds the numerical parameters, branch scale and input fixed while adding or removing the identity path. For tanh, the changed intermediate states also change activation slopes. This is an architectural counterfactual, not two separately trained systems or an equal-performance comparison.

Read the magnitude plot honestly

The vertical axis shows absolute magnitude on a base-ten logarithmic scale. A factor of ten takes the same vertical distance anywhere on that axis. This is useful when a derivative changes by many orders of magnitude; it is not a linear scale and is not a loss curve.

Positive values use circles and negative values use squares. Exact zero appears in a separate strip because log(0) is undefined. Values outside the fixed plotted range are counted and omitted rather than assigned a misleading boundary value. The exact trace retains their sign and magnitude.

Choose a state and compare its value, output sensitivity and actual loss gradient. Large forward states and large backward derivatives are related through the computation, but they are different measurements.

Global clipping scales the parameter vector after accumulation

Gradient clipping controls the size of a proposed update. It does not change the already-calculated forward trace, repair a wrong derivative, or create information where a gradient is zero.

The lab uses the global Euclidean norm of the complete parameter-gradient vector, including biases and all shared-use sums. For vector g = (g₁, …, gₖ), its norm is ‖g‖ = √(g₁² + … + gₖ²). The index k counts the actual independent parameters: two when shared, or two per connection otherwise.

For a positive limit C, the applied gradient is g_applied = g × min(1, C/‖g‖). A zero gradient stays zero without division. The Off control disables clipping; it does not mean a zero-length update.

For g = (3, 4), the norm is 5. With limit 1, multiply every component by 0.2 to obtain (0.6, 0.8), whose norm is 1. This preserves direction and differs from clipping each coordinate separately. At learning rate 0.01, the update vector is (−0.006, −0.008) and its norm is 0.01.

The live proposal table shows raw gradients, applied gradients, parameter deltas and proposed parameters together. With clipping enabled, update norm is at most learning rate × limit. That is a size bound, not a guarantee that the loss decreases. Already-small gradients remain small; clipping cannot repair vanishing sensitivity.

Compare from a complete checkpoint

Try a sixteen-step linear chain with initial weight 1.5, input 1 and target 0. Save the checkpoint. Inspect the raw proposal at rate 0.05 with clipping off, then compare limit 1 from the same checkpoint. A proposal outside the lab’s numerical bounds is rejected and leaves the preceding parameters intact. Restore or restart before trying another training branch after a stopped run.

Up to five small CPU updates run per click, with fifty accepted updates per run. There is no background training. Saved comparisons include every accepted parameter checkpoint and each update’s rate and clipping setting. These temporary records can be exported; they are not authenticated training history or independent evaluation evidence.

For guided practice, explain a shrinking, growing and exactly-zero derivative using the local trace. For implementation practice, reproduce shared-parameter accumulation, verify finite differences and test the (3, 4) clipping example before implementing larger chains. For a management decision, explain why a smaller update or a stable loss plot is insufficient evidence that the model retained useful long-range information.

Human artifact review and representative learner pilots remain pending. Continue to training diagnosis to combine gradient evidence with data and update checks.