Make numerical computation trustworthy.

The formula can be right while its evaluation fails

Three classes receive scores 2, 1 and −1. Softmax turns their relative sizes into positive values summing to one in exact arithmetic. Adding 1000 to all three scores leaves their differences unchanged, so the exact probabilities should stay the same. Try it in the lab: the naive calculation fails while the shifted calculation stays useful.

This is a numerical failure, not evidence that the task suddenly became harder or the target label changed. Repairing the arithmetic should preserve the intended mathematical question.

The workbench uses JavaScript’s binary64 floating-point numbers. It does not simulate a float32 GPU kernel. These examples illustrate how to investigate a numerical issue; the exact failure thresholds depend on the representation and implementation.

Three ways a finite representation can lose information

Overflow occurs when a result is too large for the format. Here, directly exponentiating scores around 1000 produces Infinity. Dividing Infinity by Infinity then produces NaN, meaning the arithmetic does not define a usable represented number. A NaN probability is not zero probability.

Underflow occurs when a tiny nonzero value cannot be represented at the available scale. Exponentiating sufficiently negative numbers can round to zero. A target probability that rounds to zero does not prove the event is mathematically impossible.

Rounding also affects ordinary finite values. A large number has more widely spaced representable neighbors than a small number. Adding a common offset of 10¹⁶ can change the score gaps before softmax even starts. Inspect the exact represented scores and their differences in the workbench. Subtracting the maximum afterward cannot recover distinctions already lost.

Start with a small softmax calculation

A logit is a score before normalization. For score zᵢ, softmax is pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ). The index i identifies the class whose probability is being calculated, j visits all classes, and Σ means add. The function exp(z) is e raised to z, where e is about 2.718.

For scores (2, 1, −1), the largest score is 2. Subtracting it gives (0, −1, −3). Their exponentials are approximately (1, 0.367879, 0.049787), summing to 1.417667. Dividing each by the sum gives approximately (0.705385, 0.259496, 0.035119).

For a nearby practice case, imagine three equal scores. All three exponentials are equal, so each probability is one third. The target label changes the loss and gradient, but not the probabilities produced by these scores.

Softmax normalization is not evidence that the numbers are calibrated real-world confidence. This lab supplies scores directly; it trains no classifier.

Subtract the maximum before exponentiating

Let m = maxⱼ zⱼ, the largest score. The stable calculation uses pᵢ = exp(zᵢ − m) / Σⱼ exp(zⱼ − m). In exact arithmetic, the common factor exp(−m) cancels from numerator and denominator. In computation, every shifted score is at most zero, so the largest exponential is 1 rather than an overflowing number.

The worked animation uses (1002, 1001, 999). Their shifted scores are the same (0, −1, −3), so the example preserves the useful relative differences. The live chart uses the same probability scale for naive and stable results. Invalid results are labelled unavailable rather than plotted as zero-length probability bars.

Max shifting prevents exponential overflow for these finite inputs. It does not promise that every tiny probability remains representable. Very large negative gaps can still underflow, which is why the loss needs its own stable formula.

Keep the loss in the log domain

For target class t, cross-entropy is L = −log(pₜ). Here log is the natural logarithm, the inverse of exp, and t identifies the correct class. If a represented probability has already become zero, evaluating log(0) produces an infinite loss even when the score-based mathematical loss is finite.

Use the scores directly instead: L = (m − zₜ) + log[Σⱼ exp(zⱼ − m)]. Read it as “the target’s gap below the maximum, plus the log of the shifted exponential sum.” The parenthesized gap is computed before adding the log term.

With scores (1000, 0, −1000) and target C, the target is 2000 below the maximum. Its shifted exponential rounds to zero, but its direct loss is represented as 2000 + log(1) = 2000. The true mathematical log term is positive but far below this representation’s resolution. A finite direct loss preserves the dominant error without taking a logarithm of the rounded zero.

The equivalent log-sum-exp expression is logΣexp(z) = m + logΣexp(z − m). It is useful on its own, but calculating that large rounded result and then subtracting zₜ can lose a small remaining loss. At a huge common offset, the workbench lets you compare this subtraction with the direct gap-first formula.

Optional: preserve a very small winning-class loss

For a dominant class with shifted scores (0, −40, −41), the sum is 1 plus a very small tail. Adding that tail to 1 can round back to 1. The lab computes the tail separately and uses log1p(tail), a function designed to calculate log(1 + tail) accurately for small tails.

The dominant probability can still round to exactly 1, while the retained log-domain loss is a small positive number. Similarly, its gradient is computed as minus the sum of the other class probabilities rather than subtracting 1 from a probability already rounded to 1. These are algebraically equivalent expressions with different rounding behavior.

Compare a derivative with an independent approximation

The correct score gradient is gᵢ = pᵢ − 1[i = t]. The bracket is 1 for the target class and 0 otherwise; it is often called a one-hot target. It tells the optimizer to lower the competing scores and raise the target score when taking a negative-gradient step. The lab uses the cancellation-resistant target calculation described above.

A finite difference estimates a derivative by evaluating the loss at nearby inputs. The central estimate is [L(z + h) − L(z − h)] / (2h) for one score at a time. The positive number h is the requested perturbation, and every other score stays fixed.

The checker actually divides by the separation of the represented perturbed scores, reports both displacements and checks whether they are close to the requested h. If either perturbation disappears through rounding, or differs from the request by more than 1%, it labels that coordinate inconclusive. It does not call an unchanged input a successful zero-gradient check.

Use the balanced scores, offset zero, target A and h = 0.0001. Check the correct gradient. Then keep those inputs fixed and try the candidate that omits the target subtraction, followed by the candidate with its sign reversed. The check compares different computations of the same direct shifted loss; it does not differentiate whichever loss display you selected above.

Smaller steps are not always more accurate

A large h samples too broad an interval and can miss local curvature. This is truncation error: the central difference is only an approximation to the derivative at the original point.

A very small h can make the two loss values nearly equal, so subtraction loses meaningful digits. It may even fail to change the represented input at all. This is a different failure from an incorrect analytic gradient.

Compare several perturbation sizes and inspect the errors, not just the pass label. A moderate step may work where both a large and a tiny step fail. A derivative that is extremely small can also be accepted by an absolute tolerance without establishing high relative accuracy.

State the tolerance and its limits

The checker accepts a coordinate when |a − n| ≤ atol + rtol × max(|a|, |n|). Here a is the analytic candidate, n is the numerical estimate, atol is an absolute error allowance and rtol is a relative allowance. Absolute tolerance gives an error floor near zero; relative tolerance scales with the gradient magnitude.

For a = 2 and n = 2.00001, absolute tolerance 0.0000001 and relative tolerance 0.00001 allow approximately 0.0000001 + 0.00001 × 2.00001 = 0.0000201001. The observed difference 0.00001 is within that allowance. This arithmetic explains a pass; it does not prove that the entire program is correct.

The strict, standard and loose settings are explicit teaching choices, not universal recommendations. Increasing tolerance until a failure disappears can conceal a bug. A correct derivative can also fail an overly demanding check because the numerical estimate has its own error. Record the input scale, perturbation, formula and both tolerances with the result.

Build a numerical investigation record

The workbench retains up to twelve temporary checks, including failures and inconclusive cases. Resetting controls keeps that record; reloading clears it without making the examples independent evidence. Export before leaving to retain the calculations and your explanation.

JSON has no numerical NaN or Infinity values. The export therefore uses explicit strings for those failures rather than silently turning them into null. An unresolved numerical derivative is separately represented as null with an inconclusive status.

For guided practice, explain one overflow repair and one failure it cannot repair. For implementation practice, reproduce a stable loss and gradient, test equal scores, extreme gaps and common offsets, and compare finite differences across multiple step sizes. Include cases where probabilities underflow but the loss stays finite.

For a management decision, identify which evaluation claims are unsupported when a pipeline contains NaNs, unreported clipping or loose numerical tolerances. Ask for reproducible checks tied to the actual arithmetic and data scale rather than a green status alone.

These are practice artifacts. Human review and representative learner pilots remain pending. Continue with training diagnosis to connect numerical evidence with model and data failures.