Train a tiny language model.

From an explanation to a model that learns

The earlier Transformer walkthrough lets you inspect fixed teaching weights. This lesson has a separate model whose parameters actually change. It learns from twelve original synthetic sentences such as “red dog sees red”. Each has a color, an animal, the word sees, the same color again, and an end marker. Four complete color/animal combinations are withheld from training and shown as development data. All use the same grammar and vocabulary; success here does not establish natural-language understanding.

There are two different uncertainties. At the beginning, several colors are possible. After a particular color, training offers three animals and omits one. At the repeated-color position, however, there is a definite dependency: look back to the first color. This makes it possible to learn that dependency while also learning an unhelpfully strong preference against the missing animal combination.

Follow the learned computation

The decoder adds a learned four-number token embedding to a learned four-number position embedding. These numbers form the representation at each input position. The start token and earlier reference words are inputs; each target is shifted one position ahead. A causal mask allows each attention row to use only its current and earlier inputs.

Three learned matrices produce query, key and value vectors. A query asks which available positions to mix; dot products with keys produce scores. The four-dimensional score is divided by the square root of four, which is two. Softmax converts the available scores into nonnegative weights that sum to one. The value vectors are then combined with those weights.

A learned output matrix transforms that mixture. A residual connection adds it to the original representation. A second block expands four numbers to eight, applies tanh, projects back to four and adds another residual. Tanh smoothly bounds each intermediate activation between −1 and +1. A final matrix and bias create ten logits, one for each possible output token; softmax turns them into probabilities. Start is not an output, but end is.

The model stores 254 parameters and has one attention head. Of these, 250 participate in training. Four embedding entries for end are stored but unused: end is a target, and generation stops before feeding it back as an input. Their gradients are zero. It intentionally omits layer normalization and dropout to keep the complete gradient inspectable. It is a simplified causal attention decoder, not a scaled-down claim about every feature of a deployed Transformer. Attention percentages are only part of the computation: neither a large weight nor a plausible diagram proves a token caused the final answer.

Turn a wrong prediction into an update

Imagine the correct next token receives probability one quarter. This lesson measures its cross-entropy loss using the natural logarithm:

Here p is the probability assigned to the correct target and L is its loss. Larger correct-target probability means smaller loss. The previous lesson used logarithms base two and reported bits. Nats use the natural logarithm; multiplying a bit loss by approximately 0.693 converts it to nats. Do not compare the raw numbers as though their units were identical.

We average over all five targets, including end, in each of the twelve training sentences: sixty targets per update. Backpropagation differentiates this mean through the output, feed-forward, residual, attention and embedding computations. A gradient tells us how a small parameter increase changes this objective locally. It is not a promise about the effect of a large step or about development performance.

All parameter gradients are computed from the same current model. If their combined length, called the gradient norm, exceeds one, we scale every gradient down by the same factor. This is global gradient clipping; it limits an update without changing its direction. Then stochastic gradient descent applies the update. Here each step uses the full small training set, so there is no random minibatch selection.

The symbol w is one current parameter, g its mean training gradient, η (eta), the learning rate and c the shared clipping factor between zero and one. If a weight is 0.20, its gradient is −0.10, the rate is 0.30 and no clipping is needed, the new weight is 0.20 − 0.30 × 1 × (−0.10) = 0.23. Use the live parameter inspector to connect this arithmetic to a real update.

Compare with a baseline that cannot look back

The count baseline reads the same twelve training sentences. It counts each token after its immediate predecessor, keeps document boundaries and adds one to each of ten possible output counts before normalizing. After sees, the training data contain each of four colors three times. Each color therefore gets probability (3 + 1) / (12 + 10), or about 18.2%. The denominator also includes the six other possible outputs. Equal counts break ties by token order, so its greedy copy prediction always chooses red.

The decoder can use the earlier color. It still has to learn to do so, and the outcome depends on parameters, optimization and examples. Compare losses on identical targets, then inspect the copy-position accuracy. The count baseline has different smoothing, capacity and computational cost, so this comparison does not isolate the value of attention alone.

Watch for learning and overfitting in the same run

Training loss can fall while development animal loss rises. The held-out animal pairs have no positive training examples for that particular color. A model that becomes certain they never occur can score them poorly even while learning the copying rule. The position-level table exposes this failure rather than hiding it in a mean.

Repeated development feedback is useful for selecting a checkpoint or a next experiment. It is not a secret final test. A next experiment could change the split, gather examples of missing combinations or measure a predeclared different grammar. Record that decision before looking at the new results. The lab itself does not supply independent assessment of language-model implementation or real-text performance.

Teacher-forced loss scores each reference token after its reference prefix. The displayed continuation instead supplies only the first color, animal and sees, then lets the model feed its own prediction into the next step. It stops at end or the five-position input limit. It can be wrong, fail to end or generate an inappropriate token; no grammar constraint repairs it automatically.

Keep a reproducible checkpoint

A checkpoint copies the weights, update count, seed and active learning rate. Because this optimizer has no momentum or adaptive accumulators, those values and the fixed training corpus reproduce the next update. A more complex optimizer would need its own state too. An exported update count is a record, not authenticated proof that training occurred.

Capture an early and a later model, compare both splits and each target position, and restore one to reproduce its output. The model and up to three checkpoints are temporary browser work; export the experiment before leaving. Cancellation preserves the entire pre-request model instead of showing a partially completed batch as a finished experiment.

For guided practice, explain a gradient sign and one generation failure using the trace. For implementation practice, reproduce a forward pass and check a gradient with a small positive and negative parameter perturbation before trusting a training loop. For a decision brief, choose a checkpoint using both useful behavior and the uncovered failure, and specify the additional data and evaluation needed for your use. These activities support later projects; independent review and learner pilots remain pending.

Continue in the language implementation studio to reproduce training, freeze a checkpoint and sampling plan, replay the report and compare adaptation in a local CPU package. Supplied execution and learner implementation are separate; independent project assessment remains pending.