Compare, normalize, combine.

Queries ask, keys match, values contribute

Attention lets a position combine information from other positions. Its query is compared with each key. A key helps determine a match score. The value is the information that will be combined using the resulting weight.

In a trained Transformer, learned projections produce queries, keys, and values from token representations. The lab exposes them directly so you can edit them. Its vectors are hand-authored teaching values; the token labels do not imply a trained understanding of those words.

Work through three positions

Start with q = (1,0), keys (1,0), (0,1), and (1,1), and the same vectors as values. The three dot products are 1, 0, and 1. Divide by √2 because the keys have two coordinates. This scaling helps keep score magnitudes manageable as key dimensions grow.

Softmax exponentiates scores and divides each by their total, producing nonnegative weights that sum to one. The weights are approximately 0.4011, 0.1978, and 0.4011. Then multiply each value vector by its weight and add:

The result is a mixture, not a lookup that retrieves exactly one item. Change a value while keeping queries and keys fixed: the weights stay the same, but the output changes. Attention weights alone are therefore not a complete account of what causes the model’s final prediction. Value vectors, output projections, and later layers matter too.

A mask sets an information boundary

A causal mask lets each position use only itself and earlier positions. At the first position, only the first key is allowed, so its weight is one. At the second position, the third key is blocked. The lab applies the mask before softmax; blocked entries get exactly zero probability.

During next-token training, later text contains the answers the model is supposed to predict. Allowing access to it would create a shortcut unavailable during generation. Masking is not a claim that future words are irrelevant; it prevents the model from seeing information it would not yet have.

More heads, and less repeated work

Different attention heads use different learned projections. They can learn different relationships, though there is no rule that one head must represent grammar and another facts. Their outputs are combined by another learned projection.

During causal generation, the keys and values for earlier positions can be stored in a KV cache. The next token can reuse them. This saves repeated computation but consumes memory. It does not change weights or add knowledge. The next lesson runs a tiny two-head decoder and compares actual cached and uncached calculations.