Choose how to sample.

The same model can make different choices

Suppose a trained decoder gives the next token probabilities of 60%, 30% and smaller alternatives totaling 10%. Greedy decoding always takes the largest probability. Sampling uses a random draw, so the 30% alternative can sometimes appear. Neither action changes the model’s learned weights.

Our model writes only a tiny synthetic grammar: color, animal, sees, the same color, then end. The checkpoint was trained with actual gradients in the preceding lesson. Supplied checkpoints at 0, 100, 500 and 1,000 updates let you compare what sampling can do before and after learning. You can also import a decoder experiment; only validated numeric parameters are used, and scores are recomputed. An imported claim of training is not authenticated evidence.

Follow a draw through probability intervals

Imagine placing the alternatives on a number line from zero to one. With probabilities 0.60, 0.30 and 0.10, the first token owns the interval from 0 to 0.60; the second owns 0.60 to 0.90; the remaining alternatives divide the rest. A uniform draw of 0.72 selects the second token. Intervals include their left boundary and exclude their right boundary, so a draw of exactly 0.60 also selects the second.

The lab displays the actual prefix, draw and cumulative intervals for each chosen token. Its random generator is deterministic: a seed, sample number and position identify each draw. The same settings and checkpoint reproduce the same outputs. If one sentence stops early, it does not consume draws belonging to another sentence.

Using the same seed helps compare settings, but does not keep every later context identical. If a changed setting chooses a different first token, the model sees a different prefix at the second position. Read the trace before attributing every later difference to the filtering step alone.

Temperature changes relative confidence

A logit is a model score before conversion to probabilities. Temperature divides each score before softmax. A lower positive temperature sharpens differences; a higher one flattens them. At temperature one, the probabilities are unchanged. Greedy decoding in this lab ignores temperature and random draws entirely.

For a concrete example, imagine two alternatives with probabilities 0.80 and 0.20. At temperature 0.5, their unnormalized masses are squared: 0.64 and 0.04. Renormalizing gives approximately 94.1% and 5.9%. The favorite becomes more likely, but no additional evidence made it more correct.

Connect temperature to the softmax formula

$$p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}$$

Here zᵢ is the logit for token i, T is a positive temperature, and pᵢ(T) is its new probability. The denominator adds the exponential masses for all output alternatives j so they sum to one. Equivalently, raise the original probabilities to the power 1/T and renormalize. The implementation subtracts the largest logit before exponentiating to avoid numerical overflow; that common shift does not change the probabilities.

Top-k keeps a count; top-p keeps enough mass

Top-k retains the k highest-probability tokens and gives the others zero sampling probability. Divide the retained probabilities by their sum before drawing. With k equal to one, sampling has only one possible choice and matches greedy decoding under the same tie rule. Keeping all ten outputs in this lab matches unfiltered sampling.

Top-p, also called nucleus sampling, sorts tokens by probability and retains the smallest leading set whose total reaches a chosen threshold p. It includes the token that crosses the threshold. For 0.60, 0.30 and smaller alternatives totaling 0.10, a threshold of 0.80 keeps the first two: one alone is insufficient, while two total 0.90.

The second retained token receives 1/3. Removed tokens receive zero. Top-p does not retain 80% of token types, and it does not trim the crossing token to force exactly 80% mass. A threshold of one retains every output. Equal-probability ties in this lab follow ascending token ID.

Here temperature is applied before either filter, and you select one filter at a time. Temperature can therefore change how many tokens a top-p threshold keeps. Inspect the original, tempered and final columns instead of treating the controls as interchangeable.

Separate useful diversity from varied errors

Each configuration generates sixteen complete attempts from the start marker. The lab checks grammar validity independently of the model’s scores. A sentence must contain a color, an animal, sees, the same color again and end. Reaching end is reported separately: a one-token end output terminates but is not a valid sentence.

The number of distinct strings measures one kind of diversity. Counting distinct valid strings asks a more useful question for this task. A hotter distribution may create more unique outputs and more mistakes simultaneously. Adjacent repeated tokens provide another failure diagnostic; the grammar’s intended non-adjacent color repetition is not counted as that failure.

An exact match to a training sentence is reported as overlap, not proof of memorization. This grammar has only sixteen valid sentences, all enumerated across the training and development partitions. Even a model that learned the rule can produce an exact training match. To investigate memorization more strongly, you would need a different protocol with appropriate novel cases and source-specific evidence. This widget cannot establish that conclusion from matching strings alone.

Make a matched comparison

Hold the checkpoint and seed fixed while changing one decoding control. Save up to three comparisons and inspect individual failures alongside the summary. Repeat with another seed to see whether the impression depends on the selected draws. These sixteen outputs are a small, public development sample, not a guarantee of deployment behavior or an independent learner assessment.

For guided practice, explain one token’s final probability and locate its draw interval. For implementation practice, reproduce top-k and top-p, including ties, the crossing token and zero-probability entries; verify that changing sampling never changes weights. For a decision brief, state whether the task prioritizes predictable valid output or a range of valid alternatives, select settings using both counts, and name the further task-specific checks needed.

Sampling cannot supply knowledge absent from the model or reliably repair a dependency it has not learned. If a weak checkpoint repeatedly copies the wrong color, returning to data, training and evaluation may be more useful than searching indefinitely for a flattering seed.