Connect instruction tuning, preferences and verified rewards.

Three different answers to “better according to what?”

A model can learn to imitate an example, favor one answer over another, or seek a high reward. These goals overlap only when their supervision supports the same intended behavior. None automatically makes the examples, preferences or reward correct.

Start with the lab’s prompt: What is 2 + 2? Reply with only the integer. Response A is “4”. Response B is “5”. Response C mentions “4” but ends with “5”; D declines to answer. A weak verifier searches for the character “4”, accepting both A and C. The exact task contract accepts only trimmed text equal to “4”. A mathematically equivalent string such as “4.0” is rejected by this deliberately narrow format contract. Its exactness is local to that specification, not a claim to verify all mathematical reasoning.

The lab starts three independent models with identical probabilities over these four complete responses. Each has four trainable logits: numbers converted to probabilities by softmax. It enumerates every response and computes exact gradients. It does not train a token-generating language model, discover new answers, use a sampled rollout or establish general arithmetic ability. This small setting makes the effect of each objective inspectable.

Supervised instruction tuning: imitate the target

In supervised fine-tuning, a prompt comes with a desired response. Real instruction-tuning examples contain instructions and response tokens; this lab treats each complete response as one action. Its loss is the negative log probability of the demonstrated response. If A is the demonstration and its probability is 0.10, the loss is about 2.303 nats. Increasing A’s probability to 0.50 reduces that loss to about 0.693.

For each logit, the gradient is its response probability minus one if it is the demonstration, otherwise minus zero. At starting probabilities [0.10, 0.20, 0.60, 0.10], the gradient is [−0.90, 0.20, 0.60, 0.10]. Subtracting 0.3 times that gradient increases A’s logit by 0.27 and lowers the other three. Softmax then renormalizes the probabilities.

Change the demonstration to C. The same mathematically correct optimization now learns to imitate a bad answer. The objective supplies no independent check of the demonstration’s factual or formatting quality. Supervised learning is useful because examples can teach desired behavior, not because imitation proves that behavior is desirable.

DPO-style learning: change relative odds against a reference

A preference pair identifies a preferred response and a rejected one for the same prompt. Direct Preference Optimization uses a classification-style loss built from their probability ratios relative to a fixed reference model. In this lab, p is the current probability distribution and q is the fixed starting reference. Subscripts w and l mean preferred and rejected; they do not mean independently verified correct and incorrect.

The margin m measures how much the preferred-to-rejected odds have improved relative to the reference. Beta, a positive chosen coefficient, scales that margin. At the starting reference, both ratios are one, so both logarithms are zero and the loss is ln(2), about 0.693. A lower pair loss encourages larger relative odds for the preferred response. The fixed reference is never updated alongside the candidate.

With beta one, the initial preferred-logit gradient is −0.5 and the rejected-logit gradient is +0.5. A learning rate of 0.3 changes them by +0.15 and −0.15. In this particular independent-logit model, other logits do not change, though their probabilities can change through normalization. In a language network, shared parameters can affect many other outputs. This illustration does not imply only two outputs change in real fine-tuning.

The lab implements this pair objective directly and does not fit a separate reward model. It also does not run an RL loop. Beta here is not interchangeable with a step size or with the explicit KL coefficient in the next experiment. Changing a coefficient and comparing the same number of updates is a controlled numerical comparison, not equal compute or an assurance of equal convergence. The DPO paper provides the full method and derivation; this finite-response experiment isolates its pair-loss arithmetic.

Verified rewards: optimize what the checker returns

For the reward method, a verifier assigns a reward r to each response. Expected reward is the sum of each response’s probability times its reward. If A has probability 0.10 and C has probability 0.60, the weak verifier’s expected reward is 0.70 even though exact task success is only 0.10.

The optional penalty discourages departing from the reference distribution. Its measure is KL divergence, the probability-weighted log ratio between the current and reference probabilities. All reference probabilities here are positive, so these ratios are defined.

The index i runs over the four responses. Lambda is the selected nonnegative penalty strength. Read this as “minimize negative expected reward plus a cost for changing the distribution.” Minimizing negative reward increases reward. Setting lambda to zero removes the reference penalty. Setting it to one trades reward gain against divergence measured in nats. That tradeoff is chosen by the designer; the terms do not have naturally equivalent meanings.

This is a one-step finite-action reward problem whose expectation can be computed exactly. The lab differentiates the whole sum. Large response spaces usually require sampling and additional training machinery; this is not PPO, GRPO, an implementation of a research-scale RLVR system or a claim that an exact gradient is available for arbitrary generated text. No model-based reward predictor is trained here: the verifier is programmed.

When both A and C receive reward one, the objective has no reason to prefer A for correctness. Starting with much greater probability on C makes exploitation visible. Changing to the exact verifier supplies a different learning signal. The separate exact task-success column remains fixed under both choices so that changing the proxy cannot redefine success.

A reference is an anchor, not a guarantee

The misleading reference places 60% probability on C. Penalizing departures from that reference can retain an undesirable behavior. A stronger penalty may reduce the change in probabilities while also limiting exact task success. With a positive penalty, even the best distribution for the stated objective generally need not put all its mass on correct answers.

For an optional mathematical check, the optimum of this finite regularized reward objective has probabilities proportional to the reference probabilities multiplied by exp(reward / lambda). Normalize those four numbers to sum to one. This relation requires positive lambda, fixed rewards and the ability to choose any distribution over the finite action set. It does not describe the DPO training trajectory or guarantee that a constrained language model reaches its optimum.

Compare outcomes without confusing losses

Each model’s own loss has a different definition. A smaller number in one method’s loss column does not establish that it outperforms another. Compare the common exact task-success probability, exploit probability, declared verifier reward and reference divergence. These are distribution-level quantities, not measured success rates from independent samples.

For guided practice, predict one logit update and reproduce it with the inspector. For implementation practice, implement the three scalar objectives, compare analytic gradients with finite differences, and verify that the reference stays fixed. Include the weak-verifier counterexample and a corrupted demonstration or preference.

For a decision brief, name the supervision you can obtain, audit its failure modes and state the evaluation you would require. This public one-prompt experiment cannot serve as an untouched test. Broader claims need new prompts, trustworthy task contracts, controlled comparisons and independently reviewed evidence. Captures and exports preserve your reasoning inputs; practice answers and report presence do not establish implementation proficiency or learning effectiveness.