Learn from preferences.

A choice is evidence about a comparison

Imagine choosing between two library answers. One is polite and states the right closing time; the other is blunt and wrong. Choosing the first is useful feedback, but that single comparison does not tell a model which difference mattered. It might learn to reward politeness, accuracy, a source phrase or some combination. Examples that separate those properties are needed to distinguish the explanations.

The lab supplies a fictional notice: the library closes at 6 pm and loans last 14 days. Two fictional raters label each response pair. Their choices are editable teaching data, not a survey of real people. A preference label says which response a particular rater favors for this comparison. It is not automatically a universal standard, a factual verdict or a numerical reward with meaningful units.

What information reaches the ranker?

Each answer becomes two numbers: the count of the word “please” and whether the phrase “Source: notice” appears. These features are deliberately weak and explicitly programmed. The ranker learns two weights. Multiplying each feature by its weight and adding produces a score. There is no learned language encoder, and no language model is updated in this experiment.

If the weights are 0.4 for politeness and 0.6 for the source phrase, a response with features [1, 1] scores 1.0. A response with [0, 1] scores 0.6. Their score difference is 0.4. A score is neither correctness nor a percentage. Adding the same constant to both scores would not change their difference or their predicted preference; a shared bias is therefore unnecessary here.

The flow diagram connects the raters’ labels to a loss and then to changed weights. The separate source-check column answers a different question. It compares the authored claims with the notice; it never enters the ranker’s features, labels automatically or gradient. Read the text yourself and verify why the column marks the claim correct or false.

From scores to a pairwise probability

A logistic function converts the left-minus-right score difference into a number between zero and one. A zero difference predicts 50%; a positive difference favors left, and a negative difference favors right.

Here p is the model’s predicted chance of preferring left, s_L and s_R are its scores for the left and right responses, and e is the mathematical constant approximately 2.718. With difference 0.4, the probability is about 59.9%. This is a model prediction about a comparison, not a calibrated chance that the response is true. The live probability bar always uses the full 0–100% range.

Each rater can prefer left (1), tie (0.5) or prefer right (0). We average their numbers with equal weight. Opposite choices yield 0.5, but two ties also yield 0.5. Those situations have the same training target in this model and different human meanings. Keep the individual labels visible. An average is not proof of agreement, and two fictional raters do not represent a population.

State the objective before interpreting improvement

The ranker minimizes mean pairwise cross-entropy: a loss that penalizes predicted preferences inconsistent with the supplied labels. For each pair, its target y is the two-rater mean and its probability is p.

The natural logarithm gives units of nats. At p = 0.5, the loss is approximately 0.693 for any target. When y = 1 and p = 0.8, loss is about 0.223. When y = 0.5, pushing p toward either certainty increases loss; this objective fits the disagreement instead of choosing a winning rater. Equal weighting makes using the mean target exactly equivalent to averaging these two raters’ losses, including a soft tie target.

For each weight, the pair’s gradient is (prediction − target) multiplied by that feature’s left-minus-right difference. Suppose p = 0.5, y = 1 and the feature difference is 1. The gradient is −0.5. With learning rate 0.3, a single-pair update would add 0.15 to that weight, increasing the left score. The actual lab averages gradients over all active pairs and updates both weights simultaneously. The pair inspector shows the intermediate arithmetic; do not confuse one pair’s contribution with the full applied gradient.

This is supervised fitting of a preference score. Selecting its highest-scoring response from a fixed list optimizes that proxy over the list. It is not reinforcement learning, DPO, a trained generator or a complete alignment method. The linked research illustrates the broader idea of learning from comparisons; this lab intentionally isolates a much smaller numerical mechanism.

Make the proxy fail visibly

After training on the default comparisons, the ranker can favor repeated polite words and a source phrase. Candidate g exploits those cues while giving the wrong closing time. Lower training loss and higher selected score can coexist with a false answer. This is proxy exploitation: improving the measured score without improving the intended outcome.

Add the audit pair and retrain from the same zero weights. Its labels prefer a correct ordinary answer over the repeated-politeness exploit. This can reverse the politeness weight. It does not supply general fact checking. Candidates a and h have identical features but different closing times; they receive identical scores for every possible pair of weights. No amount of label editing can recover information discarded by this representation.

The correction pair is training data once included. All examples here are already public and repeatedly inspected, so none is an untouched test. A credible follow-up would predeclare new cases, separate accurate from inaccurate answers while matching style, check multiple sources and report disagreement under a clear rubric. Improving the representation, obtaining better comparisons and adding an independent factual check address different failure mechanisms.

Apply it at your chosen depth

For guided practice, explain one gradient sign and show a counterexample to “high preference score means true.” For implementation practice, reproduce the two-weight objective and check both derivatives with small positive and negative weight perturbations. Include a test showing that identical feature vectors must tie, even when their factual labels differ.

For a decision brief, compare a default-label run, a changed-rater run and an audit-corrected run. Name the actual optimized objective, whose labels it represents, which candidate fails and what evidence would be required before using such a score. Export the weights, labels and recomputed rankings alongside your explanation. These exercises and the lesson’s practice questions do not replace independent project assessment, human artifact review or learner pilots.