Evaluate language tasks with evidence.

Decide what success means before seeing the answers

Suppose a school information service must return exactly one time, or “unknown” when its approved notice does not establish one. A polished paragraph is not automatically a successful answer: a program may need one parseable value, and a learner needs the right value for the right day.

Entering directly: a reference answer is the answer justified by the declared source and task policy. A rubric names criteria and how they are scored. Development cases help you choose or improve a system. Reserved cases are set aside before that choice; after their outcomes influence your choices, they are development evidence too.

The workbench uses eight original fictional notices and actual recorded GPT-2 completions. It compares raw output, a wrapper that extracts the first time and a source-based rule. The rule is transparent and intentionally strong for these simple templates. None of these observations estimates the quality of current assistants on general tasks.

Use several measurements without confusing them

Measurement	What it asks	What it cannot establish alone
Prediction loss	How much probability did the model give the correct reference tokens?	Whether the delivered answer completes the task
Exact task success	Does the delivered answer match the specified time or unknown?	General conversational usefulness or broader safety
Source support	Are the factual claims established by the permitted notice?	Completeness, correct formatting, or truth beyond that source
Human judgment	Is the response clear, useful and appropriate under a defined rubric?	An objective truth certificate or unbiased agreement

The club recording states the correct time but adds a question. Raw output therefore fails the exact one-value contract. The wrapper returns the time and passes. Their underlying reference-token loss is identical: the wrapper has not changed the language model.

A wrapper can also make things worse: taking the first time from a response can discard the sentence explaining that it belongs to a different day. Postprocessing needs task-level tests too. Inspect the reserved cases only after freezing your protocol.

Make the loss calculation inspectable

Imagine the correct next token has probability one half. That costs one bit of surprise. If it has probability one quarter, it costs two bits. Less probability on the correct token means more surprise. The interactive worked example includes probabilities one and zero: a certain token costs zero; a token assigned exactly zero probability has infinite loss.

Here N is the number of correct reference tokens, i is a token’s position, and pᵢ is the model probability assigned to that correct token after the prompt and earlier correct tokens. L is mean surprise in bits per token. Read the formula as “add each correct token’s surprise, then divide by the number of tokens.” For probabilities one half and one quarter, the mean is (1 + 2) / 2 = 1.5 bits. An empty reference has no mean, not a mean of zero.

The recorder uses teacher forcing: it supplies the correct earlier reference tokens when measuring the next probability. That is different from generating an answer and then checking whether it is useful. The table shows genuine probabilities for each reference token, including leading spaces. An unknown answer may be one token while a time is several tokens, so mean token loss weights tasks differently from one-vote-per-task accuracy.

Compare token loss only with compatible tokenizers, reference text, inclusion rules and averaging. The source-based rule has no token distribution; its loss is “not applicable”, never zero. Bits are a unit for this mathematical score, not the size of a compressed file.

Build a rubric that does not hide failure

The lab gives source support, format and clarity separate binary scores: one means the criterion is met, zero means it is not. Multiply each score by its weight, add the products, and divide by the sum of weights. With weights 2, 1 and 1, an unsupported but well-formatted, clear answer earns (0 × 2 + 1 × 1 + 1 × 1) / 4 = 50%.

Now give clarity all the weight. A fluent wrong answer can pass. Changing weights has changed what you reward; it has not repaired the answer. Keep exact task success and critical failures visible beside the weighted score. For a real deployment, some requirements should be mandatory gates instead of tradeable points.

The raw-text source-support labels are authored annotations of the actual recordings. They identify specific omissions and unsupported claims. They are not the output of an automatic truth detector and have not undergone independent human review.

Treat disagreement as information

Two illustrative raters interpret “clear” differently. A rates a readable first sentence positively; B requires the entire response to be a complete answer without repair. These are authored examples, not results of a human study. Switching between them demonstrates why an unspecified adjective is a weak rubric.

Write anchors such as “returns one value with no unfinished follow-up question” and “states that the requested fact is unavailable rather than substituting another day”. In a real evaluation, have reviewers score the same examples independently before discussing differences. Keep their original ratings, disagreements and reasons. Agreement can improve after clarification, but agreement itself does not establish truth.

Practice: should we average the raters?

A mean of 0.5 records a numerical compromise, but it does not tell you whether reviewers inspected the first sentence or the entire answer. First resolve the criterion’s meaning, then report agreement and any remaining disputes. Do not silently replace original ratings with a unanimous-looking score.

Construct and freeze the task suite

Choose at least one stated-answer case and one uncertain-answer case. Explain the intended task mix, the candidate, rubric anchors and exclusions before freezing. Source groups are disjoint between development and reserved cases: two questions derived from the same notice should not masquerade as independent source evidence.

Freezing locks the selected candidate, weights, clarity interpretation, threshold and case list. Reveal records the full protocol and known exposure. The first protocol and outcomes remain available after retries, reload, draft reset and progress import. Later changes do not rewrite what you originally tested.

A reserved case is only unfamiliar relative to the recorded process. The static application ships all cases publicly; it cannot prove that you have never read them elsewhere. We also do not know everything in GPT-2’s pretraining data. Separate local evaluation exposure from possible pretraining contamination. Do not call either absence proven merely because there is no matching record.

Report the numerator and denominator, individual failures, source groups and exclusions. One success in four is 25% on those four cases; it is a very weak basis for a population estimate. Repeatedly choosing protocols to maximize that same score converts the set into development material. Seek genuinely new, appropriately sampled cases for the next evaluation.

Produce evidence for your path

Guided experiment: export a comparison explaining one fluent wrong answer, one wrapper improvement and one wrapper failure. Show why the reference loss does not change when the wrapper changes.

Implementation: reproduce the local recorder and implement a different wrapper. The repository script scripts/record-language-evaluation.py pins the model revision, seven asset hashes and package versions; it makes no network download. Keep the original protocol, compare on the same development cases, then obtain new source groups for evaluation after these reserved cases are exposed. Unit-test the wrapper independently of the reference-answer file.

Decision brief: state the intended use, case mix, exact task success, rubric failures and reviewer uncertainties. Decide whether to accept, restrict or reject the candidate, and specify what additional evidence could change your decision. Neither a completed form nor a favorable weighted score proves readiness.

Continue to measure the whole workflow for retrieval, tool and cost failures, or apply your protocol to the evidence-assistant project. Human artifact review and representative learner pilots remain pending.