Judge a generated distribution.

Evaluate a set, not a favorite example

A generator returns 24 points. The first few look plausible. Before choosing it, ask what the complete set contains, what it leaves out and what the task requires. A hand-picked gallery can hide duplicates, missing groups and invalid outputs. Freeze the model, sampling settings, sample count and evaluation rules before comparing results. Keep failures in the denominator.

This lab deliberately mixes constructed counterexamples with one actual diffusion output set. Every candidate states its origin. Constructed points explain measurement failures; they are not evidence that a trained model produced those points. The reference and all evaluation data are public repeated practice, not an untouched test.

Five questions that need different evidence

Fidelity: Do outputs resemble the intended examples under a stated measure? Our first measure is mean distance to the nearest of four centers. Smaller is better under that measure, but the actual reference points lie on rings of radius 0.2 around the centers. A point exactly at a center wins the center-distance metric while missing the ring. A second measure checks the ring itself.

Diversity and coverage: Are different kinds of output represented, and how often? Four covered modes is different from balanced counts. Twenty-four distinct coordinates can still occupy one mode. Exact duplicates are only one form of repetition; tiny perturbations may create numerically different but practically identical outputs.

Memorization: Are outputs reproducing training examples? Exact matching can establish equality to known source coordinates, but a near neighbor alone does not establish copying. Similar samples can occur naturally in a narrow distribution. Training-set access, known construction, matched non-training comparisons and the similarity rule all matter.

Utility: Does the output meet a concrete task? An application may need broad coverage, requested content, reliability, a useful downstream result or human review. A general visual score cannot replace the task contract.

Provenance: What produced the output, what source material was used and what permission or attribution applies? A license is not evidence of novelty. A file hash identifies exact bytes, not model behavior or semantic correctness. Keep source records and transformations alongside the result.

Understand the numerical measures

For a point x, let d(x) be the shortest Euclidean distance to any target center. Euclidean distance is the straight-line distance on the graph. The lab displays the average over n outputs:

The vertical bars mean absolute difference. Twenty-four identical center points have center distance zero and ring residual 0.2. Twenty-four reference-ring points have center distance 0.2 and approximately zero ring residual. The winner changes because the question changes. Neither average alone measures coverage.

A point is accepted by our coarse validity check when its nearest-center distance is at most 0.45. Each accepted point contributes to that center’s count. Coverage is the number of centers with at least one accepted point. Rejected outputs stay in the denominator. This tolerance is an explicit teaching choice, not a guarantee of matching the full target distribution.

The near-training flag compares each output with all training points and asks whether the nearest distance is below a chosen threshold. The lab separately displays exact matches using a tolerance of 10⁻¹⁰, mean training distance and mean development distance. Change the flag threshold from 0.02 to 0.1 for the “new coordinates” candidate: more flags do not mean the candidate suddenly began copying. Its known construction stays the same.

Declare what matters before choosing

The broad-coverage example contract asks for at least 75% accepted points, all four modes and at least two-thirds distinct coordinates. With 24 outputs, that means at least 18 accepted and 16 distinct. With eight outputs it means at least six accepted and six distinct, because counts must be whole numbers. These are stated task rules, not a universal score.

The additional similarity-review gate asks that no more than 10% of outputs be flagged near training data. It is a gate for further review, not a legal or scientific verdict about memorization. A larger threshold may reject the known non-copying reference. Keep that uncertainty in your decision note.

The eight-point option uses a fixed prefix of the same 24-point set. It is not an independent replicate or a new random draw. Choosing the sample count after inspecting outcomes and then reporting only the favorable result biases the comparison. Use multiple predeclared seeds and report uncertainty for a real evaluation; do not infer population performance from this tiny demonstration.

Audit licensed visual examples

The photo exercise uses actual Oxford-IIIT Pet photographs already included with the course. They are clearly labelled as real source images arranged into teaching galleries, not AI-generated outputs. One gallery repeats a dog photograph four times. Another has four distinct source files but includes a cat. For the task “four dog photographs from four distinct sources,” neither gallery meets both requirements.

Inspect the images, their text alternatives and source receipts. The repeated gallery passes a dog-only content check but fails source variety. The mixed gallery has more source variety but fails the dog-only task. This does not imply that source count fully measures visual diversity: different files can contain almost identical content, while one edited source can yield multiple files.

The page retains attribution, the dataset’s current CC BY-SA 4.0 statement and the local note about the older archive README. No original source bytes are modified. These records support transparent reuse of the teaching assets; they do not authenticate generated provenance or establish permission for arbitrary future datasets.

Make a supported claim

For guided practice, begin with the center-distance winner and show two reasons it fails a broad-coverage task. Then compare copied training points with the new-coordinate reference at two similarity thresholds. Export the full comparison and explain what changed in the measurement rather than in the samples.

For implementation practice, independently compute nearest-center distance, coverage and nearest-training distance on a small hand-checkable set. Test duplicates, an invalid outlier and an empty input; specify how your implementation handles the undefined empty mean. For a decision brief, declare the intended use, thresholds, sample protocol, provenance requirements and human checks before choosing a candidate. Describe failures and what remains unmeasured.

This lab does not compute image embedding scores or establish perceptual quality. Such scores use additional representation assumptions and still need coverage, task and provenance checks. The lesson’s questions, gates and exported notes are practice evidence; independent generative-project assessment, artifact review and learner pilots remain pending.

Return to the GAN comparison to collect trained examples, or the conditioning lesson to examine task constraints.

The optional generative implementation studio now supplies a Python reference, gradient checks, resumable training and a separate learner sampler exercise. Import its artifact into the saved generative project for comparison with another mechanism.