Validate without reusing the answers.

A perfect score can answer the wrong question

Imagine a sensor reading was logged twice. You put one copy in training and the other in validation. A nearest-neighbor model finds the identical training measurement and copies its target. The error is zero, but you have not shown it can handle a new measurement group.

The experiment has 24 fictional groups, with two identical records per group. The first control splits individual rows into three folds. Inspect the two rows of the diagram: a group can have a T (training) and V (validation) at once. Change to group folds. Both copies now travel together. The larger error reveals a harder, more relevant question; it does not mean group-aware validation damaged the model.

Choose what must be new

Cross-validation repeats fitting and evaluation on several development partitions. Here three-fold row or group validation holds out one partition, fits on the other two, then rotates the held-out partition. Each record receives one validation prediction. The model and its preprocessing are refitted for every fold.

Group folds ask about measurement groups absent from fitting. They do not automatically ask about the future: earlier and later times still mix. If tomorrow differs from yesterday, training on future records can give an unrealistic estimate for an earlier deployment.

The forward-time option first trains on times 0–7 and validates on 8–15. It then trains on 0–15 and validates on 16–23. Training is strictly earlier than validation in each fold. Earlier validation observations can legitimately become training history for the later forecast. They must not flow backward into an earlier fit. This dataset also keeps groups separate across each temporal boundary; other datasets can require explicit group exclusion as well.

Predict before switching: will moving to a more realistic split necessarily lower error? No. It may expose uncertainty that the easier protocol concealed.

Forward-time validation scores 32 later records; row/group validation score all 48 records. These are different populations. Choose the intended deployment question before comparing their numbers. Group identifiers here describe repeated measurements, not independently verified identities.

Fit the entire pipeline inside the boundary

A pipeline includes every operation that learns from data, including filling missing values and changing feature scales. A correct split followed by preprocessing fitted on the whole dataset still allows validation information into training.

For each safe fold:

1. Select training and validation records.
2. Compute each feature’s missing-value fill from observed training measurements.
3. Fill missing training values, then compute the scaling values from that filled training array.
4. Reuse exactly those fitted values to transform validation measurements.
5. Predict validation targets using training neighbors only.

The deliberately leaky switch moves step 2 and the scale calculation outside the boundary, using all 48 development measurements. It does not use validation targets, but the feature distribution has still influenced the fitted pipeline. Leakage does not have to improve every score to violate the evaluation contract.

Follow one transformation

Suppose the training values are 2, missing and 6. The observed mean is (2 + 6) / 2 = 4, so filling gives 2, 4 and 6. Their squared distances from 4 are 4, 0 and 4. The population variance is (4 + 0 + 4) / 3 = 8/3; its square root is the standard deviation, about 1.633.

The transformation is z = (filled value − m) / s, where m is the fitted training mean, s is the fitted training standard deviation and z is a dimensionless scaled value. A missing validation measurement is filled with 4, then becomes (4 − 4) / 1.633 = 0. Its own validation cohort does not supply a new mean.

For a constant feature, the lab divides by 1 to avoid division by zero. If every training value is missing, no observed mean exists: the calculation rejects that feature rather than silently inventing data. With scaling off, imputation still happens, but subtraction is 0 and division is 1.

Nearby practice: with training values 1, missing and 5, what fill is used? It is 3. A newly observed validation value of 100 does not change that fill. It can change the validation error, which is a different quantity.

Inspect the prediction rather than trusting the summary

The model finds the k closest training rows in transformed two-feature space and averages their targets. Here k is a positive integer selected from 1, 3, 5 or 9. Two neighbors with targets 2 and 6 would predict (2 + 6) / 2 = 4. A target of 5 would give absolute error |5 − 4| = 1 target unit.

The workbench displays actual neighbors and distances for each selected validation record. Equal distances use original row order. Repeated training rows still have repeated weight: group-aware splitting prevents cross-boundary duplication, but does not itself deduplicate training.

Mean absolute error, or MAE, adds every absolute prediction error and divides by the number of scored records. Fold totals are combined by record count, not by an unweighted average of unequal-size fold means. The diagram uses one fixed column per group; colors are accompanied by T/V labels and exact record membership.

Cross-validation is not an unlimited answer supply

Changing k, scaling or preprocessing and choosing the best result makes the validation answers part of development. A fixed dataset does not become independent again because you rerun the calculation. More trials give more opportunities to select a setting that happened to suit these particular examples.

Use the comparison record to retain your choices, including disappointing ones. Declare a reasonable comparison allowance before tuning. This bounded workbench records at most twelve comparisons per visit, but twelve is a teaching limit, not a scientific guarantee. Resetting controls retains the record; reloading clears this temporary experiment and does not make its reused cases unseen.

Optional: separate selection from evaluation more carefully

Nested cross-validation uses inner folds to select a pipeline and outer folds to estimate the performance of that selection procedure. Every imputer, scaler and model must be fitted inside the appropriate boundary. Temporal and grouped constraints still apply. Nested validation is more computation and cannot repair a dataset that does not represent the intended use; this workbench does not implement nested fitting.

Freeze before final evaluation

A final project should reserve a separate dataset, choose its split contract and metrics in advance, then freeze the complete pipeline before examining final outcomes. Freezing includes preprocessing, model settings and decision rules. If you use final errors to revise the model, those cases become development evidence; obtain new independent evidence for a renewed final claim.

This lesson intentionally has no final-test button. Its synthetic folds are visible development practice, and its notes are not an independently assessed project. Use the classifier project to examine an existing freeze/evaluate workflow; the cumulative real-data comparison project remains a separate build item.

Apply the idea to an unfamiliar dataset: define the future user or group, list information available at prediction time, draw the fit boundary, propose a tuning allowance and justify a separate final evaluation. Export the experiment and explain a failure case. Human artifact review and learner pilots remain pending.