Test the complete system.

Passing tests answer only the questions you asked

A model produces a number and an assistant returns a plausible sentence. Neither observation proves that the system is correct. A measurement could use the wrong unit, a scaler could include evaluation data, a tool result could belong to another request, or a correct answer could accompany an unauthorized action.

Entering midway: a test supplies an input, observes behavior and compares it with an expected result. An oracle is the source of that expectation: a separately calculated answer, a trusted fixture or an explicit rule. A regression is an unwanted change from specified behavior. A mutation here is a deliberate small defect used to ask whether a test can detect it.

The lab changes actual local computations and workflow gates after loading the original artifacts. The saved artifact files remain unchanged; their digests cannot rule out a later runtime handoff defect. It does not assign pass/fail labels from a fault-name lookup table. Start with the supplied shape and happy-path tests, introduce a scaling failure and inspect why the suite remains green.

Match the test to the question

Test scope	Question	Example in this lesson	What a pass does not establish
Unit	Does one small operation satisfy its contract?	Matching versus mismatched receipt identity	The full workflow actually calls it correctly
Numerical	Are calculated values within a justified tolerance?	Packaged probabilities versus a separate sklearn record	Accuracy on new real-world examples
Data	Are schema, units, partition and fitted statistics correct?	Train-only scaling versus fitting on all rows	Representativeness of the collected dataset
Integration	Do connected components preserve meaning and contracts?	Retrieval plus conflict handling; tools plus receipt validation	Every possible task or external failure
Behavior	Does the full task satisfy its intended rules?	Supported answer and no unauthorized effect	Hidden failures outside the observed cases

These categories overlap. A request-boundary check can be both a data-contract and an integration test. The names are less important than describing the actual input, expected behavior and components exercised.

A unit test is often useful for locating a fault quickly. A broader integration test can reveal that individually sensible components were connected incorrectly. Keep both when they answer different questions; do not count duplicate assertions as independent confidence.

Check values, not only shapes

The starter shape check asks whether the transformed model input has four finite numbers. Raw centimetres, correctly standardized coordinates and many wrongly ordered vectors all satisfy that condition.

A stronger transformation test constructs each input as its stored training mean plus one stored training spread. Standardization should produce one in every coordinate: (mean + spread − mean) / spread = 1. Here “mean” is the training average and “spread” is its positive population standard deviation. All four standardized results are dimensionless.

Practice: a feature has training mean 5 cm and spread 0.5 cm. The test supplies 5.5 cm. The correct transformed value is (5.5 − 5) / 0.5 = 1. If a broken handoff sends raw 5.5 into the weights, its shape still passes but the value check fails.

Reversing the weights gives a different counterexample. The transformation still produces four ones, so that isolated test passes. The later feature-to-weight handoff is wrong. The numerical parity check exercises the whole prediction with varied inputs and exposes the changed probabilities.

Keep the numerical expectation independent

The lab compares 152 predictions with a previously recorded sklearn pipeline. That record includes ordinary Iris inputs and partial/all-missing examples. It is separate from the JavaScript prediction code under test. Expected values are not generated by calling the same altered implementation and comparing it with itself.

For each probability, the absolute difference is the distance between actual and reference values. The maximum difference is the largest distance across the cases. A difference no greater than 1e-12, or 0.000000000001, passes this bounded parity check. The tolerance was justified for this small cross-runtime computation; it is not a universal tolerance for all models or hardware.

If two probabilities are 0.600000 and 0.600001, their absolute difference is 0.000001. That exceeds 1e-12 even if their threshold label is the same. Decide whether a test is checking exact interface behavior, numerical agreement, threshold decisions or task quality before choosing a tolerance.

These are public regression cases, not an untouched accuracy test. Matching them establishes implementation agreement on those inputs. It does not prove the training data are representative or the model generalizes.

Test the data boundary as well as the calculation

The data-statistics check reconstructs mean fill and population spread from the original 89 training measurement groups. The injected leakage variant instead fits on all 150 source rows, including development/evaluation data. It still produces finite predictions, but its fitted statistics do not match the training-only protocol.

The grouping rule identifies identical measurement vectors, not verified physical specimen identity. Both the original protocol and its limitation remain relevant even when a numerical test passes.

The schema check includes a valid centimetre request and two invalid requests: unsupported millimetres and a numeric string. A validator that rejects everything must not pass simply because it rejected two bad inputs. A boundary that changes “mm” to “cm” without converting the values can accept a well-shaped but semantically wrong request.

Input validation cannot detect every plausible lie. A number already labelled cm might actually have been measured in another unit. Testing the client, measurement process and complete handoff is therefore separate from checking the model’s JSON types.

A correct happy-path answer can hide a broken workflow

The assistant’s ordinary library question has a supported closing time and fee in one current notice. Reducing retrieval to one document still answers it correctly. That same retrieval limit can miss a second current notice with a conflicting closing time.

The conflict test’s oracle considers both authored notices and expects the closing time to remain unresolved while the fee is answered. With only one retrieved source, the answer stage sees no conflict and confidently returns a time. The defect is in the composed retrieval/answer behavior, not necessarily in the code that checks the subset it received.

A separate tool scenario returns a receipt for another task. The default workflow rejects it. The testing sandbox can inject a deliberately faulty identity predicate that accepts any receipt; then a plausible meeting time enters the final answer without valid attribution. Production callers of the shared workflow retain the exact identity check. The injected function is supplied code for this exercise, not learner text executed in the browser.

Finally, the permission variant disables the runtime action gate. The normal library answer can remain correct while a document directive produces a simulated unauthorized effect. A test that checks only the final answer misses that side effect. The behavior check observes both the attempted proposal and the absence of unauthorized effects. The documents, proposal syntax and effects are programmed teaching examples, not an evaluation of a live language model or a real external system.

Use a unmodified control before interpreting detection

The unmodified implementation is the unmodified control: your selected checks should pass on the behavior they are meant to accept. If it fails, investigate the implementation, fixture, test assumption or tolerance before celebrating failures on mutated variants.

With seven programmed mutations, detecting three gives 3 / 7 ≈ 42.9% detection for this particular list. Detecting all seven gives 100% of these seven. That denominator does not include every conceivable defect or real-world input. It is not code coverage—the fraction of code exercised—and neither measure is model accuracy.

An empty selection is untested, not passing. The matrix reports no valid detection proportion when the control has not passed. An undetected mutation may indicate a missing check, an insufficient input, a weak assertion or a change that does not affect the behavior under test. Inspect expected and observed values instead of trying to maximize a score blindly.

Turn a mismatch into a repair plan

Name the first broken assumption, not just the red cell. For raw-feature bypass, reconnect the stored preprocessing before the weights. For reversed weights, restore the declared feature order. For data leakage, fit transformations on training only. For a wrong receipt, validate identity before accepting the value. Then rerun the narrow check and the relevant surrounding workflow checks.

Restoring the unmodified scenario in this lab removes a teaching mutation; it does not repair code in another project. Your own implementation needs runnable tests and a reviewed explanation of the repair. The browser’s predefined checks provide practice and examples, not independent implementation proficiency.

• Guided exploration: predict an escaped fault, add one relevant check and explain its evidence. Find a second fault that the new check still misses.
• Implementation practice: add an independent expected-value or contract test to the downloadable model or assistant starter. Introduce a controlled bug, demonstrate failure, repair it and rerun the relevant integration tests. Preserve code, inputs and test output for review.
• Decision practice: review a green suite and state which user-facing risks it actually covers. Request tests for a missing subgroup, boundary or failure condition, and define what evidence would change the release decision.

Export a note naming the tested version, cases, expected behavior, repair and remaining limitations. More testing is useful when it addresses an identified gap. Repeating an unchanged public suite without a new question does not create new independent evidence. Human artifact review and learner pilots remain pending.