Find out whether improvement generalizes.

Ask what changed, and what evidence would reveal failure

An agent that performs well on five familiar games may have learned their rules, memorized repeated situations, or exploited a shared interface. A sixth unfamiliar game tests a different claim. Define that novelty before evaluating; adding special support after inspecting its failures turns it into a development task.

Measure several properties together: accuracy on known tasks, transfer to unseen tasks, adaptation speed, reliability over long workflows, calibration of confidence, and efficiency. Report budgets for examples, training updates, tool calls, memory, and execution time where they apply. A faster incorrect answer is not necessarily an improvement.

Four levels of novelty

Start with unseen positions under familiar rules. Next change rules while retaining familiar mechanics. Then hold out an entire game family. Finally introduce changes during operation and measure detection and recovery. The lab’s four protocol cards are worked designs for the source exercise; they are not claims that all four game environments have been implemented.

Adaptation speed asks how performance changes after a specified amount of new evidence. For example, compare zero-shot success with success after 1, 5, and 20 demonstrations. Keep the post-adaptation test separate from those demonstrations. The adaptation lab makes a small parameter update inspectable. A context hint, a memory update, and a gradient update are different adaptation mechanisms and should be reported separately.

Keep hidden evaluation outside the generated training curriculum. Separate near duplicates by episode or task family where appropriate. A random row split is insufficient if adjacent rows come from the same trajectory or template.

Repeated selection can fit the evaluator

The benchmark lab compares seeded guessers, then keeps the one with the highest public accuracy. With 128 candidates and 16 public cases, candidate 77 scores 81.25% publicly but 43.75% on the 64 reserved cases. No learning occurred. Selection found a candidate whose guesses happened to match the public set.

The best public score cannot decrease when more candidates are added to the same comparison. That alone does not mean the selected model’s generalization improves. Larger evaluation sets reduce some sampling noise, but repeated tuning can still adapt to them. Freeze the choice before opening reserved data. Once results influence another choice, use new cases for a new final claim.

These deterministic hash-based guesses illustrate selection noise. The intended chance baseline is 50%; a particular finite sample need not score exactly 50%. The experiment does not establish a population accuracy theorem or model a real neural training process.

Confidence needs its own check

A confidence of 80% should correspond to about 80% correctness among comparable predictions over sufficient evidence. Accuracy alone does not show that confidence is calibrated. Inspect groups and the distribution of confidence, not just an overall average.

If the selected guesser claimed its public score, 81.25%, as the probability of each reserved answer being correct, it would exceed the observed 43.75% by 37.5 percentage points. The lab also computes mean squared probability error, called a Brier score. For a correctness event with outcome 1 or 0, square the difference between the claimed probability and that outcome, then average. Lower is better. A constant 50% forecast always has squared error 0.25 in this binary example. One small group remains noisy evidence, not a full calibration analysis.

Ablations test contributions, including interactions

An ablation removes or changes one component while keeping the comparison conditions fixed. The graph lab removes the edge index, visited-state memory, search across branches, or runtime verifier. It also removes index and memory together. Each variant actually executes on the same cases; the independent correctness definition remains fixed.

At the default budget without a defect, all components achieve 100% correctness. Removing the index reduces it to 50%; removing visited-state memory gives 75%; following only the first branch gives 62.5%. Their effects are not necessarily additive. Some components can compensate for others, and a budget can turn additional work into failure.

Removing the runtime verifier causes no wrong acceptance when all proposed results are already valid. Inject the direction defect to reveal its value: without verification, invalid routes are accepted. The independent evaluator still marks those routes wrong. The work counter excludes verifier overhead, so this comparison does not measure the verifier’s time cost.

For a larger agent, also ablate recurrent refinement, learned routing, and specialist models. Compare with the planning alternatives, fixed-rule router, and connected-model training modes. Hold task splits and resource limits fixed, include joint removals where components interact, and avoid claiming that this graph experiment measured those other architectures.

A hundred steps can expose a different failure rate

Under independent step failures, a workflow requiring 100 successful steps, each with success probability 0.99, succeeds with probability $0.99^100\approx0.366$. That is about 36.6%, despite 99% per-step accuracy.

With one independent retry and a perfect failure detector, a step succeeds with probability $1-(1-0.99)^2=0.9999$. All 100 then succeed with probability $0.9999^100\approx0.990$. That improvement consumes a larger possible call budget and depends on strong assumptions.

The lab contrasts a shared outage: 99% of runs work perfectly, while 1% cannot work at all. It has the same marginal per-step success but 99% whole-run success; retries do not fix the outage. Correlation changes the pattern of failures and does not always reduce the chance of all-success relative to independent failures. Persistent tool faults, cascading errors, imperfect checks, and recovery policies need direct end-to-end evaluation.

Improving tool reliability, checkpointing valid intermediate work, verifying outputs, and recovering from errors can improve whole-workflow success more than a small gain in isolated model accuracy. Measure their actual cost and recovery outcomes.

Report what your experiment supports

Publish the task and novelty definitions, data exposure, baselines, budgets, subgroup results, confidence checks, ablations, failures, and uncertainty. Repeat across independently generated tasks when possible; replaying an identical fixed set is not replication. A strong result on a small suite supports a bounded claim. “General intelligence” is a much broader interpretation that these teaching experiments do not establish.