Test a proposed system improvement.

Improvement can change the system, not just its weights

Imagine a puzzle agent that finds valid routes but repeatedly scans the whole map. Replacing that scan with an outgoing-edge index can reduce wasted work without retraining any network. Other proposals might improve a tool, memory index, training curriculum, algorithm, or routing rule. Each changes a different part of the system and needs suitable evidence.

In the default experiment, the original scan solves 4 of 8 cases within the 40-unit allowance. A shortcut solves 6 of 8, but fails both critical one-way cases. Its better average hides a regression. The correctly directed index solves 8 of 8 and reduces mean counted edge/neighbor reads from 26.25 to 13.125. This is a measured reduction in our defined counter, not a claim of twice the wall-clock speed on every computer.

Separate proposing from judging

A component can suggest a change, but it should not be the sole authority deciding that its change works. Otherwise it might alter the meaning of success, miss its own mistake, or optimize a convenient proxy. Our independent checker reads the original directed graph contract. It does not trust the candidate’s invented reverse links.

A useful improvement cycle is: identify a failure, propose a bounded change, test it in isolation, compare with the current version, check regression thresholds, then promote or reject. Keep a baseline under the same cases and resource limits. Compare important subgroups as well as totals. A budget exhaustion is unresolved; it is not proof that a route cannot exist.

The eight visible cases are development and release checks. They are deliberately available for inspection and debugging. Reusing them does not create independent evidence. A final generalization claim also needs newly held-out tasks, a frozen candidate, and a recorded evaluation protocol.

Three release gates

Before running a candidate, specify what is acceptable. For this demonstration: all six routine cases must be correct, both critical cases must be correct, and mean counted work must be at most 30 units. The lab also checks overall non-regression and at least one measurable improvement.

A gate can be too weak. If you lower the critical floor to zero, the faulty shortcut can pass the remaining conditions. That does not repair the shortcut; it changes what your release process accepts. The saved review records the actual thresholds so the decision remains inspectable.

For the source exercise, design three gates for a coding-system improvement that preserves other reliability. A worked answer is: pass new independent coding tasks under a fixed budget; preserve minimum scores on math, memory, and tool-use regression groups; and pass critical permission and output-contract checks. Define numerical thresholds, samples, and failure handling before seeing candidate results. The exact thresholds depend on the intended use; the small graph lab’s values are not universal deployment standards.

Version, isolate, and recover

Keep an immutable candidate identifier, baseline identifier, test-suite version, configuration, gate values, and result provenance. Try the candidate in an isolated environment with bounded resources. Retain a working previous version and a tested restoration path. If a promoted change causes a regression, rollback restores that version while the failure is investigated.

In this browser lab, promotion changes a local simulation’s active version. Its history preserves the review that allowed the change. Rollback restores the previous recorded version. It does not prove that the older version meets every new requirement, and it is not a deployment to a real service.

Why improvement can plateau

More generated examples may repeat existing knowledge. A reward can become easier to exploit than the real task. Repeated benchmark feedback can become an accidental training signal. Synthetic data can narrow the distribution and omit unfamiliar failures. Stronger proposals still need trustworthy verification and access to informative evidence.

A curriculum generator can propose solvable challenges near the learner’s current ability, but independent checks must establish that those challenges add useful novelty. Track novelty, independent task quality, important regressions, and total cost. Use new external evidence when a system’s own generated material no longer challenges it. An improvement in one bounded domain does not establish general intelligence or guarantee an unlimited recursive improvement cycle.

The candidates here are supplied programs. The learner controls their comparison; no model writes code or modifies itself. A research extension could generate candidate patches, but would still need isolated execution, independent evaluation, versioning, and rollback.