Learn from verified self-play.

Experience can be generated; correctness still needs evidence

Self-play uses an agent or copies of its policy to generate experience by playing a game. The rules can provide exact feedback even when nobody labels the best action at every position. The familiar pieces form a loop: generate experience → verify an outcome → update learned values → evaluate the new policy.

In our stone game, a separate verifier checks alternating players, legal takes, consistent remaining piles, and an empty final pile. The winner is the player who took the last stone. A policy cannot make a losing game count as a win by claiming it won.

The Q table stays fixed while one training game is generated. After verification, the learner moves each played action’s estimate toward +1 if that player won and −1 if that player lost:

$Q_{\mathrm{new}}=Q_{\mathrm{old}}+0.3\,(\text{outcome}-Q_{\mathrm{old}})$

This is a simple Monte Carlo update: it uses the completed episode’s outcome rather than a next-state value estimate. Both players’ actions supply examples. The opponent changes as the shared table changes, so the learning problem is not stationary. The estimates are not guaranteed calibrated probabilities, and more games need not improve every measurement.

Search now versus learn for later

An agent can spend extra computation during inference: search candidate moves, revise an answer, or use examples in its current context. These can improve the current result without changing its learned parameters. The lab’s practice game only reads a frozen Q table; it does not train on your moves.

Persistent learning retains a change that affects future tasks. Here the retained change is to table entries. In a neural system it could be a weight update. A lasting memory update is another kind of retained change, but it is not automatically a neural weight update. Always identify what actually changed and test whether it helped.

Systems such as AlphaZero combine self-play with search and neural policy/value learning. This small game implements none of that architecture’s neural training or tree search. It isolates the experience-and-feedback loop so you can inspect every update.

Test beyond the training partner

A policy can exploit a weak opponent without learning a robust strategy. We therefore freeze the table and compare an untrained baseline and the learned policy against a random opponent and a separately programmed optimal opponent.

Why are only six of the twelve optimal-opponent matchups winnable? A player facing a multiple of three cannot force a win against perfect play. From other piles, taking one or two can leave a multiple of three; then answer the opponent’s take with the amount that makes their two turns remove three. Across starting piles 4–9 and both seats, six assignments give the model a forced win and six a forced loss. A 50% total win rate can therefore be perfect on this balanced test.

The states are familiar. This evaluation tests different opponents and fixed rollouts, not generalization to larger piles, different game rules, or unseen puzzles. Repeated tuning against the same test also weakens its independence. A stronger research experiment would reserve new tasks and report failures as well as successes.

Why not simply trust generated answers?

If a model invents a fact and then treats that same statement as a correct label, training can reinforce the error. Agreement with itself is not an independent truth signal. Games work well when outcomes have exact rules; factual questions often require evidence outside the generator.

For a puzzle system, one useful loop is to generate candidate solutions, check every move and the final goal with an exact simulator, and retain verified solutions as training examples. Verification establishes that those particular solutions are valid. It does not establish diversity, optimality, or success on unfamiliar puzzles. Keep reserved puzzles outside the generated curriculum and compare solve rate, cost, and failure types after each update.