Copy a demonstrated action.

Copy a move before learning from a reward

An expert sees a puzzle board and supplies a useful next action. A learner takes the same board as input and predicts an action distribution. Behavior cloning fits that prediction to expert targets with supervised learning. The label says what to imitate; it is not a reward observed after the learner acts.

Here the expert is an exact shortest-path solver for a tiny 2×3 sliding puzzle. It knows the standard simulator. It labels every action that reduces exact distance by one, sharing target probability equally among tied best moves. These labels are generated from an authored environment, not collected from human demonstrations or inferred from preferences.

The final agent project uses the same policy/value architecture for a larger supervised experiment. This lesson isolates imitation on eight demonstration states and ignores the value output. It does not modify the saved project or train with reinforcement-learning returns.

Turn the board into action probabilities

Each of the six cells has six possible tile identities, including the blank. A one-hot encoding uses 36 inputs: each cell activates the entry corresponding to its tile. Sixteen tanh hidden units feed four action scores, called logits. Softmax converts those scores into probabilities.

Directions describe movement of the blank. If the blank is on the upper row, moving Up is illegal. A legal-action mask excludes that action before normalizing probabilities. If two allowed actions have equal scores, each receives one half even if some excluded action had a very high raw score.

This is a programmed contract, not a learned skill. The mask cannot determine which legal move is useful. A runtime transition check independently rejects an illegal proposal when the mask is turned off for inspection. Turning it off does not change training or the cohort metrics.

Fit the expert distribution

Suppose the expert target gives all its probability to Left, while the model assigns Left probability 0.25. The cross-entropy loss is −ln(0.25), about 1.386. Increasing that probability to 0.75 reduces the loss to about 0.288. These are loss units, not puzzle rewards.

For a state s and action a, let q(a|s) be the expert target and p(a|s) the model’s masked probability. Their cross-entropy is:

$$L(s)=-\sum_a q(a\mid s)\ln p(a\mid s)$$

Read it as: multiply each expert weight by the logarithm of the probability the learner assigns that action, sum and negate. A zero target contributes zero. If two moves tie for the expert, each has target 0.5; collapsing onto only one of them does not minimize this particular distribution-matching loss, even though that chosen move may be useful.

For each legal logit, the derivative is prediction minus target. With prediction 0.25 and target 1, it is −0.75. Gradient descent subtracts the gradient, encouraging that logit upward. Shared hidden weights mean that one update can also change predictions on other boards. The lab averages contributions across eight demonstrations and computes all gradients before changing any parameter.

The existing model has 677 parameters. This policy-only objective trains 660 of them. The 17 parameters belonging only to the value output remain unchanged, while shared hidden units change. The value output is unused; unchanged value-head parameters do not imply an unchanged value prediction.

Leave the demonstration distribution

A model may agree with experts on the states it saw and still make mistakes during its own sequence. A mistake changes the next state; future decisions can then occur in places the demonstrations did not cover. This distribution shift is a central difficulty in imitation learning.

The lab makes a controlled version visible. Starting from a demonstration board, it forces a legal non-expert move and inspects the resulting board outside the fitted eight. It then checks the learned proposal against the exact environment and expert. The example does not assert that every deviation fails: inspect the actual probabilities and action agreement.

Both cohorts are selected from the original project’s training pool. A fixed coverage rule favors demonstration states with more legal non-expert neighbors in that pool; it never uses model predictions or losses to select cases. The deviation cases are not part of this lesson’s gradient, but they are repeatedly visible development diagnostics. They are not reserved evaluation, and a destination reached by multiple deviation transitions can be counted more than once. The displayed denominator counts transitions in that cohort.

Dataset-aggregation approaches address this problem by obtaining expert labels on states visited by the learner and adding them to training. The lab does not implement that algorithm or silently train on its diagnostic cases. Such an extension needs explicit collection rules, an expert-label budget and a separate evaluation protocol.

Keep three measurements separate

Fitting loss measures how well predicted probabilities match labels on a dataset. Next-action agreement asks whether the chosen move is one the expert considers useful. Whole-puzzle success asks whether a sequence actually reaches the goal under its constraints. High agreement need not prevent a rare early error, a loop or accumulated failure over many decisions.

The lesson shows one verified proposal at a time. The project adds bounded full attempts, exact search comparisons and frozen evaluation. Neither a mask nor an accurate expert label proves that the chosen task or reward represents a useful real-world objective. If a teacher is biased, suboptimal or observing information the learner lacks, imitation can copy those limitations or fail to reproduce the behavior at all.

Build a coverage audit

For guided practice, save a model before and after training, compare both cohorts and explain one specific prediction. For implementation practice, reproduce the masked loss and a finite-difference gradient check, preserve simultaneous updates and test an intentionally dominant illegal logit. For a decision brief, identify who supplied the demonstrations, which situations they cover, whether labels depend on unavailable information and how recovery behavior will be evaluated.

Try an unfamiliar case: a warehouse operator demonstrates successful routes on clear floors, but the deployed cart encounters a blocked aisle after one wrong turn. Identify the missing states, the source of recovery labels and the evidence required before claiming safe autonomous operation. Merely adding more copies of the same clear-floor demonstrations does not establish recovery coverage.

Compare your audit with these checks

A useful audit distinguishes original and learner-induced states, includes blocked-aisle recovery with justified actions, records sensor information available to both teacher and learner, and tests complete attempts on new scenarios. Exact runtime constraints and human review remain separate from imitation loss. These are planning checks, not a certification of the learner’s implementation.

Training and three bounded captures are temporary. Export the experiment before leaving. Human review, broader device profiling and learner pilots remain pending.