Evaluate learning fairly.

Compare the evidence behind the score

A learning curve can look impressive while hiding extra data, privileged supervision, unsuccessful seeds or a reward loophole. Before choosing a model, ask what produced the score and what success actually means.

Entering midway? An environment interaction is an action taken to collect an outcome. An optimizer update changes model parameters using data. A seed fixes a pseudorandom sequence. A checkpoint copies a model at a particular stage. An evaluation tests a frozen policy without fitting it.

Henderson and colleagues examine reproducibility and reporting challenges in reinforcement learning. Here you can inspect a complete small comparison rather than infer quality from one attractive curve. Three seeds illustrate variation; they are not enough to establish broad statistical robustness.

Keep the task fixed

The policy makes two choices, A or B. Its four possible paths are AA, AB, BA and BB. Original returns are 2, −1, 0 and 10 respectively. Whole-task success is the BB path, regardless of how much training feedback another path earns.

Three logits control the probability of B at Start, After A and After B. A logit of zero gives probability one-half. Both methods begin with all three logits zero, so the initial BB probability is one-quarter. This is a small conditional policy, not a deep benchmark or a new robot controller.

The optional shaping intervention adds eight for each A. AA then earns 2 + 16 = 18, exceeding BB’s 10 while failing the task. AB earns 7, and BA earns 8. This bonus deliberately changes which behavior is rewarded. It is not a policy-invariant shaping construction.

The evaluator keeps original rewards and the BB success test unchanged. It reports shaped return, original return and success separately. A larger shaped score alone cannot establish better task performance.

Name the information each learner receives

The clone receives demonstrations from a supplied teacher that chooses B with probability 0.8 or 0.95 at each decision. This teacher already encodes useful knowledge about the BB task. Its complete 32- or 128-episode log is collected before fitting. Teacher action draws and environment decisions are counted; the effort to design the teacher is not measured.

The reward learner starts without that teacher log. Each round it samples eight fresh episodes from its current policy and updates using their outcomes. Thus, halfway through a 128-episode schedule, it has collected 64 episodes while the clone has already received 128. Matching the final episode budget does not make every earlier checkpoint data-matched.

Cloning uses labels; reward learning uses outcomes of its own exploration. Even at the final matched episode count, they have different supervision and potentially different real-world acquisition costs. Report those differences instead of implying an equal-total-cost comparison.

Seeds 7, 19 and 41 are declared before training. Teacher collection uses offsets from those seeds, and each learner’s actions depend on its own policy. Equal seed labels are bookkeeping for paired comparisons, not a promise of identical trajectories or statistical independence.

Count rereading separately from collecting

Cloning maximizes the mean log likelihood of the demonstrated choices. Each full pass rereads the entire fixed log. Choose one or four cloning passes per scheduled round. Its fixed learning rate is 0.2.

The reward learner performs one REINFORCE update per round, at the selected rate. Each episode’s weight is its shaped return minus a fixed baseline of two. The derivative for a visited B logit is chosen-B indicator minus its preceding B probability, multiplied by that weight. Contributions are averaged over eight episodes and applied simultaneously. The baseline does not change the external task.

Optional math: compare the two gradients and their denominators

For N two-decision demonstrations, cloning’s log-likelihood ascent gradient for a state’s logit averages the terms for that state over all 2N decisions:

$$g_{\mathrm{clone}}(s)=\frac{1}{2N}\sum_{i,t:s_{it}=s}\left(\mathbf 1[a_{it}=B]-p(B\mid s)\right)$$

For eight on-policy episodes, the reward gradient averages episode-weighted contributions:

$$g_{\mathrm{reward}}(s)=\frac18\sum_{i,t:s_{it}=s}(G_i-2)\left(\mathbf 1[a_{it}=B]-p(B\mid s)\right)$$

G is shaped feedback, held fixed while differentiating. Demonstrated choices are labels; their rewards do not weight the cloning update. These objectives, scales and learning rates differ, so equal update counts would not imply equal optimization effort or effect.

With 128 episodes and four cloning passes per round, the final 16-round schedule gives the clone 64 updates and 64 × 256 = 16,384 fitting-decision uses per seed. The reward learner has 16 updates, 128 collected episodes and 256 fitting-decision uses. Both collected 256 environment action decisions, but only the clone used a knowledgeable teacher and repeatedly reread its labels.

The tables separate collected episodes, accepted updates, attempted updates and fitting decisions. They exclude diagnostic arithmetic from fitting counts. No wall-clock or energy equality is claimed.

Keep unsuccessful attempts in the record

A proposal outside the ±8 logit bound is rejected. Its previous policy is retained, the failure is recorded, and its already collected episodes and fitting work stay charged. That learner stops further training; other methods and seeds continue through the declared schedule. A failed run is never silently replaced with a fresh seed.

The chart and frozen comparison include every predeclared seed. If a numerical failure occurred, the retained policy is evaluated and labelled with its failed-training status. This is not a successfully completed full-budget run. Task failures such as AA are also counted separately from optimizer failures.

The best-seed column demonstrates how cherry-picking can inflate a claim. The main summary averages all three seeds and shows their range. That range is descriptive, not a confidence interval. Results on this small public simulator do not establish general algorithm superiority.

Freeze a common checkpoint

Choose one recorded round for both methods and every seed. The freeze action copies those policies, counters and statuses. Further training cannot change the displayed result. Selecting a different round in the control also does not replace the frozen result until you explicitly freeze again.

Every learning-curve point has already exposed exact public task performance. Choosing a checkpoint after looking at those curves is development selection. A copied checkpoint is stable, but stability alone does not turn previously inspected data into an untouched test.

The exact evaluator enumerates all four paths from the three conditional probabilities. For example, BB probability is P(B at Start) × P(B after B). AA, AB and BA are task failures even when their rewards are positive.

A second check samples 32 evaluation episodes per learner using public seeds 101, 103 and 107, paired across methods. Its success count can differ from the exact probability because of sampling variation. These are 64 additional evaluation decisions per learner and training seed, not fitting data. The seed list is reused and public; it is not reserved or independent real-world evaluation.

Write a comparison someone else can audit

Begin with 128 episodes, teacher probability 0.95, four cloning passes and reward-learning rate 0.2. Train the full cohort with bonus zero, freeze the final common round and capture it. Repeat with bonus eight. Keep both the shaped reward and the unchanged BB success result, including every seed and unsuccessful outcome.

Students can explain why the clone’s fitting count increases without new demonstrations, and why reward 18 can mean failure. Engineers can reproduce a gradient and verify counters at intermediate and final checkpoints. Managers can describe the cost of acquiring a teacher, the cost of exploration and what additional task or deployment evidence would be needed. Any learner can inspect the more technical branch.

A third capture can vary the episode budget or teacher quality. State all changed settings and avoid attributing a multi-variable difference to one algorithm. Three temporary reports download as JSON with frozen policies and declared budgets. They are practice evidence, not authenticated training history or independent implementation review. Human review and learner pilots remain pending.