Know which policy produced the data.

Separate the collector from the policy you want to evaluate

A behavior policy chooses the actions that enter a log. A target policy is the policy whose return you want to estimate or improve. When they are the same, the data are on-policy. When they differ, the task is off-policy. Neither label alone tells you whether a particular estimator is reliable.

The lab uses two decisions with actions A and B. Both decisions have the same configured probability of B. Returns are 2 for AA, −1 for AB, 0 for BA and 10 for BB. There is no intermediate reward or discount, so return is simply the terminal score. The simulator knows every outcome; the estimates use only recorded trajectories and policy probabilities.

Why an ordinary log average answers a different question

If behavior always chooses A, every episode returns 2. That is evidence about AA, not about the target that always chooses B and reaches BB. Changing the target dropdown does not retroactively change the collected actions.

Even when behavior can choose B, its trajectory frequencies differ from the target’s. With independent B probability 0.5 at each decision, each of the four trajectories has probability 0.25. The expected return is (2−1+0+10)/4 = 2.75. A target always choosing B has expected return 10.

Reweight trajectories using the recorded policy

Let μ denote behavior and π denote target. For a recorded two-action trajectory, its importance weight is:

$$w=\frac{\pi(a_0\mid s_0)\pi(a_1\mid s_1)}{\mu(a_0\mid s_0)\mu(a_1\mid s_1)}$$

Here a₀ and a₁ are its actions; s₀ and s₁ are the decision states. The ratio says how much more or less probable that trajectory is under the target. The simulator’s transition and reward rules are unchanged between policies; they are not an extra source of reweighting in this experiment.

For behavior B probability 0.1, BB has probability 0.01. If the target always chooses B, its BB probability is 1, so the weight is 100. A BB return of 10 contributes 1,000 to the weighted-return sum. Other trajectories have target probability zero and contribute zero.

The ordinary importance-sampling estimate averages weighted returns over all n recorded episodes:

$$\widehat V=\frac{1}{n}\sum_{i=1}^n w_iG_i$$

Gᵢ is episode i’s return. If behavior equals target, all recorded weights are one and this becomes the ordinary sample mean. Under adequate support and the stated sampling assumptions, its expectation matches the target value; a single finite sample can still be very inaccurate.

Each row retains its actual collection probability. Collecting a new batch with a different behavior setting does not change the denominators of old rows. An adaptive collection protocol would also need to record the probabilities it actually used, not infer them from a later policy.

Missing support is different from rare evidence

If behavior assigns zero probability to an action the target may choose, some required trajectories can never appear. This violates the support requirement for this estimator. No amount of replay repairs it. The lab returns unavailable estimates instead of pretending the missing outcomes are zero.

The implementation checks support for every logged behavior policy. Some mixtures could be evaluated with a different mixture-based weighting scheme, but this lesson does not silently switch estimators. Its support warning describes the selected per-trajectory method.

With behavior B probability 0.01, BB is possible but has probability 0.0001, or one in ten thousand. Its weight under the always-B target is 10,000. A sample of 500 episodes can easily contain none. Ordinary importance sampling then returns zero for that sample; this is not evidence that the target’s true return is zero. Rare large contributions can produce high variance.

Normalization and clipping change the tradeoff

A self-normalized estimate divides the weighted-return sum by the sum of weights. It is undefined if that sum is zero. With only one BB match and an always-B target, it reports 10, but that does not turn one matched trajectory into strong general evidence about a more complex stochastic task. Self-normalization can have finite-sample bias.

The clipped comparison caps each weight at 10 before computing the ordinary mean. It reduces extreme contributions but changes the estimator and can introduce bias. Do not compare the smoother number as though it had the same guarantee as the unclipped estimator.

The effective-sample-size diagnostic is (sum of weights) squared, divided by the sum of squared weights. Equal weights give n; one nonzero weight gives one. It describes weight concentration. It is not a confidence interval, proof of independence or substitute for examining coverage and outcomes.

Replay changes computation, not collection

A replay buffer stores old transitions for reuse during learning. Repeated gradient or value updates can extract more from those transitions, but they do not create new environment interactions. The lab’s replay button only counts rereads; it does not implement a training algorithm. The dataset, estimates and unique-episode count remain unchanged.

Off-policy learning also includes methods that use sampled transitions to update values or policies. For example, Q-learning uses an observed reward and a bootstrapped target involving a maximizing next action. It is not the same operation as this whole-trajectory importance-sampling mean. The next practical lessons develop those neural updates separately.

Audit a changed policy

Save one supported on-policy case, one rare-action case and one unsupported case. For guided practice, explain each denominator and distinguish no observed matches from impossible matches. For implementation practice, reproduce the ratios for mixed behavior batches and test zero support, zero total weight and bounded replay. For a decision brief, state what action coverage, logged probabilities, environment stability and additional evaluation are needed before using the estimate.

Transfer case: a support workflow logged only automatic approvals, but a proposed policy sometimes requests additional evidence. The log contains no outcomes of those requests. Explain why repeating the approvals cannot estimate their effect, and what new evidence or justified model assumptions would be needed. A numerical output alone does not resolve the missing-data problem.

All cohorts here are exposed synthetic practice. Three captures are temporary and exportable. They do not establish independent implementation quality, real deployment readiness or learner effectiveness. Human review and learner pilots remain pending.