Limit a policy update.

Reusing a batch changes the question

A policy-gradient update uses experience collected by a particular policy. After changing the policy, that same batch is no longer fresh experience from the candidate. To reuse it, we must keep track of how the sampled action probabilities have changed.

Entering midway? A policy assigns probabilities to actions. An advantage estimate says how useful an action’s outcome looked relative to a reference. A surrogate objective is an optimization proxy; improving it is not the same as proving that task performance improved. This lesson uses complete returns minus a fixed baseline as frozen advantage estimates.

PPO alternates collection and policy optimization. Here we isolate one clipped-objective phase. Three logits describe a two-decision policy, with terminal returns AA: 2, AB: −1, BA: 0 and BB: 10. This is not a complete implementation with a learned critic, generalized advantage estimation, entropy bonuses or repeated collection rounds.

Keep the old probability in the denominator

For each sampled decision, compute the probability ratio:

$$r(\theta)=\frac{\pi_\theta(a\mid s)}{\pi_{\mathrm{old}}(a\mid s)}$$

The numerator is the current candidate’s probability of the action actually taken. The denominator is its probability under the policy that collected the batch. If B’s probability rises from 0.5 to 0.6, a sampled B action has ratio 1.2. A sampled A action at that same state has ratio 0.4/0.5 = 0.8. Do not use the B probability when the recorded choice was A.

The old denominator remains fixed across every optimization step on this batch. Resetting it to the newest candidate would hide accumulated policy movement and change the objective. Both candidates in the lab receive identical old policies, episodes and frozen advantage estimates.

This is a per-decision ratio. It is not the product of all trajectory ratios used in some off-policy evaluation estimators. The surrogate averages over the batch’s two decisions per episode.

Take the smaller of two terms

The raw surrogate contribution is ratio times advantage, rA. The clipped contribution is:

$$L^{\mathrm{clip}}=\min\left(rA,\operatorname{clip}(r,1-\epsilon,1+\epsilon)A\right)$$

The clip function limits a number to the interval from 1−ε to 1+ε. With ε = 0.2, those endpoints are 0.8 and 1.2. The minimum is applied after multiplying by the advantage, which is why its sign matters.

If A = +5 and r = 1.5, the raw term is 7.5 and the bounded-ratio term is 6. The objective uses 6 and becomes flat in that improving direction. There is no extra reward from this term for moving r farther upward.

If A = +5 and r = 0.5, the raw term is 2.5 and the bounded-ratio term is 4. The minimum is 2.5. This adverse move is still penalized; the gradient has not disappeared just because the ratio is outside the band.

For A = −5, the picture reverses. At r = 0.5, the minimum is −4, creating a plateau below the lower boundary. At r = 1.5, the minimum is −7.5 and still penalizes increasing an action with negative advantage. The probe graph lets you inspect both signs and the zero-advantage case.

Follow the real gradient, not a clamped probability

Away from a plateau, the derivative includes the frozen advantage, the probability ratio and the derivative of the current log probability. For a binary policy’s B logit, that last derivative is chosen-B indicator minus current B probability. On the improving-side plateau, the clipped term contributes zero gradient. Contributions are averaged over all sampled decisions and applied simultaneously to the three logits.

At the exact boundary the objective has a corner. This implementation selects the unclipped branch’s subgradient. The finite-difference checks use points away from those corners so an ambiguous boundary derivative does not masquerade as a numerical failure.

Clipping is applied to the objective, not to the proposed parameters or probabilities. The lab’s separate ±8 logit limit is a numerical workload safeguard; crossing it rejects the requested comparison update. It is not PPO’s clipping mechanism.

Make a large step overshoot

Use uniform initial probabilities, seed 19, one sampled episode, baseline zero, learning rate one and ε = 0.2. The sampled path is BB, with return 10. Both candidates have ratio one before the first step, so raw and clipped gradients are identical. Averaging over the two decisions gives gradient 2.5 at each visited state’s logit.

The update moves those logits to 2.5. B’s probability becomes about 0.9241, and the sampled ratio becomes approximately 1.8483—well beyond 1.2. The clipped objective is now flat for those two terms, but it did not reverse the overshoot. Other samples or shared parameters can also move a policy after a particular term becomes flat.

This is why a clip parameter is not a hard trust-region radius, a KL constraint or a safety guarantee. Additional monitoring or stopping rules are separate mechanisms; this lab does not silently add them.

Inspect more than the sampled objective

The comparison shows the objective actually optimized by each candidate, exact simulator expected return, probability ratios and old-to-current KL divergence. Raw and clipped objectives differ by definition, so their absolute values are not interchangeable success scores.

KL measures a directional difference between action distributions, in nats. The diagnostic averages each state’s old-to-current action KL using the old policy’s normalized occupancy across the two decisions: half the weight at Start, and the remaining half distributed over After A and After B according to the old first choice. It includes all actions, even those absent from the sampled batch. It is a known-model diagnostic, not a collected performance result or an optimizer constraint.

Exact expected return enumerates the four terminal paths under the candidate. It never supplies gradients. A small, lucky batch may give a poor picture of other paths. Clipping can limit some incentives while leaving sampling error and coverage problems unresolved.

Repeated optimization increases fitting work but does not collect more episodes. The lab displays both counts. Resetting to the same seed reproduces the same public batch, rather than creating independent evidence.

Produce a policy-change audit

Guided learners can explain each advantage sign and run the overshoot example. Implementation learners can reproduce the frozen per-decision denominators, gradients and simultaneous updates, then test plateau and non-plateau regions separately. Decision learners can compare outcome evidence, distribution change and collection budgets before interpreting clipping as a reliability measure.

Capture a normal comparison, the overshoot example and a run with a different seed or tilted initial policy. Record all choices and failed proposals. The three temporary captures retain both candidates and their shared source batch, settings and accepted endpoints. They do not establish independent implementation proficiency or deployment readiness. Human review and learner pilots remain pending.