Use a critic to guide learning.

Give the policy a learned reference

The actor chooses actions through a probability distribution. The critic estimates the return expected from a state under the policy. They have different jobs: a value is not an action probability, and a confident critic is not necessarily an accurate one.

Entering midway? A return combines future rewards. Bootstrapping uses an estimate inside a training target. A gradient describes how parameter changes affect an objective. The preceding policy-gradient lesson used complete sampled returns. Here the learner updates after a single transition, using the critic to estimate what follows.

The grid has nine cells. Start is 0, goal 8 gives +10, hazard 5 gives −10, and all other moves cost −1. Boundary bumps remain in the same cell and incur that cost. A one-hot location feeds two separate linear heads: four action logits per state and one value per state. There are 36 actor and nine critic coefficients. This is equivalent to state-indexed tables; it does not demonstrate generalization through a deep shared representation.

Form a target, then compare it with the current value

For a nonterminal receipt, construct a one-step value target and temporal-difference error:

$$y=r+\gamma V_w(s'),\qquad \delta=y-V_w(s),\qquad\gamma=0.9$$

Here r is the observed reward, s is the source state, s′ is the next state and w names the critic’s parameters. The error δ is used as an advantage estimate: a signal about whether this transition looks better or worse than the critic expected at the source.

For reward −1, current value 2 and next value 4, the target is −1 + 0.9×4 = 2.6. The error is 2.6−2 = 0.6. A costly immediate move can still look useful if it leads to sufficiently promising future rewards.

For a terminal goal or hazard, y is just r. There is no later return to estimate. The terminal cells’ stored value coefficients do not contribute to the target.

The exact advantage of an action compares its expected return with the state’s value under the policy. One sampled TD error is not that exact quantity. Even with an accurate critic it can be noisy; with an inaccurate critic it can also be biased.

Update the two heads from one preceding checkpoint

The actor uses gradient ascent on log action probability, weighted by the stopped advantage estimate:

$$\Delta\theta=\alpha_\pi\,\gamma^t\,\delta\,\nabla_\theta\log\pi_\theta(a\mid s)$$

The learning rate απ sets the step size. The factor γᵗ discounts later moves relative to the episode start; t begins at zero after each collection reset. In the four-action softmax, each local logit’s derivative is its chosen-action indicator minus its old probability. A positive δ increases the sampled action’s relative preference, while a negative δ discourages it.

The critic fits the stopped target with half-squared error:

$$L_V=\tfrac12(V_w(s)-y)^2,\qquad \nabla_w L_V=(V_w(s)-y)\nabla_w V_w(s)$$

For the one-hot critic, only the source state’s coefficient changes. Subtracting its gradient moves the value toward the target. The actor’s advantage and the critic’s target stay fixed while taking these derivatives. Differentiating through the bootstrap would implement a different update.

Both proposals use the same preceding model. The actor does not first wait for the critic update and then recalculate δ. The lab records pre-update probabilities, values, targets and gradients so this ordering can be checked directly. Larger actor–critic architectures may share layers, but these heads are deliberately separate.

A faulty bootstrap is different from a baseline

Before training, all values are zero. A nonterminal −1 move therefore has target −1 and advantage −1. In the read-only probe, add +5 to the next-state estimate: the target becomes −1 + 0.9×5 = 3.5. The actor’s proposed update now reinforces the same costly transition. Nothing about its reward or actual destination changed.

The training setting can inject this offset into every nonterminal next-value estimate. It is a deliberate failure experiment, not a second legitimate reward function. A terminal target ignores it. The read-only probe is separate and never modifies training settings or weights.

In the preceding REINFORCE lesson, subtracting an action-independent baseline from complete sampled returns preserved the expected gradient. Altering the next-state estimate inside a bootstrapped target is a different operation. It can change which actions appear useful and create a misleading learning signal. Even without an injected fault, a learned critic can lag behind a changing policy.

Compare outcomes using their actual definitions

Training samples from the softmax policy. The manual rollout instead picks the largest probability at each step, with fixed tie-breaking. Its displayed return is the undiscounted reward sum. Those are two different policies and reporting conventions, so they need not give the same outcomes.

A separate diagnostic repeatedly applies the exact transition rules to evaluate the frozen stochastic policy for 24 moves. It reports goal and hazard probabilities and expected discounted returns, with zero continuation value after the horizon. It never supplies targets or gradients to training. This small known simulator makes exact evaluation possible; real environments usually require additional evidence and uncertainty estimates.

The critic bootstraps at a 24-move collection limit. The receipt keeps its actual next cell and remains nonterminal; the next collection begins again at 0. Therefore the finite-horizon diagnostic is not an exact ground truth for the critic’s continuing target. Its difference from the critic mixes horizon choices with estimation error. A real task that terminates at a deadline would require corresponding terminal and state definitions.

The online changing critic, finite collection and constant step sizes here do not establish an unbiased full infinite-horizon gradient or convergence. Policy-gradient theory with function approximation gives a deeper treatment with explicit assumptions; this teaching implementation does not claim its guarantees merely by using an actor and critic.

Produce a critic audit

Guided learners can explain the zero-value −1 versus +3.5 counterexample, then inspect a terminal move. Trace one actual training update and distinguish the actor ascent direction from the critic descent direction.

Implementation learners can freeze the recorded target and advantage and finite-difference the two objectives separately. Compare no-fault and faulty runs at the same seed, actor/critic rates and transition budget, then repeat seeds 19 and 41. Keep the failures and account for changed trajectories once policies differ.

Decision learners can compare actual terminal outcomes, stochastic goal/hazard probabilities and the critic’s numerical estimates. State what is development evidence and what additional tasks, frozen evaluation and review would be needed before deployment. A high value estimate, a low TD loss or one successful greedy route is not sufficient by itself.

Three temporary captures retain complete checkpoints and settings, with explicit practice provenance. They do not certify independent implementation skill. Human review and learner pilots remain pending.