Increase the probability of useful actions.

Learn probabilities directly from experience

A stochastic policy maps a state to action probabilities. Unlike behavior cloning, it need not receive an expert’s action label. It samples actions, observes the resulting return, and adjusts the probabilities of the choices it made. Unlike Q-learning, this lesson updates the policy directly rather than fitting action values and taking their maximum.

Entering midway? A return is the combined reward from an episode. A logit is an unrestricted score converted into a probability. A gradient describes how a small parameter change affects an objective. Gradient ascent adds the gradient to increase an objective; gradient descent subtracts it to decrease a loss.

Our two-choice task has three decision states: Start, After A and After B. Each has its own logit z. Action A has reference score zero and B has score z. A two-action softmax gives:

$$p(B\mid s)=\frac{e^{z_s}}{1+e^{z_s}}=\operatorname{sigmoid}(z_s)$$

At z = 0, B has probability 0.5. Increasing z increases B’s probability. There are only three learned parameters, with no hidden network or critic. This deliberately small policy exposes the learning arithmetic.

Credit the choices in a sampled episode

Every episode makes two choices and receives one terminal return: AA gives 2, AB gives −1, BA gives 0 and BB gives 10. There is no discount or intermediate reward. The return therefore credits both choices. The second decision has a separate parameter for each first-action branch.

For an action sampled at a state with B probability p, the derivative of its log probability with respect to that state’s B logit is:

$$\frac{\partial\log\pi(a\mid s)}{\partial z_s}=\mathbf{1}[a=B]-p(B\mid s)$$

The indicator is 1 if B was chosen and 0 if A was chosen. It is bookkeeping, not another learned quantity. At p = 0.5, a B action contributes +0.5 and an A action contributes −0.5 before reward weighting.

Multiply by the episode’s return G minus a baseline b. Average over the batch, using zero for a state not visited by an episode:

$$g_s=\frac1N\sum_{i=1}^{N}(G_i-b)\sum_{t:s_{i,t}=s}\bigl(\mathbf{1}[a_{i,t}=B]-p(B\mid s)\bigr),\qquad z_s\leftarrow z_s+\alpha g_s$$

N is the number of episodes in the batch and α is the learning rate. Every contribution uses the same policy that collected that batch. Updates to all three logits are simultaneous. This is an episodic REINFORCE estimator; Williams’s original paper develops the reinforcement-weighted probability-update family.

Work through one real update

Suppose both choices are B under a uniform policy, giving return 10. With baseline 2, each visited state’s contribution is (10−2)×(1−0.5) = 4. With a single-episode batch and learning rate 0.05, Start and After B logits rise from zero to 0.2. The After A logit stays zero because that state was not visited. The new B probability in the two updated states is about 0.5498.

For an AB episode, return is −1. At the same initial probabilities and baseline, the Start contribution is (−1−2)×(0−0.5) = +1.5: it pushes away from the sampled A choice. At After A, the contribution is (−1−2)×(1−0.5) = −1.5: it pushes away from B there. A negative return-minus-baseline reverses the usual reinforcement direction.

These are sampled estimates. One unlucky outcome can discourage an otherwise useful choice. Learning a good policy depends on repeated experience, step size and task structure; a single update does not guarantee improvement.

A baseline changes variance without changing the expected gradient

The lab uses a constant baseline chosen before collection. It does not change the environment reward. Subtracting an action-independent baseline preserves the expected policy gradient because the probability-weighted log-probability derivatives sum to zero at each state.

That cancellation is an expectation over possible actions, not a claim that every sampled update is unchanged. Baselines 0, 2 and 5 can produce different individual updates and different variances. A poorly chosen baseline can increase variance. The lab enumerates all four possible paths to show the exact mean and variance of each single-episode gradient component at the current policy.

At the uniform policy with baseline zero, the mean gradient is [1.125, −0.375, 1.25]. It is not the gradient of any one episode; it averages their contributions using their probabilities. Changing the constant baseline leaves this mean unchanged. The diagnostics compute it separately and never use it to train the policy.

A baseline derived from the same sampled action or naively from the same batch needs additional care. This lesson deliberately avoids those dependencies. A learned state-value baseline is introduced separately with actor–critic methods.

More episodes reduce one kind of noise—and cost more

For independent episodes drawn from a frozen policy, averaging N gradient estimates divides their variance by N. It does not guarantee a particular sampled update will be closer to the truth, nor does it describe uncertainty across a whole learning process where the policy changes after each update.

Batch sizes 1, 8 and 32 collect different amounts of data per update. Compare batch 1 for 160 updates, batch 8 for 20, and batch 32 for 5: each collects 160 episodes per seed, but they change the policy a different number of times. Equal interaction budgets and equal update budgets answer different questions. Each episode makes two environment choices.

Old batches were drawn under old probabilities. Reusing them as though they were fresh on-policy samples generally changes the estimator. The lesson does not replay old data or silently introduce an importance-weight correction.

Separate a lucky return from a stronger policy

The three public seeds begin with exactly the same logits and differ only in sampling. The plot shows their exact expected-return curves and their mean. Expected return is calculated by summing probability × return over all four paths. It is a diagnostic available because this task is tiny and fully known, not privileged information supplied to training.

The most recent sampled batch describes the preceding policy, while the current expected return describes the updated policy. A sampled batch can be luckier or unluckier than its expectation. Read the individual returns and the gradient trace rather than assuming the plotted mean explains every episode.

Three fixed seeds are a small public teaching comparison, not a confidence interval, reserved test or proof of general performance. Keep all runs, including weak ones; repeatedly selecting settings on these seeds makes them development evidence.

Produce a learning-evidence report

Guided learners can explain one positive and one negative update, identify unvisited states, and compare the policy probabilities before and after. Implementation learners can reproduce the estimator from stored behavior probabilities and check expected gradients against finite differences of the separately enumerated objective. Decision learners can compare outcomes under equal episode budgets and explain what the additional updates and sampling cost.

Capture a small-batch run, a larger-batch run at a matched episode budget, and a baseline comparison. Report every seed, the chosen budget, observed variability and limitations of this task. Exported captures contain the selected configuration, all three models, their last pre-update policies and curves. These are temporary practice artifacts, not independent implementation assessments. Human review and learner pilots remain pending.