Explore before you decide.

Learn from consequences

In supervised learning, we supplied a desired answer for each input. In reinforcement learning, an agent chooses an action, receives a reward, and may enter a new state. Its choices affect what experience it collects. The objective is usually to increase expected accumulated reward, not to copy a labeled action on every turn.

The three-arm experiment is a bandit: each choice gives an immediate reward without changing a useful future state. Its hidden reward probabilities are 0.2, 0.5, and 0.8, matching the source example. Each outcome is either 0 or 1. A probability of 0.8 means an average of about eight successes per ten pulls over many repetitions, not a guarantee about the next ten.

Explore or exploit?

Exploitation chooses the action with the largest current estimate. Exploration tries alternatives to collect evidence. Our ε-greedy policy explores with probability ε and otherwise exploits. Exploration chooses uniformly among all three arms, including the current favorite.

With ε = 0.3, each arm receives 0.3 / 3 = 0.1 probability from exploration. The current favorite also receives the remaining 0.7, for a total probability of 0.8. ε is a design choice: too little exploration can miss a better option; continued random choices can reduce reward once good estimates are available. This lab keeps ε fixed until you change it.

At the start all estimates are zero. The tie rule chooses A. With ε = 0, the learner never discovers B or C: A’s estimate cannot fall below their initial zero. That is a property of this initialization and tie rule, not a theorem that every greedy learner always fails.

Three different value questions

An action value Q asks: “How much return do I expect if I take this action?” For this one-step bandit, we estimate it using the observed average reward for that arm. An untried arm is initialized to zero, which is a starting guess rather than an observation.

$Q_{\mathrm{new}}=Q_{\mathrm{old}}+\frac{r-Q_{\mathrm{old}}}{n}$

Here n counts this arm’s pulls, including the latest. If an arm has produced rewards 1, 0, 1, its estimate is 2/3. A fourth reward of 0 changes it to 2/3 + (0 − 2/3)/4 = 1/2.

A state value V asks: “How good is this situation under my policy?” In this bandit, the situation is always the same. Multiply each action estimate by the probability of choosing that action, then add.

$V=\sum_a\pi(a)Q(a),\qquad A(a)=Q(a)-V$

The advantage A compares an action with the policy’s baseline. A positive advantage means better than that estimated baseline, not certain success. In a multi-step task, Q and V refer to future accumulated rewards under specified behavior, not merely the next reward.

From a table to actor–critic learning

An actor selects actions using a policy. A critic estimates value and supplies feedback that can help improve the policy. Advantage estimates can guide the actor toward actions that did better than expected. Larger systems can represent the policy and critic with neural networks and update their weights using gradients.

Our bandit instead uses a small table and a programmed ε-greedy policy. Its Q, V, and A values make those ideas visible, but it does not implement a neural actor–critic optimizer. The next lesson introduces changing states and delayed consequences.