Learn from delayed rewards.

Good actions may pay off later

A useful first move in a maze may receive no positive reward. Credit assignment asks which earlier choices contributed to the eventual outcome. Looking only at immediate rewards can miss the value of a sequence.

The return combines rewards over time. A discount γ between zero and one reduces the contribution of more distant rewards:

$G_t=r_{t+1}+\gamma r_{t+2}+\gamma^2r_{t+3}+\cdots$

For rewards 0, 0, 1 with γ = 0.9, the return is 0 + 0.9×0 + 0.9²×1 = 0.81. With γ = 0, only the next reward matters. Discounting is part of the chosen objective; it is not the same thing as the learning rate α, which controls the size of an update.

Let later value teach an earlier choice

Our Q-learning table stores four action values for every grid square. After an action, we observe its reward and next state. For a nonterminal next state, the target combines that reward with the largest estimated next-state value:

$\text{target}=r+\gamma\max_{a'}Q(s',a')$ $Q(s,a)\leftarrow Q(s,a)+\alpha\,[\text{target}-Q(s,a)]$

At the terminal goal, the target is just the reward: there are no later actions. Elsewhere the target uses an estimate. This is called bootstrapping. It can spread information about a distant goal backward through visited transitions over many updates; one successful episode does not instantly teach every route.

The exploratory behavior collects experience while the update’s maximum looks toward greedy next actions. Those are different roles. Changing γ changes the objective; changing ε changes data collection. Sufficient exploration and suitable learning-rate conditions matter for theoretical convergence. This bounded demonstration does not claim that a fixed number of episodes or a fixed learning rate always converges.

The 40-step cap is a practical reset for the demo. It truncates experience rather than representing a natural terminal grid state, so the last update still includes the next-state estimate. The displayed episode score is an undiscounted sum; it is not the discounted Q target.

A high score can be the wrong success

With the intended rule, ordinary moves cost 0.1 and reaching the goal earns 5. With the flawed rule, repeatedly entering the bonus square earns extra reward. A learned loop can then be attractive even though it never reaches G. This is reward hacking: exploiting the measurement in a way that fails the intended task.

The source’s cleaning robot illustrates the same problem. Rewarding dirt pickup can encourage creating more dirt if the environment allows it. Possible changes include measuring the room’s final cleanliness, preventing repeat credit for the same dirt, and testing unintended strategies. Every fix needs evaluation; changing one reward can introduce another loophole.

The grid’s independent check asks whether the frozen policy reaches G. It stops when a state repeats because a deterministic policy in this deterministic grid would repeat the same cycle. The displayed prefix reward therefore does not represent the cycle’s infinite return. Training goal counts also differ from greedy-policy performance because exploration can accidentally reach the goal.

Apply it to a card puzzle

The source asks you to define a reinforcement-learning task for FreeCell. Before opening the example, describe the state, legal actions, terminal reward, and credit-assignment difficulty.

Compare your task definition with an example

A state records the ordered cards in tableau columns, free cells, and foundations. An action is a legal card move according to the game’s rules; legality belongs to the environment, not to the policy’s guess. A terminal win can give +1, with 0 for an unsuccessful stopped attempt. You must specify how attempts stop and whether step costs or discounting are part of the objective.

A useful early move may produce no immediate reward and only enable a win many moves later. That makes delayed credit and exploration difficult. Rewarding moves to foundations might help but can change behavior, so compare eventual solve rates on reserved deals. The small grid table does not establish that the agent can solve FreeCell.