Stabilize neural Q-learning.

A network estimates the value of each move

Imagine learning a route by trying moves and recording what happens. A Q value estimates the discounted return from a particular state and first action, followed by a continuation policy. It can be negative or larger than one: it is not a probability. The greedy policy chooses the largest predicted value; an exploration policy sometimes chooses another action to gather experience.

Entering midway? A state is the information the controller uses about the current situation. A reward is the immediate score after acting. A return combines rewards over time. A gradient tells us how a small weight change affects a loss. Here the exact location is sufficient to describe the grid dynamics; no hidden velocity or past observation is needed.

The lab replaces a Q table with a small neural network. Nine inputs indicate the current cell: exactly one is 1 and the rest are 0. Eight tanh hidden units feed four linear outputs, one per direction. Counting weights and biases gives 9×8 + 8 + 8×4 + 4 = 116 parameters. A change to shared hidden weights can alter several predictions at once.

Build a target from a real transition

Each receipt contains a source state s, action a, reward r, next state s′ and whether the task ended. For a nonterminal move, the fitted target is:

$$y=r+\gamma\max_{a'}Q_{\theta^-}(s',a'),\qquad \gamma=0.9$$

Read this as “reward now, plus 90% of the largest predicted next-action value.” The discount γ reduces the contribution of later rewards. The symbol θ⁻ names the weights of a separate target network. If a move costs −1 and that network’s largest next value is 4, the target is −1 + 0.9×4 = 2.6. If the online prediction is 1.6, prediction minus target is −1, so the loss pushes that action’s prediction upward locally.

A goal or hazard ends the task. Its target is just the received reward: +10 or −10. There is no continuation term. A boundary bump is different: it costs −1 and stays in a nonterminal cell, so it still bootstraps.

Bootstrapping means using an estimate inside the target for another estimate. No expert action labels or full optimal-value table enter training. The max chooses the value of a greedy next action even though the exploratory collector may choose another action. This is the off-policy aspect of this Q-learning update.

Differentiate the prediction, hold the target fixed

For a batch of B receipts, the objective is mean half-squared temporal-difference error:

$$L=\frac{1}{B}\sum_{i=1}^{B}\frac12\left(Q_\theta(s_i,a_i)-y_i\right)^2$$

The factor one-half makes the derivative of a squared error simply the error. Gradients flow through the online prediction Qθ, while y is held fixed for that update. All 116 parameter proposals use the same preceding network and are applied simultaneously. This is a semi-gradient update; differentiating through the target would define a different algorithm.

The lab uses ordinary gradient descent and a fixed learning rate. It is a small mechanism demonstration, not a reproduction of Atari-scale DQN, its convolutional architecture or all its optimizer and loss choices. The DQN paper studies experience replay and a separate target network in a much larger setting.

Why replay can help—and what it cannot add

Consecutive moves are related. Fitting only the newest transition can repeatedly chase one local part of the environment while changing shared predictions elsewhere. A replay buffer retains earlier transitions and samples across them. This can reuse scarce interactions and reduce the immediate sequential dependence within fitting batches, but it does not make the underlying experience independent or create missing action coverage.

The buffer holds the most recent 64 transitions. Each update draws eight with replacement, so duplicates are possible. The newest-only comparison repeats the current receipt eight times. Its averaged gradient equals fitting that receipt once; the repeated arithmetic preserves the eight-example processing count for comparison. Neither mode gets eight new environment interactions per update: each gets one.

Replay also mixes transitions collected by earlier policies. In this stationary grid, their recorded rewards and transitions remain valid. Changing dynamics would require reconsidering whether older experience still describes the task.

Slow the target rather than chasing every change

The target network starts as a copy of the online network. It remains fixed for twenty updates, then receives a complete copy after update twenty. The interval-one ablation copies after every update. Even there, the target is held constant within each gradient calculation.

A delayed target lets several updates work against the same predictor, but can become stale. Replay and delay are useful design choices, not proofs of convergence. Neural approximation, bootstrapping and off-policy data can interact badly. A small deterministic grid may solve under several settings; it need not visibly diverge whenever one stabilizer is removed. Keep that outcome rather than inventing a failure.

Distinguish termination from a collection limit

Training starts each episode at cell 0 and resets after a goal, hazard or 24 moves. A time limit is a collection boundary here, not a terminal state in the underlying task. The final receipt keeps its actual next cell and a nonterminal bootstrap; the following interaction begins a new episode. Relabelling the receipt’s next cell as the reset location would teach a transition that never happened.

This choice corresponds to learning the continuing task until a goal or hazard, while bounding collection. If a real task instead ends at a strict deadline, remaining time would need to be represented appropriately and terminal handling would differ.

Judge actions separately from the fitting objective

The rollout uses the current greedy policy without exploration or learning. Its return is the plain sum of its observed rewards, bounded to 24 moves. Q learning uses discounted targets, so this displayed undiscounted rollout return is a separate task diagnostic. Training goal counts also include exploratory choices; they do not measure the same policy as the greedy rollout.

Last-batch loss changes when sampled receipts or target predictions change. Comparing two endpoint losses as though they used one fixed test set is misleading. Inspect the route, whether it reaches the goal or hazard, and time-limit failures as well. The exposed training grid is reused for all of these diagnostics. It is not an independent final evaluation or evidence about unfamiliar environments.

Produce a controlled experiment report

For guided practice, choose one seed and train to 1,000 updates. Save the result. Restart with only replay changed, use the same budget, and compare. Repeat for the target-copy interval. Inspect one terminal receipt and one boundary bump to explain the target arithmetic.

For implementation practice, reproduce the model and stopped-target gradient; finite-difference each parameter while freezing target weights. Test target synchronization at updates 19 and 20, terminal suppression, time-limit bootstrapping and deterministic resumed training. Repeat ablations for seeds 7, 19 and 41. Preserve failures and explain what each comparison controls. Equal initial seeds do not force equal trajectories once learned policies differ.

For a decision brief, specify the interaction budget, update budget, goal outcomes and observed variation. Discuss what additional environments, seeds, reserved evaluation and operating constraints would be needed for deployment. One successful public rollout is not a release gate.

Six temporary captures export complete checkpoints, settings and endpoint histories. They support reproducible practice, not authenticated authorship or independent assessment. This lesson does not complete the separate action-learning project. Human review and learner pilots remain pending.