Learn without new interactions.

A log is not a simulator

Suppose you inherit records of a machine’s past choices. You can inspect each state, action, reward and next state, but cannot ask the machine to try new actions during training. That is the setting of offline reinforcement learning. The policy can change; the logged experience stays fixed.

Entering midway? A Q value predicts discounted future reward for an action. A Bellman target combines a recorded reward with a prediction about the next state. Behavior cloning fits recorded action choices without using their rewards. A greedy policy chooses the action with the largest predicted Q value.

Replaying old experience is not automatically fully offline learning: an online learner can mix replay with new collection. In this lesson, fitting never collects anything. Testing uses a separate known simulator and never appends its results to the training log.

Inspect a tiny route log

At Start you can Exit and receive +1 immediately, or Continue with reward zero to Branch. At Branch, Safe gives +2 and ends. The other action ends too, but its reward may be absent from the log. Discount the second reward by 0.9: Continue→Safe returns 1.8, versus Exit’s 1.

The default authored log contains 24 episodes: sixteen Exit choices and eight Continue→Safe paths. That gives 32 transition receipts. The other Branch action appears zero times. These repeated teaching records are not 24 independent real-world measurements.

The covered alternative changes two of those eight Branch choices to Other. A missing-state alternative contains only 24 Exit choices: neither Branch action nor the Continue transition is logged. Inspect all rows and counts before interpreting a learned value.

Two worlds agree on everything recorded

In one diagnostic world, Other gives −6. In the other it gives +6. Both worlds agree on every default logged receipt. Any deterministic learner receiving only this log and the same initialization must produce the same model in both worlds.

That model cannot be correct about the missing reward in both worlds just because it took more optimizer steps. Repeating the same log does not resolve the ambiguity. A penalty or initialization can express a preference under missing evidence; it does not turn the unrecorded result into an observation.

When the covered log includes Other, the two source worlds produce different recorded rewards. That is an actual evidence difference. Evaluating a covered log’s policy in the opposite world is also a changed-environment test, not an ordinary replay of its source conditions.

Fit a transparent cloning baseline

At each state, count how often each action appears. Dividing by that state’s total gives the exact maximum-likelihood categorical clone. At Start, Continue has probability 8/24 = 1/3. At Branch in the default log, Other has probability zero. The clone follows Safe whenever it continues.

Its exact expected discounted return is two-thirds times 1 plus one-third times 1.8, or about 1.2667. This baseline preserves the recorded mixture; it does not select the best recorded reward. It also does not become a greedy majority-vote policy without an explicit change in extraction rule.

At an entirely absent state, the frequency denominator is zero. The clone is undefined there and would abstain if reached. The lab reports this separately from successful termination. A zero reward assigned to stopping is accounting, not a success claim.

Watch a missing value enter the target

Both Q learners store four numbers, one for each state/action pair. By default, all start at zero except Other at Branch, which starts at 6. This deliberately optimistic guess makes the failure visible. It is not inferred from data and is not a neural generalization effect.

For a nonterminal receipt, the target is:

$$y=r+0.9\max_{a'}Q_{\mathrm{before}}(s',a')$$

A terminal receipt’s target is just its recorded reward. On the first Continue receipt, the target is 0 + 0.9 × max(0,6) = 5.4. The maximum includes the unrecorded action. Thus its unsupported optimism can influence a value for a transition that was recorded.

Plain Q-learning minimizes mean half-squared error between the four selected values and their targets. Other has no direct error term in the default log, so its value stays at 6. Safe moves toward 2. Continue moves toward 5.4. Eventually the greedy policy continues and selects Other—even though fitting error on the logged transitions can be small.

The exact simulator reveals discounted return −5.4 in the hazard world and +5.4 in the bonus world. The model did not discover which world it occupies. Try zero initialization to see how the same evidence can lead to a different learned policy.

Specify the conservative objective

The conservative candidate adds the discrete log-sum-exp regularizer from CQL(H), Equation 4:

$$L=\frac1N\sum_{i=1}^{N}\frac{(Q(s_i,a_i)-y_i)^2}{2}+\alpha\frac1N\sum_{i=1}^{N}\left[\log\sum_a e^{Q(s_i,a)}-Q(s_i,a_i)\right]$$

The first term fits stopped Bellman targets. The second penalizes large action values relative to logged actions at the same state. α sets its weight. We use a tabular, full-batch gradient version, not a full deep benchmark reproduction or a certified lower-bound procedure.

Optional math: how the penalty moves a value

For one logged row, the penalty’s derivative for action b is α times softmax(Q) for b minus an indicator that b was logged. Average these contributions over all rows. This softmax describes a gradient weighting; the deployed Q policies here remain greedy.

At a state with equal Q values, each softmax weight is 0.5. If every receipt uses Safe, the penalty increases Safe and decreases Other under gradient descent. If no receipt visits that state, it contributes nothing. The complete update also includes the Bellman-error gradient.

The scale is important: we average over receipts, so more common states contribute more. In the default log, Start accounts for 24/32 terms and Branch for 8/32. We do not silently give both states equal weight.

The implementation subtracts the largest Q before exponentiating to compute log-sum-exp stably. All four gradients use one preceding table. Targets stay fixed while differentiating, then are recomputed from the accepted table for the next update. Differentiating through the target would be a different method.

Conservatism does not create coverage

At a logged Branch state, the penalty can push the unsupported action below Safe. In the hazard world, that can avoid the failure. In the bonus world, the same preference can miss a real opportunity. The missing-action log cannot tell the learner which trade-off it is making.

Try the missing-state log. Branch contributes no objective term, so both Branch values stay at initialization. A conservative penalty applied to logged states is not a complete answer to absent states, incorrect rewards, changing environments or operating constraints.

A finite optimization run need not produce a lower bound on every true value. “Conservative” names the specified regularizer and intended bias; it does not certify the resulting policy as safe. The lab exposes the values and simulator outcomes so you can inspect those limits.

Freeze, compare and report

Both candidates begin with the same four values and use the same fixed receipts, learning rate and update count. Their targets can diverge as their values diverge. Each paired action commits only if both candidates succeed. Runs cap at 500 full-batch updates; each CPU click takes one or ten. Values outside ±50 reject a proposal rather than clipping it into a different update.

The bar chart uses fixed −8 to +8 axes, with explicit visual-clipping counts and exact numerical tables. The learned values, fitting losses and simulator returns are separate evidence. Objectives with different α values are not interchangeable task scores.

The evaluation button copies the current models and log, then exactly enumerates all terminal paths under the selected world. Later training leaves that evaluation unchanged. Click again to replace it explicitly. This known-model diagnostic is not offline policy evaluation estimated from the log and is not reserved testing.

Guided learners can reproduce the 5.4 bootstrap and explain the missing reward. Implementation learners can check all four gradients with fixed targets, then compare missing-action and missing-state behavior. Decision learners can write a coverage audit describing what remains unknown, which outcomes would need logging, and whether an external operating restriction is justified. Keep both successful and failed comparisons.

Capture up to three frozen comparisons and download the JSON. More receipt uses mean more fitting work, not more environment interactions or independent evidence. These temporary experiments do not authenticate implementation authorship or replace independent project assessment. Human review and learner pilots remain pending.