Learn AI.
PROJECT 10 · REWARD-LEARNING EXTENSION

Train an agent.
Account for every trial.

Train a small Q-network, compare three seeds against authored and exact-search baselines, then freeze a protocol and examine failures. You can complete the guided experiment without writing code. Implementation and decision paths use the same evidence with different report emphasis.

Report: draft. Storage: loading. Work saves on this device and travels in full progress backups. Known evaluation openings: 0. Five separate objective transfer cases follow the report; human review remains pending. This extension does not change existing project achievements.

Follow the steps, predict a failure and explain one receipt sequence in everyday language.

1. Set a budget and train three seeds

The 3×3 grid starts training at cell 0. Goal 8 gives +10, hazard 5 gives −10 and ends the episode; other moves cost −1. A boundary bump stays put. A 116-parameter network maps location to four action values. Discount 0.9 downweights later rewards. Every update collects one transition and fits eight samples; a replay sample can repeat a receipt.

0 start12345 hazard678 goal
Commands: Up, Right, Down, Left. Later, a changed-rule trial reverses horizontal command meanings.

Choose an update budget (for example 1,000) before inspecting comparisons. Train and capture seed 7, then 19 and 41 at exactly the same budget/settings. Keep failed runs. You can save two complete three-seed cohorts; choose one cohort using development evidence before opening evaluation.

Settings for a new run

Saved active run: seed 7, replay, target every 20, rate 0.01, exploration 0.3. 0/2,000 updates; 0 collected transitions; 0 fitting-example uses.

CPU worker batches cap at 25 updates; each request commits together at up to 200. Cancellation, timeout, hidden/offscreen controls and navigation retain the preceding saved checkpoint. Training resets at 24 moves but keeps that receipt nonterminal; goal/hazard receipts suppress bootstrap. The last fitting loss is undefined before training and is not an evaluation score.

Choose a training budget before comparing seeds.

Inspect current development starts 0 and 3

Learned greedy policy

0/2 goals; 1 hazards; 1 move limits; 0 planning cutoffs. Mean return -18.00 across all starts.

Mean execution moves 13.50, model calls 13.50, planning edge queries 0.00. Mean moves among successes: undefined (no successes).

Inspect every start and exact receipt
Start 0: hazard

Return -12; 3 executed moves; 3 model calls; 0 search expansions; 0 planning edge queries.

  1. Cell 0, command Right → 1; reward -1; continue.
  2. Cell 1, command Right → 2; reward -1; continue.
  3. Cell 2, command Down → 5; reward -10; hazard.
Start 3: time-limit

Return -24; 24 executed moves; 24 model calls; 0 search expansions; 0 planning edge queries.

  1. Cell 3, command Left → 3; reward -1; boundary bump.
  2. Cell 3, command Left → 3; reward -1; boundary bump.
  3. Cell 3, command Left → 3; reward -1; boundary bump.
  4. Cell 3, command Left → 3; reward -1; boundary bump.
  5. Cell 3, command Left → 3; reward -1; boundary bump.
  6. Cell 3, command Left → 3; reward -1; boundary bump.
  7. Cell 3, command Left → 3; reward -1; boundary bump.
  8. Cell 3, command Left → 3; reward -1; boundary bump.
  9. Cell 3, command Left → 3; reward -1; boundary bump.
  10. Cell 3, command Left → 3; reward -1; boundary bump.
  11. Cell 3, command Left → 3; reward -1; boundary bump.
  12. Cell 3, command Left → 3; reward -1; boundary bump.
  13. Cell 3, command Left → 3; reward -1; boundary bump.
  14. Cell 3, command Left → 3; reward -1; boundary bump.
  15. Cell 3, command Left → 3; reward -1; boundary bump.
  16. Cell 3, command Left → 3; reward -1; boundary bump.
  17. Cell 3, command Left → 3; reward -1; boundary bump.
  18. Cell 3, command Left → 3; reward -1; boundary bump.
  19. Cell 3, command Left → 3; reward -1; boundary bump.
  20. Cell 3, command Left → 3; reward -1; boundary bump.
  21. Cell 3, command Left → 3; reward -1; boundary bump.
  22. Cell 3, command Left → 3; reward -1; boundary bump.
  23. Cell 3, command Left → 3; reward -1; boundary bump.
  24. Cell 3, command Left → 3; reward -1; boundary bump.

Authored standard-rule policy

2/2 goals; 0 hazards; 0 move limits; 0 planning cutoffs. Mean return 7.50 across all starts.

Mean execution moves 3.50, model calls 0.00, planning edge queries 0.00. Mean moves among successes: 3.50.

Inspect every start and exact receipt
Start 0: goal

Return 7; 4 executed moves; 0 model calls; 0 search expansions; 0 planning edge queries.

  1. Cell 0, command Down → 3; reward -1; continue.
  2. Cell 3, command Down → 6; reward -1; continue.
  3. Cell 6, command Right → 7; reward -1; continue.
  4. Cell 7, command Right → 8; reward 10; goal.
Start 3: goal

Return 8; 3 executed moves; 0 model calls; 0 search expansions; 0 planning edge queries.

  1. Cell 3, command Down → 6; reward -1; continue.
  2. Cell 6, command Right → 7; reward -1; continue.
  3. Cell 7, command Right → 8; reward 10; goal.

Full-model breadth-first search

2/2 goals; 0 hazards; 0 move limits; 0 planning cutoffs. Mean return 7.50 across all starts.

Mean execution moves 3.50, model calls 0.00, planning edge queries 28.00. Mean moves among successes: 3.50.

Inspect every start and exact receipt
Start 0: goal

Return 7; 4 executed moves; 0 model calls; 7 search expansions; 28 planning edge queries.

  1. Cell 0, command Right → 1; reward -1; continue.
  2. Cell 1, command Down → 4; reward -1; continue.
  3. Cell 4, command Down → 7; reward -1; continue.
  4. Cell 7, command Right → 8; reward 10; goal.
Start 3: goal

Return 8; 3 executed moves; 0 model calls; 7 search expansions; 28 planning edge queries.

  1. Cell 3, command Right → 4; reward -1; continue.
  2. Cell 4, command Down → 7; reward -1; continue.
  3. Cell 7, command Right → 8; reward 10; goal.

2. Capture models and freeze the protocol

0/6 saved captures. Check exactly three with distinct seeds and identical settings/update budgets. Scores recompute from numeric parameters; declared training history, seed provenance and authorship are not authenticated.

Evaluation budgets to freeze

Each search expansion queries four edges. Exact search knows the entire rule model; the learned controller sees its location. The authored policy knows standard safe moves but does not adapt to horizontal reversal. Equal move limits do not mean equal computation; training, verification and runtime overhead are excluded from controller counts. A search cutoff is not proof that no route exists.

4. Explain and submit your evidence

Identify goal, hazard, boundary bumps and move/search budgets. Explain why reward, task success and legal actions are different.

Name all three captures, update counts, replay, target interval and exploration. Count collected transitions and fitting-example uses. Retain failed seeds.

Compare authored, learned and search controllers. Explain the search model-access advantage and why edge queries, model calls and moves are not equal-time units.

Report all starts for all three seeds. Distinguish locally reserved initial conditions from unseen states, and disclose any repeated evaluation.

Use an exact receipt sequence to explain a hazard, loop, cutoff or changed-rule failure. Propose one controlled intervention, including when all current runs succeed.

Explain whether evidence supports further testing or use, which conditions remain untested, and what independent implementation/human review would require.

Code is inert text. The implementation path needs code or artifact instructions, but presence does not establish correctness or independent proficiency.

Submission checks matched captured evidence, an opened evaluation and required report fields. It does not grade reasoning, code quality, reliability or mastery. The five separate objective cases below check transfer. Human review and learner pilots remain pending.

  • Freeze three trained seeds with matched settings and update budgets.
  • Explicitly open the frozen reserved-start and changed-rule evaluations.
  • Write the task report field.
  • Write the training report field.
  • Write the baselines report field.
  • Write the evaluation report field.
  • Write the failures report field.
  • Write the limits report field.

This report is a review artifact. Use My Work for a full editable progress backup and restore.

5. Try different reward-learning problems independently

These five fixed cases use different data and separate scoring arithmetic from the guided reward-learning lessons. Work without the walkthrough first. Opening and submissions are saved; the first submission is retained, with at most two practice retries. Reopening, resetting or importing an earlier backup does not erase known exposure. This is a local objective check, not authenticated exam supervision or human review of your implementation.

Readiness recap and optional explanations

A return totals rewards over a trial. A discount scales future rewards. A policy assigns action probabilities; a logit is a score used to produce those probabilities. A baseline is a reference subtracted from a return. The cases supply the arithmetic you need; use a calculator and decimal answers. Blank answers are not submissions.

You can review bootstrapped action values, policy gradients, clipped updates, offline data coverage or evaluation design without completing earlier pages. Reviewing feedback after a first attempt supports practice, not replacement first-attempt evidence.

0/5 cases passed on first submission. Latest/practice passes: 0/5. Project evidence, these outcomes and pending human review are separate.

Two warehouse receipts

A two-sample policy update

Clipping a bad action

An untried conveyor action

Report every training seed

Learn AI · Understand by doing.Loading local progress…Copyright (c) 2026 Mobi Weave, Inc. All rights reserved