Debug and resume a framework model.

Start where you need practice

Recommended preparation: P04 manual network gradients and P02 array shapes and environments. A tensor is the framework's multidimensional numerical array. Autodiff records operations so it can calculate derivatives backward. An optimizer applies those derivatives and may retain memory from earlier updates.

The supplied PyTorch model has the same 13 parameters, two inputs, three tanh hidden units and one linear output as P04. It uses CPU float64. Matching this small model is a controlled implementation check, not a framework speed benchmark.

Try → predict: are weights enough?

A model has completed seven Adam updates with 50% hidden dropout. Two continuations begin with exactly the same weights, next batch and dropout seed. One restores Adam's memory; the other resets it. Predict whether their next parameter update will match.

Settings and answers are temporary. A checkpoint is a saved training state, not just a prediction model. For this exercise it includes weights, Adam's running average and running squared-gradient average, update count, data-loader position, and the next dropout seed.

Inspect the graph before repairing it

Use the shared scalar graph to trace where derivatives come from. The browser implements an educational reverse sweep; PyTorch is used only in the downloadable exercise. Predict which contributions disappear when a hidden result is detached.

In PyTorch, detach() keeps a tensor's numerical values while disconnecting its earlier computation history. If hidden activations are detached, output weights may still receive gradients while hidden weights do not. Inspect each parameter's .grad; a missing gradient is different from a present gradient whose numerical value happens to be zero.

model.eval() changes module behavior such as dropout. torch.no_grad() disables gradient recording. They solve different problems: no_grad by itself does not turn dropout off. For deterministic inspection use both evaluation mode and no_grad, then restore the previous training mode.

Apply: a runnable CPU framework

Download framework-debugging exercise · Download P04 reference beside it · Download pinned dependencies

Use Python 3.11+ and the P02 environment instructions. On macOS/Linux:

python3 -m venv .venv
.venv/bin/python -m pip install -r framework-requirements.txt
.venv/bin/python framework-debugging.py --compare
.venv/bin/python framework-debugging.py --diagnose
.venv/bin/python framework-debugging.py --resume-example > resume-example.json
.venv/bin/python framework-debugging.py --demo --updates 7 --dropout .5 --out first.json
.venv/bin/python framework-debugging.py --demo --resume first.json --updates 33 --out resumed.json
.venv/bin/python framework-debugging.py --demo --updates 40 --dropout .5 --out uninterrupted.json
.venv/bin/python -m pip freeze > environment-freeze.txt

On Windows use py -3 -m venv .venv and .venv\Scripts\python.exe. PyTorch 2.9.1 and NumPy 2.3.5 are pinned. Verification used Python 3.12.14 on macOS ARM64; other platforms and accelerators are not verified. Package installation needs network access and the framework download is larger than NumPy alone. The experiment itself requires no network, GPU, account or paid service.

What to inspect

1. --compare compares the framework's predictions and autodiff gradients with the P04 NumPy equations at the same weights, float64 inputs and mean squared loss, with dropout off. Check every parameter, not only the loss.
2. --diagnose deliberately detaches hidden values and reports which parameters still receive gradients. It also repeats dropout-on inspection and dropout-off evaluation. Parameters are not updated by these probes.
3. --demo runs the supplied update. Complete learner_step, run --check, then use --train for your implementation. A TODO failure is expected before you implement it. Clear prior gradients, set training mode, calculate exact-shaped mean squared loss, backpropagate, and step the optimizer.
4. Compare the seven-plus-33 and uninterrupted-40 checkpoints. All model parameters, Adam moments, batch IDs and loss history should match in the verified environment. Using the same final weights alone is a weaker comparison.
5. In a separate copy, omit gradient clearing and run two updates. Explain why a single-step check can miss accumulation. Repair it, then ask a reviewer to supply a different fault without revealing its location.

Read the loader and randomness contracts

The experiment reuses P04's disjoint 16/16 synthetic grids. A real TensorDataset and DataLoader shuffle the 16 training rows into batches of four with no worker subprocesses. Every four updates completes one epoch: one pass over the training set. Development rows never enter that loader.

For absolute update t, integer division t ÷ 4 determines the epoch and its remainder determines the batch. The loader uses a local seed 177 + epoch; dropout uses 9101 + t inside an isolated randomness scope. These explicit schedules let the checkpoint restore the next batch and mask stream without a large serialized random-generator blob. This is a deliberately small reproducible recipe, not a claim that arbitrary multiprocess/accelerator loaders can be resumed this way.

Adam uses fixed coefficients 0.9 and 0.999 for its first and second moment averages, a denominator stabilizer of 10⁻⁸, no weight decay, and rates 0.001, 0.01 or 0.05. A moment is a running summary of previous gradients. The step counter matters because Adam corrects early averages for their zero initialization. Review the optimizer lesson for the arithmetic.

What the checkpoint guarantees—and what it does not

Each command accepts 1–100 updates, with at most 1,000 total. Full train/development monitoring disables dropout, while batch loss records the stochastic pre-update objective. Numerical rejection restores both the preceding model and optimizer, retains the batch/seed position, and records a stop reason. Stopped runs require an explicit restart; interrupted processes may leave no new file.

Checkpoint imports cap at 500 KB and use JSON arrays, never executable serialized model objects. Validation checks code/environment fingerprints, dimensions, finite bounds, Adam counters, batch/seed cursor, recorded batch identities and final losses recomputed from the model. Historical losses are bounded but are not authenticated or fully replayed on import. Learner-editable files are not proof of training history or independent proficiency.

Output filenames must be new. Resume takes the saved learning rate and dropout setting, so omit those command-line options. Keep supplied and learner implementations separate when resuming. Predictions use public repeated development evidence; there is no final reserved test in this studio.

Evidence and continuation

• Understand: explain the missing gradient path, the difference between eval and no_grad, and which state makes the next update reproducible.
• Implement: submit your update function, manual/framework comparisons, two-step tests, repaired failure trace, matched uninterrupted/resumed checkpoints, environment freeze and limitations. Independent review needs an unfamiliar fault and an actual replay.
• Decide: write acceptance criteria for a training restart. Distinguish numerical agreement, successful recovery, predictive quality and generalization; identify the evidence still missing.

Artifacts remain local files. This studio does not store a draft or human-review status in My Work. Human review, learner pilots and broader platform profiling remain pending.

Organize your training-diagnosis evidence · Try real-data training faults · Return to Explore

Framework references: PyTorch data loading, dropout modes, Adam, and saving and loading model state.