Reuse a representation.

Start with something the model already learned

Suppose you have only a few labeled T and L pictures. You could train every weight from the beginning, or reuse a network that has already learned from other pictures. Transfer learning reuses learned parameters for a new task. The earlier task is the source; the new task is the target. Reuse is a starting point to test, not a guarantee of useful knowledge.

This lesson supplies the small encoder from the preceding representation lesson. It learned from paired views of vertical and horizontal bars. Our new pictures are five-pixel T and L shapes. Both contain the same amount of ink at a given brightness, so a total-brightness shortcut cannot solve the task.

If you arrived directly: an encoder converts an input into features, a head converts those features into a task prediction, and training changes allowed weights to reduce a chosen error. A frozen weight stays fixed while other weights learn. You can inspect the experiment without deriving gradients or writing code.

Three ways to learn from the same labels

The architecture diagram shows 25 input pixels, six hidden units, two features and one probability. Every candidate has 173 parameters. Its first 170 form the encoder; the last three form the head.

• From scratch: start with deterministic untrained encoder weights and train all 173 parameters.
• Frozen encoder: reuse the supplied encoder without changing it. Train only the head’s two weights and bias: three parameters.
• Tune the final encoder layer: reuse the encoder, keep its first 156 parameters fixed, and train the last 14 encoder parameters plus the three head parameters: 17 parameters.

Fine-tuning means continuing to train some or all of a previously learned model on the new task. Our limited mode is one particular partial fine-tuning strategy. It does not update the first hidden layer, so it cannot freely replace the original representation.

Select one development image and compare its features before and after training. In the frozen mode its two feature values must stay identical, although the probability can change. In limited tuning, the six hidden values stay fixed while the two feature values may move. From scratch, both layers may change. The workbench computes these intermediate values from the actual current weights.

Turn two features into a prediction

A head is small enough to follow by hand. Suppose an image has features [0.4, −0.2]. With weights [2, −1] and bias 0.1, its score is 2 × 0.4 + (−1) × (−0.2) + 0.1 = 1.1. A sigmoid converts 1.1 to approximately 0.7503, or 75.0% probability of L. At a 50% threshold the prediction is L.

Here z₁ and z₂ are dimensionless feature coordinates between −1 and +1. w₁ and w₂ are head weights; b is the bias. s is the dimensionless score. p is the model’s probability of L, between 0 and 1; e is the mathematical constant approximately 2.718. Read the first equation as “multiply each feature by its weight, add the results and the bias.” Sigmoid smoothly converts this score to a probability.

Practice: keep the features fixed and change the bias from 0.1 to −1. The score becomes zero and the probability becomes 50%. Our tie rule predicts L. The head changed the decision without changing either feature. A probability is the model’s output, not a promise that it is calibrated on real photographs.

Optional implementation: gradients and freezing

We minimize mean binary cross-entropy. For one example, the score derivative is p − y, where y is 0 for T and 1 for L. The two head-weight derivatives are (p − y) × z₁ and (p − y) × z₂; the bias derivative is p − y. Average contributions over the target training examples, compute all derivatives at the preceding weights, then update permitted parameters simultaneously.

For trainable encoder layers, backpropagate through the head and tanh. The derivative of tanh with output a is 1 − a². Apply the mode’s parameter mask: frozen parameters receive no update. Training never accepts development or reserved rows. As an implementation exercise, reproduce a one-step update and assert that every frozen parameter is exactly unchanged, then compare analytic derivatives with small finite differences.

Match the comparison and name what still differs

Use the same target label budget, learning rate and number of updates for the three candidates. Start with 12 examples and rate 0.1. Train 1,000 updates and capture the result. Repeat with 4 and 36 examples. The smaller balanced sets are contained in the larger ones, but more labels also cover more positions. You cannot attribute a change purely to sample count while ignoring this coverage change.

The supplied encoder has already consumed 1,000 source-training updates on 72 pair roles. That work is additional to target training. Equal target update counts therefore do not mean equal total compute. All candidates have the same architecture, but their numbers of trainable parameters differ. Worker timings measure this target request on your device; they exclude historical source training and rendering.

If frozen features do worse, inspect the failure instead of assuming pretraining always helps. A source representation can discard distinctions the target needs. Limited adaptation may be unable to recover them; a head cannot reconstruct information absent from its input. Poor results can also reflect the optimization settings or restricted trainable capacity. To isolate the effect of initialization, a stronger follow-up would compare pretrained and untrained encoders with the same trainable mask and multiple runs. The current three-mode experiment does not implement that matched ablation.

Separate choosing from testing

Training uses brightness 0.6 and 0.9. Repeatedly visible development images use 0.7. Use development evidence to choose a candidate and write your reason. Freeze the three parameter arrays, label budget, rate and chosen mode before revealing the 18 brightness-0.8 reserved images.

Those complete input arrays are disjoint, but all splits share the same synthetic shape generator and position grid. A good reserved result supports a narrow claim about these shapes and brightness values; it does not establish real-image performance. The source checkpoint was trained only on bars, excluding all T/L target images.

The first reveal records local exposure before displaying outcomes. Later reveals, including after reset or importing an older backup, are practice. Public fixtures and editable local records do not establish authenticated independence. The app cannot know whether you saw the cases elsewhere, and local progress does not synchronize between devices. If storage fails, the interface explains that exposure is temporary.

Seeing all three reserved scores is a descriptive comparison. If you change your selection because of those scores, acknowledge that you have used this set for development. A new final evaluation needs fresh cases and a protocol decided in advance.

Produce evidence for your chosen path

For guided practice, export three equal-update budget comparisons and explain one frozen-feature prediction in your own words. Name a training success that did not carry over to development. For software practice, reproduce forward predictions and gradients, verify the frozen parameter masks, and build the matched initialization ablation described above. That code is an exercise to implement, not an automatic credential supplied by the workbench.

For a decision brief, choose a candidate using development evidence and explain the tradeoff between target labels, source cost and adaptation freedom. Before recommending it for a real use, specify a licensed dataset, a useful baseline, errors that matter to users, and a genuinely new evaluation set. Browser captures and report length do not establish the quality of that judgment; independent artifact review remains pending.

The PyTorch transfer-learning tutorial provides an optional implementation example contrasting a fixed feature extractor with fine-tuning. It uses a different, larger architecture and dataset; its results are not results of this tiny local lab.