Choose where a connected system learns.

Decide what is allowed to learn

Connecting models gives you a new choice: when the final output is wrong, which parameters should change? Start from a clear baseline and compare alternatives on the same task. More trainable parameters are not automatically a better design.

The lab uses a two-by-two encoder followed by a two-by-two adapter. These are small linear neural layers with hand-defined starting weights. There is no pretrained image model or hidden training service. The task is to reconstruct a position from the encoder’s representation, using actual gradient updates and separate monitoring inputs.

Strategy	What changes here	What stays fixed
Frozen baseline	Nothing	Both matrices
Adapter only	Four adapter weights	The encoder
Partial tuning	Four adapter weights and the encoder’s second row	The encoder’s first row
Joint training	All eight weights	The chosen data, loss, and monitoring consumer

All learning methods use the same eight varied training positions, rate 0.2, and number of update opportunities. The frozen baseline performs no updates. Updating eight parameters is more work than updating four, so this comparison controls data and steps, not total computation or wall-clock time. A larger study should report those costs separately.

Follow the gradient across a boundary

Let the encoder be E and the adapter A. The pipeline predicts A × E × input. The downstream position loss depends on both matrices. Backpropagation first computes the adapter gradient, then can propagate sensitivity through A into E.

Adapter-only training computes updates for A while holding E fixed. Partial tuning permits only selected entries of E to move. Joint training applies the chain rule throughout this differentiable pipeline. The animated trace displays the current numerical gradients and indicates which entries each strategy can update.

Optional: the encoder gradient

For input x, encoded vector z = Ex, prediction Az, and error d = Az − x, the adapter gradient is the outer product d zᵀ. Sensitivity at z is Aᵀd. The encoder gradient is (Aᵀd)xᵀ. Average both over the training batch and apply the chosen update mask. Both gradients are calculated from the same pre-update weights; the encoder must not accidentally use an already-updated adapter.

Learn the connection before changing a good specialist

With the complete encoder, both coordinates are available but rotated. Adapter-only training can learn the inverse. Joint training also solves the position task, but it can distribute the solution across both matrices. Neither matrix then has to retain its original interpretation.

The lab keeps an existing consumer that simply reads the first encoder feature. Before training it expects y for the complete encoder, or x for the encoder that discards y. This fixed reader is deliberately simple. Its separate monitoring error reveals whether a change broke the old input contract.

Joint training on the complete example can drive the new task’s error nearly to zero while increasing that existing consumer’s error. Looking only at the final position loss hides the regression. Partial tuning protects the first encoder row by design, so that particular consumer is protected here; larger networks need behavioral tests rather than assuming one frozen layer protects every old capability.

An adapter cannot invert discarded information

With the missing-y encoder, every position [x, y] becomes [x, 0]. Frozen and adapter-only methods cannot distinguish different y values with the same x. Their position loss cannot reach zero on this varied dataset.

Partial and joint tuning can restore a y-sensitive feature because the original input still contains y and those strategies may change the relevant encoder row. This is not an adapter recreating information from nothing. The learning boundary has moved upstream to a component that can still access the missing variable.

The source’s image-to-control example is analogous: an encoder trained for semantic retrieval might discard exact object position. Connecting a larger projection cannot reliably reconstruct coordinates that never reach it. You may need a different encoder output, a preserved spatial feature map, another observation, or task-specific fine-tuning.

A new checkpoint is a new interface candidate

An embedding’s width is only its shape. A new encoder checkpoint can keep the same width while rotating, scaling, or otherwise changing its learned geometry. The existing adapter may silently misinterpret the new outputs.

The lab’s synthetic replacement negates both features while preserving width. It applies this change to temporary copies, keeping the adapter fixed. Compare the resulting errors with the original pipeline. This is a controlled compatibility test, not a claim that real checkpoint changes are simple sign flips.

Version encoders, adapters, schemas, and relevant preprocessing together. Keep a reference set of meaningful behaviors and regression thresholds. Run compatibility checks before accepting a component replacement, and retain a working version for rollback. Passing type checks alone does not establish behavioral compatibility.

Some useful boundaries are not differentiable

Exact rule engines, discrete database queries, compilers, and external tools are not ordinary differentiable layers. A numerical loss does not automatically produce a useful gradient through them. A system may train predictors around such tools, imitate successful traces, use reinforcement learning, or keep those boundaries programmed. Each choice introduces assumptions and evaluation requirements.

End-to-end training is therefore an option for suitable components, not a prerequisite for building a capable modular system. Independent specialists, frozen interfaces, selected fine-tuning, and joint training can coexist.

Design a fair vision-to-control study

The source exercise compares an unchanged specialist, a learned adapter, and partial fine-tuning on one downstream task. Define the target and data split first. Start each method from the same specialist checkpoint, give each the same training examples, and choose settings using validation rather than final tests. Include the unchanged baseline and, if useful, a joint-training arm.

Report new-task error, retained specialist behavior, compatibility after replacement, trainable parameters, training resources, and inference cost. Use multiple seeds and withheld scene variations such as new positions or backgrounds. Predefine acceptable regressions and inspect failures, including cases where extra tuning provides little benefit. Equal steps alone do not make costs equal; neither does a lower new-task loss prove the whole system improved.

Start with useful independent components. Train the connections that demonstrably need improvement, and expand the training boundary only when the evidence justifies the added coupling.