Learn from relationships between examples.

Learn a relationship before a category

A vertical bar remains vertical when shifted or made dimmer. Instead of immediately asking a network to name its orientation, we can ask it to produce similar features for two related views. An encoder turns each image into a compact representation. In this lab, it turns 25 pixel values into two learned numbers.

A positive pair says “these views should stay close.” A negative pair says “these views should remain distinguishable.” Positive and negative name a relationship; they do not mean good and bad images, nor the two orientation classes.

The lab starts with two original synthetic images. It creates 36 training views using nine positions, two brightness levels and two originals. Every positive pair normally connects a transformed view to its own original. Every negative connects it to the other original. Orientation labels are not input to the encoder loss. The source relationship supplies the learning signal.

This construction makes a strong assumption: transformations of one original preserve what matters, and the two originals differ in a relevant way. In a real collection, two different photographs may show the same category. Treating all different sources as negatives can then push apart useful neighbors. Rotating a bar can also change its orientation, making that transformation a poor positive for an orientation task.

See the two views move through one encoder

Both images use the same weights. The lab has six hidden tanh units followed by two tanh feature units: 25 × 6 input weights, 6 hidden biases, 6 × 2 output weights and 2 output biases, totaling 170 parameters. Tanh keeps each coordinate between −1 and +1. The scatter plot uses these fixed limits instead of zooming a tiny cluster to fill the screen.

Inspect the selected pair’s images, hidden values and output coordinates. These are actual computed features, not hand-positioned points. A feature coordinate is not a probability or a predefined property such as width. Its meaning comes from how the learned representation responds to inputs and supports later tasks.

The updates use both branches’ contributions to the shared weights. Moving one view’s features also changes how the same encoder processes other views. That is why a collection of pair constraints can conflict.

A numerical loss you can follow

Suppose one view has features [0.5, 0.5] and another [0.8, 0.1]. Their coordinate differences are −0.3 and 0.4. Squaring and adding gives 0.09 + 0.16 = 0.25. This is squared distance; ordinary distance is its square root, 0.5.

Our teaching objective is:

z and z′ are the two feature vectors. Their subscripts identify coordinates. s is their squared distance; L₊ is a positive pair’s loss and L₋ is a negative pair’s loss. The negative margin is a distance of 1 in these dimensionless feature coordinates. A positive pair is rewarded for getting closer. A negative pair stops contributing loss once its distance reaches 1.

For our example, the positive loss is 0.5 × 0.25 = 0.125. The negative loss is 0.5 × (1 − 0.25)² = 0.28125. Practice: if squared distance reaches 1, positive loss becomes 0.5 and negative loss becomes 0. They ask for different relationships.

This is a bounded teaching contrastive loss. It is not SimCLR’s normalized contrastive objective, and the lab does not reproduce SimCLR’s architecture or results. The linked research provides a broader example of learning representations from transformed views.

Optional implementation: shared gradients

For a positive pair, the derivative with respect to the first feature vector is z − z′. For an active negative pair it is −2 × (1 − s) × (z − z′); beyond the margin it is zero. The second view receives the opposite feature derivative. Backpropagate both through the same encoder and add their weight gradients. Average across pairs before updating all 170 weights simultaneously. The numerical tests check these gradients against central finite differences.

Agreement can collapse into a constant

If every image maps to [0, 0], all positive pairs have zero loss. Yet the representation distinguishes nothing. This is representation collapse. A positive-only objective admits this solution.

Use the collapsed-encoder control. The total coordinate variance is exactly zero: every training feature equals the mean. The classifier probe receives identical features and gets 9 of 18 balanced development examples correct through its fixed tie rule.

With negatives enabled, that collapsed encoder has nonzero loss. But our fully zero initialization still has zero gradients: identical features and symmetric zero weights cannot begin separating themselves through this particular objective. Negative pairs alone are not a guarantee that every initialization will learn. Restart the varied weights for the training comparisons.

Positive-only training from varied weights usually produces small but nonzero differences. Such features can still support some correct predictions. Compare the unrounded probe with rounding to 0.05: tiny distinctions may disappear. This is a controlled precision stress test, not a simulation of a particular device’s numerical format. Low variance, exact collapse and poor prediction are related but not interchangeable claims.

Contradictory relationships cannot all be satisfied

Apply opposite-source positives while keeping opposite-source negatives. Now the same image pair appears once with “come together” and once with “stay apart.” Inspect the first two pair entries to verify the conflict.

Even a perfect optimizer cannot make both losses zero. For this objective, the best unconstrained squared distance for a contradictory pair is 0.5, giving mean loss 0.1875. A shared encoder may do worse because it must satisfy many pairs at once. Repair the supervision before treating every remaining loss as an optimization failure.

Masked prediction asks a different question

Another way to learn from unlabeled examples is to hide part of an input and predict that missing part. In masked prediction, the original hidden values provide targets, but the encoder must not receive them as visible input. For images, a decoder can predict missing pixels or patches from visible context.

For a small row [0, 1, ?, 1, 0], predicting the middle value as 1 is plausible under a contiguous-bar assumption. It is not logically forced without assumptions about the data. If only the center pixel of a bar is visible, both a horizontal and a vertical completion may fit. Reconstruction targets and ambiguity must be handled explicitly; successful reconstruction does not automatically preserve the distinctions a later task needs.

The current workbench trains the pair-based encoder. It does not train a masked decoder. Masked prediction is a different learning signal, not a claim that removing pixels and comparing feature distances implements a masked autoencoder.

Test whether the features are useful

Freeze the encoder and use labels on its training views to average one feature center for each orientation. The nearest-centroid probe assigns each new view to the closer center. This simple downstream task uses labels even though the encoder’s fitting objective does not.

The 18 development views use brightness 0.75, distinct from the training brightness levels 0.5 and 1. They are separate pixel arrays but share the same shape generator. Repeatedly inspecting these results makes them development evidence, not an untouched test. Neither good pair loss nor a good synthetic probe establishes performance on photographs.

Start with the default learning rate 0.1 and 1,000 updates. Larger rates can make the training trajectory more sensitive to numerical differences; compare exported parameters and recomputed outcomes rather than assuming all runtimes produce identical weights.

For your report, compare pair construction, feature spread and probe errors under equal update budgets. Explain which transformations should preserve the task, how a wrong pair changes the objective, and what new data would be needed before trusting the features elsewhere. Implementation practice should reproduce the shared gradients and frozen probe; decision practice should justify the data assumptions and evaluation conditions.