Compare convolution with image attention.

Two ways to connect image evidence

A local edge may be visible in a few neighboring pixels. A relationship such as “the object on the left matches the one on the right” can require distant evidence. Encoders choose how to connect these pieces. This lesson compares two mechanisms on the same synthetic image, without claiming that untrained responses measure classification quality.

The convolution uses two fixed three-by-three filters at every fitting location. One subtracts left from right; the other subtracts top from bottom. Its local connectivity and shared weights are inductive biases: built-in assumptions about useful structure before data are learned. Deep convolutional networks can combine local features over progressively larger regions; “local in one layer” does not mean a whole network can never use distant context.

Make patches into tokens

Our six-by-six image becomes nine nonoverlapping two-by-two patches. Each patch is flattened in top-left, top-right, bottom-left, bottom-right order. For example, [1, 0; 0, 1] becomes the four-number vector (1, 0, 0, 1). The order inside the patch matters: [0, 1; 1, 0] becomes (0, 1, 1, 0), not the same vector.

A vision transformer applies transformer processing to a sequence of image patches. The original research used trained patch representations and transformer components at a very different scale; our example only exposes one attention operation. It has identity query/key/value projections, no learned patch projection, no residual path, no normalization, no feed-forward layer and no classifier. These omissions are deliberate teaching choices, not a reproduction of a full trained ViT.

Trace one attention read

Each query compares with every key. Here a token itself supplies its query, key and value. Multiply matching coordinates and add them, then divide by the square root of the vector width. The width is four, so the divisor is two.

For query (1, 0, 0, 1) and the same key, the dot product is 2 and the scaled score is 1. A blank key (0, 0, 0, 0) scores 0. If these were the only two keys, softmax would assign e¹/(e¹+e⁰), about 0.731, to the first and 0.269 to the second. The live lab has nine keys, so its weights can differ from this two-key worked example.

Here i identifies a query patch and j a key/value patch; both run from 1 to 9. q, k and v are four-coordinate vectors. s is a dimensionless compatibility score, a its nonnegative normalized weight, and o a four-coordinate output. The dot means multiply matching coordinates and add. The sum over m includes all nine keys for that query. Read the output as “add each value vector multiplied by how much this query attends to it.”

Practice: if all nine scores are zero, what are their weights? Each exponential is 1, so each weight is 1/9. The output is the mean of the nine values. That is still a real calculation, but it does not mean the system has discovered a meaningful object.

What tells a patch where it belongs?

Without position information, reordering complete patch vectors only reorders the corresponding attention outputs. This is permutation equivariance. Averaging those outputs removes their order, making the mean unchanged by the patch permutation. Two differently arranged images can therefore share that summary.

Enable the position option to attach a small fixed coordinate vector to each destination slot. The lab shows exactly which numbers are added. Moving the same patch into a different slot changes its token and can change the final summary. This does not guarantee every image gets a unique representation; it supplies information the unpositioned version lacked.

Reversing whole-patch order is not the same as shifting an image one pixel. A one-pixel shift changes how pixels are grouped into patches, and the lab discards pixels falling beyond the right boundary. Always specify the transformation before claiming invariance.

Compare dependencies, then ask a task question

Toggle only the top-left input pixel. Only convolution windows that contain it can change. Attention can propagate a changed token to distant queries through its keys and values. Inspect the numerical outputs rather than assuming all reachable outputs must change for every image: zero values and cancellations can make a dependency invisible in a particular example.

Our count of 288 convolution multiplications and 648 attention score/value products describes these specific tiny implementations. It excludes several operations and compares different representations. It is not a latency benchmark, a fair training budget, or a general claim that one architecture is cheaper.

For a guided report, construct a patch rearrangement with an unchanged unpositioned mean and explain why arrangement matters for a chosen task. For implementation, reproduce normalization, patch extraction, permutation and locality checks. For a decision brief, propose matched training data, evaluation cases and measured resource constraints before recommending an architecture. The trained-model comparison and independent perception project remain separate work.