What does this neuron notice?

Multiply, add, and interpret carefully

A 5 × 5 image becomes 25 brightness inputs. The neuron multiplies each by a weight, adds the products and a bias, then applies a sigmoid. A sigmoid maps a score to a number between 0 and 1. We interpret that number as a predicted probability of the selected positive class, although it has not been calibrated on independent data.

The initial vertical template uses +1 in the center column, −0.25 elsewhere, and bias −2. A vertical line contributes 5, giving score 3 and sigmoid output about 0.953. A cross also contains the vertical stroke. Its four extra bright pixels subtract 1, leaving score 2 and output about 0.881. “Contains a vertical line” and “is exclusively a vertical line” are different tasks.

The starting weights are a hand-written teaching template. The editor lets you inspect that mechanism directly. Changing a weight by hand is not learning; the learning-step button in the lab performs an actual gradient update for the current labeled image.

How a feature can be learned

For a binary classifier, cross-entropy penalizes assigning little probability to the correct class. If the target is 1, a probability near zero is costly. If the target is 0, a probability near one is costly. With a sigmoid output and this loss, the weight gradient has a simple form:

A dark pixel has input zero, so its weight gradient is zero for this example. A bright pixel contributes the prediction error. The optimizer subtracts learning rate times each gradient, using the same old parameter values. The lab’s rate is 0.1; both weights and bias update together.

Repeated training on varied labeled examples can strengthen useful patterns. Training on just the displayed image can overfit it and alter the response to other images. Try labeling a cross as negative, taking one step, and returning to the vertical line. Lower loss on one example is not proof that the detector has learned the intended task.

Features are not always individual neurons

Hidden features emerge because they help the final objective; programmers need not name each intermediate feature. In a larger network, one feature may be spread across many dimensions, and one neuron may contribute to several features. A strong activation is evidence of a response, not a complete explanation of a concept.

Interpretability researchers measure activations and intervene on parts of the computation to test hypotheses about their roles. Methods such as sparse autoencoders try to separate mixed activations into a larger set of sparsely used features. Their outputs are useful hypotheses, not guaranteed unique dictionaries of the model’s concepts.

Transfer the source exercise to a 3 × 3 image

Give the three middle-row pixels weight +1 and the other six weight −0.25. With bias −1, a horizontal line has score 2. A plus sign adds two negatively weighted pixels, giving score 1.5. Both still activate the sigmoid strongly. To distinguish “has a horizontal stroke” from “is only a horizontal stroke,” you need a task definition and examples that separate them.