Learn AIMake a problem easier to see.
Build XOR with two hidden neurons and discover what nonlinear activations add.
By the end: Trace a hidden representation and explain why affine layers alone cannot solve XOR.
Start with something concrete.
Two switches control a lamp. The lamp is on when exactly one switch is on, and off when both match. This rule is called exclusive OR, or XOR. Try the four switch combinations in the lab.
Try the interactive model ↓If we stack two layers but use no nonlinear activation, can we separate both positive XOR corners from both negative corners with the final straight boundary?
Change one thing. Watch what follows.
- Try all four switch pairs. Read the hidden values as well as the output.
- Turn ReLU off. Which rows of the truth table fail?
- Compare the input-space diagram with hidden space. The hidden transformation changes the arrangement of the examples.
Loading this browser experiment…
A line has a limit
In the input diagram, (0,1) and (1,0) are positive. The other two corners are negative. Any single straight decision boundary leaves at least one corner on the wrong side. This is a limitation of the model family, not a lack of training time.
Two affine transformations still combine into one: if h = ax + b and y = ch + d, then y = cax + cb + d. Adding layers without nonlinearity does not remove this limit.
Two hidden features change the view
ReLU returns its input when positive, and zero otherwise. Our first hidden neuron computes ReLU(A + B). The second computes ReLU(A + B − 1). The output subtracts twice the second value from the first.
For (1,1), the hidden values are 2 and 1, so the output is 2 − 2 × 1 = 0. For (1,0), they are 1 and 0, so the output is 1. The weights in this lab are hand-chosen to expose the mechanism; no training has taken place.
A representation is useful for what comes next
A hidden layer produces a new set of features. In hidden space, a simple output rule can now distinguish the positive examples. A representation helps when relevant differences become easier for later operations to use.
Real image networks can combine edges into useful shape evidence. But a concept does not have to live in one neuron: features are often distributed and overlapping. More depth can express compositional functions efficiently, while also making optimization harder.
For A = 0 and B = 0, what are (h₁, h₂, y) with ReLU on?
Take this idea with you.
Hidden layers transform a representation. Nonlinear activations let a network express distinctions that one straight boundary cannot.