Order changes meaning.

Start with two event logs

Imagine a toy log: A means open, B means close, and a dot means a neutral time step. A B . . and B A . . contain the same events, but they describe different orders. The lab asks one specific question: did A occur before B? Each input contains exactly one A and one B. This is a deliberately constrained teaching task, not a model of real door safety or natural-language meaning.

A bag of observations replaces the sequence with counts or proportions. For both examples, the input becomes [0.25, 0.25, 0.50]: the fractions of A, B and dots. A model receiving only this vector must give the same prediction for both. This limitation applies even to a very large count-only network. Additional training cannot recover information discarded before prediction.

The lab fits a four-parameter logistic baseline to these proportions. Every length-four training input has the same vector, and half the labels are 1. A probability of 0.5 minimizes its average cross entropy. With the displayed rule “predict 1 at or above 0.5,” it gets half the balanced cases correct. That is an information limit of this representation, not proof that training is broken.

Carry a small state forward

A recurrent neural network reads one symbol at a time. After each symbol it updates a vector called its hidden state. “Hidden” means internal to the model; the lab reveals all four values. The same learned weights are reused at every time step. The state changes while reading, but the weights do not.

We encode A as [1, 0, 0], B as [0, 1, 0], and dot as [0, 0, 1]. This is one-hot encoding: one selected coordinate is 1 and the others are 0. Dot is an input, not an instruction to preserve memory unchanged.

For each hidden unit, combine the current input contribution, a weighted sum of the preceding state, and a bias. Apply tanh, a smooth function that maps the sum to a value between −1 and +1. The live diagram shows these bounded states on a fixed scale, with intermediate arithmetic below it.

A small worked example, separate from the trained playground

Consider one hand-set unit: A contributes +1, B contributes −1, and dot contributes 0. Give the preceding state a weight of 0.5, use bias 0, and start at 0.

• After A: tanh(1 + 0.5 × 0) ≈ 0.7616.
• After a dot: tanh(0 + 0.5 × 0.7616) ≈ 0.3634.
• After B: tanh(−1 + 0.5 × 0.3634) ≈ −0.6741.

Practice: reverse this sequence to B, dot, A. With these particular symmetric weights, the states have opposite signs. This demonstrates sensitivity to order; it does not prove that these hand-set weights solve every sequence. The trained four-unit model has different weights, shown in the workbench.

Connect the diagram to the formula

Here t is the current step, xₜ is its three-number one-hot input, and hₜ is the new four-number state. hₜ₋₁ is the preceding state, initially four zeros. Wₓ is a 4-by-3 matrix of input weights; Wₕ is a 4-by-4 matrix of recurrent weights; b is a four-number bias. All values in this synthetic model are dimensionless. Tanh applies separately to each of the four sums.

Read it as: “new state equals a bounded transformation of current input plus remembered state plus bias.” The matrix products are weighted sums, just like the neuron calculations in the foundations. There is one shared parameter set, not a new network for each symbol.

After the last symbol, four output weights combine the final state with one output bias. A sigmoid turns this logit into a probability between 0 and 1. The model predicts whether A came first. Its total is 12 input weights + 16 recurrent weights + 4 state biases + 4 output weights + 1 output bias = 37 parameters.

Optional implementation: learning through time

The training loss is binary cross entropy on the final prediction. Backpropagation first differentiates the output and then follows the state chain backward. At a tanh unit, multiply the arriving sensitivity by 1 − h². Every use of a shared recurrent weight contributes to the same accumulated gradient. Only after processing all training examples does the model update every weight simultaneously.

This is full backpropagation through the short sequence, not a programmed state machine. The lab checks these derivatives against central finite differences. It does not implement an LSTM, a GRU, attention or truncated backpropagation. The reset-state checkbox is an inference fault for investigation; training always uses intact recurrence.

Find where memory fails

Training uses all 12 length-four sequences under this rule. The 20 length-five development sequences are different complete inputs but come from the same synthetic generator. You can also inspect lengths six and eight. These repeatedly visible results are development and stress probes, not reserved assessment evidence.

At the initial setting, train 1,000 updates at rate 0.5. Then inspect a longer mistake instead of stopping at the training score. Extra dots can move the state away from a useful representation. A bounded state does not promise unlimited reliable memory, and good performance at one length does not establish performance at another.

Turn on the reset-state fault and compare sequences ending in the same symbol. Once each step discards its preceding state, the final prediction depends only on the last input. For A B . . and B A . ., that destroys the order distinction. Restore the intact model before interpreting the training/development scores, which always measure intact recurrence.

For a guided explanation, supply an identical-count pair and describe one hidden-state update. For implementation practice, reproduce the forward pass and gradient checks. For a decision brief, specify the event order and lengths a system must handle, compare errors and request realistic evaluation before deployment. A simple programmed order check would solve this fully specified toy rule exactly; the purpose of training here is to inspect learned state and its limits.