Give words a place in space.

A vector keeps numbers in a meaningful order

An embedding represents a token, image, object, or state as a vector: an ordered list of numbers. Order matters. The vector (1,0) is different from (0,1), just as swapping the latitude and longitude of a location changes its meaning.

Our two-dimensional drawing makes the geometry visible. Real embeddings usually have many more coordinates. The axes in this lab are simply coordinate 1 and coordinate 2. They are not universal “meaning axes,” and a real learned dimension need not correspond to a human-readable category.

Compare direction and distance

A dot product multiplies matching coordinates and adds the products. For (1,0) and (1,1), it is 1 × 1 + 0 × 1 = 1. Vector length is the square root of the sum of squared coordinates, so these lengths are 1 and √2.

Cosine similarity divides the dot product by both lengths. It compares directions: 1 means the same direction, 0 means perpendicular directions, and −1 means opposite directions. For the source example, cosine is 1/√2, about 0.7071. The zero vector has no direction, so its cosine similarity is undefined.

Euclidean distance measures the straight-line gap between the endpoints. In the same example it is 1. If we stretch B to (2,2), its direction stays the same but its endpoint moves. Cosine stays about 0.7071 while distance becomes √5. Neither measure is universally best; the choice depends on how the representation was learned and what comparison is useful.

The same token can mean different things in context

“River bank” and “bank account” share the token “bank,” but their contexts favor different interpretations. A model can start from the same token embedding and produce different contextual representations after combining information from nearby positions.

The lab illustrates this with a fixed bank vector and a 50–50 average with a river or account vector. These are hand-chosen numbers, not a trained language model. The next lesson replaces that fixed average with query-dependent attention weights. Real contextual representations emerge through trained projections and multiple transformations.

Knowledge is distributed, and retrieval can be wrong

A distributed representation spreads useful distinctions across combinations of numbers. One concept need not live in one neuron, and one neuron can contribute to multiple features. In superposition, several features occupy different, potentially interfering directions in a shared space.

Similarity can retrieve useful related items, or provide weights for an associative memory that mixes stored values. But being close in an embedding space does not establish that a claim is correct, current, or an exact answer to your question. A model can reconstruct a plausible but wrong statement from learned patterns. Exact, inspectable databases and source verification remain valuable.

Entering from an application project

A retrieval system turns the query and candidate documents into comparable number lists, chooses a scoring rule, then inspects the returned text. A score is a ranking signal, not a probability that the answer is true. In the recap experiment, compare dot product, cosine and distance on the same three notices. Then check the organization, revision and exact sentence before choosing an answer. A current notice about the wrong building and an archived notice about the right building fail for different reasons.

The geometry is hand-assigned to expose these distinctions. Actual learned retrieval must also test whether its training objective produces useful rankings on different queries. Continue to keyword and learned retrieval or claim-to-source checks without completing the rest of the language course.