Choose the pieces a model predicts.

Text needs a representation before prediction

A model cannot directly multiply by the sentence “red apples grow”. A tokenizer divides text into pieces and assigns each piece an integer ID. The model can then look up a vector for each ID. An ID is an index, not a meaning. Changing the numbering consistently does not make a word more positive or a sentence more true.

Start with the same probe in every mode. The lab keeps spaces and punctuation explicit, so you can check whether decoding the IDs restores the original text. Your probe is encoded using the fitted vocabulary; it never changes that vocabulary or the count model.

Three useful choices, three different costs

Character mode uses Unicode code points. It can reuse known letters to spell a word never seen during fitting, but sequences can become long. A code point is not always a complete visible character: an accent or emoji sequence can contain several. This teaching tokenizer does not normalize text or fall back to arbitrary bytes, so a code point absent from training becomes unknown.

Word mode keeps word spans, punctuation and whitespace as separate pieces. Familiar sentences can be short, but an unfamiliar whole word becomes an unknown ID even if its individual letters appeared elsewhere. Try “green apples sing”, then “snow”. A novel combination of known words is different from an unknown word.

Subword mode begins with code points and learns frequent adjacent pairs within word spans. It can keep common fragments together while spelling less common words from smaller known pieces. It still cannot represent every possible character in this bounded demonstration.

Production tokenizers differ in their boundary rules, normalization, byte coverage, vocabulary sizes and special markers. Do not treat this small BPE-style implementation as the exact tokenizer of a particular deployed model.

Follow one merge

Suppose the fitting documents are aaaa and aaaa. Each has three adjacent a–a pairs, so the pair count is six. Replacing non-overlapping occurrences from left to right creates aa, aa in each document: four replacements, not six. Overlapping candidate pairs share positions.

The lab repeats this process for at most the selected number of merges. It stops if no pair appears twice. Ties use a fixed string order. Merges never cross a whitespace, punctuation or document boundary. Look at the recorded pairs before interpreting a shortened sequence.

Repetition changes these statistics. Duplicating one document can make its fragments more likely to merge. Fitting a tokenizer on evaluation text also changes the representation using evaluation evidence; it is not the same as simply applying a fixed tokenizer to that text.

Unknown is a loss of information

Two different unknown fragments can receive ID zero. The lab shows their original spelling so you can diagnose the loss, but the count model receives only that shared ID. Its decoder displays a replacement symbol. It cannot reconstruct the missing spelling from zero alone.

A lower token count is therefore not automatically better. Turning a whole paragraph into one unknown token would be short and nearly useless. Compare vocabulary coverage, sequence length, exact reconstruction and the task the representation must support.

Build a baseline before a neural model

A bigram model counts which token follows which other token. It uses only the last token of your prefix, ignoring earlier context. Separate start and end markers keep document boundaries explicit. The end of one document is not followed by the first token of the next during fitting.

For a small worked example, after token A the training data contain B twice and C once. If there are five possible target IDs, add-one smoothing gives B a probability of:

Read this as: “the probability of B after A is its observed count plus one, divided by all observations after A plus the number of possible targets.” The vertical bar means given. Smoothing adds one imaginary count to every target so unobserved transitions do not receive zero probability. This is a modeling choice, not new evidence.

In the live lab the displayed counts come from its current fitting documents. The denominator includes every possible next ID except the start marker. The five displayed bars need not add to 100%, because other targets also have probability. Try two prefixes with the same last token and explain why their predictions agree.

Apply the tradeoff to a real task

For a spelling tool, losing a name to an unknown token can defeat the task. For a multilingual application, check which scripts and writing conventions receive good coverage. For a software implementation, test round trips, literal special-marker text, document boundaries and out-of-vocabulary behavior before training a larger model.

Keep a small comparison report: the same input, each mode’s pieces and length, unknown fragments, one checked probability and a limitation. These are guided practice observations, not independent implementation evidence. Token counts or per-token losses from different vocabularies do not directly measure the same prediction task, so avoid ranking model quality from those numbers alone.

Continue to embedding vectors and masked attention to see how neural models use token representations and more context. A count model is fitted from data, but it is not a trained Transformer or evidence of language understanding.