Learn AISpend a limited compute budget.
Balance model size and training data, distinguish active weights from stored weights, and measure a small browser computation.
By the end: Estimate training FLOPs and weight storage, compare allocations under a fixed budget, and distinguish a cost estimate from a measured runtime.
Start with something concrete.
You have enough ingredients for one school bake sale. More trays leave less time to bake each tray. Training has a related tradeoff: with a fixed computation budget, a larger model generally leaves less work available for training tokens. Unlike baking, the best balance also depends on the data, architecture, and how the model will be used.
Try the interactive model ↓A dense model doubles its parameter count and is trained on twice as many tokens. Under the rough six-N-D rule, how does training work change?
Change one thing. Watch what follows.
- Reproduce the source exercise: 1B parameters and 100B training tokens. Read the estimated FLOPs, then try the equal-budget presets.
- Keep the model fixed and increase training tokens in equal steps. Explain the flattening in the explicitly invented loss curve. Compare weight storage and deployment costs across presets.
- In the 50B expert example, change active parameters from 20B to 10B. Explain why storage for all weights stays the same.
- Double context length and compare pair counts with recurrent update counts. Optionally run the small CPU/WebGPU experiment and explain why its milliseconds are separate from the large-model estimates.
Loading this browser experiment…
Work, speed, and memory are different quantities
A parameter is a stored, adjustable number such as a weight or bias. A training token is one token presented during training; passing over the same data again still counts work, even when it adds little new information. A FLOP is a floating-point arithmetic operation. FLOPs count work; FLOPs per second describe a processing rate. Neither is the amount of memory needed to hold the model.
For a conventional dense language model, a useful rough training estimate is:
Here N is the parameter count and D is the number of training tokens. Forward predictions, gradient calculation, and updates all require work. The factor six is an approximation that leaves out some costs, including important attention and implementation overheads. It is not a universal formula for every neural architecture.
The source exercise uses N = 10⁹ and D = 10¹¹. The product is 6 × 10²⁰ FLOPs, or 600 EFLOPs. Doubling N while halving D preserves that estimate. It does not preserve accuracy, memory use, or deployment cost.
A fixed budget asks you to make a tradeoff
More parameters can represent more complicated functions. More varied, relevant training data can help select functions that generalize. Large models trained on too little useful data can be a poor use of a fixed budget. Smaller models trained longer may be attractive when deployment memory or repeated inference is expensive.
Empirical scaling laws describe measured relationships between quantities such as model size, data size, computation, and prediction loss. Many show diminishing returns: each additional unit of resource brings a smaller improvement. The lab uses an invented curve so you can see that shape. Its constants do not come from measurements and must not be used to plan a real training run.
A sensible allocation requires actual quality measurements on suitable validation data, then a final held-out evaluation. Data duplication, poor labels, distribution shift, and harmful shortcuts do not disappear just because a model is bigger. Model size is not a direct measure of intelligence.
Deployment has a different bill
At two bytes per weight, one billion weights require 2 decimal GB just for the weights. Training also needs gradients, optimizer state, activations, and temporary storage. Inference needs activations, working buffers, and—in a decoder—a KV cache. Lower-precision storage can save memory, but numerical accuracy and supported operations need checking.
During small-batch generation, reading weights from memory can take a large share of the time. Memory bandwidth measures how quickly bytes can be moved. Dividing bytes by bytes per second gives an ideal reading-time lower bound. Real latency includes arithmetic, caching, batching, scheduling, and communication as well. Do not turn the lab’s assumed bandwidth into a claim about your laptop’s speed.
Conditional computation does not make inactive weights disappear
A mixture-of-experts model routes a token through a subset of its expert parameters. This is conditional computation: choose which numerical paths to activate for each input.
The ebook’s example has 50B total parameters and 20B active parameters per token. At two bytes per parameter, all the weights need 100 GB of storage. The active subset represents 40 GB of weights, but the other 60 GB still exists. Storage can be distributed or moved between devices; that can introduce additional transfer costs. Router computation, shared layers, load balancing, and communication also matter. Fewer active weights do not guarantee lower latency than a smaller dense model.
Internal experts are parts of a model’s computation. They are distinct from independent specialist models that communicate through tool calls or typed interfaces, which we will study in Course 2.
Longer context invites different architectures
For L tokens, full attention compares L² query-key pairs per head. A causal mask permits only L(L+1)/2 pairs, which still grows quadratically. Doubling L roughly quadruples the pair count. Optimized attention kernels can reduce memory traffic and avoid storing a complete score matrix, so the conceptual pair count is not a claim about one implementation’s memory allocation.
A sliding window limits each token’s direct access to nearby positions. Recurrent and state-space models instead carry a state forward through the sequence. Their updates can grow linearly with sequence length when state size is fixed, but an update is not the same operation as an attention pair. Compressing history can change which information is preserved. Architecture comparisons need quality tests and real measurements, not just counts with different units.
Why the optional GPU experiment is small
Matrix multiplication contains many independent multiply-and-add calculations. A GPU can perform many such calculations concurrently, but setting up a device, compiling a shader, and transferring arrays also takes time. For a small job, the CPU can finish first.
The experiment actually multiplies bounded arrays in a browser worker. If you choose WebGPU, it also runs a compute shader and compares every result with the CPU reference. The CPU path remains usable when GPU initialization fails or a device is lost. Each run releases GPU buffers and its device; canceling or leaving the page terminates the worker.
The two timers have deliberately stated boundaries: CPU arithmetic versus GPU setup through readback and cleanup. They do not predict large-model training performance, prove which hardware is generally faster, or compare optimized libraries. Browser timing has noise; repeated runs can differ. A WebGPU adapter can also be a software implementation, so availability alone does not prove hardware acceleration. WebGPU API documentation explains the browser interface.
Two models use the same estimated training work: A has 0.5B parameters trained on 200B tokens; B has 2B parameters trained on 50B tokens. You need low weight memory at deployment. Which recommendation is supported?
Take this idea with you.
Efficient AI balances data, arithmetic, memory, and deployment needs. State your assumptions, measure actual behavior, and evaluate quality under the intended conditions.