Measure and optimize inference.

Start with the question a user cares about

A flower-identification application feels slow. Loading its model once takes time; each later prediction takes more time. A classroom uploads many measurements together. An individual user sends just one. The same optimization may help one situation and hurt another.

Entering midway: inference means using fitted parameters to make a prediction. Latency is elapsed time from a particular request arriving to its result becoming available. Throughput is completed requests per unit time. A batch groups requests for processing. A cache stores a previous result so an identical request can reuse it. Quantization represents parameters using a smaller set of numbers.

The lab runs the previous lesson’s real binary Iris classifier. Its workload is sixteen fictional measurement requests, not a held-out accuracy dataset. You can inspect probabilities and numerical error without claiming that these timings or results describe a deployed service.

Follow the measurement boundaries

Stage	What the lab measures	What it does not establish
Bundle preparation	Parse and validate local artifacts, check hashes, prepare chosen weights and requests	Network download or complete page startup
Cold pass	Sixteen requests starting with an empty cache	First-ever JavaScript execution on a fresh device
Steady passes	Repeated requests, including validation on misses, cache-key construction, lookup and loop overhead	Real arrival patterns, queueing or user latency percentiles
Single-request probes	Sixteen separately timed uncached CPU predictions, including timer overhead	User response time, queueing, network latency or a reliable population tail
Worker round trip	Worker creation, communication, setup, reference checks and all five trials	A single request’s response time
Storage accounting	Typed-array buffers and explicit numeric/text payload estimates	Browser process or peak memory

The numerical reference checks and sixteen single-request timing probes are outside the timed cohort passes. The request probes show a local median and nearest-rank 95th percentile: sort durations and take position ceiling(0.95 × 16) = 16. With so few samples this is just the maximum, not a reliable estimate of a population tail. Tiny durations may be below timer resolution. Caches start empty for each trial. The initial pass fills them, then the steady passes repeat exactly the same cohort. A cache is never reused across model or precision changes.

The first trial may trigger compilation or other runtime work. Five trials show a minimum, median and maximum rather than hiding variation behind one number. Median means the middle duration after sorting five values. These are repeated timings on one browser, not five independent devices. Repeat a matched configuration before treating a tiny difference as meaningful. A timer that reports zero cannot justify infinite throughput.

Connect the arithmetic to the claim

Consider a separate worked example: 160 requests finish in 8 milliseconds. There are 1,000 milliseconds in a second. Throughput is 160 × 1,000 / 8 = 20,000 requests per second. Amortized time is 8 / 160 = 0.05 milliseconds per request. This spreads total elapsed work across requests; it does not tell you how long a particular request waited.

In plain notation, throughput = N × 1,000 / T, where N is the number of completed requests and T is their measured elapsed time in milliseconds. Amortized time = T / N, in milliseconds per request. Both require a meaningful nonzero denominator. The lab always runs a positive request count; an unresolved zero duration produces no rate.

Practice: 320 requests finish in 8 milliseconds. What changes? Throughput doubles to 40,000 requests per second and amortized time halves to 0.025 milliseconds. You still cannot infer the slowest request or a 95th-percentile latency. A percentile describes the value below which a stated proportion of individual observations fall; aggregate duration is not that distribution.

Batching has a waiting cost

A hardware kernel can sometimes process many inputs more efficiently together. Requests may need to wait while a batch fills, however. With evenly spaced arrivals every 10 milliseconds, collecting four requests means the first waits 30 milliseconds before processing starts, the second 20, the third 10 and the fourth zero. The average collection wait is (30 + 20 + 10 + 0) / 4 = 15 milliseconds, even before model work. This is an illustrative arrival assumption, not a measurement from this lab.

Here batching groups ordinary sequential CPU loops and changes the reusable output-buffer size. It does not create SIMD, GPU parallelism, a network queue or an optimized batched kernel. The measured difference may be small, inconsistent or negative. Check unchanged predictions and equal request counts before comparing times. A tiny model need not benefit from larger batches.

Cache hits depend on the workload

With four distinct inputs repeated four times, an empty cache has four misses and twelve hits in its first sixteen-request pass. With sixteen distinct inputs, that first pass has sixteen misses. In both cases the second identical pass has sixteen hits. Try this before making the mistaken claim that “distinct” traffic never benefits from caching.

The key contains all four ordered measurements. The cache belongs to a single loaded model and precision. Caching by just one feature, or keeping old results after replacing a model, can return a fast but wrong answer. The lab has no eviction because there are at most sixteen entries. A real service needs a policy for changed inputs, changed models, capacity, expiration and any sensitive information stored in keys or values.

A hit still constructs a key and performs a lookup. For a five-parameter model, that overhead may exceed recomputing the answer. A warm repeated-input result cannot be presented as performance on an endless stream of new inputs.

Smaller numbers trade representation for error

The reference uses JavaScript’s ordinary double-precision arithmetic. The float32 mode stores weights in a Float32Array and explicitly rounds intermediate arithmetic. The exponential function still executes in JavaScript and its final probability is rounded; this is not a native float32 hardware kernel.

Quantized modes store integer codes and reconstruct approximate weights for ordinary JavaScript arithmetic. For a separate example, a scale of 0.1 maps a weight of 0.26 to the nearest code, 3, then reconstructs 3 × 0.1 = 0.3. The absolute weight error is 0.04. Multiplication and accumulation can turn that into a different output probability. A probability near the decision threshold can cross it after a small change.

The live symmetric quantizer chooses scale = largest absolute parameter / largest positive code, then code = round(parameter / scale). Parameters here are the four weights and bias. The code limit is 127 for eight-bit mode and 7 for four-bit mode. Zero parameters use a finite fallback scale. The same scale is shared by all five parameters; that simplification is not a claim about state-of-the-art quantization.

Check maximum absolute probability difference and threshold-label changes on the same inputs. Neither measure is task accuracy: the fictional requests have no ground-truth species labels. No label changes on sixteen examples does not guarantee none elsewhere. Define an acceptable error for the application and test representative, independently chosen cases before making a deployment decision.

Account for memory honestly

Five float64 parameters occupy 5 × 8 = 40 bytes in their typed-array buffer. Float32 parameters occupy 5 × 4 = 20 bytes. Five eight-bit codes occupy five bytes, but also need their scale. Five four-bit codes could ideally fit into three bytes; this implementation stores each code in an eight-bit slot and really allocates five bytes. The table reports both quantities separately.

Preprocessing, output buffers, cache entries and runtime objects also use memory. Reducing the weight buffer alone may barely change the total. The explicit estimates omit maps, object headers, temporary allocations and the browser itself. They must not be relabelled as measured RAM or deployment cost.

Apply the same evidence at different depths

• Guided exploration: predict cold misses for both cohorts, run them, and explain the observed counts. Compare two precision modes and identify one reason the faster-looking result may not generalize.
• Implementation practice: reproduce the component’s outputs with the downloadable runtime from the previous lesson. Add a full-input cache, test that changed inputs and model versions cannot reuse stale entries, and measure matched workloads. Code and independent review are needed for implementation proficiency; the browser report alone is practice.
• Decision practice: write a short recommendation with a numerical error tolerance, workload assumptions, repeated timing range, memory exclusions and a follow-up latency experiment. Consider cold starts and unpredictable input traffic before approving a change.

Keep the exported report with your assumptions. These observations support a bounded comparison on your device. They do not establish deployment readiness, accelerator performance or learner mastery. The optional compute-budget lab supplies a separate real CPU/WebGPU matrix experiment; it is not GPU training for this classifier.