How much should a small result convince you?

One sample is one view of a population

Imagine checking a school equipment model on ten requests. Nine predictions are correct. The sample is those ten observations. The population is the wider set of requests you want to make a claim about. The sampling process determines how observations enter your sample.

The sample accuracy is 9 divided by 10: 90%. It is an estimate of population accuracy, not a guarantee for the next request. A different random sample can contain different easy and difficult cases and give a different rate. This is sampling variation. The live population grid lets you inspect all 100 labels, predictions and errors before drawing samples.

Our sampler draws uniformly with replacement: after every draw, it puts the selected row back. The same ID can appear again. These are simulated independent draws from a fixed distribution, not newly collected real-world evidence. This assumption makes the uncertainty calculation meaningful in the toy experiment. Repeated measurements from the same person or site in real data may be correlated and need grouped sampling and analysis.

Count first, then report a rate

$\hat p = \frac{k}{n}=\frac{9}{10}=0.9$

Here k is the number of correct predictions, n is the number of sampled draws, and p̂, read “p-hat,” is the estimated accuracy. It is a unitless fraction from 0 to 1; multiply by 100 for a percentage. We use the hat to distinguish a sample estimate from the unknown population rate p. In this teaching lab, we reveal p so you can check the method.

A result of 90 / 100 has the same estimated accuracy as 9 / 10 but usually less random uncertainty under the same sampling assumptions. Adding examples does not guarantee that the next estimate moves closer to the truth. Look at many sample intervals, not one convenient result.

Read the interval as a procedure’s uncertainty

For 9 correct predictions in 10 independent draws, the lab’s approximate 95% Wilson interval is about 59.6% to 98.2%. That is a wide range despite the impressive 90% headline. For 90 out of 100, it is about 82.6% to 94.5%. Sample size changes how tightly the data constrain the population rate.

Each line in the plot is an interval calculated from one sample. Its dot is that sample’s accuracy. The dashed vertical line is the known population accuracy. A 95% confidence procedure is designed to contain the fixed population rate in roughly 95% of repeated intervals under its assumptions. Once calculated, one fixed interval either contains the fixed rate or it does not; “95%” is not a probability assigned to that interval. Wilson coverage is approximate and varies with the true rate and sample size.

Ten out of ten successes do not prove perfection. The Wilson interval still has a lower endpoint around 72.2%. Zero out of ten successes also has a nonzero upper endpoint. No observations means no estimate or interval. These edge cases are why we do not simply draw an arbitrary error bar around the displayed percentage.

Optional: how the Wilson interval is calculated

The lab uses z = 1.96, approximately the standard-normal cutoff for a two-sided 95% interval. With the sample estimate p̂ and count n defined above, calculate a denominator d, a shifted center c, and a half-width h:

$d=1+\frac{z^2}{n},\qquad c=\frac{\hat p+z^2/(2n)}{d}$ $h=\frac{z}{d}\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}},\qquad [c-h,\ c+h]$

For k = 9 and n = 10, d ≈ 1.384, c ≈ 0.789, and h ≈ 0.193. Subtract and add h to get approximately [0.596, 0.982]. The square root turns the uncertainty in the fraction into a width. Every quantity is unitless. This formula assumes independent binary outcomes with the same success probability; it does not correct for sample selection, dependent examples, changed deployment conditions or repeated result-based stopping.

A narrow answer can still answer the wrong question

Choose the convenience sampler: ordinary requests scoring below 0.30. At threshold 0.50, every selected case is correctly predicted ordinary. With 200 draws the interval becomes very narrow near 100%. Yet the default whole population has 89% accuracy, including missed urgent requests and false alarms that this sampler never sees.

This is selection bias: the selection process systematically leaves out relevant cases. The interval can describe the selected subgroup accurately while failing to describe the intended population. More of the same restricted data cannot solve this mismatch. Decide whether the unit should be a person, site, request or time period, and select observations to match the intended claim. See the data-quality lesson for group and time boundaries.

Base rates change what an alert means

The base rate is the fraction of actual positives in the population. At threshold 0.50 with 10 urgent requests, the fixed score patterns catch 8 and falsely flag 9 of the 90 ordinary requests. Precision is 8 / 17 ≈ 47.1%, and recall is 8 / 10 = 80%.

Change the population to 50 urgent requests. The same conditional score patterns catch 40 and falsely flag 5 of the 50 ordinary requests. Precision rises to 40 / 45 ≈ 88.9%, while recall remains 80% and the false-positive rate remains 10%. An alert can mean something different where true positives are rare. These deliberately repeated score patterns isolate the base-rate effect; they are not evidence about real equipment requests.

Plan the evidence before seeing the result

Write down the intended population, sampling unit, selection method, metric and number of observations before collecting evidence. If you keep sampling and stop only when the result looks excellent, the usual fixed-sample confidence interpretation no longer applies. The lab is for seeing the mechanism, not collecting independent evidence for a deployment claim.

For a project decision, request both a suitable sample and an uncertainty summary. Check coverage of affected groups and changes over time; a single accuracy interval cannot settle those questions. Engineers can implement and test the sampler and interval calculation. Managers can specify what evidence would change a release decision. Guided learners can explain why “9 out of 10” is informative but insufficient for a near-perfect reliability claim.