Learn AIWho receives the errors?
Trace a limited review budget through two channels and compare who is missed, reviewed unnecessarily or absent from the evidence.
By the end: Calculate subgroup rates from counts, distinguish competing criteria and justify a capacity decision with explicit limitations.
Start with something concrete.
An equipment desk reports one accuracy score for all requests. But online requests and in-person requests may experience different errors. Only eight can receive urgent review today. Who benefits from the ordering rule, and who is left waiting?
Try the interactive model ↓If four online and four desk requests get reviewed, must their review rates be equal?
Change one thing. Watch what follows.
- Use the default eight-slot ranking. Trace O8 and D2 through the thresholds and capacity rule, then compare channel recall.
- Reserve four slots per channel. Recalculate review rates and recall. Name which requests gain or lose review.
- Lower the desk threshold to 0.30 and reserve only two online slots. Explain the change in who is missed and the assumptions behind accepting it.
- Set an online threshold of 1.00 with four slots reserved online. Explain the unused slots and how changing the allocation policy would affect outcomes.
Loading this browser experiment…
One overall score hides several experiences
A fictional school equipment desk receives requests through two subgroups: online and in person. These groups describe collection channels, not protected characteristics or fixed kinds of people. A real audit would need relevant groups, intersections and input from affected users.
The lab uses 12 online requests and 8 desk requests. Six online and four desk requests truly need urgent review. Supplied scores rank requests; they are not learned probabilities. The true labels are treated as correct for the experiment. In practice, ask who supplied labels, how disagreements were resolved and whether the definition of urgency fits the intended use.
A threshold is only the first gate
The first gate admits requests whose score meets their channel’s threshold. The second gate fills limited review capacity. At the default threshold of 0.50 and capacity of eight, the highest eligible scores win, with dataset order breaking exact score ties.
The flow diagram separates all requests, eligible requests and actual reviews. Lowering a threshold may admit a request without providing a slot. Read the individual trace to distinguish “below threshold” from “capacity deferred.” Neither is an automatic permanent denial: the lab represents one review round, and later outcomes are outside its scope.
Equal counts are not equal rates
In the default ranking, six online requests and two desk requests receive review. Four of the six urgent online requests are caught; only one of the four urgent desk requests is caught. These are worked-example defaults; the workbench recomputes every count when you change controls.
Recall means the fraction of truly urgent requests that receive review. Review rate means reviewed requests divided by all requests in that channel. False-positive rate means ordinary requests receiving review divided by all ordinary requests. Precision means truly urgent requests divided by all reviewed requests. The counts are in requests; the rates are unitless fractions expressed as percentages.
Reserving four slots per channel gives equal counts, but review rates are 4 / 12 ≈ 33.3% online and 4 / 8 = 50% at the desk. Never compare rates without their denominators. If no examples of a group are present, performance is unknown. If there are no actual urgent cases, recall is undefined rather than measured zero.
Different criteria ask different questions
Equal review rates ask whether groups receive review at the same fraction. Equal recall asks whether urgent cases are caught at the same fraction. Equal false-positive rates ask whether ordinary cases receive unnecessary review at the same fraction. These criteria can favor different policies. Matching one does not establish that every relevant harm, benefit or access barrier has been addressed.
Try reserving two online and six desk slots while lowering the desk threshold to 0.30. Online recall falls to 2 / 6, while desk recall rises to 4 / 4. Explain whose urgent requests are delayed by that allocation. A decision needs a reason for the tradeoff, not just the word “fair.” The example is a way to inspect consequences, not a recommendation to use channel-specific thresholds in a real service.
Reservation has its own policy choice: this lab leaves unused reserved slots empty. Other policies could transfer them, rotate service over time or escalate exceptions. Those rules change outcomes and must be tested explicitly. The lab does not assume human review is perfect or that an urgent review guarantees equipment.
Examine the evidence behind the numbers
A group’s measured recall of 1 / 2 and another’s 10 / 20 both equal 50%, but the first is much more uncertain. Revisit sampling uncertainty before interpreting a gap or a match. The twenty synthetic examples here are development evidence, not representative samples that establish real service quality.
For a decision brief, record the population, available capacity, compared policy, per-group counts, chosen criterion, affected users and remaining uncertainty. Request evidence on missing groups, disputed labels, access barriers and appeals. Engineers can implement the two-stage allocator with tests for ties, empty eligibility and unfilled reservations. Independent assessment of an implementation or decision requires new evidence and review beyond this practice lab.
A subgroup has 2 actual urgent requests; 1 receives review. Another has 20 actual urgent requests; 10 receive review. What is justified?
Take this idea with you.
Show group counts, state the criterion you are comparing and account for capacity. Investigate missing coverage, label quality and affected users before recommending a policy; no metric automatically settles fairness.