Learn AILearn from routing evidence without fooling yourself.
Expose selection bias, compare specialists on the same contexts, and check confidence against outcomes.
By the end: Explain cost-sensitive contextual selection, controlled exploration, calibration, and policy-induced data bias.
Start with something concrete.
One specialist reports 84% success and another 82%. But the first mostly received easy questions while the second mostly received hard ones. A controller that trusts the raw totals may learn the wrong lesson from its own choices.
Try the interactive model ↓A specialist has never been tried on hard tasks. What success rate do its logs establish for hard tasks?
Change one thing. Watch what follows.
- Load the biased log. Compare the 84% and 82% raw totals with easy-only, hard-only, and the same 50/50 task mix. Explain the reversal.
- Switch between easy and hard context with failure loss 20. Which specialist has lower expected cost in each? Lower the penalty to zero and explain the changed choice.
- Reset the experiment. Stay on hard cases and follow the current rule twice. Why does it keep using fast? Try the other specialist once, then inspect the new choice and total execution cost.
- Load balanced logs and compare claimed confidence with observed hard-case success. Adjust each claim to match that context, then switch to easy cases. Explain why a single global confidence number may be misleading.
Loading this browser experiment…
Choosing a computation is a learning problem
A controller may choose a cheap model, larger reasoner, exact tool, verifier, or search procedure. The right choice depends on context: required exactness, task difficulty, input type, available time, and failure cost. Arithmetic can favor a calculator; debugging can favor inspection and tests; an image task can require a vision encoder. These are conditional decisions under actual tool contracts, not declarations of general intelligence.
A contextual bandit records the task’s features, the chosen strategy, its outcome, and its cost. Unlike a full sequential controller, it focuses on the immediate choice and feedback. A longer workflow also needs execution state and the effects of earlier actions on later opportunities.
The widget uses a small table of observed success frequencies by difficulty and specialist. It updates that table after a call, then selects the smallest execution cost + estimated failure probability × failure loss. It is not a neural router, and its supplied outcome fixtures are not real specialist models.
Why raw success rates can reverse the comparison
Load the biased log. Fast receives 90 easy calls and 10 hard calls. It succeeds on 81 easy and 3 hard calls: 84/100. Careful receives 10 easy and 90 hard calls. It succeeds on 10 easy and 72 hard calls: 82/100.
The overall numbers make fast look more reliable. Within each difficulty group, careful is better: 100% versus 90% on easy, and 80% versus 30% on hard. The controller’s assignments changed the mixture of tasks each specialist faced. This is policy-induced selection bias, with a reversal often called Simpson’s paradox.
Compare both on the same target mix. For a 50/50 mix, fast scores 0.5 × 0.9 + 0.5 × 0.3 = 0.60, while careful scores 0.5 × 1 + 0.5 × 0.8 = 0.90. The balanced preset produces those same aggregate rates because it sends the same number of easy and hard cases to both.
Changing the target mix changes expected performance. Standardizing these two observed contexts corrects this particular mix imbalance. It does not fix unmeasured differences, incorrect labels, tiny samples, or a new distribution outside the supplied cases.
Better success is not always lower cost
Fast costs 1 execution unit and careful costs 5. With failure loss 20:
| Context | Fast expected cost | Careful expected cost | Lower-cost choice |
|---|---|---|---|
| Easy | 1 + 0.1 × 20 = 3 | 5 + 0 × 20 = 5 | Fast |
| Hard | 1 + 0.7 × 20 = 15 | 5 + 0.2 × 20 = 9 | Careful |
Careful is more successful in both groups, but its extra cost is not justified on easy cases under this objective. Raise the failure penalty or change the resource constraints and the choice can change. A high-cost mathematical error can justify an exact solver even if a cheap model is usually plausible. The previous lesson’s utility model makes that tradeoff explicit.
A greedy controller can hide alternatives from itself
Reset the experiment and use hard cases with failure loss 20. Initially both unobserved success rates are assigned an explicit 50% starting estimate. This is a prior assumption, not evidence. Fast has estimated cost 11; careful has 15. The rule chooses fast.
After observing the hard cohort, fast’s rate is 30%, giving cost 15. Careful is still unobserved and still estimated at 15. The lab’s tie rule keeps fast. More calls to fast do not reveal careful’s actual performance. Deliberately trying careful reveals its 80% rate on these cases, giving estimated cost 9 and changing the choice.
Exploration costs work. One ten-case fast cohort costs 10 units; a careful cohort costs 50. The widget reports all execution units spent and caps the log at 400 calls. It never records an unselected action as a failure. Real contextual-bandit methods can use randomized exploration with recorded selection probabilities; offline comparisons may need additional assumptions and corrections. Here the alternative is tested explicitly so the information gap is visible.
The widget replays fixed ten-case cohorts. Repeating them is practice, not new independent evidence. No confidence interval should shrink merely because the same outcomes were replayed many times. Real experiments need new representative cases and uncertainty estimates appropriate to their sampling process.
Confidence needs external calibration
Calibration concerns agreement between predicted probabilities and observed frequencies across comparable cases. If a model assigns 90% confidence to many comparable answers, about 90% should be correct over an appropriate evaluation sample. This does not guarantee correctness on any individual answer.
The fast specialist’s default claim is 90%. On the supplied hard cohort, only 30% succeed: a 60-percentage-point overstatement. The careful claim is 95%, compared with 80% observed on hard cases. Using those claims directly would make fast appear to cost 3 and careful 6, reversing the evidence-based choice of careful at cost 9 versus fast at 15.
Adjusting confidence to match these observed rates is a demonstration, not a general calibration method. Evaluate probability estimates on new cases, preferably across meaningful groups and confidence ranges. A single aggregate rate can hide major context-specific errors. Self-reported certainty, repeated agreement, or fluent prose does not replace independent outcome checks.
Adaptive depth and a stopping decision
A controller can choose how long to search or reason, not just which specialist to call. Another iteration is worthwhile when its expected decision benefit exceeds its cost. Use externally checked outcomes to estimate that benefit; internal confidence can be misleading.
The source exercise gives a diagnostic costing 3 that lowers expected failure loss from 20 to 6. Its gross improvement is 14; net value is 20 − 6 − 3 = 11 if other costs remain equal. If the diagnostic consumes the deadline or leaves no budget to repair, the complete plan must be reconsidered.
Test whether learning actually helped
Freeze the candidate rule before final evaluation. Compare it with a strong deterministic baseline on new tasks, matching tools, budgets, and task mix. Record correctness, latency, execution cost, interventions, and regressions. Training and exploration costs should be reported separately as well as included where the deployment objective requires them.
The same logs used to choose a strategy or adjust its confidence cannot also be an untouched final test. More elaborate learned control is useful only when it adds measurable value. Novel combinations of task features can expose failures that familiar demonstrations miss; report those failures rather than hiding them in a favorable overall average.
After tuning a controller on these logs, which comparison best tests whether it improved?
Take this idea with you.
A controller learns from the actions it chooses. Compare comparable contexts, explore alternatives deliberately, verify probability estimates, and reserve new tasks for evaluation.