Monitor what changed.

More mobile traffic is a question, not a verdict

A service normally receives equal numbers of desktop and mobile requests. Today seven out of eight requests come from mobile devices. Perhaps a campaign brought new users. Perhaps a client update changed measurements. Perhaps the predictor still works perfectly. The input mix alone cannot distinguish those explanations.

Entering midway: a prediction is a proposed answer; a label is an observed reference answer used to check it. An error means prediction and label disagree. A cohort is a group of requests collected in the same time window. Input drift means the observed input distribution changed relative to a reference. Performance degradation means outcomes became worse according to a specified measure. Neither implies the other in every case.

The lab uses six small fictional cohorts and a programmed predictor. It is a controlled teaching example, not a fitted production model or a claim about desktop and mobile users. Advance time yourself, inspect evidence, and choose which labels or alerts deserve limited attention. No animation or background timer is required.

Read the time line before reading the score

Each window brings eight requests. Their usual labels arrive two ticks later. At tick three, window one’s full labels arrive while window three’s requests are new. The label must stay attached to its original request and source window. Plotting every arriving error as if it belonged to today’s traffic would misattribute the problem.

The final two ticks bring labels without new traffic. Before all labels arrive, the dashboard must distinguish unknown from zero. With zero labels, the fraction “errors divided by labels” is undefined. With two reviewed labels and no errors, zero out of two is an observation about those two cases—not all eight.

The lab’s early-label budget represents a faster review process. It allows at most two requested labels per tick, and zero, four or eight across the replay. Scheduled labels are free under this simulation’s assumptions. In a real system, delayed feedback can also be incomplete, incorrect or expensive. This example deliberately makes all scheduled labels eventually available and correct so you can isolate delay and selection effects.

Connect a bar to its arithmetic

The reference mobile share is 4 / 8 = 0.5, or 50%. Seven mobile requests out of eight give 7 / 8 = 0.875, or 87.5%. Their absolute difference is 0.375: 37.5 percentage points. “Absolute” means we care about the distance from the reference in either direction.

The lab raises an input-mix alert when that distance is at least the chosen 25- or 50-percentage-point threshold. With the lower threshold, 37.5 qualifies; with the higher one, it does not. This is a simple threshold rule, not a statistical test that establishes a meaningful population change. Eight cases make a small, noisy sample in real use.

Practice: six mobile requests out of eight give 75%. The difference from 50% is 25 percentage points, so it meets the lower threshold exactly. That is not the same quantity as a 25% relative increase: the relative increase from 50% to 75% is 50%. Keep the unit visible.

The error bar uses the number of observed labels as its denominator. Four errors among eight labels give 4 / 8 = 50%. Four errors among only four selected labels give 100% for that selected sample. Both statements can be true at different times without the predictor changing.

Choose labels without hiding who is missing

Reviewing convenient examples can create selection bias. Suppose two reviewed desktop requests are correct while four mobile requests have no labels. Reporting “100% accuracy” without the count and selection method invites a false conclusion about the whole cohort.

The workbench displays each subgroup’s labelled count, total count and observed error rate. A subgroup with no labels stays unknown. Increasing the minimum label count can prevent alerts based on one or two examples, but a minimum count alone does not guarantee representative sampling. Four labels from a single subgroup may still miss a failure elsewhere.

Try two policies on window three: request the two named desktop cases, then restart and request the two named mobile cases. Predict how the observed error bar changes before the scheduled labels arrive. The differing early results reflect which cases were inspected. They are not independent estimates of the same population obtained by random sampling.

Targeting suspicious cases can be useful for finding faults quickly. Use that sample to investigate the fault, while collecting separate representative evidence to estimate its prevalence. Do not silently relabel an investigation sample as a random accuracy test.

Spend attention as carefully as labels

Alerts compete for investigation. The lab gives you one, three or six investigation slots. Clicking “Investigate” records that you spent attention on a source window at that tick. It does not diagnose the cause, repair the predictor or roll back a release.

Input-mix and observed-error signals are grouped by source window. Later evidence can add a reason, but it does not change the first alert time. This prevents repeated observations of one issue from filling the entire queue in this exercise. Real alerting also requires ownership, escalation, suppression, resolution and checks that suppression is not hiding a continuing failure.

An investigation delay is the investigation tick minus the source-window tick. Reviewing window three at tick five gives a two-tick delay. This measures when attention occurred, not when a fix was deployed. A label delay may impose a minimum detection delay unless earlier review supplies useful evidence.

Practice: with one review slot, spending it on a harmless input shift leaves none for a later performance failure. Raising thresholds may reduce that unnecessary review, but it can also silence a meaningful failure. Compare policies with the same cases and explain both changes instead of maximizing a single score.

Audit outcomes after the decisions

At tick eight, the lab reveals the complete cohort outcomes. Its fixed evaluation definition calls a window degraded when at least 25% of all eight predictions are wrong. That definition stays fixed when you change alert thresholds, so you cannot improve the score merely by redefining failure.

Three different questions matter:

• False alarm: did an alert fire for a window without defined degradation? A shift alert may still correctly detect an input change; “false” here refers specifically to the performance objective.
• Missed signal: did a degraded window fail to raise any alert?
• Missed review: did a degraded window receive no investigation, even if an alert was raised?

The false-alarm count uses raised alerts as its displayed denominator. Miss counts use degraded windows. No alerts means the false-alarm proportion is undefined, not evidence of perfect monitoring. A low false-alarm count achieved by silencing everything can coexist with serious misses.

The audit cannot retroactively add an investigation. Restarting after seeing the cases is useful policy development, but it is repeated public practice, not untouched evaluation. The source code is inspectable; delayed disclosure is an instructional convention, not a security boundary.

Turn the replay into an operating decision

• Guided exploration: identify one input change without degradation and one degradation without input drift. Explain when you had enough evidence to distinguish them.
• Implementation practice: build a monitor that assigns delayed labels to stable request IDs and source windows. Test empty denominators, duplicate feedback, unknown IDs, subgroup coverage and label arrival order. Add an unfamiliar replay and retain the runnable code and results for independent review.
• Decision practice: propose alert ownership, review capacity and an escalation rule. State which sample supports prevalence estimates, which investigation could justify rollback, and what evidence would change your decision.

Export the settings, replay observations and a short justification. Connect monitoring to the reversible-deployment lesson: an alert opens an investigation; evidence and operating constraints determine the response. This guided lesson does not establish implementation proficiency. Independent artifact review, representative learner pilots and broader device testing remain pending.