Decide where AI belongs.

A useful baseline can be a rule or a better form

The equipment desk wants to identify requests where work is blocked. A rule that flags events within 24 hours is a credible starting procedure, but it answers a different question. Some near events have spare equipment; some later events involve work that is blocked now.

A baseline is a simple reference procedure against which you compare added complexity. It might be a rule, a constant prediction, a human process or the existing workflow. Compare on the same task and examples. Do not weaken the baseline merely to make an AI candidate look better.

In this lab the blocked-work answer is explicitly available, so asking that question on a clearer form could remove the need to predict it. That is an important option. In actual use the answer may be missing, ambiguous or unreliable; gather evidence about that before choosing a solution.

Keep teaching assumptions separate from measurements

The candidate’s scores and the human decisions are supplied hypothetical values. No model is trained and no real reviewers are studied here. Human review includes an error deliberately, because escalation is not an automatic correctness guarantee. The goal is to learn how to compare workflows, not establish the performance of a real product.

The rule uses an assumed 0.1 work-minute per request, the candidate 0.2 and a simulated human review 3. Work-minutes sum effort across tasks. They are not measured elapsed time, money or a claim about model inference speed. Parallel work and queues could change elapsed time; collecting labels and maintaining a system adds other costs.

Compare errors and unfinished work

On all eight supplied cases, the deadline rule makes three false alarms and misses two urgent requests. Suppose a false alarm costs 2 points and a miss costs 10:

$\text{error cost}=3\times2+2\times10=26\ \text{points}$

The counts measure mistakes among decided cases. The points express chosen consequences, not measured financial costs. Change them in the lab to inspect which assumptions matter. Do not add error points to work-minutes: they are different units. A real decision could translate both into a justified common unit, but that needs evidence and explicit assumptions.

The hybrid sends flagged ambiguous cases to a human and uses the supplied candidate for others. With only one human review available, two ambiguous cases remain pending. The decided cases happen to have no mistakes, but one urgent request remains unresolved. Zero decided-case error cost is not successful completion of the task.

When the budget rises to three reviews, the pending cases receive decisions, including a simulated human mistake. A naive error-only score could prefer the unfinished workflow. Always report coverage, unresolved work and resource use beside error counts. Queue delay and harms from deferral remain unmodeled here.

Build, buy, defer or use no AI

Build may offer control but requires implementation, data and maintenance. Buy may reduce some implementation work but still needs task-specific evaluation, operating limits and a plan for failures. Defer can be appropriate when labels, permission or evidence are inadequate. No AI can be appropriate when a clear rule or workflow change meets the need.

There is no reward for choosing the most automated option. Test relevant boundary cases, missing inputs and escalation behavior. Compare a stronger candidate only when the simpler baseline leaves an important problem unsolved. A vendor demonstration on different cases does not establish local performance.

For a decision brief, state the task, credible baseline, observed and assumed quantities, incomplete outcomes, costs, limits and next evidence. For an engineering exercise, implement the rule with tests at exactly 24 hours and with a missing deadline. Guided learners can trace one request through every option and explain why the outcomes differ. These exercises prepare you for independent project evidence; they do not certify a real deployment decision.