Measure the whole workflow.

A working component is not a working assistant

Suppose search misses the robotics schedule, but an exact local lookup returns the meeting time. The final answer can be correct while retrieval failed. Conversely, retrieving the right notice does not prevent the assistant from quoting the wrong fact, overlooking a contradiction or proposing an unauthorized action.

This lab runs nine fictional tasks through a real local sequence: score documents by keyword overlap, read the top results, extract supplied structured facts, check source policy and consistency, optionally call exact lookup or arithmetic, then assemble an answer. Its controller is programmed. It is not a generative language model and does not infer arbitrary facts from prose.

Compare useful baselines

The extractive baseline answers from retrieved current eligible facts when they agree. The tool-assisted baseline starts the same way, then tries one local call for each unresolved requested fact. Exact lookup uses a task-scoped record; arithmetic multiplies a retrieved consistent notebook price by the supplied quantity. A tool is allowed to return no result.

Both baselines receive the same task and documents. The separate evaluator knows the expected outcome for each case. That evaluator is not used by retrieval, answer construction or tool selection. Changing a runtime gate does not change what the evaluator considers correct.

The suite is public development material. Reusing it to choose settings is legitimate practice, but it cannot then serve as an untouched independent test. A project needs unfamiliar cases and review beyond these examples.

Follow failures through the stages

Observation	What it shows	What it does not show
Relevant source missed	Retrieval omitted a source needed for the question or its conflict context	Whether a tool later recovered the answer
Wrong claim returned	A returned fact disagrees with the separate case expectation	Which upstream component caused it
Answerable fact missing	The case had an expected answer that the workflow did not supply	That the workflow invented a false answer
Tool failure	A timeout or mismatched receipt stopped acceptance	That tool output would have been reliable if unchecked
Invalid document proposal	The scripted proposer tried an out-of-contract action	That the action executed
Unauthorized effect	Runtime enforcement failed in the fictional simulation	A real-world security incident

A contradiction can be a retrieval problem. With one retrieved document, the assistant may see only the 17:00 office notice. Within its observed evidence, the answer appears supported. The evaluator sees the full case, including a conflicting eligible notice, and rejects that answer. This is why “supported by retrieved text” is weaker than “correct for the task.”

Keep abstention and completion distinct

A missing holiday-hours case should remain unresolved. A conflicting closing-time case can still return its undisputed fee information. A failed tool receipt must not be accepted as a valid answer. The suite counts appropriate handling of these cases separately from complete factual answers.

An assistant that refuses all nine questions may avoid wrong claims while failing answerable tasks. The “answerable facts missing” column exposes that failure. An assistant that guesses every time may appear more complete while returning wrong claims. Report both usefulness and errors rather than rewarding either extreme.

Count work and explain timing

The lab counts documents scored, documents read and tool calls, including failed calls. For a worked example, scoring three documents, reading two and calling one tool gives 3 scored + 2 read + 1 call. Under the declared teaching costs, sequential latency is 3 × 1 + 2 × 4 + 1 × 12 = 23 ms.

The cost model omits proposal parsing, runtime/evaluation overhead and simulated unauthorized-action work. Those costs are assumptions; the functions do not actually wait for those durations. The separate measured time covers running and evaluating all nine cases on this browser’s CPU, excluding rendering. It is not a measurement of network calls, real user latency or the assumed per-stage budget.

A nearby practice case: hold scoring and reading fixed but make two tool calls. The assumed latency becomes 35 ms because one additional call adds 12 ms. A timeout still consumes its counted call. The controller makes at most one call per missing fact, so this exercise does not hide an unlimited retry loop.

Make comparisons reproducible

Each recorded run freezes its settings and results. Changing the next candidate leaves earlier records intact. Compare one change at a time: k, tools, support checking or runtime enforcement. Inspect the case trace before assigning a cause to a changed aggregate score.

A maximum of six comparisons fits this temporary workbench. Export the configuration, traces, returned evidence, evaluation and measured timing before leaving. These exports are transparent practice artifacts, not authenticated execution logs or independently assessed implementation evidence.

For guided practice, explain one recovered failure and one unresolved case. For implementation practice, reproduce the tool receipt validation and bounded arithmetic, then add a genuinely unfamiliar source/argument case. For a management brief, state which incomplete cases are acceptable, which errors block deployment and what extra review is needed. The integrated assistant project will require a frozen case suite and evidence appropriate to the chosen path.