Respond to an incident.

First separate the symptom from the failure mechanism

An alert says something deserves attention. It does not identify the cause or choose a response. A service timeout, a response for the wrong request and a valid-looking wrong prediction can all produce a failed task, but they need different handling.

Entering midway: an incident is an observed service problem being investigated and managed. Containment limits further impact. Rollback restores a previous version for future requests. A fallback is an alternative invoked under specified failure conditions. A probe checks behavior without delivering those results as new user traffic. Provenance records where data, code and artifacts came from.

The lab begins with a candidate already serving all simulated requests. This models the response to a regression that escaped release checks, not a recommendation to skip staged deployment. Three public scenarios reuse the actual local Iris model component and a separate Python reference. Faults and service costs are controlled teaching assumptions; no external service is called.

Draw the response path before changing it

Follow the workbench’s request → delivery gate → active route → conditional fallback view. Each control acts at a different point:

• Pause withholds the next requests before model delivery. It creates unmet demand, not successful service.
• Fallback tries the stable component after an unavailable candidate or rejected response identity. It adds another call and its cost.
• Rollback routes future requests directly to stable. It preserves old evidence. If delivery is paused, rollback leaves it paused.
• Resume opens a paused route after a fresh matching probe passes. New traffic still needs observation.

The simulation assumes the stable service is healthy and available. If both versions depend on the same broken upstream data source or infrastructure, switching versions may not help. A real operating plan needs evidence about shared dependencies, fallback capacity and the consequences of withheld demand.

A valid artifact can still be the wrong artifact

The scaler scenario changes preprocessing spreads and updates dependent hashes consistently. The bundle loads, its internal references match and its probabilities are finite. Yet its outputs differ from the intended reference model.

Expand the provenance panel. Compare the candidate and stable preprocessing/model digests, then inspect the numerical receipts. Hash agreement establishes byte consistency with the declared digest. It is not a signature from a trusted publisher, proof of sound training or evidence that a changed model meets the task requirements.

The independent traffic fixture contains twenty cases: eighteen valid requests and two deliberately invalid requests. The reference expects the invalid requests to be rejected. A correct rejection passes the contract check without producing a prediction. A failed parity check compares the candidate’s probability with the recorded intended probability within a tolerance of 0.000001. It measures regression agreement, not classification accuracy on a new population.

Worked impact: eighteen parity failures in twenty attempted requests give 18 / 20 = 90% contract/parity failure for this particular cohort. Among its eighteen valid prediction requests, all eighteen differ. Name the denominator and failure definition; do not report this as “90% of real users harmed” or “90% classification error.”

Recovery only handles the failures it detects

In the timeout scenario, the candidate call is unavailable before returning a model result. Fallback can send the request to stable. In the receipt scenario, valid numerical outputs carry the wrong release identity and are rejected; fallback can also recover those requests. The two invalid inputs are correctly rejected before any successful receipt is constructed, so they do not trigger receipt fallback.

The scaler scenario returns acceptable schemas and identities. Its wrong probabilities do not activate a timeout/receipt fallback. Adding retries to that same deterministic computation would repeat the mistake. Restore the intended preprocessing or roll back, then run an independent numerical check.

Cost example: a timed-out candidate is assigned 10 ms of work, and the stable simulated service call costs 7 ms. Fallback therefore costs 17 ms per request, or 340 ms summed over twenty requests. Direct stable service costs 20 × 7 = 140 ms under the same assumptions. These are invented accounting costs, not measured wall-clock latency. They omit real queueing, human work and infrastructure overhead.

Keep raw attempt failures separate from final delivery failures. Fallback may restore service while the candidate remains broken. Counting only final successes can hide a continuing defect and extra work.

Respect the boundary between diagnosis and action

An observer can inspect traffic and run diagnostic probes. In this role exercise, only an operator may change routing, pause delivery, change fallback or resume. A denied action is recorded but does not alter the service state.

The role selector is intentionally available to the learner. It is an inspectable local rule, not authentication or a real client-side security boundary. Real systems must enforce access at the trusted execution boundary, identify the responsible actor and preserve an audit record. A document, model output or dashboard label should not grant itself authority to operate a service.

For decision practice, name who owns the incident, who can authorize containment and who must review recovery. This lesson does not send messages or perform real operational actions.

Verify the actual state you intend to resume

A passing stable probe cannot approve the candidate. A probe taken before changing the active route or fallback settings does not establish behavior for the new configuration. The lab tracks configuration changes and rejects stale approval.

Try pausing, probing stable, rolling back and then resuming. The route change invalidates the earlier probe. Run a fresh stable probe after rollback. Resume can then proceed, and the next traffic cohort provides additional scoped recovery evidence. A candidate probe whose final results were rescued by fallback cannot approve resume: raw candidate failures still matter.

This deliberately conservative sequence teaches matching evidence to the state being approved. Real recovery criteria depend on the incident, system and risks; a twenty-case public parity suite is not a universal production policy.

Preserve the timeline and distinguish kinds of evidence

The timeline numbers actions; it does not measure elapsed minutes. It includes failed access attempts, containment changes, diagnostics and traffic. Diagnostic probes have their own work total and do not increase the user-traffic denominator. Withheld requests are counted separately from attempted traffic.

After rollback, inspect the first failed cohort. Its probabilities and outcomes remain unchanged. If the first twenty attempts had eighteen failures and a later twenty all pass, the cumulative record still has eighteen failures out of forty attempts. Reporting only the latest healthy window would conceal the earlier impact.

The simulation cannot model actual downstream harm, recall delivered outputs or compensate affected users. An incident report should distinguish observed technical failures, known consequences and consequences still being investigated. Do not infer affected people from a synthetic request counter.

Make the next experiment answer the cause

A useful follow-up names a hypothesis, a controlled change, a comparison and a failure criterion. For a scaler regression: hold model weights and requests fixed, restore the intended scaler, and require independent reference agreement across complete and missing-value inputs. Then add a regression check that would have exposed the handoff before release.

For a receipt problem, test task/release identity independently of plausible numerical content. For a timeout, test unavailable-service handling, fallback capacity and the failure of shared dependencies. A happy recovery case alone does not cover those boundaries.

• Guided practice: export the incident timeline, explain the chosen response and identify what remained unknown when you acted.
• Implementation practice: add a controlled failure and recovery test to the local model component or assistant starter. Preserve immutable receipts, test denied mutations and stale approval, and run an unfamiliar case for independent review.
• Decision practice: write a short report with affected version, evidence, containment rationale, owner, recovery criteria and follow-up experiment. Include the cost of waiting, withholding service and fallback under explicit assumptions.

The report field is learner-authored evidence. Its presence or length does not establish the quality of reasoning. These public cases are repeated development practice; human artifact review, representative learner pilots and the broader operating-plan project remain pending.