Capstone: evaluate and review a release

Before release, compare the candidate against a known baseline using the same versioned evaluation set. Review correctness, groundedness, refusal behavior, tool permissions, latency, and estimated cost. Inspect individual regressions even when aggregate scores improve. Confirm data access with the deployed identity, test malformed inputs and dependency failures, and verify that sensitive content is not exposed in traces. A release review should name the exact model, prompt, code, and index versions; list known limitations; and identify an owner for monitoring and rollback. Start with a limited rollout when practical, collect feedback under a privacy policy, and define the conditions for pausing or reverting. Turn release readiness into evidence rather than a subjective demo. Run a versioned evaluation set against the current baseline and candidate, inspect critical slices and individual regressions, and check the operational budget under expected load. Verify user-scoped retrieval, tool authorization, refusal behavior, trace redaction, and failure handling with the deployed identity. Record known limitations and a named owner. A release decision should state thresholds, exceptions, rollout scope, monitoring window, and the exact rollback condition so another engineer can reproduce the decision. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: Why inspect individual failures even when aggregate quality improves?