Monitoring, feedback, and rollbacks

After release, monitor application errors, latency, usage, cost, and quality signals. User feedback is useful but should be interpreted carefully: a click or rating is not automatically a verified quality label. Sample outputs for authorized review, protect sensitive records, and connect recurring failures to a maintained evaluation set. Watch for changes in source data, retrieval coverage, and user questions. Version prompts, models, application code, and indexes so a problematic change can be identified and reversed. Use staged rollout and compare the candidate with the current version on the same checks. Define rollback authority and a recovery procedure before a release. Monitor service health and product quality separately: endpoint errors can be low while groundedness silently degrades after a source change. Sample outputs for authorized review, protect feedback data, and turn verified failures into regression tests. Version the prompt, model, index, and application release together so the active configuration can be reconstructed. Roll out changes gradually where possible and define a measurable pause or rollback trigger in advance. Practice a rollback before a production incident so the team knows which version to restore and how to confirm recovery. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What makes a rollback practical?