Use-case selection and requirements

Good use cases have a clear user, a bounded task, accessible and permitted data, and a way to judge whether an answer is useful. Compare a generative approach with search, rules, or a conventional classifier. Establish quality, latency, cost, privacy, availability, and human-review requirements before implementation. High-impact decisions may require a human approval step even when automated answers are accurate on average. Turn vague goals such as “answer accurately” into a rubric: factual support, completeness, relevance, and safe handling of unknowns. Set operational limits such as a maximum response time or token budget. Include out-of-scope questions and cases where the system should abstain. A concise requirement can be tested with a set of realistic examples. For a support assistant, define which product versions are in scope, what evidence an answer must cite, how the system handles missing documentation, and when it transfers a case to a person. Measure false answers separately from unanswered questions because their impact differs. Estimate volume and request size to understand cost and latency. Also identify users who may be affected differently and any data that must not enter prompts, logs, or test fixtures. These decisions constrain the architecture before implementation begins. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: Why include out-of-scope examples in an evaluation set?