Model capabilities and selection

Models differ in language capability, context-window limits, supported modalities, structured-output behavior, latency, availability, and price. Benchmark candidates on representative examples instead of choosing solely from a leaderboard. Include difficult inputs, long context, ambiguous requests, and the expected refusal cases. A smaller or hosted model can be a better fit when it meets the target with lower latency or a clearer data boundary. Record the selected model identifier and relevant settings with each evaluation. Check current endpoint availability, region, rate limits, data handling terms, and supported features in the workspace. Separate model capability from application quality: retrieval, instructions, and validation can materially change results. Create a comparison set that reflects the actual workload rather than generic trivia. Include domain terms, noisy input, long context, multilingual examples if relevant, structured responses, and cases where the model must decline to guess. Evaluate each candidate with the same prompt and retrieval evidence, then inspect both aggregate scores and individual failures. Record endpoint name, model version, decoding settings, region, latency, and estimated cost. Confirm deployment and data-handling requirements with the current service documentation; benchmark results from a different model version or workspace may not transfer to production. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What is the strongest basis for selecting a model?