Experiment tracking and controlled comparison

An experiment changes a defined configuration and compares its results on the same evaluation examples. Record the model, prompt, retrieval settings, data snapshot, code version, and evaluation rubric so a result can be reproduced. Change one major factor at a time where practical; otherwise, an observed improvement may have several possible causes. Compare quality together with latency, cost, and failure rates. Investigate regressions by example rather than relying only on an average score. MLflow experiment tracking can help organize runs and artifacts in supported environments. Keep deployment candidates distinct from experiments and require review against acceptance thresholds before promotion. An experiment is a reproducible record of a hypothesis and a controlled change. Store a configuration snapshot with the evaluation results: code revision, prompt, model identifier, retrieval settings, input dataset version, and scoring rubric. Change one major variable at a time when you need causal clarity; use factorial comparisons when interactions matter and you can interpret them. Inspect the examples that regress, not only the mean. Track cost and latency beside quality so an apparent quality improvement does not conceal an unacceptable operational tradeoff. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What improves attribution when comparing two chunking strategies?