MLflow tracing and GenAI observability

Tracing records the path of a request through an application, such as preprocessing, retrieval, model calls, and tool execution. A trace can help identify whether a poor answer came from missing source material, prompt construction, a slow dependency, or output validation. MLflow provides GenAI tracing and evaluation capabilities; current APIs and workspace setup can change, so use the version-matched Databricks and MLflow documentation. Instrument useful metadata such as model version, prompt version, duration, and safe retrieval identifiers. Apply redaction and access controls before collecting prompts or outputs, since traces may contain sensitive data. Define retention and sampling policies, and avoid logging credentials or unrestricted user content. Instrument spans around retrieval, prompt assembly, model invocation, tool use, and response validation. Useful metadata includes model and prompt versions, elapsed time, token counts where available, and safe source identifiers. Traces can contain user text and retrieved private material, so choose redaction, access controls, sampling, and retention before enabling broad collection. Avoid storing credentials or unrestricted payloads by default. During an incident, compare traces from successful and failed requests to locate the earliest divergence. Validate instrumentation with a deliberately slow dependency and a rejected output before relying on it operationally. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What should be excluded from application traces?