Databricks AI Search and retrieval quality

Databricks AI Search (formerly Vector Search) is a platform search capability used for retrieval in applications such as RAG. Index setup, source synchronization, endpoint availability, permissions, and supported options depend on the workspace and current product configuration; consult the current product documentation before implementation. Keep index creation and querying behind a narrow application interface so configuration can be tested and changed. Measure retrieval separately from answer generation. For each test question, record whether the correct source appears in the top results, its rank, and whether irrelevant passages crowd it out. Inspect failed queries to find issues in chunking, metadata filters, source freshness, or query wording. Similarity scores should not be treated as calibrated confidence without validation. Treat an index as a serving dependency with its own lifecycle. Confirm its source synchronization behavior, query permissions, freshness expectations, filtering strategy, and supported embedding configuration in current Databricks documentation. At query time, constrain results by user authorization before sending them to a model, then retain document IDs for citation and traceability. Build a small relevance set with expected source documents and measure whether those sources appear in the top results. If retrieval fails, inspect filters, stale records, query wording, chunk quality, and ranking before changing the generation prompt. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What should a retrieval evaluation measure?