Chunking and embeddings

Chunking divides long documents into units that can be retrieved and placed into a model context. Very large chunks can dilute relevance or exceed context budgets; tiny chunks can lose definitions and relationships. Choose boundaries that respect headings, paragraphs, tables, or other document structure, then test overlap and size using real questions. Keep chunk metadata linked to its source and position. Embedding models map text into vectors intended to place semantically related items near one another. Use compatible query and document embedding configurations, and rebuild or version an index when the embedding model or preprocessing changes. Similarity is a retrieval signal, not proof that two passages are equivalent. Chunk size is a retrieval design choice, not a universal constant. A useful passage contains enough context to answer a question but stays focused enough to rank well and fit the model context budget. Compare boundaries at headings, paragraphs, and table rows; test overlap only when it preserves a needed relationship. Keep the embedding model and preprocessing version with the index metadata. If either changes, plan a compatible re-embedding and index cutover rather than mixing incomparable vectors. Evaluate retrieval with labeled questions and inspect the actual returned passages, not just vector-distance scores. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What is a common drawback of chunks that are too small?