RAG architecture and document ingestion

Retrieval-augmented generation (RAG) looks up relevant source material at question time and supplies it to a language model. A typical pipeline ingests documents, extracts text and metadata, splits content into chunks, creates searchable representations, retrieves candidates, and constructs a grounded prompt. The answer should retain enough source identity for citations and later auditing. Ingestion quality limits answer quality. Preserve document IDs, versions, timestamps, and access labels; handle tables and scanned files deliberately; and remove or quarantine malformed records. Make the pipeline repeatable so a source update can be traced to changed chunks and indexes. Do not assume every file should be indexed. For each ingested record, preserve a stable source identifier, document version, location, update time, and any access-control metadata needed later. Normalize formats deliberately: tables, headings, footnotes, and scanned pages can lose meaning if flattened carelessly. Make ingestion repeatable and observable, with counts for accepted, rejected, changed, and deleted records. Test reprocessing the same source to ensure it does not create duplicate chunks. A RAG answer can only be as current and complete as its indexed sources, so define who owns refresh schedules and how source deletion propagates. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: Why preserve source IDs and versions in a RAG pipeline?