RAG architecture and evaluation

Trace retrieval-grounded answers, test a keyword baseline and diagnose retrieval and citation failures.

By Digital Mind Engineering · Published 11 October 2026 · 4 min read

Published by DME with AI assistance. Examples are educational and identify their assumptions and limitations. Editorial approach and corrections.

Retrieval-augmented generation separates finding evidence from composing an answer. A retriever selects material from an approved collection; a generator answers from that material. This separation helps expose why an answer failed. It does not guarantee truth, and adding a vector database cannot compensate for missing, stale or unauthorized source documents.

Trace one question through the system

For a fictional internal support handbook, the flow is: authenticate the reader, resolve their permissions, retrieve eligible passages, assemble a bounded context, generate an answer with source identifiers, and check the output before returning it. Apply permissions before content reaches the generator. Filtering only the displayed citations is too late if the answer already contains restricted information.

Approved documents → versioned chunks → searchable index
Reader + permissions → eligible retrieval → evidence context
Evidence context → answer + source IDs → checks → reader

Give each chunk a document identifier, revision, title, section and access label. Keep the original document available so a reviewer can inspect the passage in context. Avoid splitting a rule from its exception or a table header from its rows. Chunk size is a tradeoff: small chunks retrieve precisely but can lose context; large chunks can dilute relevance and consume the input budget.

Start with a transparent retrieval baseline

Before adding embeddings, test keyword retrieval on a small collection. This reference experiment uses Python's standard library only. It is not a production search engine, but it establishes deterministic expected behavior for a few questions and reveals vocabulary mismatches.

import re

DOCS = {
    "restore": "Restore backups in an isolated test environment.",
    "release": "Deploy to staging before a production release.",
    "access": "Review access permissions before sharing records.",
}

def tokens(text):
    return set(re.findall(r"[a-z]+", text.lower()))

def retrieve(question, limit=2):
    query = tokens(question)
    scored = [(len(query & tokens(text)), key, text)
              for key, text in DOCS.items()]
    scored.sort(key=lambda row: (-row[0], row[1]))
    return [(key, text) for score, key, text in scored[:limit]
            if score > 0]

assert retrieve("restore backups")[0][0] == "restore"
assert retrieve("unicorn") == []

“Recover a snapshot” may not match “restore backups.” That is a useful failing test, not a reason to silently change the expected answer. Add examples of synonyms and domain abbreviations to the evaluation. Then compare keyword, vector or hybrid retrieval using the same questions and corpus versions.

Evaluate retrieval and generation separately

Create question records with an expected supporting passage and an answer rubric. Include questions with no answer in the corpus, conflicting revisions, restricted sources, and malicious instructions inside retrieved text. Keep tuning questions separate from held-out cases. Track the corpus revision so results remain interpretable after documents change.

Measure recall at the chosen retrieval limit, answer support, citation correctness, abstention behavior, latency and cost. A citation is only useful when its passage supports the claim. Check that returned source IDs belong to the actual retrieved set, then assess the content relation; an identifier check alone cannot establish support.

Plan for updates and empty results

Version the index and document parser. Remove superseded material deliberately and re-run evaluation after ingestion changes. When retrieval returns no reliable evidence, answer that the collection does not support a response and offer a useful next step. Do not turn an empty retrieval result into a confident answer from model memory.

Keep a trace of selected document IDs and evaluation outcomes with minimal personal data. Set explicit context limits and monitor truncation. For tool execution boundaries, see Building a bounded Python agent; for a measured rollout, see Choosing a first AI project.

Technical reference

The original RAG research paper describes combining retrieval with generation. The keyword baseline and failure worksheet here are original teaching examples; they do not reproduce the paper's system or benchmarks.