How do you build a RAG system with an audit trail?

The test is simple and unforgiving: pick an answer from six months ago and reconstruct it. If any of the following is missing, you cannot.

Question

The input as received, including the identity of the person who asked and the access rights that identity carried at that moment.

Retrieval set

Which chunks were returned, from which documents, at which version, with the scores. Not just the ones cited in the answer — the ones considered.

Document state

For each source: the version identifier and its validity interval, so that 'in force on the date of the question' is a computable statement rather than an assumption.

Prompt

The assembled prompt, or a hash of it plus the template identifier and the variables. A template that changed silently is the most common cause of an answer that cannot be reproduced.

Model identity

Provider, model name, version, and the generation parameters. If the endpoint is external, this is the only record you will have of what actually answered.

Output

The answer as returned, before any post-processing, and the post-processed form if they differ.

Disposition

Who saw it, whether a human approved it, and what happened next — the part most systems forget, and the part an inspector asks about first.

A log the application can edit proves nothing, because the first question anyone asks of it is whether it was edited. The practical bar is tamper-evidence: append-only storage, records chained by hash so that altering one invalidates those after it, and write credentials separated from the application's own. Cryptographic timestamping raises the bar further where the record may end up in a dispute.

This is where an event-sourced substrate earns its keep: the record of what happened is the primary artefact and the queryable views are derived from it, rather than the record being a by-product of a mutable state that somebody can update. The trade-offs of that choice, including the ones people underestimate, are in event sourcing in practice.

Retention has to be decided in the same breath. An audit record kept forever is a liability under the GDPR, and one purged on a ninety-day default is useless for an obligation that runs for years. The retention period is a governance decision with a named owner, not an infrastructure default.

The audit record is the one component that deliberately keeps everything, which makes it the component most likely to become an unlawful processing operation. Three moves keep it defensible. Keep identities as stable pseudonymous references resolvable through a separate, access-controlled mapping, rather than names in the log. Store the question verbatim only where the obligation requires it, and otherwise store a hash plus the classification. And exclude personal data from the index in the first place, so that retrieval records cannot accumulate it — the design decision behind the Federfarma deployment, where personal data is excluded at ingestion rather than filtered at answer time.

Three things, in ascending order of value. It lets you answer an inspector: what the system said, on what basis, and who acted on it. It lets you debug a regression: an answer that changed when neither the question nor the documents did means the model or the template moved, and without the record you are guessing. And it lets you improve retrieval honestly — the set of chunks considered but not cited is the dataset that tells you where the index is weak.

The organisations that get this right treat the record as a product requirement with an owner, not as observability. Observability tells you the system is up. The audit trail tells you what it did.