For a RAG product’s truthfulness evaluation, what processes should run on every pull request, what should run nightly, and why?

In a Retrieval‑Augmented Generation (RAG) system, truthfulness comprises two aspects: faithfulness (the answer stays true to the retrieved content) and factuality (the answer aligns with external ground truth). Faithfulness can be evaluated quickly using the model’s own output and does not require external data, so it should be part of the continuous integration pipeline and run on every pull request to give developers immediate feedback. Factuality checks require querying external knowledge bases and possibly performing recall‑based corrections; these are more expensive and should be scheduled as nightly jobs, sized according to the capacity of the external source. Finally, periodic human audits add a qualitative layer of verification and should be scheduled less frequently (e.g., weekly or monthly) to catch systematic issues that automated metrics might miss.