You have ground-truth relevant documents for each query...

A strong evaluation strategy begins with standard classification metrics such as precision@k, recall@k, and F1@k, defined by the intersection of retrieved documents and ground truth. However, these metrics are fundamentally order-invariant. In a RAG pipeline, the order of retrieved documents is crucial because downstream models have limited context windows. Therefore, relying solely on these metrics is insufficient as they fail to account for the rank-dependent nature of document utility.