What does it mean for a benchmark to be reproducible, a...

Benchmark reproducibility depends on locking three key pillars: protocol variance (prompt format and decoding settings), data quality (label noise), and contamination status. If any of these shift, the benchmark results become unreliable. The most effective way to ensure reproducibility is to adopt a versioned evaluation framework that pins the entire protocol as code, ensuring that every evaluation run is consistent and auditable.