Why is a safety leaderboard where lower ASR is always b...

A safety leaderboard focusing solely on Attack Success Rate (ASR) is insufficient because it ignores the model’s utility. A model that refuses all prompts achieves a 0% ASR but provides no value. Therefore, safety evaluation requires a Pareto view that balances refusal rates against capability. Furthermore, current benchmarks like HarmBench do not measure systemic harms, such as hallucinations in high-stakes contexts, which are critical to overall safety.