How would you design an evaluation system that avoids t...

To avoid Chatbot Arena’s failure modes, the system should implement a blind protocol with randomized comparison partners and strictly prohibit privileged submissions. It must utilize stratified sampling to ensure coverage across coding, factual recall, and safety-adjacent prompts, rather than relying on the distribution of site visitors. Additionally, all published scores should include confidence intervals, and there must be a clear, enforced separation between public evaluation comparisons and internal development comparisons to prevent data leakage.