How would you evaluate a code generation model for prod...

To evaluate a code generation model, use pass@1 at the system’s actual production sampling temperature as your primary accuracy metric. Supplement this with pass@10 to characterize the model’s ceiling capability. Additionally, measure latency-to-first-token and cost per accepted suggestion. Finally, report accuracy versus cost on a Pareto frontier rather than relying on a single aggregate number.