How would you evaluate an agent that writes and runs code?

Evaluating code-writing agents requires a test-suite-as-verifier approach, such as that used in SWE-Bench. It is critical to recognize the coverage gap: passing tests does not guarantee the code is correct. A senior-level evaluation must specify the scaffold version, the step budget allowed for the agent, and clarify whether the assessment is focused on the model’s performance in isolation or the full system, including all integrated tooling.