← All questions
EvaluationMedium· Evaluation & Production

How do you know your prompt change didn't break anything?

A fixed golden set with a regression gate in CI, the same discipline you’d apply to any other code change. Prompts are deployable artifacts, not casual text edits — they need to be versioned, reviewed, and gated on evaluation results rather than on someone eyeballing a handful of examples and declaring it looks fine.

This is one of the most common gaps I see in teams shipping LLM features: there’s no answer to “did this get better or worse,” just a vague sense that it “feels different,” which is exactly the kind of unanswerable question that erodes trust in the whole feature over time. A golden set with a CI gate turns “we changed the prompt and something got worse” into a number you can point at.

Code playground (Python)