Compare Two Prompt Versions Without Fooling Yourself

If version B gives a better-looking answer, I want to know why. Did I improve the prompt, or did I change the task, model, input, or grading rule? A useful comparison holds as much constant as practical. Keep both versions and their results in Build Your Personal Prompt Notebook so you remember what the change actually did.

I record: task, input, model and settings if relevant, prompt version, output, pass criteria, and failures. Then I change one instruction. For example, I add “do not invent missing reproduction steps” to a bug-triage prompt while keeping the same four test reports. I score both versions on the same criteria: required fields present, missing steps flagged, no invented evidence, and result usable by a tester. If B improves one criterion but fails another, I record the tradeoff instead of calling it universally better.

Do not choose the one example where the new prompt shines. Include a normal case, a messy case, and an edge case. If the prompt is intended for repeated work, test it more than once. Model outputs can vary, so a single run should not carry a grand claim.

Sometimes a shorter output is better because it requires less editing. Sometimes more detail is necessary because it exposes an assumption. Measure usefulness for the task, not length or elegance alone. Try it: Take two versions of a prompt you already use. Run both on the same three inputs. Score each against criteria written in advance. Save the version that helps you complete the job, and note what it still misses. Keep learning: How to Test a Prompt on More Than One Example, Build Your Personal Prompt Notebook, and Make a Bad Prompt Better in Three Edits.