How to Test a Prompt on More Than One Example

One successful example can be luck. If I plan to reuse a prompt, I need to see whether it works on different inputs and awkward cases. For coding prompts, A Tiny Evaluation Set for Your Favorite Coding Prompt gives you a compact evaluation set you can rerun after each revision.

Suppose a prompt rewrites messy bug reports into three fields: observed behavior, expected behavior, and reproduction step. I create four fictional input notes: A clear report containing all three fields. A report with no reproduction step. A note that mixes two bugs. A note containing speculation rather than observation. Before running the prompt, I decide what success looks like. The model should not invent a reproduction step for case 2; it should flag the missing detail. For case 3, it should separate the bugs or say that the note needs review. For case 4, it should label speculation.

If I change the examples and the pass rule at the same time, I cannot compare prompt versions. Save the test inputs and the expected checks. Run the same set each time you revise the prompt. OpenAI’s evaluation guidance describes using examples and grading criteria to understand model behavior. You do not need a large benchmark to start; four thoughtful cases can reveal whether a prompt is brittle.

Passing four examples does not prove a prompt will always work. It tells me whether the version handles the cases I care about so far. Add a new case whenever you encounter a new failure in real use. Try it: Take one prompt you plan to reuse. Make three normal inputs and one edge case. Write expected behavior for each, run the prompt, and record pass/fail/unclear. Revise one instruction and rerun the same set. Keep learning: A Tiny Evaluation Set for Your Favorite Coding Prompt, Compare Two Prompt Versions Without Fooling Yourself, and How to Tell Whether an AI Answer Is Any Good.