A Tiny Evaluation Set for Your Favorite Coding Prompt

If a coding prompt works once, I may want to use it again. Before calling it reliable, I test it on more than one request. I can do this without a giant benchmark. If you change the prompt, Compare Two Prompt Versions Without Fooling Yourself keeps the comparison fair by using the same cases and rubric.

For a prompt that asks AI to write small Python functions, I choose: A straightforward transformation, such as sorting scores without mutation. A boundary rule, such as a bonus with a floor at zero. Invalid input, such as a negative timer value. A task with a missing requirement that should trigger a question. I write the expected behavior before running the model. For each task I check: does the code run, does it satisfy the contract, are the boundary tests meaningful, and are assumptions visible? I save prompt version, input, output, test results, and one failure note.

When I change the coding prompt, I rerun the same four tasks. I do not declare improvement because one output looks cleaner. I count which requirements pass and inspect important regressions. If a revision fixes sorting but breaks error handling, it is not clearly better.

Test cases help, but they do not capture every design choice or security issue. For code that affects real users, review the implementation and its integration. OpenAI’s evaluation guide explains the general practice of using examples and graders to inspect model behavior; my notebook is a small manual version of that idea. Try it: Choose a prompt you use for coding. Make four non-sensitive tasks, with expected checks. Run the prompt and record pass/fail/unclear. Change one instruction and rerun the exact set. Keep learning: How to Test a Prompt on More Than One Example, Use AI to Draft Tests, Then Challenge the Tests, and Compare Two Prompt Versions Without Fooling Yourself.