Use AI to Draft Tests, Then Challenge the Tests

AI can write tests that pass while missing the failure you care about. A test suite is useful only if it distinguishes correct code from plausible wrong code. A small bank of repeatable tasks from A Tiny Evaluation Set for Your Favorite Coding Prompt lets you test whether the test-writing prompt improves.

For speed_bonus(seconds), the contract says 100 at 0, subtract 2 per second, floor at 0, and reject negative inputs. I ask: Write tests for the contract below. Cover a normal value, the floor boundary, a value past the floor, and negative input. Do not infer extra requirements. For each test, name the rule it checks. Now I can see whether every rule has a test. But coverage labels are not proof.

Imagine a buggy function that returns 100 - 2 * seconds without the floor. Would the generated tests fail it? If not, add a test past the boundary. Imagine another that sorts input in place despite promising a new list. Would the tests detect mutation? If not, add an original-input check. This is a simple form of mutation thinking: introduce a plausible defect mentally and ask whether a test catches it. You can use tools to automate such checks later, but the habit starts on paper.

Do not accept tests solely because they look sensible. Run them against the real code. Read each assertion and make sure it checks the intended value, not merely that the function returns something. The Python standard library’s unittest documentation is a reliable reference for test structure and assertions. Try it: Ask AI for five tests for a small function. Write one deliberately wrong implementation on paper. If the tests would still pass, revise the prompt to target the missed rule. Keep learning: A Tiny Evaluation Set for Your Favorite Coding Prompt, Debugging With AI: Reproduce the Bug Before Requesting a Fix, and How to Test a Prompt on More Than One Example.