The plain-English version
Testing a selected prompt on cases reserved from the prompt-editing process.
Why it matters
It helps distinguish improvement on familiar examples from improvement on new cases.
Let’s make it click
A team tunes an issue classifier on development cases, freezes a candidate, and evaluates it on a separately stored final set.
Vibe Bro’s heads-up
Repeatedly inspecting the final set and rewriting for its failures makes that set part of development.
You might also hear
- unseen prompt tests
- final evaluation set
- evaluation leakage
The docs behind it
Research sources reviewed .
Editorial review · October 28, 2026
Supported: Provider eval guidance supports task-representative test sets and separation from prompt iteration. The warning that repeated tuning consumes a held-out set is sound experimental guidance.