Your build starts here
Replace bracketed details with your own goal, context, and checks. Keep the first build small enough to inspect.
Create a calibration exercise for judging code-review comments. The relevant code excerpts are [excerpts], the review criteria are [criteria], and a small human-reviewed reference set is [labeled comments and reasons].
Define pass, fail, and insufficient-evidence outcomes with concrete anchors. A passing finding should identify a supported trigger and practical effect in the supplied code. Avoid rewarding confident language or long explanations when the claimed bug is unsupported.
Compare the candidate grader's decisions with the reference set. Include concise and verbose comments with the same substance. For pairwise choices, specify a test with candidate order swapped. Report disagreements as cases to inspect, not as automatic proof that the human or model is wrong.
Return the rubric, calibration cases, and a measurement plan. Give brief evidence-based rationales and distinguish checks that can be executed from judgments that still need review.Make this prompt yours
Use before relying on a model grader for nuanced findings. Agreement on a small sample does not prove broad reviewer reliability.
You might also hear
- judge calibration
- rubric anchors
- code review grader
The docs behind it
These references explain the underlying concepts and APIs. The prompt templates and project plans are original examples to adapt to your own setup.
Research sources reviewed .
Editorial review · October 28, 2026
Supported: Original, unexecuted judge-calibration template. It asks for human-labeled anchors and treats disagreement as evidence to inspect, consistent with documented model-judge bias and calibration limits.