Skill / instruction file
Skill / instruction file: model-graded objectives and leakage flags.
Real-world analogue: a longer instruction artifact (an agent's system prompt, a reusable "skill" file, a style guide) optimized against a rubric when there is no single gold answer to score against.
The artifact is a longer instruction file telling a model how to write
executive summaries. There is no gold answer to exact-match against, so an
LLMJudge scores outputs against a rubric, and the run record flags the
whole run as model-graded so a reviewer knows the metric is not
reference-based.
One scripted candidate also pastes a validation example's text straight into the instructions. It scores well, and the leakage heuristics flag it in the record, which is exactly the point: a gain you got by memorizing the held-out set is not a gain.
Acceptance requires a statistically significant gain (accept_significant),
a locked test split gives an unbiased final estimate, and a MaxLength
guard confirms the structured rewrite also keeps summaries tight.
Runnable script: examples/skill_file.py: reproduce offline with python examples/skill_file.py, or against the Anthropic API with python examples/skill_file.py --real.
Starting artifact
# Writing executive summaries
Write a summary of the supplied report. Try to keep it useful for a busy
reader and do not leave out anything that seems important.
Baseline validation score: 0.4000 (model-graded)
Round 1
Train score 0.4000; failures analyzed: t1.
Diagnosis: The instructions give no structure: nothing says what to lead with, how many points to keep, or any length limit, so summaries ramble.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r1-c1 |
0.4000 | none | none | rejected |
r1-c2 |
0.9000 | none | significant (gain +0.500); max_length(70) 0.00->1.00 (ok) | accepted |
r1-c3 |
0.4000 | verbatim_validation_span: candidate embeds 141 chars of validation example 'v1' (input): 'Quarterly churn fell from 9.1% to 6.4% after the onboarding ' | none | rejected |
Accepted as version 5fab3dec585a:
--- 850ddc37c512
+++ r1-c2
@@ -1,4 +1,5 @@
# Writing executive summaries
-Write a summary of the supplied report. Try to keep it useful for a busy
-reader and do not leave out anything that seems important.
+Lead with the single most important finding in the first sentence.
+Then give at most three supporting points, each with its number or date.
+Cut everything a decision-maker would not act on. Maximum 80 words.
Round 2
Train score 0.9000; failures analyzed: t1.
Diagnosis: The instructions give no structure: nothing says what to lead with, how many points to keep, or any length limit, so summaries ramble.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r2-c1 |
0.4000 | none | none | rejected |
No candidate beat the incumbent this round.
Result
Validation score 0.4000 → 0.9000 (no_improvement, 44 model calls).
Locked test split (scored once at the end): 0.9000 (validation/test gap 0.0000).
Token ledger: 6928 in / 1124 out.
Integrity: verified (fingerprint bed9a4f89f5ce601). The complete audit record for this page is regenerated on every build.