Classification prompt
Classification prompt: the flagship example.
Real-world analogue: a production labeling or extraction prompt (support-ticket triage, invoice field extraction, content moderation) tuned on labeled data.
A short sentiment-labeling prompt is optimized against a held-out set with an
ExactMatch objective. The starting prompt never states the allowed labels,
so the model answers free-form and misses. The optimizer diagnoses that from
train failures, proposes four distinct revisions, and accepts the one that
states the labels, because it wins on the validation split, not because it
looked good to anyone.
This example also exercises the measurement and governance features: acceptance
requires a statistically significant gain (accept_significant), a locked
test split gives an unbiased final estimate, the record reports per-channel
slice scores, and a MaxLength guard confirms the fix did not start
producing long, sentence-style outputs.
Runnable script: examples/classification.py: reproduce offline with python examples/classification.py, or against the Anthropic API with python examples/classification.py --real.
Starting artifact
Decide how the customer feels about the product.
Baseline validation score: 0.0000
Round 1
Train score 0.0000; failures analyzed: t1, t2.
Diagnosis: The artifact never tells the model the allowed labels or the output format, so it answers in free-form sentences that cannot exact-match.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r1-c1 |
0.0000 | none | none | rejected |
r1-c2 |
1.0000 | none | significant (gain +1.000); max_length(20) 0.00->1.00 (ok) | accepted |
r1-c3 |
0.0000 | none | none | rejected |
r1-c4 |
0.0000 | none | none | rejected |
Accepted as version 5ea29372a5cb:
--- 6a5b9f362a1d
+++ r1-c2
@@ -1 +1,2 @@
-Decide how the customer feels about the product.
\ No newline at end of file
+Decide how the customer feels about the product.
+Answer with exactly one word: "positive" or "negative".
\ No newline at end of file
Round 2
Train score 1.0000; failures analyzed: t1, t2.
Diagnosis: The artifact never tells the model the allowed labels or the output format, so it answers in free-form sentences that cannot exact-match.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r2-c1 |
0.0000 | none | none | rejected |
r2-c2 |
1.0000 | none | none | rejected |
r2-c3 |
0.0000 | none | none | rejected |
r2-c4 |
0.0000 | none | none | rejected |
No candidate beat the incumbent this round.
Result
Validation score 0.0000 → 1.0000 (no_improvement, 56 model calls).
Locked test split (scored once at the end): 1.0000 (validation/test gap 0.0000).
Per-slice scores
| slice | n | baseline | final |
|---|---|---|---|
| reviews | 2 | 0.0000 | 1.0000 |
| support | 2 | 0.0000 | 1.0000 |
Token ledger: 3337 in / 640 out.
Integrity: verified (fingerprint 694a9be1d3a062e1). The complete audit record for this page is regenerated on every build.