Context template
Context template: text-layer changes move the metric with the model fixed.
Real-world analogue: a RAG / retrieval-QA pipeline where you tune how retrieved chunks are assembled into the prompt, with the retriever and model held fixed.
The artifact here is not a prompt but a template that assembles retrieved context and a question into the final message. The model never changes; only the assembly text does. The starting template buries the context after the question and never tells the model to ground its answer, so answers drift. The accepted revision puts context first with a grounding instruction, and the token-level F1 against reference answers moves accordingly.
Acceptance requires a statistically significant gain (accept_significant),
a locked test split gives an unbiased final estimate, and the record reports
per-topic slice scores so a gain on one subject area cannot hide a loss on
another.
Runnable script: examples/context_template.py: reproduce offline with python examples/context_template.py, or against the Anthropic API with python examples/context_template.py --real.
Starting artifact
Question: {question}
You may find this background useful:
{context}
Answer:
Baseline validation score: 0.2981
Round 1
Train score 0.2889; failures analyzed: t2, t1.
Diagnosis: The template puts the context after the question and never instructs the model to ground its answer in it, so answers come from memory.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r1-c1 |
0.2981 | none | none | rejected |
r1-c2 |
1.0000 | none | significant (gain +0.702) | accepted |
r1-c3 |
0.2981 | none | none | rejected |
Accepted as version 2242fb0d30df:
--- 83adf9a5944f
+++ r1-c2
@@ -1,4 +1,7 @@
+Context:
+{context}
+
+Answer using only the facts in the context above.
+
Question: {question}
-You may find this background useful:
-{context}
Answer:
\ No newline at end of file
Round 2
Train score 1.0000; failures analyzed: t1, t2.
Diagnosis: The template puts the context after the question and never instructs the model to ground its answer in it, so answers come from memory.
| candidate | validation score | leakage flags | checks | outcome |
|---|---|---|---|---|
r2-c1 |
0.2981 | none | none | rejected |
No candidate beat the incumbent this round.
Result
Validation score 0.2981 → 1.0000 (no_improvement, 37 model calls).
Locked test split (scored once at the end): 1.0000 (validation/test gap 0.0000).
Per-slice scores
| slice | n | baseline | final |
|---|---|---|---|
| culture | 2 | 0.3472 | 1.0000 |
| science | 1 | 0.2000 | 1.0000 |
Token ledger: 3679 in / 537 out.
Integrity: verified (fingerprint 3c92808b02221ae1). The complete audit record for this page is regenerated on every build.