Establish prompt engineering baseline with Gold Standard tests

- introduce Gold Standard evaluation corpus
- document decision taxonomy
- define prompt-engineering methodology
- add regression workflow
- establish Prompt Version 2 baseline
- validate decision_simple, decision_deferred and decision_none
This commit is contained in:
2026-07-30 12:13:10 +02:00
parent 07b0d80113
commit f7ad9ba51f
43 changed files with 1288 additions and 45 deletions
+11
View File
@@ -0,0 +1,11 @@
# facts_simple
Tests extraction of objective facts from a short status update.
The transcript includes a question and a technical statement, but no decision or todo.
Typical LLM mistakes:
- Treating "Good" as approval of a decision.
- Turning "No decision needed today" into a decision.
- Missing that the scanner gateway details are technical as well as factual.
+32
View File
@@ -0,0 +1,32 @@
{
"facts": [
{
"fact": "The old scanner gateway is still running in aisle three.",
"evidence": "Tom: The old scanner gateway is still running in aisle three."
},
{
"fact": "Aisles one and two moved to the new gateway last week.",
"evidence": "Tom: Yes. Aisles one and two moved to the new gateway last week."
},
{
"fact": "The new gateway is handling live scans for receiving.",
"evidence": "Iris: The new gateway is already handling live scans for receiving."
}
],
"decisions": [],
"todos": [],
"questions": [
{
"question": "Is aisle three the only scanner gateway still left on the old gateway?",
"evidence": "Elena: Is that the only one left?"
}
],
"positions": [],
"technical": [
{
"subject": "Scanner gateway rollout",
"statement": "Aisle three remains on the old scanner gateway while aisles one and two use the new gateway.",
"evidence": "The old scanner gateway is still running in aisle three. Aisles one and two moved to the new gateway last week."
}
]
}
+19
View File
@@ -0,0 +1,19 @@
Elena: Quick status on the warehouse migration.
Tom: The old scanner gateway is still running in aisle three.
Elena: Is that the only one left?
Tom: Yes. Aisles one and two moved to the new gateway last week.
Iris: The new gateway is already handling live scans for receiving.
Elena: Good. Let's keep the rollout note factual.
Tom: No decision needed today.
Iris: Fine.
Elena: Anything else on warehouse?
Tom: No, that is all.