Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
# facts_simple
|
||||
|
||||
Tests extraction of objective facts from a short status update.
|
||||
|
||||
The transcript includes a question and a technical statement, but no decision or todo.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Treating "Good" as approval of a decision.
|
||||
- Turning "No decision needed today" into a decision.
|
||||
- Missing that the scanner gateway details are technical as well as factual.
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"facts": [
|
||||
{
|
||||
"fact": "The old scanner gateway is still running in aisle three.",
|
||||
"evidence": "Tom: The old scanner gateway is still running in aisle three."
|
||||
},
|
||||
{
|
||||
"fact": "Aisles one and two moved to the new gateway last week.",
|
||||
"evidence": "Tom: Yes. Aisles one and two moved to the new gateway last week."
|
||||
},
|
||||
{
|
||||
"fact": "The new gateway is handling live scans for receiving.",
|
||||
"evidence": "Iris: The new gateway is already handling live scans for receiving."
|
||||
}
|
||||
],
|
||||
"decisions": [],
|
||||
"todos": [],
|
||||
"questions": [
|
||||
{
|
||||
"question": "Is aisle three the only scanner gateway still left on the old gateway?",
|
||||
"evidence": "Elena: Is that the only one left?"
|
||||
}
|
||||
],
|
||||
"positions": [],
|
||||
"technical": [
|
||||
{
|
||||
"subject": "Scanner gateway rollout",
|
||||
"statement": "Aisle three remains on the old scanner gateway while aisles one and two use the new gateway.",
|
||||
"evidence": "The old scanner gateway is still running in aisle three. Aisles one and two moved to the new gateway last week."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Elena: Quick status on the warehouse migration.
|
||||
|
||||
Tom: The old scanner gateway is still running in aisle three.
|
||||
|
||||
Elena: Is that the only one left?
|
||||
|
||||
Tom: Yes. Aisles one and two moved to the new gateway last week.
|
||||
|
||||
Iris: The new gateway is already handling live scans for receiving.
|
||||
|
||||
Elena: Good. Let's keep the rollout note factual.
|
||||
|
||||
Tom: No decision needed today.
|
||||
|
||||
Iris: Fine.
|
||||
|
||||
Elena: Anything else on warehouse?
|
||||
|
||||
Tom: No, that is all.
|
||||
Reference in New Issue
Block a user