Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
# question_simple
|
||||
|
||||
Tests extraction of an explicit open question.
|
||||
|
||||
The transcript also contains a non-task: Kai says he can ask finance but explicitly does not accept it as a task yet.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Creating a todo for Kai despite his correction.
|
||||
- Missing that the group decides to leave the issue open.
|
||||
- Treating "support package" as enough information to answer the question.
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"facts": [
|
||||
{
|
||||
"fact": "The vendor invoice came in this morning.",
|
||||
"evidence": "Kai: The vendor invoice came in this morning."
|
||||
},
|
||||
{
|
||||
"fact": "The invoice line item says support package.",
|
||||
"evidence": "Kai: I don't know. The line item just says support package."
|
||||
}
|
||||
],
|
||||
"decisions": [
|
||||
{
|
||||
"decision": "The support-hours invoice issue will remain an open question for now.",
|
||||
"evidence": "Ruth: Fine. Let's leave it as an open question for now."
|
||||
}
|
||||
],
|
||||
"todos": [],
|
||||
"questions": [
|
||||
{
|
||||
"question": "Does the vendor invoice include the extra support hours from March?",
|
||||
"evidence": "Ruth: Does it include the extra support hours from March?"
|
||||
}
|
||||
],
|
||||
"positions": [],
|
||||
"technical": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Kai: The vendor invoice came in this morning.
|
||||
|
||||
Ruth: Does it include the extra support hours from March?
|
||||
|
||||
Kai: I don't know. The line item just says support package.
|
||||
|
||||
Ruth: Then that is still open.
|
||||
|
||||
Kai: I can ask finance, but I am not taking that as a task yet.
|
||||
|
||||
Ruth: Fine. Let's leave it as an open question for now.
|
||||
|
||||
Kai: Understood.
|
||||
|
||||
Ruth: Anything else on invoices?
|
||||
|
||||
Kai: No.
|
||||
|
||||
Ruth: Then next item.
|
||||
Reference in New Issue
Block a user