Establish prompt engineering baseline with Gold Standard tests

- introduce Gold Standard evaluation corpus
- document decision taxonomy
- define prompt-engineering methodology
- add regression workflow
- establish Prompt Version 2 baseline
- validate decision_simple, decision_deferred and decision_none
This commit is contained in:
2026-07-30 12:13:10 +02:00
parent 07b0d80113
commit f7ad9ba51f
43 changed files with 1288 additions and 45 deletions
+11
View File
@@ -0,0 +1,11 @@
# question_simple
Tests extraction of an explicit open question.
The transcript also contains a non-task: Kai says he can ask finance but explicitly does not accept it as a task yet.
Typical LLM mistakes:
- Creating a todo for Kai despite his correction.
- Missing that the group decides to leave the issue open.
- Treating "support package" as enough information to answer the question.
+27
View File
@@ -0,0 +1,27 @@
{
"facts": [
{
"fact": "The vendor invoice came in this morning.",
"evidence": "Kai: The vendor invoice came in this morning."
},
{
"fact": "The invoice line item says support package.",
"evidence": "Kai: I don't know. The line item just says support package."
}
],
"decisions": [
{
"decision": "The support-hours invoice issue will remain an open question for now.",
"evidence": "Ruth: Fine. Let's leave it as an open question for now."
}
],
"todos": [],
"questions": [
{
"question": "Does the vendor invoice include the extra support hours from March?",
"evidence": "Ruth: Does it include the extra support hours from March?"
}
],
"positions": [],
"technical": []
}
+19
View File
@@ -0,0 +1,19 @@
Kai: The vendor invoice came in this morning.
Ruth: Does it include the extra support hours from March?
Kai: I don't know. The line item just says support package.
Ruth: Then that is still open.
Kai: I can ask finance, but I am not taking that as a task yet.
Ruth: Fine. Let's leave it as an open question for now.
Kai: Understood.
Ruth: Anything else on invoices?
Kai: No.
Ruth: Then next item.