Establish prompt engineering baseline with Gold Standard tests

- introduce Gold Standard evaluation corpus
- document decision taxonomy
- define prompt-engineering methodology
- add regression workflow
- establish Prompt Version 2 baseline
- validate decision_simple, decision_deferred and decision_none
This commit is contained in:
2026-07-30 12:13:10 +02:00
parent 07b0d80113
commit f7ad9ba51f
43 changed files with 1288 additions and 45 deletions
+21
View File
@@ -0,0 +1,21 @@
# decision_deferred
Tests that a process decision to defer a substantive decision is still extracted
as a decision.
The transcript contains several candidate options for the weekly dashboard, but
the group does not choose any of them. Instead, Mira explicitly says not to
decide today and Jonas agrees that more input is needed.
Ground truth:
- No substantive dashboard schedule decision is reached.
- One process decision is reached: the substantive decision is deferred until
more information is available.
Typical LLM mistakes:
- Extracting Monday, Tuesday, or Friday as the chosen dashboard day.
- Treating "I like shorter" as an approval.
- Missing the deferral because the substantive decision is unresolved.
- Treating "no decision today" as equivalent to no decision at all.
@@ -0,0 +1,31 @@
{
"facts": [],
"decisions": [
{
"decision": "The substantive decision is deferred until more information is available.",
"evidence": "Mira: Okay, let's not decide this today. Jonas: Agreed, we need more input."
}
],
"todos": [
{
"task": "Lea will bring Dana's feedback about the weekly dashboard next time.",
"responsible": "Lea",
"deadline": "next time",
"evidence": "Lea: I will bring Dana's feedback next time."
}
],
"questions": [],
"positions": [
{
"speaker": "Jonas",
"position": "Jonas prefers moving the weekly dashboard to Monday morning.",
"evidence": "Jonas: I would prefer moving it to Monday morning."
},
{
"speaker": "Lea",
"position": "Lea thinks Monday is difficult for support.",
"evidence": "Lea: Monday is rough for support."
}
],
"technical": []
}
@@ -0,0 +1,19 @@
Mira: We need to talk about the weekly dashboard.
Jonas: I would prefer moving it to Monday morning.
Lea: Monday is rough for support. We usually have backlog cleanup then.
Mira: Tuesday might work, but I am not sure.
Jonas: Or we keep it on Friday and just make it shorter.
Lea: I like shorter, but I need to check with Dana first.
Mira: Okay, let's not decide this today.
Jonas: Agreed, we need more input.
Lea: I will bring Dana's feedback next time.
Mira: Thanks, that will help.