Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
# decision_deferred
|
||||
|
||||
Tests that a process decision to defer a substantive decision is still extracted
|
||||
as a decision.
|
||||
|
||||
The transcript contains several candidate options for the weekly dashboard, but
|
||||
the group does not choose any of them. Instead, Mira explicitly says not to
|
||||
decide today and Jonas agrees that more input is needed.
|
||||
|
||||
Ground truth:
|
||||
|
||||
- No substantive dashboard schedule decision is reached.
|
||||
- One process decision is reached: the substantive decision is deferred until
|
||||
more information is available.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Extracting Monday, Tuesday, or Friday as the chosen dashboard day.
|
||||
- Treating "I like shorter" as an approval.
|
||||
- Missing the deferral because the substantive decision is unresolved.
|
||||
- Treating "no decision today" as equivalent to no decision at all.
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"facts": [],
|
||||
"decisions": [
|
||||
{
|
||||
"decision": "The substantive decision is deferred until more information is available.",
|
||||
"evidence": "Mira: Okay, let's not decide this today. Jonas: Agreed, we need more input."
|
||||
}
|
||||
],
|
||||
"todos": [
|
||||
{
|
||||
"task": "Lea will bring Dana's feedback about the weekly dashboard next time.",
|
||||
"responsible": "Lea",
|
||||
"deadline": "next time",
|
||||
"evidence": "Lea: I will bring Dana's feedback next time."
|
||||
}
|
||||
],
|
||||
"questions": [],
|
||||
"positions": [
|
||||
{
|
||||
"speaker": "Jonas",
|
||||
"position": "Jonas prefers moving the weekly dashboard to Monday morning.",
|
||||
"evidence": "Jonas: I would prefer moving it to Monday morning."
|
||||
},
|
||||
{
|
||||
"speaker": "Lea",
|
||||
"position": "Lea thinks Monday is difficult for support.",
|
||||
"evidence": "Lea: Monday is rough for support."
|
||||
}
|
||||
],
|
||||
"technical": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Mira: We need to talk about the weekly dashboard.
|
||||
|
||||
Jonas: I would prefer moving it to Monday morning.
|
||||
|
||||
Lea: Monday is rough for support. We usually have backlog cleanup then.
|
||||
|
||||
Mira: Tuesday might work, but I am not sure.
|
||||
|
||||
Jonas: Or we keep it on Friday and just make it shorter.
|
||||
|
||||
Lea: I like shorter, but I need to check with Dana first.
|
||||
|
||||
Mira: Okay, let's not decide this today.
|
||||
|
||||
Jonas: Agreed, we need more input.
|
||||
|
||||
Lea: I will bring Dana's feedback next time.
|
||||
|
||||
Mira: Thanks, that will help.
|
||||
Reference in New Issue
Block a user