Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
# decision_none
|
||||
|
||||
Tests a true decision-negative meeting segment.
|
||||
|
||||
The transcript contains discussion, competing preferences, and possible options
|
||||
for the weekly dashboard. No participant approves an option, rejects an option
|
||||
on behalf of the group, assigns a follow-up, agrees to gather more information,
|
||||
or explicitly decides to defer the decision.
|
||||
|
||||
Ground truth:
|
||||
|
||||
- No decision was reached.
|
||||
- No process decision was reached.
|
||||
- No agreed next step was created.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Treating a preference as a decision.
|
||||
- Treating a proposed option as the selected option.
|
||||
- Treating the topic change as an implicit deferral decision.
|
||||
- Creating an action item for Dana even though she is only mentioned as absent.
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"facts": [],
|
||||
"decisions": [],
|
||||
"todos": [],
|
||||
"questions": [],
|
||||
"positions": [
|
||||
{
|
||||
"speaker": "Jonas",
|
||||
"position": "Jonas prefers moving the weekly dashboard to Monday morning.",
|
||||
"evidence": "Jonas: I would prefer moving it to Monday morning."
|
||||
},
|
||||
{
|
||||
"speaker": "Mira",
|
||||
"position": "Mira thinks Tuesday might work better for support.",
|
||||
"evidence": "Mira: Tuesday might work better for support."
|
||||
},
|
||||
{
|
||||
"speaker": "Lea",
|
||||
"position": "Lea suggests keeping Friday and making the dashboard shorter.",
|
||||
"evidence": "Lea: Or we keep Friday and make the dashboard shorter."
|
||||
}
|
||||
],
|
||||
"technical": []
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
Mira: We need to talk about the weekly dashboard.
|
||||
|
||||
Jonas: I would prefer moving it to Monday morning.
|
||||
|
||||
Lea: Monday is rough for support because backlog cleanup starts then.
|
||||
|
||||
Mira: Tuesday might work better for support.
|
||||
|
||||
Jonas: Tuesday is hard for sales, at least this month.
|
||||
|
||||
Lea: Or we keep Friday and make the dashboard shorter.
|
||||
|
||||
Mira: I am not convinced shorter solves the timing issue.
|
||||
|
||||
Jonas: I am not convinced Monday is actually a problem for everyone.
|
||||
|
||||
Lea: Dana might have a view, but she is not here.
|
||||
|
||||
Mira: We are circling now.
|
||||
|
||||
Jonas: Yes, I do not have anything else to add.
|
||||
|
||||
Lea: Same here.
|
||||
|
||||
Mira: Okay, let's move to the budget topic.
|
||||
Reference in New Issue
Block a user