Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
# todo_negative
|
||||
|
||||
Tests that vague "someone should" language is not an action item.
|
||||
|
||||
The transcript contains a real decision to keep the issue on the risk list, but no assigned task.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Creating a todo with "someone" as owner.
|
||||
- Assigning Paula, Ravi, or Sam even though they explicitly do not take ownership.
|
||||
- Ignoring the explicit "no owner for now" correction.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"facts": [
|
||||
{
|
||||
"fact": "The customer export takes longer than Paula expected.",
|
||||
"evidence": "Paula: The customer export takes longer than I expected."
|
||||
},
|
||||
{
|
||||
"fact": "There is no owner for the customer export issue for now.",
|
||||
"evidence": "Paula: Right, no owner for now."
|
||||
}
|
||||
],
|
||||
"decisions": [
|
||||
{
|
||||
"decision": "The customer export issue will stay on the risk list for now.",
|
||||
"evidence": "Sam: Then let's just keep it on the risk list. Paula: Right, no owner for now."
|
||||
}
|
||||
],
|
||||
"todos": [],
|
||||
"questions": [],
|
||||
"positions": [],
|
||||
"technical": []
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Paula: The customer export takes longer than I expected.
|
||||
|
||||
Ravi: Someone should probably look at it.
|
||||
|
||||
Sam: Yes, maybe after the release freeze.
|
||||
|
||||
Paula: I don't have capacity this week.
|
||||
|
||||
Ravi: Same here.
|
||||
|
||||
Sam: Then let's just keep it on the risk list.
|
||||
|
||||
Paula: Right, no owner for now.
|
||||
|
||||
Ravi: We can revisit it in planning.
|
||||
|
||||
Sam: Okay.
|
||||
|
||||
Paula: Next item.
|
||||
Reference in New Issue
Block a user