Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
# evil_meeting
|
||||
|
||||
Tests a deliberately difficult meeting with interruptions, corrections, topic switches, changed positions, absent referenced people, and near-decisions.
|
||||
|
||||
Every utterance is designed to trigger a common extraction failure. The meeting mentions Omar and Platform, but neither is a participant. It includes an explicit non-decision on migration and a real decision only on excluding FR-7 from Friday's batch.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Extracting a migration decision even though the group says no migration decision today.
|
||||
- Assigning Dana a todo even though she retracts it.
|
||||
- Treating Omar as a participant or technical owner.
|
||||
- Claiming Platform approved something despite being absent.
|
||||
- Losing the correction from "old export" to "nightly CSV job" and from API export to CSV export.
|
||||
- Treating Dana's opinion about rollout appearance as a fact or decision.
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"facts": [
|
||||
{
|
||||
"fact": "Tenant FR-7 still used the nightly CSV job yesterday.",
|
||||
"evidence": "Alex: Fine. So fact: tenant FR-7 still used the nightly CSV job yesterday."
|
||||
},
|
||||
{
|
||||
"fact": "Omar is the customer contact, not the technical owner.",
|
||||
"evidence": "Bea: Omar is the customer contact, not a participant here and not the technical owner."
|
||||
},
|
||||
{
|
||||
"fact": "Platform is the technical owner, but nobody from Platform is in the meeting.",
|
||||
"evidence": "Chen: The technical owner is still Platform, but nobody from Platform is in this call."
|
||||
},
|
||||
{
|
||||
"fact": "Friday's rollout batch still includes DE-2 and NL-4.",
|
||||
"evidence": "Alex: Good. Back to the portal rollout. Friday's batch still includes DE-2 and NL-4."
|
||||
}
|
||||
],
|
||||
"decisions": [
|
||||
{
|
||||
"decision": "FR-7 is excluded from Friday's portal rollout batch.",
|
||||
"evidence": "the portal rollout note will say FR-7 is excluded from Friday's batch. Bea: Agreed. Excluded from Friday's batch. Chen: Yes, put that in."
|
||||
}
|
||||
],
|
||||
"todos": [
|
||||
{
|
||||
"task": "Check the FR-7 mapping table.",
|
||||
"responsible": "Bea",
|
||||
"deadline": "Thursday morning",
|
||||
"evidence": "Bea: Yes, I will check it by Thursday morning."
|
||||
}
|
||||
],
|
||||
"questions": [
|
||||
{
|
||||
"question": "Can FR-7 use the v2 mapping without a customer-side field rename?",
|
||||
"evidence": "Alex: Open question: can FR-7 use the v2 mapping without a customer-side field rename?"
|
||||
},
|
||||
{
|
||||
"question": "Can Omar confirm FR-7's preferred launch window?",
|
||||
"evidence": "Dana: Also, can Omar confirm their preferred launch window?"
|
||||
}
|
||||
],
|
||||
"positions": [
|
||||
{
|
||||
"speaker": "Dana",
|
||||
"position": "Dana wants to migrate FR-7 but recognizes the mapping table may not be clean.",
|
||||
"evidence": "Dana: I want to, but we do not know if the mapping table is clean."
|
||||
},
|
||||
{
|
||||
"speaker": "Bea",
|
||||
"position": "Bea changed her earlier position and now says not to migrate FR-7 until the mapping table is checked.",
|
||||
"evidence": "I said last week we should migrate it. I am changing that. Do not migrate until the mapping table is checked."
|
||||
},
|
||||
{
|
||||
"speaker": "Dana",
|
||||
"position": "Dana thinks excluding FR-7 makes the rollout look messy.",
|
||||
"evidence": "Dana: I personally think excluding FR-7 makes the rollout look messy."
|
||||
}
|
||||
],
|
||||
"technical": [
|
||||
{
|
||||
"subject": "French export failure",
|
||||
"statement": "The old nightly CSV job failed; the new exporter was not running on tenant FR-7.",
|
||||
"evidence": "The old export failed. The new exporter was not running on that tenant."
|
||||
},
|
||||
{
|
||||
"subject": "Export mapping tables",
|
||||
"statement": "The API export uses the v2 mapping table, while the nightly CSV job uses the legacy table.",
|
||||
"evidence": "the API export uses the v2 mapping table, the nightly CSV job uses the legacy table."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
Alex: Okay, quick pass on the portal rollout. Wait, before that, the French CSV export broke again.
|
||||
|
||||
Bea: It did not break again. The old export failed. The new exporter was not running on that tenant.
|
||||
|
||||
Chen: Sorry, when you say old export, do you mean the nightly job?
|
||||
|
||||
Bea: Yes, the nightly CSV job. Not the API export.
|
||||
|
||||
Alex: Fine. So fact: tenant FR-7 still used the nightly CSV job yesterday.
|
||||
|
||||
Dana: I thought Omar owned that tenant.
|
||||
|
||||
Bea: Omar is the customer contact, not a participant here and not the technical owner.
|
||||
|
||||
Chen: The technical owner is still Platform, but nobody from Platform is in this call.
|
||||
|
||||
Alex: Should we decide to migrate FR-7 today?
|
||||
|
||||
Dana: I want to, but we do not know if the mapping table is clean.
|
||||
|
||||
Bea: Also, I said last week we should migrate it. I am changing that. Do not migrate until the mapping table is checked.
|
||||
|
||||
Chen: So no migration decision today?
|
||||
|
||||
Alex: Correct, no migration decision today.
|
||||
|
||||
Dana: But we can decide one thing: the portal rollout note will say FR-7 is excluded from Friday's batch.
|
||||
|
||||
Bea: Agreed. Excluded from Friday's batch.
|
||||
|
||||
Chen: Yes, put that in.
|
||||
|
||||
Alex: Action item: Bea checks the FR-7 mapping table by Thursday morning.
|
||||
|
||||
Bea: Yes, I will check it by Thursday morning.
|
||||
|
||||
Dana: And I will message Omar after Bea is done.
|
||||
|
||||
Alex: Hold on, after Bea is done is not a date.
|
||||
|
||||
Dana: Fair. Then no task for me yet. I need Bea's result first.
|
||||
|
||||
Chen: Technical note: the API export uses the v2 mapping table, the nightly CSV job uses the legacy table.
|
||||
|
||||
Bea: Correct.
|
||||
|
||||
Alex: Open question: can FR-7 use the v2 mapping without a customer-side field rename?
|
||||
|
||||
Dana: Also, can Omar confirm their preferred launch window?
|
||||
|
||||
Chen: Omar can answer that, but again he is not in this meeting.
|
||||
|
||||
Alex: Good. Back to the portal rollout. Friday's batch still includes DE-2 and NL-4.
|
||||
|
||||
Bea: Yes, those two are unchanged.
|
||||
|
||||
Dana: I personally think excluding FR-7 makes the rollout look messy.
|
||||
|
||||
Alex: Noted as Dana's view, not a decision.
|
||||
|
||||
Chen: And please don't write that Platform approved anything. They are absent.
|
||||
Reference in New Issue
Block a user