Establish prompt engineering baseline with Gold Standard tests

- introduce Gold Standard evaluation corpus
- document decision taxonomy
- define prompt-engineering methodology
- add regression workflow
- establish Prompt Version 2 baseline
- validate decision_simple, decision_deferred and decision_none
This commit is contained in:
2026-07-30 12:13:10 +02:00
parent 07b0d80113
commit f7ad9ba51f
43 changed files with 1288 additions and 45 deletions
+11
View File
@@ -0,0 +1,11 @@
# facts_vs_positions
Tests separation of objective facts from personal opinions.
The transcript deliberately mixes numeric facts, named non-respondents, and subjective positions about rollout health.
Typical LLM mistakes:
- Treating Marta's opinion as an objective fact.
- Treating Leo's optimism as a fact.
- Extracting a rollout decision even though the group explicitly does not decide.
@@ -0,0 +1,42 @@
{
"facts": [
{
"fact": "The pilot survey closed yesterday with 42 responses.",
"evidence": "Hannah: The pilot survey closed yesterday with 42 responses."
},
{
"fact": "The average pilot survey rating was 3.8 out of 5.",
"evidence": "Leo: The average rating was 3.8 out of 5."
},
{
"fact": "Northwind, Verdan, and Eastport did not respond to the survey.",
"evidence": "Hannah: That part is true. Northwind, Verdan, and Eastport did not respond."
}
],
"decisions": [],
"todos": [],
"questions": [
{
"question": "Why does Marta think the survey result is weaker than it looks?",
"evidence": "Leo: Why?"
}
],
"positions": [
{
"speaker": "Marta",
"position": "Marta thinks the pilot survey result is weaker than it looks.",
"evidence": "Marta: I think that is weaker than it looks."
},
{
"speaker": "Leo",
"position": "Leo feels the pilot is healthy.",
"evidence": "Leo: I still feel the pilot is healthy."
},
{
"speaker": "Marta",
"position": "Marta thinks the rollout should slow down.",
"evidence": "Marta: I disagree. My view is that we should slow down the rollout."
}
],
"technical": []
}
@@ -0,0 +1,19 @@
Hannah: The pilot survey closed yesterday with 42 responses.
Leo: The average rating was 3.8 out of 5.
Marta: I think that is weaker than it looks.
Leo: Why?
Marta: Because three enterprise customers skipped the survey entirely.
Hannah: That part is true. Northwind, Verdan, and Eastport did not respond.
Leo: I still feel the pilot is healthy.
Marta: I disagree. My view is that we should slow down the rollout.
Hannah: Let's capture both views and not decide rollout speed today.
Leo: Okay, that matches my notes.