Establish prompt engineering baseline with Gold Standard tests

- introduce Gold Standard evaluation corpus
- document decision taxonomy
- define prompt-engineering methodology
- add regression workflow
- establish Prompt Version 2 baseline
- validate decision_simple, decision_deferred and decision_none
This commit is contained in:
2026-07-30 12:13:10 +02:00
parent 07b0d80113
commit f7ad9ba51f
43 changed files with 1288 additions and 45 deletions
+11
View File
@@ -0,0 +1,11 @@
# technical_simple
Tests technical extraction with a corrected diagnosis.
The transcript contrasts two possible causes: certificate expiry and runner configuration. The latter is confirmed as the cause.
Typical LLM mistakes:
- Reporting certificate expiry as the problem.
- Creating a fix decision even though the group explicitly says no fix is decided.
- Missing the distinction between fact and technical diagnosis.
+33
View File
@@ -0,0 +1,33 @@
{
"facts": [
{
"fact": "The mobile build failed on the staging runner.",
"evidence": "Sofia: The mobile build failed again on the staging runner."
},
{
"fact": "The certificate is valid until October.",
"evidence": "Nils: The certificate itself is valid until October."
}
],
"decisions": [],
"todos": [],
"questions": [
{
"question": "Is the mobile build failing with the same error as yesterday?",
"evidence": "Nils: Same error as yesterday?"
}
],
"positions": [],
"technical": [
{
"subject": "iOS staging build",
"statement": "The iOS job fails during code signing because the runner uses the old keychain path.",
"evidence": "The iOS job now fails during code signing. The problem is that the runner uses the old keychain path."
},
{
"subject": "Failure classification",
"statement": "The failure is a runner configuration issue, not a certificate expiry issue.",
"evidence": "So it is a runner configuration issue, not a certificate expiry issue. Sofia: Exactly."
}
]
}
@@ -0,0 +1,19 @@
Sofia: The mobile build failed again on the staging runner.
Nils: Same error as yesterday?
Sofia: No, different. The iOS job now fails during code signing.
Nils: The certificate itself is valid until October.
Sofia: Right. The problem is that the runner uses the old keychain path.
Nils: So it is a runner configuration issue, not a certificate expiry issue.
Sofia: Exactly.
Nils: We are not deciding the fix today.
Sofia: Okay.
Nils: Next item.