Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
# technical_simple
|
||||
|
||||
Tests technical extraction with a corrected diagnosis.
|
||||
|
||||
The transcript contrasts two possible causes: certificate expiry and runner configuration. The latter is confirmed as the cause.
|
||||
|
||||
Typical LLM mistakes:
|
||||
|
||||
- Reporting certificate expiry as the problem.
|
||||
- Creating a fix decision even though the group explicitly says no fix is decided.
|
||||
- Missing the distinction between fact and technical diagnosis.
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"facts": [
|
||||
{
|
||||
"fact": "The mobile build failed on the staging runner.",
|
||||
"evidence": "Sofia: The mobile build failed again on the staging runner."
|
||||
},
|
||||
{
|
||||
"fact": "The certificate is valid until October.",
|
||||
"evidence": "Nils: The certificate itself is valid until October."
|
||||
}
|
||||
],
|
||||
"decisions": [],
|
||||
"todos": [],
|
||||
"questions": [
|
||||
{
|
||||
"question": "Is the mobile build failing with the same error as yesterday?",
|
||||
"evidence": "Nils: Same error as yesterday?"
|
||||
}
|
||||
],
|
||||
"positions": [],
|
||||
"technical": [
|
||||
{
|
||||
"subject": "iOS staging build",
|
||||
"statement": "The iOS job fails during code signing because the runner uses the old keychain path.",
|
||||
"evidence": "The iOS job now fails during code signing. The problem is that the runner uses the old keychain path."
|
||||
},
|
||||
{
|
||||
"subject": "Failure classification",
|
||||
"statement": "The failure is a runner configuration issue, not a certificate expiry issue.",
|
||||
"evidence": "So it is a runner configuration issue, not a certificate expiry issue. Sofia: Exactly."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Sofia: The mobile build failed again on the staging runner.
|
||||
|
||||
Nils: Same error as yesterday?
|
||||
|
||||
Sofia: No, different. The iOS job now fails during code signing.
|
||||
|
||||
Nils: The certificate itself is valid until October.
|
||||
|
||||
Sofia: Right. The problem is that the runner uses the old keychain path.
|
||||
|
||||
Nils: So it is a runner configuration issue, not a certificate expiry issue.
|
||||
|
||||
Sofia: Exactly.
|
||||
|
||||
Nils: We are not deciding the fix today.
|
||||
|
||||
Sofia: Okay.
|
||||
|
||||
Nils: Next item.
|
||||
Reference in New Issue
Block a user