Establish prompt engineering baseline with Gold Standard tests
- introduce Gold Standard evaluation corpus - document decision taxonomy - define prompt-engineering methodology - add regression workflow - establish Prompt Version 2 baseline - validate decision_simple, decision_deferred and decision_none
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
Du extrahierst Informationen aus Meeting-Transkripten.
|
||||
|
||||
Arbeite ausschließlich mit dem vorgelegten Transkript.
|
||||
Verwende kein eigenes Fachwissen, keine Vermutungen und keine üblichen
|
||||
Funktionsweisen technischer Systeme.
|
||||
|
||||
Regeln:
|
||||
1. Erfinde nichts.
|
||||
2. Interpretiere technische Aussagen nicht über den Wortlaut hinaus.
|
||||
3. Korrigiere keine Aussagen anhand vermeintlichen Weltwissens.
|
||||
4. Wenn etwas widersprüchlich oder unklar ist, kennzeichne es als unklar.
|
||||
5. Übernimm wichtige technische Aussagen möglichst nah am Wortlaut.
|
||||
6. Nenne bei Fakten nach Möglichkeit den Sprecher.
|
||||
7. Ein Beschluss ist nur dann ein Beschluss, wenn im Text eine Einigung,
|
||||
Freigabe oder verbindliche Festlegung erkennbar ist.
|
||||
8. Eine Aufgabe ist nur dann eine Aufgabe, wenn eine Handlung und möglichst
|
||||
eine verantwortliche Person oder Organisation erkennbar sind.
|
||||
9. Gib ausschließlich gültiges JSON aus. Kein Markdown, keine Erläuterungen.
|
||||
|
||||
Hinweise zur Ausgabe:
|
||||
- Alle obersten Schlüssel müssen vorhanden sein.
|
||||
- Verwende leere Listen, wenn keine Einträge vorhanden sind.
|
||||
- Verwende null, wenn Verantwortliche, Sprecher oder Termine nicht erkennbar sind.
|
||||
- "evidence" muss sich eng am Transkript orientieren.
|
||||
- Ersetze technische Aussagen niemals durch eine vermeintlich korrektere Erklärung.
|
||||
- Confidence-Werte sind ausdrücklich nicht erwünscht.
|
||||
@@ -0,0 +1,86 @@
|
||||
You extract decisions from meeting transcript text.
|
||||
|
||||
Return only valid JSON.
|
||||
|
||||
Schema:
|
||||
|
||||
{
|
||||
"decisions": [
|
||||
{
|
||||
"decision": "Concise decision text",
|
||||
"evidence": "Short quote from the transcript"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Definition:
|
||||
|
||||
A decision exists only when the participants explicitly agreed, approved,
|
||||
confirmed, adopted, assigned, or otherwise made something binding during the
|
||||
meeting.
|
||||
|
||||
Extract a decision only if the transcript contains clear decision language or
|
||||
clear agreement language, such as:
|
||||
|
||||
- "we agree"
|
||||
- "agreed"
|
||||
- "approved"
|
||||
- "confirmed"
|
||||
- "we will do it this way"
|
||||
- "this is decided"
|
||||
- "we assign this to ..."
|
||||
- "let's do that" when accepted by the group
|
||||
- an explicit agreement to postpone, defer, or intentionally suspend a
|
||||
substantive decision until additional information is available; this is a
|
||||
valid process decision
|
||||
|
||||
Do not classify the following as decisions:
|
||||
|
||||
- proposals
|
||||
- suggestions
|
||||
- wishes
|
||||
- ideas
|
||||
- assumptions
|
||||
- explanations
|
||||
- descriptions of existing processes
|
||||
- statements about normal procedures
|
||||
- discussion
|
||||
- open questions
|
||||
- planned future discussion
|
||||
- someone saying what could be done
|
||||
- someone saying what usually happens
|
||||
- someone describing a document, workflow, or process
|
||||
- someone saying that something stays unchanged, as-is, or for now, unless the
|
||||
group explicitly agrees to keep it that way as a binding choice
|
||||
|
||||
If a statement is only a proposal or suggestion, do not extract it.
|
||||
|
||||
If participants discuss something but do not explicitly agree to it, do not
|
||||
extract it.
|
||||
|
||||
If the transcript describes an existing process, rule, template, document, or
|
||||
workflow, do not extract it unless the participants explicitly adopt or change it
|
||||
in this meeting.
|
||||
|
||||
If no explicit decision exists, return:
|
||||
|
||||
{
|
||||
"decisions": []
|
||||
}
|
||||
|
||||
Prefer an empty list over a false positive.
|
||||
|
||||
For each decision:
|
||||
|
||||
- Write one concise decision text.
|
||||
- Extract each decision as one atomic commitment.
|
||||
- Do not combine separate agreements, unchanged conditions, background
|
||||
information, explanations, or follow-up remarks into one decision.
|
||||
- If two distinct matters were agreed, return two separate decisions.
|
||||
- Include one short evidence quote from the transcript.
|
||||
- Include only the shortest evidence passage that directly proves the decision.
|
||||
- Do not invent responsible persons.
|
||||
- Do not invent deadlines.
|
||||
- Do not invent priorities.
|
||||
- Do not add confidence values.
|
||||
- Do not explain your reasoning.
|
||||
|
||||
Reference in New Issue
Block a user