Improve semantic classification precision for BUG-015

This commit is contained in:
2026-08-09 16:15:35 +02:00
parent 0b24351127
commit fd7d5e1424
19 changed files with 464 additions and 15 deletions
+36
View File
@@ -375,6 +375,20 @@ extraction results into `meeting_protocol.md` for technical validation. The
planned architecture separates Canonical Meeting Knowledge from the final Output planned architecture separates Canonical Meeting Knowledge from the final Output
Views documented in `output-views.md`. Views documented in `output-views.md`.
## Extraction classification contract
Decision, Action Item and Open Question extraction shares one evidence-oriented
classification contract. It is placed after the transcript so it remains the
final classification instruction in the single multi-category extraction call.
Decisions require a settled outcome; Action Items require established work;
Open Questions require a concrete unresolved need. Unsupported candidates must
not be moved into another category.
Action existence and responsibility attribution are separate checks. A valid
Action Item may have no known owner, while a named owner requires explicit
assignment, volunteering or acceptance. These are semantic LLM classifications;
deterministic validation must not guess intent from keywords.
## Semantic Consolidator failure handling ## Semantic Consolidator failure handling
Semantic Consolidator V0 preserves every raw model response before parsing. Semantic Consolidator V0 preserves every raw model response before parsing.
@@ -396,6 +410,28 @@ an instruction not to emit an identical group more than once. Both attempts
and the detected repetition metadata are preserved. If the retry also fails, and the detected repetition metadata are preserved. If the retry also fails,
the stage fails normally; it does not make another LLM call. the stage fails normally; it does not make another LLM call.
## Working Protocol V2 renderer contract
The renderer deterministically projects consolidated items to the semantic
fields required for presentation and omits bulky provenance fields from the
LLM request. Every renderable item remains represented; structurally empty
items are recorded separately rather than turned into invented prose. The
compact renderer input is preserved as an artifact.
The exact Markdown structure is generated from the same heading constants used
by the validator and appended after the renderer input so it remains visible
within the evaluated context. Decisions, action items and open questions carry
input-derived hidden coverage markers. Strict validation requires every such
renderable priority item exactly once in its matching section and rejects
missing, duplicate, wrong-section or invented markers. Facts and technical
details remain condensable as background.
Renderer output budgeting is adaptive to required priority content and prompt
size, while an explicit `num_predict` override remains authoritative. Raw model
output is always preserved, and `working_protocol.md` is written only after
strict structure and coverage validation passes. The renderer does not retry
automatically.
--- ---
# Next Milestone # Next Milestone
+120 -1
View File
@@ -724,7 +724,7 @@ Evidence:
## EXP-0015 - Difficult synthetic meeting ## EXP-0015 - Difficult synthetic meeting
Status: Running Status: Accepted
Date or period: 2026-07-30 Date or period: 2026-07-30
@@ -1231,3 +1231,122 @@ Evidence:
- `docs/output-views.md` - `docs/output-views.md`
- `prompts/working_protocol.md` - `prompts/working_protocol.md`
- `tests/gold/responsibility_attribution_negative/` - `tests/gold/responsibility_attribution_negative/`
## EXP-0024 - Working Protocol V2 contract visibility
Status: Running
Date or period: 2026-08-09
Target:
BUG-011 renderer-only regression using an already validated Semantic
Consolidator artifact.
Hypothesis:
The renderer failures are caused by provenance-heavy 199-350 KB JSON inputs
placing the leading prompt contract outside the model's effective evaluated
context. The preserved failures all report `prompt_eval_count=16386`, while
their outputs either echo trailing JSON or produce an unconstrained generic
category summary instead of the requested Working Protocol.
Iteration 1 change:
- Project every consolidated item to rendering-relevant semantic fields while
retaining every item and its category/text/responsibility/deadline/status
information.
- Generate the exact structural contract from renderer validator constants and
append it after the compact INPUT JSON.
- Replace the independently handwritten prompt skeleton with a reference to
that authoritative appended contract.
- Enforce the existing prompt rule that emitted sections must not be empty.
This is one renderer-contract prompt iteration. It does not change extraction,
canonicalization, semantic consolidation or responsibility semantics.
Validation before LLM run:
- 15 focused renderer tests pass.
- Tests cover contract generation, compact input projection, valid and invalid
headings, missing topic sections, empty sections, wrapper cleanup, malformed
Markdown and final-file write gating.
Decision:
Iteration 1 passed structural validation and wrote `working_protocol.md`, but
the quality sanity check found that the model omitted most of the ten supplied
decisions, two open questions and several action items. The structurally valid
result therefore was not accepted as BUG-011 verification.
Iteration 2 change:
- Add input-derived hidden coverage markers for every decision, action item and
open question.
- Require every priority item exactly once in its matching section.
- Validate missing, duplicate, unknown and wrong-section markers
deterministically.
- Keep facts and technical details condensable as background.
This is the second single prompt iteration. It responds to the concrete
omission failure observed in Iteration 1 without changing upstream semantics or
inventing renderer content.
Iteration 2 pre-run validation:
- 18 focused renderer tests pass, including exact required-item coverage and
wrong-section rejection.
Decision:
Iteration 2 initially exhausted the fixed 4,096-token renderer output budget
after emitting all decisions and most action items. Adaptive renderer budgeting
resolved to 8,192 tokens for this input. The final run stopped normally after
3,709 evaluated output tokens.
The final renderer-only regression passed strict validation and wrote
`working_protocol.md`. Exact coverage was 10/10 decisions, 36/36 renderable
action items and 23/23 open questions, each once in its matching section. One
structurally empty action item whose task, responsible, deadline and evidence
were all null was recorded and excluded rather than fabricated. Optional
background markers were accepted only when they referred to real projected
input items.
Accept the compact renderer input, validator-derived trailing contract,
priority-item coverage markers, empty-section validation and adaptive renderer
output sizing as the BUG-011 baseline. This establishes structural reliability
and priority-item coverage, not complete protocol prose quality.
Evidence:
- `prompts/working_protocol.md`
- `src/meeting_lab/protocol/render_working_protocol.py`
- `tests/test_render_working_protocol.py`
- `samples/benchmarks/progeo_qwen35_9b_20260805_133942/working_protocol/`
- `samples/benchmarks/progeo_qwen35_35b_a3b_20260805_133942/working_protocol/`
## EXP-0025 — BUG-015 classification precision
Date: 2026-08-09
Target: Progeo-derived Decision, Action Item and Open Question precision cases.
Model/configuration: `qwen3.5:9B`, temperature 0, `num_ctx=32768`.
Tests were created before prompt changes. A Decision-only evidence threshold
kept the explicit Dr. Schlummer rejection and omitted an option and preference.
Adding Action and Open Question definitions improved several negatives but was
not stable: the model alternately promoted an unaccepted Textor suggestion or
moved rejected candidates into Open Questions. Moving the standalone category
prompts after the transcript made the partial Decision schema dominate and
misclassified true Action Items as Decisions in two consecutive runs.
The final iteration replaced the competing standalone category prompts with a
single unified classification contract after the transcript. It preserved the
assigned Nina task and ownerless established CET work, and prevented
cross-category leakage in the focused case, but still emitted the unaccepted
Textor suggestion as an Action Item. Further prompt iterations were stopped in
accordance with the Gold Standard methodology.
Result: partially improved, not accepted as a complete BUG-015 fix. BUG-015
remains Open; no phrase-specific deterministic filter was introduced.
+78
View File
@@ -840,6 +840,51 @@ Verified. The renderer contract is now enforced deterministically. This does
not improve semantic quality of the generated prose; it prevents invalid not improve semantic quality of the generated prose; it prevents invalid
renderer output from being accepted as a final Working Protocol. renderer output from being accepted as a final Working Protocol.
Production-blocker follow-up, 2026-08-09:
Later preserved renderer failures showed that enforcement alone did not make a
valid protocol reliably obtainable. Provenance-heavy consolidated JSON inputs
were 199-350 KB, while failed Ollama runs consistently evaluated 16,386 prompt
tokens. The leading handwritten renderer contract was therefore effectively
lost or underweighted: models echoed trailing JSON or produced generic category
summaries. Prompt and validator also duplicated the structure independently,
and the validator did not enforce the prompt's no-empty-section rule.
The renderer now:
- preserves every renderable item in a compact semantic projection while
removing source-reference and original-value bulk
- records and excludes structurally empty items instead of inventing content
- appends an authoritative contract generated from validator heading constants
- rejects empty emitted sections
- requires hidden, input-derived coverage markers for every decision, action
item and open question exactly once in the matching section
- accepts optional background markers only for real projected input items
- uses adaptive output sizing instead of the truncating fixed 4,096-token cap
- keeps explicit output-budget overrides authoritative and makes no automatic
renderer retry
Renderer-only verification used the validated consolidated input at
`/tmp/meeting-lab-bug014-regression/consolidated_extractions.json`. The final
run used `qwen3.5:9B`, temperature 0, `num_ctx=32768`, adaptive
`num_predict=8192`, and stopped normally with `eval_count=3709` and
`done_reason=stop` after 53.069 seconds. Strict validation reported no
violations and wrote:
- `/tmp/meeting-lab-bug011-renderer-final/working_protocol.md`
Coverage was complete for all renderable priority items: 10 decisions, 36
action items and 23 open questions, with no missing or duplicate markers. One
upstream action item containing only null task/responsibility/deadline/evidence
was recorded as structurally empty and not rendered. The protocol retained
substantial technical background and did not add a new named responsibility;
the only structured responsible person in the renderer input remained Marleen.
Status remains Verified for Working Protocol V2 structural validity and
priority-item coverage. This does not claim full semantic or editorial protocol
quality, topic quality, or resolution of the other renderer-related regression
bugs.
## BUG-012 ## BUG-012
ID: BUG-012 ID: BUG-012
@@ -1124,3 +1169,36 @@ source-ID occurrences and 113 unique canonical source IDs. There were no
missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of
BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups, BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups,
whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON. whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON.
## BUG-015
ID: BUG-015
Title: Extraction classification precision for Decisions, Action Items, and Open Questions
Status: Open
First observed: Progeo production benchmark and BUG-011 renderer-only output
The Progeo extraction classified proposals/options as Decisions, suggestions
and hypothetical work as Action Items, and uncertainty or discussion fragments
as Open Questions. The production prompt combined a weak shared German rule,
a standalone Decision prompt, a responsibility-only Action prompt, and no
Open-Question definition in one multi-category extraction request.
Regression fixtures now record explicit positive and negative evidence for all
three categories. Extraction uses one unified classification contract after the
transcript, with precision-first thresholds, category boundaries, evidence
requirements, and the responsibility-attribution invariant. In a seven-chunk
Progeo measurement, bare uncertainty in chunk 08 stopped becoming an Open
Question and the preference about real plant versus Technikum in chunk 16
stopped becoming a Decision. Supported examples including the explicit Dr.
Schlummer rejection and established CET work remained detectable.
BUG-015 is not Verified. With `qwen3.5:9B`, temperature 0, the focused Action
fixture still classified the unaccepted suggestion to contact Dirk Textor as
an Action Item. Earlier prompt variants also showed category leakage by turning
rejected candidates into Open Questions or Decisions. Prompt iterations were
stopped under the Gold Standard methodology rather than adding phrase-specific
filters. A general semantic classification architecture improvement remains
necessary before this bug can be marked Verified.
+55
View File
@@ -0,0 +1,55 @@
Classification contract for Decisions, Action Items, and Open Questions:
Classify each supported proposition into the one best category. Do not move an
unsupported candidate into another category merely to retain it. Prefer
omission over a false Decision or Action Item.
Decision:
A Decision requires evidence that the meeting settled, selected, approved,
rejected, or committed to an outcome. Explicit agreement, selection,
approval/rejection, commitment, or a clear statement that a decision was made
is sufficient. A proposal, suggestion, preference, possibility, option,
recommendation, consideration, or unresolved plan is not sufficient without
explicit acceptance or commitment. A personal commitment to perform work is an
Action Item, not a Decision, unless the meeting also establishes a distinct
group-level outcome.
Action Item:
An Action Item is a concrete future action that the meeting establishes as work
to be done. Explicit assignment, explicit acceptance, a personal commitment,
or clear evidence that the work is already established and underway is
sufficient. A suggestion, hypothetical action, possible next step,
recommendation, brainstorming option, or exploratory statement is not
sufficient. Do not create an Action Item merely because a statement contains an
imperative-like verb.
First decide whether the action itself exists. Only then attribute
responsibility. An Action Item may exist without a named responsible person. A
person may be named only when explicitly assigned, volunteering, or explicitly
accepting/confirming the task. Mention, proximity, expertise, role, a request,
or language that someone should/could/might act does not establish
responsibility. Do not preserve an unsupported action merely by setting
responsible to null.
Open Question:
An Open Question requires a concrete unresolved information, decision, or
clarification need that remains open after the available context. An explicit
question left unanswered or an explicit statement that clarification remains
needed is sufficient. Uncertainty, doubt, ignorance, difficulty, an
observation, an incomplete fragment, or a rhetorical question alone is not
sufficient. A question answered in the available context is not open. Do not
invent question wording or convert a rejected Decision/Action candidate into an
Open Question without independent evidence of an unresolved need.
Evidence discipline:
- Every emitted item must include the shortest transcript evidence that proves
the category threshold.
- Evidence for an Action Item must prove both the concrete work and that it was
established, not merely possible.
- Evidence for an Open Question must prove both the specific need and that it
remains unresolved.
- Use an empty list when a category has no supported item.
+12 -5
View File
@@ -11,11 +11,18 @@ Regeln:
4. Wenn etwas widersprüchlich oder unklar ist, kennzeichne es als unklar. 4. Wenn etwas widersprüchlich oder unklar ist, kennzeichne es als unklar.
5. Übernimm wichtige technische Aussagen möglichst nah am Wortlaut. 5. Übernimm wichtige technische Aussagen möglichst nah am Wortlaut.
6. Nenne bei Fakten nach Möglichkeit den Sprecher. 6. Nenne bei Fakten nach Möglichkeit den Sprecher.
7. Ein Beschluss ist nur dann ein Beschluss, wenn im Text eine Einigung, 7. Ein Beschluss ist nur dann ein Beschluss, wenn das Meeting etwas erkennbar
Freigabe oder verbindliche Festlegung erkennbar ist. geeinigt, ausgewählt, freigegeben, abgelehnt oder verbindlich festgelegt hat.
8. Eine Aufgabe ist nur dann eine Aufgabe, wenn eine Handlung und möglichst 8. Eine Aufgabe ist nur dann eine Aufgabe, wenn das Meeting eine konkrete
eine verantwortliche Person oder Organisation erkennbar sind. zukünftige Handlung als zu erledigende Arbeit festgelegt hat. Vorschläge,
9. Gib ausschließlich gültiges JSON aus. Kein Markdown, keine Erläuterungen. Möglichkeiten und hypothetische nächste Schritte genügen nicht.
9. Prüfe erst, ob die Aufgabe selbst belegt ist. Ordne danach nur bei expliziter
Zuweisung oder Annahme eine verantwortliche Person zu. Eine belegte Aufgabe
darf auch keine bekannte verantwortliche Person haben.
10. Eine offene Frage erfordert einen konkreten, weiterhin offenen Informations-,
Entscheidungs- oder Klärungsbedarf. Bloße Unsicherheit genügt nicht; eine im
verfügbaren Kontext beantwortete Frage bleibt nicht offen.
11. Gib ausschließlich gültiges JSON aus. Kein Markdown, keine Erläuterungen.
Hinweise zur Ausgabe: Hinweise zur Ausgabe:
- Alle obersten Schlüssel müssen vorhanden sein. - Alle obersten Schlüssel müssen vorhanden sein.
View File
+1 -1
View File
@@ -41,7 +41,7 @@ EXTRACTION_CATEGORIES = (
NORMALIZED_CHUNK_RE = re.compile(r"^(chunk_\d+)_normalized\.txt$") NORMALIZED_CHUNK_RE = re.compile(r"^(chunk_\d+)_normalized\.txt$")
EXTRACTION_TASK_PROMPT_NAMES = ("decisions.md", "todos.md") EXTRACTION_TASK_PROMPT_NAMES = ("classification.md",)
OUTPUT_SCHEMA = { OUTPUT_SCHEMA = {
+1 -1
View File
@@ -38,7 +38,6 @@ def build_extraction_prompt(
prompt_parts = [ prompt_parts = [
load_prompt("common.md", prompts_dir), load_prompt("common.md", prompts_dir),
meeting_context, meeting_context,
*load_existing_prompts(task_prompt_names, prompts_dir),
f"""Quelldatei: f"""Quelldatei:
{source_name} {source_name}
@@ -49,5 +48,6 @@ TRANSKRIPT:
--- BEGINN TRANSKRIPT --- --- BEGINN TRANSKRIPT ---
{transcript} {transcript}
--- ENDE TRANSKRIPT ---""", --- ENDE TRANSKRIPT ---""",
*load_existing_prompts(task_prompt_names, prompts_dir),
] ]
return "\n\n".join(part for part in prompt_parts if part).strip() + "\n" return "\n\n".join(part for part in prompt_parts if part).strip() + "\n"
@@ -0,0 +1,8 @@
# progeo_action_precision
Minimal German regression derived from Progeo chunks 2, 10, 17 and 18. It
separates suggestions and exploratory possibilities from established work and
tests explicit acceptance of responsibility.
Expected semantics: only the accepted review task and the already-established
article work are Action Items.
@@ -0,0 +1,21 @@
{
"facts": [],
"decisions": [],
"todos": [
{
"task": "Messdaten bis Freitag prüfen.",
"responsible": "Nina",
"deadline": "Freitag",
"evidence": "Nina: Ja, ich übernehme die Prüfung bis Freitag."
},
{
"task": "CET-Artikel erstellen und anschließend im Konsortium verteilen.",
"responsible": null,
"deadline": null,
"evidence": "Den CET-Artikel erstellen wir bereits und schicken ihn anschließend im Konsortium herum."
}
],
"questions": [],
"positions": [],
"technical": []
}
@@ -0,0 +1,11 @@
Martin: Die Geometrie kann man vielleicht noch optimieren. Dann würde man mal gucken, was herauskommt.
Antonius: Marleen müsste vielleicht mal äußern, was Hüsker sich unter dem Versuch vorstellt.
Tim: Wir könnten Dirk Textor vielleicht noch einmal kontaktieren.
Antonius: Nina, übernimmst du die Prüfung der Messdaten bis Freitag?
Nina: Ja, ich übernehme die Prüfung bis Freitag.
Tim: Den CET-Artikel erstellen wir bereits und schicken ihn anschließend im Konsortium herum.
@@ -0,0 +1,6 @@
# progeo_decision_precision
Minimal German regression derived from Progeo chunks 12 and 16. It contrasts
an option and a personal preference with an explicit group rejection.
Expected semantics: only the explicit rejection is a Decision.
@@ -0,0 +1,13 @@
{
"facts": [],
"decisions": [
{
"decision": "Die Gruppe lehnt die Zusammenarbeit mit Dr. Schlummer ab.",
"evidence": "Dann haben wir gesagt: Nein, die Zusammenarbeit mit Dr. Schlummer machen wir nicht. Antonius: Ja, das ist entschieden."
}
],
"todos": [],
"questions": [],
"positions": [],
"technical": []
}
@@ -0,0 +1,9 @@
Tim: Die Option steht im Raum, das Material chemisch recyceln zu lassen.
Martin: Ich würde nicht in eine reale Anlage gehen. Wenn überhaupt, können wir über ein Technikum reden.
Antonius: Das Angebot von Dr. Schlummer kostet 30.000 Euro.
Tim: Dann haben wir gesagt: Nein, die Zusammenarbeit mit Dr. Schlummer machen wir nicht.
Antonius: Ja, das ist entschieden.
@@ -0,0 +1,8 @@
# progeo_question_precision
Minimal German regression derived from Progeo chunks 8 and 18. It contrasts
uncertainty, an observation and an immediately answered question with an
explicitly unresolved information need.
Expected semantics: only the explicitly unresolved publication question is an
Open Question.
@@ -0,0 +1,13 @@
{
"facts": [],
"decisions": [],
"todos": [],
"questions": [
{
"question": "Welche Daten aus dem Energieaudit dürfen veröffentlicht werden?",
"evidence": "Welche Daten aus dem Energieaudit dürfen wir veröffentlichen? Martin: Das ist weiterhin ungeklärt."
}
],
"positions": [],
"technical": []
}
@@ -0,0 +1,11 @@
Martin: Ob sich das Waschen lohnt, weiß ich nicht.
Tim: Das lässt sich teilweise optimieren und teilweise nicht.
Antonius: Gibt es schon ein Programm für die Veranstaltung?
Tim: Ja, das Programm steht auf der Dechema-Seite und ist vollständig.
Antonius: Welche Daten aus dem Energieaudit dürfen wir veröffentlichen?
Martin: Das ist weiterhin ungeklärt. Wir müssen die Freigabe noch klären.
+36
View File
@@ -0,0 +1,36 @@
import unittest
from src.meeting_lab.extraction.extract_chunks import build_prompt
class ClassificationInvariantPromptTests(unittest.TestCase):
def setUp(self) -> None:
self.prompt = build_prompt("regression.txt", "Meeting transcript.")
def test_decision_requires_settled_outcome_and_prefers_precision(self) -> None:
self.assertIn("settled, selected, approved", self.prompt)
self.assertIn("possibility, option", self.prompt)
self.assertIn("Prefer\nomission over a false Decision or Action Item", self.prompt)
def test_action_requires_established_future_work(self) -> None:
self.assertIn("concrete future action that the meeting establishes as", self.prompt)
self.assertIn("suggestion, hypothetical action, possible next step", self.prompt.lower())
self.assertIn("Do not create an Action Item merely because", self.prompt)
self.assertIn("statement contains an", self.prompt)
self.assertIn("imperative-like verb", self.prompt)
def test_action_existence_is_separate_from_responsibility(self) -> None:
self.assertIn("First decide whether the action itself exists", self.prompt)
self.assertIn("An Action Item may exist without a named responsible", self.prompt)
self.assertIn("responsible person", self.prompt)
self.assertIn("explicitly assigned, volunteering", self.prompt)
self.assertIn("accepting/confirming the task", self.prompt)
def test_open_question_requires_unresolved_need(self) -> None:
self.assertIn("concrete unresolved information, decision, or", self.prompt)
self.assertIn("Uncertainty, doubt, ignorance, difficulty", self.prompt)
self.assertIn("answered in the available context is not open", self.prompt)
if __name__ == "__main__":
unittest.main()
+25 -7
View File
@@ -101,17 +101,20 @@ Final answer:
prompt = build_prompt("transcript.txt", "Anna: Agreed.") prompt = build_prompt("transcript.txt", "Anna: Agreed.")
self.assertIn("Du extrahierst Informationen aus Meeting-Transkripten.", prompt) self.assertIn("Du extrahierst Informationen aus Meeting-Transkripten.", prompt)
self.assertIn("You extract decisions from meeting transcript text.", prompt) self.assertIn("Classification contract for Decisions", prompt)
self.assertIn("Extract each decision as one atomic commitment.", prompt) self.assertIn("A Decision requires evidence", prompt)
self.assertIn("Anna: Agreed.", prompt) self.assertIn("Anna: Agreed.", prompt)
def test_build_prompt_includes_todo_prompt_file(self) -> None: def test_build_prompt_includes_todo_prompt_file(self) -> None:
prompt = build_prompt("transcript.txt", "Nina: I will update it.") prompt = build_prompt("transcript.txt", "Nina: I will update it.")
self.assertEqual(EXTRACTION_TASK_PROMPT_NAMES, ("decisions.md", "todos.md")) self.assertEqual(
self.assertIn("Action-item responsibility rule:", prompt) EXTRACTION_TASK_PROMPT_NAMES,
("classification.md",),
)
self.assertIn("Action Item:", prompt)
self.assertIn( self.assertIn(
"A named responsible person may be extracted only when the transcript contains", "person may be named only when explicitly assigned",
prompt, prompt,
) )
@@ -131,8 +134,9 @@ Final answer:
) )
for prompt in (no_context_prompt, context_prompt): for prompt in (no_context_prompt, context_prompt):
self.assertIn("You extract decisions from meeting transcript text.", prompt) self.assertIn("Classification contract for Decisions", prompt)
self.assertIn("Action-item responsibility rule:", prompt) self.assertIn("Action Item:", prompt)
self.assertIn("Open Question:", prompt)
def test_prompt_assembly_is_deterministic(self) -> None: def test_prompt_assembly_is_deterministic(self) -> None:
first = build_prompt("transcript.txt", "Nina: I will update it.") first = build_prompt("transcript.txt", "Nina: I will update it.")
@@ -140,6 +144,20 @@ Final answer:
self.assertEqual(first, second) self.assertEqual(first, second)
def test_classification_contracts_follow_transcript(self) -> None:
prompt = build_prompt("transcript.txt", "SOURCE_SENTINEL")
transcript_end = prompt.index("--- ENDE TRANSKRIPT ---")
self.assertGreater(prompt.index("Classification contract", transcript_end), transcript_end)
self.assertGreater(
prompt.index("Action Item:", transcript_end),
transcript_end,
)
self.assertGreater(
prompt.index("Open Question:", transcript_end),
transcript_end,
)
def test_build_protocol_groups_extraction_items(self) -> None: def test_build_protocol_groups_extraction_items(self) -> None:
with tempfile.TemporaryDirectory() as directory: with tempfile.TemporaryDirectory() as directory:
input_dir = Path(directory) input_dir = Path(directory)