Improve semantic classification precision for BUG-015
This commit is contained in:
@@ -375,6 +375,20 @@ extraction results into `meeting_protocol.md` for technical validation. The
|
|||||||
planned architecture separates Canonical Meeting Knowledge from the final Output
|
planned architecture separates Canonical Meeting Knowledge from the final Output
|
||||||
Views documented in `output-views.md`.
|
Views documented in `output-views.md`.
|
||||||
|
|
||||||
|
## Extraction classification contract
|
||||||
|
|
||||||
|
Decision, Action Item and Open Question extraction shares one evidence-oriented
|
||||||
|
classification contract. It is placed after the transcript so it remains the
|
||||||
|
final classification instruction in the single multi-category extraction call.
|
||||||
|
Decisions require a settled outcome; Action Items require established work;
|
||||||
|
Open Questions require a concrete unresolved need. Unsupported candidates must
|
||||||
|
not be moved into another category.
|
||||||
|
|
||||||
|
Action existence and responsibility attribution are separate checks. A valid
|
||||||
|
Action Item may have no known owner, while a named owner requires explicit
|
||||||
|
assignment, volunteering or acceptance. These are semantic LLM classifications;
|
||||||
|
deterministic validation must not guess intent from keywords.
|
||||||
|
|
||||||
## Semantic Consolidator failure handling
|
## Semantic Consolidator failure handling
|
||||||
|
|
||||||
Semantic Consolidator V0 preserves every raw model response before parsing.
|
Semantic Consolidator V0 preserves every raw model response before parsing.
|
||||||
@@ -396,6 +410,28 @@ an instruction not to emit an identical group more than once. Both attempts
|
|||||||
and the detected repetition metadata are preserved. If the retry also fails,
|
and the detected repetition metadata are preserved. If the retry also fails,
|
||||||
the stage fails normally; it does not make another LLM call.
|
the stage fails normally; it does not make another LLM call.
|
||||||
|
|
||||||
|
## Working Protocol V2 renderer contract
|
||||||
|
|
||||||
|
The renderer deterministically projects consolidated items to the semantic
|
||||||
|
fields required for presentation and omits bulky provenance fields from the
|
||||||
|
LLM request. Every renderable item remains represented; structurally empty
|
||||||
|
items are recorded separately rather than turned into invented prose. The
|
||||||
|
compact renderer input is preserved as an artifact.
|
||||||
|
|
||||||
|
The exact Markdown structure is generated from the same heading constants used
|
||||||
|
by the validator and appended after the renderer input so it remains visible
|
||||||
|
within the evaluated context. Decisions, action items and open questions carry
|
||||||
|
input-derived hidden coverage markers. Strict validation requires every such
|
||||||
|
renderable priority item exactly once in its matching section and rejects
|
||||||
|
missing, duplicate, wrong-section or invented markers. Facts and technical
|
||||||
|
details remain condensable as background.
|
||||||
|
|
||||||
|
Renderer output budgeting is adaptive to required priority content and prompt
|
||||||
|
size, while an explicit `num_predict` override remains authoritative. Raw model
|
||||||
|
output is always preserved, and `working_protocol.md` is written only after
|
||||||
|
strict structure and coverage validation passes. The renderer does not retry
|
||||||
|
automatically.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Next Milestone
|
# Next Milestone
|
||||||
|
|||||||
+120
-1
@@ -724,7 +724,7 @@ Evidence:
|
|||||||
|
|
||||||
## EXP-0015 - Difficult synthetic meeting
|
## EXP-0015 - Difficult synthetic meeting
|
||||||
|
|
||||||
Status: Running
|
Status: Accepted
|
||||||
|
|
||||||
Date or period: 2026-07-30
|
Date or period: 2026-07-30
|
||||||
|
|
||||||
@@ -1231,3 +1231,122 @@ Evidence:
|
|||||||
- `docs/output-views.md`
|
- `docs/output-views.md`
|
||||||
- `prompts/working_protocol.md`
|
- `prompts/working_protocol.md`
|
||||||
- `tests/gold/responsibility_attribution_negative/`
|
- `tests/gold/responsibility_attribution_negative/`
|
||||||
|
|
||||||
|
## EXP-0024 - Working Protocol V2 contract visibility
|
||||||
|
|
||||||
|
Status: Running
|
||||||
|
|
||||||
|
Date or period: 2026-08-09
|
||||||
|
|
||||||
|
Target:
|
||||||
|
|
||||||
|
BUG-011 renderer-only regression using an already validated Semantic
|
||||||
|
Consolidator artifact.
|
||||||
|
|
||||||
|
Hypothesis:
|
||||||
|
|
||||||
|
The renderer failures are caused by provenance-heavy 199-350 KB JSON inputs
|
||||||
|
placing the leading prompt contract outside the model's effective evaluated
|
||||||
|
context. The preserved failures all report `prompt_eval_count=16386`, while
|
||||||
|
their outputs either echo trailing JSON or produce an unconstrained generic
|
||||||
|
category summary instead of the requested Working Protocol.
|
||||||
|
|
||||||
|
Iteration 1 change:
|
||||||
|
|
||||||
|
- Project every consolidated item to rendering-relevant semantic fields while
|
||||||
|
retaining every item and its category/text/responsibility/deadline/status
|
||||||
|
information.
|
||||||
|
- Generate the exact structural contract from renderer validator constants and
|
||||||
|
append it after the compact INPUT JSON.
|
||||||
|
- Replace the independently handwritten prompt skeleton with a reference to
|
||||||
|
that authoritative appended contract.
|
||||||
|
- Enforce the existing prompt rule that emitted sections must not be empty.
|
||||||
|
|
||||||
|
This is one renderer-contract prompt iteration. It does not change extraction,
|
||||||
|
canonicalization, semantic consolidation or responsibility semantics.
|
||||||
|
|
||||||
|
Validation before LLM run:
|
||||||
|
|
||||||
|
- 15 focused renderer tests pass.
|
||||||
|
- Tests cover contract generation, compact input projection, valid and invalid
|
||||||
|
headings, missing topic sections, empty sections, wrapper cleanup, malformed
|
||||||
|
Markdown and final-file write gating.
|
||||||
|
|
||||||
|
Decision:
|
||||||
|
|
||||||
|
Iteration 1 passed structural validation and wrote `working_protocol.md`, but
|
||||||
|
the quality sanity check found that the model omitted most of the ten supplied
|
||||||
|
decisions, two open questions and several action items. The structurally valid
|
||||||
|
result therefore was not accepted as BUG-011 verification.
|
||||||
|
|
||||||
|
Iteration 2 change:
|
||||||
|
|
||||||
|
- Add input-derived hidden coverage markers for every decision, action item and
|
||||||
|
open question.
|
||||||
|
- Require every priority item exactly once in its matching section.
|
||||||
|
- Validate missing, duplicate, unknown and wrong-section markers
|
||||||
|
deterministically.
|
||||||
|
- Keep facts and technical details condensable as background.
|
||||||
|
|
||||||
|
This is the second single prompt iteration. It responds to the concrete
|
||||||
|
omission failure observed in Iteration 1 without changing upstream semantics or
|
||||||
|
inventing renderer content.
|
||||||
|
|
||||||
|
Iteration 2 pre-run validation:
|
||||||
|
|
||||||
|
- 18 focused renderer tests pass, including exact required-item coverage and
|
||||||
|
wrong-section rejection.
|
||||||
|
|
||||||
|
Decision:
|
||||||
|
|
||||||
|
Iteration 2 initially exhausted the fixed 4,096-token renderer output budget
|
||||||
|
after emitting all decisions and most action items. Adaptive renderer budgeting
|
||||||
|
resolved to 8,192 tokens for this input. The final run stopped normally after
|
||||||
|
3,709 evaluated output tokens.
|
||||||
|
|
||||||
|
The final renderer-only regression passed strict validation and wrote
|
||||||
|
`working_protocol.md`. Exact coverage was 10/10 decisions, 36/36 renderable
|
||||||
|
action items and 23/23 open questions, each once in its matching section. One
|
||||||
|
structurally empty action item whose task, responsible, deadline and evidence
|
||||||
|
were all null was recorded and excluded rather than fabricated. Optional
|
||||||
|
background markers were accepted only when they referred to real projected
|
||||||
|
input items.
|
||||||
|
|
||||||
|
Accept the compact renderer input, validator-derived trailing contract,
|
||||||
|
priority-item coverage markers, empty-section validation and adaptive renderer
|
||||||
|
output sizing as the BUG-011 baseline. This establishes structural reliability
|
||||||
|
and priority-item coverage, not complete protocol prose quality.
|
||||||
|
|
||||||
|
Evidence:
|
||||||
|
|
||||||
|
- `prompts/working_protocol.md`
|
||||||
|
- `src/meeting_lab/protocol/render_working_protocol.py`
|
||||||
|
- `tests/test_render_working_protocol.py`
|
||||||
|
- `samples/benchmarks/progeo_qwen35_9b_20260805_133942/working_protocol/`
|
||||||
|
- `samples/benchmarks/progeo_qwen35_35b_a3b_20260805_133942/working_protocol/`
|
||||||
|
|
||||||
|
## EXP-0025 — BUG-015 classification precision
|
||||||
|
|
||||||
|
Date: 2026-08-09
|
||||||
|
|
||||||
|
Target: Progeo-derived Decision, Action Item and Open Question precision cases.
|
||||||
|
|
||||||
|
Model/configuration: `qwen3.5:9B`, temperature 0, `num_ctx=32768`.
|
||||||
|
|
||||||
|
Tests were created before prompt changes. A Decision-only evidence threshold
|
||||||
|
kept the explicit Dr. Schlummer rejection and omitted an option and preference.
|
||||||
|
Adding Action and Open Question definitions improved several negatives but was
|
||||||
|
not stable: the model alternately promoted an unaccepted Textor suggestion or
|
||||||
|
moved rejected candidates into Open Questions. Moving the standalone category
|
||||||
|
prompts after the transcript made the partial Decision schema dominate and
|
||||||
|
misclassified true Action Items as Decisions in two consecutive runs.
|
||||||
|
|
||||||
|
The final iteration replaced the competing standalone category prompts with a
|
||||||
|
single unified classification contract after the transcript. It preserved the
|
||||||
|
assigned Nina task and ownerless established CET work, and prevented
|
||||||
|
cross-category leakage in the focused case, but still emitted the unaccepted
|
||||||
|
Textor suggestion as an Action Item. Further prompt iterations were stopped in
|
||||||
|
accordance with the Gold Standard methodology.
|
||||||
|
|
||||||
|
Result: partially improved, not accepted as a complete BUG-015 fix. BUG-015
|
||||||
|
remains Open; no phrase-specific deterministic filter was introduced.
|
||||||
|
|||||||
@@ -840,6 +840,51 @@ Verified. The renderer contract is now enforced deterministically. This does
|
|||||||
not improve semantic quality of the generated prose; it prevents invalid
|
not improve semantic quality of the generated prose; it prevents invalid
|
||||||
renderer output from being accepted as a final Working Protocol.
|
renderer output from being accepted as a final Working Protocol.
|
||||||
|
|
||||||
|
Production-blocker follow-up, 2026-08-09:
|
||||||
|
|
||||||
|
Later preserved renderer failures showed that enforcement alone did not make a
|
||||||
|
valid protocol reliably obtainable. Provenance-heavy consolidated JSON inputs
|
||||||
|
were 199-350 KB, while failed Ollama runs consistently evaluated 16,386 prompt
|
||||||
|
tokens. The leading handwritten renderer contract was therefore effectively
|
||||||
|
lost or underweighted: models echoed trailing JSON or produced generic category
|
||||||
|
summaries. Prompt and validator also duplicated the structure independently,
|
||||||
|
and the validator did not enforce the prompt's no-empty-section rule.
|
||||||
|
|
||||||
|
The renderer now:
|
||||||
|
|
||||||
|
- preserves every renderable item in a compact semantic projection while
|
||||||
|
removing source-reference and original-value bulk
|
||||||
|
- records and excludes structurally empty items instead of inventing content
|
||||||
|
- appends an authoritative contract generated from validator heading constants
|
||||||
|
- rejects empty emitted sections
|
||||||
|
- requires hidden, input-derived coverage markers for every decision, action
|
||||||
|
item and open question exactly once in the matching section
|
||||||
|
- accepts optional background markers only for real projected input items
|
||||||
|
- uses adaptive output sizing instead of the truncating fixed 4,096-token cap
|
||||||
|
- keeps explicit output-budget overrides authoritative and makes no automatic
|
||||||
|
renderer retry
|
||||||
|
|
||||||
|
Renderer-only verification used the validated consolidated input at
|
||||||
|
`/tmp/meeting-lab-bug014-regression/consolidated_extractions.json`. The final
|
||||||
|
run used `qwen3.5:9B`, temperature 0, `num_ctx=32768`, adaptive
|
||||||
|
`num_predict=8192`, and stopped normally with `eval_count=3709` and
|
||||||
|
`done_reason=stop` after 53.069 seconds. Strict validation reported no
|
||||||
|
violations and wrote:
|
||||||
|
|
||||||
|
- `/tmp/meeting-lab-bug011-renderer-final/working_protocol.md`
|
||||||
|
|
||||||
|
Coverage was complete for all renderable priority items: 10 decisions, 36
|
||||||
|
action items and 23 open questions, with no missing or duplicate markers. One
|
||||||
|
upstream action item containing only null task/responsibility/deadline/evidence
|
||||||
|
was recorded as structurally empty and not rendered. The protocol retained
|
||||||
|
substantial technical background and did not add a new named responsibility;
|
||||||
|
the only structured responsible person in the renderer input remained Marleen.
|
||||||
|
|
||||||
|
Status remains Verified for Working Protocol V2 structural validity and
|
||||||
|
priority-item coverage. This does not claim full semantic or editorial protocol
|
||||||
|
quality, topic quality, or resolution of the other renderer-related regression
|
||||||
|
bugs.
|
||||||
|
|
||||||
## BUG-012
|
## BUG-012
|
||||||
|
|
||||||
ID: BUG-012
|
ID: BUG-012
|
||||||
@@ -1124,3 +1169,36 @@ source-ID occurrences and 113 unique canonical source IDs. There were no
|
|||||||
missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of
|
missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of
|
||||||
BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups,
|
BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups,
|
||||||
whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON.
|
whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON.
|
||||||
|
|
||||||
|
## BUG-015
|
||||||
|
|
||||||
|
ID: BUG-015
|
||||||
|
|
||||||
|
Title: Extraction classification precision for Decisions, Action Items, and Open Questions
|
||||||
|
|
||||||
|
Status: Open
|
||||||
|
|
||||||
|
First observed: Progeo production benchmark and BUG-011 renderer-only output
|
||||||
|
|
||||||
|
The Progeo extraction classified proposals/options as Decisions, suggestions
|
||||||
|
and hypothetical work as Action Items, and uncertainty or discussion fragments
|
||||||
|
as Open Questions. The production prompt combined a weak shared German rule,
|
||||||
|
a standalone Decision prompt, a responsibility-only Action prompt, and no
|
||||||
|
Open-Question definition in one multi-category extraction request.
|
||||||
|
|
||||||
|
Regression fixtures now record explicit positive and negative evidence for all
|
||||||
|
three categories. Extraction uses one unified classification contract after the
|
||||||
|
transcript, with precision-first thresholds, category boundaries, evidence
|
||||||
|
requirements, and the responsibility-attribution invariant. In a seven-chunk
|
||||||
|
Progeo measurement, bare uncertainty in chunk 08 stopped becoming an Open
|
||||||
|
Question and the preference about real plant versus Technikum in chunk 16
|
||||||
|
stopped becoming a Decision. Supported examples including the explicit Dr.
|
||||||
|
Schlummer rejection and established CET work remained detectable.
|
||||||
|
|
||||||
|
BUG-015 is not Verified. With `qwen3.5:9B`, temperature 0, the focused Action
|
||||||
|
fixture still classified the unaccepted suggestion to contact Dirk Textor as
|
||||||
|
an Action Item. Earlier prompt variants also showed category leakage by turning
|
||||||
|
rejected candidates into Open Questions or Decisions. Prompt iterations were
|
||||||
|
stopped under the Gold Standard methodology rather than adding phrase-specific
|
||||||
|
filters. A general semantic classification architecture improvement remains
|
||||||
|
necessary before this bug can be marked Verified.
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
Classification contract for Decisions, Action Items, and Open Questions:
|
||||||
|
|
||||||
|
Classify each supported proposition into the one best category. Do not move an
|
||||||
|
unsupported candidate into another category merely to retain it. Prefer
|
||||||
|
omission over a false Decision or Action Item.
|
||||||
|
|
||||||
|
Decision:
|
||||||
|
|
||||||
|
A Decision requires evidence that the meeting settled, selected, approved,
|
||||||
|
rejected, or committed to an outcome. Explicit agreement, selection,
|
||||||
|
approval/rejection, commitment, or a clear statement that a decision was made
|
||||||
|
is sufficient. A proposal, suggestion, preference, possibility, option,
|
||||||
|
recommendation, consideration, or unresolved plan is not sufficient without
|
||||||
|
explicit acceptance or commitment. A personal commitment to perform work is an
|
||||||
|
Action Item, not a Decision, unless the meeting also establishes a distinct
|
||||||
|
group-level outcome.
|
||||||
|
|
||||||
|
Action Item:
|
||||||
|
|
||||||
|
An Action Item is a concrete future action that the meeting establishes as work
|
||||||
|
to be done. Explicit assignment, explicit acceptance, a personal commitment,
|
||||||
|
or clear evidence that the work is already established and underway is
|
||||||
|
sufficient. A suggestion, hypothetical action, possible next step,
|
||||||
|
recommendation, brainstorming option, or exploratory statement is not
|
||||||
|
sufficient. Do not create an Action Item merely because a statement contains an
|
||||||
|
imperative-like verb.
|
||||||
|
|
||||||
|
First decide whether the action itself exists. Only then attribute
|
||||||
|
responsibility. An Action Item may exist without a named responsible person. A
|
||||||
|
person may be named only when explicitly assigned, volunteering, or explicitly
|
||||||
|
accepting/confirming the task. Mention, proximity, expertise, role, a request,
|
||||||
|
or language that someone should/could/might act does not establish
|
||||||
|
responsibility. Do not preserve an unsupported action merely by setting
|
||||||
|
responsible to null.
|
||||||
|
|
||||||
|
Open Question:
|
||||||
|
|
||||||
|
An Open Question requires a concrete unresolved information, decision, or
|
||||||
|
clarification need that remains open after the available context. An explicit
|
||||||
|
question left unanswered or an explicit statement that clarification remains
|
||||||
|
needed is sufficient. Uncertainty, doubt, ignorance, difficulty, an
|
||||||
|
observation, an incomplete fragment, or a rhetorical question alone is not
|
||||||
|
sufficient. A question answered in the available context is not open. Do not
|
||||||
|
invent question wording or convert a rejected Decision/Action candidate into an
|
||||||
|
Open Question without independent evidence of an unresolved need.
|
||||||
|
|
||||||
|
Evidence discipline:
|
||||||
|
|
||||||
|
- Every emitted item must include the shortest transcript evidence that proves
|
||||||
|
the category threshold.
|
||||||
|
- Evidence for an Action Item must prove both the concrete work and that it was
|
||||||
|
established, not merely possible.
|
||||||
|
- Evidence for an Open Question must prove both the specific need and that it
|
||||||
|
remains unresolved.
|
||||||
|
- Use an empty list when a category has no supported item.
|
||||||
+12
-5
@@ -11,11 +11,18 @@ Regeln:
|
|||||||
4. Wenn etwas widersprüchlich oder unklar ist, kennzeichne es als unklar.
|
4. Wenn etwas widersprüchlich oder unklar ist, kennzeichne es als unklar.
|
||||||
5. Übernimm wichtige technische Aussagen möglichst nah am Wortlaut.
|
5. Übernimm wichtige technische Aussagen möglichst nah am Wortlaut.
|
||||||
6. Nenne bei Fakten nach Möglichkeit den Sprecher.
|
6. Nenne bei Fakten nach Möglichkeit den Sprecher.
|
||||||
7. Ein Beschluss ist nur dann ein Beschluss, wenn im Text eine Einigung,
|
7. Ein Beschluss ist nur dann ein Beschluss, wenn das Meeting etwas erkennbar
|
||||||
Freigabe oder verbindliche Festlegung erkennbar ist.
|
geeinigt, ausgewählt, freigegeben, abgelehnt oder verbindlich festgelegt hat.
|
||||||
8. Eine Aufgabe ist nur dann eine Aufgabe, wenn eine Handlung und möglichst
|
8. Eine Aufgabe ist nur dann eine Aufgabe, wenn das Meeting eine konkrete
|
||||||
eine verantwortliche Person oder Organisation erkennbar sind.
|
zukünftige Handlung als zu erledigende Arbeit festgelegt hat. Vorschläge,
|
||||||
9. Gib ausschließlich gültiges JSON aus. Kein Markdown, keine Erläuterungen.
|
Möglichkeiten und hypothetische nächste Schritte genügen nicht.
|
||||||
|
9. Prüfe erst, ob die Aufgabe selbst belegt ist. Ordne danach nur bei expliziter
|
||||||
|
Zuweisung oder Annahme eine verantwortliche Person zu. Eine belegte Aufgabe
|
||||||
|
darf auch keine bekannte verantwortliche Person haben.
|
||||||
|
10. Eine offene Frage erfordert einen konkreten, weiterhin offenen Informations-,
|
||||||
|
Entscheidungs- oder Klärungsbedarf. Bloße Unsicherheit genügt nicht; eine im
|
||||||
|
verfügbaren Kontext beantwortete Frage bleibt nicht offen.
|
||||||
|
11. Gib ausschließlich gültiges JSON aus. Kein Markdown, keine Erläuterungen.
|
||||||
|
|
||||||
Hinweise zur Ausgabe:
|
Hinweise zur Ausgabe:
|
||||||
- Alle obersten Schlüssel müssen vorhanden sein.
|
- Alle obersten Schlüssel müssen vorhanden sein.
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ EXTRACTION_CATEGORIES = (
|
|||||||
|
|
||||||
NORMALIZED_CHUNK_RE = re.compile(r"^(chunk_\d+)_normalized\.txt$")
|
NORMALIZED_CHUNK_RE = re.compile(r"^(chunk_\d+)_normalized\.txt$")
|
||||||
|
|
||||||
EXTRACTION_TASK_PROMPT_NAMES = ("decisions.md", "todos.md")
|
EXTRACTION_TASK_PROMPT_NAMES = ("classification.md",)
|
||||||
|
|
||||||
|
|
||||||
OUTPUT_SCHEMA = {
|
OUTPUT_SCHEMA = {
|
||||||
|
|||||||
@@ -38,7 +38,6 @@ def build_extraction_prompt(
|
|||||||
prompt_parts = [
|
prompt_parts = [
|
||||||
load_prompt("common.md", prompts_dir),
|
load_prompt("common.md", prompts_dir),
|
||||||
meeting_context,
|
meeting_context,
|
||||||
*load_existing_prompts(task_prompt_names, prompts_dir),
|
|
||||||
f"""Quelldatei:
|
f"""Quelldatei:
|
||||||
{source_name}
|
{source_name}
|
||||||
|
|
||||||
@@ -49,5 +48,6 @@ TRANSKRIPT:
|
|||||||
--- BEGINN TRANSKRIPT ---
|
--- BEGINN TRANSKRIPT ---
|
||||||
{transcript}
|
{transcript}
|
||||||
--- ENDE TRANSKRIPT ---""",
|
--- ENDE TRANSKRIPT ---""",
|
||||||
|
*load_existing_prompts(task_prompt_names, prompts_dir),
|
||||||
]
|
]
|
||||||
return "\n\n".join(part for part in prompt_parts if part).strip() + "\n"
|
return "\n\n".join(part for part in prompt_parts if part).strip() + "\n"
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# progeo_action_precision
|
||||||
|
|
||||||
|
Minimal German regression derived from Progeo chunks 2, 10, 17 and 18. It
|
||||||
|
separates suggestions and exploratory possibilities from established work and
|
||||||
|
tests explicit acceptance of responsibility.
|
||||||
|
|
||||||
|
Expected semantics: only the accepted review task and the already-established
|
||||||
|
article work are Action Items.
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"facts": [],
|
||||||
|
"decisions": [],
|
||||||
|
"todos": [
|
||||||
|
{
|
||||||
|
"task": "Messdaten bis Freitag prüfen.",
|
||||||
|
"responsible": "Nina",
|
||||||
|
"deadline": "Freitag",
|
||||||
|
"evidence": "Nina: Ja, ich übernehme die Prüfung bis Freitag."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task": "CET-Artikel erstellen und anschließend im Konsortium verteilen.",
|
||||||
|
"responsible": null,
|
||||||
|
"deadline": null,
|
||||||
|
"evidence": "Den CET-Artikel erstellen wir bereits und schicken ihn anschließend im Konsortium herum."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"questions": [],
|
||||||
|
"positions": [],
|
||||||
|
"technical": []
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
Martin: Die Geometrie kann man vielleicht noch optimieren. Dann würde man mal gucken, was herauskommt.
|
||||||
|
|
||||||
|
Antonius: Marleen müsste vielleicht mal äußern, was Hüsker sich unter dem Versuch vorstellt.
|
||||||
|
|
||||||
|
Tim: Wir könnten Dirk Textor vielleicht noch einmal kontaktieren.
|
||||||
|
|
||||||
|
Antonius: Nina, übernimmst du die Prüfung der Messdaten bis Freitag?
|
||||||
|
|
||||||
|
Nina: Ja, ich übernehme die Prüfung bis Freitag.
|
||||||
|
|
||||||
|
Tim: Den CET-Artikel erstellen wir bereits und schicken ihn anschließend im Konsortium herum.
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# progeo_decision_precision
|
||||||
|
|
||||||
|
Minimal German regression derived from Progeo chunks 12 and 16. It contrasts
|
||||||
|
an option and a personal preference with an explicit group rejection.
|
||||||
|
|
||||||
|
Expected semantics: only the explicit rejection is a Decision.
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"facts": [],
|
||||||
|
"decisions": [
|
||||||
|
{
|
||||||
|
"decision": "Die Gruppe lehnt die Zusammenarbeit mit Dr. Schlummer ab.",
|
||||||
|
"evidence": "Dann haben wir gesagt: Nein, die Zusammenarbeit mit Dr. Schlummer machen wir nicht. Antonius: Ja, das ist entschieden."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"todos": [],
|
||||||
|
"questions": [],
|
||||||
|
"positions": [],
|
||||||
|
"technical": []
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
Tim: Die Option steht im Raum, das Material chemisch recyceln zu lassen.
|
||||||
|
|
||||||
|
Martin: Ich würde nicht in eine reale Anlage gehen. Wenn überhaupt, können wir über ein Technikum reden.
|
||||||
|
|
||||||
|
Antonius: Das Angebot von Dr. Schlummer kostet 30.000 Euro.
|
||||||
|
|
||||||
|
Tim: Dann haben wir gesagt: Nein, die Zusammenarbeit mit Dr. Schlummer machen wir nicht.
|
||||||
|
|
||||||
|
Antonius: Ja, das ist entschieden.
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# progeo_question_precision
|
||||||
|
|
||||||
|
Minimal German regression derived from Progeo chunks 8 and 18. It contrasts
|
||||||
|
uncertainty, an observation and an immediately answered question with an
|
||||||
|
explicitly unresolved information need.
|
||||||
|
|
||||||
|
Expected semantics: only the explicitly unresolved publication question is an
|
||||||
|
Open Question.
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"facts": [],
|
||||||
|
"decisions": [],
|
||||||
|
"todos": [],
|
||||||
|
"questions": [
|
||||||
|
{
|
||||||
|
"question": "Welche Daten aus dem Energieaudit dürfen veröffentlicht werden?",
|
||||||
|
"evidence": "Welche Daten aus dem Energieaudit dürfen wir veröffentlichen? Martin: Das ist weiterhin ungeklärt."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"positions": [],
|
||||||
|
"technical": []
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
Martin: Ob sich das Waschen lohnt, weiß ich nicht.
|
||||||
|
|
||||||
|
Tim: Das lässt sich teilweise optimieren und teilweise nicht.
|
||||||
|
|
||||||
|
Antonius: Gibt es schon ein Programm für die Veranstaltung?
|
||||||
|
|
||||||
|
Tim: Ja, das Programm steht auf der Dechema-Seite und ist vollständig.
|
||||||
|
|
||||||
|
Antonius: Welche Daten aus dem Energieaudit dürfen wir veröffentlichen?
|
||||||
|
|
||||||
|
Martin: Das ist weiterhin ungeklärt. Wir müssen die Freigabe noch klären.
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
import unittest
|
||||||
|
|
||||||
|
from src.meeting_lab.extraction.extract_chunks import build_prompt
|
||||||
|
|
||||||
|
|
||||||
|
class ClassificationInvariantPromptTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.prompt = build_prompt("regression.txt", "Meeting transcript.")
|
||||||
|
|
||||||
|
def test_decision_requires_settled_outcome_and_prefers_precision(self) -> None:
|
||||||
|
self.assertIn("settled, selected, approved", self.prompt)
|
||||||
|
self.assertIn("possibility, option", self.prompt)
|
||||||
|
self.assertIn("Prefer\nomission over a false Decision or Action Item", self.prompt)
|
||||||
|
|
||||||
|
def test_action_requires_established_future_work(self) -> None:
|
||||||
|
self.assertIn("concrete future action that the meeting establishes as", self.prompt)
|
||||||
|
self.assertIn("suggestion, hypothetical action, possible next step", self.prompt.lower())
|
||||||
|
self.assertIn("Do not create an Action Item merely because", self.prompt)
|
||||||
|
self.assertIn("statement contains an", self.prompt)
|
||||||
|
self.assertIn("imperative-like verb", self.prompt)
|
||||||
|
|
||||||
|
def test_action_existence_is_separate_from_responsibility(self) -> None:
|
||||||
|
self.assertIn("First decide whether the action itself exists", self.prompt)
|
||||||
|
self.assertIn("An Action Item may exist without a named responsible", self.prompt)
|
||||||
|
self.assertIn("responsible person", self.prompt)
|
||||||
|
self.assertIn("explicitly assigned, volunteering", self.prompt)
|
||||||
|
self.assertIn("accepting/confirming the task", self.prompt)
|
||||||
|
|
||||||
|
def test_open_question_requires_unresolved_need(self) -> None:
|
||||||
|
self.assertIn("concrete unresolved information, decision, or", self.prompt)
|
||||||
|
self.assertIn("Uncertainty, doubt, ignorance, difficulty", self.prompt)
|
||||||
|
self.assertIn("answered in the available context is not open", self.prompt)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -101,17 +101,20 @@ Final answer:
|
|||||||
prompt = build_prompt("transcript.txt", "Anna: Agreed.")
|
prompt = build_prompt("transcript.txt", "Anna: Agreed.")
|
||||||
|
|
||||||
self.assertIn("Du extrahierst Informationen aus Meeting-Transkripten.", prompt)
|
self.assertIn("Du extrahierst Informationen aus Meeting-Transkripten.", prompt)
|
||||||
self.assertIn("You extract decisions from meeting transcript text.", prompt)
|
self.assertIn("Classification contract for Decisions", prompt)
|
||||||
self.assertIn("Extract each decision as one atomic commitment.", prompt)
|
self.assertIn("A Decision requires evidence", prompt)
|
||||||
self.assertIn("Anna: Agreed.", prompt)
|
self.assertIn("Anna: Agreed.", prompt)
|
||||||
|
|
||||||
def test_build_prompt_includes_todo_prompt_file(self) -> None:
|
def test_build_prompt_includes_todo_prompt_file(self) -> None:
|
||||||
prompt = build_prompt("transcript.txt", "Nina: I will update it.")
|
prompt = build_prompt("transcript.txt", "Nina: I will update it.")
|
||||||
|
|
||||||
self.assertEqual(EXTRACTION_TASK_PROMPT_NAMES, ("decisions.md", "todos.md"))
|
self.assertEqual(
|
||||||
self.assertIn("Action-item responsibility rule:", prompt)
|
EXTRACTION_TASK_PROMPT_NAMES,
|
||||||
|
("classification.md",),
|
||||||
|
)
|
||||||
|
self.assertIn("Action Item:", prompt)
|
||||||
self.assertIn(
|
self.assertIn(
|
||||||
"A named responsible person may be extracted only when the transcript contains",
|
"person may be named only when explicitly assigned",
|
||||||
prompt,
|
prompt,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -131,8 +134,9 @@ Final answer:
|
|||||||
)
|
)
|
||||||
|
|
||||||
for prompt in (no_context_prompt, context_prompt):
|
for prompt in (no_context_prompt, context_prompt):
|
||||||
self.assertIn("You extract decisions from meeting transcript text.", prompt)
|
self.assertIn("Classification contract for Decisions", prompt)
|
||||||
self.assertIn("Action-item responsibility rule:", prompt)
|
self.assertIn("Action Item:", prompt)
|
||||||
|
self.assertIn("Open Question:", prompt)
|
||||||
|
|
||||||
def test_prompt_assembly_is_deterministic(self) -> None:
|
def test_prompt_assembly_is_deterministic(self) -> None:
|
||||||
first = build_prompt("transcript.txt", "Nina: I will update it.")
|
first = build_prompt("transcript.txt", "Nina: I will update it.")
|
||||||
@@ -140,6 +144,20 @@ Final answer:
|
|||||||
|
|
||||||
self.assertEqual(first, second)
|
self.assertEqual(first, second)
|
||||||
|
|
||||||
|
def test_classification_contracts_follow_transcript(self) -> None:
|
||||||
|
prompt = build_prompt("transcript.txt", "SOURCE_SENTINEL")
|
||||||
|
|
||||||
|
transcript_end = prompt.index("--- ENDE TRANSKRIPT ---")
|
||||||
|
self.assertGreater(prompt.index("Classification contract", transcript_end), transcript_end)
|
||||||
|
self.assertGreater(
|
||||||
|
prompt.index("Action Item:", transcript_end),
|
||||||
|
transcript_end,
|
||||||
|
)
|
||||||
|
self.assertGreater(
|
||||||
|
prompt.index("Open Question:", transcript_end),
|
||||||
|
transcript_end,
|
||||||
|
)
|
||||||
|
|
||||||
def test_build_protocol_groups_extraction_items(self) -> None:
|
def test_build_protocol_groups_extraction_items(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
input_dir = Path(directory)
|
input_dir = Path(directory)
|
||||||
|
|||||||
Reference in New Issue
Block a user