65 changed files with 8112 additions and 18 deletions
+37
View File
@@ -1,5 +1,18 @@
# Project Knowledge # Project Knowledge
## Meeting language in direct protocols
Direct protocol prompts derive their explicit output language from persisted
Meeting Context `meeting.language`, including regeneration and diarized fallback.
Missing context/language explicitly defaults to `de`; omitted language is accepted
without modifying context data. Protocol runtime `output_language` is derived
provenance, not a separate setting. No transcript/context translation is performed.
Prompt evolution (2026-09-11): replaced unconditional German output with the
meeting-language instruction in the shared prompt builder. Names remain verbatim.
Validation uses mocked generation for German/English, plain/diarized inputs,
regeneration and legacy contexts; no live model or extraction gold run is involved.
This is a compact operational summary of the current Meeting Lab state. This is a compact operational summary of the current Meeting Lab state.
## Objective ## Objective
@@ -25,7 +38,28 @@ Implemented:
- Prompt loading from `src/meeting_lab/llm/prompts.py`. - Prompt loading from `src/meeting_lab/llm/prompts.py`.
- Meeting Context V1 loading, validation and optional extraction prompt - Meeting Context V1 loading, validation and optional extraction prompt
injection with minimal extraction JSON provenance. injection with minimal extraction JSON provenance.
- FFmpeg-backed WAV, FLAC and M4A preparation into a per-run canonical mono
16 kHz signed PCM16 WAV artifact before transcription or diarization. Audio
preparation always runs. Optional loudness normalization defaults to on and
currently uses the isolated FFmpeg filter
`loudnorm=I=-16:LRA=11:TP=-1.5`. This is a conservative speech-recording
default and may be revisited after empirical comparison without changing the
orchestration API.
- Interim Markdown protocol generation in `src/meeting_lab/protocol/`. - Interim Markdown protocol generation in `src/meeting_lab/protocol/`.
- Direct protocol prompt input protection: diarized transcripts are rendered as
compact adjacent-speaker blocks without per-segment timestamps. Every source
segment remains represented in order. A deterministic heuristic enforces a
configurable safe input budget, falls back to complete plain transcript text
when necessary, and fails before any Ollama request if even that input is too
large. Silent head/tail truncation is prohibited.
- The `qwen3.8:27b` direct-protocol stage explicitly requests `num_ctx=32768`
and `think=false`; the practical prompt target is approximately 29,000 tokens.
A 31,038-token synthetic prompt passed, but larger prompts are not assumed safe
from the model's advertised 262,144-token native context alone.
- `regenerate_mvp_protocol` updates the run's validated Meeting Context and
regenerates protocol artifacts from the existing diarized transcript when
available. It never reruns audio preparation, Whisper or Pyannote, and it
preserves anonymous speaker labels in the source transcript.
- Non-LLM unit tests for chunking, extraction helpers, protocol rendering and - Non-LLM unit tests for chunking, extraction helpers, protocol rendering and
gold-test runner validation. gold-test runner validation.
- Meeting Context V1 scaffold and documentation for manually maintained - Meeting Context V1 scaffold and documentation for manually maintained
@@ -139,6 +173,9 @@ departments only when they are explicitly supplied as metadata. It must not be
used to infer responsibilities. In the current implementation this context can used to infer responsibilities. In the current implementation this context can
be injected into chunk extraction prompts as authoritative metadata, and only be injected into chunk extraction prompts as authoritative metadata, and only
minimal provenance is written to extraction JSON. minimal provenance is written to extraction JSON.
The implemented MVP statuses are exactly `present` and `mentioned_only`.
Legacy entries without a status receive collection-appropriate defaults. Only
present participants may be targets of explicit `SPEAKER_XX` mappings.
A `responsible` or future `owner` / `assignee` value may be recorded only when A `responsible` or future `owner` / `assignee` value may be recorded only when
source evidence explicitly assigns, accepts or confirms responsibility. If the source evidence explicitly assigns, accepts or confirms responsibility. If the
+7
View File
@@ -1,5 +1,12 @@
# Meeting Lab # Meeting Lab
The direct protocol uses saved Meeting Context `meeting.language`: `de` requests
German output and `en` requests English output, for plain and diarized transcripts
and protocol-only regeneration. Meeting Assistant supplies the same selection to
Whisper. Missing context/language retains German output for older runs. Effective
output language is recorded as `output_language` in protocol runtime metadata.
Transcript text, authored context, names and speaker mappings are not translated.
Experimentierumgebung zur Entwicklung eines lokalen Diskussionsanalyzers für Meetingtranskripte. Experimentierumgebung zur Entwicklung eines lokalen Diskussionsanalyzers für Meetingtranskripte.
## Ziel ## Ziel
+62
View File
@@ -0,0 +1,62 @@
# Optional Speaker Diarization
The direct-protocol MVP keeps speaker diarization disabled by default. Enable
anonymous Community-1 speaker labels with `--diarization auto`, `gpu`, or
`cpu`:
```bash
python3 scripts/run_mvp_meeting.py meeting.wav \
--whisper-model /path/to/ggml-model.bin \
--diarization auto
```
Native mode (the default runtime) requires a compatible local PyTorch and
`pyannote.audio==4.0.7`. For isolated ROCm/CUDA environments, select the
container runtime and provide its image and hardware arguments explicitly:
```bash
python3 scripts/run_mvp_meeting.py meeting.wav \
--whisper-model /path/to/ggml-model.bin \
--diarization gpu \
--diarization-runtime container \
--diarization-container-image IMAGE \
--diarization-container-arg=--device=/dev/kfd \
--diarization-container-arg=--device=/dev/dri \
--diarization-container-arg=--group-add \
--diarization-container-arg=video
```
The container receives `HF_TOKEN` by environment-variable name only. It mounts
the source audio and repository read-only and writes diarization artifacts into
the current run directory. Meeting Lab loads mono 16 kHz PCM16 WAV with
Python's `wave` module and sends an in-memory tensor to pyannote, avoiding its
torchcodec file decoder.
Anonymous `SPEAKER_XX` labels are aligned to Whisper segments by maximum
temporal overlap with Community-1 exclusive diarization. The original Whisper
transcript is preserved; the derived transcript under `diarization/` is used as
the direct-protocol generator's source input.
The full diarized JSON and timestamped text remain immutable audit artifacts,
but their per-segment formatting is too verbose for a full-meeting LLM prompt:
timestamps and repeated speaker labels can more than double input size. For
protocol generation, Meeting Lab deterministically groups only adjacent
segments assigned to the same anonymous speaker and omits timestamps. A later
return by the same speaker starts a new block, and unassigned segments remain
under `SPEAKER_UNASSIGNED`. `protocol/transcript_input.txt` preserves the exact
derived representation sent to prompt construction.
Before contacting Ollama, Meeting Lab conservatively estimates prompt tokens
from UTF-8 byte count without adding a model tokenizer dependency. The default safe
budget is 29,000 estimated tokens within the explicitly configured 32,768-token
Ollama context. The estimate is calibrated against the currently validated
German BPD input and is configurable through
`MvpMeetingConfig.protocol_safe_input_token_budget` or
`--protocol-safe-input-token-budget`.
If compact diarized input exceeds the budget, the generator deterministically
uses the complete plain segment transcript and records the fallback. If that
also exceeds the budget, generation fails before model lookup or generation;
it never truncates, chunks, summarizes, retries, or makes multiple protocol
calls implicitly. Full diarization artifacts are never overwritten by this
selection.
+440
View File
@@ -1747,6 +1747,446 @@ additional semantic category is justified by this result.
Artifacts are preserved under Artifacts are preserved under
`artifacts/experiments/request_acceptance_gold_v0/20260820_qwen35_9b_single_run/`. `artifacts/experiments/request_acceptance_gold_v0/20260820_qwen35_9b_single_run/`.
## EXP-0033 — Collective Commitment Gold V0
Status: Experimental; architecturally successful with one contained
recognition false positive
Date: 2026-08-20
This isolated second-stage experiment tested whether an explicit collective
first-person commitment can establish an action without inventing an individual
owner. It used ten synthetic cases containing one minimal V3-style observation
each. Evidence Observation V3 was neither called nor changed, and the accepted
Request/Acceptance mechanism remained unchanged and independent.
The strict semantic schema contains exactly `observation_id`,
`commitment_form` and `normalized_action_text`. `commitment_form` is closed to
`individual_first_person`, `collective_first_person` and `none`. The model
cannot output responsibility, ownership, requested actor, establishment,
Action Item, protocol, confidence, relations, graphs, decisions or unresolved
issues. Deterministic code validates schema and provenance, requires collective
commitment plus non-empty action text, applies bounded deadline consistency and
explicit-negation gates, and only then sets `status: established`,
`commitment_scope: collective` and `responsible_person: null`.
Gold results:
- CC-01 explicit collective commitment: PASS; established, due `nächste
Woche`, no person.
- CC-02 individual commitment: PASS; correctly routed out of the collective
path.
- CC-03 tentative collective possibility: PASS; unestablished.
- CC-04 collective suggestion: PASS; unestablished.
- CC-05 impersonal necessity: PASS; unestablished.
- CC-06 passive future statement: PASS; unestablished.
- CC-07 collective rejection: PARTIAL. The model incorrectly returned
`collective_first_person`, but the deterministic negation gate detected
`nicht` and prevented establishment.
- CC-08 qualified collective commitment: PASS; established with `nur im
Technikum` preserved, null due and no person.
- CC-09 collective commitment without deadline: PASS; established with null
due and no person.
- CC-10 speaker ownership trap: PASS; established collectively while Martin
remained only the speaker and was not assigned ownership.
Configuration: exactly ten successful sequential `qwen3.5:9B` calls, one per
case, temperature 0, `think=false`, `num_ctx=16384`, `num_predict=1024`, no
retries, no voting and no prompt change. There were zero technical failed
calls. Aggregate runner time was 10.504 seconds; summed per-call time was 10.500
seconds, with 4,267 prompt-evaluation tokens and 415 evaluation tokens.
The outcome was nine PASS, one PARTIAL and zero FAIL. There was one recognition
false positive and no recognition false negatives. No qualifier was lost, no
individual owner was invented, and no responsibility or status field leaked
into recognition. Bounded due handling preserved `nächste Woche` verbatim and
returned null when no deadline was present.
Conclusion: the collective-commitment path is architecturally successful for
this narrow Gold set. The deterministic negation gate contained the only model
error, and every successful collective result necessarily retained
`responsible_person: null`. This does not justify a generic commitment system,
production integration, group identity inference or another semantic category.
Artifacts are preserved under
`artifacts/experiments/collective_commitment_gold_v0/20260820_qwen35_9b_single_run/`.
## EXP-0034 — Explicit Rejection Gold V0
Status: Failed architecturally
Date: 2026-08-20
This isolated Stage-2 experiment tested the narrow evidence fact that a
concrete action, option, proposal or future course was explicitly rejected,
abandoned, discontinued or ruled out. It used twelve synthetic cases containing
one self-contained observation or one local target/rejection pair. Evidence
Observation V3 was not called or changed. The accepted Request/Acceptance and
Collective Commitment paths remained unchanged and were not invoked.
The strict semantic schema contains exactly `rejection_observation_id`,
`target_observation_id`, `rejection_form` and
`normalized_rejected_action_text`. `rejection_form` is closed to
`explicit_action_rejection` and `none`. A positive recognition requires a
known local target and non-empty normalized target; `none` requires both target
and normalized text to be null. Decision, outcome, topic-closure,
responsibility, ownership, protocol, confidence and graph fields are forbidden.
Target resolution is limited to the same observation or one earlier supplied
observation. Deterministic code validates schema, IDs, ordering and complete
provenance before emitting the narrow status `explicitly_rejected`.
`explicitly_rejected` means rejected by the cited evidence only. It is not yet
a final meeting decision or final topic outcome, does not close a topic, and
does not supersede an earlier commitment.
Gold results:
- RJ-01 explicit collective rejection with local target: PASS.
- RJ-02 explicit non-pursuit with paired target: PASS.
- RJ-03 self-contained collaboration rejection: FAIL. The model returned
`none`, producing one recognition false negative.
- RJ-04 personal preference: FAIL. The model promoted the preference to an
explicit rejection and derived an unsupported rejection.
- RJ-05 concern: PASS; remained a non-rejection.
- RJ-06 uncertainty: PASS; remained a non-rejection.
- RJ-07 negative recommendation: FAIL. The model promoted advice to an
explicit rejection and derived an unsupported rejection.
- RJ-08 deferral: PASS; remained a non-rejection.
- RJ-09 factual negation: PASS; remained a non-rejection.
- RJ-10 temporary non-action: FAIL. The model treated `erstmal noch nicht` as
abandonment and derived an unsupported rejection.
- RJ-11 explicit rejection with material scope: PASS. Real-plant and
Druckversuch scope were preserved.
- RJ-12 rejection plus positive alternative: PASS. Only the real-plant option
was rejected; the Technikum alternative was not absorbed.
Configuration: exactly twelve successful sequential `qwen3.5:9B` calls, one
per case, temperature 0, `think=false`, `num_ctx=16384`,
`num_predict=1024`, no retries, no voting and no prompt changes. There were zero
technical failed calls. Aggregate runner time was 15.518 seconds; summed
per-call time was 15.493 seconds, with 6,972 prompt-evaluation tokens and 681
evaluation tokens.
The outcome was eight PASS, zero PARTIAL and four FAIL. Recognition produced
three false positives (RJ-04, RJ-07 and RJ-10) and one false negative (RJ-03).
There were four strict target-field expectation mismatches: three were
consequences of false-positive rejection objects populating otherwise locally
correct antecedents, and one was the missing self-contained RJ-03 target. No
derived positive selected the wrong concrete antecedent. Qualifier-loss count
was zero, positive-alternative absorption count was zero, and no responsibility,
decision, outcome or topic-closure field leaked into model output.
Conclusion: the experiment is not architecturally successful. Deterministic
structural gates cannot contain a semantically well-formed false-positive
rejection with valid local target and provenance. The model did distinguish
concern, uncertainty, deferral and factual negation, and it handled scoped and
alternative-bearing positives correctly, but it did not reliably separate
explicit rejection from personal preference, advice or temporary non-action.
The current binary recognition `explicit_action_rejection | none` is
insufficient for reliable generalization.
No production integration, generic rejection system, prompt tuning or
cross-pattern reconciliation is justified.
Artifacts are preserved under
`artifacts/experiments/explicit_rejection_gold_v0/20260820_qwen35_9b_single_run/`.
## EXP-0035 — Negative Act Form V0
Status: Experimental; successful for form classification with normalization
limitations
Date: 2026-08-20
EXP-0034 failed because the binary `explicit_action_rejection | none` question
collapsed materially different negative acts. It missed self-contained
non-pursuit and promoted personal preference, recommendation and temporary
non-action to rejection. This isolated follow-up tested only whether those
evidence-near forms can be distinguished before any normative derivation. It
does not derive rejection, decision, outcome, topic closure, responsibility or
protocol status, and EXP-0034 remained unchanged.
The strict output schema contains exactly `observation_id`,
`negative_act_form` and `normalized_action_text`. The closed form vocabulary is
`explicit_non_pursuit`, `personal_preference`, `recommendation`,
`temporary_non_action` and `none`. Non-`none` forms require non-empty normalized
action text; `none` requires null. Rejection, status, decision, outcome,
responsibility and other normative fields are forbidden recursively. Local
context may resolve a candidate observation's pronoun, but the schema contains
no target relation and the experiment exposes no derivation function.
Gold results:
- NA-01 explicit non-pursuit: PARTIAL. The form was correct; `working with Dr.
Schlummer` omitted the continuation aspect from normalization.
- NA-02 paraphrased explicit non-pursuit: PASS.
- NA-03 personal preference: PARTIAL. The form was correct, but normalization
repeated `Ich würde das nicht machen` instead of resolving the real-plant
trial target.
- NA-04 negative recommendation: PARTIAL. The form was correct; the normalized
English action used the loose rendering `real asset` for `reale Anlage`.
- NA-05 temporary non-action: PASS.
- NA-06 concern only: PASS with `none` and null action text.
- NA-07 uncertainty: PASS with `none` and null action text.
- NA-08 factual negation: PASS with `none` and null action text.
Expected-versus-actual form confusion was entirely diagonal:
| Expected form | Actual form | Count |
| --- | --- | ---: |
| `explicit_non_pursuit` | `explicit_non_pursuit` | 2 |
| `personal_preference` | `personal_preference` | 1 |
| `recommendation` | `recommendation` | 1 |
| `temporary_non_action` | `temporary_non_action` | 1 |
| `none` | `none` | 3 |
Configuration: exactly eight successful sequential `qwen3.5:9B` calls, one
per case, temperature 0, `think=false`, `num_ctx=16384`,
`num_predict=1024`, no retries, no voting and no prompt changes. There were zero
technical failures. Aggregate runner time was 8.688 seconds; summed per-call
time was 8.686 seconds, with 4,183 prompt-evaluation tokens and 310 evaluation
tokens.
The result was five PASS, three PARTIAL and zero FAIL. All eight
`negative_act_form` classifications matched Gold. There was no unsupported
semantic strengthening and no rejection, status, decision, outcome,
responsibility or topic-closure leakage. Normalized action meaning was fully
acceptable in five cases and imperfect in three.
Conclusion: the finer evidence-near form vocabulary successfully distinguished
the four semantic boundaries that defeated the binary rejection experiment in
this small Gold set. The result supports separating negative-act-form
recognition from later normative derivation, but local target normalization is
not yet uniformly reliable. It does not justify modifying EXP-0034, deriving
rejection, production integration or beginning cross-pattern reconciliation.
Artifacts are preserved under
`artifacts/experiments/negative_act_form_v0/20260820_qwen35_9b_single_run/`.
## EXP-0036 — Controlled Rejection Derivation V1
Status: Experimental; architecturally unsuccessful
Date: 2026-08-20
This isolated experiment followed the failed binary rejection baseline
(EXP-0034) and successful Negative Act Form classification (EXP-0035). Its V1
hypothesis was to classify the negative act first, resolve its local target in
a separate semantic call, and only then derive `explicitly_rejected`
deterministically. It did not modify either predecessor or any accepted Stage-2
pattern, and it has no production integration.
The target recognizer emitted exactly `candidate_observation_id`,
`target_observation_id`, and `normalized_target_text`. Only
`explicit_non_pursuit` was deterministically eligible. Personal preference,
recommendation, temporary non-action, and `none` could never derive rejection,
even with a valid target. Provenance, local membership, ordering, non-empty
target text, and strict non-normative output were additional gates.
`explicitly_rejected` means rejected by the cited evidence only, not a final
decision, topic outcome, permanent state, or closure.
The run reused five exact accepted Negative Act Form outputs and made three new
Negative Act calls plus eight target-resolution calls. All calls used
`qwen3.5:9B`, temperature 0, `think=false`, `num_ctx=16384`,
`num_predict=1024`, no retries, voting, or prompt changes.
| Case | Negative Act expected / actual | Target result | Verdict |
| --- | --- | --- | --- |
| CR-01 | `explicit_non_pursuit` / same | Model returned the string `"null"` as an unknown ID; self-contained target was not linked | FAIL |
| CR-02 | `explicit_non_pursuit` / same | `obs_1`, external solution and continuation preserved | PASS |
| CR-03 | `personal_preference` / same | `obs_1`; eligibility gate prevented rejection | PASS |
| CR-04 | `recommendation` / same | `obs_1`; eligibility gate prevented rejection | PASS |
| CR-05 | `temporary_non_action` / same | `obs_1`; eligibility gate prevented rejection | PASS |
| CR-06 | `none` / same | Null target; final non-rejection was correct, but expected local target was unresolved | FAIL |
| CR-07 | `explicit_non_pursuit` / same | `obs_1`; real-plant and pressure-test scope survived, but normalization remained proposition-like | PARTIAL |
| CR-08 | `explicit_non_pursuit` / same | `obs_1`; real-plant scope preserved and Technikum alternative excluded | PASS |
Result: five PASS, one PARTIAL, two FAIL. All eight Negative Act forms were
correct. There were no false-positive rejections: the valid targets in CR-03,
CR-04, and CR-05 could not override their ineligible forms. There was one
false-negative rejection, CR-01, caused by invalid target output. Target
resolution missed two expected links (invalid CR-01 and null CR-06), so the
wrong/unresolved-target count was two. CR-06 exposed a strategy flaw: a
non-eligible Negative Act form should not be required to pass target resolution
when it cannot derive rejection. Qualifier-loss count was zero. CR-08 isolated
the positive alternative successfully. No individual owner, responsibility,
decision, outcome, topic-closure, or LLM-emitted rejection status appeared.
There were 3 new Negative Act calls, 8 target calls, 5 accepted classification
reuses, zero technical call failures, and one structural target-validation
failure. Aggregate runner time was 11.951 seconds.
Conclusion: negative-act-form gating is promising and successfully contains
the semantic false positives that defeated EXP-0034, but the experiment is not
architecturally successful. The current target-resolution strategy failed the
required self-contained positive CR-01 and unnecessarily evaluated the
ineligible CR-06 path; it is not reliable enough for rejection derivation.
Artifacts are preserved under
`artifacts/experiments/controlled_rejection_v1/20260820_qwen35_9b_single_run/`.
## EXP-0037 — Target Resolution V0
Status: Experimental; FAILED for target resolution
Date: 2026-08-20
Controlled Rejection V1 showed that fine-grained Negative Act Form eligibility
contained false-positive rejection, but its target strategy failed a
self-contained positive and unnecessarily resolved a target for an ineligible
`none` form. This isolated experiment tested target resolution only. It
contains no rejection derivation, status, decision, outcome, responsibility,
topic closure, or production integration.
Eligibility was deterministic: only `explicit_non_pursuit` could reach the
resolver. TR-05 personal preference, TR-06 recommendation, TR-07 temporary
non-action, and TR-08 `none` stopped before prompt construction and recorded an
explicit skipped-call artifact. This hard gate worked in all four cases.
The target schema contained exactly `candidate_observation_id`,
`target_observation_id`, and `normalized_target_text`, with local IDs,
same-or-earlier ordering, unique evidence provenance, null consistency, and
recursive normative-field exclusion. TR-01 used the self-contained strategy:
the prompt stated that linkage was deterministically fixed to the candidate and
requested semantic normalization only. TR-02 through TR-04 used paired local
resolution. No original transcript or new Negative Act classification call was
used.
| Case | Form / eligible | Call | Target result | Verdict |
| --- | --- | --- | --- | --- |
| TR-01 | `explicit_non_pursuit` / yes | yes | Returned string `"null"`; required same-observation target unresolved | FAIL |
| TR-02 | `explicit_non_pursuit` / yes | yes | Returned string `"null"`; `obs_1` unresolved | FAIL |
| TR-03 | `explicit_non_pursuit` / yes | yes | Returned string `"null"`; scoped `obs_1` unresolved | FAIL |
| TR-04 | `explicit_non_pursuit` / yes | yes | Returned string `"null"`; real-plant target unresolved | FAIL |
| TR-05 | `personal_preference` / no | no | Deterministically skipped | PASS |
| TR-06 | `recommendation` / no | no | Deterministically skipped | PASS |
| TR-07 | `temporary_non_action` / no | no | Deterministically skipped | PASS |
| TR-08 | `none` / no | no | Deterministically skipped | PASS |
Result: four PASS, zero PARTIAL, four FAIL. Exactly four successful Ollama
calls were made, all for eligible cases; there were zero technical call
failures and four structural validation failures. All four raw responses used
the JSON string `"null"` as target ID rather than a supplied observation ID or
JSON null. Wrong-target count and unresolved-target count were therefore four.
No qualifier-preservation claim can be made because no eligible positive target
passed validation. TR-04 alternative isolation likewise could not be
established. No rejection, status, decision, outcome, responsibility, or other
normative leakage occurred, and no rejection derivation was performed.
Configuration: `qwen3.5:9B`, temperature 0, `think=false`,
`num_ctx=16384`, `num_predict=1024`, no retries, voting, or prompt changes.
Aggregate runner time was 4.034 seconds.
Conclusion: eligibility gating is successful and should be retained; it fully
prevents unnecessary target calls for ineligible Negative Act forms. Target
Resolution V0 itself failed structurally across all eligible cases. Neither the
self-contained nor paired strategy produced a valid target, and merely
instructing deterministic self-linkage in the semantic prompt did not make the
linkage structurally deterministic. The repeated `"null"` string pattern
requires diagnosis before changing the architecture or prompt. No rejection
derivation is justified by this result.
Artifacts are preserved under
`artifacts/experiments/target_resolution_v0/20260820_qwen35_9b_single_run/`.
## EXP-0038 — Target Resolution V1 Diagnostic
Status: Experimental; linkage boundary successful, normalization incomplete
Date: 2026-08-20
Forensics on failed Target Resolution V0 found a definite prompt defect: its
illustrative value `"observation ID or null"` placed both alternatives inside
a JSON string. V0 also sent only `format: "json"`, which enforced JSON syntax
but not field types. This isolated diagnostic changed only the linkage/output
boundary. It contains no rejection derivation or normative semantics.
Ollama 0.32.6 accepted a true JSON Schema object in `format`. TR1-V1 removed
target selection from the model output entirely and deterministically linked
the self-contained candidate to itself. TR2-V1 through TR4-V1 used a closed
allowed-ID list, an enum of those IDs plus JSON null, typed positive and null
examples, recursive strict validation, and one fixed paired prompt. Linkage and
normalization were persisted separately.
| Case | Strategy / ID source | Target | Normalized target | Verdict |
| --- | --- | --- | --- | --- |
| TR1-V1 | self-contained / deterministic | `obs_1` | `Mit Dr. Schlummer arbeiten wir nicht weiter.` retained negation instead of a positive action meaning | FAIL |
| TR2-V1 | paired / LLM | `obs_1` | `externe Lösung weiterverfolgen` | PASS |
| TR3-V1 | paired / LLM | `obs_1` | `reale Anlage zur Diskussion` lost `Druckversuch` purpose and the `nutzen` action | FAIL |
| TR4-V1 | paired / LLM | `obs_1` | `Versuch in der realen Anlage durchführen`; Technikum excluded | PASS |
Result: two PASS, zero PARTIAL, two FAIL. All four responses passed their true
JSON Schemas. Every resulting target was `obs_1`; wrong-target and
unresolved-target counts were zero. The string `"null"` recurrence count was
zero, and there were zero structural validation failures. TR1 preserved the
collaboration, person, and continuation wording but failed positive-action
normalization by retaining negation. TR3 had one material scope loss. TR4
preserved real-plant scope and isolated the Technikum alternative. No
normative leakage occurred.
Configuration: exactly four `qwen3.5:9B` calls, temperature 0,
`think=false`, `num_ctx=16384`, `num_predict=1024`, no retries, voting, or
prompt tuning. Aggregate runner time was 4.557 seconds.
Conclusion: the V0 string-null failure was primarily a linkage/output-boundary
failure rather than evidence that observation-ID linkage is semantically
impossible. True typed schemas, closed ID lists, and deterministic self-linkage
eliminated every structural and target-ID failure. The experiment still fails
its complete acceptance criterion because target normalization is not reliably
positive or scope-preserving. These results justify separating linkage from
normalization, but not deriving rejection or integrating a new pipeline.
Artifacts are preserved under
`artifacts/experiments/target_resolution_v1_diagnostic/20260820_qwen35_9b_single_run/`.
## EXP-0039 — Target Normalization V0
Status: Experimental; normalization improved but incomplete
Date: 2026-08-20
Target Resolution V1 established correct linkage for all four narrow cases and
eliminated structural ID failures with deterministic self-linkage, closed ID
lists, and true JSON Schemas. Its remaining failures were normalization-only.
This isolated follow-up therefore accepted candidate and target IDs as fixed
input and tested only reconstruction of the positive German action meaning. It
contains no target selection, Negative Act classification, eligibility logic,
rejection derivation, or production integration.
The strict output schema contained exactly `candidate_observation_id`,
`target_observation_id`, and `normalized_target_text`. Both IDs were constrained
to their supplied values with JSON Schema `const`; normalized text was a
non-empty string and null was disallowed. The one fixed prompt required removal
of negative polarity, preservation of action, continuation, material scope and
source language, and exclusion of separate alternatives.
| Case | Actual normalized target | Verdict |
| --- | --- | --- |
| TN-01 | `Mit Dr. Schlummer zusammenarbeiten` | FAIL: positive polarity and collaboration survived, but continuation was lost |
| TN-02 | `externe Lösung weiterverfolgen` | PASS |
| TN-03 | `reale Anlage für den Druckversuch nutzen` | PASS |
| TN-04 | `Versuch in der realen Anlage durchführen` | PASS; Technikum alternative excluded |
Result: three PASS, zero PARTIAL, one FAIL. All four outputs passed strict
schema validation and copied both fixed IDs exactly, so changed-ID count was
zero. Polarity-error count was zero: even TN-01 removed rejection and negation.
Action/continuation-loss count was one (TN-01); material purpose/location
scope-loss count was zero; alternative-absorption count was zero. There was no
unsupported strengthening or normative leakage.
Configuration: exactly four `qwen3.5:9B` calls, temperature 0,
`think=false`, true JSON Schema, `num_ctx=16384`, `num_predict=1024`, no
retries, voting, or prompt tuning. Aggregate runner time was 5.066 seconds.
Conclusion: isolating normalization solved the polarity and scoped-action
failures seen in Target Resolution V1 for three of four cases, including exact
pressure-test scope and alternative isolation. Continuation semantics remain
unreliable in the self-contained collaboration case, so Target Normalization
V0 does not meet its full acceptance criterion. The result does not justify
rejection derivation or production integration.
Artifacts are preserved under
`artifacts/experiments/target_normalization_v0/20260820_qwen35_9b_single_run/`.
## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype ## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype
Date: 2026-08-11 Date: 2026-08-11
+7 -3
View File
@@ -107,8 +107,10 @@ or aliases are corrected.
`department`: Organizational unit. Optional and nullable. `department`: Organizational unit. Optional and nullable.
`attendance_status`: `present` for participants. This distinguishes attendees `attendance_status`: exactly `present` for participants or `mentioned_only` for
from mentioned people. people who are relevant but did not attend. For backward compatibility, a
missing status defaults to `present` in `participants` and `mentioned_only` in
`mentioned_people`.
`mentioned_people`: People discussed or referenced but not present. They are `mentioned_people`: People discussed or referenced but not present. They are
not participants and must not be treated as speakers. not participants and must not be treated as speakers.
@@ -153,7 +155,7 @@ mentioned_people:
aliases: [] aliases: []
role: null role: null
department: null department: null
attendance_status: "not_present" attendance_status: "mentioned_only"
notes: "Wurde erwaehnt, war aber nicht anwesend." notes: "Wurde erwaehnt, war aber nicht anwesend."
organization: organization:
@@ -282,6 +284,8 @@ The current validator checks that:
- participant ids and mentioned-person ids do not collide - participant ids and mentioned-person ids do not collide
- referenced departments exist in `organization.departments` - referenced departments exist in `organization.departments`
- `attendance_status` values are valid - `attendance_status` values are valid
- speaker mappings reference present participants only; mentioned-only people
cannot be diarized speakers
- participants are marked `present` - participants are marked `present`
- mentioned people are not marked `present` - mentioned people are not marked `present`
+229
View File
@@ -0,0 +1,229 @@
# Protocol Generation Decision
## Executive Summary
Meeting Lab tested ten protocol-generation and runtime variants against the
same 93.5-minute reference meeting, `project_process_meeting`. The current
evidence does not support a fully automatic protocol. The best practical local
baseline remains one direct call to `qwen3.6:35B-A3B`, followed by informed
human review. It is fast and produces readable, broadly useful Markdown, but it
still overstates consensus, compresses unresolved process boundaries, misses
some challenge and follow-up paths, and can infer unsafe ownership. Its current
quality verdict is **C — promising but insufficient**.
Additional prompting, review, Meeting Map, hierarchical, diarized, dense-model
and 70B-scale variants did not produce a reliable step change. Some improved
individual dimensions, but none reached a stable B result or removed the need
for substantive review. This is a current evidence-based product choice, not a
permanent architecture decision.
## Experiments Compared
All protocol rows used the complete cleaned transcript and meeting context
unless stated otherwise. A dash means that the artifact did not record the
number; it is not an estimate. Verdicts marked “assessment” are comparative
assessments of saved outputs because those older artifact directories contain
no formal `quality_review.json`.
| Experiment | Model | Architecture | LLM calls | Prompt tokens | Runtime | Human editing | Verdict | Main strength | Main failure | MVP | Research |
| --- | --- | --- | ---: | ---: | ---: | --- | --- | --- | --- | --- | --- |
| Direct one-shot baseline | Qwen3.6 35B-A3B | Direct full-context protocol | 1 | 18,385 | 72.5 s cold; about 38 s inference | Not recorded | C | Fast, readable, broad topic outline | Consensus and ownership overpromotion; missing boundaries and follow-up | **Yes, with review** | Baseline |
| Conservative one-shot | Qwen3.6 35B-A3B | Direct with stronger safety instructions | 1 | 19,120 | 39.5 s warm | Not recorded | C (assessment) | Better uncertainty and pending-feedback language | Still invents or upgrades named follow-up actions | No | Limited |
| Draft → review | Qwen3.6 35B-A3B | Conservative draft plus review call | 2 | 39,071 total | About 110.4 s summed | Not recorded | C (assessment) | Removes some unsafe named attribution | Does not reliably restore omitted content; empty/weak action sections remain | No | Limited |
| Meeting Map → protocol | Qwen3.6 35B-A3B | Semantic map followed by rendering | 2 | 39,129 total | 134.8 s | Not recorded | C (assessment) | Explicit intermediate structure | Map errors propagate: false consensus and named ownership remain | No | Yes |
| Hierarchical notes → protocol | Qwen3.6 35B-A3B | Five chunk-note calls plus synthesis | 6 | 32,145 total | 251.1 s | Not recorded | C (assessment) | Highest recall in several detailed/open topics | Amplifies unsupported speaker/name interpretations and confirmed actions | No | Yes |
| Segment-level anonymous diarization | Qwen3.6 35B-A3B | One-shot over 1,154 labeled segments | 1 | 39,308 | 125.4 s | 25–35 min | C | Preserves some filtered-idea challenge structure | Token count more than doubled; actions and deadlines became less safe | No | No further protocol tests |
| Turn-merged anonymous diarization | Qwen3.6 35B-A3B | One-shot over 435 merged turns | 1 | 22,490 | 101.2 s | 25–35 min | C | Corrected token inflation; recovered some topic and feedback detail | Still did not beat raw input; unsafe actions/deadlines persisted | No | UI/search only |
| Dense one-shot | Qwen3.5 27B | Direct full-context protocol | 1 | 18,385 | 181.2 s | Not recorded | C (assessment) | Somewhat better recall of process details | Much slower; no material overall quality gain | No | No |
| 70B scale one-shot | Llama 3.3 70B Q3_K_S | Dense, 64% CPU / 36% GPU | 1 | 21,156 | 470.1 s warm | 45–60 min | D | Technically proved a 70B hybrid load can run | Severe coverage loss, invented governance, internal contradiction | No | Negative scale result |
| Ollama vs native llama.cpp | Qwen3.6 35B-A3B | Same Q4_K_M GGUF; ROCm/Vulkan servers | 1 per backend | 18,385 | ROCm 38.4 s; Vulkan 42.3 s | N/A | Runtime only | Native ROCm reached 51.38 generated tok/s | No meaningful end-to-end advantage; more operational complexity | Ollama | Runtime reference |
The draft-review total combines the saved conservative draft call and the
saved review call. Its review metadata itself reports only the one new review
call (19,951 prompt tokens and 70.8 seconds). The direct baseline's 72.5-second
wall time includes a 34.5-second cold load; its measured prompt evaluation plus
generation was 37.8 seconds. These distinctions explain apparent runtime
differences between otherwise similar Qwen3.6 calls.
### Recurring quality patterns
- **Topic coverage and factual accuracy:** Direct Qwen3.6 captures the main
process but misses the second review after enrichment, project reporting and
parts of the filtered-idea challenge path. Hierarchical processing recalls
more detail but introduces too many unsupported interpretations. Llama 3.3
loses most of the meeting and invents a governance role for the
Geschäftsführung.
- **Consensus and unresolved boundaries:** Every broad one-shot family remains
vulnerable to turning discussion or a working direction into agreement. The
unresolved boundary between central coordination and autonomous department
work, and the uncertainty around universal filter criteria, are especially
fragile.
- **Visibility, veto and reconsideration:** No approach consistently preserves
initial filtering, later cross-functional challenge, reconsideration after
enrichment and the return through the project cycle together.
- **Stakeholder feedback:** Pending Jovana and Björn feedback is an important
quality probe. Some variants preserve both; segment-level diarization drops
Björn, while Llama 3.3 drops both.
- **Actions and attribution:** Added structure does not guarantee safety.
Conservative, reviewed, Meeting Map, hierarchical and diarized outputs still
promote proposals or expected work into confirmed actions, infer owners from
roles or conversational context, or invent deadlines. Human review remains
mandatory.
## Model Findings
### Qwen3.6:35B-A3B
Qwen3.6 is the best overall local practical baseline. Its Q4_K_M model is
operationally fast on the RX 9070/CPU hybrid setup, follows the requested
Markdown form and usually provides a useful first draft. It remains verdict C:
larger context and fluent synthesis do not reliably protect evidence strength,
responsibility attribution or unresolved process boundaries.
### Qwen3.5:27b dense
The dense 27B run recalled some process details better than the MoE baseline,
but took 181.2 seconds and generated at 8.45 tokens/s. The gains did not amount
to a material overall quality improvement. This result does not prove that
dense models are generally inferior; it shows that this dense model is not a
better product choice on this hardware and meeting.
### Llama 3.3 70B Q3_K_S
Llama 3.3 70B was technically runnable at 32k context with a 42 GB loaded
footprint and a 64% CPU / 36% GPU split. Its 7m50s warm meeting run produced a
very short, materially worse protocol: one critical invented governance claim,
five new major errors and an estimated 45–60 minutes of editing. Raw parameter
count alone is therefore insufficient. The older model generation and
aggressive Q3 quantization are plausible contributors, but this experiment
does not isolate or prove either cause.
## Runtime Findings
The native comparison reused the exact 23,938,321,664-byte Qwen3.6 Q4_K_M GGUF
that Ollama uses. The tested `llama-server` binary was the llama.cpp runtime
shipped with the installed Ollama distribution, not an independent source
build.
Native ROCm processed the reference request in 38.4 seconds and generated at
51.38 tokens/s. Vulkan took 42.3 seconds and generated at 46.73 tokens/s. The
comparable Ollama baseline generated at 44.68 tokens/s, with about 37.8 seconds
of prompt evaluation plus generation when load time is excluded. Output token
counts differed, so generation throughput alone is not an end-to-end quality or
latency comparison.
Native ROCm gained some generation throughput, but did not provide a meaningful
end-to-end advantage for this workload. Vulkan required more host spill and was
not preferable. Ollama already provides the relevant llama.cpp runtime
components, model lifecycle and API integration; it remains the preferred
routine Meeting Lab runtime.
## Diarization Findings
### Technical feasibility
Pyannote `speaker-diarization-community-1` successfully processed the
93.5-minute meeting on CPU in 1,647 seconds (about 27m27s), an RTF of 0.293. It
detected four anonymous clusters and assigned 1,145 of 1,154 Whisper segments
(99.2%). Peak RSS was about 3.3 GiB. This establishes technical feasibility; it
does not establish speaker identity or diarization accuracy against labeled
ground truth.
### Protocol-quality impact
Annotating every Whisper segment increased the Qwen prompt from 18,385 to
39,308 tokens, confounding speaker structure with fragmentation and token
inflation. Deterministic turn merging reduced 1,154 segments to 435 turns and
the complete prompt to 22,490 tokens. That controlled the main representation
confound, but the resulting protocol still did not materially outperform the
raw transcript and remained verdict C.
Anonymous diarization is therefore not justified as a mandatory MVP
protocol-quality feature. This does **not** mean diarization is generally
useless. It may remain valuable for speaker-aware UI, navigation and search,
participation statistics, traceability, or later carefully validated real-name
mapping.
## Semantic Research Findings
The semantic experiments provide architectural evidence, but should not
dominate the product decision:
- **Evidence Observation V3** is a strong evidence-near candidate stage. With
Qwen3.5 9B it achieved 8 PASS, 1 PARTIAL and 0 FAIL while preserving hedges,
alternatives, requests, commitments and boundaries in natural language.
- **Request/Acceptance** and **Collective Commitment** show that narrow semantic
recognition followed by deterministic provenance, ordering, addressee,
negation and deadline gates can safely derive limited consequences. Model
recognition errors were contained without inventing individual ownership.
- **Explicit Rejection** failed when reduced to a coarse binary recognition
problem: semantically valid false positives passed structural gates.
- **Negative Act Form** worked better by distinguishing non-pursuit, personal
preference, recommendation and temporary non-action before any normative
derivation. All eight form classifications matched Gold, although normalized
action text was imperfect in three cases.
- **Target Resolution V0** failed because prompt examples and a weak JSON
boundary encouraged the string `"null"` instead of typed linkage.
**Target Resolution V1** fixed all linkage/ID failures with deterministic
self-linkage, closed ID lists and true JSON Schema, but normalization remained
incomplete.
- **Target Normalization V0** improved polarity and scope preservation to 3/4
PASS, but still lost continuation meaning in the collaboration case.
These findings support Meeting Lab as a research and validation track. They do
not yet justify placing a multi-stage semantic pipeline on the MVP critical
path.
## Current MVP Decision
The current product path is:
```text
Audio
-> transcription
-> direct qwen3.6:35B-A3B protocol generation through Ollama
-> informed human review
-> final protocol
```
The first MVP should treat the generated protocol as an editable draft, not an
authoritative semantic record. Human review must specifically check consensus,
unresolved boundaries, competing positions, action status, owners, deadlines
and pending stakeholder feedback.
Diarization is optional and deferred. The semantic research pipeline remains
in Meeting Lab, outside the MVP critical path. Ollama remains the default local
runtime.
## Rejected / Deferred Directions
- Do not continue Qwen3.6 prompt variants as the main quality strategy.
- Do not add draft-review, Meeting Map or hierarchical generation to the MVP;
their added calls and complexity did not deliver reliable quality gains.
- Do not continue anonymous-diarization protocol experiments. Revisit
diarization for UI, search, statistics or traceability instead.
- Do not use Qwen3.5 27B or Llama 3.3 70B Q3_K_S as the routine protocol model.
- Do not replace Ollama with a manually managed native llama.cpp service for
this workload.
- Retain semantic experiments, but defer production integration and broad
semantic consolidation.
## Open Questions
1. How does a genuinely newer, materially stronger model perform when a useful
quantization fits the available RAM/VRAM without severe swap?
2. If project policy permits, what quality ceiling does the unchanged reference
prompt achieve with a commercial frontier model?
3. What is the measured reviewer time and correction distribution once the
direct Qwen3.6 draft path is exercised in an end-to-end MVP workflow?
4. Which non-protocol product benefits justify revisiting diarization later?
No further Qwen3.6 prompt variants, anonymous-diarization protocol runs or old
70B Q3 scale tests are recommended.
## Recommended Next Product Step
Build the practical end-to-end MVP around direct Qwen3.6 generation and an
explicit human review handoff. Measure reviewer time and correction categories
in real use. Keep the experiment artifacts and semantic Gold work as validation
evidence, but do not block the first product loop on broader research stages.
@@ -75,7 +75,7 @@ mentioned_people:
aliases: [] aliases: []
role: null role: null
department: null department: null
attendance_status: "not_present" attendance_status: "mentioned_only"
notes: null notes: null
organization: organization:
@@ -64,7 +64,7 @@ mentioned_people:
- "Giovana" - "Giovana"
role: "Leiterin Business Development" role: "Leiterin Business Development"
department_id: "bd" department_id: "bd"
attendance_status: "not_present" attendance_status: "mentioned_only"
notes: null notes: null
organization: organization:
@@ -4,6 +4,8 @@
schema_version: "1" schema_version: "1"
meeting: meeting:
# Stable identifier used for provenance across corrections and later runs.
meeting_id: ""
# Human-readable title for the meeting. # Human-readable title for the meeting.
title: "" title: ""
# Dominant meeting language, for example "de" or "en". # Dominant meeting language, for example "de" or "en".
@@ -28,6 +30,12 @@ participants:
attendance_status: "present" attendance_status: "present"
notes: null notes: null
# Optional authoritative mapping from diarization labels to actual participants.
# Add entries only after a human or trusted external process confirms identity.
# Never infer mappings from conversational context. Unmapped labels stay anonymous.
speaker_mappings: {}
# SPEAKER_00: "participant-id"
mentioned_people: mentioned_people:
# People discussed or referenced but not present in the meeting. # People discussed or referenced but not present in the meeting.
# Mentioned people are not speakers and must not become responsible persons # Mentioned people are not speakers and must not become responsible persons
@@ -37,7 +45,7 @@ mentioned_people:
aliases: [] aliases: []
role: null role: null
department: null department: null
attendance_status: "not_present" attendance_status: "mentioned_only"
notes: null notes: null
organization: organization:
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
"""Repository entry point for the collective-commitment Gold experiment."""
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_collective import main # noqa: E402
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,7 @@
#!/usr/bin/env python3
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path: sys.path.insert(0, str(ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_rejection_v1 import main
if __name__ == "__main__": raise SystemExit(main())
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env python3
"""Run the one-call direct protocol MVP from compact Whisper JSON."""
from __future__ import annotations
import argparse
import json
import re
import shutil
import sys
import time
from datetime import datetime
from pathlib import Path
from typing import Any, Callable
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.llm.ollama import DEFAULT_ENDPOINT # noqa: E402
from src.meeting_lab.protocol.generate_direct_protocol import ( # noqa: E402
DEFAULT_MODEL,
DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
DirectProtocolResult,
generate_direct_protocol,
)
DEFAULT_OUTPUT_ROOT = Path("meeting_data/runs")
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate one direct protocol from compact Whisper JSON.")
parser.add_argument("transcript", type=Path)
parser.add_argument("--context", type=Path)
parser.add_argument("--output-root", type=Path, default=DEFAULT_OUTPUT_ROOT)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--ollama-endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument(
"--safe-input-token-budget",
type=int,
default=DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
)
return parser.parse_args(argv)
def create_unique_run_dir(
output_root: Path,
transcript_stem: str,
now: Callable[[], datetime] = datetime.now,
) -> Path:
safe_stem = re.sub(r"[^A-Za-z0-9_.-]+", "_", transcript_stem).strip("._-") or "meeting"
base = output_root / f"{safe_stem}_{now().strftime('%Y%m%d_%H%M%S')}"
candidate = base
suffix = 1
while candidate.exists():
candidate = output_root / f"{base.name}_{suffix:02d}"
suffix += 1
candidate.mkdir(parents=True)
return candidate
def write_json(path: Path, data: Any) -> None:
path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def persist_result(run_dir: Path, result: DirectProtocolResult) -> Path:
protocol_dir = run_dir / "protocol"
protocol_dir.mkdir()
(protocol_dir / "exact_prompt.txt").write_text(result.exact_prompt, encoding="utf-8")
write_json(protocol_dir / "raw_response.json", result.raw_response)
write_json(protocol_dir / "runtime_metadata.json", result.runtime_metadata)
transcript_input = getattr(result, "transcript_input", None)
if transcript_input is not None:
(protocol_dir / "transcript_input.txt").write_text(
transcript_input, encoding="utf-8"
)
protocol_path = run_dir / "protocol.md"
protocol_path.write_text(result.protocol_text, encoding="utf-8")
return protocol_path
def run(args: argparse.Namespace) -> tuple[int, Path, Path | None]:
run_dir = create_unique_run_dir(args.output_root, args.transcript.stem)
timestamp = datetime.now().astimezone().isoformat(timespec="seconds")
started = time.perf_counter()
protocol_path: Path | None = None
metadata: dict[str, Any] = {
"run_id": run_dir.name,
"timestamp": timestamp,
"transcript_path": str(args.transcript.resolve()),
"context_path": str(args.context.resolve()) if args.context else None,
"model": args.model,
"ollama_endpoint": args.ollama_endpoint,
"status": "running",
"total_runtime_seconds": None,
"final_protocol_path": None,
}
try:
transcript_dir = run_dir / "transcript"
transcript_dir.mkdir()
if not args.transcript.is_file():
raise FileNotFoundError(f"Transcript file does not exist: {args.transcript}")
preserved_transcript = transcript_dir / "transcript.json"
shutil.copy2(args.transcript, preserved_transcript)
preserved_context: Path | None = None
if args.context is not None:
if not args.context.is_file():
raise FileNotFoundError(f"Meeting Context file does not exist: {args.context}")
context_dir = run_dir / "context"
context_dir.mkdir()
preserved_context = context_dir / "meeting_context.yaml"
shutil.copy2(args.context, preserved_context)
write_json(
run_dir / "input_manifest.json",
{
"transcript_source": str(args.transcript.resolve()),
"transcript_copy": str(preserved_transcript.resolve()),
"context_source": str(args.context.resolve()) if args.context else None,
"context_copy": str(preserved_context.resolve()) if preserved_context else None,
},
)
result = generate_direct_protocol(
preserved_transcript,
preserved_context,
model=args.model,
endpoint=args.ollama_endpoint,
safe_input_token_budget=args.safe_input_token_budget,
)
protocol_path = persist_result(run_dir, result)
metadata["status"] = "completed"
metadata["final_protocol_path"] = str(protocol_path.resolve())
except Exception as exc:
metadata["status"] = "failed"
metadata["failure"] = f"{type(exc).__name__}: {exc}"
print(f"Error: {metadata['failure']}", file=sys.stderr)
finally:
metadata["total_runtime_seconds"] = round(time.perf_counter() - started, 3)
write_json(run_dir / "run_metadata.json", metadata)
return (0 if metadata["status"] == "completed" else 2), run_dir, protocol_path
def main(argv: list[str] | None = None) -> int:
code, _run_dir, protocol_path = run(parse_args(argv))
if protocol_path is not None:
print(protocol_path)
return code
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
"""Repository entry point for the explicit-rejection Gold experiment."""
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_rejection import main # noqa: E402
if __name__ == "__main__":
raise SystemExit(main())
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
"""CLI adapter for the reusable Meeting Lab MVP orchestration API."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.llm.ollama import DEFAULT_ENDPOINT # noqa: E402
from src.meeting_lab.models.meeting_context import MeetingContext # noqa: E402
from src.meeting_lab.orchestration.mvp import ( # noqa: E402
DEFAULT_DIARIZATION_MODEL,
DEFAULT_MODEL,
DEFAULT_OUTPUT_ROOT,
DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
MvpMeetingConfig,
create_unique_run_dir,
run_mvp_meeting,
)
from src.meeting_lab.progress import ProgressSink # noqa: E402
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Transcribe one meeting and generate one direct protocol."
)
parser.add_argument("audio_file", type=Path)
parser.add_argument("--whisper-model", type=Path, required=True)
parser.add_argument("--whisper-executable", default="whisper-cli")
parser.add_argument("--ffmpeg-executable", default="ffmpeg")
parser.add_argument(
"--audio-normalization",
action=argparse.BooleanOptionalAction,
default=True,
help=(
"Enable FFmpeg loudness normalization during canonical audio preparation "
"(default: enabled)."
),
)
parser.add_argument("--context", type=Path)
parser.add_argument("--output-root", type=Path, default=DEFAULT_OUTPUT_ROOT)
parser.add_argument("--language", default="de")
parser.add_argument(
"--threads",
default="auto",
help="Thread count or 'auto' for physical CPU cores (default: auto).",
)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--ollama-endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument(
"--protocol-safe-input-token-budget",
type=int,
default=DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
help="Conservative estimated prompt-token limit before any Ollama request.",
)
parser.add_argument(
"--diarization",
choices=("auto", "gpu", "cpu", "off"),
default="off",
help="Optional Community-1 diarization device mode (default: off).",
)
parser.add_argument(
"--diarization-runtime",
choices=("native", "container"),
default="native",
help="Run pyannote in this Python environment or an explicit container.",
)
parser.add_argument(
"--diarization-container-image",
help="Container image required with --diarization-runtime container.",
)
parser.add_argument(
"--diarization-container-arg",
action="append",
default=[],
help="Additional docker argument; repeat and use = for values beginning with --.",
)
return parser.parse_args(argv)
def config_from_args(args: argparse.Namespace) -> MvpMeetingConfig:
return MvpMeetingConfig(
audio_file=args.audio_file,
whisper_model=args.whisper_model,
whisper_executable=args.whisper_executable,
ffmpeg_executable=args.ffmpeg_executable,
audio_normalization=args.audio_normalization,
context_file=args.context,
output_root=args.output_root,
language=args.language,
threads=args.threads,
model=args.model,
ollama_endpoint=args.ollama_endpoint,
protocol_safe_input_token_budget=args.protocol_safe_input_token_budget,
diarization=args.diarization,
diarization_runtime=args.diarization_runtime,
diarization_container_image=args.diarization_container_image,
diarization_container_args=tuple(args.diarization_container_arg),
)
def run(
args: argparse.Namespace,
*,
context_override: MeetingContext | dict[str, Any] | None = None,
progress_sink: ProgressSink | None = None,
) -> tuple[int, Path | None, Path | None]:
"""Compatibility wrapper for existing Python callers of the CLI module."""
result = run_mvp_meeting(
config_from_args(args),
meeting_context=context_override,
progress_sink=progress_sink,
)
return result.exit_code, result.run_dir, result.protocol_path
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
if args.diarization == "off":
print("Diarization: disabled")
else:
print(
f"Diarization: enabled; backend=pyannote.audio; "
f"model={DEFAULT_DIARIZATION_MODEL}; requested_device={args.diarization}; "
f"runtime={args.diarization_runtime}"
)
code, run_dir, protocol_path = run(args)
if run_dir is not None and args.diarization != "off":
metadata_path = run_dir / "diarization" / "metadata.json"
if metadata_path.is_file():
details = json.loads(metadata_path.read_text(encoding="utf-8"))
print(
f"Diarization result: device={details.get('actual_device')}; "
f"device_name={details.get('device_name') or 'n/a'}; "
f"runtime={details.get('runtime_seconds'):.3f}s; "
f"speakers={details.get('speaker_count')}; artifacts={metadata_path.parent}"
)
if protocol_path is not None:
print(protocol_path)
return code
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
"""Repository entry point for the Negative Act Form experiment."""
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_negative_act import main # noqa: E402
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,7 @@
#!/usr/bin/env python3
import sys
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path: sys.path.insert(0,str(ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_target_normalization import main
if __name__=="__main__": raise SystemExit(main())
@@ -0,0 +1,7 @@
#!/usr/bin/env python3
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path: sys.path.insert(0, str(ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_target_resolution import main
if __name__ == "__main__": raise SystemExit(main())
@@ -0,0 +1,7 @@
#!/usr/bin/env python3
import sys
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path: sys.path.insert(0,str(ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_target_resolution_v1 import main
if __name__=="__main__": raise SystemExit(main())
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Transcribe one audio file with whisper.cpp; do not generate a protocol."""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.transcription.whisper import ( # noqa: E402
TranscriptionError,
transcribe_audio,
)
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Create a compact Meeting Lab transcript with whisper.cpp.")
parser.add_argument("audio_file", type=Path)
parser.add_argument("--model", type=Path, required=True, help="Path to a whisper.cpp GGML model.")
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--language", default="auto", help="Language code or 'auto' (default: auto).")
parser.add_argument("--threads", default="auto", help="Thread count or 'auto' for physical CPU cores (default: auto).")
parser.add_argument("--whisper-executable", default="whisper-cli", help="whisper.cpp CLI executable (default: whisper-cli).")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
try:
result = transcribe_audio(
args.audio_file,
args.model,
args.output_dir,
args.language,
executable=args.whisper_executable,
threads=args.threads,
)
except TranscriptionError as exc:
print(f"Error: {exc}")
return 1
print(f"Transcript: {result.transcript_json}")
print(f"Runtime: {result.runtime_seconds:.3f} seconds")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+5
View File
@@ -0,0 +1,5 @@
"""Canonical audio preparation boundary."""
from .preparation import AudioPreparationError, PreparedAudio, prepare_audio
__all__ = ["AudioPreparationError", "PreparedAudio", "prepare_audio"]
+190
View File
@@ -0,0 +1,190 @@
"""Prepare supported recordings for deterministic downstream processing."""
from __future__ import annotations
import os
import shutil
import subprocess
import wave
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from pathlib import Path
SUPPORTED_EXTENSIONS = {".wav", ".flac", ".m4a"}
CANONICAL_SAMPLE_RATE = 16_000
CANONICAL_CHANNELS = 1
CANONICAL_SAMPLE_WIDTH_BYTES = 2
CANONICAL_CODEC = "pcm_s16le"
DEFAULT_NORMALIZATION_FILTER = "loudnorm=I=-16:LRA=11:TP=-1.5"
DEFAULT_NORMALIZATION_METHOD = "ffmpeg_loudnorm"
class AudioPreparationError(RuntimeError):
"""Raised when source audio cannot be prepared as canonical WAV."""
@dataclass(frozen=True)
class PreparedAudio:
source_path: Path
source_format: str
prepared_path: Path
method: str
ffmpeg_executable: str
normalization_enabled: bool = True
normalization_method: str | None = DEFAULT_NORMALIZATION_METHOD
normalization_filter: str | None = DEFAULT_NORMALIZATION_FILTER
def metadata(self) -> dict[str, object]:
return {
"original_source_path": str(self.source_path.resolve()),
"original_source_name": self.source_path.name,
"original_format": self.source_format,
"prepared_audio_path": str(self.prepared_path.resolve()),
"preparation_method": self.method,
"ffmpeg_executable": self.ffmpeg_executable,
"normalization_enabled": self.normalization_enabled,
"normalization_method": self.normalization_method,
"normalization_filter": self.normalization_filter,
"canonical_output": {
"container": "wav",
"codec": CANONICAL_CODEC,
"channels": CANONICAL_CHANNELS,
"sample_rate_hz": CANONICAL_SAMPLE_RATE,
"bits_per_sample": CANONICAL_SAMPLE_WIDTH_BYTES * 8,
},
}
Runner = Callable[..., subprocess.CompletedProcess[str]]
def prepare_audio(
source_path: Path,
prepared_path: Path,
*,
ffmpeg_executable: str = "ffmpeg",
normalization_enabled: bool = True,
runner: Runner = subprocess.run,
) -> PreparedAudio:
"""Create and validate a canonical mono 16 kHz signed PCM16 WAV artifact."""
source_path = Path(source_path)
prepared_path = Path(prepared_path)
source_format = source_path.suffix.lower()
if not source_path.is_file():
raise AudioPreparationError(f"Source audio does not exist: {source_path}")
if source_format not in SUPPORTED_EXTENSIONS:
supported = ", ".join(sorted(SUPPORTED_EXTENSIONS))
raise AudioPreparationError(
f"Unsupported audio format {source_format or '<none>'!r}; supported: {supported}."
)
resolved_executable = _resolve_executable(ffmpeg_executable)
prepared_path.parent.mkdir(parents=True, exist_ok=True)
temporary_path = prepared_path.with_name(f".{prepared_path.name}.tmp.wav")
command_parts = [
resolved_executable,
"-nostdin",
"-hide_banner",
"-loglevel",
"error",
"-y",
"-i",
str(source_path),
"-map_metadata",
"-1",
"-vn",
]
if normalization_enabled:
command_parts.extend(("-af", DEFAULT_NORMALIZATION_FILTER))
command_parts.extend(
(
"-ac",
str(CANONICAL_CHANNELS),
"-ar",
str(CANONICAL_SAMPLE_RATE),
"-c:a",
CANONICAL_CODEC,
"-fflags",
"+bitexact",
str(temporary_path),
)
)
command: Sequence[str] = tuple(command_parts)
try:
completed = runner(command, capture_output=True, text=True, check=False)
except OSError as exc:
raise AudioPreparationError(f"Could not run FFmpeg: {exc}") from exc
if completed.returncode != 0:
detail = (
completed.stderr or completed.stdout or "no diagnostic output"
).strip()
raise AudioPreparationError(
f"FFmpeg failed to prepare {source_path.name} (exit {completed.returncode}): "
f"{detail}"
)
try:
_validate_canonical_wav(temporary_path)
os.replace(temporary_path, prepared_path)
except Exception:
temporary_path.unlink(missing_ok=True)
raise
return PreparedAudio(
source_path=source_path,
source_format=source_format.removeprefix("."),
prepared_path=prepared_path,
method="ffmpeg",
ffmpeg_executable=resolved_executable,
normalization_enabled=normalization_enabled,
normalization_method=(
DEFAULT_NORMALIZATION_METHOD if normalization_enabled else None
),
normalization_filter=(
DEFAULT_NORMALIZATION_FILTER if normalization_enabled else None
),
)
def _resolve_executable(executable: str) -> str:
value = executable.strip()
if not value:
raise AudioPreparationError("FFmpeg executable must not be empty.")
if Path(value).parent != Path("."):
path = Path(value)
if path.is_file() and os.access(path, os.X_OK):
return str(path)
raise AudioPreparationError(f"FFmpeg executable is not available: {value}")
resolved = shutil.which(value)
if resolved is None:
raise AudioPreparationError(
f"FFmpeg executable {value!r} was not found on PATH. Install FFmpeg or "
"configure its executable path."
)
return resolved
def _validate_canonical_wav(path: Path) -> None:
try:
with wave.open(str(path), "rb") as recording:
properties = (
recording.getnchannels(),
recording.getframerate(),
recording.getsampwidth(),
recording.getcomptype(),
)
except (OSError, EOFError, wave.Error) as exc:
raise AudioPreparationError(
f"FFmpeg did not produce a readable WAV file: {path}: {exc}"
) from exc
expected = (
CANONICAL_CHANNELS,
CANONICAL_SAMPLE_RATE,
CANONICAL_SAMPLE_WIDTH_BYTES,
"NONE",
)
if properties != expected:
raise AudioPreparationError(
"Prepared audio is not canonical mono 16 kHz PCM16 WAV: "
f"channels={properties[0]}, sample_rate={properties[1]}, "
f"sample_width={properties[2]}, compression={properties[3]}."
)
@@ -0,0 +1,349 @@
#!/usr/bin/env python3
"""Isolated collective-commitment Gold reliability experiment."""
from __future__ import annotations
import argparse
import json
import re
import time
from pathlib import Path
from typing import Any
from .experiment_h import (
DEFAULT_ENDPOINT,
DEFAULT_MODEL,
DerivationValidationError,
OBSERVATION_KEYS,
call_ollama,
)
GOLD_SCHEMA_VERSION = "experimental-collective-commitment-gold-v0"
RECOGNITION_KEYS = {"observation_id", "commitment_form", "normalized_action_text"}
COMMITMENT_FORMS = {"individual_first_person", "collective_first_person", "none"}
FORBIDDEN_LLM_KEYS = {
"responsible_person", "responsibility", "responsibility_scope",
"requested_actor", "owner", "ownership", "assignee", "status",
"established", "action_item", "protocol", "protocol_category", "decision",
"unresolved_issue", "confidence", "relation", "relations", "graph",
}
WEEKDAYS = {
"monday": "Montag", "montag": "Montag", "tuesday": "Dienstag",
"dienstag": "Dienstag", "wednesday": "Mittwoch", "mittwoch": "Mittwoch",
"thursday": "Donnerstag", "donnerstag": "Donnerstag", "friday": "Freitag",
"freitag": "Freitag", "saturday": "Samstag", "samstag": "Samstag",
"sunday": "Sonntag", "sonntag": "Sonntag",
}
PROMPT_TEMPLATE = """Recognize only the explicit first-person commitment form and concise action meaning in the supplied single V3-style observation.
Answer only:
1. What explicit first-person commitment form is present?
- individual_first_person: the speaker explicitly commits themself personally.
- collective_first_person: the speaker explicitly commits a "we" group.
- none: there is no explicit first-person commitment.
2. What is the concise normalized action meaning?
Tentative possibility is not commitment. Suggestion or recommendation is not commitment. Impersonal necessity is not commitment. Passive future wording is not commitment. Rejection or negation is not positive commitment. Speaker identity does not convert collective "we" into individual commitment.
Preserve material limitations such as "nur im Technikum", "nur als Versuch", or "nur 20 Meter" in normalized_action_text. Keep normalized action text in the observation language. When commitment_form is not "none", normalized_action_text must be a non-empty string. When commitment_form is "none", normalized_action_text may be a non-empty action meaning or null.
Do not infer who is responsible. Do not decide whether an action is established. Do not output responsibility, responsibility scope, requested actor, owner, assignee, status, established, Action Item, protocol, confidence, semantic relations, graphs, decisions, or unresolved issues.
Return exactly this JSON shape and no additional fields:
{{
"observation_id": "observation ID",
"commitment_form": "individual_first_person | collective_first_person | none",
"normalized_action_text": "concise action meaning" | null
}}
V3-style observation:
{observation_json}
"""
def _exact_keys(value: dict[str, Any], required: set[str], location: str) -> None:
missing = required - value.keys()
unknown = value.keys() - required
if missing:
raise DerivationValidationError(f"{location} missing required keys: {sorted(missing)}")
if unknown:
raise DerivationValidationError(f"{location} has unknown keys: {sorted(unknown)}")
def _nonempty_text(value: Any, location: str) -> str:
if not isinstance(value, str) or not value.strip():
raise DerivationValidationError(f"{location} must be a non-empty string")
return value.strip()
def _validate_observations(observations: Any) -> None:
if not isinstance(observations, list) or not observations:
raise DerivationValidationError("observations must be a non-empty list")
seen_observations: set[str] = set()
seen_evidence: set[str] = set()
for index, observation in enumerate(observations):
location = f"observations[{index}]"
if not isinstance(observation, dict):
raise DerivationValidationError(f"{location} must be an object")
_exact_keys(observation, OBSERVATION_KEYS, location)
observation_id = _nonempty_text(observation["observation_id"], f"{location}.observation_id")
evidence_id = _nonempty_text(observation["evidence_id"], f"{location}.evidence_id")
if observation_id in seen_observations or evidence_id in seen_evidence:
raise DerivationValidationError("observation and evidence provenance must be unique")
seen_observations.add(observation_id)
seen_evidence.add(evidence_id)
_nonempty_text(observation["content"], f"{location}.content")
_nonempty_text(observation["speaker"], f"{location}.speaker")
for field in ("named_person", "addressee"):
if observation[field] is not None:
_nonempty_text(observation[field], f"{location}.{field}")
def load_gold_cases(path: Path) -> list[dict[str, Any]]:
data = json.loads(path.read_text(encoding="utf-8-sig"))
if not isinstance(data, dict):
raise DerivationValidationError("Gold fixture must be an object")
_exact_keys(data, {"schema_version", "cases"}, "Gold fixture")
if data["schema_version"] != GOLD_SCHEMA_VERSION:
raise DerivationValidationError("unexpected Gold fixture schema_version")
cases = data["cases"]
if not isinstance(cases, list) or not cases:
raise DerivationValidationError("Gold fixture cases must be a non-empty list")
seen: set[str] = set()
for case in cases:
_exact_keys(case, {"case_id", "description", "observations", "expected_recognition", "expected_result"}, "Gold case")
case_id = _nonempty_text(case["case_id"], "Gold case.case_id")
if case_id in seen:
raise DerivationValidationError(f"duplicate case ID: {case_id}")
seen.add(case_id)
_validate_observations(case["observations"])
if len(case["observations"]) != 1:
raise DerivationValidationError("collective Gold cases require exactly one observation")
return cases
def build_prompt(observations: list[dict[str, Any]]) -> str:
_validate_observations(observations)
if len(observations) != 1:
raise DerivationValidationError("collective recognition requires exactly one observation")
return PROMPT_TEMPLATE.format(
observation_json=json.dumps(observations[0], ensure_ascii=False, indent=2)
)
def parse_model_json(raw_text: str) -> dict[str, Any]:
data = json.loads(raw_text)
if not isinstance(data, dict):
raise DerivationValidationError("semantic recognition must be an object")
return data
def _reject_forbidden_keys(value: Any, location: str = "output") -> None:
if isinstance(value, dict):
forbidden = FORBIDDEN_LLM_KEYS.intersection(value)
if forbidden:
raise DerivationValidationError(
f"{location} contains forbidden semantic keys: {sorted(forbidden)}"
)
for key, item in value.items():
_reject_forbidden_keys(item, f"{location}.{key}")
elif isinstance(value, list):
for index, item in enumerate(value):
_reject_forbidden_keys(item, f"{location}[{index}]")
def validate_recognition(data: Any, observations: list[dict[str, Any]]) -> dict[str, Any]:
_validate_observations(observations)
if not isinstance(data, dict):
raise DerivationValidationError("semantic recognition must be an object")
_reject_forbidden_keys(data)
_exact_keys(data, RECOGNITION_KEYS, "output")
observation_id = _nonempty_text(data["observation_id"], "output.observation_id")
if observation_id not in {item["observation_id"] for item in observations}:
raise DerivationValidationError("recognition references unknown observation")
if data["commitment_form"] not in COMMITMENT_FORMS:
raise DerivationValidationError("commitment_form has an unsupported value")
action_text = data["normalized_action_text"]
if action_text is not None:
_nonempty_text(action_text, "output.normalized_action_text")
if data["commitment_form"] != "none" and action_text is None:
raise DerivationValidationError("non-none commitment requires normalized_action_text")
return data
def _bounded_due(observations: list[dict[str, Any]]) -> tuple[str | None, bool]:
due_forms: set[str] = set()
for observation in observations:
content = observation["content"].casefold()
for token in re.findall(r"\b[A-Za-zÄÖÜäöü]+\b", content):
if token in WEEKDAYS:
due_forms.add(WEEKDAYS[token])
if re.search(r"\bnächste\s+woche\b", content):
due_forms.add("nächste Woche")
return (next(iter(due_forms)) if len(due_forms) == 1 else None, len(due_forms) <= 1)
def _has_explicit_negation(observations: list[dict[str, Any]]) -> bool:
return any(
re.search(r"\b(?:nicht|kein(?:e|en|er|es)?|nein|not|no)\b", item["content"], re.IGNORECASE)
for item in observations
)
def derive_collective_action(
observations: list[dict[str, Any]], recognition: dict[str, Any]
) -> tuple[dict[str, bool], dict[str, Any] | None]:
validate_recognition(recognition, observations)
by_id = {item["observation_id"]: item for item in observations}
observation = by_id.get(recognition["observation_id"])
due, deadline_consistent = _bounded_due(observations)
gates = {
"recognition_schema_valid": True,
"observation_exists": observation is not None,
"provenance_valid_and_unique": observation is not None and len({item["evidence_id"] for item in observations}) == len(observations),
"collective_commitment_form": recognition["commitment_form"] == "collective_first_person",
"normalized_action_present": isinstance(recognition["normalized_action_text"], str) and bool(recognition["normalized_action_text"].strip()),
"deadline_supported_and_consistent": deadline_consistent,
"no_explicit_negation": not _has_explicit_negation(observations),
}
if not all(gates.values()):
return gates, None
return gates, {
"action_id": "action_1",
"content": recognition["normalized_action_text"].strip(),
"status": "established",
"commitment_scope": "collective",
"responsible_person": None,
"due": due,
"support": {
"commitment": {
"observation_id": observation["observation_id"],
"evidence_id": observation["evidence_id"],
}
},
}
def _concepts_present(text: str | None, concepts: list[list[str]]) -> bool:
if not concepts:
return True
if not isinstance(text, str):
return False
folded = text.casefold()
return all(any(alias.casefold() in folded for alias in alternatives) for alternatives in concepts)
def evaluate_case(case: dict[str, Any], recognition: dict[str, Any]) -> dict[str, Any]:
validate_recognition(recognition, case["observations"])
gates, result = derive_collective_action(case["observations"], recognition)
expected_recognition = case["expected_recognition"]
expected_result = case["expected_result"]
form_correct = recognition["commitment_form"] == expected_recognition["commitment_form"]
action_correct = _concepts_present(recognition["normalized_action_text"], expected_recognition["action_concepts"])
qualifier_preserved = _concepts_present(recognition["normalized_action_text"], expected_recognition["qualifier_concepts"])
established = result is not None
final_correct = established == expected_result["established"]
if result is not None:
final_correct = final_correct and result["due"] == expected_result["due"] and result["responsible_person"] is None and result["commitment_scope"] == "collective"
owner_correct = result is None or result["responsible_person"] is None
automatic_failure = (established and not expected_result["established"]) or not owner_correct or (established and not qualifier_preserved)
semantic_correct = form_correct and action_correct and qualifier_preserved
classification = "FAIL" if automatic_failure or not final_correct else ("PASS" if semantic_correct else "PARTIAL")
return {
"case_id": case["case_id"], "classification": classification,
"commitment_form_correct": form_correct,
"normalized_action_meaning_correct": action_correct,
"material_qualifier_preserved": qualifier_preserved,
"deterministic_gates_correct": final_correct,
"final_result_correct": final_correct,
"responsible_person_correctly_null": owner_correct,
"due_correct": result is None or result["due"] == expected_result["due"],
"unsupported_semantic_strengthening": recognition["commitment_form"] == "collective_first_person" and expected_recognition["commitment_form"] != "collective_first_person",
"responsibility_status_leakage": False,
"gates": gates, "result": result,
}
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def run_gold(args: argparse.Namespace) -> dict[str, Any]:
cases = load_gold_cases(args.cases)
args.output.mkdir(parents=True, exist_ok=False)
_write_json(args.output / "gold_cases.json", {"schema_version": GOLD_SCHEMA_VERSION, "cases": cases})
evaluations: list[dict[str, Any]] = []
successful_calls = 0
technical_failures = 0
started = time.perf_counter()
for case in cases:
case_dir = args.output / case["case_id"].lower()
case_dir.mkdir()
observations = case["observations"]
_write_json(case_dir / "v3_style_input_observations.json", observations)
prompt = build_prompt(observations)
(case_dir / "prompt.txt").write_text(prompt, encoding="utf-8")
try:
raw, metadata = call_ollama(args.endpoint, args.model, prompt, args.timeout, args.num_ctx, args.num_predict)
successful_calls += 1
except Exception as exc: # one recorded attempt; never retry
technical_failures += 1
failure = {"case_id": case["case_id"], "classification": "FAIL", "technical_failure": True, "error_type": type(exc).__name__, "error": str(exc)}
_write_json(case_dir / "ollama_metadata.json", {"model": args.model, "configuration": {"temperature": 0, "think": False, "num_ctx": args.num_ctx, "num_predict": args.num_predict, "retries": 0}, "technical_failure": failure})
_write_json(case_dir / "structural_validation.json", {"valid": False, "error": str(exc)})
_write_json(case_dir / "deterministic_gate_results.json", {})
_write_json(case_dir / "final_derived_result.json", None)
_write_json(case_dir / "evaluation.json", failure)
evaluations.append(failure)
continue
(case_dir / "raw_model_response.txt").write_text(raw + "\n", encoding="utf-8")
_write_json(case_dir / "ollama_metadata.json", metadata)
try:
parsed = parse_model_json(raw)
_write_json(case_dir / "parsed_semantic_recognition.json", parsed)
evaluation = evaluate_case(case, parsed)
validation = {"valid": True, "error": None}
gates, result = derive_collective_action(observations, parsed)
except (DerivationValidationError, json.JSONDecodeError) as exc:
validation = {"valid": False, "error_type": type(exc).__name__, "error": str(exc)}
evaluation = {"case_id": case["case_id"], "classification": "FAIL", "error": str(exc), "responsibility_status_leakage": "forbidden" in str(exc)}
gates, result = {}, None
_write_json(case_dir / "structural_validation.json", validation)
_write_json(case_dir / "deterministic_gate_results.json", gates)
_write_json(case_dir / "final_derived_result.json", result)
_write_json(case_dir / "evaluation.json", evaluation)
evaluations.append(evaluation)
summary = {
"experiment": "collective_commitment_gold_v0", "model": args.model,
"successful_llm_call_count": successful_calls,
"technical_failed_call_count": technical_failures,
"runtime_seconds": round(time.perf_counter() - started, 3),
"counts": {label: sum(item["classification"] == label for item in evaluations) for label in ("PASS", "PARTIAL", "FAIL")},
"evaluations": evaluations,
}
_write_json(args.output / "summary.json", summary)
return summary
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run isolated collective-commitment Gold experiment")
parser.add_argument("cases", type=Path)
parser.add_argument("-o", "--output", type=Path, required=True)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument("--timeout", type=int, default=300)
parser.add_argument("--num-ctx", type=int, default=16384)
parser.add_argument("--num-predict", type=int, default=1024)
return parser.parse_args()
def main() -> int:
summary = run_gold(parse_args())
print(json.dumps(summary, ensure_ascii=False, indent=2))
return 0 if summary["counts"]["FAIL"] == 0 and summary["technical_failed_call_count"] == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,277 @@
#!/usr/bin/env python3
"""Isolated evidence-near Negative Act Form classification experiment."""
from __future__ import annotations
import argparse
import json
import time
from pathlib import Path
from typing import Any
from .experiment_h import (
DEFAULT_ENDPOINT,
DEFAULT_MODEL,
DerivationValidationError,
OBSERVATION_KEYS,
build_ollama_payload,
call_ollama,
)
GOLD_SCHEMA_VERSION = "experimental-negative-act-form-gold-v0"
RECOGNITION_KEYS = {"observation_id", "negative_act_form", "normalized_action_text"}
NEGATIVE_ACT_FORMS = {
"explicit_non_pursuit", "personal_preference", "recommendation",
"temporary_non_action", "none",
}
FORBIDDEN_LLM_KEYS = {
"rejection_form", "explicitly_rejected", "status", "decision", "outcome",
"topic_status", "responsible_person", "responsibility", "owner",
"requested_actor", "action_item", "protocol", "protocol_category",
"confidence", "relation", "relations", "graph", "unresolved_issue",
}
PROMPT_TEMPLATE = """Classify only the negative semantic form expressed by the candidate observation, using earlier supplied V3-style observations only as local context for pronouns or shortened references.
The candidate observation is {candidate_observation_id}.
Choose exactly one negative_act_form:
- explicit_non_pursuit: explicitly states that an action, option, collaboration, or course will not be continued or pursued. This is stronger than preference, advice, or temporary delay.
- personal_preference: the speaker states what they personally would or would not do, without establishing collective non-pursuit.
- recommendation: the speaker advises for or against an action without establishing abandonment.
- temporary_non_action: the action is postponed, deferred, or explicitly not done for now without abandonment.
- none: none of those four forms is present, including mere concern, uncertainty, negative sentiment, or factual negation.
Do not collapse non-pursuit into temporary non-action. Do not convert a personal conditional preference into collective non-pursuit. Do not convert advice into non-pursuit. Speaker identity does not change personal preference into collective non-pursuit.
When the form is not none, return concise normalized action meaning. Resolve a pronoun only from the supplied local context. If its target is genuinely ambiguous, return none rather than guessing. When the form is none, normalized_action_text must be null. Keep normalized action text in the observation language.
Do not derive or output rejection, status, decision, outcome, topic closure, responsibility, ownership, Action Item, protocol category, confidence, relations, graphs, or unresolved issues.
Return exactly this JSON shape and no additional fields:
{{
"observation_id": "{candidate_observation_id}",
"negative_act_form": "explicit_non_pursuit | personal_preference | recommendation | temporary_non_action | none",
"normalized_action_text": "concise action meaning" | null
}}
V3-style observations:
{observations_json}
"""
def _exact_keys(value: dict[str, Any], required: set[str], location: str) -> None:
missing = required - value.keys()
unknown = value.keys() - required
if missing:
raise DerivationValidationError(f"{location} missing required keys: {sorted(missing)}")
if unknown:
raise DerivationValidationError(f"{location} has unknown keys: {sorted(unknown)}")
def _nonempty_text(value: Any, location: str) -> str:
if not isinstance(value, str) or not value.strip():
raise DerivationValidationError(f"{location} must be a non-empty string")
return value.strip()
def _validate_observations(observations: Any) -> None:
if not isinstance(observations, list) or not observations:
raise DerivationValidationError("observations must be a non-empty list")
seen_observations: set[str] = set()
seen_evidence: set[str] = set()
for index, observation in enumerate(observations):
location = f"observations[{index}]"
if not isinstance(observation, dict):
raise DerivationValidationError(f"{location} must be an object")
_exact_keys(observation, OBSERVATION_KEYS, location)
observation_id = _nonempty_text(observation["observation_id"], f"{location}.observation_id")
evidence_id = _nonempty_text(observation["evidence_id"], f"{location}.evidence_id")
if observation_id in seen_observations or evidence_id in seen_evidence:
raise DerivationValidationError("observation and evidence provenance must be unique")
seen_observations.add(observation_id)
seen_evidence.add(evidence_id)
_nonempty_text(observation["content"], f"{location}.content")
_nonempty_text(observation["speaker"], f"{location}.speaker")
for field in ("named_person", "addressee"):
if observation[field] is not None:
_nonempty_text(observation[field], f"{location}.{field}")
def load_gold_cases(path: Path) -> list[dict[str, Any]]:
data = json.loads(path.read_text(encoding="utf-8-sig"))
if not isinstance(data, dict):
raise DerivationValidationError("Gold fixture must be an object")
_exact_keys(data, {"schema_version", "cases"}, "Gold fixture")
if data["schema_version"] != GOLD_SCHEMA_VERSION:
raise DerivationValidationError("unexpected Gold fixture schema_version")
cases = data["cases"]
if not isinstance(cases, list) or not cases:
raise DerivationValidationError("Gold fixture cases must be a non-empty list")
seen: set[str] = set()
for case in cases:
_exact_keys(case, {"case_id", "description", "observations", "expected"}, "Gold case")
case_id = _nonempty_text(case["case_id"], "Gold case.case_id")
if case_id in seen:
raise DerivationValidationError(f"duplicate case ID: {case_id}")
seen.add(case_id)
_validate_observations(case["observations"])
if len(case["observations"]) not in (1, 2):
raise DerivationValidationError("Negative Act cases require one or two observations")
return cases
def build_prompt(case: dict[str, Any]) -> str:
observations = case["observations"]
_validate_observations(observations)
candidate_id = observations[-1]["observation_id"]
return PROMPT_TEMPLATE.format(
candidate_observation_id=candidate_id,
observations_json=json.dumps(observations, ensure_ascii=False, indent=2),
)
def parse_model_json(raw_text: str) -> dict[str, Any]:
data = json.loads(raw_text)
if not isinstance(data, dict):
raise DerivationValidationError("semantic classification must be an object")
return data
def _reject_forbidden_keys(value: Any, location: str = "output") -> None:
if isinstance(value, dict):
forbidden = FORBIDDEN_LLM_KEYS.intersection(value)
if forbidden:
raise DerivationValidationError(f"{location} contains forbidden semantic keys: {sorted(forbidden)}")
for key, item in value.items():
_reject_forbidden_keys(item, f"{location}.{key}")
elif isinstance(value, list):
for index, item in enumerate(value):
_reject_forbidden_keys(item, f"{location}[{index}]")
def validate_classification(data: Any, observations: list[dict[str, Any]]) -> dict[str, Any]:
_validate_observations(observations)
if not isinstance(data, dict):
raise DerivationValidationError("semantic classification must be an object")
_reject_forbidden_keys(data)
_exact_keys(data, RECOGNITION_KEYS, "output")
observation_id = _nonempty_text(data["observation_id"], "output.observation_id")
if observation_id not in {item["observation_id"] for item in observations}:
raise DerivationValidationError("classification references unknown observation")
form = data["negative_act_form"]
if form not in NEGATIVE_ACT_FORMS:
raise DerivationValidationError("negative_act_form has an unsupported value")
action_text = data["normalized_action_text"]
if form == "none":
if action_text is not None:
raise DerivationValidationError("none form requires null normalized_action_text")
else:
_nonempty_text(action_text, "output.normalized_action_text")
return data
def _concepts_present(text: str | None, concepts: list[list[str]]) -> bool:
if not concepts:
return text is None
if not isinstance(text, str):
return False
folded = text.casefold()
return all(any(alias.casefold() in folded for alias in alternatives) for alternatives in concepts)
def evaluate_case(case: dict[str, Any], classification: dict[str, Any]) -> dict[str, Any]:
validate_classification(classification, case["observations"])
expected = case["expected"]
observation_correct = classification["observation_id"] == expected["observation_id"]
form_correct = classification["negative_act_form"] == expected["negative_act_form"]
action_correct = _concepts_present(classification["normalized_action_text"], expected["action_concepts"])
unsupported_strengthening = expected["negative_act_form"] == "none" and classification["negative_act_form"] != "none"
classification_label = "PASS" if observation_correct and form_correct and action_correct else ("PARTIAL" if observation_correct and form_correct else "FAIL")
return {
"case_id": case["case_id"], "classification": classification_label,
"expected_negative_act_form": expected["negative_act_form"],
"actual_negative_act_form": classification["negative_act_form"],
"observation_id_correct": observation_correct,
"normalized_action_meaning_correct": action_correct,
"unsupported_semantic_strengthening": unsupported_strengthening,
"normative_leakage": False,
}
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def run_experiment(args: argparse.Namespace) -> dict[str, Any]:
cases = load_gold_cases(args.cases)
args.output.mkdir(parents=True, exist_ok=False)
_write_json(args.output / "gold_cases.json", {"schema_version": GOLD_SCHEMA_VERSION, "cases": cases})
evaluations: list[dict[str, Any]] = []
successful_calls = 0
technical_failures = 0
started = time.perf_counter()
for case in cases:
case_dir = args.output / case["case_id"].lower()
case_dir.mkdir()
observations = case["observations"]
_write_json(case_dir / "v3_style_input_observations.json", observations)
prompt = build_prompt(case)
(case_dir / "prompt.txt").write_text(prompt, encoding="utf-8")
try:
raw, metadata = call_ollama(args.endpoint, args.model, prompt, args.timeout, args.num_ctx, args.num_predict)
successful_calls += 1
except Exception as exc: # one recorded attempt; never retry
technical_failures += 1
failure = {"case_id": case["case_id"], "classification": "FAIL", "technical_failure": True, "error_type": type(exc).__name__, "error": str(exc)}
_write_json(case_dir / "ollama_metadata.json", {"model": args.model, "configuration": {"temperature": 0, "think": False, "num_ctx": args.num_ctx, "num_predict": args.num_predict, "retries": 0}, "technical_failure": failure})
_write_json(case_dir / "structural_validation.json", {"valid": False, "error": str(exc)})
_write_json(case_dir / "evaluation.json", failure)
evaluations.append(failure)
continue
(case_dir / "raw_model_response.txt").write_text(raw + "\n", encoding="utf-8")
_write_json(case_dir / "ollama_metadata.json", metadata)
try:
parsed = parse_model_json(raw)
_write_json(case_dir / "parsed_semantic_classification.json", parsed)
evaluation = evaluate_case(case, parsed)
validation = {"valid": True, "error": None}
except (DerivationValidationError, json.JSONDecodeError) as exc:
validation = {"valid": False, "error_type": type(exc).__name__, "error": str(exc)}
evaluation = {"case_id": case["case_id"], "classification": "FAIL", "error": str(exc), "normative_leakage": "forbidden" in str(exc)}
_write_json(case_dir / "structural_validation.json", validation)
_write_json(case_dir / "evaluation.json", evaluation)
evaluations.append(evaluation)
summary = {
"experiment": "negative_act_form_v0", "model": args.model,
"successful_llm_call_count": successful_calls,
"technical_failed_call_count": technical_failures,
"runtime_seconds": round(time.perf_counter() - started, 3),
"counts": {label: sum(item["classification"] == label for item in evaluations) for label in ("PASS", "PARTIAL", "FAIL")},
"evaluations": evaluations,
}
_write_json(args.output / "summary.json", summary)
return summary
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run isolated Negative Act Form experiment")
parser.add_argument("cases", type=Path)
parser.add_argument("-o", "--output", type=Path, required=True)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument("--timeout", type=int, default=300)
parser.add_argument("--num-ctx", type=int, default=16384)
parser.add_argument("--num-predict", type=int, default=1024)
return parser.parse_args()
def main() -> int:
summary = run_experiment(parse_args())
print(json.dumps(summary, ensure_ascii=False, indent=2))
return 0 if summary["counts"]["FAIL"] == 0 and summary["technical_failed_call_count"] == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,347 @@
#!/usr/bin/env python3
"""Isolated explicit-action-rejection Gold reliability experiment."""
from __future__ import annotations
import argparse
import json
import time
from pathlib import Path
from typing import Any
from .experiment_h import (
DEFAULT_ENDPOINT,
DEFAULT_MODEL,
DerivationValidationError,
OBSERVATION_KEYS,
call_ollama,
)
GOLD_SCHEMA_VERSION = "experimental-explicit-rejection-gold-v0"
RECOGNITION_KEYS = {
"rejection_observation_id", "target_observation_id", "rejection_form",
"normalized_rejected_action_text",
}
REJECTION_FORMS = {"explicit_action_rejection", "none"}
FORBIDDEN_LLM_KEYS = {
"decision", "decision_status", "outcome", "topic_status", "closed",
"agreement", "responsible_person", "responsibility", "responsibility_scope",
"owner", "ownership", "assignee", "requested_actor", "status",
"explicitly_rejected", "action_item", "protocol", "protocol_category",
"confidence", "relation", "relations", "graph", "unresolved_issue",
}
PROMPT_TEMPLATE = """Recognize only whether the candidate rejection observation explicitly rejects a concrete action, option, proposal, or future course of action in this small local set of V3-style observations.
Answer only:
1. Does the candidate rejection observation explicitly reject, abandon, discontinue, or rule out a concrete action, option, proposal, or future course of action?
2. If yes, which supplied observation identifies the rejected target?
3. What is the concise normalized meaning of the rejected action or option?
The candidate rejection observation is {rejection_observation_id}.
Use explicit_action_rejection only for an asserted rejection, abandonment, discontinuation, or non-pursuit with a concrete locally resolvable target. Personal preference is not meeting-level explicit rejection. Concern or objection without refusal is not rejection. Uncertainty is not rejection. Negative recommendation or advice is not established rejection. Deferral is not rejection. "Not yet" or temporary non-action is not abandonment. Factual negation is not action rejection. Lack of commitment is not rejection.
The rejected target may be self-contained in the candidate observation or introduced by one earlier supplied observation. Choose only among supplied observation IDs. If the target is ambiguous or unresolved, return rejection_form none. Preserve material scope limitations in normalized_rejected_action_text. Ignore a separate positive alternative when describing the rejected target. Keep normalized text in the observation language.
Do not infer responsibility, ownership, decision status, final outcome, topic closure, protocol status, confidence, relations, graphs, or unresolved issues. Do not answer whether this was finally decided, what the meeting outcome was, who is responsible, or whether the topic is closed.
Return exactly this JSON shape and no additional fields:
{{
"rejection_observation_id": "{rejection_observation_id}",
"target_observation_id": "supplied observation ID" | null,
"rejection_form": "explicit_action_rejection | none",
"normalized_rejected_action_text": "concise rejected target" | null
}}
For rejection_form none, target_observation_id and normalized_rejected_action_text must both be null.
V3-style observations:
{observations_json}
"""
def _exact_keys(value: dict[str, Any], required: set[str], location: str) -> None:
missing = required - value.keys()
unknown = value.keys() - required
if missing:
raise DerivationValidationError(f"{location} missing required keys: {sorted(missing)}")
if unknown:
raise DerivationValidationError(f"{location} has unknown keys: {sorted(unknown)}")
def _nonempty_text(value: Any, location: str) -> str:
if not isinstance(value, str) or not value.strip():
raise DerivationValidationError(f"{location} must be a non-empty string")
return value.strip()
def _validate_observations(observations: Any) -> None:
if not isinstance(observations, list) or not observations:
raise DerivationValidationError("observations must be a non-empty list")
seen_observations: set[str] = set()
seen_evidence: set[str] = set()
for index, observation in enumerate(observations):
location = f"observations[{index}]"
if not isinstance(observation, dict):
raise DerivationValidationError(f"{location} must be an object")
_exact_keys(observation, OBSERVATION_KEYS, location)
observation_id = _nonempty_text(observation["observation_id"], f"{location}.observation_id")
evidence_id = _nonempty_text(observation["evidence_id"], f"{location}.evidence_id")
if observation_id in seen_observations:
raise DerivationValidationError("observation IDs must be unique")
if evidence_id in seen_evidence:
raise DerivationValidationError("evidence provenance must be unique and consistent")
seen_observations.add(observation_id)
seen_evidence.add(evidence_id)
_nonempty_text(observation["content"], f"{location}.content")
_nonempty_text(observation["speaker"], f"{location}.speaker")
for field in ("named_person", "addressee"):
if observation[field] is not None:
_nonempty_text(observation[field], f"{location}.{field}")
def load_gold_cases(path: Path) -> list[dict[str, Any]]:
data = json.loads(path.read_text(encoding="utf-8-sig"))
if not isinstance(data, dict):
raise DerivationValidationError("Gold fixture must be an object")
_exact_keys(data, {"schema_version", "cases"}, "Gold fixture")
if data["schema_version"] != GOLD_SCHEMA_VERSION:
raise DerivationValidationError("unexpected Gold fixture schema_version")
cases = data["cases"]
if not isinstance(cases, list) or not cases:
raise DerivationValidationError("Gold fixture cases must be a non-empty list")
seen: set[str] = set()
for case in cases:
_exact_keys(case, {"case_id", "description", "observations", "expected_recognition", "expected_result"}, "Gold case")
case_id = _nonempty_text(case["case_id"], "Gold case.case_id")
if case_id in seen:
raise DerivationValidationError(f"duplicate case ID: {case_id}")
seen.add(case_id)
_validate_observations(case["observations"])
if len(case["observations"]) not in (1, 2):
raise DerivationValidationError("rejection Gold cases require one or two observations")
return cases
def build_prompt(case: dict[str, Any]) -> str:
observations = case["observations"]
_validate_observations(observations)
rejection_observation_id = observations[-1]["observation_id"]
return PROMPT_TEMPLATE.format(
rejection_observation_id=rejection_observation_id,
observations_json=json.dumps(observations, ensure_ascii=False, indent=2),
)
def parse_model_json(raw_text: str) -> dict[str, Any]:
data = json.loads(raw_text)
if not isinstance(data, dict):
raise DerivationValidationError("semantic recognition must be an object")
return data
def _reject_forbidden_keys(value: Any, location: str = "output") -> None:
if isinstance(value, dict):
forbidden = FORBIDDEN_LLM_KEYS.intersection(value)
if forbidden:
raise DerivationValidationError(f"{location} contains forbidden semantic keys: {sorted(forbidden)}")
for key, item in value.items():
_reject_forbidden_keys(item, f"{location}.{key}")
elif isinstance(value, list):
for index, item in enumerate(value):
_reject_forbidden_keys(item, f"{location}[{index}]")
def validate_recognition(data: Any, observations: list[dict[str, Any]]) -> dict[str, Any]:
_validate_observations(observations)
if not isinstance(data, dict):
raise DerivationValidationError("semantic recognition must be an object")
_reject_forbidden_keys(data)
_exact_keys(data, RECOGNITION_KEYS, "output")
rejection_id = _nonempty_text(data["rejection_observation_id"], "output.rejection_observation_id")
known_ids = {item["observation_id"] for item in observations}
if rejection_id not in known_ids:
raise DerivationValidationError("unknown rejection observation ID")
form = data["rejection_form"]
if form not in REJECTION_FORMS:
raise DerivationValidationError("rejection_form has an unsupported value")
target_id = data["target_observation_id"]
action_text = data["normalized_rejected_action_text"]
if form == "none":
if target_id is not None:
raise DerivationValidationError("none rejection must have null target_observation_id")
if action_text is not None:
raise DerivationValidationError("none rejection must have null normalized_rejected_action_text")
else:
target_id = _nonempty_text(target_id, "output.target_observation_id")
if target_id not in known_ids:
raise DerivationValidationError("unknown target observation ID")
_nonempty_text(action_text, "output.normalized_rejected_action_text")
return data
def derive_rejection(
observations: list[dict[str, Any]], recognition: dict[str, Any]
) -> tuple[dict[str, bool], dict[str, Any] | None]:
validate_recognition(recognition, observations)
by_id = {item["observation_id"]: item for item in observations}
positions = {item["observation_id"]: index for index, item in enumerate(observations)}
rejection = by_id.get(recognition["rejection_observation_id"])
target_id = recognition["target_observation_id"]
target = by_id.get(target_id) if target_id is not None else None
gates = {
"recognition_schema_valid": True,
"explicit_action_rejection": recognition["rejection_form"] == "explicit_action_rejection",
"rejection_observation_exists": rejection is not None,
"target_observation_exists": target is not None,
"observation_ids_valid_and_unique": len(by_id) == len(observations),
"evidence_provenance_valid_unique_consistent": len({item["evidence_id"] for item in observations}) == len(observations),
"target_same_or_before_rejection": target is not None and rejection is not None and positions[target["observation_id"]] <= positions[rejection["observation_id"]],
"normalized_rejected_action_present": isinstance(recognition["normalized_rejected_action_text"], str) and bool(recognition["normalized_rejected_action_text"].strip()),
"target_local_to_case": target_id in by_id if target_id is not None else False,
"schema_state_consistent": recognition["rejection_form"] == "explicit_action_rejection" and target_id is not None,
"referenced_provenance_available": target is not None and rejection is not None and bool(target["evidence_id"]) and bool(rejection["evidence_id"]),
}
if not all(gates.values()):
return gates, None
return gates, {
"rejection_id": "rejection_1",
"content": recognition["normalized_rejected_action_text"].strip(),
"status": "explicitly_rejected",
"support": {
"target": {"observation_id": target["observation_id"], "evidence_id": target["evidence_id"]},
"rejection": {"observation_id": rejection["observation_id"], "evidence_id": rejection["evidence_id"]},
},
}
def _concepts_present(text: str | None, concepts: list[list[str]]) -> bool:
if not concepts:
return True
if not isinstance(text, str):
return False
folded = text.casefold()
return all(any(alias.casefold() in folded for alias in alternatives) for alternatives in concepts)
def _contains_forbidden_concept(text: str | None, concepts: list[str]) -> bool:
return isinstance(text, str) and any(concept.casefold() in text.casefold() for concept in concepts)
def evaluate_case(case: dict[str, Any], recognition: dict[str, Any]) -> dict[str, Any]:
validate_recognition(recognition, case["observations"])
gates, result = derive_rejection(case["observations"], recognition)
expected = case["expected_recognition"]
expected_result = case["expected_result"]
form_correct = recognition["rejection_form"] == expected["rejection_form"]
rejection_observation_correct = recognition["rejection_observation_id"] == expected["rejection_observation_id"]
target_correct = recognition["target_observation_id"] == expected["target_observation_id"]
action_correct = _concepts_present(recognition["normalized_rejected_action_text"], expected["action_concepts"])
qualifier_preserved = _concepts_present(recognition["normalized_rejected_action_text"], expected["qualifier_concepts"])
alternative_absorbed = _contains_forbidden_concept(recognition["normalized_rejected_action_text"], expected["forbidden_action_concepts"])
derived = result is not None
final_correct = derived == expected_result["explicitly_rejected"]
if result is not None:
final_correct = final_correct and result["status"] == "explicitly_rejected"
semantic_correct = form_correct and rejection_observation_correct and target_correct and action_correct and qualifier_preserved and not alternative_absorbed
automatic_failure = (derived and not expected_result["explicitly_rejected"]) or (derived and not target_correct) or (derived and not qualifier_preserved) or alternative_absorbed
classification = "FAIL" if automatic_failure or not final_correct else ("PASS" if semantic_correct else "PARTIAL")
return {
"case_id": case["case_id"], "classification": classification,
"rejection_form_correct": form_correct,
"rejection_observation_correct": rejection_observation_correct,
"target_observation_correct": target_correct,
"normalized_rejected_action_correct": action_correct,
"material_qualifiers_preserved": qualifier_preserved,
"positive_alternative_absorbed": alternative_absorbed,
"deterministic_gates_correct": final_correct,
"final_result_correct": final_correct,
"unsupported_semantic_strengthening": recognition["rejection_form"] == "explicit_action_rejection" and expected["rejection_form"] == "none",
"normative_leakage": False,
"gates": gates, "result": result,
}
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def run_gold(args: argparse.Namespace) -> dict[str, Any]:
cases = load_gold_cases(args.cases)
args.output.mkdir(parents=True, exist_ok=False)
_write_json(args.output / "gold_cases.json", {"schema_version": GOLD_SCHEMA_VERSION, "cases": cases})
evaluations: list[dict[str, Any]] = []
successful_calls = 0
technical_failures = 0
started = time.perf_counter()
for case in cases:
case_dir = args.output / case["case_id"].lower()
case_dir.mkdir()
observations = case["observations"]
_write_json(case_dir / "v3_style_input_observations.json", observations)
prompt = build_prompt(case)
(case_dir / "prompt.txt").write_text(prompt, encoding="utf-8")
try:
raw, metadata = call_ollama(args.endpoint, args.model, prompt, args.timeout, args.num_ctx, args.num_predict)
successful_calls += 1
except Exception as exc: # one recorded attempt; never retry
technical_failures += 1
failure = {"case_id": case["case_id"], "classification": "FAIL", "technical_failure": True, "error_type": type(exc).__name__, "error": str(exc)}
_write_json(case_dir / "ollama_metadata.json", {"model": args.model, "configuration": {"temperature": 0, "think": False, "num_ctx": args.num_ctx, "num_predict": args.num_predict, "retries": 0}, "technical_failure": failure})
_write_json(case_dir / "structural_validation.json", {"valid": False, "error": str(exc)})
_write_json(case_dir / "deterministic_gate_results.json", {})
_write_json(case_dir / "final_derived_result.json", None)
_write_json(case_dir / "evaluation.json", failure)
evaluations.append(failure)
continue
(case_dir / "raw_model_response.txt").write_text(raw + "\n", encoding="utf-8")
_write_json(case_dir / "ollama_metadata.json", metadata)
try:
parsed = parse_model_json(raw)
_write_json(case_dir / "parsed_semantic_recognition.json", parsed)
evaluation = evaluate_case(case, parsed)
validation = {"valid": True, "error": None}
gates, result = derive_rejection(observations, parsed)
except (DerivationValidationError, json.JSONDecodeError) as exc:
validation = {"valid": False, "error_type": type(exc).__name__, "error": str(exc)}
evaluation = {"case_id": case["case_id"], "classification": "FAIL", "error": str(exc), "normative_leakage": "forbidden" in str(exc)}
gates, result = {}, None
_write_json(case_dir / "structural_validation.json", validation)
_write_json(case_dir / "deterministic_gate_results.json", gates)
_write_json(case_dir / "final_derived_result.json", result)
_write_json(case_dir / "evaluation.json", evaluation)
evaluations.append(evaluation)
summary = {
"experiment": "explicit_rejection_gold_v0", "model": args.model,
"successful_llm_call_count": successful_calls,
"technical_failed_call_count": technical_failures,
"runtime_seconds": round(time.perf_counter() - started, 3),
"counts": {label: sum(item["classification"] == label for item in evaluations) for label in ("PASS", "PARTIAL", "FAIL")},
"evaluations": evaluations,
}
_write_json(args.output / "summary.json", summary)
return summary
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run isolated explicit-rejection Gold experiment")
parser.add_argument("cases", type=Path)
parser.add_argument("-o", "--output", type=Path, required=True)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument("--timeout", type=int, default=300)
parser.add_argument("--num-ctx", type=int, default=16384)
parser.add_argument("--num-predict", type=int, default=1024)
return parser.parse_args()
def main() -> int:
summary = run_gold(parse_args())
print(json.dumps(summary, ensure_ascii=False, indent=2))
return 0 if summary["counts"]["FAIL"] == 0 and summary["technical_failed_call_count"] == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
"""Isolated controlled rejection derivation V1 experiment."""
from __future__ import annotations
import argparse, json, time
from pathlib import Path
from typing import Any
from .experiment_h import DEFAULT_ENDPOINT, DEFAULT_MODEL, DerivationValidationError, OBSERVATION_KEYS, call_ollama
from .experiment_negative_act import build_prompt as build_negative_prompt, parse_model_json, validate_classification
SCHEMA_VERSION="experimental-controlled-rejection-v1"
TARGET_KEYS={"candidate_observation_id","target_observation_id","normalized_target_text"}
FORBIDDEN={"rejection_form","negative_act_form","explicitly_rejected","status","decision","outcome","topic_status","closed","responsible_person","responsibility","owner","requested_actor","action_item","protocol_category","confidence","relation","relations","graph","unresolved_issue"}
PROMPT="""Resolve only the concrete local action or option referred to by the candidate negative act. The candidate is {candidate}. Choose only a supplied observation ID. Use the same observation for a self-contained target. If no unique local target exists, return null for both target fields. Preserve source-language meaning and material scope such as purpose and location. Preserve continuation when non-pursuit concerns continuing something. Ignore any separate positive alternative. Do not classify the negative act and do not output rejection, status, decision, outcome, responsibility, protocol concepts, confidence, relations, or graphs. Return exactly JSON with candidate_observation_id, target_observation_id, normalized_target_text and no other fields.\nObservations:\n{observations}"""
def _keys(v,r,loc):
if not isinstance(v,dict): raise DerivationValidationError(f"{loc} must be an object")
if set(v)!=r: raise DerivationValidationError(f"{loc} keys invalid: missing={sorted(r-set(v))}, unknown={sorted(set(v)-r)}")
def _text(v,loc):
if not isinstance(v,str) or not v.strip(): raise DerivationValidationError(f"{loc} must be non-empty")
return v.strip()
def _forbidden(v,loc="output"):
if isinstance(v,dict):
bad=FORBIDDEN & set(v)
if bad: raise DerivationValidationError(f"{loc} contains forbidden fields: {sorted(bad)}")
for k,x in v.items(): _forbidden(x,f"{loc}.{k}")
elif isinstance(v,list):
for i,x in enumerate(v): _forbidden(x,f"{loc}[{i}]")
def validate_observations(obs):
if not isinstance(obs,list) or not obs: raise DerivationValidationError("observations must be non-empty")
ids=set(); evid=set()
for i,o in enumerate(obs):
_keys(o,OBSERVATION_KEYS,f"observations[{i}]"); oid=_text(o["observation_id"],"observation_id"); eid=_text(o["evidence_id"],"evidence_id")
if oid in ids or eid in evid: raise DerivationValidationError("provenance must be unique")
ids.add(oid); evid.add(eid); _text(o["content"],"content"); _text(o["speaker"],"speaker")
return ids
def validate_target(data,obs):
ids=validate_observations(obs); _forbidden(data); _keys(data,TARGET_KEYS,"target output")
candidate=_text(data["candidate_observation_id"],"candidate_observation_id")
if candidate not in ids: raise DerivationValidationError("unknown candidate observation")
target=data["target_observation_id"]; normalized=data["normalized_target_text"]
if target is None:
if normalized is not None: raise DerivationValidationError("null target requires null text")
else:
target=_text(target,"target_observation_id")
if target not in ids: raise DerivationValidationError("unknown target observation")
_text(normalized,"normalized_target_text")
return data
def build_target_prompt(case):
validate_observations(case["observations"])
return PROMPT.format(candidate=case["expected"]["candidate_observation_id"],observations=json.dumps(case["observations"],ensure_ascii=False,indent=2))
def derive(obs,negative,target):
ids=validate_observations(obs); validate_classification(negative,obs); validate_target(target,obs)
candidate=negative["observation_id"]
if target["candidate_observation_id"]!=candidate: raise DerivationValidationError("candidate outputs disagree")
positions={o["observation_id"]:i for i,o in enumerate(obs)}; tid=target["target_observation_id"]
gates={"negative_act_valid":True,"eligible_explicit_non_pursuit":negative["negative_act_form"]=="explicit_non_pursuit","candidate_exists":candidate in ids,"target_valid":True,"target_present":tid is not None,"target_exists":tid in ids if tid else False,"target_not_after_candidate":positions[tid]<=positions[candidate] if tid else False,"provenance_valid_unique":True,"normalized_target_nonempty":bool(target["normalized_target_text"] and target["normalized_target_text"].strip()),"same_isolated_case":tid in ids if tid else False,"no_forbidden_fields":True}
established=all(gates.values())
result=None
if established:
byid={o["observation_id"]:o for o in obs}
result={"rejection_id":"rejection_1","content":target["normalized_target_text"].strip(),"status":"explicitly_rejected","support":{"target":{"observation_id":tid,"evidence_id":byid[tid]["evidence_id"]},"negative_act":{"observation_id":candidate,"evidence_id":byid[candidate]["evidence_id"]}}}
return {"gates":gates,"derived_result":result}
def load_cases(path):
data=json.loads(path.read_text(encoding="utf-8")); _keys(data,{"schema_version","cases"},"fixture")
if data["schema_version"]!=SCHEMA_VERSION: raise DerivationValidationError("wrong schema version")
return data["cases"]
def _concepts(text,groups):
folded=(text or "").casefold(); return all(any(x.casefold() in folded for x in g) for g in groups)
def evaluate(case,negative,target,derivation):
e=case["expected"]; text=target["normalized_target_text"]
form=negative["negative_act_form"]==e["negative_act_form"]; target_ok=target["target_observation_id"]==e["target_observation_id"]
action=_concepts(text,e["action_concepts"]); material=_concepts(text,e["material_concepts"]); forbidden=any(x.casefold() in (text or "").casefold() for x in e["forbidden_concepts"])
final=(derivation["derived_result"] is not None)==e["explicitly_rejected"]
label="PASS" if form and target_ok and action and material and not forbidden and final else ("PARTIAL" if form and target_ok and material and not forbidden and final else "FAIL")
return {"case_id":case["case_id"],"classification":label,"expected_negative_act_form":e["negative_act_form"],"actual_negative_act_form":negative["negative_act_form"],"expected_target_observation_id":e["target_observation_id"],"actual_target_observation_id":target["target_observation_id"],"normalized_target_text":text,"normalized_action_correct":action,"material_scope_preserved":material,"alternative_absorbed":forbidden,"final_rejection_correct":final}
def _write(p,v): p.write_text(json.dumps(v,ensure_ascii=False,indent=2)+"\n",encoding="utf-8")
def run(args):
cases=load_cases(args.cases); args.output.mkdir(parents=True,exist_ok=False); _write(args.output/"gold_cases.json",{"schema_version":SCHEMA_VERSION,"cases":cases})
evals=[]; naf_calls=target_calls=technical_failures=structural_failures=0; start=time.perf_counter()
for case in cases:
d=args.output/case["case_id"].lower(); d.mkdir(); obs=case["observations"]; _write(d/"v3_style_input_observations.json",obs)
try:
source=case["negative_act_source"]
if source=="live":
np=build_negative_prompt(case); (d/"negative_act_prompt.txt").write_text(np,encoding="utf-8"); raw,nmeta=call_ollama(args.endpoint,args.model,np,args.timeout,args.num_ctx,args.num_predict); naf_calls+=1; (d/"negative_act_raw_response.txt").write_text(raw+"\n",encoding="utf-8"); negative=parse_model_json(raw)
_write(d/"negative_act_ollama_metadata.json",nmeta); _write(d/"negative_act_source.json",{"kind":"live_call"})
else:
sd=args.negative_act_artifacts/source.lower(); accepted=json.loads((sd/"v3_style_input_observations.json").read_text());
if accepted!=obs: raise DerivationValidationError(f"{source} observations do not exactly match")
negative=json.loads((sd/"parsed_semantic_classification.json").read_text()); _write(d/"negative_act_source.json",{"kind":"accepted_artifact_reuse","case_id":source,"path":str(sd)})
validate_classification(negative,obs); _write(d/"negative_act_classification.json",negative)
tp=build_target_prompt(case); (d/"target_prompt.txt").write_text(tp,encoding="utf-8"); traw,tmeta=call_ollama(args.endpoint,args.model,tp,args.timeout,args.num_ctx,args.num_predict); target_calls+=1; (d/"target_raw_response.txt").write_text(traw+"\n",encoding="utf-8"); _write(d/"target_ollama_metadata.json",tmeta); target=parse_model_json(traw); _write(d/"target_recognition.json",target); validate_target(target,obs)
derivation=derive(obs,negative,target); _write(d/"deterministic_gate_results.json",derivation["gates"]); _write(d/"final_derived_result.json",derivation["derived_result"]); ev=evaluate(case,negative,target,derivation)
_write(d/"structural_validation.json",{"valid":True})
except Exception as exc:
structural_failures+=1; ev={"case_id":case["case_id"],"classification":"FAIL","technical_or_validation_failure":str(exc)}; _write(d/"structural_validation.json",{"valid":False,"error":str(exc)})
_write(d/"evaluation.json",ev); evals.append(ev)
summary={"experiment":"controlled_rejection_v1","model":args.model,"negative_act_llm_call_count":naf_calls,"reused_negative_act_count":len(cases)-naf_calls,"target_resolution_llm_call_count":target_calls,"technical_failed_call_count":technical_failures,"structural_validation_failure_count":structural_failures,"runtime_seconds":round(time.perf_counter()-start,3),"counts":{x:sum(e["classification"]==x for e in evals) for x in ["PASS","PARTIAL","FAIL"]},"evaluations":evals}; _write(args.output/"summary.json",summary); return summary
def main():
p=argparse.ArgumentParser(); p.add_argument("cases",type=Path); p.add_argument("-o","--output",type=Path,required=True); p.add_argument("--negative-act-artifacts",type=Path,default=Path("artifacts/experiments/negative_act_form_v0/20260820_qwen35_9b_single_run")); p.add_argument("--model",default=DEFAULT_MODEL); p.add_argument("--endpoint",default=DEFAULT_ENDPOINT); p.add_argument("--timeout",type=int,default=300); p.add_argument("--num-ctx",type=int,default=16384); p.add_argument("--num-predict",type=int,default=1024); args=p.parse_args(); print(json.dumps(run(args),ensure_ascii=False,indent=2)); return 0
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Target Normalization V0: reconstruct target text with fixed linkage."""
from __future__ import annotations
import argparse,json,time
from pathlib import Path
from typing import Any,Callable
import requests
from .experiment_h import DEFAULT_ENDPOINT,DEFAULT_MODEL,DerivationValidationError,OBSERVATION_KEYS
SCHEMA_VERSION="experimental-target-normalization-v0"
OUTPUT_KEYS={"candidate_observation_id","target_observation_id","normalized_target_text"}
LINK_KEYS={"candidate_observation_id","target_observation_id"}
FORBIDDEN={"negative_act_form","rejection_form","explicitly_rejected","status","decision","outcome","topic_status","responsible_person","responsibility","owner","requested_actor","action_item","protocol_category","confidence","relation","relations","graph","unresolved_issue"}
PROMPT="""The candidate and target observation IDs below are already resolved. Copy both IDs exactly; do not perform target selection. Reconstruct only the concrete POSITIVE action or option meaning targeted by the negative act. Remove rejection and negation polarity while preserving the underlying positive action. Preserve German source language, material qualifiers, purpose, location, named people, and continuation. Exclude separate positive alternatives. Do not summarize the discussion or infer rejection, decision, outcome, status, responsibility, ownership, protocol relevance, confidence, relations, graphs, or topic state. Return only the JSON-Schema-conforming object; null is not permitted.\n\nExample A observations: [{{"observation_id":"obs_a","content":"Mit Frau Beispiel arbeiten wir nicht weiter."}}]\nFixed IDs: candidate=obs_a, target=obs_a\nOutput: {{"candidate_observation_id":"obs_a","target_observation_id":"obs_a","normalized_target_text":"Zusammenarbeit mit Frau Beispiel fortsetzen"}}\n\nExample B observations: [{{"observation_id":"obs_a","content":"Für den Druckversuch steht die reale Anlage zur Diskussion."}},{{"observation_id":"obs_b","content":"Die reale Anlage nutzen wir dafür nicht."}}]\nFixed IDs: candidate=obs_b, target=obs_a\nOutput: {{"candidate_observation_id":"obs_b","target_observation_id":"obs_a","normalized_target_text":"reale Anlage für den Druckversuch nutzen"}}\n\nFixed candidate_observation_id: {candidate}\nFixed target_observation_id: {target}\nV3-style observations:\n{observations}"""
def _keys(value,required,where):
if not isinstance(value,dict): raise DerivationValidationError(f"{where} must be an object")
if set(value)!=required: raise DerivationValidationError(f"{where} keys invalid: missing={sorted(required-set(value))}, unknown={sorted(set(value)-required)}")
def _text(value,where):
if not isinstance(value,str) or not value.strip(): raise DerivationValidationError(f"{where} must be non-empty")
return value.strip()
def _reject_forbidden(value,where="output"):
if isinstance(value,dict):
bad=FORBIDDEN & set(value)
if bad: raise DerivationValidationError(f"{where} contains forbidden fields: {sorted(bad)}")
for key,item in value.items(): _reject_forbidden(item,f"{where}.{key}")
elif isinstance(value,list):
for index,item in enumerate(value): _reject_forbidden(item,f"{where}[{index}]")
def validate_observations(observations):
if not isinstance(observations,list) or not observations: raise DerivationValidationError("observations must be non-empty")
ids=set(); evidence=set()
for index,item in enumerate(observations):
_keys(item,OBSERVATION_KEYS,f"observations[{index}]"); oid=_text(item["observation_id"],"observation_id"); eid=_text(item["evidence_id"],"evidence_id")
if oid in ids or eid in evidence: raise DerivationValidationError("observation/evidence provenance must be unique")
ids.add(oid); evidence.add(eid); _text(item["content"],"content"); _text(item["speaker"],"speaker")
return ids
def validate_linkage(case):
ids=validate_observations(case["observations"]); linkage=case["fixed_linkage"]; _keys(linkage,LINK_KEYS,"fixed_linkage")
for field in LINK_KEYS:
if _text(linkage[field],field) not in ids: raise DerivationValidationError(f"{field} is unknown")
return linkage
def output_schema(case):
link=validate_linkage(case)
return {"type":"object","additionalProperties":False,"required":["candidate_observation_id","target_observation_id","normalized_target_text"],"properties":{"candidate_observation_id":{"const":link["candidate_observation_id"]},"target_observation_id":{"const":link["target_observation_id"]},"normalized_target_text":{"type":"string","minLength":1}}}
def build_prompt(case):
link=validate_linkage(case)
return PROMPT.format(candidate=link["candidate_observation_id"],target=link["target_observation_id"],observations=json.dumps(case["observations"],ensure_ascii=False,indent=2))
def validate_output(data,case):
_reject_forbidden(data); _keys(data,OUTPUT_KEYS,"output"); link=validate_linkage(case)
if data["candidate_observation_id"]!=link["candidate_observation_id"]: raise DerivationValidationError("candidate ID changed")
if data["target_observation_id"]!=link["target_observation_id"]: raise DerivationValidationError("target ID changed")
_text(data["normalized_target_text"],"normalized_target_text"); return data
def build_payload(model,prompt,schema,num_ctx,num_predict): return {"model":model,"prompt":prompt,"think":False,"stream":False,"format":schema,"options":{"temperature":0,"num_ctx":num_ctx,"num_predict":num_predict}}
def call_schema(endpoint,model,prompt,schema,timeout,num_ctx,num_predict):
started=time.perf_counter(); response=requests.post(endpoint,json=build_payload(model,prompt,schema,num_ctx,num_predict),timeout=timeout); elapsed=time.perf_counter()-started; response.raise_for_status(); body=response.json(); raw=body.get("response")
if not isinstance(raw,str) or not raw.strip(): raise ValueError("Ollama returned no usable response")
return raw.strip(),{"model":body.get("model",model),"elapsed_seconds":round(elapsed,3),"total_duration_ns":body.get("total_duration"),"prompt_eval_count":body.get("prompt_eval_count"),"eval_count":body.get("eval_count"),"configuration":{"temperature":0,"think":False,"format":"json_schema_object","num_ctx":num_ctx,"num_predict":num_predict,"retries":0}}
def _concepts(text,groups):
folded=text.casefold(); return all(any(alias.casefold() in folded for alias in group) for group in groups)
def evaluate(case,output):
validate_output(output,case); expected=case["expected"]; text=output["normalized_target_text"]; folded=text.casefold(); action=_concepts(text,expected["action_concepts"]); scope=_concepts(text,expected["material_concepts"]); forbidden=[x for x in expected["forbidden_concepts"] if x.casefold() in folded]; positive=not any(x in forbidden for x in ("nicht","beenden")); german=any(x.casefold() in folded for x in expected["german_markers"]); alternative=not any(x.casefold() in folded for x in ("technikum","stattdessen")); strengthening=False
label="PASS" if positive and action and scope and german and alternative and not forbidden and not strengthening else "FAIL"
return {"case_id":case["case_id"],"classification":label,"expected_normalized_target_text":expected["normalized_target_text"],"actual_normalized_target_text":text,"positive_polarity_correct":positive,"action_semantics_preserved":action,"material_scope_preserved":scope,"source_language_preserved":german,"separate_alternative_excluded":alternative,"forbidden_semantics_present":forbidden,"unsupported_strengthening":strengthening,"normative_leakage":False,"candidate_id_unchanged":True,"target_id_unchanged":True}
def load_cases(path):
data=json.loads(path.read_text(encoding="utf-8")); _keys(data,{"schema_version","cases"},"fixture")
if data["schema_version"]!=SCHEMA_VERSION: raise DerivationValidationError("wrong schema version")
for case in data["cases"]: validate_linkage(case)
return data["cases"]
def _write(path,value): path.write_text(json.dumps(value,ensure_ascii=False,indent=2)+"\n",encoding="utf-8")
def run(args,caller:Callable=call_schema):
cases=load_cases(args.cases); args.output.mkdir(parents=True,exist_ok=False); _write(args.output/"gold_cases.json",{"schema_version":SCHEMA_VERSION,"cases":cases}); evaluations=[]; calls=failures=0; started=time.perf_counter()
for case in cases:
folder=args.output/case["case_id"].lower(); folder.mkdir(); _write(folder/"v3_style_input_observations.json",case["observations"]); _write(folder/"fixed_linkage.json",case["fixed_linkage"]); schema=output_schema(case); _write(folder/"ollama_json_schema.json",schema); prompt=build_prompt(case); (folder/"prompt.txt").write_text(prompt,encoding="utf-8")
try:
raw,metadata=caller(args.endpoint,args.model,prompt,schema,args.timeout,args.num_ctx,args.num_predict); calls+=1; (folder/"raw_model_response.txt").write_text(raw+"\n",encoding="utf-8"); _write(folder/"ollama_metadata.json",metadata); parsed=json.loads(raw); _write(folder/"parsed_response.json",parsed); validate_output(parsed,case); _write(folder/"structural_validation.json",{"valid":True}); _write(folder/"normalized_target_result.json",{"normalized_target_text":parsed["normalized_target_text"]}); evaluation=evaluate(case,parsed)
except Exception as exc:
failures+=1; _write(folder/"structural_validation.json",{"valid":False,"error":str(exc)}); evaluation={"case_id":case["case_id"],"classification":"FAIL","error":str(exc),"normative_leakage":False}
_write(folder/"evaluation.json",evaluation); evaluations.append(evaluation)
summary={"experiment":"target_normalization_v0","model":args.model,"llm_call_count":calls,"structural_validation_failure_count":failures,"runtime_seconds":round(time.perf_counter()-started,3),"counts":{x:sum(e["classification"]==x for e in evaluations) for x in ["PASS","PARTIAL","FAIL"]},"evaluations":evaluations}; _write(args.output/"summary.json",summary); return summary
def main():
p=argparse.ArgumentParser(); p.add_argument("cases",type=Path); p.add_argument("-o","--output",type=Path,required=True); p.add_argument("--model",default=DEFAULT_MODEL); p.add_argument("--endpoint",default=DEFAULT_ENDPOINT); p.add_argument("--timeout",type=int,default=300); p.add_argument("--num-ctx",type=int,default=16384); p.add_argument("--num-predict",type=int,default=1024); print(json.dumps(run(p.parse_args()),ensure_ascii=False,indent=2)); return 0
@@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""Isolated local target-resolution experiment; performs no rejection derivation."""
from __future__ import annotations
import argparse, json, time
from pathlib import Path
from typing import Any, Callable
from .experiment_h import DEFAULT_ENDPOINT, DEFAULT_MODEL, DerivationValidationError, OBSERVATION_KEYS, call_ollama
from .experiment_negative_act import validate_classification
SCHEMA_VERSION="experimental-target-resolution-v0"
TARGET_KEYS={"candidate_observation_id","target_observation_id","normalized_target_text"}
FORBIDDEN={"negative_act_form","rejection_form","explicitly_rejected","status","decision","outcome","topic_status","closed","responsible_person","responsibility","owner","requested_actor","action_item","protocol_category","confidence","relation","relations","graph","unresolved_issue"}
PROMPT="""Resolve and normalize only the concrete action or option referred to by the candidate negative act. Candidate: {candidate}. Strategy: {instruction} Choose only a supplied observation ID. If no unique local target exists, use JSON null for both target fields. Preserve German source meaning, continuation, purpose, location, and other material scope. Do not translate Anlage as asset. Ignore separate positive alternatives. Do not output negative-act form, rejection, status, decision, outcome, responsibility, topic closure, protocol concepts, confidence, relations, or graphs. Return exactly this JSON object with no additional fields: {{"candidate_observation_id":"{candidate}","target_observation_id":"observation ID or null","normalized_target_text":"concise positive action in German or null"}}\nObservations:\n{observations}"""
def _keys(v:Any, required:set[str], where:str):
if not isinstance(v,dict): raise DerivationValidationError(f"{where} must be an object")
if set(v)!=required: raise DerivationValidationError(f"{where} keys invalid: missing={sorted(required-set(v))}, unknown={sorted(set(v)-required)}")
def _text(v:Any, where:str):
if not isinstance(v,str) or not v.strip(): raise DerivationValidationError(f"{where} must be non-empty")
return v.strip()
def _reject_forbidden(v:Any, where="output"):
if isinstance(v,dict):
bad=FORBIDDEN & set(v)
if bad: raise DerivationValidationError(f"{where} contains forbidden fields: {sorted(bad)}")
for k,x in v.items(): _reject_forbidden(x,f"{where}.{k}")
elif isinstance(v,list):
for i,x in enumerate(v): _reject_forbidden(x,f"{where}[{i}]")
def validate_observations(obs):
if not isinstance(obs,list) or not obs: raise DerivationValidationError("observations must be non-empty")
ids=set(); evidence=set()
for i,o in enumerate(obs):
_keys(o,OBSERVATION_KEYS,f"observations[{i}]"); oid=_text(o["observation_id"],"observation_id"); eid=_text(o["evidence_id"],"evidence_id")
if oid in ids or eid in evidence: raise DerivationValidationError("observation/evidence provenance must be unique")
ids.add(oid); evidence.add(eid); _text(o["content"],"content"); _text(o["speaker"],"speaker")
return ids
def eligibility(negative,obs):
validate_classification(negative,obs)
eligible=negative["negative_act_form"]=="explicit_non_pursuit"
return {"eligible_for_target_resolution":eligible,"reason":None if eligible else "negative_act_form_not_explicit_non_pursuit"}
def validate_target(data,obs,candidate):
ids=validate_observations(obs); _reject_forbidden(data); _keys(data,TARGET_KEYS,"target output")
if _text(data["candidate_observation_id"],"candidate_observation_id")!=candidate: raise DerivationValidationError("candidate observation mismatch")
if candidate not in ids: raise DerivationValidationError("unknown candidate observation")
target=data["target_observation_id"]; normalized=data["normalized_target_text"]
if target is None:
if normalized is not None: raise DerivationValidationError("null target requires null text")
else:
target=_text(target,"target_observation_id")
if target not in ids: raise DerivationValidationError("unknown target observation")
if [o["observation_id"] for o in obs].index(target)>[o["observation_id"] for o in obs].index(candidate): raise DerivationValidationError("target must not occur after candidate")
_text(normalized,"normalized_target_text")
return data
def build_prompt(case):
gate=eligibility(case["negative_act"],case["observations"])
if not gate["eligible_for_target_resolution"]: raise DerivationValidationError("ineligible case must not build a target prompt")
candidate=case["negative_act"]["observation_id"]
instruction=(f"The target linkage is deterministically fixed to {candidate}; output that exact target ID and only normalize its positive action meaning." if case["strategy"]=="self_contained" else "Resolve the unique preceding local observation that supplies the referenced action.")
return PROMPT.format(candidate=candidate,instruction=instruction,observations=json.dumps(case["observations"],ensure_ascii=False,indent=2))
def _concepts(text,groups):
folded=(text or "").casefold(); return all(any(alias.casefold() in folded for alias in group) for group in groups)
def evaluate(case,gate,called,target):
e=case["expected"]; eligible=gate["eligible_for_target_resolution"]==e["eligible"]; call_ok=called==e["eligible"]
if not e["eligible"]:
label="PASS" if eligible and call_ok and target is None else "FAIL"
return {"case_id":case["case_id"],"classification":label,"negative_act_form":case["negative_act"]["negative_act_form"],"eligible":gate["eligible_for_target_resolution"],"target_resolution_call_made":called,"expected_target_observation_id":None,"actual_target_observation_id":None,"normalized_target_text":None,"material_scope_preserved":True,"alternative_isolation":True,"normative_leakage":False}
text=target["normalized_target_text"]; target_ok=target["target_observation_id"]==e["target_observation_id"]; concepts=_concepts(text,e["concepts"]); material=_concepts(text,e["material_concepts"]); isolated=not any(x.casefold() in (text or "").casefold() for x in e["forbidden_concepts"])
label="PASS" if eligible and call_ok and target_ok and concepts and material and isolated else ("PARTIAL" if eligible and call_ok and target_ok and material and isolated else "FAIL")
return {"case_id":case["case_id"],"classification":label,"negative_act_form":case["negative_act"]["negative_act_form"],"eligible":gate["eligible_for_target_resolution"],"target_resolution_call_made":called,"expected_target_observation_id":e["target_observation_id"],"actual_target_observation_id":target["target_observation_id"],"normalized_target_text":text,"normalized_action_correct":concepts,"material_scope_preserved":material,"alternative_isolation":isolated,"normative_leakage":False}
def load_cases(path):
data=json.loads(path.read_text(encoding="utf-8")); _keys(data,{"schema_version","cases"},"fixture")
if data["schema_version"]!=SCHEMA_VERSION: raise DerivationValidationError("unexpected schema version")
for case in data["cases"]: validate_observations(case["observations"]); validate_classification(case["negative_act"],case["observations"])
return data["cases"]
def _write(path,value): path.write_text(json.dumps(value,ensure_ascii=False,indent=2)+"\n",encoding="utf-8")
def run(args, resolver:Callable=call_ollama):
cases=load_cases(args.cases); args.output.mkdir(parents=True,exist_ok=False); _write(args.output/"gold_cases.json",{"schema_version":SCHEMA_VERSION,"cases":cases})
evaluations=[]; calls=technical_failures=structural_failures=0; started=time.perf_counter()
for case in cases:
folder=args.output/case["case_id"].lower(); folder.mkdir(); _write(folder/"v3_style_input_observations.json",case["observations"]); _write(folder/"negative_act_form.json",case["negative_act"])
gate=eligibility(case["negative_act"],case["observations"]); _write(folder/"eligibility.json",gate)
if not gate["eligible_for_target_resolution"]:
skipped={"call_made":False,"reason":gate["reason"]}; _write(folder/"target_resolution_skipped.json",skipped); ev=evaluate(case,gate,False,None)
else:
prompt=build_prompt(case); (folder/"prompt.txt").write_text(prompt,encoding="utf-8")
try:
raw,meta=resolver(args.endpoint,args.model,prompt,args.timeout,args.num_ctx,args.num_predict); calls+=1; (folder/"raw_model_response.txt").write_text(raw+"\n",encoding="utf-8"); _write(folder/"ollama_metadata.json",meta); parsed=json.loads(raw); _write(folder/"parsed_target_resolution.json",parsed); validate_target(parsed,case["observations"],case["negative_act"]["observation_id"]); _write(folder/"structural_validation.json",{"valid":True}); ev=evaluate(case,gate,True,parsed)
except Exception as exc:
structural_failures+=1; _write(folder/"structural_validation.json",{"valid":False,"error":str(exc)}); ev={"case_id":case["case_id"],"classification":"FAIL","negative_act_form":case["negative_act"]["negative_act_form"],"eligible":True,"target_resolution_call_made":True,"error":str(exc)}
_write(folder/"evaluation.json",ev); evaluations.append(ev)
summary={"experiment":"target_resolution_v0","model":args.model,"target_resolution_llm_call_count":calls,"technical_failed_call_count":technical_failures,"structural_validation_failure_count":structural_failures,"runtime_seconds":round(time.perf_counter()-started,3),"counts":{x:sum(e["classification"]==x for e in evaluations) for x in ["PASS","PARTIAL","FAIL"]},"evaluations":evaluations}; _write(args.output/"summary.json",summary); return summary
def main():
p=argparse.ArgumentParser(); p.add_argument("cases",type=Path); p.add_argument("-o","--output",type=Path,required=True); p.add_argument("--model",default=DEFAULT_MODEL); p.add_argument("--endpoint",default=DEFAULT_ENDPOINT); p.add_argument("--timeout",type=int,default=300); p.add_argument("--num-ctx",type=int,default=16384); p.add_argument("--num-predict",type=int,default=1024); print(json.dumps(run(p.parse_args()),ensure_ascii=False,indent=2)); return 0
@@ -0,0 +1,107 @@
#!/usr/bin/env python3
"""Target Resolution V1 diagnostic: linkage and normalization only."""
from __future__ import annotations
import argparse,json,time
from pathlib import Path
from typing import Any,Callable
import requests
from .experiment_h import DEFAULT_ENDPOINT,DEFAULT_MODEL,DerivationValidationError,OBSERVATION_KEYS
from .experiment_negative_act import validate_classification
SCHEMA_VERSION="experimental-target-resolution-v1-diagnostic"
SELF_KEYS={"candidate_observation_id","normalized_target_text"}; PAIRED_KEYS={"candidate_observation_id","target_observation_id","normalized_target_text"}
FORBIDDEN={"negative_act_form","rejection_form","explicitly_rejected","status","decision","outcome","responsible_person","responsibility","owner","requested_actor","action_item","protocol_category","confidence","relation","relations","graph","topic_status","closed","unresolved_issue"}
SELF_PROMPT="""Normalize only the concrete positive action meaning in the self-contained candidate observation. The target linkage is already deterministic and is not your task. Preserve German, collaboration, named people, and continuation meaning. Do not output a target ID, rejection, status, decision, outcome, responsibility, ownership, protocol concepts, confidence, relations, graphs, or topic closure. Return only the schema-conforming object.\nCandidate observation ID: {candidate}\nObservation:\n{observations}"""
PAIRED_PROMPT="""Resolve and normalize only the concrete local action or option referred to by the candidate negative act. Choose exactly one listed allowed target observation ID, or use JSON null only when no unique local target exists. Never return the string \"null\". Preserve German source language and all material purpose/location scope. Do not absorb a separate positive alternative. Do not output negative-act form, rejection, status, decision, outcome, responsibility, ownership, protocol concepts, confidence, relations, graphs, or topic closure.\nAllowed target observation IDs:\n{allowed}\nConcrete positive typed example:\n{{"candidate_observation_id":"obs_2","target_observation_id":"obs_1","normalized_target_text":"externe Lösung weiterverfolgen"}}\nActual JSON-null example:\n{{"candidate_observation_id":"obs_2","target_observation_id":null,"normalized_target_text":null}}\nReturn only the schema-conforming object.\nCandidate observation ID: {candidate}\nObservations:\n{observations}"""
def _keys(value,required,where):
if not isinstance(value,dict): raise DerivationValidationError(f"{where} must be an object")
if set(value)!=required: raise DerivationValidationError(f"{where} keys invalid: missing={sorted(required-set(value))}, unknown={sorted(set(value)-required)}")
def _text(value,where):
if not isinstance(value,str) or not value.strip(): raise DerivationValidationError(f"{where} must be non-empty")
return value.strip()
def _forbidden(value,where="output"):
if isinstance(value,dict):
bad=FORBIDDEN & set(value)
if bad: raise DerivationValidationError(f"{where} contains forbidden fields: {sorted(bad)}")
for key,item in value.items(): _forbidden(item,f"{where}.{key}")
elif isinstance(value,list):
for index,item in enumerate(value): _forbidden(item,f"{where}[{index}]")
def validate_observations(obs):
if not isinstance(obs,list) or not obs: raise DerivationValidationError("observations must be non-empty")
ids=[]; evidence=set()
for index,item in enumerate(obs):
_keys(item,OBSERVATION_KEYS,f"observations[{index}]"); oid=_text(item["observation_id"],"observation_id"); eid=_text(item["evidence_id"],"evidence_id")
if oid in ids or eid in evidence: raise DerivationValidationError("observation/evidence provenance must be unique")
ids.append(oid); evidence.add(eid); _text(item["content"],"content"); _text(item["speaker"],"speaker")
return ids
def allowed_ids(case): return validate_observations(case["observations"])
def deterministic_self_link(case):
if case["strategy"]!="self_contained": raise DerivationValidationError("self-linkage requires self-contained strategy")
ids=validate_observations(case["observations"]); candidate=case["negative_act"]["observation_id"]
if candidate not in ids: raise DerivationValidationError("unknown candidate")
return {"linkage_source":"deterministic","candidate_observation_id":candidate,"target_observation_id":candidate}
def output_schema(case):
candidate=case["negative_act"]["observation_id"]
if case["strategy"]=="self_contained":
return {"type":"object","additionalProperties":False,"required":["candidate_observation_id","normalized_target_text"],"properties":{"candidate_observation_id":{"const":candidate},"normalized_target_text":{"type":"string","minLength":1}}}
ids=allowed_ids(case)
return {"type":"object","additionalProperties":False,"required":["candidate_observation_id","target_observation_id","normalized_target_text"],"properties":{"candidate_observation_id":{"const":candidate},"target_observation_id":{"enum":ids+[None]},"normalized_target_text":{"type":["string","null"]}},"allOf":[{"if":{"properties":{"target_observation_id":{"type":"null"}}},"then":{"properties":{"normalized_target_text":{"type":"null"}}},"else":{"properties":{"normalized_target_text":{"type":"string","minLength":1}}}}]}
def build_prompt(case):
validate_classification(case["negative_act"],case["observations"]); candidate=case["negative_act"]["observation_id"]
if case["strategy"]=="self_contained": return SELF_PROMPT.format(candidate=candidate,observations=json.dumps(case["observations"],ensure_ascii=False,indent=2))
return PAIRED_PROMPT.format(candidate=candidate,allowed=json.dumps(allowed_ids(case),ensure_ascii=False),observations=json.dumps(case["observations"],ensure_ascii=False,indent=2))
def validate_semantic_output(data,case):
_forbidden(data); candidate=case["negative_act"]["observation_id"]
if case["strategy"]=="self_contained":
_keys(data,SELF_KEYS,"self output")
if data["candidate_observation_id"]!=candidate: raise DerivationValidationError("candidate mismatch")
_text(data["normalized_target_text"],"normalized_target_text")
else:
_keys(data,PAIRED_KEYS,"paired output")
if data["candidate_observation_id"]!=candidate: raise DerivationValidationError("candidate mismatch")
target=data["target_observation_id"]
if target=="null": raise DerivationValidationError('string "null" is forbidden')
if target is None:
if data["normalized_target_text"] is not None: raise DerivationValidationError("null target requires null text")
else:
if target not in allowed_ids(case): raise DerivationValidationError("target is not an allowed ID")
ids=allowed_ids(case)
if ids.index(target)>ids.index(candidate): raise DerivationValidationError("target must not occur after candidate")
_text(data["normalized_target_text"],"normalized_target_text")
return data
def combine(case,semantic):
validate_semantic_output(semantic,case)
if case["strategy"]=="self_contained":
link=deterministic_self_link(case); return {**link,"normalized_target_text":semantic["normalized_target_text"]}
return {"linkage_source":"llm","candidate_observation_id":semantic["candidate_observation_id"],"target_observation_id":semantic["target_observation_id"],"normalized_target_text":semantic["normalized_target_text"]}
def build_payload(model,prompt,schema,num_ctx,num_predict):
return {"model":model,"prompt":prompt,"think":False,"stream":False,"format":schema,"options":{"temperature":0,"num_ctx":num_ctx,"num_predict":num_predict}}
def call_schema(endpoint,model,prompt,schema,timeout,num_ctx,num_predict):
started=time.perf_counter(); response=requests.post(endpoint,json=build_payload(model,prompt,schema,num_ctx,num_predict),timeout=timeout); elapsed=time.perf_counter()-started; response.raise_for_status(); body=response.json(); raw=body.get("response")
if not isinstance(raw,str) or not raw.strip(): raise ValueError("Ollama returned no usable response")
meta={"model":body.get("model",model),"elapsed_seconds":round(elapsed,3),"total_duration_ns":body.get("total_duration"),"prompt_eval_count":body.get("prompt_eval_count"),"eval_count":body.get("eval_count"),"configuration":{"temperature":0,"think":False,"format":"json_schema_object","num_ctx":num_ctx,"num_predict":num_predict,"retries":0}}
return raw.strip(),meta
def _concepts(text,groups):
folded=(text or "").casefold(); return all(any(x.casefold() in folded for x in group) for group in groups)
def evaluate(case,semantic,combined):
expected=case["expected"]; text=combined["normalized_target_text"]; target=combined["target_observation_id"]; concepts=_concepts(text,expected["concepts"]); material=_concepts(text,expected["material_concepts"]); isolated=not any(x.casefold() in (text or "").casefold() for x in expected["forbidden_concepts"]); recurrence=semantic.get("target_observation_id")=="null"; correct=target==expected["target_observation_id"]
label="PASS" if correct and concepts and material and isolated and not recurrence else ("PARTIAL" if correct and material and isolated and not recurrence else "FAIL")
return {"case_id":case["case_id"],"classification":label,"strategy":case["strategy"],"target_id_decision_source":combined["linkage_source"],"expected_target_observation_id":expected["target_observation_id"],"actual_target_observation_id":target,"normalized_target_text":text,"continuation_or_action_preserved":concepts,"material_scope_preserved":material,"alternative_isolated":isolated,"schema_valid":True,"string_null_recurrence":recurrence,"normative_leakage":False}
def load_cases(path):
data=json.loads(path.read_text(encoding="utf-8")); _keys(data,{"schema_version","cases"},"fixture")
if data["schema_version"]!=SCHEMA_VERSION: raise DerivationValidationError("wrong schema version")
return data["cases"]
def _write(path,value): path.write_text(json.dumps(value,ensure_ascii=False,indent=2)+"\n",encoding="utf-8")
def run(args,caller:Callable=call_schema):
cases=load_cases(args.cases); args.output.mkdir(parents=True,exist_ok=False); _write(args.output/"gold_cases.json",{"schema_version":SCHEMA_VERSION,"cases":cases}); evaluations=[]; calls=failures=0; started=time.perf_counter()
for case in cases:
folder=args.output/case["case_id"].lower(); folder.mkdir(); _write(folder/"v3_style_input_observations.json",case["observations"]); _write(folder/"negative_act_form.json",case["negative_act"]); _write(folder/"eligibility.json",{"eligible_for_target_resolution":True,"reason":None}); _write(folder/"deterministic_strategy.json",{"strategy":case["strategy"],"target_id_decision_source":"deterministic" if case["strategy"]=="self_contained" else "llm"}); _write(folder/"allowed_target_ids.json",allowed_ids(case)); schema=output_schema(case); _write(folder/"ollama_json_schema.json",schema); prompt=build_prompt(case); (folder/"prompt.txt").write_text(prompt,encoding="utf-8")
try:
raw,meta=caller(args.endpoint,args.model,prompt,schema,args.timeout,args.num_ctx,args.num_predict); calls+=1; (folder/"raw_model_response.txt").write_text(raw+"\n",encoding="utf-8"); _write(folder/"ollama_metadata.json",meta); semantic=json.loads(raw); _write(folder/"parsed_semantic_output.json",semantic); validate_semantic_output(semantic,case); combined=combine(case,semantic); _write(folder/"structural_validation.json",{"valid":True}); _write(folder/"deterministic_linkage_result.json",{k:combined[k] for k in ("linkage_source","candidate_observation_id","target_observation_id")}); _write(folder/"normalized_target_result.json",{"normalized_target_text":combined["normalized_target_text"]}); evaluation=evaluate(case,semantic,combined)
except Exception as exc:
failures+=1; _write(folder/"structural_validation.json",{"valid":False,"error":str(exc)}); evaluation={"case_id":case["case_id"],"classification":"FAIL","strategy":case["strategy"],"schema_valid":False,"error":str(exc)}
_write(folder/"evaluation.json",evaluation); evaluations.append(evaluation)
summary={"experiment":"target_resolution_v1_diagnostic","model":args.model,"llm_call_count":calls,"structural_validation_failure_count":failures,"runtime_seconds":round(time.perf_counter()-started,3),"counts":{x:sum(e["classification"]==x for e in evaluations) for x in ["PASS","PARTIAL","FAIL"]},"evaluations":evaluations}; _write(args.output/"summary.json",summary); return summary
def main():
p=argparse.ArgumentParser(); p.add_argument("cases",type=Path); p.add_argument("-o","--output",type=Path,required=True); p.add_argument("--model",default=DEFAULT_MODEL); p.add_argument("--endpoint",default=DEFAULT_ENDPOINT); p.add_argument("--timeout",type=int,default=300); p.add_argument("--num-ctx",type=int,default=16384); p.add_argument("--num-predict",type=int,default=1024); print(json.dumps(run(p.parse_args()),ensure_ascii=False,indent=2)); return 0
+20
View File
@@ -0,0 +1,20 @@
"""Optional speaker diarization and transcript alignment."""
from src.meeting_lab.diarization.alignment import align_transcript, write_diarized_transcript
from src.meeting_lab.diarization.backend import (
DEFAULT_MODEL,
DiarizationError,
DiarizationResult,
diarize_audio,
select_device,
)
__all__ = [
"DEFAULT_MODEL",
"DiarizationError",
"DiarizationResult",
"align_transcript",
"diarize_audio",
"select_device",
"write_diarized_transcript",
]
+129
View File
@@ -0,0 +1,129 @@
"""Deterministic Whisper-segment alignment to anonymous diarization turns."""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
class AlignmentError(ValueError):
"""Raised when transcript or diarization inputs are malformed."""
def _number(value: Any, description: str) -> float:
if not isinstance(value, (int, float)) or isinstance(value, bool):
raise AlignmentError(f"{description} must be a number.")
return float(value)
def _validated_turns(turns: list[dict[str, Any]]) -> list[dict[str, Any]]:
validated = []
for index, turn in enumerate(turns):
if not isinstance(turn, dict):
raise AlignmentError(f"Diarization turn {index} must be an object.")
start = _number(turn.get("start"), f"Diarization turn {index} start")
end = _number(turn.get("end"), f"Diarization turn {index} end")
speaker = turn.get("speaker_id", turn.get("speaker"))
if end < start:
raise AlignmentError(f"Diarization turn {index} ends before it starts.")
if not isinstance(speaker, str) or not speaker.startswith("SPEAKER_"):
raise AlignmentError(
f"Diarization turn {index} must have an anonymous SPEAKER_ label."
)
validated.append({"start": start, "end": end, "speaker_id": speaker})
return validated
def align_transcript(
transcript: dict[str, Any], exclusive_turns: list[dict[str, Any]]
) -> dict[str, Any]:
"""Return a derived transcript using maximum exclusive-turn overlap per segment."""
if not isinstance(transcript, dict) or not isinstance(transcript.get("segments"), list):
raise AlignmentError("Whisper transcript must contain a 'segments' list.")
turns = _validated_turns(exclusive_turns)
aligned_segments: list[dict[str, Any]] = []
for index, source in enumerate(transcript["segments"]):
if not isinstance(source, dict):
raise AlignmentError(f"Transcript segment {index} must be an object.")
start = _number(source.get("start"), f"Transcript segment {index} start")
end = _number(source.get("end"), f"Transcript segment {index} end")
if end < start:
raise AlignmentError(f"Transcript segment {index} ends before it starts.")
overlap_by_speaker: dict[str, float] = {}
for turn in turns:
overlap = max(0.0, min(end, turn["end"]) - max(start, turn["start"]))
if overlap:
speaker = turn["speaker_id"]
overlap_by_speaker[speaker] = overlap_by_speaker.get(speaker, 0.0) + overlap
speaker_id = None
overlap_seconds = 0.0
if overlap_by_speaker:
speaker_id, overlap_seconds = min(
overlap_by_speaker.items(), key=lambda item: (-item[1], item[0])
)
duration = end - start
aligned = dict(source)
aligned.update(
{
"speaker_id": speaker_id,
"speaker_overlap_seconds": round(overlap_seconds, 6),
"speaker_overlap_ratio": round(
overlap_seconds / duration if duration > 0 else 0.0, 6
),
}
)
aligned_segments.append(aligned)
return {
"text": diarized_transcript_text(aligned_segments, include_end=True),
"segments": aligned_segments,
"speaker_labels_anonymous": True,
"alignment_source": "exclusive_diarization",
}
def _timestamp(seconds: float) -> str:
milliseconds = int(round(seconds * 1000))
hours, remainder = divmod(milliseconds, 3_600_000)
minutes, remainder = divmod(remainder, 60_000)
secs, millis = divmod(remainder, 1000)
return f"{hours:02d}:{minutes:02d}:{secs:02d}.{millis:03d}"
def diarized_transcript_text(
segments: list[dict[str, Any]], *, include_end: bool = True
) -> str:
lines = []
for segment in segments:
start = _timestamp(float(segment["start"]))
end = _timestamp(float(segment["end"]))
speaker = segment.get("speaker_id") or "SPEAKER_UNASSIGNED"
timestamp = f"[{start} - {end}]" if include_end else f"[{start}]"
lines.append(f"{timestamp} {speaker}: {str(segment.get('text', '')).strip()}")
return "\n".join(lines) + ("\n" if lines else "")
def write_diarized_transcript(
transcript_path: Path,
exclusive_turns_path: Path,
output_dir: Path,
) -> tuple[Path, Path]:
"""Read source artifacts and write a separate speaker-aware transcript pair."""
transcript = json.loads(Path(transcript_path).read_text(encoding="utf-8-sig"))
turns = json.loads(Path(exclusive_turns_path).read_text(encoding="utf-8"))
if not isinstance(turns, list):
raise AlignmentError("Exclusive diarization turns must contain a JSON list.")
derived = align_transcript(transcript, turns)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
json_path = output_dir / "transcript_diarized.json"
text_path = output_dir / "transcript_diarized.txt"
json_path.write_text(
json.dumps(derived, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
)
text_path.write_text(
diarized_transcript_text(derived["segments"], include_end=True), encoding="utf-8"
)
return json_path, text_path
+314
View File
@@ -0,0 +1,314 @@
"""pyannote Community-1 backend with native and isolated-container runtimes."""
from __future__ import annotations
import importlib.metadata
import json
import os
import subprocess
import time
import wave
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Literal, Sequence
DEFAULT_MODEL = "pyannote/speaker-diarization-community-1"
PYANNOTE_VERSION = "4.0.7"
DeviceMode = Literal["auto", "gpu", "cpu"]
RuntimeMode = Literal["native", "container"]
class DiarizationError(RuntimeError):
"""Raised when diarization configuration or execution fails."""
@dataclass(frozen=True)
class DiarizationResult:
output_dir: Path
metadata_path: Path
ordinary_rttm: Path
exclusive_rttm: Path
turns_json: Path
exclusive_turns_json: Path
metadata: dict[str, Any]
def _host_uid() -> int:
getter = getattr(os, "getuid", None)
if getter is None:
raise DiarizationError("Container diarization requires host UID discovery.")
return int(getter())
def _host_gid() -> int:
getter = getattr(os, "getgid", None)
if getter is None:
raise DiarizationError("Container diarization requires host GID discovery.")
return int(getter())
def select_device(mode: DeviceMode, torch_module: Any) -> tuple[Any, str | None]:
"""Resolve CPU/GPU without depending on the GPU vendor."""
if mode == "cpu":
return torch_module.device("cpu"), None
if mode not in ("auto", "gpu"):
raise DiarizationError(f"Unsupported diarization device mode: {mode}")
try:
available = bool(torch_module.cuda.is_available())
if available:
name = str(torch_module.cuda.get_device_name(0))
probe = torch_module.zeros(1, device="cuda")
del probe
return torch_module.device("cuda"), name
except Exception as exc:
if mode == "gpu":
raise DiarizationError(f"Requested PyTorch GPU is not usable: {exc}") from exc
if mode == "gpu":
raise DiarizationError("Requested PyTorch GPU is unavailable.")
return torch_module.device("cpu"), None
def _load_pcm_wave(audio_path: Path, torch_module: Any) -> tuple[Any, int, float, dict[str, Any]]:
try:
with wave.open(str(audio_path), "rb") as source:
channels = source.getnchannels()
sample_rate = source.getframerate()
sample_width = source.getsampwidth()
frame_count = source.getnframes()
pcm = bytearray(source.readframes(frame_count))
except (OSError, wave.Error) as exc:
raise DiarizationError(f"Cannot read PCM WAV input {audio_path}: {exc}") from exc
if channels != 1 or sample_rate != 16000 or sample_width != 2:
raise DiarizationError(
"Diarization currently requires mono 16 kHz signed 16-bit PCM WAV; "
f"got channels={channels}, sample_rate={sample_rate}, sample_width={sample_width}."
)
waveform = torch_module.frombuffer(pcm, dtype=torch_module.int16).to(
torch_module.float32
)
waveform = (waveform / 32768.0).reshape(channels, frame_count)
duration = frame_count / sample_rate
validation = {
"waveform_dtype": str(waveform.dtype),
"waveform_shape": list(waveform.shape),
"sample_rate": sample_rate,
"sample_count": frame_count,
"duration_seconds": duration,
"min_sample_value": waveform.min().item(),
"max_sample_value": waveform.max().item(),
"audio_loading": "python_wave_pcm16",
}
return waveform, sample_rate, duration, validation
def _turns(annotation: Any) -> list[dict[str, Any]]:
return [
{
"start": segment.start,
"end": segment.end,
"speaker_id": speaker,
}
for segment, _track, speaker in annotation.itertracks(yield_label=True)
]
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def _result_from_output(output_dir: Path) -> DiarizationResult:
metadata_path = output_dir / "metadata.json"
try:
metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise DiarizationError(f"Cannot read diarization metadata: {exc}") from exc
return DiarizationResult(
output_dir=output_dir,
metadata_path=metadata_path,
ordinary_rttm=output_dir / "diarization.rttm",
exclusive_rttm=output_dir / "exclusive_diarization.rttm",
turns_json=output_dir / "turns.json",
exclusive_turns_json=output_dir / "exclusive_turns.json",
metadata=metadata,
)
def _require_writable_output(output_dir: Path) -> None:
unwritable = [
path
for path in (output_dir, *output_dir.rglob("*"))
if not os.access(path, os.W_OK)
]
if unwritable:
rendered = ", ".join(str(path) for path in unwritable[:3])
if len(unwritable) > 3:
rendered += f", and {len(unwritable) - 3} more"
raise DiarizationError(
f"Container diarization artifacts are not writable by the host user: {rendered}"
)
def run_native_pyannote(
audio_path: Path,
output_dir: Path,
device_mode: DeviceMode,
*,
model: str = DEFAULT_MODEL,
) -> DiarizationResult:
"""Run one local pyannote inference using an in-memory waveform mapping."""
try:
import torch
from pyannote.audio import Pipeline
except ImportError as exc:
raise DiarizationError(
f"Native diarization requires pyannote.audio=={PYANNOTE_VERSION} and PyTorch."
) from exc
token = os.environ.get("HF_TOKEN")
if not token:
raise DiarizationError("HF_TOKEN is required for the pyannote model.")
output_dir.mkdir(parents=True, exist_ok=True)
waveform, sample_rate, duration, audio_metadata = _load_pcm_wave(audio_path, torch)
device, device_name = select_device(device_mode, torch)
try:
pipeline = Pipeline.from_pretrained(model, token=token)
pipeline.to(device)
started = time.perf_counter()
output = pipeline(
{
"waveform": waveform,
"sample_rate": sample_rate,
"uri": audio_path.stem,
}
)
runtime = time.perf_counter() - started
except Exception as exc:
raise DiarizationError(f"pyannote diarization failed: {type(exc).__name__}: {exc}") from exc
ordinary = getattr(output, "speaker_diarization", output)
exclusive = getattr(output, "exclusive_speaker_diarization", None)
if exclusive is None:
raise DiarizationError("Community-1 did not return exclusive diarization.")
ordinary_turns = _turns(ordinary)
exclusive_turns = _turns(exclusive)
with (output_dir / "diarization.rttm").open("w", encoding="utf-8") as handle:
ordinary.write_rttm(handle)
with (output_dir / "exclusive_diarization.rttm").open(
"w", encoding="utf-8"
) as handle:
exclusive.write_rttm(handle)
_write_json(output_dir / "turns.json", ordinary_turns)
_write_json(output_dir / "exclusive_turns.json", exclusive_turns)
speakers = sorted({turn["speaker_id"] for turn in ordinary_turns})
actual_device = str(device)
metadata = {
"backend": "pyannote.audio",
"model": model,
"pyannote_version": importlib.metadata.version("pyannote.audio"),
"torch_version": torch.__version__,
"hip_version": getattr(torch.version, "hip", None),
"cuda_version": getattr(torch.version, "cuda", None),
"runtime_adapter": "native",
"requested_device_mode": device_mode,
"actual_device": actual_device,
"device_name": device_name if actual_device == "cuda" else None,
"audio_duration_seconds": duration,
"runtime_seconds": runtime,
"rtf": runtime / duration,
"speaker_count": len(speakers),
"speaker_labels": speakers,
"turn_count": len(ordinary_turns),
"exclusive_turn_count": len(exclusive_turns),
"audio": audio_metadata,
"credentials_persisted": False,
"output_files": {
"ordinary_rttm": "diarization.rttm",
"exclusive_rttm": "exclusive_diarization.rttm",
"turns": "turns.json",
"exclusive_turns": "exclusive_turns.json",
},
}
_write_json(output_dir / "metadata.json", metadata)
return _result_from_output(output_dir)
def run_container_pyannote(
audio_path: Path,
output_dir: Path,
device_mode: DeviceMode,
*,
image: str,
container_args: Sequence[str] = (),
runner: Callable[..., subprocess.CompletedProcess[str]] = subprocess.run,
uid_getter: Callable[[], int] = _host_uid,
gid_getter: Callable[[], int] = _host_gid,
) -> DiarizationResult:
"""Run the same backend in an explicitly configured disposable container."""
if not image.strip():
raise DiarizationError("A diarization container image is required.")
output_dir.mkdir(parents=True, exist_ok=True)
host_uid = uid_getter()
host_gid = gid_getter()
if host_uid < 0 or host_gid < 0:
raise DiarizationError("Host UID and GID must be non-negative integers.")
command = [
"docker", "run", "--rm", "--ipc=host", "--shm-size=8g", "-e", "HF_TOKEN",
*container_args,
"-v", f"{Path(audio_path).resolve()}:/input/audio.wav:ro",
"-v", f"{output_dir.resolve()}:/output:rw",
"-v", f"{Path(__file__).resolve().parents[3]}:/work/meeting-lab:ro",
"-w", "/work/meeting-lab",
image,
"/bin/bash", "-lc",
(
"inference_status=0; "
f"python -m pip install --disable-pip-version-check pyannote.audio=={PYANNOTE_VERSION} "
"> /output/pip-install.log 2>&1 && "
"python -m src.meeting_lab.diarization.container_entry "
f"/input/audio.wav /output --device {device_mode} || inference_status=$?; "
f"chown -R {host_uid}:{host_gid} /output || exit $?; "
"chmod -R u+rwX /output || exit $?; "
'exit "$inference_status"'
),
]
try:
completed = runner(command, check=False, capture_output=True, text=True)
except OSError as exc:
raise DiarizationError(f"Could not start diarization container: {exc}") from exc
(output_dir / "container_stdout.log").write_text(completed.stdout, encoding="utf-8")
(output_dir / "container_stderr.log").write_text(completed.stderr, encoding="utf-8")
if completed.returncode != 0:
detail = completed.stderr.strip() or completed.stdout.strip() or "no diagnostic output"
raise DiarizationError(
f"Diarization container failed with exit code {completed.returncode}: {detail}"
)
_require_writable_output(output_dir)
result = _result_from_output(output_dir)
metadata = dict(result.metadata)
metadata["runtime_adapter"] = "container"
_write_json(result.metadata_path, metadata)
return _result_from_output(output_dir)
def diarize_audio(
audio_path: Path,
output_dir: Path,
device_mode: DeviceMode,
*,
runtime: RuntimeMode = "native",
container_image: str | None = None,
container_args: Sequence[str] = (),
) -> DiarizationResult:
if runtime == "native":
return run_native_pyannote(audio_path, output_dir, device_mode)
if runtime == "container":
return run_container_pyannote(
audio_path,
output_dir,
device_mode,
image=container_image or "",
container_args=container_args,
)
raise DiarizationError(f"Unsupported diarization runtime: {runtime}")
@@ -0,0 +1,22 @@
"""Internal entry point for the isolated pyannote container adapter."""
from __future__ import annotations
import argparse
from pathlib import Path
from src.meeting_lab.diarization.backend import run_native_pyannote
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("audio", type=Path)
parser.add_argument("output", type=Path)
parser.add_argument("--device", choices=("auto", "gpu", "cpu"), required=True)
args = parser.parse_args()
run_native_pyannote(args.audio, args.output, args.device)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+95
View File
@@ -0,0 +1,95 @@
"""Minimal Ollama client behavior used by the direct protocol MVP."""
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Any
import requests
DEFAULT_ENDPOINT = "http://127.0.0.1:11434"
class OllamaError(RuntimeError):
"""Raised when Ollama cannot safely complete the requested operation."""
@dataclass(frozen=True)
class OllamaGeneration:
raw_response: dict[str, Any]
text: str
client_wall_time_seconds: float
def ollama_base_url(endpoint: str) -> str:
endpoint = endpoint.rstrip("/")
return endpoint.rsplit("/api/", 1)[0] if "/api/" in endpoint else endpoint
def generate_url(endpoint: str) -> str:
return f"{ollama_base_url(endpoint)}/api/generate"
def require_model(endpoint: str, model: str, timeout: int = 10) -> dict[str, Any]:
base_url = ollama_base_url(endpoint)
try:
response = requests.get(f"{base_url}/api/tags", timeout=timeout)
response.raise_for_status()
data = response.json()
except (requests.RequestException, ValueError) as exc:
raise OllamaError(f"Ollama endpoint is not reachable at {base_url}: {exc}") from exc
models = data.get("models") if isinstance(data, dict) else None
if not isinstance(models, list):
raise OllamaError("Ollama /api/tags returned a malformed response.")
installed = {
item.get("name")
for item in models
if isinstance(item, dict) and isinstance(item.get("name"), str)
}
if model not in installed:
raise OllamaError(f"Requested model is not installed in Ollama: {model}")
return {"base_url": base_url, "model": model, "installed": True}
def generate_once(
endpoint: str,
model: str,
prompt: str,
*,
timeout: int,
num_ctx: int,
num_predict: int,
) -> OllamaGeneration:
payload = {
"model": model,
"prompt": prompt,
"think": False,
"stream": False,
"options": {
"temperature": 0.0,
"num_ctx": num_ctx,
"num_predict": num_predict,
},
}
started = time.perf_counter()
try:
response = requests.post(generate_url(endpoint), json=payload, timeout=timeout)
response.raise_for_status()
data = response.json()
except requests.RequestException as exc:
raise OllamaError(f"Ollama generation request failed: {exc}") from exc
except ValueError as exc:
raise OllamaError("Ollama generation response is not valid JSON.") from exc
wall_time = time.perf_counter() - started
if not isinstance(data, dict):
raise OllamaError("Ollama generation response must be a JSON object.")
text = data.get("response")
if not isinstance(text, str):
raise OllamaError("Ollama generation response has no string 'response' field.")
if not text.strip():
raise OllamaError("Ollama returned an empty protocol.")
return OllamaGeneration(data, text, wall_time)
+119 -11
View File
@@ -3,13 +3,15 @@
from __future__ import annotations from __future__ import annotations
import ast import ast
import copy
import re
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
SUPPORTED_SCHEMA_VERSIONS = {"1"} SUPPORTED_SCHEMA_VERSIONS = {"1"}
VALID_ATTENDANCE_STATUSES = {"present", "not_present", "absent"} VALID_ATTENDANCE_STATUSES = {"present", "mentioned_only"}
class MeetingContextValidationError(ValueError): class MeetingContextValidationError(ValueError):
@@ -29,6 +31,21 @@ class MeetingContext:
def meeting_id(self) -> str: def meeting_id(self) -> str:
return str(self.data["meeting"]["meeting_id"]) return str(self.data["meeting"]["meeting_id"])
@property
def speaker_mappings(self) -> dict[str, str]:
mappings = self.data.get("speaker_mappings")
return dict(mappings) if isinstance(mappings, dict) else {}
def participant_for_speaker(self, speaker_label: str) -> dict[str, Any] | None:
"""Resolve only an explicit authoritative mapping; never infer identity."""
participant_id = self.speaker_mappings.get(speaker_label)
if participant_id is None:
return None
for participant in self.data.get("participants", []):
if participant.get("participant_id") == participant_id:
return participant
return None
def provenance(self) -> dict[str, str]: def provenance(self) -> dict[str, str]:
return { return {
"meeting_id": self.meeting_id, "meeting_id": self.meeting_id,
@@ -42,8 +59,43 @@ def load_meeting_context(path: Path) -> MeetingContext:
if not isinstance(loaded, dict): if not isinstance(loaded, dict):
raise MeetingContextValidationError("Meeting Context must be a YAML object.") raise MeetingContextValidationError("Meeting Context must be a YAML object.")
validate_meeting_context(loaded) normalized = _with_attendance_defaults(loaded)
return MeetingContext(data=loaded, source_file=path) validate_meeting_context(normalized)
return MeetingContext(data=normalized, source_file=path)
def create_meeting_context(
data: dict[str, Any], *, source_file: Path = Path("<generated>")
) -> MeetingContext:
"""Validate structured data and return an immutable context boundary."""
validated = _with_attendance_defaults(data)
validate_meeting_context(validated)
return MeetingContext(data=validated, source_file=source_file)
def serialize_meeting_context_yaml(context: MeetingContext) -> str:
"""Serialize validated Meeting Context data deterministically as YAML."""
validate_meeting_context(context.data)
try:
import yaml # type: ignore[import-not-found]
except ModuleNotFoundError as exc:
raise MeetingContextValidationError(
"PyYAML is required to write Meeting Context YAML."
) from exc
return yaml.safe_dump(
context.data,
allow_unicode=True,
sort_keys=False,
default_flow_style=False,
)
def write_meeting_context(context: MeetingContext, path: Path) -> Path:
"""Persist a validated context without changing its schema or semantics."""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(serialize_meeting_context_yaml(context), encoding="utf-8")
return path
def validate_meeting_context(data: dict[str, Any]) -> None: def validate_meeting_context(data: dict[str, Any]) -> None:
@@ -56,7 +108,9 @@ def validate_meeting_context(data: dict[str, Any]) -> None:
meeting = _require_mapping(data, "meeting") meeting = _require_mapping(data, "meeting")
_require_non_empty_string(meeting, "meeting.meeting_id") _require_non_empty_string(meeting, "meeting.meeting_id")
_require_non_empty_string(meeting, "meeting.title") _require_non_empty_string(meeting, "meeting.title")
_require_non_empty_string(meeting, "meeting.language") # Legacy contexts may omit language; protocol generation defaults to German.
if "language" in meeting:
_require_non_empty_string(meeting, "meeting.language")
organization = _optional_mapping(data.get("organization"), "organization") organization = _optional_mapping(data.get("organization"), "organization")
departments = _optional_list(organization.get("departments"), "organization.departments") departments = _optional_list(organization.get("departments"), "organization.departments")
@@ -74,10 +128,26 @@ def validate_meeting_context(data: dict[str, Any]) -> None:
+ ", ".join(collisions) + ", ".join(collisions)
) )
speaker_mappings = _optional_mapping(
data.get("speaker_mappings"), "speaker_mappings"
)
for speaker_label, participant_id in speaker_mappings.items():
if not isinstance(speaker_label, str) or re.fullmatch(
r"SPEAKER_\d+", speaker_label
) is None:
raise MeetingContextValidationError(
f"Invalid diarization speaker label: {speaker_label!r}."
)
if not isinstance(participant_id, str) or participant_id not in participant_ids:
raise MeetingContextValidationError(
f"speaker_mappings.{speaker_label} references unknown participant: "
f"{participant_id!r}."
)
for index, participant in enumerate(participants): for index, participant in enumerate(participants):
item_path = f"participants[{index}]" item_path = f"participants[{index}]"
_validate_attendance(participant, item_path) status = _validate_attendance(participant, item_path, default="present")
if participant.get("attendance_status") != "present": if status != "present":
raise MeetingContextValidationError( raise MeetingContextValidationError(
f"{item_path}.attendance_status must be 'present'." f"{item_path}.attendance_status must be 'present'."
) )
@@ -85,10 +155,10 @@ def validate_meeting_context(data: dict[str, Any]) -> None:
for index, person in enumerate(mentioned_people): for index, person in enumerate(mentioned_people):
item_path = f"mentioned_people[{index}]" item_path = f"mentioned_people[{index}]"
_validate_attendance(person, item_path) status = _validate_attendance(person, item_path, default="mentioned_only")
if person.get("attendance_status") == "present": if status != "mentioned_only":
raise MeetingContextValidationError( raise MeetingContextValidationError(
f"{item_path}.attendance_status must not be 'present'." f"{item_path}.attendance_status must be 'mentioned_only'."
) )
_validate_department_reference(person, item_path, department_ids) _validate_department_reference(person, item_path, department_ids)
@@ -131,6 +201,28 @@ def render_meeting_context_for_prompt(context: MeetingContext) -> str:
for participant in participants: for participant in participants:
lines.append(_render_person_line(participant, "participant_id", departments_by_id)) lines.append(_render_person_line(participant, "participant_id", departments_by_id))
speaker_mappings = context.speaker_mappings
if speaker_mappings:
participants_by_id = {
participant["participant_id"]: participant
for participant in participants
if isinstance(participant, dict) and participant.get("participant_id")
}
lines.extend(
[
"",
"Confirmed diarization speaker mappings (authoritative):",
"- Use only these explicit mappings. Never infer identities for other speaker labels.",
"- Unmapped SPEAKER_XX labels must remain anonymous.",
]
)
for speaker_label, participant_id in sorted(speaker_mappings.items()):
participant = participants_by_id[participant_id]
lines.append(
f"- {speaker_label}: {_text(participant.get('display_name'))} "
f"(participant_id: {participant_id})"
)
mentioned_people = _optional_list(data.get("mentioned_people"), "mentioned_people") mentioned_people = _optional_list(data.get("mentioned_people"), "mentioned_people")
if mentioned_people: if mentioned_people:
lines.extend(["", "Mentioned but absent people:"]) lines.extend(["", "Mentioned but absent people:"])
@@ -245,12 +337,28 @@ def _collect_unique_ids(items: list[Any], key: str, path: str) -> set[str]:
return ids return ids
def _validate_attendance(item: dict[str, Any], path: str) -> None: def _validate_attendance(item: dict[str, Any], path: str, *, default: str) -> str:
status = item.get("attendance_status") status = item.get("attendance_status", default)
if status not in VALID_ATTENDANCE_STATUSES: if status not in VALID_ATTENDANCE_STATUSES:
raise MeetingContextValidationError( raise MeetingContextValidationError(
f"{path}.attendance_status has invalid value: {status!r}." f"{path}.attendance_status has invalid value: {status!r}."
) )
return status
def _with_attendance_defaults(data: dict[str, Any]) -> dict[str, Any]:
normalized = copy.deepcopy(data)
participants = normalized.get("participants")
if isinstance(participants, list):
for participant in participants:
if isinstance(participant, dict):
participant.setdefault("attendance_status", "present")
mentioned_people = normalized.get("mentioned_people")
if isinstance(mentioned_people, list):
for person in mentioned_people:
if isinstance(person, dict):
person.setdefault("attendance_status", "mentioned_only")
return normalized
def _validate_department_reference( def _validate_department_reference(
+17
View File
@@ -0,0 +1,17 @@
"""Reusable orchestration APIs for Meeting Lab applications and CLIs."""
from src.meeting_lab.orchestration.mvp import (
DEFAULT_OUTPUT_ROOT,
MvpMeetingConfig,
MvpRunResult,
create_unique_run_dir,
run_mvp_meeting,
)
__all__ = [
"DEFAULT_OUTPUT_ROOT",
"MvpMeetingConfig",
"MvpRunResult",
"create_unique_run_dir",
"run_mvp_meeting",
]
+467
View File
@@ -0,0 +1,467 @@
"""Reusable audio-to-direct-protocol MVP orchestration."""
from __future__ import annotations
import json
import re
import shutil
import sys
import time
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from src.meeting_lab.audio import prepare_audio
from src.meeting_lab.diarization import (
DEFAULT_MODEL as DEFAULT_DIARIZATION_MODEL,
diarize_audio,
write_diarized_transcript,
)
from src.meeting_lab.llm.ollama import DEFAULT_ENDPOINT
from src.meeting_lab.models.meeting_context import (
MeetingContext,
create_meeting_context,
load_meeting_context,
validate_meeting_context,
write_meeting_context,
)
from src.meeting_lab.progress import ProgressEvent, ProgressSink, ProgressStatus
from src.meeting_lab.protocol.generate_direct_protocol import (
DEFAULT_MODEL,
DEFAULT_NUM_CTX,
DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
DirectProtocolResult,
generate_direct_protocol,
load_compact_transcript,
)
from src.meeting_lab.transcription.whisper import transcribe_audio
DEFAULT_OUTPUT_ROOT = Path("meeting_data/runs")
ContextInput = MeetingContext | Mapping[str, Any]
@dataclass(frozen=True)
class MvpMeetingConfig:
audio_file: Path
whisper_model: Path
whisper_executable: str = "whisper-cli"
ffmpeg_executable: str = "ffmpeg"
audio_normalization: bool = True
context_file: Path | None = None
output_root: Path = DEFAULT_OUTPUT_ROOT
language: str = "de"
threads: str | int = "auto"
model: str = DEFAULT_MODEL
ollama_endpoint: str = DEFAULT_ENDPOINT
protocol_num_ctx: int = DEFAULT_NUM_CTX
protocol_safe_input_token_budget: int = DEFAULT_SAFE_INPUT_TOKEN_BUDGET
diarization: str = "off"
diarization_runtime: str = "native"
diarization_container_image: str | None = None
diarization_container_args: Sequence[str] = ()
@dataclass(frozen=True)
class MvpRunResult:
exit_code: int
run_dir: Path | None
protocol_path: Path | None
def regenerate_mvp_protocol(
run_dir: Path,
*,
meeting_context: ContextInput,
model: str = DEFAULT_MODEL,
ollama_endpoint: str = DEFAULT_ENDPOINT,
protocol_num_ctx: int = DEFAULT_NUM_CTX,
protocol_safe_input_token_budget: int = DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
progress_sink: ProgressSink | None = None,
) -> MvpRunResult:
"""Regenerate only protocol artifacts from an existing completed run."""
started = time.perf_counter()
run_dir = Path(run_dir)
context = _effective_context(meeting_context)
if context is None:
raise ValueError("Meeting Context is required for protocol regeneration.")
if protocol_num_ctx <= 0:
raise ValueError("Protocol Ollama context size must be positive.")
if protocol_safe_input_token_budget <= 0:
raise ValueError("Protocol safe input token budget must be positive.")
diarized_transcript = run_dir / "diarization" / "transcript_diarized.json"
plain_transcript = run_dir / "transcript" / "transcript.json"
transcript_path = (
diarized_transcript if diarized_transcript.is_file() else plain_transcript
)
if not transcript_path.is_file():
raise FileNotFoundError(
f"Existing run has no protocol transcript artifact: {run_dir}"
)
context_path = run_dir / "context" / "meeting_context.yaml"
write_meeting_context(context, context_path)
_emit(progress_sink, "protocol_generation", "started", started)
try:
result = generate_direct_protocol(
transcript_path,
context_path,
model=model,
endpoint=ollama_endpoint,
num_ctx=protocol_num_ctx,
safe_input_token_budget=protocol_safe_input_token_budget,
)
protocol_path = _persist_protocol(run_dir, result)
except Exception as exc:
_emit(
progress_sink,
"failed",
"failed",
started,
message=f"protocol_generation: {type(exc).__name__}: {exc}",
)
raise
_emit(progress_sink, "protocol_generation", "completed", started)
_emit(progress_sink, "completed", "completed", started)
return MvpRunResult(0, run_dir, protocol_path)
def create_unique_run_dir(
output_root: Path,
meeting_name: str,
now: Callable[[], datetime] = datetime.now,
) -> Path:
safe_name = re.sub(r"[^A-Za-z0-9_.-]+", "_", meeting_name).strip("._-") or "meeting"
base = output_root / f"{safe_name}_{now().strftime('%Y%m%d_%H%M%S')}"
candidate = base
suffix = 1
while candidate.exists():
candidate = output_root / f"{base.name}_{suffix:02d}"
suffix += 1
candidate.mkdir(parents=True)
return candidate
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def _effective_context(value: ContextInput | None) -> MeetingContext | None:
if value is None:
return None
if isinstance(value, MeetingContext):
validate_meeting_context(value.data)
return value
if isinstance(value, Mapping):
return create_meeting_context(dict(value), source_file=Path("<programmatic>"))
raise TypeError("meeting_context must be MeetingContext, mapping, or None.")
def _validate_inputs(
config: MvpMeetingConfig, meeting_context: MeetingContext | None
) -> None:
if not config.audio_file.is_file():
raise FileNotFoundError(f"Audio file does not exist: {config.audio_file}")
if not config.whisper_model.is_file():
raise FileNotFoundError(f"Whisper model does not exist: {config.whisper_model}")
if meeting_context is not None and config.context_file is not None:
raise ValueError("Use either a context file or a programmatic Meeting Context, not both.")
if meeting_context is not None:
validate_meeting_context(meeting_context.data)
elif config.context_file is not None:
if not config.context_file.is_file():
raise FileNotFoundError(
f"Meeting Context file does not exist: {config.context_file}"
)
load_meeting_context(config.context_file)
if config.diarization not in ("off", "auto", "gpu", "cpu"):
raise ValueError(f"Unsupported diarization mode: {config.diarization}")
if config.diarization_runtime not in ("native", "container"):
raise ValueError(
f"Unsupported diarization runtime: {config.diarization_runtime}"
)
if (
config.diarization != "off"
and config.diarization_runtime == "container"
and not config.diarization_container_image
):
raise ValueError("A diarization container image is required.")
if config.protocol_safe_input_token_budget <= 0:
raise ValueError("Protocol safe input token budget must be positive.")
if config.protocol_num_ctx <= 0:
raise ValueError("Protocol Ollama context size must be positive.")
def _emit(
sink: ProgressSink | None,
stage: str,
status: ProgressStatus,
overall_started: float,
*,
message: str | None = None,
) -> None:
if sink is not None:
sink(
ProgressEvent(
stage=stage,
status=status,
elapsed_seconds=time.perf_counter() - overall_started,
message=message,
)
)
def _persist_protocol(run_dir: Path, result: DirectProtocolResult) -> Path:
protocol_dir = run_dir / "protocol"
protocol_dir.mkdir(exist_ok=True)
(protocol_dir / "exact_prompt.txt").write_text(result.exact_prompt, encoding="utf-8")
_write_json(protocol_dir / "raw_response.json", result.raw_response)
_write_json(protocol_dir / "runtime_metadata.json", result.runtime_metadata)
transcript_input = getattr(result, "transcript_input", None)
if transcript_input is not None:
(protocol_dir / "transcript_input.txt").write_text(
transcript_input, encoding="utf-8"
)
protocol_path = run_dir / "protocol.md"
protocol_path.write_text(result.protocol_text, encoding="utf-8")
return protocol_path
def run_mvp_meeting(
config: MvpMeetingConfig,
*,
meeting_context: ContextInput | None = None,
progress_sink: ProgressSink | None = None,
) -> MvpRunResult:
"""Run the existing MVP directly, without subprocess or GUI dependencies."""
overall_started = time.perf_counter()
validation_started = time.perf_counter()
_emit(progress_sink, "preparing", "started", overall_started)
try:
effective_context = _effective_context(meeting_context)
_validate_inputs(config, effective_context)
except Exception as exc:
_emit(
progress_sink,
"failed",
"failed",
overall_started,
message=f"preparing: {type(exc).__name__}: {exc}",
)
print(f"Error: {type(exc).__name__}: {exc}", file=sys.stderr)
return MvpRunResult(2, None, None)
validation_runtime = time.perf_counter() - validation_started
run_dir = create_unique_run_dir(config.output_root, config.audio_file.stem)
timestamp = datetime.now().astimezone().isoformat(timespec="seconds")
transcript_path = run_dir / "transcript" / "transcript.json"
protocol_path = run_dir / "protocol.md"
stage_runtimes: dict[str, float | None] = {
"validation": round(validation_runtime, 3),
"setup": None,
"audio_preparation": None,
"whisper": None,
"transcript_validation": None,
"protocol": None,
}
if config.diarization != "off":
stage_runtimes["diarization"] = None
stage_runtimes["diarization_alignment"] = None
metadata: dict[str, Any] = {
"run_id": run_dir.name,
"timestamp": timestamp,
"input_audio": str(config.audio_file.resolve()),
"audio_preparation": None,
"transcript_output": str(transcript_path.resolve()),
"protocol_output": str(protocol_path.resolve()),
"whisper_model": str(config.whisper_model.resolve()),
"model": config.model,
"ollama_endpoint": config.ollama_endpoint,
"status": "running",
"stage_runtimes_seconds": stage_runtimes,
"total_runtime_seconds": None,
"failure": None,
"diarization": {
"enabled": config.diarization != "off",
"backend": "pyannote.audio" if config.diarization != "off" else None,
"model": DEFAULT_DIARIZATION_MODEL if config.diarization != "off" else None,
"requested_device_mode": config.diarization,
"runtime": config.diarization_runtime if config.diarization != "off" else None,
"metadata_path": None,
"transcript_diarized": None,
},
}
current_stage = "preparing"
stage_started = time.perf_counter()
try:
audio_dir = run_dir / "audio"
transcript_dir = run_dir / "transcript"
context_dir = run_dir / "context"
protocol_dir = run_dir / "protocol"
audio_dir.mkdir()
transcript_dir.mkdir()
context_dir.mkdir()
protocol_dir.mkdir()
_write_json(
audio_dir / "input_manifest.json",
{
"source_file": str(config.audio_file.resolve()),
"filename": config.audio_file.name,
"size_bytes": config.audio_file.stat().st_size,
},
)
preparation_started = time.perf_counter()
current_stage = "audio_preparation"
stage_started = preparation_started
prepared_audio = prepare_audio(
config.audio_file,
audio_dir / "prepared.wav",
ffmpeg_executable=config.ffmpeg_executable,
normalization_enabled=config.audio_normalization,
)
stage_runtimes["audio_preparation"] = round(
time.perf_counter() - preparation_started, 3
)
current_stage = "preparing"
preparation_metadata = prepared_audio.metadata()
metadata["audio_preparation"] = preparation_metadata
_write_json(audio_dir / "preparation_metadata.json", preparation_metadata)
_write_json(
audio_dir / "input_manifest.json",
{
"source_file": str(config.audio_file.resolve()),
"filename": config.audio_file.name,
"size_bytes": config.audio_file.stat().st_size,
"format": config.audio_file.suffix.lower().removeprefix("."),
"prepared_audio": preparation_metadata,
},
)
preserved_context: Path | None = None
if effective_context is not None:
preserved_context = context_dir / "meeting_context.yaml"
write_meeting_context(effective_context, preserved_context)
elif config.context_file is not None:
preserved_context = context_dir / "meeting_context.yaml"
shutil.copy2(config.context_file, preserved_context)
stage_runtimes["setup"] = round(time.perf_counter() - stage_started, 3)
_emit(progress_sink, "preparing", "completed", overall_started)
current_stage = "transcription"
stage_started = time.perf_counter()
_emit(progress_sink, "transcription", "started", overall_started)
transcription = transcribe_audio(
prepared_audio.prepared_path,
config.whisper_model,
transcript_dir,
config.language,
executable=config.whisper_executable,
threads=config.threads,
)
stage_runtimes["whisper"] = round(time.perf_counter() - stage_started, 3)
_emit(progress_sink, "transcription", "completed", overall_started)
stage_started = time.perf_counter()
load_compact_transcript(transcription.transcript_json)
stage_runtimes["transcript_validation"] = round(
time.perf_counter() - stage_started, 3
)
protocol_transcript = transcription.transcript_json
if config.diarization != "off":
current_stage = "diarization"
stage_started = time.perf_counter()
_emit(progress_sink, "diarization", "started", overall_started)
diarization_dir = run_dir / "diarization"
diarization = diarize_audio(
prepared_audio.prepared_path,
diarization_dir,
config.diarization,
runtime=config.diarization_runtime,
container_image=config.diarization_container_image,
container_args=config.diarization_container_args,
)
stage_runtimes["diarization"] = round(
time.perf_counter() - stage_started, 3
)
metadata["diarization"].update(
{
"actual_device": diarization.metadata.get("actual_device"),
"device_name": diarization.metadata.get("device_name"),
"runtime_seconds": diarization.metadata.get("runtime_seconds"),
"speaker_count": diarization.metadata.get("speaker_count"),
"metadata_path": str(diarization.metadata_path.resolve()),
}
)
stage_started = time.perf_counter()
protocol_transcript, diarized_text = write_diarized_transcript(
transcription.transcript_json,
diarization.exclusive_turns_json,
diarization_dir,
)
load_compact_transcript(protocol_transcript)
stage_runtimes["diarization_alignment"] = round(
time.perf_counter() - stage_started, 3
)
metadata["diarization"].update(
{
"transcript_diarized": str(protocol_transcript.resolve()),
"transcript_diarized_text": str(diarized_text.resolve()),
}
)
_emit(progress_sink, "diarization", "completed", overall_started)
current_stage = "protocol_generation"
stage_started = time.perf_counter()
_emit(progress_sink, "protocol_generation", "started", overall_started)
result = generate_direct_protocol(
protocol_transcript,
preserved_context,
model=config.model,
endpoint=config.ollama_endpoint,
num_ctx=config.protocol_num_ctx,
safe_input_token_budget=config.protocol_safe_input_token_budget,
)
stage_runtimes["protocol"] = round(time.perf_counter() - stage_started, 3)
protocol_path = _persist_protocol(run_dir, result)
_emit(progress_sink, "protocol_generation", "completed", overall_started)
metadata["status"] = "completed"
_emit(progress_sink, "completed", "completed", overall_started)
except Exception as exc:
metadata_stage = {
"preparing": "setup",
"audio_preparation": "audio_preparation",
"transcription": "whisper",
"diarization": "diarization",
"protocol_generation": "protocol",
}.get(current_stage, current_stage)
runtime_key = metadata_stage
if runtime_key in stage_runtimes and stage_runtimes[runtime_key] is None:
stage_runtimes[runtime_key] = round(time.perf_counter() - stage_started, 3)
metadata["status"] = "failed"
metadata["failure"] = {
"stage": metadata_stage,
"type": type(exc).__name__,
"message": str(exc),
}
protocol_path = None
_emit(
progress_sink,
"failed",
"failed",
overall_started,
message=f"{current_stage}: {type(exc).__name__}: {exc}",
)
print(f"Error: {type(exc).__name__}: {exc}", file=sys.stderr)
finally:
metadata["total_runtime_seconds"] = round(time.perf_counter() - overall_started, 3)
_write_json(run_dir / "run_metadata.json", metadata)
exit_code = 0 if metadata["status"] == "completed" else 2
return MvpRunResult(exit_code, run_dir, protocol_path)
+21
View File
@@ -0,0 +1,21 @@
"""Small observer boundary for long-running Meeting Lab operations."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Literal
ProgressStatus = Literal["started", "completed", "failed"]
@dataclass(frozen=True)
class ProgressEvent:
stage: str
status: ProgressStatus
elapsed_seconds: float
progress: float | None = None
message: str | None = None
ProgressSink = Callable[[ProgressEvent], None]
@@ -0,0 +1,36 @@
"""Prompt construction for the direct transcript-to-protocol MVP."""
from __future__ import annotations
DIRECT_PROTOCOL_INSTRUCTION = """Erstelle aus dem vollständigen Transkript und dem Meeting-Kontext ein vollständiges, strukturiertes und professionelles internes Besprechungsprotokoll.
Das Protokoll muss themenorientiert sein, nicht chronologisch und nicht nach technischen Kategorien gegliedert. Beginne mit # Meeting Protocol. Verwende für jedes kohärente Thema eine Überschrift ## <Thema> und darunter eine strukturierte Synthese der Diskussion. Bewahre relevante Diskussionsverläufe, unterschiedliche Positionen, offene Punkte und Entscheidungsgrundlagen. Dokumentiere die wesentlichen Inhalte nachvollziehbar und fasse Themenblöcke so zusammen, dass auch Personen, die nicht am Meeting teilgenommen haben, den Kontext und die Entwicklung der Diskussion verstehen können. Nenne Entscheidungen oder abgestimmte Positionen nur, wenn sie tatsächlich belegt sind. Führe Maßnahmen nur auf, wenn eine konkrete zukünftige Handlung gestützt ist; nenne verantwortliche Personen und Fristen ausschließlich bei expliziter Zuweisung, Annahme oder Bestätigung im Transkript. Vorschläge, Einwände, Möglichkeiten und vorläufige Ideen sind keine Entscheidungen oder Verpflichtungen. Bewahre relevante Einschränkungen und ungelöste Meinungsverschiedenheiten. Nenne offene Punkte nur, wenn sie wirklich offen bleiben. Nicht jedes Thema benötigt Entscheidungen, Maßnahmen oder offene Punkte.
Erzeuge keine reine Wiedergabe des Transkripts und verlängere das Protokoll nicht unnötig durch Wiederholungen. Synthetisiere zusammengehörige Aussagen, entferne Füllwörter und Gesprächsrauschen und erfinde keine Fakten, Entscheidungen, Zustimmungen, Verantwortlichen oder Fristen. Gib kein JSON, keine internen Labels und keine Analyse oder Denkprotokolle aus. Das Ergebnis soll als Markdown-Protokoll nach geringfügiger menschlicher Redaktion intern versendbar sein. Eine kompakte themenübergreifende Maßnahmenliste am Ende ist optional, wenn sie nützlich und vollständig belegt ist."""
COMPACT_DIARIZED_PROTOCOL_INSTRUCTION = """Erstelle aus dem vollständigen Transkript und Meeting-Kontext ein vollständiges, professionelles internes Besprechungsprotokoll. Das Transkript ist in aufeinanderfolgende anonyme Sprecherblöcke gegliedert.
Beginne mit # Meeting Protocol. Gliedere themenorientiert mit ## <Thema> und synthetisiere je Thema den relevanten Diskussionsverlauf, Kontext, unterschiedliche Positionen, Entscheidungsgrundlagen, Einschränkungen und ungelöste Meinungsverschiedenheiten so, dass Dritte ihn nachvollziehen können. Nenne Entscheidungen nur bei Beleg. Nenne Maßnahmen, Verantwortliche und Fristen nur bei expliziter Zuweisung, Annahme oder Bestätigung; Vorschläge sind keine Verpflichtungen.
Entferne nur Wiederholungen, Füllwörter und Gesprächsrauschen. Erfinde keine Fakten oder Identitäten. Gib kein JSON, keine Sprecherlabels und kein Denkprotokoll aus. Eine belegte themenübergreifende Maßnahmenliste am Ende ist optional."""
MAPPED_SPEAKER_ATTRIBUTION_INSTRUCTION = """Nutze die autoritativen SPEAKER_XX-zu-Teilnehmer-Zuordnungen im Meeting-Kontext, um ausdrücklich belegte Aussagen, Positionen, Entscheidungen, Zuweisungen und angenommene persönliche Verpflichtungen namentlich zuzuordnen. Eine ausdrückliche Ich-Zusage eines zugeordneten Sprechers belegt persönliche Verantwortung. Unterscheide stets den Sprecher einer Aussage von darin nur erwähnten Personen. Leite für nicht zugeordnete Sprecher keine Identität ab und erfinde keine persönliche Verantwortung. Gib die technischen SPEAKER_XX-Bezeichnungen nicht im nutzerseitigen Protokoll aus."""
def build_direct_protocol_prompt(
transcript: str,
meeting_context: str | None = None,
*,
instruction: str = DIRECT_PROTOCOL_INSTRUCTION,
meeting_language: str = "de",
) -> str:
context = meeting_context.strip() if meeting_context else "Kein Meeting-Kontext bereitgestellt."
language = {"de": "German", "en": "English"}.get(meeting_language, meeting_language)
return (
f"{instruction}\n\n"
f"Write the meeting protocol in {language}. "
"Preserve speaker and person names exactly as supplied.\n\n"
f"MEETING-KONTEXT:\n{context}\n\n"
f"VOLLSTAENDIGES TRANSKRIPT:\n{transcript.strip()}\n"
)
@@ -0,0 +1,246 @@
"""One-call direct protocol generation from a compact Whisper transcript."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable
from src.meeting_lab.llm.ollama import (
DEFAULT_ENDPOINT,
OllamaGeneration,
generate_once,
require_model,
)
from src.meeting_lab.models.meeting_context import (
MeetingContext,
load_meeting_context,
render_meeting_context_for_prompt,
)
from src.meeting_lab.protocol.direct_protocol_prompt import (
COMPACT_DIARIZED_PROTOCOL_INSTRUCTION,
MAPPED_SPEAKER_ATTRIBUTION_INSTRUCTION,
build_direct_protocol_prompt,
)
from src.meeting_lab.protocol.transcript_input import (
TranscriptInputError,
compact_diarized_transcript,
plain_segment_transcript,
)
DEFAULT_MODEL = "qwen3.6:35B-A3B"
DEFAULT_NUM_CTX = 32768
DEFAULT_NUM_PREDICT = 8192
DEFAULT_TIMEOUT = 1800
DEFAULT_SAFE_INPUT_TOKEN_BUDGET = 29_000
ESTIMATED_UTF8_BYTES_PER_TOKEN = 4.4
class DirectProtocolError(ValueError):
"""Raised for invalid direct-protocol inputs or model output."""
@dataclass(frozen=True)
class DirectProtocolResult:
protocol_text: str
exact_prompt: str
model_metadata: dict[str, Any]
runtime_metadata: dict[str, Any]
raw_response: dict[str, Any]
transcript_input: str | None = None
@dataclass(frozen=True)
class SelectedTranscriptInput:
text: str
prompt: str
representation: str
estimated_input_tokens: int
safe_input_token_budget: int
fallback_used: bool
diarization_enabled: bool
def load_compact_transcript(path: Path) -> str:
data = _load_transcript_document(path)
text = data.get("text")
if not isinstance(text, str) or not text.strip():
raise DirectProtocolError("Transcript top-level 'text' must be a non-empty string.")
return text
def _load_transcript_document(path: Path) -> dict[str, Any]:
if not path.is_file():
raise DirectProtocolError(f"Transcript file does not exist: {path}")
try:
data = json.loads(path.read_text(encoding="utf-8-sig"))
except json.JSONDecodeError as exc:
raise DirectProtocolError(f"Transcript is not valid JSON: {path}: {exc}") from exc
if not isinstance(data, dict):
raise DirectProtocolError("Transcript JSON must contain a top-level object.")
if "text" not in data:
raise DirectProtocolError("Transcript JSON must contain top-level 'text'.")
return data
def estimate_input_tokens(prompt: str) -> int:
"""Estimate tokens without adding a model-specific tokenizer dependency."""
byte_count = len(prompt.encode("utf-8"))
return max(1, int(byte_count / ESTIMATED_UTF8_BYTES_PER_TOKEN + 0.999999))
def select_transcript_input(
transcript: dict[str, Any],
rendered_context: str | None,
*,
safe_input_token_budget: int = DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
meeting_language: str = "de",
) -> SelectedTranscriptInput:
"""Select complete prompt input without allowing silent tail truncation."""
if safe_input_token_budget <= 0:
raise DirectProtocolError("Safe protocol input token budget must be positive.")
diarization_enabled = transcript.get("speaker_labels_anonymous") is True
if diarization_enabled:
try:
compact = compact_diarized_transcript(transcript.get("segments"))
plain_text = plain_segment_transcript(transcript.get("segments"))
except TranscriptInputError as exc:
raise DirectProtocolError(str(exc)) from exc
instruction = COMPACT_DIARIZED_PROTOCOL_INSTRUCTION
if (
rendered_context
and "Confirmed diarization speaker mappings" in rendered_context
):
instruction = f"{instruction}\n\n{MAPPED_SPEAKER_ATTRIBUTION_INSTRUCTION}"
compact_prompt = build_direct_protocol_prompt(
compact.text,
rendered_context,
instruction=instruction,
meeting_language=meeting_language,
)
compact_estimate = estimate_input_tokens(compact_prompt)
if compact_estimate <= safe_input_token_budget:
return SelectedTranscriptInput(
text=compact.text,
prompt=compact_prompt,
representation="diarized_compact",
estimated_input_tokens=compact_estimate,
safe_input_token_budget=safe_input_token_budget,
fallback_used=False,
diarization_enabled=True,
)
representation = "plain_transcript_fallback"
fallback_used = True
else:
plain_text = transcript.get("text")
if not isinstance(plain_text, str) or not plain_text.strip():
raise DirectProtocolError("Transcript top-level 'text' must be a non-empty string.")
representation = "plain_transcript"
fallback_used = False
plain_prompt = build_direct_protocol_prompt(
plain_text, rendered_context, meeting_language=meeting_language
)
plain_estimate = estimate_input_tokens(plain_prompt)
if plain_estimate > safe_input_token_budget:
raise DirectProtocolError(
"Protocol prompt/input is too large for the configured safe input budget "
f"({plain_estimate} estimated tokens > {safe_input_token_budget}). "
"No LLM request was made; silent truncation is not allowed."
)
return SelectedTranscriptInput(
text=plain_text,
prompt=plain_prompt,
representation=representation,
estimated_input_tokens=plain_estimate,
safe_input_token_budget=safe_input_token_budget,
fallback_used=fallback_used,
diarization_enabled=diarization_enabled,
)
def generate_direct_protocol(
transcript_path: Path,
context_path: Path | None = None,
*,
model: str = DEFAULT_MODEL,
endpoint: str = DEFAULT_ENDPOINT,
timeout: int = DEFAULT_TIMEOUT,
num_ctx: int = DEFAULT_NUM_CTX,
num_predict: int = DEFAULT_NUM_PREDICT,
safe_input_token_budget: int = DEFAULT_SAFE_INPUT_TOKEN_BUDGET,
model_check: Callable[[str, str, int], dict[str, Any]] = require_model,
generation_call: Callable[..., OllamaGeneration] = generate_once,
) -> DirectProtocolResult:
transcript = _load_transcript_document(transcript_path)
context: MeetingContext | None = (
load_meeting_context(context_path) if context_path is not None else None
)
rendered_context = render_meeting_context_for_prompt(context) if context else None
# Older runs without a meeting language retain the historical German output.
meeting_language = (
str(context.data["meeting"].get("language") or "de") if context else "de"
)
selected = select_transcript_input(
transcript,
rendered_context,
safe_input_token_budget=safe_input_token_budget,
meeting_language=meeting_language,
)
model_metadata = model_check(endpoint, model, 10)
generation = generation_call(
endpoint,
model,
selected.prompt,
timeout=timeout,
num_ctx=num_ctx,
num_predict=num_predict,
)
data = generation.raw_response
runtime_metadata = {
"output_language": meeting_language,
"model": model,
"prompt_token_count": data.get("prompt_eval_count"),
"output_token_count": data.get("eval_count"),
"prompt_evaluation_duration_ns": data.get("prompt_eval_duration"),
"generation_duration_ns": data.get("eval_duration"),
"total_ollama_duration_ns": data.get("total_duration"),
"client_wall_time_seconds": generation.client_wall_time_seconds,
"completion_reason": data.get("done_reason"),
"done": data.get("done"),
"request_count": 1,
"temperature": 0.0,
"think": False,
"num_ctx": num_ctx,
"num_predict": num_predict,
"selected_transcript_representation": selected.representation,
"estimated_input_tokens": selected.estimated_input_tokens,
"safe_input_token_budget": selected.safe_input_token_budget,
"input_token_estimation_method": "utf8_bytes_divided_by_4.4",
"fallback_used": selected.fallback_used,
"diarization_enabled": selected.diarization_enabled,
"speaker_attribution_available": (
True
if selected.representation == "diarized_compact"
else False
if selected.representation == "plain_transcript_fallback"
else None
),
"speaker_attribution_loss_reason": (
"plain_transcript_fallback"
if selected.representation == "plain_transcript_fallback"
else None
),
"speaker_mapping_count": len(context.speaker_mappings) if context else 0,
}
return DirectProtocolResult(
protocol_text=generation.text,
exact_prompt=selected.prompt,
model_metadata=model_metadata,
runtime_metadata=runtime_metadata,
raw_response=data,
transcript_input=selected.text,
)
@@ -0,0 +1,97 @@
"""Deterministic transcript representations for one-call protocol prompts."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
class TranscriptInputError(ValueError):
"""Raised when a transcript cannot be represented without content loss."""
@dataclass(frozen=True)
class SpeakerBlock:
"""One contiguous run of transcript segments assigned to one speaker."""
speaker_id: str
segment_texts: tuple[str, ...]
@dataclass(frozen=True)
class CompactDiarizedTranscript:
"""Compact prompt text plus structural evidence of segment preservation."""
text: str
blocks: tuple[SpeakerBlock, ...]
source_segment_count: int
@property
def represented_segment_count(self) -> int:
return sum(len(block.segment_texts) for block in self.blocks)
@property
def segment_texts(self) -> tuple[str, ...]:
return tuple(text for block in self.blocks for text in block.segment_texts)
def normalize_segment_text(value: Any, index: int) -> str:
"""Normalize formatting whitespace while retaining all semantic text."""
if not isinstance(value, str):
raise TranscriptInputError(f"Transcript segment {index} text must be a string.")
return " ".join(value.split())
def compact_diarized_transcript(segments: Any) -> CompactDiarizedTranscript:
"""Group only adjacent same-speaker segments and omit repeated timestamps."""
if not isinstance(segments, list) or not segments:
raise TranscriptInputError(
"Diarized transcript must contain a non-empty 'segments' list."
)
mutable_blocks: list[tuple[str, list[str]]] = []
source_texts: list[str] = []
for index, segment in enumerate(segments):
if not isinstance(segment, dict):
raise TranscriptInputError(f"Transcript segment {index} must be an object.")
speaker = segment.get("speaker_id") or "SPEAKER_UNASSIGNED"
if not isinstance(speaker, str) or not speaker.startswith("SPEAKER_"):
raise TranscriptInputError(
f"Transcript segment {index} must use an anonymous SPEAKER_ label."
)
text = normalize_segment_text(segment.get("text"), index)
source_texts.append(text)
if mutable_blocks and mutable_blocks[-1][0] == speaker:
mutable_blocks[-1][1].append(text)
else:
mutable_blocks.append((speaker, [text]))
blocks = tuple(
SpeakerBlock(speaker_id=speaker, segment_texts=tuple(texts))
for speaker, texts in mutable_blocks
)
rendered = "\n".join(
f"{block.speaker_id}: {' '.join(block.segment_texts)}" for block in blocks
)
result = CompactDiarizedTranscript(
text=rendered + "\n",
blocks=blocks,
source_segment_count=len(segments),
)
if result.represented_segment_count != len(segments):
raise TranscriptInputError("Compact diarized transcript lost source segments.")
if result.segment_texts != tuple(source_texts):
raise TranscriptInputError("Compact diarized transcript changed segment order or text.")
return result
def plain_segment_transcript(segments: Any) -> str:
"""Reconstruct plain transcript text from every segment in source order."""
if not isinstance(segments, list) or not segments:
raise TranscriptInputError("Transcript must contain a non-empty 'segments' list.")
texts = []
for index, segment in enumerate(segments):
if not isinstance(segment, dict):
raise TranscriptInputError(f"Transcript segment {index} must be an object.")
texts.append(normalize_segment_text(segment.get("text"), index))
return " ".join(texts)
@@ -0,0 +1,5 @@
"""Audio transcription support for the direct-protocol MVP."""
from .whisper import TranscriptionError, TranscriptionResult, transcribe_audio
__all__ = ["TranscriptionError", "TranscriptionResult", "transcribe_audio"]
+290
View File
@@ -0,0 +1,290 @@
"""Isolated whisper.cpp wrapper producing Meeting Lab compact transcripts."""
from __future__ import annotations
import json
import os
import platform
import re
import shutil
import subprocess
import tempfile
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable, Sequence
BACKEND = "whisper.cpp"
RAW_FILENAME = "whisper_raw.json"
TRANSCRIPT_FILENAME = "transcript.json"
TEXT_FILENAME = "transcript.txt"
METADATA_FILENAME = "runtime_metadata.json"
DEFAULT_THREADS = "auto"
class TranscriptionError(RuntimeError):
"""Raised when parameters, Whisper execution, or output are invalid."""
@dataclass(frozen=True)
class TranscriptionResult:
output_dir: Path
raw_output: Path
transcript_json: Path
transcript_text: Path
runtime_metadata: Path
runtime_seconds: float
def _logical_cpu_count() -> int:
"""Return the available logical CPU count as a last-resort fallback."""
if hasattr(os, "sched_getaffinity"):
try:
count = len(os.sched_getaffinity(0))
if count > 0:
return count
except OSError:
pass
return os.cpu_count() or 1
def _linux_physical_core_count() -> int | None:
affinity = None
if hasattr(os, "sched_getaffinity"):
try:
affinity = os.sched_getaffinity(0)
except OSError:
pass
cores: set[tuple[str, str]] = set()
for cpu_dir in Path("/sys/devices/system/cpu").glob("cpu[0-9]*"):
try:
cpu_number = int(cpu_dir.name[3:])
if affinity is not None and cpu_number not in affinity:
continue
topology = cpu_dir / "topology"
package = (topology / "physical_package_id").read_text().strip()
core = (topology / "core_id").read_text().strip()
cores.add((package, core))
except (OSError, ValueError):
continue
return len(cores) or None
def _darwin_physical_core_count() -> int | None:
try:
completed = subprocess.run(
("sysctl", "-n", "hw.physicalcpu"),
check=False,
capture_output=True,
text=True,
)
count = int(completed.stdout.strip())
return count if completed.returncode == 0 and count > 0 else None
except (OSError, ValueError):
return None
def physical_core_count() -> int:
"""Detect physical cores where supported, falling back to available threads."""
system = platform.system()
detected = _linux_physical_core_count() if system == "Linux" else None
if system == "Darwin":
detected = _darwin_physical_core_count()
return detected or _logical_cpu_count()
def resolve_threads(
threads: str | int,
detector: Callable[[], int] = physical_core_count,
) -> int:
if isinstance(threads, bool):
raise TranscriptionError("Threads must be 'auto' or a positive integer.")
if threads == "auto":
count = detector()
else:
try:
count = int(threads)
except (TypeError, ValueError) as exc:
raise TranscriptionError("Threads must be 'auto' or a positive integer.") from exc
if count <= 0:
raise TranscriptionError("Threads must be 'auto' or a positive integer.")
return count
def _vulkan_support(raw: dict[str, Any]) -> bool | None:
system_info = raw.get("systeminfo")
if not isinstance(system_info, str) or "VULKAN" not in system_info.upper():
return None
return re.search(r"VULKAN\s*=\s*1", system_info, re.IGNORECASE) is not None
def _json_object(path: Path) -> dict[str, Any]:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise TranscriptionError(f"Cannot read Whisper JSON output {path}: {exc}") from exc
if not isinstance(data, dict):
raise TranscriptionError("Whisper JSON output must contain a top-level object.")
return data
def compact_transcript(raw: dict[str, Any]) -> dict[str, Any]:
"""Convert whisper.cpp JSON without linguistic cleanup or reordering."""
entries = raw.get("transcription")
if not isinstance(entries, list):
raise TranscriptionError("Whisper JSON output must contain a 'transcription' list.")
segments: list[dict[str, Any]] = []
for index, entry in enumerate(entries):
if not isinstance(entry, dict):
raise TranscriptionError(f"transcription[{index}] must be an object.")
offsets = entry.get("offsets")
if not isinstance(offsets, dict):
raise TranscriptionError(f"transcription[{index}].offsets must be an object.")
start_ms = offsets.get("from")
end_ms = offsets.get("to")
if not isinstance(start_ms, (int, float)) or isinstance(start_ms, bool):
raise TranscriptionError(f"transcription[{index}].offsets.from must be a number.")
if not isinstance(end_ms, (int, float)) or isinstance(end_ms, bool):
raise TranscriptionError(f"transcription[{index}].offsets.to must be a number.")
if end_ms < start_ms:
raise TranscriptionError(
f"transcription[{index}].offsets.to must be greater than or equal to offsets.from."
)
text_value = entry.get("text", "")
if not isinstance(text_value, str):
raise TranscriptionError(f"transcription[{index}].text must be a string.")
text = text_value.strip()
if text:
segments.append(
{
"id": len(segments),
"start": float(start_ms) / 1000.0,
"end": float(end_ms) / 1000.0,
"text": text,
}
)
return {"text": " ".join(item["text"] for item in segments), "segments": segments}
def _timestamp(seconds: float) -> str:
milliseconds = int(round(seconds * 1000))
hours, remainder = divmod(milliseconds, 3_600_000)
minutes, remainder = divmod(remainder, 60_000)
secs, millis = divmod(remainder, 1000)
return f"{hours:02d}:{minutes:02d}:{secs:02d}.{millis:03d}"
def transcript_text(transcript: dict[str, Any]) -> str:
lines = [
f"[{_timestamp(item['start'])} - {_timestamp(item['end'])}] {item['text']}"
for item in transcript["segments"]
]
return "\n".join(lines) + ("\n" if lines else "")
def _write_json(path: Path, value: dict[str, Any]) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def transcribe_audio(
audio_path: Path,
model_path: Path,
output_dir: Path,
language: str = "auto",
*,
executable: str = "whisper-cli",
threads: str | int = DEFAULT_THREADS,
thread_detector: Callable[[], int] = physical_core_count,
runner: Callable[..., subprocess.CompletedProcess[str]] = subprocess.run,
monotonic: Callable[[], float] = time.monotonic,
now: Callable[[], datetime] = lambda: datetime.now(timezone.utc),
) -> TranscriptionResult:
"""Run one whisper.cpp call and write raw, compact, text, and metadata outputs."""
audio_path = Path(audio_path)
model_path = Path(model_path)
output_dir = Path(output_dir)
if not audio_path.is_file():
raise TranscriptionError(f"Audio file does not exist: {audio_path}")
if not model_path.is_file():
raise TranscriptionError(f"Whisper model does not exist: {model_path}")
if not isinstance(language, str) or not language.strip():
raise TranscriptionError("Language must be a non-empty string.")
if not executable.strip():
raise TranscriptionError("Whisper executable must be a non-empty string.")
if output_dir.exists() and not output_dir.is_dir():
raise TranscriptionError(f"Output directory path is not a directory: {output_dir}")
thread_count = resolve_threads(threads, thread_detector)
output_dir.mkdir(parents=True, exist_ok=True)
raw_output = output_dir / RAW_FILENAME
started_at = now().astimezone(timezone.utc)
started = monotonic()
with tempfile.TemporaryDirectory(prefix=".whisper-", dir=output_dir) as temp_name:
temporary_prefix = Path(temp_name) / "whisper_raw"
command: Sequence[str] = (
executable,
"-m", str(model_path),
"-f", str(audio_path),
"-l", language.strip(),
"-t", str(thread_count),
"-fa",
"-oj",
"-of", str(temporary_prefix),
)
try:
completed = runner(command, check=False, capture_output=True, text=True)
except OSError as exc:
raise TranscriptionError(f"Could not start {BACKEND}: {exc}") from exc
runtime_seconds = monotonic() - started
temporary_raw = temporary_prefix.with_suffix(".json")
if temporary_raw.is_file():
shutil.copyfile(temporary_raw, raw_output)
if completed.returncode != 0:
detail = completed.stderr.strip() or completed.stdout.strip() or "no diagnostic output"
raise TranscriptionError(
f"{BACKEND} failed with exit code {completed.returncode}: {detail}"
)
if not raw_output.is_file():
raise TranscriptionError(f"{BACKEND} completed without producing JSON output.")
raw_data = _json_object(raw_output)
transcript = compact_transcript(raw_data)
transcript_json_path = output_dir / TRANSCRIPT_FILENAME
transcript_text_path = output_dir / TEXT_FILENAME
metadata_path = output_dir / METADATA_FILENAME
_write_json(transcript_json_path, transcript)
transcript_text_path.write_text(transcript_text(transcript), encoding="utf-8")
duration = max((item["end"] for item in transcript["segments"]), default=None)
metadata = {
"input_file": str(audio_path.resolve()),
"model": str(model_path.resolve()),
"backend": BACKEND,
"whisper_executable": executable,
"language": language.strip(),
"threads": thread_count,
"threads_option": str(threads),
"flash_attention": True,
"vulkan_support_detected": _vulkan_support(raw_data),
"duration_seconds": duration,
"runtime_seconds": runtime_seconds,
"timestamp": started_at.isoformat(),
"output_files": {
"whisper_raw": RAW_FILENAME,
"transcript_json": TRANSCRIPT_FILENAME,
"transcript_text": TEXT_FILENAME,
"runtime_metadata": METADATA_FILENAME,
},
}
_write_json(metadata_path, metadata)
return TranscriptionResult(
output_dir=output_dir,
raw_output=raw_output,
transcript_json=transcript_json_path,
transcript_text=transcript_text_path,
runtime_metadata=metadata_path,
runtime_seconds=runtime_seconds,
)
@@ -0,0 +1,95 @@
{
"schema_version": "experimental-collective-commitment-gold-v0",
"cases": [
{
"case_id": "CC-01",
"description": "Explicit collective commitment",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Ja, wir testen nächste Woche 20 Meter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "collective_first_person", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": true, "due": "nächste Woche"}
},
{
"case_id": "CC-02",
"description": "Individual commitment",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Ja, ich teste nächste Woche 20 Meter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "individual_first_person", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-03",
"description": "Tentative collective possibility",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten nächste Woche 20 Meter testen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "none", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-04",
"description": "Collective suggestion",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Vielleicht sollten wir nächste Woche 20 Meter testen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "none", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-05",
"description": "Impersonal necessity",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Man müsste nächste Woche 20 Meter testen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "none", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-06",
"description": "Passive future statement",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Nächste Woche werden 20 Meter getestet.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "none", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-07",
"description": "Collective rejection",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Nein, das testen wir nächste Woche nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "none", "action_concepts": [], "qualifier_concepts": []},
"expected_result": {"established": false, "due": null}
},
{
"case_id": "CC-08",
"description": "Collective commitment with qualifier",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Ja, wir testen 20 Meter, aber nur im Technikum.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "collective_first_person", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": [["nur", "only"], ["technikum", "technical facility", "technical center", "technical centre"]]},
"expected_result": {"established": true, "due": null}
},
{
"case_id": "CC-09",
"description": "Collective commitment without deadline",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Ja, wir testen 20 Meter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "collective_first_person", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": true, "due": null}
},
{
"case_id": "CC-10",
"description": "Speaker ownership trap",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Ja, wir testen nächste Woche 20 Meter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"commitment_form": "collective_first_person", "action_concepts": [["20"], ["meter", "metre"]], "qualifier_concepts": []},
"expected_result": {"established": true, "due": "nächste Woche"}
}
]
}
@@ -0,0 +1,13 @@
{
"schema_version": "experimental-controlled-rejection-v1",
"cases": [
{"case_id":"CR-01","description":"self-contained non-pursuit","negative_act_source":"NA-01","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.","speaker":"Martin","named_person":"Dr. Schlummer","addressee":null}],"expected":{"negative_act_form":"explicit_non_pursuit","candidate_observation_id":"obs_1","target_observation_id":"obs_1","action_concepts":[["Schlummer"],["Zusammenarbeit","arbeiten"],["fortsetzen","weiter"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":true}},
{"case_id":"CR-02","description":"paired non-pursuit","negative_act_source":"live","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Eine Möglichkeit wäre, die externe Lösung weiterzuverfolgen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das verfolgen wir nicht weiter.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"explicit_non_pursuit","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["externe Lösung"],["weiterverfolgen","weiter verfolgen"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":true}},
{"case_id":"CR-03","description":"personal preference","negative_act_source":"NA-03","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die reale Anlage für den Versuch nutzen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Ich würde das nicht machen.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"personal_preference","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["reale Anlage"],["Versuch"],["nutzen"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":false}},
{"case_id":"CR-04","description":"recommendation","negative_act_source":"NA-04","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die reale Anlage verwenden.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Ich würde eher davon abraten.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"recommendation","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["reale Anlage"],["verwenden","nutzen"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":false}},
{"case_id":"CR-05","description":"temporary non-action","negative_act_source":"NA-05","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die Waschstufe einbauen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir erstmal noch nicht.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"temporary_non_action","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["Waschstufe"],["einbauen"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":false}},
{"case_id":"CR-06","description":"concern","negative_act_source":"NA-06","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten das neue Material einsetzen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das wäre kritisch.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"none","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["neue Material","neues Material"],["einsetzen"]],"material_concepts":[],"forbidden_concepts":[],"explicitly_rejected":false}},
{"case_id":"CR-07","description":"scoped explicit rejection","negative_act_source":"live","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Für den Druckversuch steht die reale Anlage zur Diskussion.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Die reale Anlage nutzen wir dafür nicht.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"explicit_non_pursuit","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["Anlage"],["nutzen"]],"material_concepts":[["real"],["Druckversuch"]],"forbidden_concepts":[],"explicitly_rejected":true}},
{"case_id":"CR-08","description":"rejection plus alternative","negative_act_source":"live","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten den Versuch in der realen Anlage durchführen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir nicht; wir testen stattdessen im Technikum.","speaker":"Martin","named_person":null,"addressee":null}],"expected":{"negative_act_form":"explicit_non_pursuit","candidate_observation_id":"obs_2","target_observation_id":"obs_1","action_concepts":[["Versuch"],["durchführen"]],"material_concepts":[["real"],["Anlage"]],"forbidden_concepts":["Technikum"],"explicitly_rejected":true}}
]
}
+111
View File
@@ -0,0 +1,111 @@
{
"schema_version": "experimental-explicit-rejection-gold-v0",
"cases": [
{
"case_id": "RJ-01", "description": "Explicit collective rejection with local target",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage für den Versuch nutzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Nein, das machen wir nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "explicit_action_rejection", "rejection_observation_id": "obs_2", "target_observation_id": "obs_1", "action_concepts": [["real"], ["anlage", "plant"], ["versuch", "trial", "test"]], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": true}
},
{
"case_id": "RJ-02", "description": "Explicit non-pursuit with paired target",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Eine Möglichkeit wäre, die externe Lösung weiterzuverfolgen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das verfolgen wir nicht weiter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "explicit_action_rejection", "rejection_observation_id": "obs_2", "target_observation_id": "obs_1", "action_concepts": [["extern"], ["lösung", "solution"], ["weiter", "pursu", "continu"]], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": true}
},
{
"case_id": "RJ-03", "description": "Self-contained collaboration rejection",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.", "speaker": "Martin", "named_person": "Dr. Schlummer", "addressee": null}
],
"expected_recognition": {"rejection_form": "explicit_action_rejection", "rejection_observation_id": "obs_1", "target_observation_id": "obs_1", "action_concepts": [["schlummer"], ["arbeit", "collabor"], ["weiter", "fortsetz", "continu"]], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": true}
},
{
"case_id": "RJ-04", "description": "Personal preference",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage für den Versuch nutzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde das nicht machen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-05", "description": "Concern",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten das neue Material einsetzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das wäre kritisch.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-06", "description": "Uncertainty",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Eine Möglichkeit wäre, die Waschstufe einzubauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich weiß nicht, ob das sinnvoll ist.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-07", "description": "Negative recommendation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage verwenden.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde eher davon abraten.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-08", "description": "Deferral",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die externe Lösung einsetzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das entscheiden wir nächste Woche.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-09", "description": "Factual negation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Das Material ist nicht verfügbar.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_1", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-10", "description": "Temporary non-action",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die Waschstufe einbauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das machen wir erstmal noch nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "none", "rejection_observation_id": "obs_2", "target_observation_id": null, "action_concepts": [], "qualifier_concepts": [], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": false}
},
{
"case_id": "RJ-11", "description": "Explicit rejection with material scope",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Für den Druckversuch steht die reale Anlage zur Diskussion.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Die reale Anlage nutzen wir dafür nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "explicit_action_rejection", "rejection_observation_id": "obs_2", "target_observation_id": "obs_1", "action_concepts": [["real"], ["anlage", "plant"]], "qualifier_concepts": [["druckversuch", "dafür", "pressure test"]], "forbidden_action_concepts": []},
"expected_result": {"explicitly_rejected": true}
},
{
"case_id": "RJ-12", "description": "Rejection plus positive alternative",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten den Versuch in der realen Anlage durchführen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das machen wir nicht; wir testen stattdessen im Technikum.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected_recognition": {"rejection_form": "explicit_action_rejection", "rejection_observation_id": "obs_2", "target_observation_id": "obs_1", "action_concepts": [["versuch", "trial", "test"], ["real"], ["anlage", "plant"]], "qualifier_concepts": [["real"], ["anlage", "plant"]], "forbidden_action_concepts": ["technikum", "technical facility", "technical center", "technical centre"]},
"expected_result": {"explicitly_rejected": true}
}
]
}
@@ -0,0 +1,66 @@
{
"schema_version": "experimental-negative-act-form-gold-v0",
"cases": [
{
"case_id": "NA-01", "description": "Explicit non-pursuit",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.", "speaker": "Martin", "named_person": "Dr. Schlummer", "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "explicit_non_pursuit", "action_concepts": [["schlummer"], ["arbeit", "collabor"], ["weiter", "fortsetz", "continu"]]}
},
{
"case_id": "NA-02", "description": "Explicit non-pursuit paraphrase",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Die externe Lösung verfolgen wir nicht weiter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "explicit_non_pursuit", "action_concepts": [["extern"], ["lösung", "solution"], ["weiter", "pursu", "continu"]]}
},
{
"case_id": "NA-03", "description": "Personal preference with local context",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage für den Versuch nutzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde das nicht machen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "personal_preference", "action_concepts": [["real"], ["anlage", "plant"], ["versuch", "trial", "test"]]}
},
{
"case_id": "NA-04", "description": "Negative recommendation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage verwenden.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde eher davon abraten.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "recommendation", "action_concepts": [["real"], ["anlage", "plant"], ["verwend", "use"]]}
},
{
"case_id": "NA-05", "description": "Temporary non-action",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die Waschstufe einbauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das machen wir erstmal noch nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "temporary_non_action", "action_concepts": [["waschstufe", "washing stage"], ["einbau", "install"]]}
},
{
"case_id": "NA-06", "description": "Concern only",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten das neue Material einsetzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das wäre kritisch.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "none", "action_concepts": []}
},
{
"case_id": "NA-07", "description": "Uncertainty",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Eine Möglichkeit wäre, die Waschstufe einzubauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich weiß nicht, ob das sinnvoll ist.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "none", "action_concepts": []}
},
{
"case_id": "NA-08", "description": "Factual negation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Das Material ist nicht verfügbar.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "none", "action_concepts": []}
}
]
}
@@ -0,0 +1,9 @@
{
"schema_version":"experimental-target-normalization-v0",
"cases":[
{"case_id":"TN-01","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.","speaker":"Martin","named_person":"Dr. Schlummer","addressee":null}],"fixed_linkage":{"candidate_observation_id":"obs_1","target_observation_id":"obs_1"},"expected":{"normalized_target_text":"Zusammenarbeit mit Dr. Schlummer fortsetzen","action_concepts":[["Zusammenarbeit","arbeiten"],["Schlummer"],["fortsetzen","weiter"]],"material_concepts":[],"forbidden_concepts":["nicht","beenden"],"german_markers":["Zusammenarbeit","arbeiten","fortsetzen"]}},
{"case_id":"TN-02","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Eine Möglichkeit wäre, die externe Lösung weiterzuverfolgen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das verfolgen wir nicht weiter.","speaker":"Martin","named_person":null,"addressee":null}],"fixed_linkage":{"candidate_observation_id":"obs_2","target_observation_id":"obs_1"},"expected":{"normalized_target_text":"externe Lösung weiterverfolgen","action_concepts":[["externe Lösung"],["weiterverfolgen","weiter verfolgen"]],"material_concepts":[],"forbidden_concepts":["nicht"],"german_markers":["Lösung","verfolgen"]}},
{"case_id":"TN-03","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Für den Druckversuch steht die reale Anlage zur Diskussion.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Die reale Anlage nutzen wir dafür nicht.","speaker":"Martin","named_person":null,"addressee":null}],"fixed_linkage":{"candidate_observation_id":"obs_2","target_observation_id":"obs_1"},"expected":{"normalized_target_text":"reale Anlage für den Druckversuch nutzen","action_concepts":[["Anlage"],["nutzen"]],"material_concepts":[["real"],["Druckversuch"]],"forbidden_concepts":["nicht","zur Diskussion"],"german_markers":["Anlage","Druckversuch","nutzen"]}},
{"case_id":"TN-04","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten den Versuch in der realen Anlage durchführen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir nicht; wir testen stattdessen im Technikum.","speaker":"Martin","named_person":null,"addressee":null}],"fixed_linkage":{"candidate_observation_id":"obs_2","target_observation_id":"obs_1"},"expected":{"normalized_target_text":"Versuch in der realen Anlage durchführen","action_concepts":[["Versuch"],["durchführen"]],"material_concepts":[["real"],["Anlage"]],"forbidden_concepts":["nicht","Technikum","stattdessen"],"german_markers":["Versuch","Anlage","durchführen"]}}
]
}
@@ -0,0 +1,13 @@
{
"schema_version": "experimental-target-resolution-v0",
"cases": [
{"case_id":"TR-01","description":"self-contained continuation target","strategy":"self_contained","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.","speaker":"Martin","named_person":"Dr. Schlummer","addressee":null}],"negative_act":{"observation_id":"obs_1","negative_act_form":"explicit_non_pursuit","normalized_action_text":"working with Dr. Schlummer"},"expected":{"eligible":true,"target_observation_id":"obs_1","concepts":[["Zusammenarbeit","arbeiten"],["Schlummer"],["fortsetzen","weiter"]],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR-02","description":"paired pronoun target","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Eine Möglichkeit wäre, die externe Lösung weiterzuverfolgen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das verfolgen wir nicht weiter.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"verfolgen wir nicht weiter"},"expected":{"eligible":true,"target_observation_id":"obs_1","concepts":[["externe Lösung"],["weiterverfolgen","weiter verfolgen"]],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR-03","description":"scoped location and purpose target","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Für den Druckversuch steht die reale Anlage zur Diskussion.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Die reale Anlage nutzen wir dafür nicht.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"reale Anlage dafür nicht nutzen"},"expected":{"eligible":true,"target_observation_id":"obs_1","concepts":[["Anlage"],["nutzen"]],"material_concepts":[["real"],["Druckversuch"]],"forbidden_concepts":[]}},
{"case_id":"TR-04","description":"rejection plus alternative","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten den Versuch in der realen Anlage durchführen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir nicht; wir testen stattdessen im Technikum.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"Versuch in der realen Anlage nicht durchführen"},"expected":{"eligible":true,"target_observation_id":"obs_1","concepts":[["Versuch"],["durchführen"]],"material_concepts":[["real"],["Anlage"]],"forbidden_concepts":["Technikum"]}},
{"case_id":"TR-05","description":"personal preference","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die reale Anlage für den Versuch nutzen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Ich würde das nicht machen.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"personal_preference","normalized_action_text":"Ich würde das nicht machen"},"expected":{"eligible":false,"target_observation_id":null,"concepts":[],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR-06","description":"recommendation","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die reale Anlage verwenden.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Ich würde eher davon abraten.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"recommendation","normalized_action_text":"advise against using the real asset"},"expected":{"eligible":false,"target_observation_id":null,"concepts":[],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR-07","description":"temporary non-action","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten die Waschstufe einbauen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir erstmal noch nicht.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"temporary_non_action","normalized_action_text":"install the washing stage"},"expected":{"eligible":false,"target_observation_id":null,"concepts":[],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR-08","description":"concern only","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten das neue Material einsetzen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das wäre kritisch.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"none","normalized_action_text":null},"expected":{"eligible":false,"target_observation_id":null,"concepts":[],"material_concepts":[],"forbidden_concepts":[]}}
]
}
@@ -0,0 +1,9 @@
{
"schema_version":"experimental-target-resolution-v1-diagnostic",
"cases":[
{"case_id":"TR1-V1","strategy":"self_contained","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.","speaker":"Martin","named_person":"Dr. Schlummer","addressee":null}],"negative_act":{"observation_id":"obs_1","negative_act_form":"explicit_non_pursuit","normalized_action_text":"working with Dr. Schlummer"},"expected":{"target_observation_id":"obs_1","concepts":[["Zusammenarbeit","arbeiten"],["Schlummer"],["fortsetzen","weiter"]],"material_concepts":[],"forbidden_concepts":["nicht"]}},
{"case_id":"TR2-V1","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Eine Möglichkeit wäre, die externe Lösung weiterzuverfolgen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das verfolgen wir nicht weiter.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"verfolgen wir nicht weiter"},"expected":{"target_observation_id":"obs_1","concepts":[["externe Lösung"],["weiterverfolgen","weiter verfolgen"]],"material_concepts":[],"forbidden_concepts":[]}},
{"case_id":"TR3-V1","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Für den Druckversuch steht die reale Anlage zur Diskussion.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Die reale Anlage nutzen wir dafür nicht.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"reale Anlage dafür nicht nutzen"},"expected":{"target_observation_id":"obs_1","concepts":[["Anlage"],["nutzen"]],"material_concepts":[["real"],["Druckversuch"]],"forbidden_concepts":[]}},
{"case_id":"TR4-V1","strategy":"paired","observations":[{"observation_id":"obs_1","evidence_id":"e1","content":"Martin: Wir könnten den Versuch in der realen Anlage durchführen.","speaker":"Martin","named_person":null,"addressee":null},{"observation_id":"obs_2","evidence_id":"e2","content":"Martin: Das machen wir nicht; wir testen stattdessen im Technikum.","speaker":"Martin","named_person":null,"addressee":null}],"negative_act":{"observation_id":"obs_2","negative_act_form":"explicit_non_pursuit","normalized_action_text":"Versuch in der realen Anlage nicht durchführen"},"expected":{"target_observation_id":"obs_1","concepts":[["Versuch"],["durchführen"]],"material_concepts":[["real"],["Anlage"]],"forbidden_concepts":["Technikum"]}}
]
}
+230
View File
@@ -0,0 +1,230 @@
import subprocess
import tempfile
import unittest
import wave
from pathlib import Path
from unittest.mock import patch
from src.meeting_lab.audio.preparation import (
DEFAULT_NORMALIZATION_FILTER,
DEFAULT_NORMALIZATION_METHOD,
AudioPreparationError,
prepare_audio,
)
def _write_wav(
path: Path, *, channels: int = 1, sample_rate: int = 16_000, sample_width: int = 2
) -> None:
with wave.open(str(path), "wb") as recording:
recording.setnchannels(channels)
recording.setsampwidth(sample_width)
recording.setframerate(sample_rate)
recording.writeframes(b"\x00" * channels * sample_width * 32)
def _successful_runner(commands: list[list[str]]):
def run(command, **kwargs):
commands.append(list(command))
_write_wav(Path(command[-1]))
return subprocess.CompletedProcess(command, 0, "", "")
return run
class AudioPreparationTests(unittest.TestCase):
def test_supported_inputs_are_prepared_with_normalization_on_and_off(self) -> None:
for suffix in (".wav", ".flac", ".m4a"):
for normalization_enabled in (True, False):
with (
self.subTest(
suffix=suffix, normalization_enabled=normalization_enabled
),
tempfile.TemporaryDirectory() as directory,
):
root = Path(directory)
source = root / f"meeting{suffix}"
if suffix == ".wav":
_write_wav(source)
else:
source.write_bytes(b"original encoded audio")
original = source.read_bytes()
destination = root / "run" / "audio" / "prepared.wav"
commands: list[list[str]] = []
with patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
):
result = prepare_audio(
source,
destination,
normalization_enabled=normalization_enabled,
runner=_successful_runner(commands),
)
self.assertEqual(source.read_bytes(), original)
self.assertEqual(result.prepared_path, destination)
with wave.open(str(destination), "rb") as recording:
self.assertEqual(recording.getnchannels(), 1)
self.assertEqual(recording.getframerate(), 16_000)
self.assertEqual(recording.getsampwidth(), 2)
self.assertEqual(recording.getcomptype(), "NONE")
self.assertEqual(commands[0][commands[0].index("-ac") + 1], "1")
self.assertEqual(commands[0][commands[0].index("-ar") + 1], "16000")
self.assertEqual(
commands[0][commands[0].index("-c:a") + 1], "pcm_s16le"
)
self.assertEqual("-af" in commands[0], normalization_enabled)
if normalization_enabled:
self.assertEqual(
commands[0][commands[0].index("-af") + 1],
DEFAULT_NORMALIZATION_FILTER,
)
self.assertEqual(
result.normalization_enabled, normalization_enabled
)
def test_normalization_defaults_to_on_and_explicit_on_matches(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "meeting.wav"
_write_wav(source)
commands: list[list[str]] = []
with patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
):
default = prepare_audio(
source, root / "default.wav", runner=_successful_runner(commands)
)
explicit = prepare_audio(
source,
root / "explicit.wav",
normalization_enabled=True,
runner=_successful_runner(commands),
)
self.assertTrue(default.normalization_enabled)
self.assertTrue(explicit.normalization_enabled)
self.assertEqual(
commands[0][commands[0].index("-af") + 1],
commands[1][commands[1].index("-af") + 1],
)
def test_noncanonical_wav_is_normalized(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "stereo-48k.wav"
_write_wav(source, channels=2, sample_rate=48_000)
destination = root / "prepared.wav"
with patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
):
prepare_audio(source, destination, runner=_successful_runner([]))
with wave.open(str(destination), "rb") as recording:
self.assertEqual(
(recording.getnchannels(), recording.getframerate()), (1, 16_000)
)
def test_ffmpeg_missing_has_actionable_error(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "meeting.flac"
source.write_bytes(b"audio")
with (
patch(
"src.meeting_lab.audio.preparation.shutil.which", return_value=None
),
self.assertRaisesRegex(AudioPreparationError, "not found on PATH"),
):
prepare_audio(source, root / "prepared.wav")
def test_ffmpeg_failure_includes_diagnostic_and_preserves_source(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "meeting.m4a"
source.write_bytes(b"original")
def fail(command, **kwargs):
return subprocess.CompletedProcess(command, 1, "", "decoder exploded")
with (
patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
),
self.assertRaisesRegex(AudioPreparationError, "decoder exploded"),
):
prepare_audio(source, root / "prepared.wav", runner=fail)
self.assertEqual(source.read_bytes(), b"original")
self.assertFalse((root / "prepared.wav").exists())
def test_prepared_audio_metadata_is_traceable(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "unknown_meeting.flac"
source.write_bytes(b"source")
destination = root / "audio" / "prepared.wav"
with patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
):
result = prepare_audio(
source, destination, runner=_successful_runner([])
)
metadata = result.metadata()
self.assertEqual(metadata["original_source_name"], "unknown_meeting.flac")
self.assertEqual(metadata["original_format"], "flac")
self.assertEqual(
metadata["prepared_audio_path"], str(destination.resolve())
)
self.assertEqual(metadata["preparation_method"], "ffmpeg")
self.assertTrue(metadata["normalization_enabled"])
self.assertEqual(
metadata["normalization_method"], DEFAULT_NORMALIZATION_METHOD
)
self.assertEqual(
metadata["normalization_filter"], DEFAULT_NORMALIZATION_FILTER
)
self.assertEqual(
metadata["canonical_output"],
{
"container": "wav",
"codec": "pcm_s16le",
"channels": 1,
"sample_rate_hz": 16_000,
"bits_per_sample": 16,
},
)
def test_disabled_normalization_metadata_has_no_method_or_filter(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "meeting.m4a"
source.write_bytes(b"source")
with patch(
"src.meeting_lab.audio.preparation.shutil.which",
return_value="/usr/bin/ffmpeg",
):
result = prepare_audio(
source,
root / "prepared.wav",
normalization_enabled=False,
runner=_successful_runner([]),
)
metadata = result.metadata()
self.assertFalse(metadata["normalization_enabled"])
self.assertIsNone(metadata["normalization_method"])
self.assertIsNone(metadata["normalization_filter"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,200 @@
import json
import tempfile
import unittest
from copy import deepcopy
from pathlib import Path
from src.meeting_lab.controlled_semantic_derivation.experiment_collective import (
DerivationValidationError,
build_prompt,
derive_collective_action,
evaluate_case,
load_gold_cases,
validate_recognition,
)
GOLD_PATH = Path("tests/gold/collective_commitment_v0/cases.json")
def recognition_for(case):
form = case["expected_recognition"]["commitment_form"]
action = "20 Meter testen"
if case["case_id"] == "CC-08":
action = "20 Meter testen, aber nur im Technikum"
return {
"observation_id": "obs_1",
"commitment_form": form,
"normalized_action_text": action,
}
class CollectiveCommitmentGoldExperimentTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.cases = load_gold_cases(GOLD_PATH)
cls.by_id = {case["case_id"]: case for case in cls.cases}
def test_fixture_contains_exactly_cc_01_through_cc_10(self):
self.assertEqual(list(self.by_id), [f"CC-{number:02d}" for number in range(1, 11)])
def test_all_cases_are_single_minimal_v3_style_observations(self):
keys = {"observation_id", "evidence_id", "content", "speaker", "named_person", "addressee"}
for case in self.cases:
with self.subTest(case=case["case_id"]):
self.assertEqual(len(case["observations"]), 1)
self.assertEqual(set(case["observations"][0]), keys)
def test_cc_01_establishes_collective_action_without_person_and_with_due(self):
case = self.by_id["CC-01"]
gates, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertTrue(all(gates.values()))
self.assertEqual(result["status"], "established")
self.assertEqual(result["commitment_scope"], "collective")
self.assertIsNone(result["responsible_person"])
self.assertEqual(result["due"], "nächste Woche")
def test_individual_commitment_routes_out_of_collective_path(self):
self._assert_unestablished("CC-02", "collective_commitment_form")
def test_tentative_suggestion_impersonal_and_passive_remain_unestablished(self):
for case_id in ("CC-03", "CC-04", "CC-05", "CC-06"):
with self.subTest(case=case_id):
self._assert_unestablished(case_id, "collective_commitment_form")
def test_rejection_remains_unestablished_and_negation_gate_is_negative(self):
case = self.by_id["CC-07"]
gates, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertFalse(gates["collective_commitment_form"])
self.assertFalse(gates["no_explicit_negation"])
self.assertIsNone(result)
def test_qualifier_case_establishes_preserves_limit_and_has_no_due(self):
case = self.by_id["CC-08"]
_, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertIsNotNone(result)
self.assertIn("nur im Technikum", result["content"])
self.assertIsNone(result["due"])
self.assertIsNone(result["responsible_person"])
def test_collective_without_deadline_establishes_with_null_due(self):
case = self.by_id["CC-09"]
_, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertIsNotNone(result)
self.assertIsNone(result["due"])
def test_speaker_ownership_trap_never_assigns_martin(self):
case = self.by_id["CC-10"]
_, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertIsNotNone(result)
self.assertEqual(case["observations"][0]["speaker"], "Martin")
self.assertIsNone(result["responsible_person"])
def test_changing_only_speaker_cannot_create_individual_owner(self):
case = deepcopy(self.by_id["CC-01"])
for speaker in ("Martin", "Clara", "Antonius"):
case["observations"][0]["speaker"] = speaker
_, result = derive_collective_action(case["observations"], recognition_for(case))
with self.subTest(speaker=speaker):
self.assertIsNotNone(result)
self.assertIsNone(result["responsible_person"])
def test_none_and_individual_forms_never_establish(self):
case = self.by_id["CC-01"]
for form in ("none", "individual_first_person"):
recognition = recognition_for(case)
recognition["commitment_form"] = form
_, result = derive_collective_action(case["observations"], recognition)
with self.subTest(form=form):
self.assertIsNone(result)
def test_non_none_commitment_requires_action_text(self):
case = self.by_id["CC-01"]
recognition = recognition_for(case)
recognition["normalized_action_text"] = None
with self.assertRaisesRegex(DerivationValidationError, "requires normalized_action_text"):
validate_recognition(recognition, case["observations"])
def test_none_commitment_allows_null_action_text_but_never_establishes(self):
case = self.by_id["CC-03"]
recognition = recognition_for(case)
recognition["normalized_action_text"] = None
_, result = derive_collective_action(case["observations"], recognition)
self.assertIsNone(result)
def test_unknown_observation_id_is_rejected(self):
case = self.by_id["CC-01"]
recognition = recognition_for(case)
recognition["observation_id"] = "obs_99"
with self.assertRaisesRegex(DerivationValidationError, "unknown observation"):
validate_recognition(recognition, case["observations"])
def test_inconsistent_duplicate_evidence_provenance_is_rejected(self):
fixture = json.loads(GOLD_PATH.read_text())
fixture["cases"][0]["observations"].append({
"observation_id": "obs_2", "evidence_id": "e1", "content": "Martin: Zusatz.",
"speaker": "Martin", "named_person": None, "addressee": None,
})
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "cases.json"
path.write_text(json.dumps(fixture), encoding="utf-8")
with self.assertRaisesRegex(DerivationValidationError, "provenance must be unique"):
load_gold_cases(path)
def test_conflicting_deadlines_prevent_establishment(self):
case = deepcopy(self.by_id["CC-01"])
case["observations"][0]["content"] += " Bis Mittwoch."
gates, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertFalse(gates["deadline_supported_and_consistent"])
self.assertIsNone(result)
def test_forbidden_fields_are_rejected_recursively(self):
case = self.by_id["CC-01"]
forbidden = (
"responsible_person", "responsibility", "responsibility_scope",
"requested_actor", "owner", "ownership", "assignee", "status",
"established", "action_item", "protocol_category", "decision",
"unresolved_issue", "confidence", "relation", "relations", "graph",
)
for field in forbidden:
recognition = recognition_for(case)
recognition["wrapper"] = {field: "forbidden"}
with self.subTest(field=field), self.assertRaisesRegex(DerivationValidationError, "forbidden semantic keys"):
validate_recognition(recognition, case["observations"])
def test_unknown_schema_field_is_rejected(self):
case = self.by_id["CC-01"]
recognition = recognition_for(case)
recognition["explanation"] = "extra"
with self.assertRaisesRegex(DerivationValidationError, "unknown keys"):
validate_recognition(recognition, case["observations"])
def test_provenance_survives_and_successes_always_have_null_person(self):
for case_id in ("CC-01", "CC-08", "CC-09", "CC-10"):
case = self.by_id[case_id]
_, result = derive_collective_action(case["observations"], recognition_for(case))
with self.subTest(case=case_id):
self.assertEqual(result["support"]["commitment"], {"observation_id": "obs_1", "evidence_id": "e1"})
self.assertIsNone(result["responsible_person"])
def test_all_expected_recognitions_have_correct_final_outcome(self):
for case in self.cases:
evaluation = evaluate_case(case, recognition_for(case))
with self.subTest(case=case["case_id"]):
self.assertEqual(evaluation["classification"], "PASS")
def test_prompt_is_fixed_narrow_and_contains_no_gold_expectation(self):
prompt = build_prompt(self.by_id["CC-01"]["observations"])
self.assertIn("commitment_form", prompt)
self.assertNotIn("expected_result", prompt)
self.assertNotIn("Who is responsible", prompt)
def _assert_unestablished(self, case_id, failed_gate):
case = self.by_id[case_id]
gates, result = derive_collective_action(case["observations"], recognition_for(case))
self.assertFalse(gates[failed_gate])
self.assertIsNone(result)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,82 @@
import copy, json, unittest
from pathlib import Path
from src.meeting_lab.controlled_semantic_derivation.experiment_h import DerivationValidationError
from src.meeting_lab.controlled_semantic_derivation.experiment_rejection_v1 import (
build_target_prompt, derive, load_cases, validate_target,
)
CASES=load_cases(Path("tests/gold/controlled_rejection_v1/cases.json"))
BY_ID={c["case_id"]:c for c in CASES}
def negative(case, form=None):
return {"observation_id":case["expected"]["candidate_observation_id"],"negative_act_form":form or case["expected"]["negative_act_form"],"normalized_action_text":None if (form or case["expected"]["negative_act_form"])=="none" else "semantische Aktion"}
def target(case, oid=None, text="konkrete Zielhandlung"):
return {"candidate_observation_id":case["expected"]["candidate_observation_id"],"target_observation_id":oid or case["expected"]["target_observation_id"],"normalized_target_text":text}
class ControlledRejectionV1Tests(unittest.TestCase):
def test_positive_forms_derive_and_provenance_survives(self):
for cid in ("CR-01","CR-02","CR-07","CR-08"):
c=BY_ID[cid]; out=derive(c["observations"],negative(c),target(c))
self.assertEqual(out["derived_result"]["status"],"explicitly_rejected")
self.assertEqual(out["derived_result"]["support"]["target"]["evidence_id"],"e1")
self.assertNotIn("responsible_person",json.dumps(out["derived_result"]))
def test_noneligible_form_never_derives_even_with_target(self):
for form in ["personal_preference","recommendation","temporary_non_action","none"]:
c=BY_ID["CR-03"]; self.assertIsNone(derive(c["observations"],negative(c,form),target(c))["derived_result"])
def test_missing_target_prevents_derivation(self):
c=BY_ID["CR-02"]; t=target(c); t.update(target_observation_id=None,normalized_target_text=None)
self.assertIsNone(derive(c["observations"],negative(c),t)["derived_result"])
def test_unknown_ids_rejected(self):
for field in ["candidate_observation_id","target_observation_id"]:
c=BY_ID["CR-02"]; t=target(c); t[field]="obs_unknown"
with self.assertRaises(DerivationValidationError): validate_target(t,c["observations"])
def test_target_after_candidate_cannot_derive(self):
c=copy.deepcopy(BY_ID["CR-02"]); n={"observation_id":"obs_1","negative_act_form":"explicit_non_pursuit","normalized_action_text":"x"}; t={"candidate_observation_id":"obs_1","target_observation_id":"obs_2","normalized_target_text":"x"}
self.assertIsNone(derive(c["observations"],n,t)["derived_result"])
def test_same_observation_target_allowed(self):
c=BY_ID["CR-01"]; self.assertIsNotNone(derive(c["observations"],negative(c),target(c))["derived_result"])
def test_duplicate_provenance_rejected(self):
for field in ["observation_id","evidence_id"]:
c=copy.deepcopy(BY_ID["CR-02"]); c["observations"][1][field]=c["observations"][0][field]
with self.assertRaises(DerivationValidationError): derive(c["observations"],negative(BY_ID["CR-02"]),target(BY_ID["CR-02"]))
def test_target_text_constraints(self):
c=BY_ID["CR-02"]
with self.assertRaises(DerivationValidationError): validate_target(target(c,text=""),c["observations"])
t=target(c); t["target_observation_id"]=None
with self.assertRaises(DerivationValidationError): validate_target(t,c["observations"])
def test_null_target_accepts_only_null_text(self):
c=BY_ID["CR-02"]; t=target(c); t.update(target_observation_id=None,normalized_target_text=None)
self.assertEqual(validate_target(t,c["observations"]),t)
def test_forbidden_and_unknown_fields_rejected(self):
for extra in [{"status":"rejected"},{"nested":{"decision":True}},{"extra":1}]:
c=BY_ID["CR-02"]; t=target(c); t.update(extra)
with self.assertRaises(DerivationValidationError): validate_target(t,c["observations"])
def test_candidate_outputs_must_agree(self):
c=BY_ID["CR-02"]; t=target(c); t["candidate_observation_id"]="obs_1"
with self.assertRaises(DerivationValidationError): derive(c["observations"],negative(c),t)
def test_scope_and_alternative_fixture_contract(self):
self.assertLessEqual({"real","Druckversuch"},{x for group in BY_ID["CR-07"]["expected"]["material_concepts"] for x in group})
self.assertEqual(BY_ID["CR-08"]["expected"]["forbidden_concepts"],["Technikum"])
def test_target_prompt_is_semantic_only_and_fixed(self):
prompt=build_target_prompt(BY_ID["CR-08"])
self.assertIn("Ignore any separate positive alternative",prompt)
self.assertIn("Do not classify the negative act",prompt)
def test_accepted_negative_act_inputs_are_exactly_reused(self):
root=Path("artifacts/experiments/negative_act_form_v0/20260820_qwen35_9b_single_run")
for cid,nid in (("CR-01","NA-01"),("CR-03","NA-03"),("CR-04","NA-04"),("CR-05","NA-05"),("CR-06","NA-06")):
accepted=json.loads((root/nid.lower()/"v3_style_input_observations.json").read_text())
self.assertEqual(BY_ID[cid]["observations"],accepted)
+222
View File
@@ -0,0 +1,222 @@
import json
import os
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from src.meeting_lab.diarization.alignment import align_transcript, write_diarized_transcript
from src.meeting_lab.diarization.backend import (
DiarizationError,
diarize_audio,
run_container_pyannote,
select_device,
)
class FakeCuda:
def __init__(self, available: bool, *, name: str = "Test GPU", failure=None):
self.available = available
self.name = name
self.failure = failure
def is_available(self):
return self.available
def get_device_name(self, index):
if self.failure:
raise self.failure
return self.name
class FakeTorch:
def __init__(self, available: bool, *, failure=None):
self.cuda = FakeCuda(available, failure=failure)
self.probes = []
def device(self, name):
return name
def zeros(self, size, *, device):
self.probes.append(device)
if self.cuda.failure:
raise self.cuda.failure
return [0]
class DeviceSelectionTests(unittest.TestCase):
def test_auto_selects_usable_gpu(self):
torch = FakeTorch(True)
self.assertEqual(select_device("auto", torch), ("cuda", "Test GPU"))
self.assertEqual(torch.probes, ["cuda"])
def test_auto_falls_back_to_cpu(self):
self.assertEqual(select_device("auto", FakeTorch(False)), ("cpu", None))
self.assertEqual(
select_device("auto", FakeTorch(True, failure=RuntimeError("probe"))),
("cpu", None),
)
def test_explicit_cpu_does_not_probe_gpu(self):
torch = FakeTorch(True)
self.assertEqual(select_device("cpu", torch), ("cpu", None))
self.assertEqual(torch.probes, [])
def test_explicit_gpu_fails_when_unavailable(self):
with self.assertRaisesRegex(DiarizationError, "GPU is unavailable"):
select_device("gpu", FakeTorch(False))
class AlignmentTests(unittest.TestCase):
def test_exclusive_overlap_assigns_anonymous_speakers(self):
transcript = {
"text": "Original unchanged text.",
"segments": [
{"id": 0, "start": 0.0, "end": 4.0, "text": "Hallo"},
{"id": 1, "start": 4.0, "end": 6.0, "text": "Antwort"},
],
}
original = json.loads(json.dumps(transcript))
turns = [
{"start": 0.0, "end": 3.0, "speaker_id": "SPEAKER_00"},
{"start": 3.0, "end": 6.0, "speaker_id": "SPEAKER_01"},
]
derived = align_transcript(transcript, turns)
self.assertEqual(transcript, original)
self.assertEqual(derived["segments"][0]["speaker_id"], "SPEAKER_00")
self.assertEqual(derived["segments"][0]["speaker_overlap_seconds"], 3.0)
self.assertEqual(derived["segments"][1]["speaker_id"], "SPEAKER_01")
self.assertIn("SPEAKER_00: Hallo", derived["text"])
self.assertTrue(derived["speaker_labels_anonymous"])
self.assertEqual(derived["alignment_source"], "exclusive_diarization")
def test_speaker_aware_transcript_is_a_separate_artifact(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
source = root / "transcript.json"
turns = root / "exclusive_turns.json"
source_text = json.dumps(
{"text": "Original", "segments": [{"start": 0, "end": 1, "text": "Hi"}]}
)
source.write_text(source_text, encoding="utf-8")
turns.write_text(
json.dumps([{"start": 0, "end": 1, "speaker_id": "SPEAKER_07"}]),
encoding="utf-8",
)
json_path, text_path = write_diarized_transcript(source, turns, root / "derived")
self.assertEqual(source.read_text(encoding="utf-8"), source_text)
self.assertNotEqual(json_path, source)
self.assertIn("SPEAKER_07", json_path.read_text(encoding="utf-8"))
self.assertIn("SPEAKER_07", text_path.read_text(encoding="utf-8"))
class ContainerAdapterTests(unittest.TestCase):
def test_container_configuration_and_metadata_do_not_persist_token(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio = root / "audio.wav"
output = root / "output"
audio.write_bytes(b"audio")
observed = {}
def fake_runner(command, **kwargs):
observed["command"] = command
output.mkdir(exist_ok=True)
metadata = {
"backend": "pyannote.audio",
"model": "pyannote/speaker-diarization-community-1",
"requested_device_mode": "gpu",
"actual_device": "cuda",
"credentials_persisted": False,
}
(output / "metadata.json").write_text(json.dumps(metadata))
return SimpleNamespace(returncode=0, stdout="ok", stderr="")
with patch.dict("os.environ", {"HF_TOKEN": "secret-token"}):
result = run_container_pyannote(
audio,
output,
"gpu",
image="test/image",
container_args=("--device=/dev/test",),
runner=fake_runner,
uid_getter=lambda: 2345,
gid_getter=lambda: 3456,
)
command = observed["command"]
shell_command = command[-1]
persisted = "".join(
path.read_text(encoding="utf-8")
for path in output.iterdir()
if path.is_file()
)
self.assertNotIn("secret-token", persisted)
self.assertNotIn("secret-token", command)
self.assertIn("HF_TOKEN", command)
self.assertIn("chown -R 2345:3456 /output", shell_command)
self.assertIn("chmod -R u+rwX /output", shell_command)
self.assertNotIn("1000:1000", shell_command)
device_index = command.index("--device=/dev/test")
self.assertLess(device_index, command.index("test/image"))
self.assertFalse(result.metadata["credentials_persisted"])
self.assertEqual(result.metadata["runtime_adapter"], "container")
self.assertTrue(
all(os.access(path, os.W_OK) for path in (output, *output.rglob("*")))
)
def test_unwritable_container_artifact_is_rejected_before_metadata_update(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio = root / "audio.wav"
output = root / "output"
audio.write_bytes(b"audio")
def fake_runner(command, **kwargs):
output.mkdir(exist_ok=True)
metadata = output / "metadata.json"
metadata.write_text("{}", encoding="utf-8")
metadata.chmod(0o444)
return SimpleNamespace(returncode=0, stdout="", stderr="")
with patch(
"src.meeting_lab.diarization.backend.os.access",
side_effect=lambda path, mode: Path(path).name != "metadata.json",
):
with self.assertRaisesRegex(DiarizationError, "not writable"):
run_container_pyannote(
audio,
output,
"cpu",
image="test/image",
runner=fake_runner,
)
def test_orchestrator_dispatches_runtime(self):
with patch(
"src.meeting_lab.diarization.backend.run_container_pyannote"
) as container:
diarize_audio(
Path("audio.wav"),
Path("out"),
"cpu",
runtime="container",
container_image="image",
container_args=("--arg",),
)
container.assert_called_once_with(
Path("audio.wav"),
Path("out"),
"cpu",
image="image",
container_args=("--arg",),
)
if __name__ == "__main__":
unittest.main()
+321
View File
@@ -0,0 +1,321 @@
import json
import tempfile
import unittest
from datetime import datetime
from pathlib import Path
from unittest.mock import Mock, patch
import requests
from scripts import run_direct_protocol
from src.meeting_lab.llm import ollama
from src.meeting_lab.llm.ollama import OllamaError, OllamaGeneration
from src.meeting_lab.protocol.generate_direct_protocol import (
DirectProtocolError,
generate_direct_protocol,
load_compact_transcript,
)
VALID_CONTEXT = """schema_version: "1"
meeting:
meeting_id: "test-meeting"
title: "Test Meeting"
language: "de"
participants: []
mentioned_people: []
organization:
departments: []
known_entities: {}
"""
def write_transcript(path: Path, text: str = "Wir besprechen den Projektstatus.") -> None:
path.write_text(json.dumps({"text": text, "segments": []}), encoding="utf-8")
def generation(text: str = "# Meeting Protocol\n\n## Status\nUnveraendert.") -> OllamaGeneration:
return OllamaGeneration(
raw_response={
"response": text,
"done": True,
"done_reason": "stop",
"prompt_eval_count": 123,
"eval_count": 17,
"prompt_eval_duration": 1000,
"eval_duration": 2000,
"total_duration": 4000,
},
text=text,
client_wall_time_seconds=0.25,
)
class TranscriptLoadingTests(unittest.TestCase):
def test_valid_transcript_is_accepted(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "transcript.json"
write_transcript(path)
self.assertEqual(load_compact_transcript(path), "Wir besprechen den Projektstatus.")
def test_missing_top_level_text_is_rejected(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "transcript.json"
path.write_text('{"segments": []}', encoding="utf-8")
with self.assertRaisesRegex(DirectProtocolError, "top-level 'text'"):
load_compact_transcript(path)
def test_empty_transcript_is_rejected(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "transcript.json"
write_transcript(path, " \n")
with self.assertRaisesRegex(DirectProtocolError, "non-empty string"):
load_compact_transcript(path)
def test_malformed_json_is_rejected(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "transcript.json"
path.write_text("{", encoding="utf-8")
with self.assertRaisesRegex(DirectProtocolError, "not valid JSON"):
load_compact_transcript(path)
class GeneratorTests(unittest.TestCase):
def test_prompt_requires_contextual_discussion_density_without_transcript_replay(self) -> None:
with tempfile.TemporaryDirectory() as directory:
transcript = Path(directory) / "transcript.json"
write_transcript(transcript)
result = generate_direct_protocol(
transcript,
model_check=Mock(return_value={}),
generation_call=Mock(return_value=generation()),
)
self.assertIn("vollständiges, strukturiertes", result.exact_prompt)
self.assertIn("relevante Diskussionsverläufe", result.exact_prompt)
self.assertIn("unterschiedliche Positionen", result.exact_prompt)
self.assertIn("Entscheidungsgrundlagen", result.exact_prompt)
self.assertIn("nicht am Meeting teilgenommen haben", result.exact_prompt)
self.assertIn("keine reine Wiedergabe des Transkripts", result.exact_prompt)
self.assertIn("nicht unnötig durch Wiederholungen", result.exact_prompt)
def test_optional_context_absent_and_generation_called_once(self) -> None:
with tempfile.TemporaryDirectory() as directory:
transcript = Path(directory) / "transcript.json"
write_transcript(transcript)
check = Mock(return_value={"model": "qwen3.6:35B-A3B"})
call = Mock(return_value=generation())
result = generate_direct_protocol(
transcript,
model_check=check,
generation_call=call,
)
self.assertIn("Kein Meeting-Kontext", result.exact_prompt)
self.assertEqual(check.call_count, 1)
self.assertEqual(call.call_count, 1)
self.assertEqual(call.call_args.args[1], "qwen3.6:35B-A3B")
self.assertEqual(call.call_args.kwargs["num_ctx"], 32768)
self.assertEqual(result.runtime_metadata["request_count"], 1)
self.assertEqual(result.runtime_metadata["prompt_token_count"], 123)
self.assertFalse(result.runtime_metadata["think"])
self.assertEqual(result.runtime_metadata["temperature"], 0.0)
def test_valid_context_is_loaded_and_rendered(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = root / "transcript.json"
context = root / "context.yaml"
write_transcript(transcript)
context.write_text(VALID_CONTEXT, encoding="utf-8")
result = generate_direct_protocol(
transcript,
context,
model_check=Mock(return_value={}),
generation_call=Mock(return_value=generation()),
)
self.assertIn("MEETING CONTEXT V1", result.exact_prompt)
self.assertIn("Test Meeting", result.exact_prompt)
def test_invalid_context_is_rejected_before_network_calls(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = root / "transcript.json"
context = root / "context.yaml"
write_transcript(transcript)
context.write_text("schema_version: wrong", encoding="utf-8")
check = Mock()
call = Mock()
with self.assertRaisesRegex(ValueError, "schema_version"):
generate_direct_protocol(
transcript,
context,
model_check=check,
generation_call=call,
)
check.assert_not_called()
call.assert_not_called()
class OllamaTests(unittest.TestCase):
def test_unavailable_endpoint_failure(self) -> None:
with patch.object(ollama.requests, "get", side_effect=requests.ConnectionError("down")):
with self.assertRaisesRegex(OllamaError, "not reachable"):
ollama.require_model("http://127.0.0.1:11434", "model")
def test_missing_model_failure(self) -> None:
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = {"models": [{"name": "other:model"}]}
with patch.object(ollama.requests, "get", return_value=response):
with self.assertRaisesRegex(OllamaError, "not installed"):
ollama.require_model("http://127.0.0.1:11434", "model")
def test_request_settings_and_raw_response(self) -> None:
raw = {"response": "# Meeting Protocol", "done": True}
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = raw
with patch.object(ollama.requests, "post", return_value=response) as post:
result = ollama.generate_once(
"http://localhost:11434",
"qwen3.8:27b",
"prompt",
timeout=30,
num_ctx=32768,
num_predict=8192,
)
self.assertEqual(post.call_count, 1)
payload = post.call_args.kwargs["json"]
self.assertEqual(payload["model"], "qwen3.8:27b")
self.assertEqual(payload["options"]["temperature"], 0.0)
self.assertEqual(payload["options"]["num_ctx"], 32768)
self.assertFalse(payload["think"])
self.assertFalse(payload["stream"])
self.assertEqual(result.raw_response, raw)
def test_malformed_response_failure_without_retry(self) -> None:
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = {"message": "missing response"}
with patch.object(ollama.requests, "post", return_value=response) as post:
with self.assertRaisesRegex(OllamaError, "no string 'response'"):
ollama.generate_once("url", "model", "prompt", timeout=1, num_ctx=1, num_predict=1)
self.assertEqual(post.call_count, 1)
def test_empty_response_failure(self) -> None:
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = {"response": " "}
with patch.object(ollama.requests, "post", return_value=response):
with self.assertRaisesRegex(OllamaError, "empty protocol"):
ollama.generate_once("url", "model", "prompt", timeout=1, num_ctx=1, num_predict=1)
class DirectProtocolCliTests(unittest.TestCase):
def test_artifacts_are_preserved_and_protocol_is_untouched(self) -> None:
protocol_text = "# Meeting Protocol\n\nExact output. \n"
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = root / "source.json"
context = root / "source.yaml"
write_transcript(transcript)
context.write_text(VALID_CONTEXT, encoding="utf-8")
args = run_direct_protocol.parse_args(
[str(transcript), "--context", str(context), "--output-root", str(root / "runs")]
)
with patch.object(
run_direct_protocol,
"generate_direct_protocol",
return_value=type("Result", (), {
"protocol_text": protocol_text,
"exact_prompt": "exact prompt\n",
"transcript_input": "selected transcript\n",
"raw_response": {"response": protocol_text},
"runtime_metadata": {"request_count": 1},
})(),
) as generator:
code, run_dir, protocol_path = run_direct_protocol.run(args)
self.assertEqual(code, 0)
self.assertEqual(generator.call_count, 1)
self.assertEqual(protocol_path.read_text(encoding="utf-8"), protocol_text)
self.assertEqual(
(run_dir / "protocol/exact_prompt.txt").read_text(encoding="utf-8"),
"exact prompt\n",
)
self.assertEqual(
(run_dir / "protocol/transcript_input.txt").read_text(encoding="utf-8"),
"selected transcript\n",
)
self.assertEqual(
json.loads((run_dir / "protocol/raw_response.json").read_text())["response"],
protocol_text,
)
self.assertEqual(
json.loads((run_dir / "protocol/runtime_metadata.json").read_text())["request_count"],
1,
)
self.assertTrue((run_dir / "transcript/transcript.json").is_file())
self.assertTrue((run_dir / "context/meeting_context.yaml").is_file())
self.assertTrue((run_dir / "input_manifest.json").is_file())
self.assertEqual(json.loads((run_dir / "run_metadata.json").read_text())["status"], "completed")
def test_unique_run_directories_do_not_overwrite(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
fixed = datetime(2026, 8, 20, 12, 0, 0)
first = run_direct_protocol.create_unique_run_dir(root, "meeting", lambda: fixed)
marker = first / "keep.txt"
marker.write_text("keep", encoding="utf-8")
second = run_direct_protocol.create_unique_run_dir(root, "meeting", lambda: fixed)
self.assertEqual(first.name, "meeting_20260820_120000")
self.assertEqual(second.name, "meeting_20260820_120000_01")
self.assertEqual(marker.read_text(encoding="utf-8"), "keep")
def test_failure_after_directory_creation_preserves_metadata(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = run_direct_protocol.parse_args(
[str(root / "missing.json"), "--output-root", str(root / "runs")]
)
code, run_dir, protocol_path = run_direct_protocol.run(args)
metadata = json.loads((run_dir / "run_metadata.json").read_text())
self.assertEqual(code, 2)
self.assertIsNone(protocol_path)
self.assertEqual(metadata["status"], "failed")
self.assertIn("does not exist", metadata["failure"])
def test_semantic_pipeline_functions_are_never_invoked(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = root / "source.json"
write_transcript(transcript)
args = run_direct_protocol.parse_args(
[str(transcript), "--output-root", str(root / "runs")]
)
fake_result = type("Result", (), {
"protocol_text": "# Meeting Protocol",
"exact_prompt": "prompt",
"raw_response": {"response": "# Meeting Protocol"},
"runtime_metadata": {},
})()
with (
patch("src.meeting_lab.extraction.extract_chunks.extract_input") as extraction,
patch("src.meeting_lab.consolidation.consolidate_facts.call_ollama") as consolidation,
patch.object(run_direct_protocol, "generate_direct_protocol", return_value=fake_result),
):
code, _run_dir, _protocol_path = run_direct_protocol.run(args)
self.assertEqual(code, 0)
extraction.assert_not_called()
consolidation.assert_not_called()
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,210 @@
import json
import tempfile
import unittest
from copy import deepcopy
from pathlib import Path
from src.meeting_lab.controlled_semantic_derivation.experiment_rejection import (
DerivationValidationError,
build_prompt,
derive_rejection,
evaluate_case,
load_gold_cases,
validate_recognition,
)
GOLD_PATH = Path("tests/gold/explicit_rejection_v0/cases.json")
POSITIVE_TEXT = {
"RJ-01": "reale Anlage für den Versuch nutzen",
"RJ-02": "externe Lösung weiterverfolgen",
"RJ-03": "Zusammenarbeit mit Dr. Schlummer fortsetzen",
"RJ-11": "reale Anlage für den Druckversuch nutzen",
"RJ-12": "Versuch in der realen Anlage durchführen",
}
def recognition_for(case):
expected = case["expected_recognition"]
positive = expected["rejection_form"] == "explicit_action_rejection"
return {
"rejection_observation_id": expected["rejection_observation_id"],
"target_observation_id": expected["target_observation_id"] if positive else None,
"rejection_form": expected["rejection_form"],
"normalized_rejected_action_text": POSITIVE_TEXT.get(case["case_id"]) if positive else None,
}
class ExplicitRejectionGoldExperimentTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.cases = load_gold_cases(GOLD_PATH)
cls.by_id = {case["case_id"]: case for case in cls.cases}
def test_fixture_contains_exactly_rj_01_through_rj_12(self):
self.assertEqual(list(self.by_id), [f"RJ-{number:02d}" for number in range(1, 13)])
def test_cases_use_only_minimal_v3_style_observations(self):
keys = {"observation_id", "evidence_id", "content", "speaker", "named_person", "addressee"}
for case in self.cases:
with self.subTest(case=case["case_id"]):
self.assertIn(len(case["observations"]), (1, 2))
self.assertTrue(all(set(item) == keys for item in case["observations"]))
def test_rj_01_derives_target_and_both_provenance_paths(self):
case = self.by_id["RJ-01"]
gates, result = derive_rejection(case["observations"], recognition_for(case))
self.assertTrue(all(gates.values()))
self.assertEqual(result["status"], "explicitly_rejected")
self.assertEqual(result["support"]["target"], {"observation_id": "obs_1", "evidence_id": "e1"})
self.assertEqual(result["support"]["rejection"], {"observation_id": "obs_2", "evidence_id": "e2"})
def test_rj_02_requires_paired_target_and_derives_abandonment(self):
case = self.by_id["RJ-02"]
_, result = derive_rejection(case["observations"], recognition_for(case))
self.assertIn("externe Lösung", result["content"])
with self.assertRaisesRegex(DerivationValidationError, "unknown target"):
derive_rejection(case["observations"][1:], recognition_for(case))
def test_rj_03_supports_same_observation_target_and_rejection(self):
case = self.by_id["RJ-03"]
_, result = derive_rejection(case["observations"], recognition_for(case))
self.assertEqual(result["support"]["target"], result["support"]["rejection"])
self.assertIn("Dr. Schlummer", result["content"])
def test_all_required_negative_cases_remain_non_rejections(self):
for case_id in ("RJ-04", "RJ-05", "RJ-06", "RJ-07", "RJ-08", "RJ-09", "RJ-10"):
case = self.by_id[case_id]
gates, result = derive_rejection(case["observations"], recognition_for(case))
with self.subTest(case=case_id):
self.assertFalse(gates["explicit_action_rejection"])
self.assertIsNone(result)
def test_rj_11_derives_and_preserves_location_and_purpose_scope(self):
case = self.by_id["RJ-11"]
_, result = derive_rejection(case["observations"], recognition_for(case))
self.assertIsNotNone(result)
self.assertIn("reale Anlage", result["content"])
self.assertIn("Druckversuch", result["content"])
def test_rj_12_rejects_only_real_plant_action_and_not_alternative(self):
case = self.by_id["RJ-12"]
_, result = derive_rejection(case["observations"], recognition_for(case))
self.assertIn("realen Anlage", result["content"])
self.assertNotIn("Technikum", result["content"])
def test_separate_target_cannot_follow_rejection(self):
case = deepcopy(self.by_id["RJ-01"])
case["observations"].reverse()
gates, result = derive_rejection(case["observations"], recognition_for(case))
self.assertFalse(gates["target_same_or_before_rejection"])
self.assertIsNone(result)
def test_unknown_target_observation_id_is_rejected(self):
case = self.by_id["RJ-01"]
recognition = recognition_for(case)
recognition["target_observation_id"] = "obs_99"
with self.assertRaisesRegex(DerivationValidationError, "unknown target"):
validate_recognition(recognition, case["observations"])
def test_unknown_rejection_observation_id_is_rejected(self):
case = self.by_id["RJ-01"]
recognition = recognition_for(case)
recognition["rejection_observation_id"] = "obs_99"
with self.assertRaisesRegex(DerivationValidationError, "unknown rejection"):
validate_recognition(recognition, case["observations"])
def test_duplicate_observation_ids_are_rejected(self):
fixture = json.loads(GOLD_PATH.read_text())
fixture["cases"][0]["observations"][1]["observation_id"] = "obs_1"
self._assert_bad_fixture(fixture, "observation IDs must be unique")
def test_inconsistent_evidence_provenance_is_rejected(self):
fixture = json.loads(GOLD_PATH.read_text())
fixture["cases"][0]["observations"][1]["evidence_id"] = "e1"
self._assert_bad_fixture(fixture, "evidence provenance must be unique")
def test_none_rejects_populated_target_or_action(self):
case = self.by_id["RJ-04"]
for field, value, message in (
("target_observation_id", "obs_1", "null target"),
("normalized_rejected_action_text", "Anlage nutzen", "null normalized"),
):
recognition = recognition_for(case)
recognition[field] = value
with self.subTest(field=field), self.assertRaisesRegex(DerivationValidationError, message):
validate_recognition(recognition, case["observations"])
def test_explicit_rejection_requires_normalized_target_text(self):
case = self.by_id["RJ-01"]
for value in (None, ""):
recognition = recognition_for(case)
recognition["normalized_rejected_action_text"] = value
with self.subTest(value=value), self.assertRaises(DerivationValidationError):
validate_recognition(recognition, case["observations"])
def test_unknown_schema_fields_are_rejected(self):
case = self.by_id["RJ-01"]
recognition = recognition_for(case)
recognition["explanation"] = "extra"
with self.assertRaisesRegex(DerivationValidationError, "unknown keys"):
validate_recognition(recognition, case["observations"])
def test_forbidden_normative_fields_are_rejected_recursively(self):
case = self.by_id["RJ-01"]
fields = (
"decision", "decision_status", "outcome", "topic_status", "closed",
"agreement", "responsible_person", "responsibility", "responsibility_scope",
"owner", "ownership", "assignee", "requested_actor", "status",
"explicitly_rejected", "action_item", "protocol_category", "confidence",
"relation", "relations", "graph", "unresolved_issue",
)
for field in fields:
recognition = recognition_for(case)
recognition["wrapper"] = {field: "forbidden"}
with self.subTest(field=field), self.assertRaisesRegex(DerivationValidationError, "forbidden semantic keys"):
validate_recognition(recognition, case["observations"])
def test_speaker_identity_creates_no_ownership_or_responsibility(self):
case = deepcopy(self.by_id["RJ-03"])
for speaker in ("Martin", "Clara", "Antonius"):
case["observations"][0]["speaker"] = speaker
_, result = derive_rejection(case["observations"], recognition_for(case))
with self.subTest(speaker=speaker):
self.assertNotIn("responsible_person", result)
self.assertNotIn("owner", result)
def test_all_expected_recognitions_evaluate_as_pass(self):
for case in self.cases:
evaluation = evaluate_case(case, recognition_for(case))
with self.subTest(case=case["case_id"]):
self.assertEqual(evaluation["classification"], "PASS")
def test_rj_11_qualifier_loss_and_rj_12_alternative_absorption_fail(self):
rj11 = self.by_id["RJ-11"]
recognition = recognition_for(rj11)
recognition["normalized_rejected_action_text"] = "reale Anlage nutzen"
self.assertEqual(evaluate_case(rj11, recognition)["classification"], "FAIL")
rj12 = self.by_id["RJ-12"]
recognition = recognition_for(rj12)
recognition["normalized_rejected_action_text"] += "; stattdessen im Technikum testen"
self.assertEqual(evaluate_case(rj12, recognition)["classification"], "FAIL")
def test_prompt_is_fixed_narrow_and_does_not_expose_gold_expectation(self):
prompt = build_prompt(self.by_id["RJ-01"])
self.assertIn("candidate rejection observation is obs_2", prompt)
self.assertNotIn("expected_result", prompt)
self.assertNotIn("Who is responsible", prompt)
def _assert_bad_fixture(self, fixture, message):
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "cases.json"
path.write_text(json.dumps(fixture), encoding="utf-8")
with self.assertRaisesRegex(DerivationValidationError, message):
load_gold_cases(path)
if __name__ == "__main__":
unittest.main()
+79
View File
@@ -11,8 +11,10 @@ from src.meeting_lab.extraction.extract_chunks import (
) )
from src.meeting_lab.models.meeting_context import ( from src.meeting_lab.models.meeting_context import (
MeetingContextValidationError, MeetingContextValidationError,
create_meeting_context,
load_meeting_context, load_meeting_context,
render_meeting_context_for_prompt, render_meeting_context_for_prompt,
serialize_meeting_context_yaml,
validate_meeting_context, validate_meeting_context,
) )
@@ -70,6 +72,38 @@ class MeetingContextTests(unittest.TestCase):
with self.assertRaisesRegex(MeetingContextValidationError, "invalid value"): with self.assertRaisesRegex(MeetingContextValidationError, "invalid value"):
validate_meeting_context(data) validate_meeting_context(data)
def test_missing_participant_attendance_defaults_to_present(self) -> None:
data = copy.deepcopy(self.context.data)
del data["participants"][0]["attendance_status"]
context = create_meeting_context(data)
self.assertEqual(
context.data["participants"][0]["attendance_status"], "present"
)
def test_explicit_mentioned_only_is_preserved(self) -> None:
data = copy.deepcopy(self.context.data)
data["mentioned_people"][0]["attendance_status"] = "mentioned_only"
context = create_meeting_context(data)
self.assertEqual(
context.data["mentioned_people"][0]["attendance_status"],
"mentioned_only",
)
self.assertIn(
"Mentioned but absent people:", render_meeting_context_for_prompt(context)
)
def test_mentioned_only_person_cannot_be_a_diarized_speaker(self) -> None:
data = copy.deepcopy(self.context.data)
mentioned_id = data["mentioned_people"][0]["person_id"]
data["speaker_mappings"] = {"SPEAKER_00": mentioned_id}
with self.assertRaisesRegex(MeetingContextValidationError, "unknown participant"):
validate_meeting_context(data)
def test_prompt_representation_is_deterministic(self) -> None: def test_prompt_representation_is_deterministic(self) -> None:
first = render_meeting_context_for_prompt(self.context) first = render_meeting_context_for_prompt(self.context)
second = render_meeting_context_for_prompt(self.context) second = render_meeting_context_for_prompt(self.context)
@@ -192,6 +226,51 @@ class MeetingContextTests(unittest.TestCase):
self.assertNotIn("responsible: Björn", prompt_context) self.assertNotIn("responsible: Björn", prompt_context)
self.assertNotIn("responsible: Jovana", prompt_context) self.assertNotIn("responsible: Jovana", prompt_context)
def test_existing_context_without_speaker_mappings_remains_valid(self) -> None:
self.assertEqual(self.context.speaker_mappings, {})
self.assertIsNone(self.context.participant_for_speaker("SPEAKER_00"))
def test_explicit_speaker_mapping_is_authoritative(self) -> None:
data = copy.deepcopy(self.context.data)
participant = data["participants"][0]
data["speaker_mappings"] = {"SPEAKER_03": participant["participant_id"]}
context = create_meeting_context(data)
rendered = render_meeting_context_for_prompt(context)
self.assertEqual(
context.participant_for_speaker("SPEAKER_03")["participant_id"],
participant["participant_id"],
)
self.assertIsNone(context.participant_for_speaker("SPEAKER_04"))
self.assertIn("Confirmed diarization speaker mappings (authoritative)", rendered)
self.assertIn("Unmapped SPEAKER_XX labels must remain anonymous", rendered)
def test_speaker_mapping_must_reference_existing_participant(self) -> None:
data = copy.deepcopy(self.context.data)
data["speaker_mappings"] = {"SPEAKER_00": "unknown-person"}
with self.assertRaisesRegex(MeetingContextValidationError, "unknown participant"):
validate_meeting_context(data)
def test_speaker_mapping_label_must_use_pyannote_shape(self) -> None:
data = copy.deepcopy(self.context.data)
data["speaker_mappings"] = {
"Martin": data["participants"][0]["participant_id"]
}
with self.assertRaisesRegex(MeetingContextValidationError, "speaker label"):
validate_meeting_context(data)
def test_generated_context_yaml_is_deterministic_and_round_trips(self) -> None:
first = serialize_meeting_context_yaml(self.context)
second = serialize_meeting_context_yaml(self.context)
self.assertEqual(first, second)
SCRATCH_DIR.mkdir(exist_ok=True)
path = SCRATCH_DIR / "generated_context.yaml"
path.write_text(first, encoding="utf-8")
loaded = load_meeting_context(path)
self.assertEqual(loaded.data, self.context.data)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+164
View File
@@ -0,0 +1,164 @@
"""Meeting-language regression tests without media or model execution."""
import json
import tempfile
import unittest
from functools import partial
from pathlib import Path
from unittest.mock import Mock, patch
from src.meeting_lab.llm.ollama import OllamaGeneration
from src.meeting_lab.models.meeting_context import load_meeting_context
from src.meeting_lab.orchestration.mvp import regenerate_mvp_protocol
from src.meeting_lab.protocol.direct_protocol_prompt import build_direct_protocol_prompt
from src.meeting_lab.protocol.generate_direct_protocol import (
estimate_input_tokens,
generate_direct_protocol,
select_transcript_input,
)
class MeetingLanguageTests(unittest.TestCase):
def test_diarized_plain_fallback_retains_english_instruction(self) -> None:
segments = [
{
"start": index,
"end": index + 1,
"text": "Word.",
"speaker_id": f"SPEAKER_{index % 2:02d}",
}
for index in range(200)
]
plain = " ".join(segment["text"] for segment in segments)
budget = estimate_input_tokens(
build_direct_protocol_prompt(plain, meeting_language="en")
)
selected = select_transcript_input(
{"text": plain, "segments": segments, "speaker_labels_anonymous": True},
None,
meeting_language="en",
safe_input_token_budget=budget,
)
self.assertEqual(selected.representation, "plain_transcript_fallback")
self.assertIn("Write the meeting protocol in English.", selected.prompt)
self.assertEqual(selected.text, plain)
def test_generation_and_regeneration_preserve_language_and_inputs(self) -> None:
for language, expected in (
("de", "German"),
("en", "English"),
(None, "German"),
):
for diarized in (False, True):
with (
self.subTest(language=language, diarized=diarized),
tempfile.TemporaryDirectory() as directory,
):
run = Path(directory)
context_path = run / "context" / "meeting_context.yaml"
context_path.parent.mkdir()
context_text = (
'schema_version: "1"\nmeeting:\n'
" meeting_id: test\n title: Mixed terminology\n"
+ (f" language: {language}\n" if language else "")
+ ' notes: "Freigabe für Product X"\n'
"participants:\n - participant_id: person\n"
' display_name: "Jörg Müller"\n'
"speaker_mappings:\n SPEAKER_00: person\n"
)
context_path.write_text(context_text, encoding="utf-8")
transcript = run / ("diarization" if diarized else "transcript")
transcript.mkdir()
transcript /= (
"transcript_diarized.json" if diarized else "transcript.json"
)
transcript.write_text(
json.dumps(
{
"text": "I will check Product X.",
"speaker_labels_anonymous": diarized,
"segments": [
{
"id": 0,
"start": 0,
"end": 1,
"speaker_id": "SPEAKER_00",
"text": "I will check Product X.",
}
],
}
),
encoding="utf-8",
)
original = transcript.read_bytes()
call = Mock(
return_value=OllamaGeneration(
text="# Meeting Protocol",
raw_response={},
client_wall_time_seconds=0.1,
)
)
generate = partial(
generate_direct_protocol,
model_check=Mock(return_value={}),
generation_call=call,
)
result = generate(transcript, context_path)
self.assertIn(
f"Write the meeting protocol in {expected}.",
result.exact_prompt,
)
self.assertNotIn("in deutscher Sprache", result.exact_prompt)
self.assertNotIn("auf Deutsch", result.exact_prompt)
self.assertIn("Jörg Müller", result.exact_prompt)
self.assertIn("Mixed terminology", result.exact_prompt)
self.assertIn("I will check Product X.", result.transcript_input)
self.assertEqual(
result.runtime_metadata["output_language"], language or "de"
)
self.assertEqual(
context_path.read_text(encoding="utf-8"), context_text
)
context = load_meeting_context(context_path)
with (
patch(
"src.meeting_lab.orchestration.mvp.generate_direct_protocol",
side_effect=generate,
),
patch(
"src.meeting_lab.orchestration.mvp.transcribe_audio",
side_effect=AssertionError("Retranscription is forbidden"),
),
):
regenerate_mvp_protocol(run, meeting_context=context)
metadata = json.loads(
(run / "protocol" / "runtime_metadata.json").read_text()
)
self.assertEqual(metadata["output_language"], language or "de")
self.assertIn(
f"Write the meeting protocol in {expected}.",
call.call_args.args[2],
)
self.assertEqual(transcript.read_bytes(), original)
self.assertEqual(
load_meeting_context(context_path).data, context.data
)
self.assertEqual(context.speaker_mappings, {"SPEAKER_00": "person"})
def test_no_context_defaults_to_german(self) -> None:
with tempfile.TemporaryDirectory() as directory:
transcript = Path(directory) / "transcript.json"
transcript.write_text('{"text": "English source text."}')
result = generate_direct_protocol(
transcript,
model_check=Mock(return_value={}),
generation_call=Mock(
return_value=OllamaGeneration(
text="Protocol",
raw_response={},
client_wall_time_seconds=0.1,
)
),
)
self.assertIn("Write the meeting protocol in German.", result.exact_prompt)
self.assertEqual(result.runtime_metadata["output_language"], "de")
+310
View File
@@ -0,0 +1,310 @@
import json
import shutil
import subprocess
import tempfile
import unittest
from dataclasses import replace
from pathlib import Path
from unittest.mock import patch
from scripts import run_mvp_meeting as cli
from src.meeting_lab.audio import PreparedAudio
from src.meeting_lab.models.meeting_context import load_meeting_context
from src.meeting_lab.orchestration import mvp as mvp_api
from src.meeting_lab.orchestration.mvp import MvpMeetingConfig, MvpRunResult
from src.meeting_lab.protocol.generate_direct_protocol import DirectProtocolResult
from src.meeting_lab.transcription.whisper import TranscriptionError, TranscriptionResult
def context_data():
return {
"schema_version": "1",
"meeting": {
"meeting_id": "programmatic-test",
"title": "Programmatic Test",
"language": "de",
"date": None,
"objective": "API prüfen",
"notes": "",
},
"participants": [
{
"participant_id": "person-1",
"display_name": "Test Person",
"aliases": [],
"role": "Projektleitung",
"department": None,
"attendance_status": "present",
"notes": None,
}
],
"speaker_mappings": {"SPEAKER_00": "person-1"},
"mentioned_people": [],
"organization": {"name": "Example", "departments": []},
"known_entities": {},
"context_rules": {"do_not_infer_responsibilities": True},
}
def fake_transcribe(audio, model, output, language, **kwargs):
output.mkdir(parents=True, exist_ok=True)
raw = output / "whisper_raw.json"
transcript = output / "transcript.json"
text = output / "transcript.txt"
metadata = output / "runtime_metadata.json"
raw.write_text('{"transcription": []}\n', encoding="utf-8")
transcript.write_text(
json.dumps(
{
"text": "Ein kurzer Besprechungstext.",
"segments": [
{
"id": 0,
"start": 0.0,
"end": 1.0,
"text": "Ein kurzer Besprechungstext.",
}
],
}
),
encoding="utf-8",
)
text.write_text("Ein kurzer Besprechungstext.\n", encoding="utf-8")
metadata.write_text('{"runtime_seconds": 0.1}\n', encoding="utf-8")
return TranscriptionResult(output, raw, transcript, text, metadata, 0.1)
def fake_prepare(source, destination, **kwargs):
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source, destination)
return PreparedAudio(source, source.suffix.removeprefix("."), destination, "ffmpeg", "ffmpeg")
def fake_protocol(transcript, context, **kwargs):
rendered_context = load_meeting_context(context)
assert rendered_context.meeting_id == "programmatic-test"
return DirectProtocolResult(
protocol_text="# Meeting Protocol\n",
exact_prompt="prompt",
model_metadata={"model": kwargs["model"]},
runtime_metadata={"request_count": 1},
raw_response={"response": "# Meeting Protocol\n"},
)
class MvpApiTests(unittest.TestCase):
def config(self, root: Path):
audio = root / "meeting.wav"
model = root / "model.bin"
audio.write_bytes(b"audio")
model.write_bytes(b"model")
return MvpMeetingConfig(
audio_file=audio,
whisper_model=model,
output_root=root / "runs",
model="test:model",
)
def test_programmatic_context_is_persisted_without_source_yaml(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.config(root)
events = []
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe),
patch.object(mvp_api, "prepare_audio", side_effect=fake_prepare),
patch.object(
mvp_api, "generate_direct_protocol", side_effect=fake_protocol
) as protocol_generator,
patch.object(subprocess, "run") as subprocess_run,
):
result = mvp_api.run_mvp_meeting(
config, meeting_context=context_data(), progress_sink=events.append
)
self.assertEqual(result.exit_code, 0)
context_path = result.run_dir / "context/meeting_context.yaml"
self.assertTrue(context_path.is_file())
persisted = load_meeting_context(context_path)
self.assertEqual(persisted.meeting_id, "programmatic-test")
self.assertEqual(persisted.speaker_mappings, {"SPEAKER_00": "person-1"})
subprocess_run.assert_not_called()
self.assertEqual(
[(event.stage, event.status) for event in events],
[
("preparing", "started"),
("preparing", "completed"),
("transcription", "started"),
("transcription", "completed"),
("protocol_generation", "started"),
("protocol_generation", "completed"),
("completed", "completed"),
],
)
self.assertTrue(all(event.progress is None for event in events))
self.assertEqual(protocol_generator.call_args.kwargs["num_ctx"], 32_768)
self.assertEqual(
protocol_generator.call_args.kwargs["safe_input_token_budget"],
29_000,
)
def test_protocol_only_regeneration_reuses_diarized_artifacts(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
run_dir = root / "existing-run"
diarization_dir = run_dir / "diarization"
diarization_dir.mkdir(parents=True)
source = diarization_dir / "transcript_diarized.json"
source.write_text(
json.dumps(
{
"text": "SPEAKER_00: Existing statement.\n",
"segments": [
{
"start": 0.0,
"end": 1.0,
"speaker_id": "SPEAKER_00",
"text": "Existing statement.",
}
],
"speaker_labels_anonymous": True,
}
),
encoding="utf-8",
)
source_before = source.read_bytes()
mapped_context = context_data()
mapped_context["speaker_mappings"] = {"SPEAKER_00": "person-1"}
with (
patch.object(mvp_api, "prepare_audio") as preparation,
patch.object(mvp_api, "transcribe_audio") as transcription,
patch.object(mvp_api, "diarize_audio") as diarization,
patch.object(
mvp_api, "generate_direct_protocol", side_effect=fake_protocol
) as protocol,
):
result = mvp_api.regenerate_mvp_protocol(
run_dir,
meeting_context=mapped_context,
model="qwen3.8:27b",
protocol_num_ctx=32_768,
protocol_safe_input_token_budget=29_000,
)
self.assertEqual(result.exit_code, 0)
self.assertEqual(result.protocol_path, run_dir / "protocol.md")
preparation.assert_not_called()
transcription.assert_not_called()
diarization.assert_not_called()
self.assertEqual(protocol.call_args.args[0], source)
self.assertEqual(protocol.call_args.kwargs["num_ctx"], 32_768)
self.assertEqual(source.read_bytes(), source_before)
persisted = load_meeting_context(run_dir / "context/meeting_context.yaml")
self.assertEqual(persisted.speaker_mappings, {"SPEAKER_00": "person-1"})
def test_failure_emits_terminal_failure_event(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.config(root)
events = []
with patch.object(
mvp_api,
"transcribe_audio",
side_effect=TranscriptionError("stopped"),
), patch.object(mvp_api, "prepare_audio", side_effect=fake_prepare):
result = mvp_api.run_mvp_meeting(config, progress_sink=events.append)
self.assertEqual(result.exit_code, 2)
self.assertEqual(events[-1].stage, "failed")
self.assertEqual(events[-1].status, "failed")
self.assertIn("transcription", events[-1].message)
metadata = json.loads((result.run_dir / "run_metadata.json").read_text())
self.assertEqual(metadata["status"], "failed")
def test_cli_defaults_and_wrapper_delegate_without_subprocess(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.config(root)
args = cli.parse_args(
[str(config.audio_file), "--whisper-model", str(config.whisper_model)]
)
expected = MvpRunResult(0, root / "run", root / "run/protocol.md")
with patch.object(cli, "run_mvp_meeting", return_value=expected) as api:
actual = cli.run(args, context_override=context_data())
self.assertEqual(actual, (0, expected.run_dir, expected.protocol_path))
delegated = api.call_args.args[0]
self.assertEqual(delegated.diarization, "off")
self.assertEqual(delegated.language, "de")
self.assertEqual(delegated.whisper_executable, "whisper-cli")
self.assertEqual(delegated.ffmpeg_executable, "ffmpeg")
self.assertTrue(delegated.audio_normalization)
self.assertEqual(delegated.protocol_num_ctx, 32_768)
self.assertEqual(delegated.protocol_safe_input_token_budget, 29_000)
self.assertEqual(api.call_args.kwargs["meeting_context"], context_data())
def test_cli_explicit_audio_normalization_values_are_propagated(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.config(root)
enabled = cli.config_from_args(
cli.parse_args(
[
str(config.audio_file),
"--whisper-model",
str(config.whisper_model),
"--audio-normalization",
]
)
)
disabled = cli.config_from_args(
cli.parse_args(
[
str(config.audio_file),
"--whisper-model",
str(config.whisper_model),
"--no-audio-normalization",
]
)
)
self.assertTrue(enabled.audio_normalization)
self.assertFalse(disabled.audio_normalization)
def test_transcription_receives_prepared_wav_for_encoded_inputs(self):
for suffix in (".flac", ".m4a"):
with self.subTest(suffix=suffix), tempfile.TemporaryDirectory() as directory:
root = Path(directory)
config = self.config(root)
encoded = config.audio_file.with_suffix(suffix)
config.audio_file.rename(encoded)
config = replace(config, audio_file=encoded)
received = []
def capture_transcribe(audio, *args, received_paths=received, **kwargs):
received_paths.append(audio)
return fake_transcribe(audio, *args, **kwargs)
with (
patch.object(mvp_api, "prepare_audio", side_effect=fake_prepare),
patch.object(mvp_api, "transcribe_audio", side_effect=capture_transcribe),
patch.object(mvp_api, "generate_direct_protocol", side_effect=fake_protocol),
):
result = mvp_api.run_mvp_meeting(
config, meeting_context=context_data()
)
self.assertEqual(result.exit_code, 0)
self.assertEqual(received, [result.run_dir / "audio" / "prepared.wav"])
manifest = json.loads(
(result.run_dir / "audio" / "input_manifest.json").read_text()
)
self.assertEqual(manifest["format"], suffix.removeprefix("."))
self.assertEqual(
manifest["prepared_audio"]["prepared_audio_path"],
str((result.run_dir / "audio" / "prepared.wav").resolve()),
)
if __name__ == "__main__":
unittest.main()
+444
View File
@@ -0,0 +1,444 @@
import json
import shutil
import tempfile
import unittest
from datetime import datetime
from pathlib import Path
from unittest.mock import Mock, patch
from scripts import run_mvp_meeting
from src.meeting_lab.audio import PreparedAudio
from src.meeting_lab.audio.preparation import (
DEFAULT_NORMALIZATION_FILTER,
DEFAULT_NORMALIZATION_METHOD,
)
from src.meeting_lab.orchestration import mvp as mvp_api
from src.meeting_lab.protocol.generate_direct_protocol import DirectProtocolResult
from src.meeting_lab.diarization.backend import DiarizationResult
from src.meeting_lab.transcription.whisper import TranscriptionError, TranscriptionResult
VALID_CONTEXT = """schema_version: "1"
meeting:
meeting_id: "mvp-test"
title: "MVP Test"
language: "de"
participants: []
mentioned_people: []
organization:
departments: []
known_entities: {}
"""
def protocol_result(model: str = "chosen:model") -> DirectProtocolResult:
text = "# Protokoll\n\nUnverändert. \n"
return DirectProtocolResult(
protocol_text=text,
exact_prompt="exact prompt\n",
model_metadata={"model": model},
runtime_metadata={"model": model, "request_count": 1, "client_wall_time_seconds": 0.5},
raw_response={"response": text, "done": True},
transcript_input="selected transcript\n",
)
def fake_transcribe(
audio_path: Path,
model_path: Path,
output_dir: Path,
language: str,
*,
executable: str,
threads: str | int,
) -> TranscriptionResult:
output_dir.mkdir(parents=True, exist_ok=True)
raw = output_dir / "whisper_raw.json"
transcript = output_dir / "transcript.json"
text = output_dir / "transcript.txt"
metadata = output_dir / "runtime_metadata.json"
raw.write_text('{"transcription": []}\n', encoding="utf-8")
transcript.write_text(
json.dumps(
{
"text": "Besprechungstext.",
"segments": [
{"id": 0, "start": 0.0, "end": 1.0, "text": "Besprechungstext."}
],
}
)
+ "\n",
encoding="utf-8",
)
text.write_text("Besprechungstext.\n", encoding="utf-8")
metadata.write_text('{"runtime_seconds": 1.25}\n', encoding="utf-8")
return TranscriptionResult(output_dir, raw, transcript, text, metadata, 1.25)
def fake_diarize(audio_path, output_dir, device_mode, **kwargs):
output_dir.mkdir(parents=True, exist_ok=True)
metadata = {
"backend": "pyannote.audio",
"model": "pyannote/speaker-diarization-community-1",
"requested_device_mode": device_mode,
"actual_device": "cuda",
"device_name": "Fake GPU",
"runtime_seconds": 2.5,
"speaker_count": 1,
"credentials_persisted": False,
}
paths = {
"metadata": output_dir / "metadata.json",
"ordinary": output_dir / "diarization.rttm",
"exclusive": output_dir / "exclusive_diarization.rttm",
"turns": output_dir / "turns.json",
"exclusive_turns": output_dir / "exclusive_turns.json",
}
paths["metadata"].write_text(json.dumps(metadata), encoding="utf-8")
paths["ordinary"].write_text("", encoding="utf-8")
paths["exclusive"].write_text("", encoding="utf-8")
paths["turns"].write_text("[]", encoding="utf-8")
paths["exclusive_turns"].write_text(
json.dumps([{"start": 0, "end": 10, "speaker_id": "SPEAKER_00"}]),
encoding="utf-8",
)
return DiarizationResult(
output_dir,
paths["metadata"],
paths["ordinary"],
paths["exclusive"],
paths["turns"],
paths["exclusive_turns"],
metadata,
)
def fake_prepare(source, destination, **kwargs):
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source, destination)
normalization_enabled = kwargs.get("normalization_enabled", True)
return PreparedAudio(
source,
source.suffix.removeprefix("."),
destination,
"ffmpeg",
kwargs.get("ffmpeg_executable", "ffmpeg"),
normalization_enabled,
DEFAULT_NORMALIZATION_METHOD if normalization_enabled else None,
DEFAULT_NORMALIZATION_FILTER if normalization_enabled else None,
)
class MvpOrchestratorTests(unittest.TestCase):
def setUp(self) -> None:
patcher = patch.object(mvp_api, "prepare_audio", side_effect=fake_prepare)
self.prepare_audio = patcher.start()
self.addCleanup(patcher.stop)
def create_inputs(self, root: Path) -> tuple[Path, Path, Path]:
audio = root / "team meeting.wav"
whisper_model = root / "ggml-model.bin"
context = root / "source-context.yaml"
audio.write_bytes(b"audio")
whisper_model.write_bytes(b"model")
context.write_text(VALID_CONTEXT, encoding="utf-8")
return audio, whisper_model, context
def args(self, root: Path, extra: list[str] | None = None):
audio, whisper_model, context = self.create_inputs(root)
values = [
str(audio),
"--whisper-model", str(whisper_model),
"--context", str(context),
"--output-root", str(root / "runs"),
"--language", "de",
"--model", "chosen:model",
"--ollama-endpoint", "http://ollama.test:11434",
]
if extra:
values.extend(extra)
return run_mvp_meeting.parse_args(values)
def test_successful_full_orchestration_and_artifact_layout(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe) as whisper,
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
) as protocol,
):
code, run_dir, protocol_path = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
self.assertIsNotNone(run_dir)
self.assertEqual(protocol_path, run_dir / "protocol.md")
expected = {
"run_metadata.json",
"audio/input_manifest.json",
"audio/prepared.wav",
"audio/preparation_metadata.json",
"transcript/whisper_raw.json",
"transcript/transcript.json",
"transcript/transcript.txt",
"transcript/runtime_metadata.json",
"context/meeting_context.yaml",
"protocol/exact_prompt.txt",
"protocol/transcript_input.txt",
"protocol/raw_response.json",
"protocol/runtime_metadata.json",
"protocol.md",
}
self.assertTrue(all((run_dir / item).is_file() for item in expected))
metadata = json.loads((run_dir / "run_metadata.json").read_text())
self.assertEqual(metadata["status"], "completed")
self.assertEqual(metadata["model"], "chosen:model")
self.assertIsNone(metadata["failure"])
self.assertEqual(whisper.call_count, 1)
self.assertEqual(
whisper.call_args.args[0], run_dir / "audio" / "prepared.wav"
)
self.assertEqual(protocol.call_count, 1)
self.assertTrue(
self.prepare_audio.call_args.kwargs["normalization_enabled"]
)
preparation = json.loads(
(run_dir / "audio" / "preparation_metadata.json").read_text()
)
self.assertTrue(preparation["normalization_enabled"])
def test_cli_can_disable_audio_normalization_without_bypassing_preparation(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root, ["--no-audio-normalization"])
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe) as whisper,
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
),
):
code, run_dir, _ = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
self.prepare_audio.assert_called_once()
self.assertFalse(
self.prepare_audio.call_args.kwargs["normalization_enabled"]
)
self.assertEqual(
whisper.call_args.args[0], run_dir / "audio" / "prepared.wav"
)
preparation = json.loads(
(run_dir / "audio" / "preparation_metadata.json").read_text()
)
self.assertFalse(preparation["normalization_enabled"])
self.assertIsNone(preparation["normalization_method"])
self.assertIsNone(preparation["normalization_filter"])
def test_context_model_endpoint_and_whisper_options_are_forwarded(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(
root,
[
"--whisper-executable",
"/tools/whisper-cli",
"--ffmpeg-executable",
"/tools/ffmpeg",
"--threads",
"4",
],
)
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe) as whisper,
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
) as protocol,
):
code, run_dir, _ = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
self.assertEqual(whisper.call_args.args[3], "de")
self.assertEqual(whisper.call_args.kwargs["executable"], "/tools/whisper-cli")
self.assertEqual(whisper.call_args.kwargs["threads"], "4")
self.assertEqual(
self.prepare_audio.call_args.kwargs["ffmpeg_executable"],
"/tools/ffmpeg",
)
self.assertEqual(protocol.call_args.args[1], run_dir / "context/meeting_context.yaml")
self.assertEqual(protocol.call_args.kwargs["model"], "chosen:model")
self.assertEqual(
protocol.call_args.kwargs["endpoint"], "http://ollama.test:11434"
)
def test_diarization_is_off_by_default_and_preserves_protocol_input(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe),
patch.object(mvp_api, "diarize_audio") as diarization,
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
) as protocol,
):
code, run_dir, _ = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
diarization.assert_not_called()
self.assertEqual(protocol.call_args.args[0], run_dir / "transcript/transcript.json")
self.assertFalse((run_dir / "diarization").exists())
def test_diarization_cli_propagates_and_uses_derived_protocol_input(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(
root,
[
"--diarization", "gpu",
"--diarization-runtime", "container",
"--diarization-container-image", "rocm/test",
"--diarization-container-arg=--device=/dev/kfd",
],
)
with (
patch.object(
mvp_api, "transcribe_audio", side_effect=fake_transcribe
),
patch.object(
mvp_api, "diarize_audio", side_effect=fake_diarize
) as diarization,
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
) as protocol,
):
code, run_dir, _ = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
self.assertEqual(diarization.call_args.args[2], "gpu")
self.assertEqual(
diarization.call_args.args[0], run_dir / "audio" / "prepared.wav"
)
self.assertEqual(diarization.call_args.kwargs["runtime"], "container")
self.assertEqual(
diarization.call_args.kwargs["container_args"], ("--device=/dev/kfd",)
)
derived = run_dir / "diarization/transcript_diarized.json"
self.assertEqual(protocol.call_args.args[0], derived)
self.assertIn("SPEAKER_00", derived.read_text(encoding="utf-8"))
self.assertEqual(
json.loads((run_dir / "transcript/transcript.json").read_text())["text"],
"Besprechungstext.",
)
run_metadata = json.loads((run_dir / "run_metadata.json").read_text())
self.assertTrue(run_metadata["diarization"]["enabled"])
self.assertNotIn("HF_TOKEN", json.dumps(run_metadata))
def test_whisper_failure_is_recorded_and_protocol_is_not_called(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
with (
patch.object(
mvp_api,
"transcribe_audio",
side_effect=TranscriptionError("whisper stopped"),
),
patch.object(mvp_api, "generate_direct_protocol") as protocol,
):
code, run_dir, protocol_path = run_mvp_meeting.run(args)
metadata = json.loads((run_dir / "run_metadata.json").read_text())
self.assertEqual(code, 2)
self.assertIsNone(protocol_path)
self.assertEqual(metadata["status"], "failed")
self.assertEqual(metadata["failure"]["stage"], "whisper")
self.assertIn("whisper stopped", metadata["failure"]["message"])
protocol.assert_not_called()
self.assertTrue((run_dir / "audio/input_manifest.json").is_file())
def test_protocol_failure_preserves_transcript_and_failure_metadata(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe),
patch.object(
mvp_api,
"generate_direct_protocol",
side_effect=ValueError("generation stopped"),
),
):
code, run_dir, protocol_path = run_mvp_meeting.run(args)
metadata = json.loads((run_dir / "run_metadata.json").read_text())
self.assertEqual(code, 2)
self.assertIsNone(protocol_path)
self.assertEqual(metadata["failure"]["stage"], "protocol")
self.assertTrue((run_dir / "transcript/transcript.json").is_file())
self.assertFalse((run_dir / "protocol.md").exists())
def test_unique_run_directories_do_not_overwrite(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
fixed = datetime(2026, 8, 21, 9, 15, 30)
first = run_mvp_meeting.create_unique_run_dir(root, "team meeting", lambda: fixed)
marker = first / "keep.txt"
marker.write_text("keep", encoding="utf-8")
second = run_mvp_meeting.create_unique_run_dir(root, "team meeting", lambda: fixed)
self.assertEqual(first.name, "team_meeting_20260821_091530")
self.assertEqual(second.name, "team_meeting_20260821_091530_01")
self.assertEqual(marker.read_text(), "keep")
def test_semantic_pipeline_functions_are_never_invoked(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
with (
patch.object(mvp_api, "transcribe_audio", side_effect=fake_transcribe),
patch.object(
mvp_api,
"generate_direct_protocol",
return_value=protocol_result(),
),
patch("src.meeting_lab.chunking.chunk_transcript.build_chunks") as chunking,
patch("src.meeting_lab.extraction.extract_chunks.extract_input") as extraction,
patch("src.meeting_lab.consolidation.consolidate_facts.call_ollama") as consolidation,
):
code, _, _ = run_mvp_meeting.run(args)
self.assertEqual(code, 0)
chunking.assert_not_called()
extraction.assert_not_called()
consolidation.assert_not_called()
def test_main_returns_nonzero_for_whisper_failure(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
args = self.args(root)
argv = [
str(args.audio_file),
"--whisper-model", str(args.whisper_model),
"--context", str(args.context),
"--output-root", str(args.output_root),
]
with patch.object(
mvp_api,
"transcribe_audio",
side_effect=TranscriptionError("failed"),
):
self.assertEqual(run_mvp_meeting.main(argv), 2)
if __name__ == "__main__":
unittest.main()
+162
View File
@@ -0,0 +1,162 @@
import argparse
import json
import tempfile
import unittest
from copy import deepcopy
from pathlib import Path
from unittest.mock import patch
import src.meeting_lab.controlled_semantic_derivation.experiment_negative_act as module
from src.meeting_lab.controlled_semantic_derivation.experiment_negative_act import (
DerivationValidationError,
build_ollama_payload,
build_prompt,
evaluate_case,
load_gold_cases,
parse_model_json,
run_experiment,
validate_classification,
)
GOLD_PATH = Path("tests/gold/negative_act_form_v0/cases.json")
FORM_TEXT = {
"NA-01": "Zusammenarbeit mit Dr. Schlummer fortsetzen",
"NA-02": "externe Lösung weiterverfolgen",
"NA-03": "reale Anlage für den Versuch nutzen",
"NA-04": "reale Anlage verwenden",
"NA-05": "Waschstufe einbauen",
}
def classification_for(case):
expected = case["expected"]
return {
"observation_id": expected["observation_id"],
"negative_act_form": expected["negative_act_form"],
"normalized_action_text": FORM_TEXT.get(case["case_id"]),
}
class NegativeActFormExperimentTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.cases = load_gold_cases(GOLD_PATH)
cls.by_id = {case["case_id"]: case for case in cls.cases}
def test_fixture_contains_exactly_na_01_through_na_08(self):
self.assertEqual(list(self.by_id), [f"NA-{number:02d}" for number in range(1, 9)])
def test_exact_schema_is_accepted(self):
case = self.by_id["NA-01"]
self.assertEqual(validate_classification(classification_for(case), case["observations"]), classification_for(case))
def test_unknown_field_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["explanation"] = "extra"
with self.assertRaisesRegex(DerivationValidationError, "unknown keys"):
validate_classification(classification, case["observations"])
def test_invalid_enum_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["negative_act_form"] = "rejection"
with self.assertRaisesRegex(DerivationValidationError, "unsupported value"):
validate_classification(classification, case["observations"])
def test_non_none_requires_normalized_action_text(self):
case = self.by_id["NA-01"]
for value in (None, ""):
classification = classification_for(case)
classification["normalized_action_text"] = value
with self.subTest(value=value), self.assertRaises(DerivationValidationError):
validate_classification(classification, case["observations"])
def test_none_requires_null_normalized_action_text(self):
case = self.by_id["NA-06"]
classification = classification_for(case)
self.assertIsNone(classification["normalized_action_text"])
classification["normalized_action_text"] = "Material einsetzen"
with self.assertRaisesRegex(DerivationValidationError, "requires null"):
validate_classification(classification, case["observations"])
def test_forbidden_normative_fields_are_rejected_recursively(self):
case = self.by_id["NA-01"]
fields = (
"rejection_form", "explicitly_rejected", "status", "decision", "outcome",
"topic_status", "responsible_person", "responsibility", "owner",
"requested_actor", "action_item", "protocol_category", "confidence",
"relation", "relations", "graph", "unresolved_issue",
)
for field in fields:
classification = classification_for(case)
classification["wrapper"] = {field: "forbidden"}
with self.subTest(field=field), self.assertRaisesRegex(DerivationValidationError, "forbidden semantic keys"):
validate_classification(classification, case["observations"])
def test_unknown_observation_id_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["observation_id"] = "obs_99"
with self.assertRaisesRegex(DerivationValidationError, "unknown observation"):
validate_classification(classification, case["observations"])
def test_malformed_json_is_rejected(self):
with self.assertRaises(json.JSONDecodeError):
parse_model_json("{bad json")
def test_all_expected_classifications_evaluate_as_pass(self):
for case in self.cases:
evaluation = evaluate_case(case, classification_for(case))
with self.subTest(case=case["case_id"]):
self.assertEqual(evaluation["classification"], "PASS")
def test_fixed_prompt_contains_candidate_and_no_gold_expectation(self):
prompt = build_prompt(self.by_id["NA-03"])
self.assertIn("candidate observation is obs_2", prompt)
self.assertNotIn("expected", prompt)
self.assertNotIn("Who is responsible", prompt)
def test_fixed_model_configuration(self):
payload = build_ollama_payload("qwen3.5:9B", "prompt", 16384, 1024)
self.assertFalse(payload["think"])
self.assertFalse(payload["stream"])
self.assertEqual(payload["options"]["temperature"], 0)
def test_no_rejection_or_status_derivation_function_exists(self):
public_names = {name for name in dir(module) if not name.startswith("_")}
self.assertNotIn("derive_rejection", public_names)
self.assertFalse(any(name.startswith("derive_") for name in public_names))
def test_artifacts_preserve_semantic_classification_only(self):
case = self.by_id["NA-01"]
raw = json.dumps(classification_for(case), ensure_ascii=False)
with tempfile.TemporaryDirectory() as temporary:
output = Path(temporary) / "run"
args = argparse.Namespace(
cases=GOLD_PATH, output=output, model="qwen3.5:9B",
endpoint="http://unused", timeout=1, num_ctx=16384, num_predict=1024,
)
with patch.object(module, "load_gold_cases", return_value=[deepcopy(case)]), patch.object(
module, "call_ollama", return_value=(raw, {"model": "qwen3.5:9B"})
):
summary = run_experiment(args)
self.assertEqual(summary["successful_llm_call_count"], 1)
case_dir = output / "na-01"
for filename in (
"v3_style_input_observations.json", "prompt.txt", "raw_model_response.txt",
"parsed_semantic_classification.json", "structural_validation.json",
"evaluation.json", "ollama_metadata.json",
):
self.assertTrue((case_dir / filename).is_file(), filename)
self.assertFalse((case_dir / "final_derived_result.json").exists())
self.assertFalse((case_dir / "deterministic_gate_results.json").exists())
parsed = json.loads((case_dir / "parsed_semantic_classification.json").read_text())
self.assertEqual(set(parsed), {"observation_id", "negative_act_form", "normalized_action_text"})
if __name__ == "__main__":
unittest.main()
+315
View File
@@ -0,0 +1,315 @@
import json
import tempfile
import unittest
from pathlib import Path
from unittest.mock import Mock, patch
from src.meeting_lab.diarization.alignment import diarized_transcript_text
from src.meeting_lab.llm import ollama
from src.meeting_lab.llm.ollama import OllamaGeneration
from src.meeting_lab.protocol.direct_protocol_prompt import (
COMPACT_DIARIZED_PROTOCOL_INSTRUCTION,
build_direct_protocol_prompt,
)
from src.meeting_lab.protocol.generate_direct_protocol import (
DirectProtocolError,
estimate_input_tokens,
generate_direct_protocol,
)
from src.meeting_lab.protocol.transcript_input import compact_diarized_transcript
def segments() -> list[dict[str, object]]:
return [
{"id": 0, "start": 0.0, "end": 1.0, "text": "First.", "speaker_id": "SPEAKER_01"},
{"id": 1, "start": 1.0, "end": 2.0, "text": "Second.", "speaker_id": "SPEAKER_01"},
{"id": 2, "start": 2.0, "end": 3.0, "text": "Third.", "speaker_id": "SPEAKER_04"},
{"id": 3, "start": 3.0, "end": 4.0, "text": "Unassigned.", "speaker_id": None},
{"id": 4, "start": 4.0, "end": 5.0, "text": "Last.", "speaker_id": "SPEAKER_01"},
]
def diarized_document(repetitions: int = 1) -> dict[str, object]:
source = segments() * repetitions
return {
"text": diarized_transcript_text(source),
"segments": source,
"speaker_labels_anonymous": True,
"alignment_source": "exclusive_diarization",
}
def completed_generation() -> OllamaGeneration:
text = "# Meeting Protocol\n\nComplete."
return OllamaGeneration(
raw_response={
"response": text,
"done": True,
"done_reason": "stop",
"prompt_eval_count": 100,
"eval_count": 10,
},
text=text,
client_wall_time_seconds=0.1,
)
class CompactDiarizedTranscriptTests(unittest.TestCase):
def test_adjacent_segments_group_and_transitions_remain_separate(self) -> None:
compact = compact_diarized_transcript(segments())
self.assertEqual(
[block.speaker_id for block in compact.blocks],
["SPEAKER_01", "SPEAKER_04", "SPEAKER_UNASSIGNED", "SPEAKER_01"],
)
self.assertEqual(compact.blocks[0].segment_texts, ("First.", "Second."))
self.assertEqual(compact.blocks[-1].segment_texts, ("Last.",))
self.assertEqual(compact.text.count("SPEAKER_01:"), 2)
def test_every_segment_text_and_order_are_preserved(self) -> None:
source = segments()
compact = compact_diarized_transcript(source)
self.assertEqual(compact.source_segment_count, len(source))
self.assertEqual(compact.represented_segment_count, len(source))
self.assertEqual(
compact.segment_texts,
tuple(str(segment["text"]) for segment in source),
)
self.assertEqual(compact.segment_texts[0], "First.")
self.assertEqual(compact.segment_texts[-1], "Last.")
self.assertIn("SPEAKER_UNASSIGNED: Unassigned.", compact.text)
def test_compact_form_is_materially_smaller_than_per_segment_format(self) -> None:
source = [
{
"start": index,
"end": index + 1,
"text": "Repeated transcript content.",
"speaker_id": "SPEAKER_01",
}
for index in range(100)
]
compact = compact_diarized_transcript(source).text
verbose = diarized_transcript_text(source)
self.assertLess(len(compact), len(verbose) * 0.6)
class ProtocolInputBudgetTests(unittest.TestCase):
def _write(self, root: Path, document: dict[str, object]) -> Path:
path = root / "transcript.json"
path.write_text(json.dumps(document), encoding="utf-8")
return path
def test_token_estimate_uses_utf8_bytes_for_non_ascii_safety(self) -> None:
self.assertEqual(estimate_input_tokens("ä" * 44), 20)
def test_compact_diarized_representation_selected_within_budget(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
document = diarized_document()
transcript = self._write(root, document)
compact = compact_diarized_transcript(document["segments"]).text
budget = estimate_input_tokens(
build_direct_protocol_prompt(
compact,
instruction=COMPACT_DIARIZED_PROTOCOL_INSTRUCTION,
)
)
call = Mock(return_value=completed_generation())
result = generate_direct_protocol(
transcript,
safe_input_token_budget=budget,
model_check=Mock(return_value={}),
generation_call=call,
)
self.assertEqual(
result.runtime_metadata["selected_transcript_representation"],
"diarized_compact",
)
self.assertFalse(result.runtime_metadata["fallback_used"])
self.assertTrue(result.runtime_metadata["diarization_enabled"])
self.assertEqual(result.runtime_metadata["safe_input_token_budget"], budget)
self.assertEqual(result.transcript_input, compact)
self.assertEqual(call.call_count, 1)
def test_mapped_speakers_and_statements_reach_final_ollama_payload(self) -> None:
diarized = {
"text": "",
"segments": [
{
"start": 0.0,
"end": 1.0,
"speaker_id": "SPEAKER_00",
"text": "We will run the trial on Wednesday.",
},
{
"start": 1.0,
"end": 2.0,
"speaker_id": "SPEAKER_01",
"text": "I will prepare the raw materials before then.",
},
{
"start": 2.0,
"end": 3.0,
"speaker_id": "SPEAKER_00",
"text": "Good. Anna owns the material preparation.",
},
],
"speaker_labels_anonymous": True,
"alignment_source": "exclusive_diarization",
}
context = {
"schema_version": "1",
"meeting": {
"meeting_id": "speaker-test",
"title": "Speaker test",
"language": "en",
},
"participants": [
{"participant_id": "martin", "display_name": "Martin"},
{"participant_id": "anna", "display_name": "Anna"},
],
"speaker_mappings": {"SPEAKER_00": "martin", "SPEAKER_01": "anna"},
"mentioned_people": [],
"organization": {"departments": []},
"known_entities": {},
}
response = Mock()
response.raise_for_status.return_value = None
response.json.return_value = {"response": "# Meeting Protocol\n", "done": True}
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = self._write(root, diarized)
context_path = root / "context.yaml"
context_path.write_text(json.dumps(context), encoding="utf-8")
with patch.object(ollama.requests, "post", return_value=response) as post:
result = generate_direct_protocol(
transcript,
context_path,
model="qwen3.8:27b",
model_check=Mock(return_value={}),
)
prompt = post.call_args.kwargs["json"]["prompt"]
self.assertEqual(result.exact_prompt, prompt)
self.assertIn("- SPEAKER_00: Martin (participant_id: martin)", prompt)
self.assertIn("- SPEAKER_01: Anna (participant_id: anna)", prompt)
self.assertIn("SPEAKER_00: We will run the trial on Wednesday.", prompt)
self.assertIn(
"SPEAKER_01: I will prepare the raw materials before then.", prompt
)
self.assertIn("SPEAKER_00: Good. Anna owns the material preparation.", prompt)
self.assertNotIn("Martin: We will run the trial on Wednesday.", prompt)
self.assertNotIn("Anna: I will prepare the raw materials before then.", prompt)
self.assertIn("autoritativen SPEAKER_XX-zu-Teilnehmer-Zuordnungen", prompt)
self.assertIn("Ich-Zusage", prompt)
self.assertIn("nur erwähnten Personen", prompt)
self.assertIn("keine persönliche Verantwortung", prompt)
def test_plain_fallback_selected_when_diarized_compact_exceeds_budget(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
alternating_segments = [
{
"id": index,
"start": float(index),
"end": float(index + 1),
"text": "Word.",
"speaker_id": f"SPEAKER_{index % 2:02d}",
}
for index in range(200)
]
document = {
"text": diarized_transcript_text(alternating_segments),
"segments": alternating_segments,
"speaker_labels_anonymous": True,
"alignment_source": "exclusive_diarization",
}
transcript = self._write(root, document)
compact = compact_diarized_transcript(document["segments"]).text
plain = " ".join(str(segment["text"]) for segment in document["segments"])
compact_estimate = estimate_input_tokens(
build_direct_protocol_prompt(
compact,
instruction=COMPACT_DIARIZED_PROTOCOL_INSTRUCTION,
)
)
plain_estimate = estimate_input_tokens(build_direct_protocol_prompt(plain))
self.assertLess(plain_estimate, compact_estimate)
result = generate_direct_protocol(
transcript,
safe_input_token_budget=plain_estimate,
model_check=Mock(return_value={}),
generation_call=Mock(return_value=completed_generation()),
)
self.assertEqual(
result.runtime_metadata["selected_transcript_representation"],
"plain_transcript_fallback",
)
self.assertTrue(result.runtime_metadata["fallback_used"])
self.assertEqual(result.transcript_input, plain)
self.assertNotIn("SPEAKER_00", result.exact_prompt)
self.assertNotIn("SPEAKER_01", result.exact_prompt)
self.assertFalse(result.runtime_metadata["speaker_attribution_available"])
self.assertEqual(
result.runtime_metadata["speaker_attribution_loss_reason"],
"plain_transcript_fallback",
)
def test_oversized_plain_transcript_fails_before_any_network_call(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = self._write(
root,
{"text": "large input " * 1000, "segments": []},
)
model_check = Mock()
generation_call = Mock()
with self.assertRaisesRegex(
DirectProtocolError,
"No LLM request was made; silent truncation is not allowed",
):
generate_direct_protocol(
transcript,
safe_input_token_budget=1,
model_check=model_check,
generation_call=generation_call,
)
model_check.assert_not_called()
generation_call.assert_not_called()
def test_existing_plain_path_and_metadata_remain_direct(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
transcript = self._write(
root,
{"text": "Plain original transcript.", "segments": []},
)
result = generate_direct_protocol(
transcript,
model_check=Mock(return_value={}),
generation_call=Mock(return_value=completed_generation()),
)
self.assertEqual(result.transcript_input, "Plain original transcript.")
self.assertEqual(
result.runtime_metadata["selected_transcript_representation"],
"plain_transcript",
)
self.assertFalse(result.runtime_metadata["fallback_used"])
self.assertFalse(result.runtime_metadata["diarization_enabled"])
self.assertIsNone(result.runtime_metadata["speaker_attribution_available"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,51 @@
import argparse,copy,json,tempfile,unittest
from pathlib import Path
from unittest.mock import Mock
import src.meeting_lab.controlled_semantic_derivation.experiment_target_normalization as module
from src.meeting_lab.controlled_semantic_derivation.experiment_h import DerivationValidationError
CASES=module.load_cases(Path("tests/gold/target_normalization_v0/cases.json")); BY={c["case_id"]:c for c in CASES}
def output(case,text=None):
link=case["fixed_linkage"]; return {"candidate_observation_id":link["candidate_observation_id"],"target_observation_id":link["target_observation_id"],"normalized_target_text":text or case["expected"]["normalized_target_text"]}
class TargetNormalizationTests(unittest.TestCase):
def test_exact_id_copying_accepted(self):
for case in CASES: self.assertEqual(module.validate_output(output(case),case),output(case))
def test_changed_candidate_rejected(self):
case=BY["TN-02"]; data=output(case); data["candidate_observation_id"]="obs_1"
with self.assertRaises(DerivationValidationError): module.validate_output(data,case)
def test_changed_target_rejected(self):
case=BY["TN-02"]; data=output(case); data["target_observation_id"]="obs_2"
with self.assertRaises(DerivationValidationError): module.validate_output(data,case)
def test_empty_and_null_text_rejected(self):
case=BY["TN-01"]
for value in ("",None):
data=output(case); data["normalized_target_text"]=value
with self.assertRaises(DerivationValidationError): module.validate_output(data,case)
def test_unknown_and_forbidden_fields_rejected(self):
case=BY["TN-01"]
for extra in ({"extra":1},{"status":"x"},{"nested":{"decision":True}}):
data=output(case); data.update(extra)
with self.assertRaises(DerivationValidationError): module.validate_output(data,case)
def test_true_schema_fixes_both_ids_and_disallows_null(self):
case=BY["TN-02"]; schema=module.output_schema(case); self.assertEqual(schema["properties"]["candidate_observation_id"]["const"],"obs_2"); self.assertEqual(schema["properties"]["target_observation_id"]["const"],"obs_1"); self.assertEqual(schema["properties"]["normalized_target_text"]["type"],"string"); self.assertFalse(schema["additionalProperties"])
def test_payload_uses_schema_object(self):
schema=module.output_schema(BY["TN-01"]); payload=module.build_payload("qwen3.5:9B","p",schema,16384,1024); self.assertIs(payload["format"],schema); self.assertIsInstance(payload["format"],dict)
def test_duplicate_ids_and_evidence_rejected(self):
for field in ("observation_id","evidence_id"):
case=copy.deepcopy(BY["TN-02"]); case["observations"][1][field]=case["observations"][0][field]
with self.assertRaises(DerivationValidationError): module.validate_linkage(case)
def test_prompt_is_fixed_normalization_only(self):
first=module.build_prompt(BY["TN-01"]); second=module.build_prompt(BY["TN-02"]); self.assertIn("do not perform target selection",first); self.assertIn("concrete POSITIVE action",first); self.assertEqual(first.split("Fixed candidate_observation_id:")[0],second.split("Fixed candidate_observation_id:")[0])
def test_no_target_selection_or_rejection_derivation_exists(self):
self.assertFalse(hasattr(module,"select_target")); self.assertFalse(hasattr(module,"derive")); self.assertNotIn("explicitly_rejected",module.OUTPUT_KEYS); self.assertNotIn("status",module.OUTPUT_KEYS)
def test_artifacts_preserve_fixed_linkage(self):
case=BY["TN-01"]; fixture={"schema_version":module.SCHEMA_VERSION,"cases":[case]}; caller=Mock(return_value=(json.dumps(output(case)),{"model":"qwen3.5:9B"}))
with tempfile.TemporaryDirectory() as tmp:
root=Path(tmp); path=root/"cases.json"; path.write_text(json.dumps(fixture)); out=root/"out"; args=argparse.Namespace(cases=path,output=out,endpoint="x",model="qwen3.5:9B",timeout=1,num_ctx=16384,num_predict=1024); summary=module.run(args,caller); self.assertEqual(summary["llm_call_count"],1); self.assertEqual(json.loads((out/"tn-01"/"fixed_linkage.json").read_text()),case["fixed_linkage"]); self.assertTrue((out/"tn-01"/"normalized_target_result.json").exists())
def test_negative_polarity_fails_evaluation(self):
case=BY["TN-01"]; self.assertEqual(module.evaluate(case,output(case,"Mit Dr. Schlummer arbeiten wir nicht weiter"))["classification"],"FAIL")
def test_material_scope_and_alternative_contract(self):
self.assertEqual(module.evaluate(BY["TN-03"],output(BY["TN-03"]))["classification"],"PASS"); self.assertEqual(module.evaluate(BY["TN-04"],output(BY["TN-04"]))["classification"],"PASS")
if __name__=="__main__": unittest.main()
@@ -0,0 +1,75 @@
import argparse, copy, json, tempfile, unittest
from pathlib import Path
from unittest.mock import Mock
from src.meeting_lab.controlled_semantic_derivation.experiment_h import DerivationValidationError
import src.meeting_lab.controlled_semantic_derivation.experiment_target_resolution as module
GOLD=Path("tests/gold/target_resolution_v0/cases.json")
CASES=module.load_cases(GOLD); BY_ID={c["case_id"]:c for c in CASES}
def target(case, target_id=None, text="konkrete Zielhandlung"):
return {"candidate_observation_id":case["negative_act"]["observation_id"],"target_observation_id":target_id if target_id is not None else case["expected"]["target_observation_id"],"normalized_target_text":text}
class TargetResolutionTests(unittest.TestCase):
def test_eligibility_enum_boundary(self):
for cid in ("TR-01","TR-02","TR-03","TR-04"):
self.assertTrue(module.eligibility(BY_ID[cid]["negative_act"],BY_ID[cid]["observations"])["eligible_for_target_resolution"])
for cid in ("TR-05","TR-06","TR-07","TR-08"):
gate=module.eligibility(BY_ID[cid]["negative_act"],BY_ID[cid]["observations"])
self.assertFalse(gate["eligible_for_target_resolution"]); self.assertEqual(gate["reason"],"negative_act_form_not_explicit_non_pursuit")
def test_ineligible_cases_never_call_resolver_and_record_skip(self):
fixture={"schema_version":module.SCHEMA_VERSION,"cases":[BY_ID[x] for x in ("TR-05","TR-06","TR-07","TR-08")]}; resolver=Mock()
with tempfile.TemporaryDirectory() as tmp:
root=Path(tmp); cases=root/"cases.json"; cases.write_text(json.dumps(fixture)); out=root/"out"
summary=module.run(argparse.Namespace(cases=cases,output=out,endpoint="x",model="qwen3.5:9B",timeout=1,num_ctx=16384,num_predict=1024),resolver)
self.assertEqual(summary["target_resolution_llm_call_count"],0); resolver.assert_not_called()
for cid in ("tr-05","tr-06","tr-07","tr-08"):
skipped=json.loads((out/cid/"target_resolution_skipped.json").read_text()); self.assertFalse(skipped["call_made"])
def test_self_contained_target_equals_candidate(self):
c=BY_ID["TR-01"]; self.assertEqual(module.validate_target(target(c,text="Zusammenarbeit mit Dr. Schlummer fortsetzen"),c["observations"],"obs_1")["target_observation_id"],"obs_1")
def test_paired_target_precedes_candidate(self):
c=BY_ID["TR-02"]; self.assertEqual(module.validate_target(target(c,text="externe Lösung weiterverfolgen"),c["observations"],"obs_2")["target_observation_id"],"obs_1")
def test_target_after_candidate_rejected(self):
c=copy.deepcopy(BY_ID["TR-02"]); data={"candidate_observation_id":"obs_1","target_observation_id":"obs_2","normalized_target_text":"x"}
with self.assertRaises(DerivationValidationError): module.validate_target(data,c["observations"],"obs_1")
def test_unknown_candidate_and_target_rejected(self):
c=BY_ID["TR-02"]
with self.assertRaises(DerivationValidationError): module.validate_target({"candidate_observation_id":"missing","target_observation_id":"obs_1","normalized_target_text":"x"},c["observations"],"missing")
with self.assertRaises(DerivationValidationError): module.validate_target({"candidate_observation_id":"obs_2","target_observation_id":"missing","normalized_target_text":"x"},c["observations"],"obs_2")
def test_duplicate_observation_and_evidence_ids_rejected(self):
for field in ("observation_id","evidence_id"):
obs=copy.deepcopy(BY_ID["TR-02"]["observations"]); obs[1][field]=obs[0][field]
with self.assertRaises(DerivationValidationError): module.validate_observations(obs)
def test_null_and_non_null_text_constraints(self):
c=BY_ID["TR-02"]
valid={"candidate_observation_id":"obs_2","target_observation_id":None,"normalized_target_text":None}; self.assertEqual(module.validate_target(valid,c["observations"],"obs_2"),valid)
for bad in ({"candidate_observation_id":"obs_2","target_observation_id":None,"normalized_target_text":"x"},{"candidate_observation_id":"obs_2","target_observation_id":"obs_1","normalized_target_text":""}):
with self.assertRaises(DerivationValidationError): module.validate_target(bad,c["observations"],"obs_2")
def test_unknown_and_recursive_forbidden_fields_rejected(self):
c=BY_ID["TR-02"]
for extra in ({"extra":1},{"nested":{"status":"rejected"}}):
data=target(c,text="x"); data.update(extra)
with self.assertRaises(DerivationValidationError): module.validate_target(data,c["observations"],"obs_2")
def test_self_contained_prompt_fixes_linkage_deterministically(self):
prompt=module.build_prompt(BY_ID["TR-01"]); self.assertIn("deterministically fixed to obs_1",prompt); self.assertIn("only normalize",prompt)
def test_ineligible_prompt_is_impossible(self):
with self.assertRaises(DerivationValidationError): module.build_prompt(BY_ID["TR-05"])
def test_experiment_has_no_rejection_derivation(self):
self.assertFalse(hasattr(module,"derive")); self.assertNotIn("explicitly_rejected",module.TARGET_KEYS); self.assertNotIn("status",module.TARGET_KEYS)
def test_evaluation_preserves_scope_and_alternative_contract(self):
c=BY_ID["TR-04"]; gate=module.eligibility(c["negative_act"],c["observations"]); result=module.evaluate(c,gate,True,target(c,text="Versuch in der realen Anlage durchführen")); self.assertEqual(result["classification"],"PASS"); self.assertTrue(result["alternative_isolation"])
if __name__=="__main__": unittest.main()
@@ -0,0 +1,59 @@
import argparse,copy,json,tempfile,unittest
from pathlib import Path
from unittest.mock import Mock
import src.meeting_lab.controlled_semantic_derivation.experiment_target_resolution_v1 as module
from src.meeting_lab.controlled_semantic_derivation.experiment_h import DerivationValidationError
CASES=module.load_cases(Path("tests/gold/target_resolution_v1/cases.json")); BY={c["case_id"]:c for c in CASES}
def self_output(text="Zusammenarbeit mit Dr. Schlummer fortsetzen"): return {"candidate_observation_id":"obs_1","normalized_target_text":text}
def paired(case,target="obs_1",text="externe Lösung weiterverfolgen"): return {"candidate_observation_id":case["negative_act"]["observation_id"],"target_observation_id":target,"normalized_target_text":text}
class TargetResolutionV1Tests(unittest.TestCase):
def test_self_linkage_is_deterministic_and_equals_candidate(self):
result=module.deterministic_self_link(BY["TR1-V1"]); self.assertEqual(result["linkage_source"],"deterministic"); self.assertEqual(result["target_observation_id"],result["candidate_observation_id"])
def test_self_llm_output_has_no_target_id(self):
self.assertEqual(module.SELF_KEYS,{"candidate_observation_id","normalized_target_text"}); self.assertNotIn("target_observation_id",module.output_schema(BY["TR1-V1"])["properties"])
def test_self_combination_records_link_and_normalization_separately(self):
combined=module.combine(BY["TR1-V1"],self_output()); self.assertEqual(combined["target_observation_id"],"obs_1"); self.assertIn("fortsetzen",combined["normalized_target_text"])
def test_self_link_rejects_paired_strategy(self):
with self.assertRaises(DerivationValidationError): module.deterministic_self_link(BY["TR2-V1"])
def test_paired_schema_enumerates_allowed_ids_and_null(self):
schema=module.output_schema(BY["TR2-V1"]); self.assertEqual(schema["properties"]["target_observation_id"]["enum"],["obs_1","obs_2",None]); self.assertFalse(schema["additionalProperties"])
def test_allowed_obs1_and_obs2_are_structurally_accepted(self):
case=BY["TR2-V1"]
module.validate_semantic_output(paired(case,"obs_1"),case); module.validate_semantic_output(paired(case,"obs_2"),case)
def test_unknown_and_string_null_targets_rejected(self):
case=BY["TR2-V1"]
for value in ("obs_9","null"):
with self.assertRaises(DerivationValidationError): module.validate_semantic_output(paired(case,value),case)
def test_json_null_accepted_and_requires_null_text(self):
case=BY["TR2-V1"]; valid=paired(case,None,None); self.assertEqual(module.validate_semantic_output(valid,case),valid)
with self.assertRaises(DerivationValidationError): module.validate_semantic_output(paired(case,None,"x"),case)
def test_non_null_requires_nonempty_text(self):
with self.assertRaises(DerivationValidationError): module.validate_semantic_output(paired(BY["TR2-V1"],"obs_1",""),BY["TR2-V1"])
def test_target_after_candidate_rejected(self):
case=copy.deepcopy(BY["TR2-V1"]); case["negative_act"]["observation_id"]="obs_1"
with self.assertRaises(DerivationValidationError): module.validate_semantic_output({"candidate_observation_id":"obs_1","target_observation_id":"obs_2","normalized_target_text":"x"},case)
def test_duplicate_ids_and_evidence_rejected(self):
for field in ("observation_id","evidence_id"):
obs=copy.deepcopy(BY["TR2-V1"]["observations"]); obs[1][field]=obs[0][field]
with self.assertRaises(DerivationValidationError): module.validate_observations(obs)
def test_forbidden_and_unknown_fields_rejected(self):
case=BY["TR2-V1"]
for extra in ({"status":"x"},{"nested":{"decision":True}},{"extra":1}):
data=paired(case); data.update(extra)
with self.assertRaises(DerivationValidationError): module.validate_semantic_output(data,case)
def test_payload_uses_true_schema_object(self):
schema=module.output_schema(BY["TR2-V1"]); payload=module.build_payload("qwen3.5:9B","p",schema,16384,1024); self.assertIs(payload["format"],schema); self.assertIsInstance(payload["format"],dict); self.assertEqual(payload["options"]["temperature"],0)
def test_prompt_has_typed_examples_and_allowed_ids(self):
prompt=module.build_prompt(BY["TR2-V1"]); self.assertIn('["obs_1", "obs_2"]',prompt); self.assertIn('"target_observation_id":null',prompt); self.assertIn('Never return the string "null"',prompt); self.assertNotIn('observation ID or null',prompt)
def test_self_normalization_must_be_positive(self):
case=BY["TR1-V1"]; semantic=self_output("Mit Dr. Schlummer arbeiten wir nicht weiter."); combined=module.combine(case,semantic); self.assertEqual(module.evaluate(case,semantic,combined)["classification"],"FAIL")
def test_no_rejection_derivation_exists(self):
self.assertFalse(hasattr(module,"derive")); self.assertNotIn("status",module.PAIRED_KEYS); self.assertNotIn("explicitly_rejected",module.PAIRED_KEYS)
def test_runner_artifacts_distinguish_linkage_and_normalization(self):
case=BY["TR1-V1"]; fixture={"schema_version":module.SCHEMA_VERSION,"cases":[case]}; caller=Mock(return_value=(json.dumps(self_output()),{"model":"qwen3.5:9B"}))
with tempfile.TemporaryDirectory() as tmp:
root=Path(tmp); path=root/"cases.json"; path.write_text(json.dumps(fixture)); out=root/"out"; args=argparse.Namespace(cases=path,output=out,endpoint="x",model="qwen3.5:9B",timeout=1,num_ctx=16384,num_predict=1024); summary=module.run(args,caller); self.assertEqual(summary["llm_call_count"],1); self.assertTrue((out/"tr1-v1"/"deterministic_linkage_result.json").exists()); self.assertTrue((out/"tr1-v1"/"normalized_target_result.json").exists())
if __name__=="__main__": unittest.main()
+221
View File
@@ -0,0 +1,221 @@
import json
import subprocess
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import Mock, patch
from scripts import transcribe_meeting
from src.meeting_lab.transcription.whisper import (
TranscriptionError,
compact_transcript,
transcribe_audio,
)
RAW_RESULT = {
"params": {"language": "de"},
"transcription": [
{
"timestamps": {"from": "00:00:00,000", "to": "00:00:01,200"},
"offsets": {"from": 0, "to": 1200},
"text": " Hallo ",
},
{
"timestamps": {"from": "00:00:01,200", "to": "00:00:02,500"},
"offsets": {"from": 1200, "to": 2500},
"text": "Welt",
},
],
}
class FakeRunner:
def __init__(self, *, returncode: int = 0, write_raw: bool = True) -> None:
self.returncode = returncode
self.write_raw = write_raw
self.commands: list[list[str]] = []
def __call__(self, command, **kwargs):
command = list(command)
self.commands.append(command)
if self.write_raw:
prefix = Path(command[command.index("-of") + 1])
prefix.with_suffix(".json").write_text(
json.dumps(RAW_RESULT, ensure_ascii=False), encoding="utf-8"
)
return subprocess.CompletedProcess(
command, self.returncode, stdout="", stderr="backend failure"
)
class TranscriptionTests(unittest.TestCase):
def create_inputs(self, root: Path) -> tuple[Path, Path]:
audio = root / "meeting.wav"
model = root / "ggml-model.bin"
audio.write_bytes(b"audio")
model.write_bytes(b"model")
return audio, model
def test_missing_audio_file_is_rejected_before_execution(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
model = root / "model.bin"
model.write_bytes(b"model")
runner = FakeRunner()
with self.assertRaisesRegex(TranscriptionError, "Audio file does not exist"):
transcribe_audio(root / "missing.wav", model, root / "out", runner=runner)
self.assertEqual(runner.commands, [])
def test_invalid_parameters_are_rejected(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
with self.assertRaisesRegex(TranscriptionError, "model does not exist"):
transcribe_audio(audio, root / "missing.bin", root / "out")
with self.assertRaisesRegex(TranscriptionError, "Language must"):
transcribe_audio(audio, model, root / "out", " ")
with self.assertRaisesRegex(TranscriptionError, "executable must"):
transcribe_audio(audio, model, root / "out", executable="")
with self.assertRaisesRegex(TranscriptionError, "Threads must"):
transcribe_audio(audio, model, root / "out", threads="zero")
def test_auto_threads_use_detected_physical_core_count_and_flash_attention(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
runner = FakeRunner()
result = transcribe_audio(
audio,
model,
root / "output",
runner=runner,
thread_detector=lambda: 6,
)
command = runner.commands[0]
self.assertEqual(command[command.index("-t") + 1], "6")
self.assertIn("-fa", command)
metadata = json.loads(result.runtime_metadata.read_text())
self.assertEqual(metadata["threads"], 6)
self.assertEqual(metadata["threads_option"], "auto")
self.assertTrue(metadata["flash_attention"])
self.assertEqual(metadata["whisper_executable"], "whisper-cli")
def test_explicit_thread_count_is_forwarded_unchanged(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
runner = FakeRunner()
result = transcribe_audio(
audio, model, root / "output", threads=12, runner=runner
)
command = runner.commands[0]
self.assertEqual(command[command.index("-t") + 1], "12")
metadata = json.loads(result.runtime_metadata.read_text())
self.assertEqual(metadata["threads"], 12)
self.assertEqual(metadata["threads_option"], "12")
def test_cli_forwards_explicit_thread_count_to_wrapper(self) -> None:
fake_result = Mock(
transcript_json=Path("output/transcript.json"), runtime_seconds=1.0
)
with patch.object(
transcribe_meeting, "transcribe_audio", return_value=fake_result
) as transcribe:
code = transcribe_meeting.main(
[
"meeting.wav",
"--model", "model.bin",
"--output-dir", "output",
"--threads", "6",
]
)
self.assertEqual(code, 0)
self.assertEqual(transcribe.call_args.kwargs["threads"], "6")
def test_compact_transcript_has_established_schema(self) -> None:
self.assertEqual(
compact_transcript(RAW_RESULT),
{
"text": "Hallo Welt",
"segments": [
{"id": 0, "start": 0.0, "end": 1.2, "text": "Hallo"},
{"id": 1, "start": 1.2, "end": 2.5, "text": "Welt"},
],
},
)
def test_success_preserves_raw_output_and_creates_metadata(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
runner = FakeRunner()
times = iter([10.0, 12.25])
result = transcribe_audio(
audio,
model,
root / "output",
"de",
runner=runner,
monotonic=lambda: next(times),
now=lambda: datetime(2026, 8, 21, 8, 30, tzinfo=timezone.utc),
)
self.assertEqual(json.loads(result.raw_output.read_text()), RAW_RESULT)
self.assertEqual(json.loads(result.transcript_json.read_text())["text"], "Hallo Welt")
self.assertEqual(
result.transcript_text.read_text(),
"[00:00:00.000 - 00:00:01.200] Hallo\n"
"[00:00:01.200 - 00:00:02.500] Welt\n",
)
metadata = json.loads(result.runtime_metadata.read_text())
self.assertEqual(metadata["backend"], "whisper.cpp")
self.assertEqual(metadata["language"], "de")
self.assertEqual(metadata["duration_seconds"], 2.5)
self.assertEqual(metadata["runtime_seconds"], 2.25)
self.assertEqual(metadata["timestamp"], "2026-08-21T08:30:00+00:00")
self.assertEqual(metadata["input_file"], str(audio.resolve()))
self.assertEqual(metadata["model"], str(model.resolve()))
def test_output_paths_are_deterministic(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
output = root / "chosen-output"
result = transcribe_audio(audio, model, output, runner=FakeRunner())
self.assertEqual(result.raw_output, output / "whisper_raw.json")
self.assertEqual(result.transcript_json, output / "transcript.json")
self.assertEqual(result.transcript_text, output / "transcript.txt")
self.assertEqual(result.runtime_metadata, output / "runtime_metadata.json")
self.assertEqual(
set(json.loads(result.runtime_metadata.read_text())["output_files"].values()),
{"whisper_raw.json", "transcript.json", "transcript.txt", "runtime_metadata.json"},
)
def test_backend_failure_raises_and_preserves_available_raw_output(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
output = root / "output"
with self.assertRaisesRegex(TranscriptionError, "exit code 7.*backend failure"):
transcribe_audio(audio, model, output, runner=FakeRunner(returncode=7))
self.assertEqual(json.loads((output / "whisper_raw.json").read_text()), RAW_RESULT)
self.assertFalse((output / "transcript.json").exists())
self.assertFalse((output / "runtime_metadata.json").exists())
def test_missing_backend_output_is_reported(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
audio, model = self.create_inputs(root)
with self.assertRaisesRegex(TranscriptionError, "without producing JSON"):
transcribe_audio(
audio, model, root / "output", runner=FakeRunner(write_raw=False)
)
if __name__ == "__main__":
unittest.main()