martin/meeting-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document validation architecture and renderer faithfulness findings

Browse Source 
- document Entity Registry and Meeting Context V2 architecture
- preserve meeting_context.yaml as the authoritative meeting-specific input
- define immutable authoritative metadata across all pipeline stages
- restrict Constraint Repair to deterministic structured-data operations
- record BUG-003 root cause and deferred entity-verification resolution
- document BUG-005 attendance-consistency design
- add BUG-006 renderer faithfulness root-cause analysis
- distinguish Engineering Readiness from Practical Usability
- update the persistent regression bug tracker
This commit is contained in:
admin
2026-08-03 16:05:47 +02:00
parent 06f0e7e651
commit 9446c6e0be
13 changed files with 2208 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/bug-003-root-cause.md
+265
View File
@@ -0,0 +1,265 @@
# BUG-003 Root Cause Analysis
## Observed Behaviour
The final working protocol contains two names that were flagged as not
belonging to the real meeting:
- `Guido`
- `Noah`
Observed output:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md:36`
contains `Einbinden der Fachbereiche (stellvertretend durch Guido) zur
Definition von Prüfsteinen.`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md:44`
contains `Sollte ein eigener Prozess für Noah (Business Development)
benötigt werden oder reicht der bestehende?`
## Evidence
The names are present in the final renderer raw response as well as
`working_protocol.md`:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/raw_model_response.txt:36`
contains `Guido`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/raw_model_response.txt:44`
contains `Noah`
They are also present before rendering in the repaired consolidated input:
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1047`
contains the open question text with `Noah`
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1048`
contains evidence `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1537`
contains the action item text with `Guido`
They are present before semantic consolidation in the canonicalized output:
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:411`
contains the open question text with `Noah`
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:412`
contains evidence `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:1401`
contains the action item text with `Guido`
They are present before canonicalization in extraction output and raw model
responses:
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_02_extraction.json:18`
contains the extracted open question with `Noah`
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_02_extraction.raw.txt:76`
contains the same raw model output question with `Noah`
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_07_extraction.json:11`
contains the extracted todo with `Guido`
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_07_extraction.raw.txt:53`
contains the same raw model output todo with `Guido`
They are present before extraction in the reconstructed normalized chunks:
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_02_normalized.txt:79`
contains `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_07_normalized.txt:213`
contains `Guido`
They are present before normalization in reconstructed raw chunks:
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_02.txt:79`
contains `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_07.txt:213`
contains `Guido`
They are present in the source cleaned Whisper transcript used to reconstruct
the chunks:
- `samples/real_live/project_process_meeting/transcript/meeting_speech_cleaned.json:5783`
has segment text `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `samples/real_live/project_process_meeting/transcript/meeting_speech_cleaned.json:26315`
has segment text `Guido`
Segment-level verification:
- `Noah`: segment `id=182`, `start=899.6400000000001`, `end=901.94`,
text `Oder brauchen wir einen eigenen Prozess bei Noah?`
- `Guido`: segment `id=776`, `start=4157.86`, `end=4160.14`, text `Guido`
The original Whisper transcript also contains both names:
- `samples/real_live/project_process_meeting/transcript/meeting_speech.json:1`
contains both `Noah` and `Guido` in the top-level transcript text and segment
data.
Meeting Context does not contain either name:
- `rg -n "Guido|Noah" samples/real_live/project_process_meeting/meeting_context.yaml`
returned no matches.
Prompt files do not contain either name:
- `rg -n "Guido|Noah" prompts`
returned no matches.
## Earliest Pipeline Stage Containing the Names
The earliest verified pipeline artifact containing the names is the Whisper
transcript stage:
- `samples/real_live/project_process_meeting/transcript/meeting_speech.json`
- `samples/real_live/project_process_meeting/transcript/meeting_speech_cleaned.json`
For the reconstructed benchmark specifically, the earliest input used by the
TODO 4 pipeline is:
- `samples/real_live/project_process_meeting/transcript/meeting_speech_cleaned.json`
The reconstructed chunks preserve the names from that cleaned Whisper JSON.
Extraction, canonicalization, consolidation and rendering propagate them.
## Repository Search Results
Repository search for `Guido|Noah` found these occurrence groups:
- `docs/regression-bugs.md`: tracker entry for BUG-003 mentions both names.
- `tests/gold/responsibility_attribution_negative/transcript.txt`: contains
`Noah` in a synthetic gold scenario.
- `tests/gold/responsibility_attribution_negative/expected.json`: contains
expected `Noah` entries for that synthetic gold scenario.
- `samples/real_live/project_process_meeting/transcript/meeting_speech.json`:
contains `Noah` and `Guido`.
- `samples/real_live/project_process_meeting/transcript/meeting_speech_cleaned.json`:
contains `Noah` and `Guido`.
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_02.txt`
and `chunk_02_normalized.txt`: contain `Noah`.
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_07.txt`
and `chunk_07_normalized.txt`: contain `Guido`.
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_02_extraction.json`
and `chunk_02_extraction.raw.txt`: contain `Noah`.
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_07_extraction.json`
and `chunk_07_extraction.raw.txt`: contain `Guido`.
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json`:
contains both names.
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`:
contains both names.
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/raw_model_response.txt`
and `working_protocol.md`: contain both names.
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`:
mentions `Noah` in the evaluation notes.
Search results did not find `Guido` or `Noah` in:
- `prompts/`
- `samples/real_live/project_process_meeting/meeting_context.yaml`
- `src/`
## Prompt Inspection
Renderer prompt:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/metadata.json`
records `prompt_file: "prompts/working_protocol.md"` and
`input_source:
"samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json"`.
- `prompts/working_protocol.md` contains no few-shot examples and no `Guido` or
`Noah` occurrences.
Extraction prompt assembly:
- `src/meeting_lab/llm/prompts.py` builds extraction prompts from
`prompts/common.md`, optional Meeting Context, explicitly requested task
prompts and the provided transcript.
- `src/meeting_lab/extraction/extract_chunks.py` calls that shared builder with
`decisions.md` and `todos.md` as task prompts.
Semantic Consolidator prompt assembly:
- `src/meeting_lab/consolidation/consolidate_facts.py` builds its prompt from
`prompts/consolidate_facts.md` plus the canonicalized fact payload.
Gold/test prompt inspection:
- `tests/gold/responsibility_attribution_negative/` contains synthetic `Noah`
examples.
- `scripts/run_gold_test.py` is a gold-test runner and reads
`scenario_dir / "transcript.txt"` when explicitly invoked.
- The inspected production metadata for the TODO 4 extraction run records
`input_path` values under
`samples/benchmarks/meeting_context_v1/source_reconstruction/`, not
`tests/gold/`.
- The inspected renderer metadata records only the repaired consolidated JSON
and `prompts/working_protocol.md`.
No evidence was found that few-shot examples, embedded examples, test
transcripts or gold scenarios became part of the production prompt for this
benchmark run.
## Pipeline Verification
The persisted metadata verifies the production execution path:
- `chunk_02_extraction.metadata.json` input:
`samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_02_normalized.txt`
- `chunk_07_extraction.metadata.json` input:
`samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_07_normalized.txt`
- both extraction metadata files record Meeting Context source:
`samples/real_live/project_process_meeting/meeting_context.yaml`
- `canonicalizer/metadata.json` records input directory:
`samples/benchmarks/meeting_context_v1/full_context_run`
- `semantic_consolidator_repair_v1/report.md` records output:
`samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
- `final_protocol_with_context/metadata.json` records renderer input:
`samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
and prompt file `prompts/working_protocol.md`
No execution metadata points to unrelated examples, templates, tests or gold
scenarios.
## Verified Root Cause
The verified root cause for BUG-003 in this pipeline run is upstream transcript
contamination: `Guido` and `Noah` already exist in the Whisper transcript and
the cleaned Whisper transcript used as the benchmark source.
The later stages did not introduce these names from prompts, Meeting Context,
gold tests or renderer examples. They propagated names already present in the
pipeline input:
`meeting_speech.json`
-> `meeting_speech_cleaned.json`
-> reconstructed chunks
-> extraction JSON
-> canonicalized JSON
-> repaired consolidated JSON
-> final working protocol
What cannot be established from repository artifacts alone:
- whether Whisper hallucinated these names from audio
- whether the source audio actually contains words that sound like these names
- what the correct intended tokens should be
Those questions require audio-level or human-transcript verification and are
outside the evidence available in this repository trace.
## Confidence
High
The conclusion that the final protocol names originated before extraction is
directly supported by persisted source transcript, chunk, extraction,
canonicalization, consolidation and renderer artifacts.
The confidence does not extend to identifying the correct replacement words.
That remains unverified.
## Recommended Fix
Conceptually, add a transcript/source-quality validation step before extraction
that flags person or organization names not present in Meeting Context or an
approved alias list. The step should preserve the original transcript text, but
mark suspicious entity mentions for review before they become structured
knowledge and final protocol content.
Do not treat gold scenarios or prompt examples as the root cause for this bug;
the evidence does not support that.