Files
meeting-lab/docs/bug-005-root-cause.md
T
admin 9446c6e0be Document validation architecture and renderer faithfulness findings
- document Entity Registry and Meeting Context V2 architecture
- preserve meeting_context.yaml as the authoritative meeting-specific input
- define immutable authoritative metadata across all pipeline stages
- restrict Constraint Repair to deterministic structured-data operations
- record BUG-003 root cause and deferred entity-verification resolution
- document BUG-005 attendance-consistency design
- add BUG-006 renderer faithfulness root-cause analysis
- distinguish Engineering Readiness from Practical Usability
- update the persistent regression bug tracker
2026-08-03 16:05:47 +02:00

199 lines
7.5 KiB
Markdown

# BUG-005 Root Cause Analysis
## Observed Behaviour
The final working protocol treats Björn as one of the "fehlende Teilnehmer":
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md:53`
This contradicts Meeting Context, which lists Björn as an actual participant
with `attendance_status: "present"`:
- `samples/real_live/project_process_meeting/meeting_context.yaml:45`
- `samples/real_live/project_process_meeting/meeting_context.yaml:52`
## Evidence
Meeting Context marks Björn present:
```text
participant_id: "bjoern"
display_name: "Björn"
role: "Leiter Marketing"
attendance_status: "present"
```
The extraction prompt path does include Meeting Context:
- `src/meeting_lab/extraction/extract_chunks.py:161` renders Meeting Context
for the prompt when it is supplied.
- `src/meeting_lab/models/meeting_context.py:108` renders the heading
`MEETING CONTEXT V1 (AUTHORITATIVE METADATA)`.
- `src/meeting_lab/models/meeting_context.py:112` renders the rule
`The participant list is authoritative.`
- `src/meeting_lab/models/meeting_context.py:128` to
`src/meeting_lab/models/meeting_context.py:132` renders actual
participants.
Execution metadata confirms chunk 08 used the Meeting Context file:
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:29`
to `:32`
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:34`
to `:38`
The source chunk does not say Björn was absent from the meeting. It says
Giovanna and Björn had not yet understood or reviewed the discussed mail or
process state:
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:211`
to `:215`
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:229`
to `:245`
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:257`
to `:267`
The earliest generated artifact that treats Björn as absent is the raw chunk
08 extraction response:
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:4`
summarizes "Einbeziehung fehlender Teilnehmer wie Jovana und Björn".
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:6`
to `:10` lists participants as only Martin, Lars and Malte.
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:64`
to `:68` emits the open question
`Wie sollen fehlende Teilnehmer (Jovana, Björn) in den Prozess einbezogen werden?`
The persisted extraction JSON keeps that open question:
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:14`
to `:16`
The persisted extraction JSON records only Meeting Context provenance, not the
participant list or attendance status:
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:23`
to `:27`
The normalizer drops raw response fields such as `participants` and `topics`:
- `src/meeting_lab/extraction/extract_chunks.py:318` to `:365` returns only
normalized `facts`, `decisions`, `todos`, `questions`, `positions` and
`technical`.
- `src/meeting_lab/extraction/extract_chunks.py:432` to `:434` adds only
provenance from Meeting Context after normalization.
Canonicalizer preserves the bad open question:
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:1633`
to `:1645`
Semantic Consolidator output also preserves the bad open question:
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1669`
to `:1681`
The renderer receives the repaired consolidated JSON as its only recorded
input:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/metadata.json:2`
to `:4`
The renderer prompt requires using only the provided input:
- `prompts/working_protocol.md:7` to `:12`
Therefore the renderer did not receive Meeting Context attendance metadata
that would let it distinguish a present participant from a mentioned-only or
absent person.
## Stage-by-Stage Verification
1. Does extraction receive Björn as a present participant?
Yes. The extraction code renders Meeting Context into the prompt when supplied,
and the chunk 08 metadata records
`samples/real_live/project_process_meeting/meeting_context.yaml` as the Meeting
Context source.
2. Does extraction output preserve that information?
No. The raw chunk 08 response lists participants as Martin, Lars and Malte
only, despite Björn being present in Meeting Context. The official persisted
extraction JSON contains only a minimal `context` provenance object and does
not preserve the authoritative participant list or attendance status.
3. Does Canonicalizer preserve it?
No. Canonicalizer input lacks the participant list and attendance status. The
Canonicalizer preserves the already bad open question from chunk 08.
4. Does Semantic Consolidator preserve it?
No. The repaired consolidated JSON lacks Meeting Context participant metadata
and preserves the bad open question.
5. Does the Renderer receive enough information to distinguish present from
mentioned-only participants?
No. Renderer metadata records only
`semantic_consolidator_repair_v1/consolidated_extractions.json` as input, and
that file contains no Meeting Context participant list or attendance status.
## Earliest Failing Pipeline Stage
The earliest failing stage is chunk extraction for
`chunk_08_normalized.txt`.
More precisely, the raw model response for chunk 08 is the first artifact that
both omits Björn from the participant list and groups Björn with absent or
"fehlende" participants.
## Verified Root Cause
The immediate root cause is ignored Meeting Context metadata during extraction
for chunk 08. The extraction model received authoritative Meeting Context, but
interpreted transcript evidence about not having reviewed or understood an
email/process state as meeting absence.
The downstream cause is missing propagation of participant attendance metadata.
After extraction, only Meeting Context provenance is persisted. Canonicalizer,
Semantic Consolidator and Renderer do not receive the authoritative participant
list or attendance status, so they cannot detect or repair the contradiction.
This is not primarily a renderer inference bug. The renderer preserved an open
question already present in its input, and its prompt explicitly restricts it
to the supplied consolidated input.
## Cause Classification
- Missing propagation: verified.
- Ignored metadata: verified at extraction.
- Prompt wording: not verified as the direct cause.
- Renderer inference: disproved as the earliest cause.
- Lost provenance: partially verified; provenance remains, but the substantive
participant metadata is not propagated.
- Another cause: not identified.
## Confidence
High.
The conclusion is supported by Meeting Context, extraction metadata, raw chunk
08 model output, persisted extraction JSON, Canonicalizer output, Semantic
Consolidator output and renderer metadata.
## Recommended Fix
Concept only:
Propagate authoritative Meeting Context participant metadata, including
attendance status, beyond extraction as structured data. Add validation that
flags contradictions where generated items classify an actual participant as
absent or mentioned-only. The validation should run before rendering, and
ideally immediately after extraction so the contradiction is caught at the
earliest stage.
Do not rely on the Working Protocol Renderer to correct attendance semantics
from prose-only consolidated items.