- document Entity Registry and Meeting Context V2 architecture - preserve meeting_context.yaml as the authoritative meeting-specific input - define immutable authoritative metadata across all pipeline stages - restrict Constraint Repair to deterministic structured-data operations - record BUG-003 root cause and deferred entity-verification resolution - document BUG-005 attendance-consistency design - add BUG-006 renderer faithfulness root-cause analysis - distinguish Engineering Readiness from Practical Usability - update the persistent regression bug tracker
199 lines
7.5 KiB
Markdown
199 lines
7.5 KiB
Markdown
# BUG-005 Root Cause Analysis
|
|
|
|
## Observed Behaviour
|
|
|
|
The final working protocol treats Björn as one of the "fehlende Teilnehmer":
|
|
|
|
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md:53`
|
|
|
|
This contradicts Meeting Context, which lists Björn as an actual participant
|
|
with `attendance_status: "present"`:
|
|
|
|
- `samples/real_live/project_process_meeting/meeting_context.yaml:45`
|
|
- `samples/real_live/project_process_meeting/meeting_context.yaml:52`
|
|
|
|
## Evidence
|
|
|
|
Meeting Context marks Björn present:
|
|
|
|
```text
|
|
participant_id: "bjoern"
|
|
display_name: "Björn"
|
|
role: "Leiter Marketing"
|
|
attendance_status: "present"
|
|
```
|
|
|
|
The extraction prompt path does include Meeting Context:
|
|
|
|
- `src/meeting_lab/extraction/extract_chunks.py:161` renders Meeting Context
|
|
for the prompt when it is supplied.
|
|
- `src/meeting_lab/models/meeting_context.py:108` renders the heading
|
|
`MEETING CONTEXT V1 (AUTHORITATIVE METADATA)`.
|
|
- `src/meeting_lab/models/meeting_context.py:112` renders the rule
|
|
`The participant list is authoritative.`
|
|
- `src/meeting_lab/models/meeting_context.py:128` to
|
|
`src/meeting_lab/models/meeting_context.py:132` renders actual
|
|
participants.
|
|
|
|
Execution metadata confirms chunk 08 used the Meeting Context file:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:29`
|
|
to `:32`
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:34`
|
|
to `:38`
|
|
|
|
The source chunk does not say Björn was absent from the meeting. It says
|
|
Giovanna and Björn had not yet understood or reviewed the discussed mail or
|
|
process state:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:211`
|
|
to `:215`
|
|
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:229`
|
|
to `:245`
|
|
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:257`
|
|
to `:267`
|
|
|
|
The earliest generated artifact that treats Björn as absent is the raw chunk
|
|
08 extraction response:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:4`
|
|
summarizes "Einbeziehung fehlender Teilnehmer wie Jovana und Björn".
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:6`
|
|
to `:10` lists participants as only Martin, Lars and Malte.
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:64`
|
|
to `:68` emits the open question
|
|
`Wie sollen fehlende Teilnehmer (Jovana, Björn) in den Prozess einbezogen werden?`
|
|
|
|
The persisted extraction JSON keeps that open question:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:14`
|
|
to `:16`
|
|
|
|
The persisted extraction JSON records only Meeting Context provenance, not the
|
|
participant list or attendance status:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:23`
|
|
to `:27`
|
|
|
|
The normalizer drops raw response fields such as `participants` and `topics`:
|
|
|
|
- `src/meeting_lab/extraction/extract_chunks.py:318` to `:365` returns only
|
|
normalized `facts`, `decisions`, `todos`, `questions`, `positions` and
|
|
`technical`.
|
|
- `src/meeting_lab/extraction/extract_chunks.py:432` to `:434` adds only
|
|
provenance from Meeting Context after normalization.
|
|
|
|
Canonicalizer preserves the bad open question:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:1633`
|
|
to `:1645`
|
|
|
|
Semantic Consolidator output also preserves the bad open question:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1669`
|
|
to `:1681`
|
|
|
|
The renderer receives the repaired consolidated JSON as its only recorded
|
|
input:
|
|
|
|
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/metadata.json:2`
|
|
to `:4`
|
|
|
|
The renderer prompt requires using only the provided input:
|
|
|
|
- `prompts/working_protocol.md:7` to `:12`
|
|
|
|
Therefore the renderer did not receive Meeting Context attendance metadata
|
|
that would let it distinguish a present participant from a mentioned-only or
|
|
absent person.
|
|
|
|
## Stage-by-Stage Verification
|
|
|
|
1. Does extraction receive Björn as a present participant?
|
|
|
|
Yes. The extraction code renders Meeting Context into the prompt when supplied,
|
|
and the chunk 08 metadata records
|
|
`samples/real_live/project_process_meeting/meeting_context.yaml` as the Meeting
|
|
Context source.
|
|
|
|
2. Does extraction output preserve that information?
|
|
|
|
No. The raw chunk 08 response lists participants as Martin, Lars and Malte
|
|
only, despite Björn being present in Meeting Context. The official persisted
|
|
extraction JSON contains only a minimal `context` provenance object and does
|
|
not preserve the authoritative participant list or attendance status.
|
|
|
|
3. Does Canonicalizer preserve it?
|
|
|
|
No. Canonicalizer input lacks the participant list and attendance status. The
|
|
Canonicalizer preserves the already bad open question from chunk 08.
|
|
|
|
4. Does Semantic Consolidator preserve it?
|
|
|
|
No. The repaired consolidated JSON lacks Meeting Context participant metadata
|
|
and preserves the bad open question.
|
|
|
|
5. Does the Renderer receive enough information to distinguish present from
|
|
mentioned-only participants?
|
|
|
|
No. Renderer metadata records only
|
|
`semantic_consolidator_repair_v1/consolidated_extractions.json` as input, and
|
|
that file contains no Meeting Context participant list or attendance status.
|
|
|
|
## Earliest Failing Pipeline Stage
|
|
|
|
The earliest failing stage is chunk extraction for
|
|
`chunk_08_normalized.txt`.
|
|
|
|
More precisely, the raw model response for chunk 08 is the first artifact that
|
|
both omits Björn from the participant list and groups Björn with absent or
|
|
"fehlende" participants.
|
|
|
|
## Verified Root Cause
|
|
|
|
The immediate root cause is ignored Meeting Context metadata during extraction
|
|
for chunk 08. The extraction model received authoritative Meeting Context, but
|
|
interpreted transcript evidence about not having reviewed or understood an
|
|
email/process state as meeting absence.
|
|
|
|
The downstream cause is missing propagation of participant attendance metadata.
|
|
After extraction, only Meeting Context provenance is persisted. Canonicalizer,
|
|
Semantic Consolidator and Renderer do not receive the authoritative participant
|
|
list or attendance status, so they cannot detect or repair the contradiction.
|
|
|
|
This is not primarily a renderer inference bug. The renderer preserved an open
|
|
question already present in its input, and its prompt explicitly restricts it
|
|
to the supplied consolidated input.
|
|
|
|
## Cause Classification
|
|
|
|
- Missing propagation: verified.
|
|
- Ignored metadata: verified at extraction.
|
|
- Prompt wording: not verified as the direct cause.
|
|
- Renderer inference: disproved as the earliest cause.
|
|
- Lost provenance: partially verified; provenance remains, but the substantive
|
|
participant metadata is not propagated.
|
|
- Another cause: not identified.
|
|
|
|
## Confidence
|
|
|
|
High.
|
|
|
|
The conclusion is supported by Meeting Context, extraction metadata, raw chunk
|
|
08 model output, persisted extraction JSON, Canonicalizer output, Semantic
|
|
Consolidator output and renderer metadata.
|
|
|
|
## Recommended Fix
|
|
|
|
Concept only:
|
|
|
|
Propagate authoritative Meeting Context participant metadata, including
|
|
attendance status, beyond extraction as structured data. Add validation that
|
|
flags contradictions where generated items classify an actual participant as
|
|
absent or mentioned-only. The validation should run before rendering, and
|
|
ideally immediately after extraction so the contradiction is caught at the
|
|
earliest stage.
|
|
|
|
Do not rely on the Working Protocol Renderer to correct attendance semantics
|
|
from prose-only consolidated items.
|