Document validation architecture and renderer faithfulness findings
- document Entity Registry and Meeting Context V2 architecture - preserve meeting_context.yaml as the authoritative meeting-specific input - define immutable authoritative metadata across all pipeline stages - restrict Constraint Repair to deterministic structured-data operations - record BUG-003 root cause and deferred entity-verification resolution - document BUG-005 attendance-consistency design - add BUG-006 renderer faithfulness root-cause analysis - distinguish Engineering Readiness from Practical Usability - update the persistent regression bug tracker
This commit is contained in:
@@ -0,0 +1,198 @@
|
||||
# BUG-005 Root Cause Analysis
|
||||
|
||||
## Observed Behaviour
|
||||
|
||||
The final working protocol treats Björn as one of the "fehlende Teilnehmer":
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md:53`
|
||||
|
||||
This contradicts Meeting Context, which lists Björn as an actual participant
|
||||
with `attendance_status: "present"`:
|
||||
|
||||
- `samples/real_live/project_process_meeting/meeting_context.yaml:45`
|
||||
- `samples/real_live/project_process_meeting/meeting_context.yaml:52`
|
||||
|
||||
## Evidence
|
||||
|
||||
Meeting Context marks Björn present:
|
||||
|
||||
```text
|
||||
participant_id: "bjoern"
|
||||
display_name: "Björn"
|
||||
role: "Leiter Marketing"
|
||||
attendance_status: "present"
|
||||
```
|
||||
|
||||
The extraction prompt path does include Meeting Context:
|
||||
|
||||
- `src/meeting_lab/extraction/extract_chunks.py:161` renders Meeting Context
|
||||
for the prompt when it is supplied.
|
||||
- `src/meeting_lab/models/meeting_context.py:108` renders the heading
|
||||
`MEETING CONTEXT V1 (AUTHORITATIVE METADATA)`.
|
||||
- `src/meeting_lab/models/meeting_context.py:112` renders the rule
|
||||
`The participant list is authoritative.`
|
||||
- `src/meeting_lab/models/meeting_context.py:128` to
|
||||
`src/meeting_lab/models/meeting_context.py:132` renders actual
|
||||
participants.
|
||||
|
||||
Execution metadata confirms chunk 08 used the Meeting Context file:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:29`
|
||||
to `:32`
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.metadata.json:34`
|
||||
to `:38`
|
||||
|
||||
The source chunk does not say Björn was absent from the meeting. It says
|
||||
Giovanna and Björn had not yet understood or reviewed the discussed mail or
|
||||
process state:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:211`
|
||||
to `:215`
|
||||
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:229`
|
||||
to `:245`
|
||||
- `samples/benchmarks/meeting_context_v1/source_reconstruction/chunk_08_normalized.txt:257`
|
||||
to `:267`
|
||||
|
||||
The earliest generated artifact that treats Björn as absent is the raw chunk
|
||||
08 extraction response:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:4`
|
||||
summarizes "Einbeziehung fehlender Teilnehmer wie Jovana und Björn".
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:6`
|
||||
to `:10` lists participants as only Martin, Lars and Malte.
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.raw.txt:64`
|
||||
to `:68` emits the open question
|
||||
`Wie sollen fehlende Teilnehmer (Jovana, Björn) in den Prozess einbezogen werden?`
|
||||
|
||||
The persisted extraction JSON keeps that open question:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:14`
|
||||
to `:16`
|
||||
|
||||
The persisted extraction JSON records only Meeting Context provenance, not the
|
||||
participant list or attendance status:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/full_context_run/chunk_08_extraction.json:23`
|
||||
to `:27`
|
||||
|
||||
The normalizer drops raw response fields such as `participants` and `topics`:
|
||||
|
||||
- `src/meeting_lab/extraction/extract_chunks.py:318` to `:365` returns only
|
||||
normalized `facts`, `decisions`, `todos`, `questions`, `positions` and
|
||||
`technical`.
|
||||
- `src/meeting_lab/extraction/extract_chunks.py:432` to `:434` adds only
|
||||
provenance from Meeting Context after normalization.
|
||||
|
||||
Canonicalizer preserves the bad open question:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/canonicalizer/canonicalized_extractions.json:1633`
|
||||
to `:1645`
|
||||
|
||||
Semantic Consolidator output also preserves the bad open question:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json:1669`
|
||||
to `:1681`
|
||||
|
||||
The renderer receives the repaired consolidated JSON as its only recorded
|
||||
input:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/metadata.json:2`
|
||||
to `:4`
|
||||
|
||||
The renderer prompt requires using only the provided input:
|
||||
|
||||
- `prompts/working_protocol.md:7` to `:12`
|
||||
|
||||
Therefore the renderer did not receive Meeting Context attendance metadata
|
||||
that would let it distinguish a present participant from a mentioned-only or
|
||||
absent person.
|
||||
|
||||
## Stage-by-Stage Verification
|
||||
|
||||
1. Does extraction receive Björn as a present participant?
|
||||
|
||||
Yes. The extraction code renders Meeting Context into the prompt when supplied,
|
||||
and the chunk 08 metadata records
|
||||
`samples/real_live/project_process_meeting/meeting_context.yaml` as the Meeting
|
||||
Context source.
|
||||
|
||||
2. Does extraction output preserve that information?
|
||||
|
||||
No. The raw chunk 08 response lists participants as Martin, Lars and Malte
|
||||
only, despite Björn being present in Meeting Context. The official persisted
|
||||
extraction JSON contains only a minimal `context` provenance object and does
|
||||
not preserve the authoritative participant list or attendance status.
|
||||
|
||||
3. Does Canonicalizer preserve it?
|
||||
|
||||
No. Canonicalizer input lacks the participant list and attendance status. The
|
||||
Canonicalizer preserves the already bad open question from chunk 08.
|
||||
|
||||
4. Does Semantic Consolidator preserve it?
|
||||
|
||||
No. The repaired consolidated JSON lacks Meeting Context participant metadata
|
||||
and preserves the bad open question.
|
||||
|
||||
5. Does the Renderer receive enough information to distinguish present from
|
||||
mentioned-only participants?
|
||||
|
||||
No. Renderer metadata records only
|
||||
`semantic_consolidator_repair_v1/consolidated_extractions.json` as input, and
|
||||
that file contains no Meeting Context participant list or attendance status.
|
||||
|
||||
## Earliest Failing Pipeline Stage
|
||||
|
||||
The earliest failing stage is chunk extraction for
|
||||
`chunk_08_normalized.txt`.
|
||||
|
||||
More precisely, the raw model response for chunk 08 is the first artifact that
|
||||
both omits Björn from the participant list and groups Björn with absent or
|
||||
"fehlende" participants.
|
||||
|
||||
## Verified Root Cause
|
||||
|
||||
The immediate root cause is ignored Meeting Context metadata during extraction
|
||||
for chunk 08. The extraction model received authoritative Meeting Context, but
|
||||
interpreted transcript evidence about not having reviewed or understood an
|
||||
email/process state as meeting absence.
|
||||
|
||||
The downstream cause is missing propagation of participant attendance metadata.
|
||||
After extraction, only Meeting Context provenance is persisted. Canonicalizer,
|
||||
Semantic Consolidator and Renderer do not receive the authoritative participant
|
||||
list or attendance status, so they cannot detect or repair the contradiction.
|
||||
|
||||
This is not primarily a renderer inference bug. The renderer preserved an open
|
||||
question already present in its input, and its prompt explicitly restricts it
|
||||
to the supplied consolidated input.
|
||||
|
||||
## Cause Classification
|
||||
|
||||
- Missing propagation: verified.
|
||||
- Ignored metadata: verified at extraction.
|
||||
- Prompt wording: not verified as the direct cause.
|
||||
- Renderer inference: disproved as the earliest cause.
|
||||
- Lost provenance: partially verified; provenance remains, but the substantive
|
||||
participant metadata is not propagated.
|
||||
- Another cause: not identified.
|
||||
|
||||
## Confidence
|
||||
|
||||
High.
|
||||
|
||||
The conclusion is supported by Meeting Context, extraction metadata, raw chunk
|
||||
08 model output, persisted extraction JSON, Canonicalizer output, Semantic
|
||||
Consolidator output and renderer metadata.
|
||||
|
||||
## Recommended Fix
|
||||
|
||||
Concept only:
|
||||
|
||||
Propagate authoritative Meeting Context participant metadata, including
|
||||
attendance status, beyond extraction as structured data. Add validation that
|
||||
flags contradictions where generated items classify an actual participant as
|
||||
absent or mentioned-only. The validation should run before rendering, and
|
||||
ideally immediately after extraction so the contradiction is caught at the
|
||||
earliest stage.
|
||||
|
||||
Do not rely on the Working Protocol Renderer to correct attendance semantics
|
||||
from prose-only consolidated items.
|
||||
Reference in New Issue
Block a user