martin/meeting-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
18beb3385f53c4f8cd532e3e65b6fbe475a23cd3
meeting-lab/docs/bug-006-root-cause.md
T
admin 9446c6e0be Document validation architecture and renderer faithfulness findings 
- document Entity Registry and Meeting Context V2 architecture
- preserve meeting_context.yaml as the authoritative meeting-specific input
- define immutable authoritative metadata across all pipeline stages
- restrict Constraint Repair to deterministic structured-data operations
- record BUG-003 root cause and deferred entity-verification resolution
- document BUG-005 attendance-consistency design
- add BUG-006 renderer faithfulness root-cause analysis
- distinguish Engineering Readiness from Practical Usability
- update the persistent regression bug tracker
2026-08-03 16:05:47 +02:00

427 lines
16 KiB
Markdown
Raw Blame History

# BUG-006 Root Cause Analysis
## Observed Behaviour
In benchmark `meeting_context_v1/e2e_current_20260803_151000`, the Working
Protocol Renderer output is not a faithful view of the matching consolidated
representation.
Compared files:
- Consolidated input:
`samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- Rendered output:
`samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- Raw renderer response:
`samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/raw_model_response.txt`
Observed problems:
- consolidated decisions are omitted from the rendered `Decisions` sections
- consolidated open questions are omitted from rendered `Open Questions`
sections
- consolidated facts are rendered as decisions
- some consolidated action items are omitted
- some action-item and decision meanings are merged into other rendered
sections
The raw renderer response and `working_protocol.md` are byte-identical
(`cmp` exit code `0`), so the mismatch is already present in the LLM renderer
response. It is not introduced by Markdown file writing.
## Renderer Input And Prompt Evidence
Renderer metadata records the exact input and prompt:
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/metadata.json:2`
to `:4`
The renderer prompt explicitly says:
- use only the provided input:
`prompts/working_protocol.md:9` to `:10`
- preserve all decisions, action items and open questions:
`prompts/working_protocol.md:24` to `:26`
- remove presentation-level redundancy:
`prompts/working_protocol.md:23`
- do not repeat information merely because it appears in multiple categories:
`prompts/working_protocol.md:65`
- condense everything else:
`prompts/working_protocol.md:75`
The prompt is therefore present and relevant, but the direct observed failure
occurs in the model-generated renderer response.
## Faithfulness Analysis
### Rendered Section: Prozessrahmen und Projektideen-Eingang / Background
Status: modified.
Rendered lines:
- `working_protocol.md:7`
- `working_protocol.md:9`
The section condenses several consolidated facts and decisions into background
paragraphs. Examples:
- `decision_0001` is restated in background and again as a decision.
- `fact`: `Der Prozess ist ein F&E-Prozess gewesen, einfach historisch.`
- `fact`: `Die Bearbeitung der Projekte muss und kann nur in den
Fachabteilungen passieren.`
- `fact`: `Bevor F&E ein Projekt durchdenkt, müssen im Vorfeld Kriterien wie
Marktexistenz und Zulassungen geprüft werden.`
- `decision_0003` and `decision_0005` are partially restated as background.
This is a condensation, not a renderer-only invention.
### Rendered Section: Prozessrahmen und Projektideen-Eingang / Decisions
Status: partially faithful, partially omitted.
Faithfully or near-faithfully preserved:
- `decision_0001` -> `working_protocol.md:13`
- `decision_0002` -> `working_protocol.md:14`
- `decision_0003` -> `working_protocol.md:15`
- `decision_0005` -> `working_protocol.md:16`
- `decision_0006` -> `working_protocol.md:17`
- `decision_0009` -> `working_protocol.md:18`
- `decision_0011` -> `working_protocol.md:19`
Omitted as decisions:
- `decision_0004`: `Festlegung eines Reporting-Zyklus für abgelehnte
Projekte.`
- Consolidated evidence: `consolidated_extractions.json:1217` to `:1227`
- Rendered only as background: `working_protocol.md:72`
- `decision_0007`: `Zusammengetragen und beantwortete Fragen führen zur
Sitzung mit fünf Leuten...`
- Consolidated evidence: `consolidated_extractions.json:1329` to `:1339`
- No matching rendered decision.
- `decision_0008`: `Es wird vereinbart, den bestehenden Prozessrahmen zu
nutzen und auf die Bedürfnisse der F&E abzustimmen...`
- Consolidated evidence present in the input.
- No matching rendered decision.
- `decision_0010`: `Es wird vereinbart, dass betriebliche
Verbesserungsvorschläge... ausgeschleust werden.`
- Consolidated evidence present in the input.
- No matching rendered decision.
- `decision_0012`: `Die Diskussion wird beendet und die Änderungen werden
verschickt; Abwarten auf Rückmeldung von Johanna/Björn.`
- Consolidated evidence: `consolidated_extractions.json:1747` to `:1757`
- No matching rendered decision.
### Rendered Section: Prozessrahmen und Projektideen-Eingang / Action Items
Status: partially faithful, partially omitted, with upstream semantic problems
preserved.
Rendered lines:
- `working_protocol.md:23` to `:32`
Preserved action items include:
- `action_item_0001` -> `working_protocol.md:23`
- `action_item_0002` -> `working_protocol.md:24`
- `action_item_0003` -> `working_protocol.md:25`
- `action_item_0004` -> `working_protocol.md:26`
- `action_item_0005` -> `working_protocol.md:27`
- `action_item_0006` -> `working_protocol.md:28`
- `action_item_0007` -> `working_protocol.md:29`
- `action_item_0009` -> `working_protocol.md:30`
- `action_item_0011` -> `working_protocol.md:31`
- `action_item_0012` -> `working_protocol.md:32`
Omitted or merged action items:
- `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes
(z.B. Turkmenistan) führen.`
- Present in consolidated input.
- No matching rendered action item.
- `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.`
- Present in consolidated input.
- Its meaning overlaps with rendered decision `working_protocol.md:18` and
action item `working_protocol.md:30`, but it is not preserved as its own
action item.
Important boundary:
Items such as `Feedback zu den Kriterien...`, `Einbinden der Fachbereiche...`
and `Mail an Jovana und Björn...` are questionable todos, but they are already
`action_item` entries in the consolidated input:
- `consolidated_extractions.json:1007` to `:1019`
- `consolidated_extractions.json:1537` to `:1549`
- `consolidated_extractions.json:1651` to `:1663`
The renderer preserves those upstream action-item categories. It does not
create those todos from facts or prose.
### Rendered Section: Prozessrahmen und Projektideen-Eingang / Open Questions
Status: partially faithful, partially omitted.
Rendered lines:
- `working_protocol.md:36` to `:47`
Preserved open questions:
- `open_question_0001` -> `working_protocol.md:36`
- `open_question_0002` -> `working_protocol.md:37`
- `open_question_0003` -> `working_protocol.md:38`
- `open_question_0004` -> `working_protocol.md:39`
- `open_question_0005` -> `working_protocol.md:40`
- `open_question_0006` -> `working_protocol.md:41`
- `open_question_0007` -> `working_protocol.md:42`
- `open_question_0009` -> `working_protocol.md:43`
- `open_question_0010` -> `working_protocol.md:44`
- `open_question_0011` -> `working_protocol.md:45`
- `open_question_0012` -> `working_protocol.md:46`
- `open_question_0013` -> `working_protocol.md:47`
Omitted open questions:
- `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen und wie
wird der Prozess konkret implementiert?`
- Consolidated evidence: `consolidated_extractions.json:1387` to `:1397`
- No matching rendered open question.
- `open_question_0014`: `Welche Prüfsteine sind relevant für die
Fachabteilung?`
- Consolidated evidence: `consolidated_extractions.json:1765` to `:1775`
- No matching rendered open question.
### Rendered Section: Projektkategorisierung und Status / Background
Status: modified.
Rendered line:
- `working_protocol.md:53`
This condenses:
- fact: `Der Leiter F&E führt die Projektliste...`
- `technical_detail_0010`: project type definition
- fact: `Ein Projekt wandert direkt wieder in Business Development...`
No renderer-only invention was verified in this section.
### Rendered Section: Projektkategorisierung und Status / Decisions
Status: category-changed.
Rendered lines:
- `working_protocol.md:57`
- `working_protocol.md:58`
Both rendered decisions are facts in the consolidated input:
- fact: `Im Zweifel gehen Projekte durch.`
- Consolidated evidence: `consolidated_extractions.json:775` to `:784`
- Rendered as decision: `working_protocol.md:57`
- fact: `Ein Projekt kann abgelehnt werden, aber es muss sichergestellt sein,
dass relevante Projekte nicht weggeschmissen werden.`
- Consolidated evidence: `consolidated_extractions.json:755` to `:764`
- Rendered as decision: `working_protocol.md:58`
This is the clearest category change in the renderer output.
### Rendered Section: Projektkategorisierung und Status / Action Items
Status: omitted.
Rendered line:
- `working_protocol.md:62`: `*Keine.*`
The consolidated input contains at least one action item that belongs to this
topic area:
- `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes
(z.B. Turkmenistan) führen.`
The renderer omitted it.
### Rendered Section: Projektkategorisierung und Status / Open Questions
Status: faithful to rendered grouping, but unfaithful to complete input.
Rendered line:
- `working_protocol.md:66`: `*Keine.*`
The complete consolidated input still contains project-process open questions,
including omitted `open_question_0008` and `open_question_0014`. They were not
rendered elsewhere.
### Rendered Section: Technische Details und Infrastruktur / Background
Status: modified.
Rendered line:
- `working_protocol.md:72`
This condenses facts and technical details:
- fact: `Das Netzwerk bei Gemeinden ist momentan langsam.`
- `technical_detail_0006`: `Das Netzwerk ist langsam (wie zu ISDN-Zeiten).`
- `technical_detail_0007`: `Der Rechner schmiert ab...`
- fact: `Das Dokument ist das aktuelle Projektdeckblatt...`
- fact: `Es gibt einen Reporting-Zyklus.`
- fact: `Sekretariate sind sehr abweisend...`
The problem is not invention; the problem is that `decision_0004`
(`Festlegung eines Reporting-Zyklus...`) is downgraded from decision to
background.
### Rendered Section: Technische Details und Infrastruktur / Decisions
Status: omitted.
Rendered line:
- `working_protocol.md:76`: `*Keine.*`
The consolidated input contains `decision_0004`, which is about a reporting
cycle and is rendered only as background.
### Rendered Section: Technische Details und Infrastruktur / Action Items
Status: faithful to rendered items.
Rendered lines:
- `working_protocol.md:80`
- `working_protocol.md:81`
These correspond to:
- `action_item_0013`
- `action_item_0014`
The semantic quality of `action_item_0014` is questionable, but that issue is
upstream because the consolidated input already categorizes it as an action
item.
### Rendered Section: Technische Details und Infrastruktur / Open Questions
Status: omitted relative to complete input.
Rendered line:
- `working_protocol.md:85`: `*Keine.*`
No technical open questions are clearly required here, but the overall rendered
protocol still omits consolidated open questions elsewhere.
## Exact Mismatches
| Consolidated item | Category before | Rendered text | Category after | Result |
| --- | --- | --- | --- | --- |
| `decision_0004`: `Festlegung eines Reporting-Zyklus...` | decision | `Es gibt einen Reporting-Zyklus...` | background | category weakened |
| `decision_0007`: `...Sitzung mit fünf Leuten...` | decision | none | omitted | omitted |
| `decision_0008`: `...Prozessrahmen nutzen...` | decision | none | omitted | omitted |
| `decision_0010`: `...Verbesserungsvorschläge... ausgeschleust...` | decision | none | omitted | omitted |
| `decision_0012`: `Die Diskussion wird beendet...` | decision | none | omitted | omitted |
| `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen...` | open_question | none | omitted | omitted |
| `open_question_0014`: `Welche Prüfsteine sind relevant...` | open_question | none | omitted | omitted |
| fact: `Im Zweifel gehen Projekte durch.` | fact | `Im Zweifel gehen Projekte durch.` | decision | promoted |
| fact: `Ein Projekt kann abgelehnt werden...` | fact | same meaning | decision | promoted |
| `action_item_0008`: `Diskussion über die Relevanz... Turkmenistan...` | action_item | none | omitted | omitted |
| `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.` | action_item | overlapped by decision/action item text | merged / not preserved as action item | modified |
## Invented Content Check
No renderer-only invented people were verified in this comparison. `Guido` and
`Noah` appear in the rendered protocol, but they are already present in the
consolidated input. This belongs to BUG-003, not BUG-006.
No clear renderer-only invented responsibility was verified in this comparison.
Questionable todos in the rendered protocol are already categorized as action
items in the consolidated input.
## Merged And Split Items
Verified merges:
- Multiple facts and decisions are merged into background paragraphs in
`working_protocol.md:7`, `:9`, `:53` and `:72`.
- `action_item_0010` is not preserved as a separate action item and overlaps
with rendered filter-criteria decision/action-item wording.
Verified splits:
- No clear split from one consolidated item into multiple materially different
rendered items was identified.
## Earliest Point Where The Mismatch Occurs
The earliest point is the Working Protocol Renderer LLM response.
Evidence:
- `metadata.json` records renderer input as
`semantic_consolidator/consolidated_extractions.json` and prompt file
`prompts/working_protocol.md`.
- `raw_model_response.txt` already contains the omissions and category
changes.
- `raw_model_response.txt` and `working_protocol.md` are byte-identical.
Therefore the mismatch is not introduced by renderer post-processing, output
normalization or Markdown generation.
## Verified Root Cause
Verified cause:
The single-pass LLM Working Protocol Renderer does not reliably preserve the
category and coverage constraints of the consolidated input. It generates a
Markdown view that condenses and reorganizes the input, but the generated raw
response omits required items and changes some categories.
Cause classification:
- Renderer prompt: not proven as the sole root cause. The prompt includes
explicit preservation rules, but also includes condensation and redundancy
instructions. The evidence proves the model output violates the preservation
rules; it does not prove which prompt sentence caused the violation.
- Renderer post-processing: disproved. Raw response and final Markdown are
identical.
- Renderer output normalization: disproved. No separate normalization changed
the raw response.
- Markdown generation: disproved. The final file is the raw Markdown response.
- Another verified cause: single-pass LLM renderer generation violates
structural faithfulness requirements.
## Confidence
High for the location of the failure and for disproving post-processing,
normalization and Markdown generation as causes.
Medium for root-cause classification beyond that. The evidence identifies the
renderer LLM response as the failing point, but does not isolate one prompt
sentence as the cause.
## Conceptual Repair Strategy
Do not attempt to repair this with free-form post-processing.
Conceptually, renderer faithfulness needs a structural coverage contract:
- every consolidated decision must be accounted for as a decision
- every consolidated action item must be accounted for as an action item
- every consolidated open question must be accounted for as an open question
- facts may appear in background, but must not be promoted to decisions
- omissions and category changes should be validator-detectable before the
protocol is accepted
The renderer output should be validated against the consolidated input by item
ID or another stable structured reference, rather than relying on prose-only
Markdown to preserve category semantics implicitly.
Reference in New Issue View Git Blame Copy Permalink