# BUG-006 Root Cause Analysis ## Observed Behaviour In benchmark `meeting_context_v1/e2e_current_20260803_151000`, the Working Protocol Renderer output is not a faithful view of the matching consolidated representation. Compared files: - Consolidated input: `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json` - Rendered output: `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md` - Raw renderer response: `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/raw_model_response.txt` Observed problems: - consolidated decisions are omitted from the rendered `Decisions` sections - consolidated open questions are omitted from rendered `Open Questions` sections - consolidated facts are rendered as decisions - some consolidated action items are omitted - some action-item and decision meanings are merged into other rendered sections The raw renderer response and `working_protocol.md` are byte-identical (`cmp` exit code `0`), so the mismatch is already present in the LLM renderer response. It is not introduced by Markdown file writing. ## Renderer Input And Prompt Evidence Renderer metadata records the exact input and prompt: - `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/metadata.json:2` to `:4` The renderer prompt explicitly says: - use only the provided input: `prompts/working_protocol.md:9` to `:10` - preserve all decisions, action items and open questions: `prompts/working_protocol.md:24` to `:26` - remove presentation-level redundancy: `prompts/working_protocol.md:23` - do not repeat information merely because it appears in multiple categories: `prompts/working_protocol.md:65` - condense everything else: `prompts/working_protocol.md:75` The prompt is therefore present and relevant, but the direct observed failure occurs in the model-generated renderer response. ## Faithfulness Analysis ### Rendered Section: Prozessrahmen und Projektideen-Eingang / Background Status: modified. Rendered lines: - `working_protocol.md:7` - `working_protocol.md:9` The section condenses several consolidated facts and decisions into background paragraphs. Examples: - `decision_0001` is restated in background and again as a decision. - `fact`: `Der Prozess ist ein F&E-Prozess gewesen, einfach historisch.` - `fact`: `Die Bearbeitung der Projekte muss und kann nur in den Fachabteilungen passieren.` - `fact`: `Bevor F&E ein Projekt durchdenkt, müssen im Vorfeld Kriterien wie Marktexistenz und Zulassungen geprüft werden.` - `decision_0003` and `decision_0005` are partially restated as background. This is a condensation, not a renderer-only invention. ### Rendered Section: Prozessrahmen und Projektideen-Eingang / Decisions Status: partially faithful, partially omitted. Faithfully or near-faithfully preserved: - `decision_0001` -> `working_protocol.md:13` - `decision_0002` -> `working_protocol.md:14` - `decision_0003` -> `working_protocol.md:15` - `decision_0005` -> `working_protocol.md:16` - `decision_0006` -> `working_protocol.md:17` - `decision_0009` -> `working_protocol.md:18` - `decision_0011` -> `working_protocol.md:19` Omitted as decisions: - `decision_0004`: `Festlegung eines Reporting-Zyklus für abgelehnte Projekte.` - Consolidated evidence: `consolidated_extractions.json:1217` to `:1227` - Rendered only as background: `working_protocol.md:72` - `decision_0007`: `Zusammengetragen und beantwortete Fragen führen zur Sitzung mit fünf Leuten...` - Consolidated evidence: `consolidated_extractions.json:1329` to `:1339` - No matching rendered decision. - `decision_0008`: `Es wird vereinbart, den bestehenden Prozessrahmen zu nutzen und auf die Bedürfnisse der F&E abzustimmen...` - Consolidated evidence present in the input. - No matching rendered decision. - `decision_0010`: `Es wird vereinbart, dass betriebliche Verbesserungsvorschläge... ausgeschleust werden.` - Consolidated evidence present in the input. - No matching rendered decision. - `decision_0012`: `Die Diskussion wird beendet und die Änderungen werden verschickt; Abwarten auf Rückmeldung von Johanna/Björn.` - Consolidated evidence: `consolidated_extractions.json:1747` to `:1757` - No matching rendered decision. ### Rendered Section: Prozessrahmen und Projektideen-Eingang / Action Items Status: partially faithful, partially omitted, with upstream semantic problems preserved. Rendered lines: - `working_protocol.md:23` to `:32` Preserved action items include: - `action_item_0001` -> `working_protocol.md:23` - `action_item_0002` -> `working_protocol.md:24` - `action_item_0003` -> `working_protocol.md:25` - `action_item_0004` -> `working_protocol.md:26` - `action_item_0005` -> `working_protocol.md:27` - `action_item_0006` -> `working_protocol.md:28` - `action_item_0007` -> `working_protocol.md:29` - `action_item_0009` -> `working_protocol.md:30` - `action_item_0011` -> `working_protocol.md:31` - `action_item_0012` -> `working_protocol.md:32` Omitted or merged action items: - `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes (z.B. Turkmenistan) führen.` - Present in consolidated input. - No matching rendered action item. - `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.` - Present in consolidated input. - Its meaning overlaps with rendered decision `working_protocol.md:18` and action item `working_protocol.md:30`, but it is not preserved as its own action item. Important boundary: Items such as `Feedback zu den Kriterien...`, `Einbinden der Fachbereiche...` and `Mail an Jovana und Björn...` are questionable todos, but they are already `action_item` entries in the consolidated input: - `consolidated_extractions.json:1007` to `:1019` - `consolidated_extractions.json:1537` to `:1549` - `consolidated_extractions.json:1651` to `:1663` The renderer preserves those upstream action-item categories. It does not create those todos from facts or prose. ### Rendered Section: Prozessrahmen und Projektideen-Eingang / Open Questions Status: partially faithful, partially omitted. Rendered lines: - `working_protocol.md:36` to `:47` Preserved open questions: - `open_question_0001` -> `working_protocol.md:36` - `open_question_0002` -> `working_protocol.md:37` - `open_question_0003` -> `working_protocol.md:38` - `open_question_0004` -> `working_protocol.md:39` - `open_question_0005` -> `working_protocol.md:40` - `open_question_0006` -> `working_protocol.md:41` - `open_question_0007` -> `working_protocol.md:42` - `open_question_0009` -> `working_protocol.md:43` - `open_question_0010` -> `working_protocol.md:44` - `open_question_0011` -> `working_protocol.md:45` - `open_question_0012` -> `working_protocol.md:46` - `open_question_0013` -> `working_protocol.md:47` Omitted open questions: - `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen und wie wird der Prozess konkret implementiert?` - Consolidated evidence: `consolidated_extractions.json:1387` to `:1397` - No matching rendered open question. - `open_question_0014`: `Welche Prüfsteine sind relevant für die Fachabteilung?` - Consolidated evidence: `consolidated_extractions.json:1765` to `:1775` - No matching rendered open question. ### Rendered Section: Projektkategorisierung und Status / Background Status: modified. Rendered line: - `working_protocol.md:53` This condenses: - fact: `Der Leiter F&E führt die Projektliste...` - `technical_detail_0010`: project type definition - fact: `Ein Projekt wandert direkt wieder in Business Development...` No renderer-only invention was verified in this section. ### Rendered Section: Projektkategorisierung und Status / Decisions Status: category-changed. Rendered lines: - `working_protocol.md:57` - `working_protocol.md:58` Both rendered decisions are facts in the consolidated input: - fact: `Im Zweifel gehen Projekte durch.` - Consolidated evidence: `consolidated_extractions.json:775` to `:784` - Rendered as decision: `working_protocol.md:57` - fact: `Ein Projekt kann abgelehnt werden, aber es muss sichergestellt sein, dass relevante Projekte nicht weggeschmissen werden.` - Consolidated evidence: `consolidated_extractions.json:755` to `:764` - Rendered as decision: `working_protocol.md:58` This is the clearest category change in the renderer output. ### Rendered Section: Projektkategorisierung und Status / Action Items Status: omitted. Rendered line: - `working_protocol.md:62`: `*Keine.*` The consolidated input contains at least one action item that belongs to this topic area: - `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes (z.B. Turkmenistan) führen.` The renderer omitted it. ### Rendered Section: Projektkategorisierung und Status / Open Questions Status: faithful to rendered grouping, but unfaithful to complete input. Rendered line: - `working_protocol.md:66`: `*Keine.*` The complete consolidated input still contains project-process open questions, including omitted `open_question_0008` and `open_question_0014`. They were not rendered elsewhere. ### Rendered Section: Technische Details und Infrastruktur / Background Status: modified. Rendered line: - `working_protocol.md:72` This condenses facts and technical details: - fact: `Das Netzwerk bei Gemeinden ist momentan langsam.` - `technical_detail_0006`: `Das Netzwerk ist langsam (wie zu ISDN-Zeiten).` - `technical_detail_0007`: `Der Rechner schmiert ab...` - fact: `Das Dokument ist das aktuelle Projektdeckblatt...` - fact: `Es gibt einen Reporting-Zyklus.` - fact: `Sekretariate sind sehr abweisend...` The problem is not invention; the problem is that `decision_0004` (`Festlegung eines Reporting-Zyklus...`) is downgraded from decision to background. ### Rendered Section: Technische Details und Infrastruktur / Decisions Status: omitted. Rendered line: - `working_protocol.md:76`: `*Keine.*` The consolidated input contains `decision_0004`, which is about a reporting cycle and is rendered only as background. ### Rendered Section: Technische Details und Infrastruktur / Action Items Status: faithful to rendered items. Rendered lines: - `working_protocol.md:80` - `working_protocol.md:81` These correspond to: - `action_item_0013` - `action_item_0014` The semantic quality of `action_item_0014` is questionable, but that issue is upstream because the consolidated input already categorizes it as an action item. ### Rendered Section: Technische Details und Infrastruktur / Open Questions Status: omitted relative to complete input. Rendered line: - `working_protocol.md:85`: `*Keine.*` No technical open questions are clearly required here, but the overall rendered protocol still omits consolidated open questions elsewhere. ## Exact Mismatches | Consolidated item | Category before | Rendered text | Category after | Result | | --- | --- | --- | --- | --- | | `decision_0004`: `Festlegung eines Reporting-Zyklus...` | decision | `Es gibt einen Reporting-Zyklus...` | background | category weakened | | `decision_0007`: `...Sitzung mit fünf Leuten...` | decision | none | omitted | omitted | | `decision_0008`: `...Prozessrahmen nutzen...` | decision | none | omitted | omitted | | `decision_0010`: `...Verbesserungsvorschläge... ausgeschleust...` | decision | none | omitted | omitted | | `decision_0012`: `Die Diskussion wird beendet...` | decision | none | omitted | omitted | | `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen...` | open_question | none | omitted | omitted | | `open_question_0014`: `Welche Prüfsteine sind relevant...` | open_question | none | omitted | omitted | | fact: `Im Zweifel gehen Projekte durch.` | fact | `Im Zweifel gehen Projekte durch.` | decision | promoted | | fact: `Ein Projekt kann abgelehnt werden...` | fact | same meaning | decision | promoted | | `action_item_0008`: `Diskussion über die Relevanz... Turkmenistan...` | action_item | none | omitted | omitted | | `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.` | action_item | overlapped by decision/action item text | merged / not preserved as action item | modified | ## Invented Content Check No renderer-only invented people were verified in this comparison. `Guido` and `Noah` appear in the rendered protocol, but they are already present in the consolidated input. This belongs to BUG-003, not BUG-006. No clear renderer-only invented responsibility was verified in this comparison. Questionable todos in the rendered protocol are already categorized as action items in the consolidated input. ## Merged And Split Items Verified merges: - Multiple facts and decisions are merged into background paragraphs in `working_protocol.md:7`, `:9`, `:53` and `:72`. - `action_item_0010` is not preserved as a separate action item and overlaps with rendered filter-criteria decision/action-item wording. Verified splits: - No clear split from one consolidated item into multiple materially different rendered items was identified. ## Earliest Point Where The Mismatch Occurs The earliest point is the Working Protocol Renderer LLM response. Evidence: - `metadata.json` records renderer input as `semantic_consolidator/consolidated_extractions.json` and prompt file `prompts/working_protocol.md`. - `raw_model_response.txt` already contains the omissions and category changes. - `raw_model_response.txt` and `working_protocol.md` are byte-identical. Therefore the mismatch is not introduced by renderer post-processing, output normalization or Markdown generation. ## Verified Root Cause Verified cause: The single-pass LLM Working Protocol Renderer does not reliably preserve the category and coverage constraints of the consolidated input. It generates a Markdown view that condenses and reorganizes the input, but the generated raw response omits required items and changes some categories. Cause classification: - Renderer prompt: not proven as the sole root cause. The prompt includes explicit preservation rules, but also includes condensation and redundancy instructions. The evidence proves the model output violates the preservation rules; it does not prove which prompt sentence caused the violation. - Renderer post-processing: disproved. Raw response and final Markdown are identical. - Renderer output normalization: disproved. No separate normalization changed the raw response. - Markdown generation: disproved. The final file is the raw Markdown response. - Another verified cause: single-pass LLM renderer generation violates structural faithfulness requirements. ## Confidence High for the location of the failure and for disproving post-processing, normalization and Markdown generation as causes. Medium for root-cause classification beyond that. The evidence identifies the renderer LLM response as the failing point, but does not isolate one prompt sentence as the cause. ## Conceptual Repair Strategy Do not attempt to repair this with free-form post-processing. Conceptually, renderer faithfulness needs a structural coverage contract: - every consolidated decision must be accounted for as a decision - every consolidated action item must be accounted for as an action item - every consolidated open question must be accounted for as an open question - facts may appear in background, but must not be promoted to decisions - omissions and category changes should be validator-detectable before the protocol is accepted The renderer output should be validated against the consolidated input by item ID or another stable structured reference, rather than relying on prose-only Markdown to preserve category semantics implicitly.