meeting-lab/docs/bug-006-root-cause.md

BUG-006 Root Cause Analysis

Observed Behaviour

In benchmark meeting_context_v1/e2e_current_20260803_151000, the Working Protocol Renderer output is not a faithful view of the matching consolidated representation.

Compared files:

• Consolidated input: samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json
• Rendered output: samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md
• Raw renderer response: samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/raw_model_response.txt

Observed problems:

• consolidated decisions are omitted from the rendered Decisions sections
• consolidated open questions are omitted from rendered Open Questions sections
• consolidated facts are rendered as decisions
• some consolidated action items are omitted
• some action-item and decision meanings are merged into other rendered sections

The raw renderer response and working_protocol.md are byte-identical (cmp exit code 0), so the mismatch is already present in the LLM renderer response. It is not introduced by Markdown file writing.

Renderer Input And Prompt Evidence

Renderer metadata records the exact input and prompt:

• samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/metadata.json:2 to :4

The renderer prompt explicitly says:

• use only the provided input: prompts/working_protocol.md:9 to :10
• preserve all decisions, action items and open questions: prompts/working_protocol.md:24 to :26
• remove presentation-level redundancy: prompts/working_protocol.md:23
• do not repeat information merely because it appears in multiple categories: prompts/working_protocol.md:65
• condense everything else: prompts/working_protocol.md:75

The prompt is therefore present and relevant, but the direct observed failure occurs in the model-generated renderer response.

Faithfulness Analysis

Rendered Section: Prozessrahmen und Projektideen-Eingang / Background

Status: modified.

Rendered lines:

• working_protocol.md:7
• working_protocol.md:9

The section condenses several consolidated facts and decisions into background paragraphs. Examples:

• decision_0001 is restated in background and again as a decision.
• fact: Der Prozess ist ein F&E-Prozess gewesen, einfach historisch.
• fact: Die Bearbeitung der Projekte muss und kann nur in den Fachabteilungen passieren.
• fact: Bevor F&E ein Projekt durchdenkt, müssen im Vorfeld Kriterien wie Marktexistenz und Zulassungen geprüft werden.
• decision_0003 and decision_0005 are partially restated as background.

This is a condensation, not a renderer-only invention.

Rendered Section: Prozessrahmen und Projektideen-Eingang / Decisions

Status: partially faithful, partially omitted.

Faithfully or near-faithfully preserved:

• decision_0001 -> working_protocol.md:13
• decision_0002 -> working_protocol.md:14
• decision_0003 -> working_protocol.md:15
• decision_0005 -> working_protocol.md:16
• decision_0006 -> working_protocol.md:17
• decision_0009 -> working_protocol.md:18
• decision_0011 -> working_protocol.md:19

Omitted as decisions:

• decision_0004: Festlegung eines Reporting-Zyklus für abgelehnte Projekte.
  • Consolidated evidence: consolidated_extractions.json:1217 to :1227
  • Rendered only as background: working_protocol.md:72
• decision_0007: Zusammengetragen und beantwortete Fragen führen zur Sitzung mit fünf Leuten...
  • Consolidated evidence: consolidated_extractions.json:1329 to :1339
  • No matching rendered decision.
• decision_0008: Es wird vereinbart, den bestehenden Prozessrahmen zu nutzen und auf die Bedürfnisse der F&E abzustimmen...
  • Consolidated evidence present in the input.
  • No matching rendered decision.
• decision_0010: Es wird vereinbart, dass betriebliche Verbesserungsvorschläge... ausgeschleust werden.
  • Consolidated evidence present in the input.
  • No matching rendered decision.
• decision_0012: Die Diskussion wird beendet und die Änderungen werden verschickt; Abwarten auf Rückmeldung von Johanna/Björn.
  • Consolidated evidence: consolidated_extractions.json:1747 to :1757
  • No matching rendered decision.

Rendered Section: Prozessrahmen und Projektideen-Eingang / Action Items

Status: partially faithful, partially omitted, with upstream semantic problems preserved.

Rendered lines:

• working_protocol.md:23 to :32

Preserved action items include:

• action_item_0001 -> working_protocol.md:23
• action_item_0002 -> working_protocol.md:24
• action_item_0003 -> working_protocol.md:25
• action_item_0004 -> working_protocol.md:26
• action_item_0005 -> working_protocol.md:27
• action_item_0006 -> working_protocol.md:28
• action_item_0007 -> working_protocol.md:29
• action_item_0009 -> working_protocol.md:30
• action_item_0011 -> working_protocol.md:31
• action_item_0012 -> working_protocol.md:32

Omitted or merged action items:

• action_item_0008: Diskussion über die Relevanz eines speziellen Marktes (z.B. Turkmenistan) führen.
  • Present in consolidated input.
  • No matching rendered action item.
• action_item_0010: Filterkriterien für verschiedene Fälle... definieren.
  • Present in consolidated input.
  • Its meaning overlaps with rendered decision working_protocol.md:18 and action item working_protocol.md:30, but it is not preserved as its own action item.

Important boundary:

Items such as Feedback zu den Kriterien..., Einbinden der Fachbereiche... and Mail an Jovana und Björn... are questionable todos, but they are already action_item entries in the consolidated input:

• consolidated_extractions.json:1007 to :1019
• consolidated_extractions.json:1537 to :1549
• consolidated_extractions.json:1651 to :1663

The renderer preserves those upstream action-item categories. It does not create those todos from facts or prose.

Rendered Section: Prozessrahmen und Projektideen-Eingang / Open Questions

Status: partially faithful, partially omitted.

Rendered lines:

• working_protocol.md:36 to :47

Preserved open questions:

• open_question_0001 -> working_protocol.md:36
• open_question_0002 -> working_protocol.md:37
• open_question_0003 -> working_protocol.md:38
• open_question_0004 -> working_protocol.md:39
• open_question_0005 -> working_protocol.md:40
• open_question_0006 -> working_protocol.md:41
• open_question_0007 -> working_protocol.md:42
• open_question_0009 -> working_protocol.md:43
• open_question_0010 -> working_protocol.md:44
• open_question_0011 -> working_protocol.md:45
• open_question_0012 -> working_protocol.md:46
• open_question_0013 -> working_protocol.md:47

Omitted open questions:

• open_question_0008: Wer soll die Rolle des Gatekeepers übernehmen und wie wird der Prozess konkret implementiert?
  • Consolidated evidence: consolidated_extractions.json:1387 to :1397
  • No matching rendered open question.
• open_question_0014: Welche Prüfsteine sind relevant für die Fachabteilung?
  • Consolidated evidence: consolidated_extractions.json:1765 to :1775
  • No matching rendered open question.

Rendered Section: Projektkategorisierung und Status / Background

Status: modified.

Rendered line:

• working_protocol.md:53

This condenses:

• fact: Der Leiter F&E führt die Projektliste...
• technical_detail_0010: project type definition
• fact: Ein Projekt wandert direkt wieder in Business Development...

No renderer-only invention was verified in this section.

Rendered Section: Projektkategorisierung und Status / Decisions

Status: category-changed.

Rendered lines:

• working_protocol.md:57
• working_protocol.md:58

Both rendered decisions are facts in the consolidated input:

• fact: Im Zweifel gehen Projekte durch.
  • Consolidated evidence: consolidated_extractions.json:775 to :784
  • Rendered as decision: working_protocol.md:57
• fact: Ein Projekt kann abgelehnt werden, aber es muss sichergestellt sein, dass relevante Projekte nicht weggeschmissen werden.
  • Consolidated evidence: consolidated_extractions.json:755 to :764
  • Rendered as decision: working_protocol.md:58

This is the clearest category change in the renderer output.

Rendered Section: Projektkategorisierung und Status / Action Items

Status: omitted.

Rendered line:

• working_protocol.md:62: *Keine.*

The consolidated input contains at least one action item that belongs to this topic area:

• action_item_0008: Diskussion über die Relevanz eines speziellen Marktes (z.B. Turkmenistan) führen.

The renderer omitted it.

Rendered Section: Projektkategorisierung und Status / Open Questions

Status: faithful to rendered grouping, but unfaithful to complete input.

Rendered line:

• working_protocol.md:66: *Keine.*

The complete consolidated input still contains project-process open questions, including omitted open_question_0008 and open_question_0014. They were not rendered elsewhere.

Rendered Section: Technische Details und Infrastruktur / Background

Status: modified.

Rendered line:

• working_protocol.md:72

This condenses facts and technical details:

• fact: Das Netzwerk bei Gemeinden ist momentan langsam.
• technical_detail_0006: Das Netzwerk ist langsam (wie zu ISDN-Zeiten).
• technical_detail_0007: Der Rechner schmiert ab...
• fact: Das Dokument ist das aktuelle Projektdeckblatt...
• fact: Es gibt einen Reporting-Zyklus.
• fact: Sekretariate sind sehr abweisend...

The problem is not invention; the problem is that decision_0004 (Festlegung eines Reporting-Zyklus...) is downgraded from decision to background.

Rendered Section: Technische Details und Infrastruktur / Decisions

Status: omitted.

Rendered line:

• working_protocol.md:76: *Keine.*

The consolidated input contains decision_0004, which is about a reporting cycle and is rendered only as background.

Rendered Section: Technische Details und Infrastruktur / Action Items

Status: faithful to rendered items.

Rendered lines:

• working_protocol.md:80
• working_protocol.md:81

These correspond to:

• action_item_0013
• action_item_0014

The semantic quality of action_item_0014 is questionable, but that issue is upstream because the consolidated input already categorizes it as an action item.

Rendered Section: Technische Details und Infrastruktur / Open Questions

Status: omitted relative to complete input.

Rendered line:

• working_protocol.md:85: *Keine.*

No technical open questions are clearly required here, but the overall rendered protocol still omits consolidated open questions elsewhere.

Exact Mismatches

Consolidated item	Category before	Rendered text	Category after	Result
decision_0004: Festlegung eines Reporting-Zyklus...	decision	Es gibt einen Reporting-Zyklus...	background	category weakened
decision_0007: ...Sitzung mit fünf Leuten...	decision	none	omitted	omitted
decision_0008: ...Prozessrahmen nutzen...	decision	none	omitted	omitted
decision_0010: ...Verbesserungsvorschläge... ausgeschleust...	decision	none	omitted	omitted
decision_0012: Die Diskussion wird beendet...	decision	none	omitted	omitted
open_question_0008: Wer soll die Rolle des Gatekeepers übernehmen...	open_question	none	omitted	omitted
open_question_0014: Welche Prüfsteine sind relevant...	open_question	none	omitted	omitted
fact: Im Zweifel gehen Projekte durch.	fact	Im Zweifel gehen Projekte durch.	decision	promoted
fact: Ein Projekt kann abgelehnt werden...	fact	same meaning	decision	promoted
action_item_0008: Diskussion über die Relevanz... Turkmenistan...	action_item	none	omitted	omitted
action_item_0010: Filterkriterien für verschiedene Fälle... definieren.	action_item	overlapped by decision/action item text	merged / not preserved as action item	modified

Invented Content Check

No renderer-only invented people were verified in this comparison. Guido and Noah appear in the rendered protocol, but they are already present in the consolidated input. This belongs to BUG-003, not BUG-006.

No clear renderer-only invented responsibility was verified in this comparison. Questionable todos in the rendered protocol are already categorized as action items in the consolidated input.

Merged And Split Items

Verified merges:

• Multiple facts and decisions are merged into background paragraphs in working_protocol.md:7, :9, :53 and :72.
• action_item_0010 is not preserved as a separate action item and overlaps with rendered filter-criteria decision/action-item wording.

Verified splits:

• No clear split from one consolidated item into multiple materially different rendered items was identified.

Earliest Point Where The Mismatch Occurs

The earliest point is the Working Protocol Renderer LLM response.

Evidence:

• metadata.json records renderer input as semantic_consolidator/consolidated_extractions.json and prompt file prompts/working_protocol.md.
• raw_model_response.txt already contains the omissions and category changes.
• raw_model_response.txt and working_protocol.md are byte-identical.

Therefore the mismatch is not introduced by renderer post-processing, output normalization or Markdown generation.

Verified Root Cause

Verified cause:

The single-pass LLM Working Protocol Renderer does not reliably preserve the category and coverage constraints of the consolidated input. It generates a Markdown view that condenses and reorganizes the input, but the generated raw response omits required items and changes some categories.

Cause classification:

• Renderer prompt: not proven as the sole root cause. The prompt includes explicit preservation rules, but also includes condensation and redundancy instructions. The evidence proves the model output violates the preservation rules; it does not prove which prompt sentence caused the violation.
• Renderer post-processing: disproved. Raw response and final Markdown are identical.
• Renderer output normalization: disproved. No separate normalization changed the raw response.
• Markdown generation: disproved. The final file is the raw Markdown response.
• Another verified cause: single-pass LLM renderer generation violates structural faithfulness requirements.

Confidence

High for the location of the failure and for disproving post-processing, normalization and Markdown generation as causes.

Medium for root-cause classification beyond that. The evidence identifies the renderer LLM response as the failing point, but does not isolate one prompt sentence as the cause.

Conceptual Repair Strategy

Do not attempt to repair this with free-form post-processing.

Conceptually, renderer faithfulness needs a structural coverage contract:

• every consolidated decision must be accounted for as a decision
• every consolidated action item must be accounted for as an action item
• every consolidated open question must be accounted for as an open question
• facts may appear in background, but must not be promoted to decisions
• omissions and category changes should be validator-detectable before the protocol is accepted

The renderer output should be validated against the consolidated input by item ID or another stable structured reference, rather than relying on prose-only Markdown to preserve category semantics implicitly.