Document validation architecture and renderer faithfulness findings

- document Entity Registry and Meeting Context V2 architecture - preserve meeting_context.yaml as the authoritative meeting-specific input - define immutable authoritative metadata across all pipeline stages - restrict Constraint Repair to deterministic structured-data operations - record BUG-003 root cause and deferred entity-verification resolution - document BUG-005 attendance-consistency design - add BUG-006 renderer faithfulness root-cause analysis - distinguish Engineering Readiness from Practical Usability - update the persistent regression bug tracker
2026-08-03 16:05:47 +02:00
parent 06f0e7e651
commit 9446c6e0be
13 changed files with 2208 additions and 0 deletions
@@ -0,0 +1,426 @@
+# BUG-006 Root Cause Analysis
+
+## Observed Behaviour
+
+In benchmark `meeting_context_v1/e2e_current_20260803_151000`, the Working
+Protocol Renderer output is not a faithful view of the matching consolidated
+representation.
+
+Compared files:
+
+- Consolidated input:
+  `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
+- Rendered output:
+  `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
+- Raw renderer response:
+  `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/raw_model_response.txt`
+
+Observed problems:
+
+- consolidated decisions are omitted from the rendered `Decisions` sections
+- consolidated open questions are omitted from rendered `Open Questions`
+  sections
+- consolidated facts are rendered as decisions
+- some consolidated action items are omitted
+- some action-item and decision meanings are merged into other rendered
+  sections
+
+The raw renderer response and `working_protocol.md` are byte-identical
+(`cmp` exit code `0`), so the mismatch is already present in the LLM renderer
+response. It is not introduced by Markdown file writing.
+
+## Renderer Input And Prompt Evidence
+
+Renderer metadata records the exact input and prompt:
+
+- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/metadata.json:2`
+  to `:4`
+
+The renderer prompt explicitly says:
+
+- use only the provided input:
+  `prompts/working_protocol.md:9` to `:10`
+- preserve all decisions, action items and open questions:
+  `prompts/working_protocol.md:24` to `:26`
+- remove presentation-level redundancy:
+  `prompts/working_protocol.md:23`
+- do not repeat information merely because it appears in multiple categories:
+  `prompts/working_protocol.md:65`
+- condense everything else:
+  `prompts/working_protocol.md:75`
+
+The prompt is therefore present and relevant, but the direct observed failure
+occurs in the model-generated renderer response.
+
+## Faithfulness Analysis
+
+### Rendered Section: Prozessrahmen und Projektideen-Eingang / Background
+
+Status: modified.
+
+Rendered lines:
+
+- `working_protocol.md:7`
+- `working_protocol.md:9`
+
+The section condenses several consolidated facts and decisions into background
+paragraphs. Examples:
+
+- `decision_0001` is restated in background and again as a decision.
+- `fact`: `Der Prozess ist ein F&E-Prozess gewesen, einfach historisch.`
+- `fact`: `Die Bearbeitung der Projekte muss und kann nur in den
+  Fachabteilungen passieren.`
+- `fact`: `Bevor F&E ein Projekt durchdenkt, müssen im Vorfeld Kriterien wie
+  Marktexistenz und Zulassungen geprüft werden.`
+- `decision_0003` and `decision_0005` are partially restated as background.
+
+This is a condensation, not a renderer-only invention.
+
+### Rendered Section: Prozessrahmen und Projektideen-Eingang / Decisions
+
+Status: partially faithful, partially omitted.
+
+Faithfully or near-faithfully preserved:
+
+- `decision_0001` -> `working_protocol.md:13`
+- `decision_0002` -> `working_protocol.md:14`
+- `decision_0003` -> `working_protocol.md:15`
+- `decision_0005` -> `working_protocol.md:16`
+- `decision_0006` -> `working_protocol.md:17`
+- `decision_0009` -> `working_protocol.md:18`
+- `decision_0011` -> `working_protocol.md:19`
+
+Omitted as decisions:
+
+- `decision_0004`: `Festlegung eines Reporting-Zyklus für abgelehnte
+  Projekte.`
+  - Consolidated evidence: `consolidated_extractions.json:1217` to `:1227`
+  - Rendered only as background: `working_protocol.md:72`
+- `decision_0007`: `Zusammengetragen und beantwortete Fragen führen zur
+  Sitzung mit fünf Leuten...`
+  - Consolidated evidence: `consolidated_extractions.json:1329` to `:1339`
+  - No matching rendered decision.
+- `decision_0008`: `Es wird vereinbart, den bestehenden Prozessrahmen zu
+  nutzen und auf die Bedürfnisse der F&E abzustimmen...`
+  - Consolidated evidence present in the input.
+  - No matching rendered decision.
+- `decision_0010`: `Es wird vereinbart, dass betriebliche
+  Verbesserungsvorschläge... ausgeschleust werden.`
+  - Consolidated evidence present in the input.
+  - No matching rendered decision.
+- `decision_0012`: `Die Diskussion wird beendet und die Änderungen werden
+  verschickt; Abwarten auf Rückmeldung von Johanna/Björn.`
+  - Consolidated evidence: `consolidated_extractions.json:1747` to `:1757`
+  - No matching rendered decision.
+
+### Rendered Section: Prozessrahmen und Projektideen-Eingang / Action Items
+
+Status: partially faithful, partially omitted, with upstream semantic problems
+preserved.
+
+Rendered lines:
+
+- `working_protocol.md:23` to `:32`
+
+Preserved action items include:
+
+- `action_item_0001` -> `working_protocol.md:23`
+- `action_item_0002` -> `working_protocol.md:24`
+- `action_item_0003` -> `working_protocol.md:25`
+- `action_item_0004` -> `working_protocol.md:26`
+- `action_item_0005` -> `working_protocol.md:27`
+- `action_item_0006` -> `working_protocol.md:28`
+- `action_item_0007` -> `working_protocol.md:29`
+- `action_item_0009` -> `working_protocol.md:30`
+- `action_item_0011` -> `working_protocol.md:31`
+- `action_item_0012` -> `working_protocol.md:32`
+
+Omitted or merged action items:
+
+- `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes
+  (z.B. Turkmenistan) führen.`
+  - Present in consolidated input.
+  - No matching rendered action item.
+- `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.`
+  - Present in consolidated input.
+  - Its meaning overlaps with rendered decision `working_protocol.md:18` and
+    action item `working_protocol.md:30`, but it is not preserved as its own
+    action item.
+
+Important boundary:
+
+Items such as `Feedback zu den Kriterien...`, `Einbinden der Fachbereiche...`
+and `Mail an Jovana und Björn...` are questionable todos, but they are already
+`action_item` entries in the consolidated input:
+
+- `consolidated_extractions.json:1007` to `:1019`
+- `consolidated_extractions.json:1537` to `:1549`
+- `consolidated_extractions.json:1651` to `:1663`
+
+The renderer preserves those upstream action-item categories. It does not
+create those todos from facts or prose.
+
+### Rendered Section: Prozessrahmen und Projektideen-Eingang / Open Questions
+
+Status: partially faithful, partially omitted.
+
+Rendered lines:
+
+- `working_protocol.md:36` to `:47`
+
+Preserved open questions:
+
+- `open_question_0001` -> `working_protocol.md:36`
+- `open_question_0002` -> `working_protocol.md:37`
+- `open_question_0003` -> `working_protocol.md:38`
+- `open_question_0004` -> `working_protocol.md:39`
+- `open_question_0005` -> `working_protocol.md:40`
+- `open_question_0006` -> `working_protocol.md:41`
+- `open_question_0007` -> `working_protocol.md:42`
+- `open_question_0009` -> `working_protocol.md:43`
+- `open_question_0010` -> `working_protocol.md:44`
+- `open_question_0011` -> `working_protocol.md:45`
+- `open_question_0012` -> `working_protocol.md:46`
+- `open_question_0013` -> `working_protocol.md:47`
+
+Omitted open questions:
+
+- `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen und wie
+  wird der Prozess konkret implementiert?`
+  - Consolidated evidence: `consolidated_extractions.json:1387` to `:1397`
+  - No matching rendered open question.
+- `open_question_0014`: `Welche Prüfsteine sind relevant für die
+  Fachabteilung?`
+  - Consolidated evidence: `consolidated_extractions.json:1765` to `:1775`
+  - No matching rendered open question.
+
+### Rendered Section: Projektkategorisierung und Status / Background
+
+Status: modified.
+
+Rendered line:
+
+- `working_protocol.md:53`
+
+This condenses:
+
+- fact: `Der Leiter F&E führt die Projektliste...`
+- `technical_detail_0010`: project type definition
+- fact: `Ein Projekt wandert direkt wieder in Business Development...`
+
+No renderer-only invention was verified in this section.
+
+### Rendered Section: Projektkategorisierung und Status / Decisions
+
+Status: category-changed.
+
+Rendered lines:
+
+- `working_protocol.md:57`
+- `working_protocol.md:58`
+
+Both rendered decisions are facts in the consolidated input:
+
+- fact: `Im Zweifel gehen Projekte durch.`
+  - Consolidated evidence: `consolidated_extractions.json:775` to `:784`
+  - Rendered as decision: `working_protocol.md:57`
+- fact: `Ein Projekt kann abgelehnt werden, aber es muss sichergestellt sein,
+  dass relevante Projekte nicht weggeschmissen werden.`
+  - Consolidated evidence: `consolidated_extractions.json:755` to `:764`
+  - Rendered as decision: `working_protocol.md:58`
+
+This is the clearest category change in the renderer output.
+
+### Rendered Section: Projektkategorisierung und Status / Action Items
+
+Status: omitted.
+
+Rendered line:
+
+- `working_protocol.md:62`: `*Keine.*`
+
+The consolidated input contains at least one action item that belongs to this
+topic area:
+
+- `action_item_0008`: `Diskussion über die Relevanz eines speziellen Marktes
+  (z.B. Turkmenistan) führen.`
+
+The renderer omitted it.
+
+### Rendered Section: Projektkategorisierung und Status / Open Questions
+
+Status: faithful to rendered grouping, but unfaithful to complete input.
+
+Rendered line:
+
+- `working_protocol.md:66`: `*Keine.*`
+
+The complete consolidated input still contains project-process open questions,
+including omitted `open_question_0008` and `open_question_0014`. They were not
+rendered elsewhere.
+
+### Rendered Section: Technische Details und Infrastruktur / Background
+
+Status: modified.
+
+Rendered line:
+
+- `working_protocol.md:72`
+
+This condenses facts and technical details:
+
+- fact: `Das Netzwerk bei Gemeinden ist momentan langsam.`
+- `technical_detail_0006`: `Das Netzwerk ist langsam (wie zu ISDN-Zeiten).`
+- `technical_detail_0007`: `Der Rechner schmiert ab...`
+- fact: `Das Dokument ist das aktuelle Projektdeckblatt...`
+- fact: `Es gibt einen Reporting-Zyklus.`
+- fact: `Sekretariate sind sehr abweisend...`
+
+The problem is not invention; the problem is that `decision_0004`
+(`Festlegung eines Reporting-Zyklus...`) is downgraded from decision to
+background.
+
+### Rendered Section: Technische Details und Infrastruktur / Decisions
+
+Status: omitted.
+
+Rendered line:
+
+- `working_protocol.md:76`: `*Keine.*`
+
+The consolidated input contains `decision_0004`, which is about a reporting
+cycle and is rendered only as background.
+
+### Rendered Section: Technische Details und Infrastruktur / Action Items
+
+Status: faithful to rendered items.
+
+Rendered lines:
+
+- `working_protocol.md:80`
+- `working_protocol.md:81`
+
+These correspond to:
+
+- `action_item_0013`
+- `action_item_0014`
+
+The semantic quality of `action_item_0014` is questionable, but that issue is
+upstream because the consolidated input already categorizes it as an action
+item.
+
+### Rendered Section: Technische Details und Infrastruktur / Open Questions
+
+Status: omitted relative to complete input.
+
+Rendered line:
+
+- `working_protocol.md:85`: `*Keine.*`
+
+No technical open questions are clearly required here, but the overall rendered
+protocol still omits consolidated open questions elsewhere.
+
+## Exact Mismatches
+
+| Consolidated item | Category before | Rendered text | Category after | Result |
+| --- | --- | --- | --- | --- |
+| `decision_0004`: `Festlegung eines Reporting-Zyklus...` | decision | `Es gibt einen Reporting-Zyklus...` | background | category weakened |
+| `decision_0007`: `...Sitzung mit fünf Leuten...` | decision | none | omitted | omitted |
+| `decision_0008`: `...Prozessrahmen nutzen...` | decision | none | omitted | omitted |
+| `decision_0010`: `...Verbesserungsvorschläge... ausgeschleust...` | decision | none | omitted | omitted |
+| `decision_0012`: `Die Diskussion wird beendet...` | decision | none | omitted | omitted |
+| `open_question_0008`: `Wer soll die Rolle des Gatekeepers übernehmen...` | open_question | none | omitted | omitted |
+| `open_question_0014`: `Welche Prüfsteine sind relevant...` | open_question | none | omitted | omitted |
+| fact: `Im Zweifel gehen Projekte durch.` | fact | `Im Zweifel gehen Projekte durch.` | decision | promoted |
+| fact: `Ein Projekt kann abgelehnt werden...` | fact | same meaning | decision | promoted |
+| `action_item_0008`: `Diskussion über die Relevanz... Turkmenistan...` | action_item | none | omitted | omitted |
+| `action_item_0010`: `Filterkriterien für verschiedene Fälle... definieren.` | action_item | overlapped by decision/action item text | merged / not preserved as action item | modified |
+
+## Invented Content Check
+
+No renderer-only invented people were verified in this comparison. `Guido` and
+`Noah` appear in the rendered protocol, but they are already present in the
+consolidated input. This belongs to BUG-003, not BUG-006.
+
+No clear renderer-only invented responsibility was verified in this comparison.
+Questionable todos in the rendered protocol are already categorized as action
+items in the consolidated input.
+
+## Merged And Split Items
+
+Verified merges:
+
+- Multiple facts and decisions are merged into background paragraphs in
+  `working_protocol.md:7`, `:9`, `:53` and `:72`.
+- `action_item_0010` is not preserved as a separate action item and overlaps
+  with rendered filter-criteria decision/action-item wording.
+
+Verified splits:
+
+- No clear split from one consolidated item into multiple materially different
+  rendered items was identified.
+
+## Earliest Point Where The Mismatch Occurs
+
+The earliest point is the Working Protocol Renderer LLM response.
+
+Evidence:
+
+- `metadata.json` records renderer input as
+  `semantic_consolidator/consolidated_extractions.json` and prompt file
+  `prompts/working_protocol.md`.
+- `raw_model_response.txt` already contains the omissions and category
+  changes.
+- `raw_model_response.txt` and `working_protocol.md` are byte-identical.
+
+Therefore the mismatch is not introduced by renderer post-processing, output
+normalization or Markdown generation.
+
+## Verified Root Cause
+
+Verified cause:
+
+The single-pass LLM Working Protocol Renderer does not reliably preserve the
+category and coverage constraints of the consolidated input. It generates a
+Markdown view that condenses and reorganizes the input, but the generated raw
+response omits required items and changes some categories.
+
+Cause classification:
+
+- Renderer prompt: not proven as the sole root cause. The prompt includes
+  explicit preservation rules, but also includes condensation and redundancy
+  instructions. The evidence proves the model output violates the preservation
+  rules; it does not prove which prompt sentence caused the violation.
+- Renderer post-processing: disproved. Raw response and final Markdown are
+  identical.
+- Renderer output normalization: disproved. No separate normalization changed
+  the raw response.
+- Markdown generation: disproved. The final file is the raw Markdown response.
+- Another verified cause: single-pass LLM renderer generation violates
+  structural faithfulness requirements.
+
+## Confidence
+
+High for the location of the failure and for disproving post-processing,
+normalization and Markdown generation as causes.
+
+Medium for root-cause classification beyond that. The evidence identifies the
+renderer LLM response as the failing point, but does not isolate one prompt
+sentence as the cause.
+
+## Conceptual Repair Strategy
+
+Do not attempt to repair this with free-form post-processing.
+
+Conceptually, renderer faithfulness needs a structural coverage contract:
+
+- every consolidated decision must be accounted for as a decision
+- every consolidated action item must be accounted for as an action item
+- every consolidated open question must be accounted for as an open question
+- facts may appear in background, but must not be promoted to decisions
+- omissions and category changes should be validator-detectable before the
+  protocol is accepted
+
+The renderer output should be validated against the consolidated input by item
+ID or another stable structured reference, rather than relying on prose-only
+Markdown to preserve category semantics implicitly.