Document validation architecture and renderer faithfulness findings
- document Entity Registry and Meeting Context V2 architecture - preserve meeting_context.yaml as the authoritative meeting-specific input - define immutable authoritative metadata across all pipeline stages - restrict Constraint Repair to deterministic structured-data operations - record BUG-003 root cause and deferred entity-verification resolution - document BUG-005 attendance-consistency design - add BUG-006 renderer faithfulness root-cause analysis - distinguish Engineering Readiness from Practical Usability - update the persistent regression bug tracker
This commit is contained in:
@@ -0,0 +1,478 @@
|
||||
# Regression Bug Tracker
|
||||
|
||||
Living tracker for real bugs discovered during end-to-end Meeting Lab
|
||||
evaluation.
|
||||
|
||||
Purpose:
|
||||
|
||||
- prevent forgotten regressions
|
||||
- document known root causes
|
||||
- record fixes when they happen
|
||||
- verify that bugs never silently return
|
||||
|
||||
## BUG-001
|
||||
|
||||
ID: BUG-001
|
||||
|
||||
Title: Renderer reverses the intended process direction
|
||||
|
||||
Pipeline stage: Working Protocol Renderer
|
||||
|
||||
Severity: High
|
||||
|
||||
Status: Open
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/final_protocol_with_context`
|
||||
|
||||
Description:
|
||||
|
||||
The generated working protocol phrases the process direction as if the
|
||||
existing F&E process is broadly taken over for all project types. The intended
|
||||
meaning from the evaluation is more constrained: the existing F&E process is a
|
||||
framework that may be adapted or extended for broader project handling.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
The protocol should preserve the direction and uncertainty of the source
|
||||
discussion: F&E process/framework adapted for other project types where
|
||||
appropriate, without implying unconditional adoption for all project types.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The protocol states that the existing F&E process is generally adopted for all
|
||||
project types.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
Unknown.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Open. Documented from end-to-end evaluation output.
|
||||
|
||||
Notes:
|
||||
|
||||
Do not treat this entry as a prompt-change instruction. It records the observed
|
||||
bug only.
|
||||
|
||||
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
|
||||
the issue remains. The new protocol still says the existing F&E process is
|
||||
basically taken over for all project types.
|
||||
|
||||
## BUG-002
|
||||
|
||||
ID: BUG-002
|
||||
|
||||
Title: Jovana todo is strengthened beyond the meeting content
|
||||
|
||||
Pipeline stage: Extraction / Working Protocol Renderer
|
||||
|
||||
Severity: High
|
||||
|
||||
Status: Open
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/final_protocol_with_context`
|
||||
|
||||
Description:
|
||||
|
||||
The generated output strengthens a proposal or discussion about Jovana
|
||||
assembling criteria into an assigned todo. The evaluation identified this as
|
||||
overstating the meeting content.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
The pipeline should preserve the weaker source meaning unless the transcript
|
||||
explicitly assigns, accepts or confirms responsibility.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The consolidated input and final protocol include an action item assigning
|
||||
criterion compilation to Jovana.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
Unknown.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `prompts/todos.md`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Open. Documented from end-to-end evaluation output.
|
||||
|
||||
Notes:
|
||||
|
||||
This bug is subject to the responsibility attribution invariant in
|
||||
`AGENTS.md`.
|
||||
|
||||
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
|
||||
the issue remains. The consolidated output still contains `Zusammenstellen der
|
||||
Kriterien durch Jovana` with `responsible: "Jovana"`, and the final protocol
|
||||
renders it as an action item.
|
||||
|
||||
## BUG-003
|
||||
|
||||
ID: BUG-003
|
||||
|
||||
Title: Unexpected synthetic person names appear in generated protocols
|
||||
|
||||
Pipeline stage: Extraction / Working Protocol Renderer
|
||||
|
||||
Severity: Medium
|
||||
|
||||
Status: Root Cause Identified
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/final_protocol_with_context`
|
||||
|
||||
Description:
|
||||
|
||||
The generated protocol contains unexpected or risky names/aliases such as
|
||||
`Guido` and `Noah`. These names were flagged during evaluation as synthetic or
|
||||
inconsistent in the generated protocol context.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
The protocol should use only names supported by the transcript, consolidated
|
||||
input or Meeting Context, and should preserve uncertainty when a name or alias
|
||||
is unclear.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The final protocol includes names or name variants that were flagged as
|
||||
unexpected in evaluation.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
The unexpected names already exist in the Whisper transcript before extraction.
|
||||
They are propagated through the pipeline and are not introduced by prompts,
|
||||
gold tests or the renderer.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
|
||||
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/real_live/project_process_meeting/meeting_context.yaml`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Root Cause Identified. Planned resolution is deferred until Meeting Context V2
|
||||
/ Entity Registry implementation.
|
||||
|
||||
Notes:
|
||||
|
||||
Resolution strategy:
|
||||
|
||||
Introduce an interactive entity verification step after Whisper transcription:
|
||||
|
||||
```text
|
||||
Whisper
|
||||
↓
|
||||
Entity Detection
|
||||
↓
|
||||
User Verification
|
||||
↓
|
||||
Meeting Context Builder
|
||||
↓
|
||||
Extraction
|
||||
```
|
||||
|
||||
Expected effect:
|
||||
|
||||
Unknown or suspicious person names are detected before extraction begins and
|
||||
can be classified by the user as participant, mentioned person, external
|
||||
person, transcription error or ignore.
|
||||
|
||||
Implementation status:
|
||||
|
||||
Architecture accepted. Implementation deferred.
|
||||
|
||||
Regression test:
|
||||
|
||||
Not yet possible before Meeting Context V2 exists.
|
||||
|
||||
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
|
||||
the issue remains. `Guido` and `Noah` both appear in the generated working
|
||||
protocol.
|
||||
|
||||
## BUG-004
|
||||
|
||||
ID: BUG-004
|
||||
|
||||
Title: Rhetorical question extracted as an open question
|
||||
|
||||
Pipeline stage: Extraction
|
||||
|
||||
Severity: Medium
|
||||
|
||||
Status: Open
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/final_protocol_with_context`
|
||||
|
||||
Description:
|
||||
|
||||
A rhetorical or discussion-framing question is extracted and later rendered as
|
||||
an open question. This makes the protocol imply that the meeting left a real
|
||||
follow-up question unresolved.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
Only genuine unresolved questions should be extracted as open questions.
|
||||
Rhetorical questions or conversational framing should not become protocol open
|
||||
questions.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The final protocol includes at least one open question identified during
|
||||
evaluation as rhetorical rather than genuinely open.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
Unknown.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `prompts/questions.md`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Open. Documented from end-to-end evaluation output.
|
||||
|
||||
Notes:
|
||||
|
||||
No specific fix has been proposed.
|
||||
|
||||
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
|
||||
the issue remains. The final protocol still contains discussion-framing or
|
||||
underspecified questions as open protocol questions, including the question
|
||||
about exactly who is responsible for defining filter criteria.
|
||||
|
||||
## BUG-005
|
||||
|
||||
ID: BUG-005
|
||||
|
||||
Title: Renderer treats Björn as absent although Meeting Context marks him present
|
||||
|
||||
Pipeline stage: Working Protocol Renderer
|
||||
|
||||
Severity: Medium
|
||||
|
||||
Status: Investigating
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/final_protocol_with_context`
|
||||
|
||||
Description:
|
||||
|
||||
The generated protocol asks how missing participants such as Jovana and Björn
|
||||
should be included in the process, even though Meeting Context marks Björn as
|
||||
present. This creates a misleading participant-status implication.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
The protocol should not mark or imply Björn as absent when Meeting Context
|
||||
records him as present. If the source discussion concerns whether a person had
|
||||
reviewed materials or been included in a process, that should not be converted
|
||||
into meeting absence.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The final protocol renders an open question about including missing
|
||||
participants, including Björn.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
The extraction model receives Meeting Context but still classifies Björn with
|
||||
missing participants in chunk 08. Downstream artifacts preserve only minimal
|
||||
Meeting Context provenance, not structured participant attendance metadata, so
|
||||
Canonicalizer, Semantic Consolidator and Renderer cannot correct the conflict.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/real_live/project_process_meeting/meeting_context.yaml`
|
||||
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
|
||||
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `docs/bug-005-root-cause.md`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Investigating. Root-cause evidence has been documented in
|
||||
`docs/bug-005-root-cause.md`, but no fix has been implemented or verified.
|
||||
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
|
||||
the issue remains.
|
||||
|
||||
Notes:
|
||||
|
||||
This bug is distinct from responsibility attribution. It concerns participant
|
||||
presence/status handling.
|
||||
|
||||
Recommended resolution remains a Meeting Context constraint validator over
|
||||
structured participant metadata before rendering.
|
||||
|
||||
## BUG-006
|
||||
|
||||
ID: BUG-006
|
||||
|
||||
Title: Renderer drops required consolidated items and changes item categories
|
||||
|
||||
Pipeline stage: Working Protocol Renderer
|
||||
|
||||
Severity: High
|
||||
|
||||
Status: Open
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/e2e_current_20260803_151000`
|
||||
|
||||
Description:
|
||||
|
||||
The Working Protocol Renderer does not preserve every consolidated decision and
|
||||
open question even though the renderer prompt requires preserving all
|
||||
decisions, action items and open questions. It also renders some consolidated
|
||||
facts as decisions.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
Every consolidated decision and open question should be represented in the
|
||||
final protocol unless there is an explicit, validated reason to omit it.
|
||||
Background facts must not be promoted into decisions.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The consolidated input contains decisions such as `Festlegung eines
|
||||
Reporting-Zyklus für abgelehnte Projekte`, `Zusammengetragen und beantwortete
|
||||
Fragen führen zur Sitzung mit fünf Leuten...`, and `Die Diskussion wird
|
||||
beendet und die Änderungen werden verschickt...`; these are not preserved as
|
||||
decisions in the final protocol. The consolidated input also contains open
|
||||
questions such as `Wer soll die Rolle des Gatekeepers übernehmen...` and
|
||||
`Welche Prüfsteine sind relevant für die Fachabteilung?`; these are omitted
|
||||
from the final protocol. Conversely, consolidated facts such as `Im Zweifel
|
||||
gehen Projekte durch` and `Ein Projekt kann abgelehnt werden...` are rendered
|
||||
as decisions.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
Unknown.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Open. Documented from current end-to-end benchmark output.
|
||||
|
||||
Notes:
|
||||
|
||||
This is a renderer faithfulness problem independent from whether the upstream
|
||||
extracted items are themselves correct.
|
||||
|
||||
## BUG-007
|
||||
|
||||
ID: BUG-007
|
||||
|
||||
Title: Non-committed discussion items are extracted and rendered as todos
|
||||
|
||||
Pipeline stage: Extraction / Working Protocol Renderer
|
||||
|
||||
Severity: High
|
||||
|
||||
Status: Open
|
||||
|
||||
Date discovered: 2026-08-03
|
||||
|
||||
Version first observed: `meeting_context_v1/e2e_current_20260803_151000`
|
||||
|
||||
Description:
|
||||
|
||||
The pipeline extracts and renders several action items that are not clearly
|
||||
assigned, accepted or confirmed commitments in the meeting evidence.
|
||||
|
||||
Expected behaviour:
|
||||
|
||||
Only explicit assignments, accepted responsibilities or confirmed follow-up
|
||||
actions should become todos. Discussion, examples, proposals, broad process
|
||||
needs or unclear alternatives should remain facts/questions or be marked
|
||||
unclear where supported.
|
||||
|
||||
Actual behaviour:
|
||||
|
||||
The current consolidated output and final protocol contain todos such as
|
||||
`Feedback zu den Kriterien und Änderungen am Prozess abwarten`, `Einbinden der
|
||||
Fachbereiche (stellvertretend durch Guido) zur Definition von Prüfsteinen`,
|
||||
and `Mail an Jovana und Björn erneut senden oder Prozessanpassung vornehmen`.
|
||||
The evidence for these items is discussion-level or ambiguous rather than a
|
||||
clear todo commitment.
|
||||
|
||||
Likely root cause:
|
||||
|
||||
Unknown.
|
||||
|
||||
Related files:
|
||||
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
|
||||
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
|
||||
- `prompts/todos.md`
|
||||
- `prompts/working_protocol.md`
|
||||
|
||||
Regression test available (yes/no): no
|
||||
|
||||
Current status:
|
||||
|
||||
Open. Documented from current end-to-end benchmark output.
|
||||
|
||||
Notes:
|
||||
|
||||
This generalizes BUG-002 beyond the specific Jovana assignment case and should
|
||||
be evaluated against the responsibility attribution invariant.
|
||||
Reference in New Issue
Block a user