martin/meeting-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9446c6e0bedbf7a8e9ba4986c3ebaf276784a292
meeting-lab/docs/regression-bugs.md
T
admin 9446c6e0be Document validation architecture and renderer faithfulness findings 
- document Entity Registry and Meeting Context V2 architecture
- preserve meeting_context.yaml as the authoritative meeting-specific input
- define immutable authoritative metadata across all pipeline stages
- restrict Constraint Repair to deterministic structured-data operations
- record BUG-003 root cause and deferred entity-verification resolution
- document BUG-005 attendance-consistency design
- add BUG-006 renderer faithfulness root-cause analysis
- distinguish Engineering Readiness from Practical Usability
- update the persistent regression bug tracker
2026-08-03 16:05:47 +02:00

479 lines
14 KiB
Markdown
Raw Blame History

# Regression Bug Tracker
Living tracker for real bugs discovered during end-to-end Meeting Lab
evaluation.
Purpose:
- prevent forgotten regressions
- document known root causes
- record fixes when they happen
- verify that bugs never silently return
## BUG-001
ID: BUG-001
Title: Renderer reverses the intended process direction
Pipeline stage: Working Protocol Renderer
Severity: High
Status: Open
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/final_protocol_with_context`
Description:
The generated working protocol phrases the process direction as if the
existing F&E process is broadly taken over for all project types. The intended
meaning from the evaluation is more constrained: the existing F&E process is a
framework that may be adapted or extended for broader project handling.
Expected behaviour:
The protocol should preserve the direction and uncertainty of the source
discussion: F&E process/framework adapted for other project types where
appropriate, without implying unconditional adoption for all project types.
Actual behaviour:
The protocol states that the existing F&E process is generally adopted for all
project types.
Likely root cause:
Unknown.
Related files:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Open. Documented from end-to-end evaluation output.
Notes:
Do not treat this entry as a prompt-change instruction. It records the observed
bug only.
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
the issue remains. The new protocol still says the existing F&E process is
basically taken over for all project types.
## BUG-002
ID: BUG-002
Title: Jovana todo is strengthened beyond the meeting content
Pipeline stage: Extraction / Working Protocol Renderer
Severity: High
Status: Open
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/final_protocol_with_context`
Description:
The generated output strengthens a proposal or discussion about Jovana
assembling criteria into an assigned todo. The evaluation identified this as
overstating the meeting content.
Expected behaviour:
The pipeline should preserve the weaker source meaning unless the transcript
explicitly assigns, accepts or confirms responsibility.
Actual behaviour:
The consolidated input and final protocol include an action item assigning
criterion compilation to Jovana.
Likely root cause:
Unknown.
Related files:
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `prompts/todos.md`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Open. Documented from end-to-end evaluation output.
Notes:
This bug is subject to the responsibility attribution invariant in
`AGENTS.md`.
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
the issue remains. The consolidated output still contains `Zusammenstellen der
Kriterien durch Jovana` with `responsible: "Jovana"`, and the final protocol
renders it as an action item.
## BUG-003
ID: BUG-003
Title: Unexpected synthetic person names appear in generated protocols
Pipeline stage: Extraction / Working Protocol Renderer
Severity: Medium
Status: Root Cause Identified
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/final_protocol_with_context`
Description:
The generated protocol contains unexpected or risky names/aliases such as
`Guido` and `Noah`. These names were flagged during evaluation as synthetic or
inconsistent in the generated protocol context.
Expected behaviour:
The protocol should use only names supported by the transcript, consolidated
input or Meeting Context, and should preserve uncertainty when a name or alias
is unclear.
Actual behaviour:
The final protocol includes names or name variants that were flagged as
unexpected in evaluation.
Likely root cause:
The unexpected names already exist in the Whisper transcript before extraction.
They are propagated through the pipeline and are not introduced by prompts,
gold tests or the renderer.
Related files:
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/real_live/project_process_meeting/meeting_context.yaml`
Regression test available (yes/no): no
Current status:
Root Cause Identified. Planned resolution is deferred until Meeting Context V2
/ Entity Registry implementation.
Notes:
Resolution strategy:
Introduce an interactive entity verification step after Whisper transcription:
```text
Whisper
↓
Entity Detection
↓
User Verification
↓
Meeting Context Builder
↓
Extraction
```
Expected effect:
Unknown or suspicious person names are detected before extraction begins and
can be classified by the user as participant, mentioned person, external
person, transcription error or ignore.
Implementation status:
Architecture accepted. Implementation deferred.
Regression test:
Not yet possible before Meeting Context V2 exists.
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
the issue remains. `Guido` and `Noah` both appear in the generated working
protocol.
## BUG-004
ID: BUG-004
Title: Rhetorical question extracted as an open question
Pipeline stage: Extraction
Severity: Medium
Status: Open
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/final_protocol_with_context`
Description:
A rhetorical or discussion-framing question is extracted and later rendered as
an open question. This makes the protocol imply that the meeting left a real
follow-up question unresolved.
Expected behaviour:
Only genuine unresolved questions should be extracted as open questions.
Rhetorical questions or conversational framing should not become protocol open
questions.
Actual behaviour:
The final protocol includes at least one open question identified during
evaluation as rhetorical rather than genuinely open.
Likely root cause:
Unknown.
Related files:
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `prompts/questions.md`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Open. Documented from end-to-end evaluation output.
Notes:
No specific fix has been proposed.
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
the issue remains. The final protocol still contains discussion-framing or
underspecified questions as open protocol questions, including the question
about exactly who is responsible for defining filter criteria.
## BUG-005
ID: BUG-005
Title: Renderer treats Björn as absent although Meeting Context marks him present
Pipeline stage: Working Protocol Renderer
Severity: Medium
Status: Investigating
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/final_protocol_with_context`
Description:
The generated protocol asks how missing participants such as Jovana and Björn
should be included in the process, even though Meeting Context marks Björn as
present. This creates a misleading participant-status implication.
Expected behaviour:
The protocol should not mark or imply Björn as absent when Meeting Context
records him as present. If the source discussion concerns whether a person had
reviewed materials or been included in a process, that should not be converted
into meeting absence.
Actual behaviour:
The final protocol renders an open question about including missing
participants, including Björn.
Likely root cause:
The extraction model receives Meeting Context but still classifies Björn with
missing participants in chunk 08. Downstream artifacts preserve only minimal
Meeting Context provenance, not structured participant attendance metadata, so
Canonicalizer, Semantic Consolidator and Renderer cannot correct the conflict.
Related files:
- `samples/real_live/project_process_meeting/meeting_context.yaml`
- `samples/benchmarks/meeting_context_v1/semantic_consolidator_repair_v1/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/working_protocol.md`
- `samples/benchmarks/meeting_context_v1/final_protocol_with_context/comparison.md`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `docs/bug-005-root-cause.md`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Investigating. Root-cause evidence has been documented in
`docs/bug-005-root-cause.md`, but no fix has been implemented or verified.
Current benchmark `meeting_context_v1/e2e_current_20260803_151000` confirms
the issue remains.
Notes:
This bug is distinct from responsibility attribution. It concerns participant
presence/status handling.
Recommended resolution remains a Meeting Context constraint validator over
structured participant metadata before rendering.
## BUG-006
ID: BUG-006
Title: Renderer drops required consolidated items and changes item categories
Pipeline stage: Working Protocol Renderer
Severity: High
Status: Open
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/e2e_current_20260803_151000`
Description:
The Working Protocol Renderer does not preserve every consolidated decision and
open question even though the renderer prompt requires preserving all
decisions, action items and open questions. It also renders some consolidated
facts as decisions.
Expected behaviour:
Every consolidated decision and open question should be represented in the
final protocol unless there is an explicit, validated reason to omit it.
Background facts must not be promoted into decisions.
Actual behaviour:
The consolidated input contains decisions such as `Festlegung eines
Reporting-Zyklus für abgelehnte Projekte`, `Zusammengetragen und beantwortete
Fragen führen zur Sitzung mit fünf Leuten...`, and `Die Diskussion wird
beendet und die Änderungen werden verschickt...`; these are not preserved as
decisions in the final protocol. The consolidated input also contains open
questions such as `Wer soll die Rolle des Gatekeepers übernehmen...` and
`Welche Prüfsteine sind relevant für die Fachabteilung?`; these are omitted
from the final protocol. Conversely, consolidated facts such as `Im Zweifel
gehen Projekte durch` and `Ein Projekt kann abgelehnt werden...` are rendered
as decisions.
Likely root cause:
Unknown.
Related files:
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Open. Documented from current end-to-end benchmark output.
Notes:
This is a renderer faithfulness problem independent from whether the upstream
extracted items are themselves correct.
## BUG-007
ID: BUG-007
Title: Non-committed discussion items are extracted and rendered as todos
Pipeline stage: Extraction / Working Protocol Renderer
Severity: High
Status: Open
Date discovered: 2026-08-03
Version first observed: `meeting_context_v1/e2e_current_20260803_151000`
Description:
The pipeline extracts and renders several action items that are not clearly
assigned, accepted or confirmed commitments in the meeting evidence.
Expected behaviour:
Only explicit assignments, accepted responsibilities or confirmed follow-up
actions should become todos. Discussion, examples, proposals, broad process
needs or unclear alternatives should remain facts/questions or be marked
unclear where supported.
Actual behaviour:
The current consolidated output and final protocol contain todos such as
`Feedback zu den Kriterien und Änderungen am Prozess abwarten`, `Einbinden der
Fachbereiche (stellvertretend durch Guido) zur Definition von Prüfsteinen`,
and `Mail an Jovana und Björn erneut senden oder Prozessanpassung vornehmen`.
The evidence for these items is discussion-level or ambiguous rather than a
clear todo commitment.
Likely root cause:
Unknown.
Related files:
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/semantic_consolidator/consolidated_extractions.json`
- `samples/benchmarks/meeting_context_v1/e2e_current_20260803_151000/working_protocol/working_protocol.md`
- `prompts/todos.md`
- `prompts/working_protocol.md`
Regression test available (yes/no): no
Current status:
Open. Documented from current end-to-end benchmark output.
Notes:
This generalizes BUG-002 beyond the specific Jovana assignment case and should
be evaluated against the responsibility attribution invariant.
Reference in New Issue View Git Blame Copy Permalink