Repair hallucinated Semantic Consolidator source IDs

2026-08-09 14:59:27 +02:00
parent 58effcafc5
commit 0b24351127
4 changed files with 240 additions and 11 deletions
@@ -1038,6 +1038,89 @@ invalid only because group 89 contained unknown `fact_0114`.

 The stage therefore failed cleanly after the single retry and preserved both
 attempts. No third LLM call occurred. The remaining unknown-ID failure is not a
-repetitive generation loop and is not silently repaired by the existing
-coverage repair. This verification does not claim that general Semantic
-Consolidator output quality is solved.
+repetitive generation loop; its deterministic repair is tracked separately as
+BUG-014. This verification does not claim that general Semantic Consolidator
+output quality is solved.
+
+## BUG-014
+
+ID: BUG-014
+
+Title: Semantic Consolidator hallucinates unknown source item IDs
+
+Pipeline stage: Semantic Consolidator deterministic coverage repair
+
+Severity: High
+
+Status: Verified
+
+Date discovered: 2026-08-09
+
+Version first observed: BUG-013 consolidator-only regression
+
+Description:
+
+The parseable BUG-013 retry response contained a semantic group at index 104
+with `source_item_ids` equal to `fact_0113` and unknown `fact_0114`. The
+canonicalized input contains 113 facts ending at `fact_0113`; no `fact_0114`
+exists.
+
+The group canonical text, "Ich habe ihn heute Morgen im Büro angetroffen.", is
+directly supported by valid `fact_0113`. The unknown ID adds no identifiable
+canonical source content and appears to be an additional hallucinated ID, not
+a safely correctable typo or a reference that can be mapped to another input
+fact.
+
+Invariant:
+
+Every `source_item_id` emitted by the Semantic Consolidator must refer to an
+existing canonical input fact. Strict validation must continue to reject any
+unknown ID that survives deterministic repair.
+
+Deterministic repair rule:
+
+- Remove every string source ID that is not present in the canonical fact-ID
+  set. Never map it to a numerically nearby or semantically guessed ID.
+- Preserve all valid source IDs in the group.
+- Preserve the group when at least one valid source ID remains.
+- Remove the group when unknown-ID removal leaves it empty.
+- Apply existing duplicate-ID removal to surviving valid IDs.
+- Restore genuinely missing canonical facts as singletons through the existing
+  coverage repair.
+- Preserve the original group text and merge reason; do not rewrite semantics.
+
+Related files:
+
+- `/tmp/meeting-lab-bug013-regression/raw_model_response_retry.txt`
+- `samples/benchmarks/progeo_north_linux_20260809_124954/canonicalizer/canonicalized_extractions.json`
+- `src/meeting_lab/consolidation/consolidate_facts.py`
+- `tests/test_consolidate_facts.py`
+
+Regression test available (yes/no): yes
+
+Current status:
+
+Verified with a consolidator-only regression using the preserved canonicalized
+input. Extraction, canonicalization, full benchmark orchestration and rendering
+were not run.
+
+The first attempt reproduced BUG-013: `eval_count=18654`,
+`done_reason=length`, invalid JSON and a detected 192-group consecutive
+repetition run. This triggered one controlled retry. The retry returned 105
+parseable groups with 154 source-ID occurrences, `eval_count=7362`,
+`done_reason=stop`, and no repetition loop.
+
+Strict source-coverage inspection before repair found one unknown ID
+(`fact_0114` in group 104), duplicate IDs and four missing canonical IDs.
+Deterministic repair made 64 changes:
+
+- 44 `remove_duplicate_source_id`
+- 1 `remove_unknown_source_id`
+- 15 `remove_empty_group`
+- 4 `restore_missing_source_id_as_singleton`
+
+Strict validation after repair passed with 94 groups containing exactly 113
+source-ID occurrences and 113 unique canonical source IDs. There were no
+missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of
+BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups,
+whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON.