Repair hallucinated Semantic Consolidator source IDs

docs/architecture.md

@@ -379,8 +379,13 @@ Views documented in `output-views.md`.
 
 Semantic Consolidator V0 preserves every raw model response before parsing.
 Its normal path accepts parseable grouping JSON and leaves duplicate source-ID
-and missing-source-ID correction to the existing deterministic coverage
-repair.
+unknown-source-ID and missing-source-ID correction to the deterministic
+coverage repair. Unknown source IDs are removed without attempting numeric or
+semantic remapping. A group retains its valid IDs and remains present when at
+least one valid ID survives. A group containing only unknown IDs is removed
+after it becomes empty. Existing coverage repair then restores every genuinely
+missing canonical fact as a singleton. Strict validation runs against the
+repaired result and still rejects any unknown ID that survives this process.
 
 An invalid or truncated response is not retried by default. One controlled
 retry is allowed only when deterministic inspection finds at least three

docs/regression-bugs.md

@@ -1038,6 +1038,89 @@ invalid only because group 89 contained unknown `fact_0114`.
 
 The stage therefore failed cleanly after the single retry and preserved both
 attempts. No third LLM call occurred. The remaining unknown-ID failure is not a
-repetitive generation loop and is not silently repaired by the existing
-coverage repair. This verification does not claim that general Semantic
-Consolidator output quality is solved.
+repetitive generation loop; its deterministic repair is tracked separately as
+BUG-014. This verification does not claim that general Semantic Consolidator
+output quality is solved.
+
+## BUG-014
+
+ID: BUG-014
+
+Title: Semantic Consolidator hallucinates unknown source item IDs
+
+Pipeline stage: Semantic Consolidator deterministic coverage repair
+
+Severity: High
+
+Status: Verified
+
+Date discovered: 2026-08-09
+
+Version first observed: BUG-013 consolidator-only regression
+
+Description:
+
+The parseable BUG-013 retry response contained a semantic group at index 104
+with `source_item_ids` equal to `fact_0113` and unknown `fact_0114`. The
+canonicalized input contains 113 facts ending at `fact_0113`; no `fact_0114`
+exists.
+
+The group canonical text, "Ich habe ihn heute Morgen im Büro angetroffen.", is
+directly supported by valid `fact_0113`. The unknown ID adds no identifiable
+canonical source content and appears to be an additional hallucinated ID, not
+a safely correctable typo or a reference that can be mapped to another input
+fact.
+
+Invariant:
+
+Every `source_item_id` emitted by the Semantic Consolidator must refer to an
+existing canonical input fact. Strict validation must continue to reject any
+unknown ID that survives deterministic repair.
+
+Deterministic repair rule:
+
+- Remove every string source ID that is not present in the canonical fact-ID
+  set. Never map it to a numerically nearby or semantically guessed ID.
+- Preserve all valid source IDs in the group.
+- Preserve the group when at least one valid source ID remains.
+- Remove the group when unknown-ID removal leaves it empty.
+- Apply existing duplicate-ID removal to surviving valid IDs.
+- Restore genuinely missing canonical facts as singletons through the existing
+  coverage repair.
+- Preserve the original group text and merge reason; do not rewrite semantics.
+
+Related files:
+
+- `/tmp/meeting-lab-bug013-regression/raw_model_response_retry.txt`
+- `samples/benchmarks/progeo_north_linux_20260809_124954/canonicalizer/canonicalized_extractions.json`
+- `src/meeting_lab/consolidation/consolidate_facts.py`
+- `tests/test_consolidate_facts.py`
+
+Regression test available (yes/no): yes
+
+Current status:
+
+Verified with a consolidator-only regression using the preserved canonicalized
+input. Extraction, canonicalization, full benchmark orchestration and rendering
+were not run.
+
+The first attempt reproduced BUG-013: `eval_count=18654`,
+`done_reason=length`, invalid JSON and a detected 192-group consecutive
+repetition run. This triggered one controlled retry. The retry returned 105
+parseable groups with 154 source-ID occurrences, `eval_count=7362`,
+`done_reason=stop`, and no repetition loop.
+
+Strict source-coverage inspection before repair found one unknown ID
+(`fact_0114` in group 104), duplicate IDs and four missing canonical IDs.
+Deterministic repair made 64 changes:
+
+- 44 `remove_duplicate_source_id`
+- 1 `remove_unknown_source_id`
+- 15 `remove_empty_group`
+- 4 `restore_missing_source_id_as_singleton`
+
+Strict validation after repair passed with 94 groups containing exactly 113
+source-ID occurrences and 113 unique canonical source IDs. There were no
+missing IDs, unknown IDs or duplicate occurrences. BUG-014 is independent of
+BUG-013: BUG-013 detects invalid JSON caused by runaway repeated groups,
+whereas BUG-014 repairs unknown source IDs in parseable model grouping JSON.