martin/meeting-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Handle repetitive Semantic Consolidator output loops

Browse Source
This commit is contained in:
admin
2026-08-09 14:33:46 +02:00
parent 3a5850430b
commit 58effcafc5
5 changed files with 462 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+16
View File
@@ -375,6 +375,22 @@ extraction results into `meeting_protocol.md` for technical validation. The
planned architecture separates Canonical Meeting Knowledge from the final Output
Views documented in `output-views.md`.
## Semantic Consolidator failure handling
Semantic Consolidator V0 preserves every raw model response before parsing.
Its normal path accepts parseable grouping JSON and leaves duplicate source-ID
and missing-source-ID correction to the existing deterministic coverage
repair.
An invalid or truncated response is not retried by default. One controlled
retry is allowed only when deterministic inspection finds at least three
consecutive complete groups with an identical structural signature:
`canonical_text`, ordered `source_item_ids` and `merge_reason`. The retry keeps
the same model, temperature, context window and generation limit and adds only
an instruction not to emit an identical group more than once. Both attempts
and the detected repetition metadata are preserved. If the retry also fails,
the stage fails normally; it does not make another LLM call.
---
# Next Milestone
docs/regression-bugs.md
+110
View File
@@ -931,3 +931,113 @@ BUG-011 and does not invalidate this chunking hypothesis.
This verifies the hypothesis for this transcript/configuration only. It does
not claim the general chunker is fixed.
## BUG-013
ID: BUG-013
Title: Semantic Consolidator repetitive output loop exhausts generation budget
Pipeline stage: Semantic Consolidator
Severity: High
Status: Verified
Date discovered: 2026-08-09
Version first observed: `progeo_north_linux_20260809_124954`
Description:
The Semantic Consolidator entered a pathological generation loop that emitted
the same complete semantic group 192 consecutive times. The repeated group had
the stable structural signature consisting of the same `canonical_text`, the
same ordered `source_item_ids` (`fact_0004`, `fact_0019`) and the same
`merge_reason`. Generation then truncated at an incomplete
`"canonical_text":` field.
Expected behaviour:
The consolidator should detect a structural repeated-group loop
deterministically. Invalid JSON should receive at most one controlled retry
only when that loop signature is present. The retry must use the same model,
temperature and context configuration and add only an instruction preventing
duplicate group emission. Unrelated malformed JSON must not be retried.
Actual behaviour:
The first model response was invalid JSON at line 974, column 24, character
71184. Semantic Consolidator runtime was 277.910 seconds. The raw response had
71,184 characters (71,568 UTF-8 bytes), ended at `"canonical_text":`, and
could not reach deterministic source-coverage repair because it was not
parseable JSON.
Model metadata:
- Resolved `num_predict`: 19,532
- `num_ctx`: 32,768
- `prompt_eval_count`: 14,114
- `eval_count`: 18,654
- `done_reason`: `length`
- `eval_count` did not equal resolved `num_predict`; it exactly consumed the
remaining evaluated context (`32768 - 14114 = 18654`)
- Complete groups before truncation: 194
- Consecutive identical groups: 192, starting at group index 2
Root cause:
The model produced a degenerate repeated semantic-group sequence until the
remaining context budget was exhausted. BUG-010 adaptive response sizing was
working as designed; increasing `num_predict` would not address this failure
mode.
Implemented handling:
- Extract complete group objects from a response even when its outer JSON is
truncated.
- Identify groups by normalized `canonical_text`, ordered `source_item_ids`
and normalized `merge_reason`.
- Classify three or more consecutive identical complete groups as a loop; two
identical occurrences remain an ordinary duplicate handled by deterministic
coverage repair when the JSON is valid.
- Preserve first-attempt raw text, raw Ollama JSON and repetition metadata.
- Retry at most once only when JSON parsing fails and a loop is detected.
- Preserve separate retry artifacts and fail normally if retry parsing fails.
Related files:
- `samples/benchmarks/progeo_north_linux_20260809_124954/`
- `samples/benchmarks/progeo_north_linux_20260809_124954/semantic_consolidator/raw_model_response.txt`
- `samples/benchmarks/progeo_north_linux_20260809_124954/semantic_consolidator/raw_ollama_response.json`
- `src/meeting_lab/consolidation/consolidate_facts.py`
- `scripts/run_meeting.py`
- `tests/test_consolidate_facts.py`
Regression test available (yes/no): yes
Current status:
Verified for the repetitive-loop failure mode. A consolidator-only regression
used the preserved canonicalized input and did not run extraction,
canonicalization or rendering.
The first attempt reproduced the original failure signature: 71,184 response
characters, `eval_count=18654`, `done_reason=length`, invalid JSON at character
71184, and a detected 192-group consecutive repetition run. This triggered the
only permitted retry.
The retry used the same model and generation configuration, returned parseable
JSON after 7,362 evaluated tokens with `done_reason=stop`, and contained no
repetition loop (longest identical consecutive run: 1). Its source-coverage
validation before repair was invalid: 105 groups, 154 observed source-ID
occurrences, duplicate IDs, four missing IDs and one unknown ID. Deterministic
repair made 63 changes: 44 duplicate-ID removals, 15 empty-group removals and
four missing-ID singleton restorations. Validation after repair remained
invalid only because group 89 contained unknown `fact_0114`.
The stage therefore failed cleanly after the single retry and preserved both
attempts. No third LLM call occurred. The remaining unknown-ID failure is not a
repetitive generation loop and is not silently repaired by the existing
coverage repair. This verification does not claim that general Semantic
Consolidator output quality is solved.
scripts/run_meeting.py
+6 -7
View File
@@ -43,7 +43,7 @@ from src.meeting_lab.consolidation.canonicalize import ( # noqa: E402
from src.meeting_lab.consolidation.consolidate_facts import ( # noqa: E402
build_consolidated_output,
build_consolidation_prompt,
call_ollama as call_consolidator_ollama,
call_ollama_with_repetition_retry,
fact_items,
parse_model_json,
repair_model_group_coverage,
@@ -665,7 +665,7 @@ def run_semantic_consolidator_stage(
prompt_token_estimate=prompt_token_estimate,
num_ctx=num_ctx,
)
raw_text, response_data, runtime = call_consolidator_ollama(
raw_text, response_data, runtime, attempts = call_ollama_with_repetition_retry(
endpoint=endpoint,
model=model,
prompt=prompt,
@@ -674,15 +674,11 @@ def run_semantic_consolidator_stage(
num_predict=resolved_num_predict,
think=False,
progress_interval=progress_interval,
output_dir=output_dir,
)
raw_response_path = output_dir / "raw_model_response.txt"
raw_json_path = output_dir / "raw_ollama_response.json"
output_path = output_dir / "consolidated_extractions.json"
report_path = output_dir / "report.md"
repair_metadata_path = output_dir / "repair_metadata.json"
raw_response_path.write_text(raw_text + "\n", encoding="utf-8")
write_json(raw_json_path, response_data)
model_output = parse_model_json(raw_text)
expected_fact_ids = {item["item_id"] for item in facts}
validator_before = source_coverage_report_from_groups(
@@ -734,9 +730,12 @@ def run_semantic_consolidator_stage(
groups=groups,
output_path=output_path,
repair_changes=repair_changes,
llm_call_count=len(attempts),
)
return {
"runtime_seconds": runtime,
"llm_call_count": len(attempts),
"attempts": attempts,
"fact_count": len(facts),
"num_predict": resolved_num_predict,
"validator_before": validator_before,
src/meeting_lab/consolidation/consolidate_facts.py
+179 -4
View File
@@ -6,6 +6,7 @@ from __future__ import annotations
import argparse
import copy
import json
import re
import sys
import time
from collections import Counter
@@ -34,6 +35,15 @@ OUTPUT_TOKEN_ESTIMATE_CHARS = 4
OUTPUT_GROUP_OVERHEAD_CHARS = 320
OUTPUT_SAFETY_MARGIN = 1.35
PROMPT_NAME = "consolidate_facts.md"
REPETITION_LOOP_MIN_CONSECUTIVE_GROUPS = 3
REPETITION_RETRY_INSTRUCTION = """
RETRY SAFETY INSTRUCTION:
Emit each semantic group exactly once. Before returning the JSON, check the
groups array for identical groups using canonical_text, source_item_ids and
merge_reason. If an identical group is already present, do not emit it again.
Finish the complete JSON object without repeating any group.
""".rstrip()
class ConsolidationValidationError(ValueError):
@@ -343,6 +353,169 @@ def parse_model_json(text: str) -> dict[str, Any]:
return parsed
def group_repetition_signature(group: dict[str, Any]) -> tuple[str, tuple[str, ...], str] | None:
canonical_text = group.get("canonical_text")
source_item_ids = group.get("source_item_ids")
merge_reason = group.get("merge_reason")
if (
not isinstance(canonical_text, str)
or not isinstance(source_item_ids, list)
or not all(isinstance(item_id, str) for item_id in source_item_ids)
or not isinstance(merge_reason, str)
):
return None
normalize = lambda value: re.sub(r"\s+", " ", value).strip()
return (
normalize(canonical_text),
tuple(source_item_ids),
normalize(merge_reason),
)
def completed_group_objects(text: str) -> list[dict[str, Any]]:
"""Extract complete JSON group objects even when the outer response is truncated."""
groups_key = re.search(r'"groups"\s*:\s*\[', text)
if groups_key is None:
return []
decoder = json.JSONDecoder()
groups: list[dict[str, Any]] = []
position = groups_key.end()
while position < len(text):
while position < len(text) and text[position] in " \t\r\n,":
position += 1
if position >= len(text) or text[position] == "]":
break
if text[position] != "{":
break
try:
value, position = decoder.raw_decode(text, position)
except json.JSONDecodeError:
break
if not isinstance(value, dict):
break
groups.append(value)
return groups
def detect_repetitive_group_loop(
text: str,
minimum_consecutive: int = REPETITION_LOOP_MIN_CONSECUTIVE_GROUPS,
) -> dict[str, Any]:
groups = completed_group_objects(text)
longest_run = 0
longest_signature: tuple[str, tuple[str, ...], str] | None = None
run_start = 0
previous: tuple[str, tuple[str, ...], str] | None = None
current_run = 0
for index, group in enumerate(groups):
signature = group_repetition_signature(group)
if signature is not None and signature == previous:
current_run += 1
else:
current_run = 1 if signature is not None else 0
run_start = index
if current_run > longest_run:
longest_run = current_run
longest_signature = signature
longest_run_start = run_start
previous = signature
detected = longest_signature is not None and longest_run >= minimum_consecutive
signature_data = None
if longest_signature is not None:
signature_data = {
"canonical_text": longest_signature[0],
"source_item_ids": list(longest_signature[1]),
"merge_reason": longest_signature[2],
}
return {
"detected": detected,
"threshold": minimum_consecutive,
"completed_group_count": len(groups),
"longest_consecutive_run": longest_run,
"run_start_group_index": longest_run_start if longest_run else None,
"signature": signature_data,
}
def call_ollama_with_repetition_retry(
*,
endpoint: str,
model: str,
prompt: str,
timeout: int,
num_ctx: int,
num_predict: int,
think: bool,
progress_interval: int,
output_dir: Path,
call_fn: Any = None,
) -> tuple[str, dict[str, Any], float, list[dict[str, Any]]]:
"""Make one call, retrying once only for invalid JSON with a detected group loop."""
if call_fn is None:
call_fn = call_ollama
output_dir.mkdir(parents=True, exist_ok=True)
attempts: list[dict[str, Any]] = []
total_runtime = 0.0
for attempt_number in (1, 2):
attempt_prompt = (
prompt
if attempt_number == 1
else f"{prompt}\n{REPETITION_RETRY_INSTRUCTION}\n"
)
raw_text, response_data, runtime = call_fn(
endpoint=endpoint,
model=model,
prompt=attempt_prompt,
timeout=timeout,
num_ctx=num_ctx,
num_predict=num_predict,
think=think,
progress_interval=progress_interval,
)
total_runtime += runtime
suffix = "" if attempt_number == 1 else "_retry"
(output_dir / f"raw_model_response{suffix}.txt").write_text(
raw_text + "\n", encoding="utf-8"
)
write_json(output_dir / f"raw_ollama_response{suffix}.json", response_data)
repetition = detect_repetitive_group_loop(raw_text)
attempt_metadata = {
"attempt": attempt_number,
"retry_instruction_added": attempt_number == 2,
"runtime_seconds": runtime,
"raw_response_chars": len(raw_text),
"raw_response_bytes": len(raw_text.encode("utf-8")),
"resolved_num_predict": num_predict,
"eval_count": response_data.get("eval_count"),
"done_reason": response_data.get("done_reason"),
"repetition": repetition,
}
attempts.append(attempt_metadata)
try:
parse_model_json(raw_text)
except ConsolidationValidationError as exc:
attempt_metadata["parse_error"] = str(exc)
write_json(
output_dir / "repetition_retry_metadata.json",
{"bug": "BUG-013", "attempts": attempts},
)
if attempt_number == 1 and repetition["detected"]:
continue
raise
write_json(
output_dir / "repetition_retry_metadata.json",
{"bug": "BUG-013", "attempts": attempts},
)
return raw_text, response_data, total_runtime, attempts
raise AssertionError("repetition retry loop exhausted unexpectedly")
def validate_model_groups(
model_output: dict[str, Any],
expected_fact_ids: set[str],
@@ -626,6 +799,7 @@ def write_report(
groups: list[dict[str, Any]],
output_path: Path,
repair_changes: list[dict[str, Any]] | None = None,
llm_call_count: int = 1,
) -> None:
merged = [group for group in groups if len(group["source_item_ids"]) > 1]
singletons = [group for group in groups if len(group["source_item_ids"]) == 1]
@@ -634,7 +808,7 @@ def write_report(
"",
"- Scope: facts only",
f"- Model: `{model}`",
"- LLM call count: 1",
f"- LLM call count: {llm_call_count}",
f"- Runtime: {runtime:.3f} seconds",
f"- Fact item count: {fact_count}",
f"- Prompt characters: {prompt_chars}",
@@ -693,10 +867,10 @@ def main() -> int:
print(f"Estimated prompt size chars: {prompt_chars}")
print(f"Estimated prompt tokens: {prompt_token_estimate}")
print(f"Resolved num_predict: {num_predict}")
print("Expected LLM call count: 1")
print("Expected LLM call count: 1; at most 2 only after detected BUG-013 loop")
print("Expected runtime: 5-10 minutes on current local benchmark basis")
raw_text, _response_data, runtime = call_ollama(
raw_text, _response_data, runtime, attempts = call_ollama_with_repetition_retry(
endpoint=args.endpoint,
model=args.model,
prompt=prompt,
@@ -705,8 +879,8 @@ def main() -> int:
num_predict=num_predict,
think=args.think,
progress_interval=args.progress_interval,
output_dir=args.output_dir,
)
raw_response_path.write_text(raw_text + "\n", encoding="utf-8")
model_output = parse_model_json(raw_text)
model_output, repair_changes = repair_model_group_coverage(model_output, facts)
if repair_changes:
@@ -736,6 +910,7 @@ def main() -> int:
groups=groups,
output_path=output_path,
repair_changes=repair_changes,
llm_call_count=len(attempts),
)
except requests.ConnectionError as exc:
print(f"Error: Ollama is not reachable at {args.endpoint}: {exc}", file=sys.stderr)
tests/test_consolidate_facts.py
+151
View File
@@ -1,6 +1,8 @@
import unittest
import json
import tempfile
from pathlib import Path
from unittest.mock import Mock
from src.meeting_lab.consolidation.consolidate_facts import (
ConsolidationValidationError,
@@ -8,6 +10,8 @@ from src.meeting_lab.consolidation.consolidate_facts import (
DEFAULT_NUM_PREDICT,
build_consolidated_output,
build_ollama_payload,
call_ollama_with_repetition_retry,
detect_repetitive_group_loop,
estimate_response_tokens,
fact_items,
parse_model_json,
@@ -74,6 +78,14 @@ def canonicalized_fixture():
class ConsolidateFactsTests(unittest.TestCase):
@staticmethod
def group(text="A", ids=None, reason="Singleton."):
return {
"canonical_text": text,
"source_item_ids": ids or ["fact_0001"],
"merge_reason": reason,
}
def test_payload_construction_disables_streaming_and_thinking_by_default(self):
payload = build_ollama_payload(
model="qwen3.5:9B",
@@ -174,6 +186,145 @@ class ConsolidateFactsTests(unittest.TestCase):
self.assertEqual(len(groups), 2)
def test_valid_normal_consolidation_has_no_repetition_loop(self):
text = json.dumps(
{"groups": [self.group("A"), self.group("B", ["fact_0002"])]}
)
detection = detect_repetitive_group_loop(text)
self.assertFalse(detection["detected"])
self.assertEqual(detection["completed_group_count"], 2)
def test_ordinary_duplicate_group_is_not_a_repetition_loop(self):
duplicate = self.group()
text = json.dumps({"groups": [duplicate, duplicate]})
detection = detect_repetitive_group_loop(text)
self.assertFalse(detection["detected"])
self.assertEqual(detection["longest_consecutive_run"], 2)
def test_repetitive_identical_group_sequence_is_detected_structurally(self):
repeated = self.group("Same fact", ["fact_0001", "fact_0002"], "Equivalent.")
text = json.dumps({"groups": [repeated, repeated, repeated]})
detection = detect_repetitive_group_loop(text)
self.assertTrue(detection["detected"])
self.assertEqual(detection["longest_consecutive_run"], 3)
self.assertEqual(
detection["signature"],
{
"canonical_text": "Same fact",
"source_item_ids": ["fact_0001", "fact_0002"],
"merge_reason": "Equivalent.",
},
)
def test_truncated_response_after_repetitive_loop_is_detected(self):
repeated = json.dumps(self.group(), separators=(",", ":"))
text = '{"groups":[' + ",".join([repeated] * 4) + ',{"canonical_text":'
detection = detect_repetitive_group_loop(text)
self.assertTrue(detection["detected"])
self.assertEqual(detection["completed_group_count"], 4)
def test_detected_invalid_loop_gets_one_controlled_retry(self):
repeated = json.dumps(self.group(), separators=(",", ":"))
truncated = '{"groups":[' + ",".join([repeated] * 3) + ',{"canonical_text":'
valid = json.dumps({"groups": [self.group()]})
call_fn = Mock(
side_effect=[
(truncated, {"eval_count": 100, "done_reason": "length"}, 1.0),
(valid, {"eval_count": 20, "done_reason": "stop"}, 0.5),
]
)
with tempfile.TemporaryDirectory() as directory:
raw_text, _data, runtime, attempts = call_ollama_with_repetition_retry(
endpoint="http://example.invalid",
model="test-model",
prompt="base prompt",
timeout=1,
num_ctx=1000,
num_predict=100,
think=False,
progress_interval=1,
output_dir=Path(directory),
call_fn=call_fn,
)
self.assertTrue((Path(directory) / "raw_model_response.txt").exists())
self.assertTrue((Path(directory) / "raw_model_response_retry.txt").exists())
metadata = json.loads(
(Path(directory) / "repetition_retry_metadata.json").read_text()
)
self.assertEqual(raw_text, valid)
self.assertEqual(runtime, 1.5)
self.assertEqual(len(attempts), 2)
self.assertEqual(call_fn.call_count, 2)
self.assertEqual(metadata["attempts"][0]["repetition"]["detected"], True)
first_prompt = call_fn.call_args_list[0].kwargs["prompt"]
retry_prompt = call_fn.call_args_list[1].kwargs["prompt"]
self.assertEqual(first_prompt, "base prompt")
self.assertIn("Emit each semantic group exactly once", retry_prompt)
def test_unrelated_malformed_json_is_not_retried(self):
call_fn = Mock(return_value=("{invalid", {"done_reason": "length"}, 0.1))
with tempfile.TemporaryDirectory() as directory:
with self.assertRaises(ConsolidationValidationError):
call_ollama_with_repetition_retry(
endpoint="http://example.invalid",
model="test-model",
prompt="prompt",
timeout=1,
num_ctx=1000,
num_predict=100,
think=False,
progress_interval=1,
output_dir=Path(directory),
call_fn=call_fn,
)
self.assertEqual(call_fn.call_count, 1)
def test_retry_failure_preserves_both_attempts_and_raises(self):
repeated = json.dumps(self.group(), separators=(",", ":"))
truncated = '{"groups":[' + ",".join([repeated] * 3) + ',{"canonical_text":'
call_fn = Mock(
side_effect=[
(truncated, {"done_reason": "length"}, 1.0),
(truncated, {"done_reason": "length"}, 1.0),
]
)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
with self.assertRaises(ConsolidationValidationError):
call_ollama_with_repetition_retry(
endpoint="http://example.invalid",
model="test-model",
prompt="prompt",
timeout=1,
num_ctx=1000,
num_predict=100,
think=False,
progress_interval=1,
output_dir=root,
call_fn=call_fn,
)
self.assertTrue((root / "raw_model_response.txt").exists())
self.assertTrue((root / "raw_ollama_response.json").exists())
self.assertTrue((root / "raw_model_response_retry.txt").exists())
self.assertTrue((root / "raw_ollama_response_retry.json").exists())
metadata = json.loads((root / "repetition_retry_metadata.json").read_text())
self.assertEqual(call_fn.call_count, 2)
self.assertEqual(len(metadata["attempts"]), 2)
self.assertIn("parse_error", metadata["attempts"][1])
def test_no_missing_source_ids(self):
with self.assertRaisesRegex(ConsolidationValidationError, "Missing"):
validate_model_groups(