Add negative act form experiment

This commit is contained in:
2026-08-20 12:19:55 +02:00
parent 97a22f3ebb
commit 0d4b426021
5 changed files with 593 additions and 0 deletions
+72
View File
@@ -1890,6 +1890,78 @@ cross-pattern reconciliation is justified.
Artifacts are preserved under Artifacts are preserved under
`artifacts/experiments/explicit_rejection_gold_v0/20260820_qwen35_9b_single_run/`. `artifacts/experiments/explicit_rejection_gold_v0/20260820_qwen35_9b_single_run/`.
## EXP-0035 — Negative Act Form V0
Status: Experimental; successful for form classification with normalization
limitations
Date: 2026-08-20
EXP-0034 failed because the binary `explicit_action_rejection | none` question
collapsed materially different negative acts. It missed self-contained
non-pursuit and promoted personal preference, recommendation and temporary
non-action to rejection. This isolated follow-up tested only whether those
evidence-near forms can be distinguished before any normative derivation. It
does not derive rejection, decision, outcome, topic closure, responsibility or
protocol status, and EXP-0034 remained unchanged.
The strict output schema contains exactly `observation_id`,
`negative_act_form` and `normalized_action_text`. The closed form vocabulary is
`explicit_non_pursuit`, `personal_preference`, `recommendation`,
`temporary_non_action` and `none`. Non-`none` forms require non-empty normalized
action text; `none` requires null. Rejection, status, decision, outcome,
responsibility and other normative fields are forbidden recursively. Local
context may resolve a candidate observation's pronoun, but the schema contains
no target relation and the experiment exposes no derivation function.
Gold results:
- NA-01 explicit non-pursuit: PARTIAL. The form was correct; `working with Dr.
Schlummer` omitted the continuation aspect from normalization.
- NA-02 paraphrased explicit non-pursuit: PASS.
- NA-03 personal preference: PARTIAL. The form was correct, but normalization
repeated `Ich würde das nicht machen` instead of resolving the real-plant
trial target.
- NA-04 negative recommendation: PARTIAL. The form was correct; the normalized
English action used the loose rendering `real asset` for `reale Anlage`.
- NA-05 temporary non-action: PASS.
- NA-06 concern only: PASS with `none` and null action text.
- NA-07 uncertainty: PASS with `none` and null action text.
- NA-08 factual negation: PASS with `none` and null action text.
Expected-versus-actual form confusion was entirely diagonal:
| Expected form | Actual form | Count |
| --- | --- | ---: |
| `explicit_non_pursuit` | `explicit_non_pursuit` | 2 |
| `personal_preference` | `personal_preference` | 1 |
| `recommendation` | `recommendation` | 1 |
| `temporary_non_action` | `temporary_non_action` | 1 |
| `none` | `none` | 3 |
Configuration: exactly eight successful sequential `qwen3.5:9B` calls, one
per case, temperature 0, `think=false`, `num_ctx=16384`,
`num_predict=1024`, no retries, no voting and no prompt changes. There were zero
technical failures. Aggregate runner time was 8.688 seconds; summed per-call
time was 8.686 seconds, with 4,183 prompt-evaluation tokens and 310 evaluation
tokens.
The result was five PASS, three PARTIAL and zero FAIL. All eight
`negative_act_form` classifications matched Gold. There was no unsupported
semantic strengthening and no rejection, status, decision, outcome,
responsibility or topic-closure leakage. Normalized action meaning was fully
acceptable in five cases and imperfect in three.
Conclusion: the finer evidence-near form vocabulary successfully distinguished
the four semantic boundaries that defeated the binary rejection experiment in
this small Gold set. The result supports separating negative-act-form
recognition from later normative derivation, but local target normalization is
not yet uniformly reliable. It does not justify modifying EXP-0034, deriving
rejection, production integration or beginning cross-pattern reconciliation.
Artifacts are preserved under
`artifacts/experiments/negative_act_form_v0/20260820_qwen35_9b_single_run/`.
## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype ## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype
Date: 2026-08-11 Date: 2026-08-11
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
"""Repository entry point for the Negative Act Form experiment."""
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from src.meeting_lab.controlled_semantic_derivation.experiment_negative_act import main # noqa: E402
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,277 @@
#!/usr/bin/env python3
"""Isolated evidence-near Negative Act Form classification experiment."""
from __future__ import annotations
import argparse
import json
import time
from pathlib import Path
from typing import Any
from .experiment_h import (
DEFAULT_ENDPOINT,
DEFAULT_MODEL,
DerivationValidationError,
OBSERVATION_KEYS,
build_ollama_payload,
call_ollama,
)
GOLD_SCHEMA_VERSION = "experimental-negative-act-form-gold-v0"
RECOGNITION_KEYS = {"observation_id", "negative_act_form", "normalized_action_text"}
NEGATIVE_ACT_FORMS = {
"explicit_non_pursuit", "personal_preference", "recommendation",
"temporary_non_action", "none",
}
FORBIDDEN_LLM_KEYS = {
"rejection_form", "explicitly_rejected", "status", "decision", "outcome",
"topic_status", "responsible_person", "responsibility", "owner",
"requested_actor", "action_item", "protocol", "protocol_category",
"confidence", "relation", "relations", "graph", "unresolved_issue",
}
PROMPT_TEMPLATE = """Classify only the negative semantic form expressed by the candidate observation, using earlier supplied V3-style observations only as local context for pronouns or shortened references.
The candidate observation is {candidate_observation_id}.
Choose exactly one negative_act_form:
- explicit_non_pursuit: explicitly states that an action, option, collaboration, or course will not be continued or pursued. This is stronger than preference, advice, or temporary delay.
- personal_preference: the speaker states what they personally would or would not do, without establishing collective non-pursuit.
- recommendation: the speaker advises for or against an action without establishing abandonment.
- temporary_non_action: the action is postponed, deferred, or explicitly not done for now without abandonment.
- none: none of those four forms is present, including mere concern, uncertainty, negative sentiment, or factual negation.
Do not collapse non-pursuit into temporary non-action. Do not convert a personal conditional preference into collective non-pursuit. Do not convert advice into non-pursuit. Speaker identity does not change personal preference into collective non-pursuit.
When the form is not none, return concise normalized action meaning. Resolve a pronoun only from the supplied local context. If its target is genuinely ambiguous, return none rather than guessing. When the form is none, normalized_action_text must be null. Keep normalized action text in the observation language.
Do not derive or output rejection, status, decision, outcome, topic closure, responsibility, ownership, Action Item, protocol category, confidence, relations, graphs, or unresolved issues.
Return exactly this JSON shape and no additional fields:
{{
"observation_id": "{candidate_observation_id}",
"negative_act_form": "explicit_non_pursuit | personal_preference | recommendation | temporary_non_action | none",
"normalized_action_text": "concise action meaning" | null
}}
V3-style observations:
{observations_json}
"""
def _exact_keys(value: dict[str, Any], required: set[str], location: str) -> None:
missing = required - value.keys()
unknown = value.keys() - required
if missing:
raise DerivationValidationError(f"{location} missing required keys: {sorted(missing)}")
if unknown:
raise DerivationValidationError(f"{location} has unknown keys: {sorted(unknown)}")
def _nonempty_text(value: Any, location: str) -> str:
if not isinstance(value, str) or not value.strip():
raise DerivationValidationError(f"{location} must be a non-empty string")
return value.strip()
def _validate_observations(observations: Any) -> None:
if not isinstance(observations, list) or not observations:
raise DerivationValidationError("observations must be a non-empty list")
seen_observations: set[str] = set()
seen_evidence: set[str] = set()
for index, observation in enumerate(observations):
location = f"observations[{index}]"
if not isinstance(observation, dict):
raise DerivationValidationError(f"{location} must be an object")
_exact_keys(observation, OBSERVATION_KEYS, location)
observation_id = _nonempty_text(observation["observation_id"], f"{location}.observation_id")
evidence_id = _nonempty_text(observation["evidence_id"], f"{location}.evidence_id")
if observation_id in seen_observations or evidence_id in seen_evidence:
raise DerivationValidationError("observation and evidence provenance must be unique")
seen_observations.add(observation_id)
seen_evidence.add(evidence_id)
_nonempty_text(observation["content"], f"{location}.content")
_nonempty_text(observation["speaker"], f"{location}.speaker")
for field in ("named_person", "addressee"):
if observation[field] is not None:
_nonempty_text(observation[field], f"{location}.{field}")
def load_gold_cases(path: Path) -> list[dict[str, Any]]:
data = json.loads(path.read_text(encoding="utf-8-sig"))
if not isinstance(data, dict):
raise DerivationValidationError("Gold fixture must be an object")
_exact_keys(data, {"schema_version", "cases"}, "Gold fixture")
if data["schema_version"] != GOLD_SCHEMA_VERSION:
raise DerivationValidationError("unexpected Gold fixture schema_version")
cases = data["cases"]
if not isinstance(cases, list) or not cases:
raise DerivationValidationError("Gold fixture cases must be a non-empty list")
seen: set[str] = set()
for case in cases:
_exact_keys(case, {"case_id", "description", "observations", "expected"}, "Gold case")
case_id = _nonempty_text(case["case_id"], "Gold case.case_id")
if case_id in seen:
raise DerivationValidationError(f"duplicate case ID: {case_id}")
seen.add(case_id)
_validate_observations(case["observations"])
if len(case["observations"]) not in (1, 2):
raise DerivationValidationError("Negative Act cases require one or two observations")
return cases
def build_prompt(case: dict[str, Any]) -> str:
observations = case["observations"]
_validate_observations(observations)
candidate_id = observations[-1]["observation_id"]
return PROMPT_TEMPLATE.format(
candidate_observation_id=candidate_id,
observations_json=json.dumps(observations, ensure_ascii=False, indent=2),
)
def parse_model_json(raw_text: str) -> dict[str, Any]:
data = json.loads(raw_text)
if not isinstance(data, dict):
raise DerivationValidationError("semantic classification must be an object")
return data
def _reject_forbidden_keys(value: Any, location: str = "output") -> None:
if isinstance(value, dict):
forbidden = FORBIDDEN_LLM_KEYS.intersection(value)
if forbidden:
raise DerivationValidationError(f"{location} contains forbidden semantic keys: {sorted(forbidden)}")
for key, item in value.items():
_reject_forbidden_keys(item, f"{location}.{key}")
elif isinstance(value, list):
for index, item in enumerate(value):
_reject_forbidden_keys(item, f"{location}[{index}]")
def validate_classification(data: Any, observations: list[dict[str, Any]]) -> dict[str, Any]:
_validate_observations(observations)
if not isinstance(data, dict):
raise DerivationValidationError("semantic classification must be an object")
_reject_forbidden_keys(data)
_exact_keys(data, RECOGNITION_KEYS, "output")
observation_id = _nonempty_text(data["observation_id"], "output.observation_id")
if observation_id not in {item["observation_id"] for item in observations}:
raise DerivationValidationError("classification references unknown observation")
form = data["negative_act_form"]
if form not in NEGATIVE_ACT_FORMS:
raise DerivationValidationError("negative_act_form has an unsupported value")
action_text = data["normalized_action_text"]
if form == "none":
if action_text is not None:
raise DerivationValidationError("none form requires null normalized_action_text")
else:
_nonempty_text(action_text, "output.normalized_action_text")
return data
def _concepts_present(text: str | None, concepts: list[list[str]]) -> bool:
if not concepts:
return text is None
if not isinstance(text, str):
return False
folded = text.casefold()
return all(any(alias.casefold() in folded for alias in alternatives) for alternatives in concepts)
def evaluate_case(case: dict[str, Any], classification: dict[str, Any]) -> dict[str, Any]:
validate_classification(classification, case["observations"])
expected = case["expected"]
observation_correct = classification["observation_id"] == expected["observation_id"]
form_correct = classification["negative_act_form"] == expected["negative_act_form"]
action_correct = _concepts_present(classification["normalized_action_text"], expected["action_concepts"])
unsupported_strengthening = expected["negative_act_form"] == "none" and classification["negative_act_form"] != "none"
classification_label = "PASS" if observation_correct and form_correct and action_correct else ("PARTIAL" if observation_correct and form_correct else "FAIL")
return {
"case_id": case["case_id"], "classification": classification_label,
"expected_negative_act_form": expected["negative_act_form"],
"actual_negative_act_form": classification["negative_act_form"],
"observation_id_correct": observation_correct,
"normalized_action_meaning_correct": action_correct,
"unsupported_semantic_strengthening": unsupported_strengthening,
"normative_leakage": False,
}
def _write_json(path: Path, value: Any) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def run_experiment(args: argparse.Namespace) -> dict[str, Any]:
cases = load_gold_cases(args.cases)
args.output.mkdir(parents=True, exist_ok=False)
_write_json(args.output / "gold_cases.json", {"schema_version": GOLD_SCHEMA_VERSION, "cases": cases})
evaluations: list[dict[str, Any]] = []
successful_calls = 0
technical_failures = 0
started = time.perf_counter()
for case in cases:
case_dir = args.output / case["case_id"].lower()
case_dir.mkdir()
observations = case["observations"]
_write_json(case_dir / "v3_style_input_observations.json", observations)
prompt = build_prompt(case)
(case_dir / "prompt.txt").write_text(prompt, encoding="utf-8")
try:
raw, metadata = call_ollama(args.endpoint, args.model, prompt, args.timeout, args.num_ctx, args.num_predict)
successful_calls += 1
except Exception as exc: # one recorded attempt; never retry
technical_failures += 1
failure = {"case_id": case["case_id"], "classification": "FAIL", "technical_failure": True, "error_type": type(exc).__name__, "error": str(exc)}
_write_json(case_dir / "ollama_metadata.json", {"model": args.model, "configuration": {"temperature": 0, "think": False, "num_ctx": args.num_ctx, "num_predict": args.num_predict, "retries": 0}, "technical_failure": failure})
_write_json(case_dir / "structural_validation.json", {"valid": False, "error": str(exc)})
_write_json(case_dir / "evaluation.json", failure)
evaluations.append(failure)
continue
(case_dir / "raw_model_response.txt").write_text(raw + "\n", encoding="utf-8")
_write_json(case_dir / "ollama_metadata.json", metadata)
try:
parsed = parse_model_json(raw)
_write_json(case_dir / "parsed_semantic_classification.json", parsed)
evaluation = evaluate_case(case, parsed)
validation = {"valid": True, "error": None}
except (DerivationValidationError, json.JSONDecodeError) as exc:
validation = {"valid": False, "error_type": type(exc).__name__, "error": str(exc)}
evaluation = {"case_id": case["case_id"], "classification": "FAIL", "error": str(exc), "normative_leakage": "forbidden" in str(exc)}
_write_json(case_dir / "structural_validation.json", validation)
_write_json(case_dir / "evaluation.json", evaluation)
evaluations.append(evaluation)
summary = {
"experiment": "negative_act_form_v0", "model": args.model,
"successful_llm_call_count": successful_calls,
"technical_failed_call_count": technical_failures,
"runtime_seconds": round(time.perf_counter() - started, 3),
"counts": {label: sum(item["classification"] == label for item in evaluations) for label in ("PASS", "PARTIAL", "FAIL")},
"evaluations": evaluations,
}
_write_json(args.output / "summary.json", summary)
return summary
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run isolated Negative Act Form experiment")
parser.add_argument("cases", type=Path)
parser.add_argument("-o", "--output", type=Path, required=True)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--endpoint", default=DEFAULT_ENDPOINT)
parser.add_argument("--timeout", type=int, default=300)
parser.add_argument("--num-ctx", type=int, default=16384)
parser.add_argument("--num-predict", type=int, default=1024)
return parser.parse_args()
def main() -> int:
summary = run_experiment(parse_args())
print(json.dumps(summary, ensure_ascii=False, indent=2))
return 0 if summary["counts"]["FAIL"] == 0 and summary["technical_failed_call_count"] == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,66 @@
{
"schema_version": "experimental-negative-act-form-gold-v0",
"cases": [
{
"case_id": "NA-01", "description": "Explicit non-pursuit",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Mit Dr. Schlummer arbeiten wir nicht weiter.", "speaker": "Martin", "named_person": "Dr. Schlummer", "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "explicit_non_pursuit", "action_concepts": [["schlummer"], ["arbeit", "collabor"], ["weiter", "fortsetz", "continu"]]}
},
{
"case_id": "NA-02", "description": "Explicit non-pursuit paraphrase",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Die externe Lösung verfolgen wir nicht weiter.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "explicit_non_pursuit", "action_concepts": [["extern"], ["lösung", "solution"], ["weiter", "pursu", "continu"]]}
},
{
"case_id": "NA-03", "description": "Personal preference with local context",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage für den Versuch nutzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde das nicht machen.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "personal_preference", "action_concepts": [["real"], ["anlage", "plant"], ["versuch", "trial", "test"]]}
},
{
"case_id": "NA-04", "description": "Negative recommendation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die reale Anlage verwenden.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich würde eher davon abraten.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "recommendation", "action_concepts": [["real"], ["anlage", "plant"], ["verwend", "use"]]}
},
{
"case_id": "NA-05", "description": "Temporary non-action",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten die Waschstufe einbauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das machen wir erstmal noch nicht.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "temporary_non_action", "action_concepts": [["waschstufe", "washing stage"], ["einbau", "install"]]}
},
{
"case_id": "NA-06", "description": "Concern only",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Wir könnten das neue Material einsetzen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Das wäre kritisch.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "none", "action_concepts": []}
},
{
"case_id": "NA-07", "description": "Uncertainty",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Eine Möglichkeit wäre, die Waschstufe einzubauen.", "speaker": "Martin", "named_person": null, "addressee": null},
{"observation_id": "obs_2", "evidence_id": "e2", "content": "Martin: Ich weiß nicht, ob das sinnvoll ist.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_2", "negative_act_form": "none", "action_concepts": []}
},
{
"case_id": "NA-08", "description": "Factual negation",
"observations": [
{"observation_id": "obs_1", "evidence_id": "e1", "content": "Martin: Das Material ist nicht verfügbar.", "speaker": "Martin", "named_person": null, "addressee": null}
],
"expected": {"observation_id": "obs_1", "negative_act_form": "none", "action_concepts": []}
}
]
}
+162
View File
@@ -0,0 +1,162 @@
import argparse
import json
import tempfile
import unittest
from copy import deepcopy
from pathlib import Path
from unittest.mock import patch
import src.meeting_lab.controlled_semantic_derivation.experiment_negative_act as module
from src.meeting_lab.controlled_semantic_derivation.experiment_negative_act import (
DerivationValidationError,
build_ollama_payload,
build_prompt,
evaluate_case,
load_gold_cases,
parse_model_json,
run_experiment,
validate_classification,
)
GOLD_PATH = Path("tests/gold/negative_act_form_v0/cases.json")
FORM_TEXT = {
"NA-01": "Zusammenarbeit mit Dr. Schlummer fortsetzen",
"NA-02": "externe Lösung weiterverfolgen",
"NA-03": "reale Anlage für den Versuch nutzen",
"NA-04": "reale Anlage verwenden",
"NA-05": "Waschstufe einbauen",
}
def classification_for(case):
expected = case["expected"]
return {
"observation_id": expected["observation_id"],
"negative_act_form": expected["negative_act_form"],
"normalized_action_text": FORM_TEXT.get(case["case_id"]),
}
class NegativeActFormExperimentTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.cases = load_gold_cases(GOLD_PATH)
cls.by_id = {case["case_id"]: case for case in cls.cases}
def test_fixture_contains_exactly_na_01_through_na_08(self):
self.assertEqual(list(self.by_id), [f"NA-{number:02d}" for number in range(1, 9)])
def test_exact_schema_is_accepted(self):
case = self.by_id["NA-01"]
self.assertEqual(validate_classification(classification_for(case), case["observations"]), classification_for(case))
def test_unknown_field_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["explanation"] = "extra"
with self.assertRaisesRegex(DerivationValidationError, "unknown keys"):
validate_classification(classification, case["observations"])
def test_invalid_enum_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["negative_act_form"] = "rejection"
with self.assertRaisesRegex(DerivationValidationError, "unsupported value"):
validate_classification(classification, case["observations"])
def test_non_none_requires_normalized_action_text(self):
case = self.by_id["NA-01"]
for value in (None, ""):
classification = classification_for(case)
classification["normalized_action_text"] = value
with self.subTest(value=value), self.assertRaises(DerivationValidationError):
validate_classification(classification, case["observations"])
def test_none_requires_null_normalized_action_text(self):
case = self.by_id["NA-06"]
classification = classification_for(case)
self.assertIsNone(classification["normalized_action_text"])
classification["normalized_action_text"] = "Material einsetzen"
with self.assertRaisesRegex(DerivationValidationError, "requires null"):
validate_classification(classification, case["observations"])
def test_forbidden_normative_fields_are_rejected_recursively(self):
case = self.by_id["NA-01"]
fields = (
"rejection_form", "explicitly_rejected", "status", "decision", "outcome",
"topic_status", "responsible_person", "responsibility", "owner",
"requested_actor", "action_item", "protocol_category", "confidence",
"relation", "relations", "graph", "unresolved_issue",
)
for field in fields:
classification = classification_for(case)
classification["wrapper"] = {field: "forbidden"}
with self.subTest(field=field), self.assertRaisesRegex(DerivationValidationError, "forbidden semantic keys"):
validate_classification(classification, case["observations"])
def test_unknown_observation_id_is_rejected(self):
case = self.by_id["NA-01"]
classification = classification_for(case)
classification["observation_id"] = "obs_99"
with self.assertRaisesRegex(DerivationValidationError, "unknown observation"):
validate_classification(classification, case["observations"])
def test_malformed_json_is_rejected(self):
with self.assertRaises(json.JSONDecodeError):
parse_model_json("{bad json")
def test_all_expected_classifications_evaluate_as_pass(self):
for case in self.cases:
evaluation = evaluate_case(case, classification_for(case))
with self.subTest(case=case["case_id"]):
self.assertEqual(evaluation["classification"], "PASS")
def test_fixed_prompt_contains_candidate_and_no_gold_expectation(self):
prompt = build_prompt(self.by_id["NA-03"])
self.assertIn("candidate observation is obs_2", prompt)
self.assertNotIn("expected", prompt)
self.assertNotIn("Who is responsible", prompt)
def test_fixed_model_configuration(self):
payload = build_ollama_payload("qwen3.5:9B", "prompt", 16384, 1024)
self.assertFalse(payload["think"])
self.assertFalse(payload["stream"])
self.assertEqual(payload["options"]["temperature"], 0)
def test_no_rejection_or_status_derivation_function_exists(self):
public_names = {name for name in dir(module) if not name.startswith("_")}
self.assertNotIn("derive_rejection", public_names)
self.assertFalse(any(name.startswith("derive_") for name in public_names))
def test_artifacts_preserve_semantic_classification_only(self):
case = self.by_id["NA-01"]
raw = json.dumps(classification_for(case), ensure_ascii=False)
with tempfile.TemporaryDirectory() as temporary:
output = Path(temporary) / "run"
args = argparse.Namespace(
cases=GOLD_PATH, output=output, model="qwen3.5:9B",
endpoint="http://unused", timeout=1, num_ctx=16384, num_predict=1024,
)
with patch.object(module, "load_gold_cases", return_value=[deepcopy(case)]), patch.object(
module, "call_ollama", return_value=(raw, {"model": "qwen3.5:9B"})
):
summary = run_experiment(args)
self.assertEqual(summary["successful_llm_call_count"], 1)
case_dir = output / "na-01"
for filename in (
"v3_style_input_observations.json", "prompt.txt", "raw_model_response.txt",
"parsed_semantic_classification.json", "structural_validation.json",
"evaluation.json", "ollama_metadata.json",
):
self.assertTrue((case_dir / filename).is_file(), filename)
self.assertFalse((case_dir / "final_derived_result.json").exists())
self.assertFalse((case_dir / "deterministic_gate_results.json").exists())
parsed = json.loads((case_dir / "parsed_semantic_classification.json").read_text())
self.assertEqual(set(parsed), {"observation_id", "negative_act_form", "normalized_action_text"})
if __name__ == "__main__":
unittest.main()