diff --git a/docs/experiments.md b/docs/experiments.md index 7b0d963..c3c7922 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -1568,6 +1568,65 @@ be the candidate first semantic stage. A later bounded experiment may examine controlled semantic interpretation, but no derivation stage, production integration or Progeo run was implemented here. +## EXP-0030 — V3 model comparison: qwen3.5:9B vs qwen3.6:35B-A3B + +Date: 2026-08-19 + +This controlled comparison reran the accepted V3 minimal semantic-preservation +experiment unchanged with the locally installed `qwen3.6:35B-A3B`. It used the +same implementation, A-I fixture, evidence, prompt, minimal schema, temperature +0, `think=false`, `num_ctx=16384`, `num_predict=4096`, no retries, no voting and +one call per case. The completed run made exactly nine calls in 96.392 seconds. +Artifacts are preserved under +`artifacts/experiments/evidence_observations_v3/20260819_v3_qwen36_35b_a3b_single_run/`. + +Structural validation passed 8/9 cases. D was semantically faithful but invalid +because the model copied transcript speakers Antonius and Martin into +`named_person`, although those names were not explicitly named within their +utterances. Human semantic preservation produced 7 PASS, 1 PARTIAL and 1 FAIL: + +| Case | Verdict | Main result | +| --- | --- | --- | +| A | PASS | Preserved `can`, `perhaps`, tentative `would`, explicit `then` and no commitment, but changed the content language to English. | +| B | PARTIAL | Preserved both approaches overall, but removed `Oder` from the 20-20 observation and locally strengthened it into collective planned conduct. | +| C | PASS | Preserved Tim's tentative personal Textor contact and possible follow-up without established work. | +| D | PASS | Preserved possibility, uncertainty, process and energy consequence; structural failure was confined to invalid speaker-as-named-person values. | +| E | PASS | Preserved cost, collective rejection/non-pursuit and later determination, though the confirmation dropped explicit `Ja`. | +| F | FAIL | Preserved quantity, timing and trial/series boundaries, but changed collective `wir` into “Martin suggests” and “Tim agrees,” inventing individual proposal/agreement meaning. | +| G | PASS | Preserved hypothetical risk, collective recipient `wir`, Martin's personal stance, `wenn überhaupt`, conditional Technikum, impersonal `man müsste` and no decision/owner. | +| H | PASS | Preserved request, addressee, Friday, explicit acceptance and future personal commitment without a responsibility field; content was English. | +| I | PASS | Preserved local/pure-production and comparison boundaries, five-degree difference, upstream effort, publication purpose, unresolved permission and clarification without assignment. | + +Direct comparison: + +| Measure | `qwen3.5:9B` | `qwen3.6:35B-A3B` | +| --- | ---: | ---: | +| Structurally valid | 9/9 | 8/9 | +| Human PASS | 8 | 7 | +| Human PARTIAL | 1 | 1 | +| Human FAIL | 0 | 1 | +| Observations | 28 | 30 | +| Runtime | 40.074 s | 96.392 s | +| LLM calls | 9 | 9 | +| Prompt-evaluation tokens | 6,268 | 6,268 | +| Evaluation tokens | 2,590 | 2,718 | + +The larger model fixed the 9B weakness in G by preserving collective `wir`, and +it split I's compound production/effort observations more cleanly. Those gains +did not offset regressions: B was locally strengthened, F materially converted +collective conduct into individual agreement, D violated the participant +schema, observation count increased, and runtime was 2.4 times higher. Both +models preserved German consistently in six of nine cases, but in different +cases; the 35B-A3B model changed A, F and H to English, while 9B changed C, F +and H wholly or partly to English. + +Result: **D — REGRESSION.** `qwen3.6:35B-A3B` does not materially improve the +accepted minimal V3 first-stage preservation over `qwen3.5:9B`; it is worse on +the A-I comparison because of the F ownership-adjacent strengthening and lower +structural validity. This conclusion applies only to the minimal V3 first +stage and does not determine model choice for any later semantic derivation. +No production integration, derivation implementation or Progeo run occurred. + ## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype Date: 2026-08-11