Document V3 model comparison

2026-08-19 15:55:01 +02:00
parent 18beb3385f
commit 4ffd4c1c5d
1 changed files with 59 additions and 0 deletions
@@ -1568,6 +1568,65 @@ be the candidate first semantic stage. A later bounded experiment may examine
 controlled semantic interpretation, but no derivation stage, production
 integration or Progeo run was implemented here.
 ## EXP-0030 — V3 model comparison: qwen3.5:9B vs qwen3.6:35B-A3B
 Date: 2026-08-19
 This controlled comparison reran the accepted V3 minimal semantic-preservation
 experiment unchanged with the locally installed `qwen3.6:35B-A3B`. It used the
 same implementation, A-I fixture, evidence, prompt, minimal schema, temperature
 0, `think=false`, `num_ctx=16384`, `num_predict=4096`, no retries, no voting and
 one call per case. The completed run made exactly nine calls in 96.392 seconds.
 Artifacts are preserved under
 `artifacts/experiments/evidence_observations_v3/20260819_v3_qwen36_35b_a3b_single_run/`.
 Structural validation passed 8/9 cases. D was semantically faithful but invalid
 because the model copied transcript speakers Antonius and Martin into
 `named_person`, although those names were not explicitly named within their
 utterances. Human semantic preservation produced 7 PASS, 1 PARTIAL and 1 FAIL:
 | Case | Verdict | Main result |
 | --- | --- | --- |
 | A | PASS | Preserved `can`, `perhaps`, tentative `would`, explicit `then` and no commitment, but changed the content language to English. |
 | B | PARTIAL | Preserved both approaches overall, but removed `Oder` from the 20-20 observation and locally strengthened it into collective planned conduct. |
 | C | PASS | Preserved Tim's tentative personal Textor contact and possible follow-up without established work. |
 | D | PASS | Preserved possibility, uncertainty, process and energy consequence; structural failure was confined to invalid speaker-as-named-person values. |
 | E | PASS | Preserved cost, collective rejection/non-pursuit and later determination, though the confirmation dropped explicit `Ja`. |
 | F | FAIL | Preserved quantity, timing and trial/series boundaries, but changed collective `wir` into “Martin suggests” and “Tim agrees,” inventing individual proposal/agreement meaning. |
 | G | PASS | Preserved hypothetical risk, collective recipient `wir`, Martin's personal stance, `wenn überhaupt`, conditional Technikum, impersonal `man müsste` and no decision/owner. |
 | H | PASS | Preserved request, addressee, Friday, explicit acceptance and future personal commitment without a responsibility field; content was English. |
 | I | PASS | Preserved local/pure-production and comparison boundaries, five-degree difference, upstream effort, publication purpose, unresolved permission and clarification without assignment. |
 Direct comparison:
 | Measure | `qwen3.5:9B` | `qwen3.6:35B-A3B` |
 | --- | ---: | ---: |
 | Structurally valid | 9/9 | 8/9 |
 | Human PASS | 8 | 7 |
 | Human PARTIAL | 1 | 1 |
 | Human FAIL | 0 | 1 |
 | Observations | 28 | 30 |
 | Runtime | 40.074 s | 96.392 s |
 | LLM calls | 9 | 9 |
 | Prompt-evaluation tokens | 6,268 | 6,268 |
 | Evaluation tokens | 2,590 | 2,718 |
 The larger model fixed the 9B weakness in G by preserving collective `wir`, and
 it split I's compound production/effort observations more cleanly. Those gains
 did not offset regressions: B was locally strengthened, F materially converted
 collective conduct into individual agreement, D violated the participant
 schema, observation count increased, and runtime was 2.4 times higher. Both
 models preserved German consistently in six of nine cases, but in different
 cases; the 35B-A3B model changed A, F and H to English, while 9B changed C, F
 and H wholly or partly to English.
 Result: **D — REGRESSION.** `qwen3.6:35B-A3B` does not materially improve the
 accepted minimal V3 first-stage preservation over `qwen3.5:9B`; it is worse on
 the A-I comparison because of the F ownership-adjacent strengthening and lower
 structural validity. This conclusion applies only to the minimal V3 first
 stage and does not determine model choice for any later semantic derivation.
 No production integration, derivation implementation or Progeo run occurred.
 ## EXP-0026 — Topic-oriented Discussion Subject reconstruction V2 prototype
 Date: 2026-08-11