meeting-lab/docs/protocol_generation_decision.md

# Protocol Generation Decision

## Executive Summary

Meeting Lab tested ten protocol-generation and runtime variants against the
same 93.5-minute reference meeting, `project_process_meeting`. The current
evidence does not support a fully automatic protocol. The best practical local
baseline remains one direct call to `qwen3.6:35B-A3B`, followed by informed
human review. It is fast and produces readable, broadly useful Markdown, but it
still overstates consensus, compresses unresolved process boundaries, misses
some challenge and follow-up paths, and can infer unsafe ownership. Its current
quality verdict is **C — promising but insufficient**.

Additional prompting, review, Meeting Map, hierarchical, diarized, dense-model
and 70B-scale variants did not produce a reliable step change. Some improved
individual dimensions, but none reached a stable B result or removed the need
for substantive review. This is a current evidence-based product choice, not a
permanent architecture decision.

## Experiments Compared

All protocol rows used the complete cleaned transcript and meeting context
unless stated otherwise. A dash means that the artifact did not record the
number; it is not an estimate. Verdicts marked “assessment” are comparative
assessments of saved outputs because those older artifact directories contain
no formal `quality_review.json`.

| Experiment | Model | Architecture | LLM calls | Prompt tokens | Runtime | Human editing | Verdict | Main strength | Main failure | MVP | Research |
| --- | --- | --- | ---: | ---: | ---: | --- | --- | --- | --- | --- | --- |
| Direct one-shot baseline | Qwen3.6 35B-A3B | Direct full-context protocol | 1 | 18,385 | 72.5 s cold; about 38 s inference | Not recorded | C | Fast, readable, broad topic outline | Consensus and ownership overpromotion; missing boundaries and follow-up | **Yes, with review** | Baseline |
| Conservative one-shot | Qwen3.6 35B-A3B | Direct with stronger safety instructions | 1 | 19,120 | 39.5 s warm | Not recorded | C (assessment) | Better uncertainty and pending-feedback language | Still invents or upgrades named follow-up actions | No | Limited |
| Draft → review | Qwen3.6 35B-A3B | Conservative draft plus review call | 2 | 39,071 total | About 110.4 s summed | Not recorded | C (assessment) | Removes some unsafe named attribution | Does not reliably restore omitted content; empty/weak action sections remain | No | Limited |
| Meeting Map → protocol | Qwen3.6 35B-A3B | Semantic map followed by rendering | 2 | 39,129 total | 134.8 s | Not recorded | C (assessment) | Explicit intermediate structure | Map errors propagate: false consensus and named ownership remain | No | Yes |
| Hierarchical notes → protocol | Qwen3.6 35B-A3B | Five chunk-note calls plus synthesis | 6 | 32,145 total | 251.1 s | Not recorded | C (assessment) | Highest recall in several detailed/open topics | Amplifies unsupported speaker/name interpretations and confirmed actions | No | Yes |
| Segment-level anonymous diarization | Qwen3.6 35B-A3B | One-shot over 1,154 labeled segments | 1 | 39,308 | 125.4 s | 25–35 min | C | Preserves some filtered-idea challenge structure | Token count more than doubled; actions and deadlines became less safe | No | No further protocol tests |
| Turn-merged anonymous diarization | Qwen3.6 35B-A3B | One-shot over 435 merged turns | 1 | 22,490 | 101.2 s | 25–35 min | C | Corrected token inflation; recovered some topic and feedback detail | Still did not beat raw input; unsafe actions/deadlines persisted | No | UI/search only |
| Dense one-shot | Qwen3.5 27B | Direct full-context protocol | 1 | 18,385 | 181.2 s | Not recorded | C (assessment) | Somewhat better recall of process details | Much slower; no material overall quality gain | No | No |
| 70B scale one-shot | Llama 3.3 70B Q3_K_S | Dense, 64% CPU / 36% GPU | 1 | 21,156 | 470.1 s warm | 45–60 min | D | Technically proved a 70B hybrid load can run | Severe coverage loss, invented governance, internal contradiction | No | Negative scale result |
| Ollama vs native llama.cpp | Qwen3.6 35B-A3B | Same Q4_K_M GGUF; ROCm/Vulkan servers | 1 per backend | 18,385 | ROCm 38.4 s; Vulkan 42.3 s | N/A | Runtime only | Native ROCm reached 51.38 generated tok/s | No meaningful end-to-end advantage; more operational complexity | Ollama | Runtime reference |

The draft-review total combines the saved conservative draft call and the
saved review call. Its review metadata itself reports only the one new review
call (19,951 prompt tokens and 70.8 seconds). The direct baseline's 72.5-second
wall time includes a 34.5-second cold load; its measured prompt evaluation plus
generation was 37.8 seconds. These distinctions explain apparent runtime
differences between otherwise similar Qwen3.6 calls.

### Recurring quality patterns

- **Topic coverage and factual accuracy:** Direct Qwen3.6 captures the main
  process but misses the second review after enrichment, project reporting and
  parts of the filtered-idea challenge path. Hierarchical processing recalls
  more detail but introduces too many unsupported interpretations. Llama 3.3
  loses most of the meeting and invents a governance role for the
  Geschäftsführung.
- **Consensus and unresolved boundaries:** Every broad one-shot family remains
  vulnerable to turning discussion or a working direction into agreement. The
  unresolved boundary between central coordination and autonomous department
  work, and the uncertainty around universal filter criteria, are especially
  fragile.
- **Visibility, veto and reconsideration:** No approach consistently preserves
  initial filtering, later cross-functional challenge, reconsideration after
  enrichment and the return through the project cycle together.
- **Stakeholder feedback:** Pending Jovana and Björn feedback is an important
  quality probe. Some variants preserve both; segment-level diarization drops
  Björn, while Llama 3.3 drops both.
- **Actions and attribution:** Added structure does not guarantee safety.
  Conservative, reviewed, Meeting Map, hierarchical and diarized outputs still
  promote proposals or expected work into confirmed actions, infer owners from
  roles or conversational context, or invent deadlines. Human review remains
  mandatory.

## Model Findings

### Qwen3.6:35B-A3B

Qwen3.6 is the best overall local practical baseline. Its Q4_K_M model is
operationally fast on the RX 9070/CPU hybrid setup, follows the requested
Markdown form and usually provides a useful first draft. It remains verdict C:
larger context and fluent synthesis do not reliably protect evidence strength,
responsibility attribution or unresolved process boundaries.

### Qwen3.5:27b dense

The dense 27B run recalled some process details better than the MoE baseline,
but took 181.2 seconds and generated at 8.45 tokens/s. The gains did not amount
to a material overall quality improvement. This result does not prove that
dense models are generally inferior; it shows that this dense model is not a
better product choice on this hardware and meeting.

### Llama 3.3 70B Q3_K_S

Llama 3.3 70B was technically runnable at 32k context with a 42 GB loaded
footprint and a 64% CPU / 36% GPU split. Its 7m50s warm meeting run produced a
very short, materially worse protocol: one critical invented governance claim,
five new major errors and an estimated 45–60 minutes of editing. Raw parameter
count alone is therefore insufficient. The older model generation and
aggressive Q3 quantization are plausible contributors, but this experiment
does not isolate or prove either cause.

## Runtime Findings

The native comparison reused the exact 23,938,321,664-byte Qwen3.6 Q4_K_M GGUF
that Ollama uses. The tested `llama-server` binary was the llama.cpp runtime
shipped with the installed Ollama distribution, not an independent source
build.

Native ROCm processed the reference request in 38.4 seconds and generated at
51.38 tokens/s. Vulkan took 42.3 seconds and generated at 46.73 tokens/s. The
comparable Ollama baseline generated at 44.68 tokens/s, with about 37.8 seconds
of prompt evaluation plus generation when load time is excluded. Output token
counts differed, so generation throughput alone is not an end-to-end quality or
latency comparison.

Native ROCm gained some generation throughput, but did not provide a meaningful
end-to-end advantage for this workload. Vulkan required more host spill and was
not preferable. Ollama already provides the relevant llama.cpp runtime
components, model lifecycle and API integration; it remains the preferred
routine Meeting Lab runtime.

## Diarization Findings

### Technical feasibility

Pyannote `speaker-diarization-community-1` successfully processed the
93.5-minute meeting on CPU in 1,647 seconds (about 27m27s), an RTF of 0.293. It
detected four anonymous clusters and assigned 1,145 of 1,154 Whisper segments
(99.2%). Peak RSS was about 3.3 GiB. This establishes technical feasibility; it
does not establish speaker identity or diarization accuracy against labeled
ground truth.

### Protocol-quality impact

Annotating every Whisper segment increased the Qwen prompt from 18,385 to
39,308 tokens, confounding speaker structure with fragmentation and token
inflation. Deterministic turn merging reduced 1,154 segments to 435 turns and
the complete prompt to 22,490 tokens. That controlled the main representation
confound, but the resulting protocol still did not materially outperform the
raw transcript and remained verdict C.

Anonymous diarization is therefore not justified as a mandatory MVP
protocol-quality feature. This does **not** mean diarization is generally
useless. It may remain valuable for speaker-aware UI, navigation and search,
participation statistics, traceability, or later carefully validated real-name
mapping.

## Semantic Research Findings

The semantic experiments provide architectural evidence, but should not
dominate the product decision:

- **Evidence Observation V3** is a strong evidence-near candidate stage. With
  Qwen3.5 9B it achieved 8 PASS, 1 PARTIAL and 0 FAIL while preserving hedges,
  alternatives, requests, commitments and boundaries in natural language.
- **Request/Acceptance** and **Collective Commitment** show that narrow semantic
  recognition followed by deterministic provenance, ordering, addressee,
  negation and deadline gates can safely derive limited consequences. Model
  recognition errors were contained without inventing individual ownership.
- **Explicit Rejection** failed when reduced to a coarse binary recognition
  problem: semantically valid false positives passed structural gates.
- **Negative Act Form** worked better by distinguishing non-pursuit, personal
  preference, recommendation and temporary non-action before any normative
  derivation. All eight form classifications matched Gold, although normalized
  action text was imperfect in three cases.
- **Target Resolution V0** failed because prompt examples and a weak JSON
  boundary encouraged the string `"null"` instead of typed linkage.
  **Target Resolution V1** fixed all linkage/ID failures with deterministic
  self-linkage, closed ID lists and true JSON Schema, but normalization remained
  incomplete.
- **Target Normalization V0** improved polarity and scope preservation to 3/4
  PASS, but still lost continuation meaning in the collaboration case.

These findings support Meeting Lab as a research and validation track. They do
not yet justify placing a multi-stage semantic pipeline on the MVP critical
path.

## Current MVP Decision

The current product path is:

```text
Audio
-> transcription
-> direct qwen3.6:35B-A3B protocol generation through Ollama
-> informed human review
-> final protocol
```

The first MVP should treat the generated protocol as an editable draft, not an
authoritative semantic record. Human review must specifically check consensus,
unresolved boundaries, competing positions, action status, owners, deadlines
and pending stakeholder feedback.

Diarization is optional and deferred. The semantic research pipeline remains
in Meeting Lab, outside the MVP critical path. Ollama remains the default local
runtime.

## Rejected / Deferred Directions

- Do not continue Qwen3.6 prompt variants as the main quality strategy.
- Do not add draft-review, Meeting Map or hierarchical generation to the MVP;
  their added calls and complexity did not deliver reliable quality gains.
- Do not continue anonymous-diarization protocol experiments. Revisit
  diarization for UI, search, statistics or traceability instead.
- Do not use Qwen3.5 27B or Llama 3.3 70B Q3_K_S as the routine protocol model.
- Do not replace Ollama with a manually managed native llama.cpp service for
  this workload.
- Retain semantic experiments, but defer production integration and broad
  semantic consolidation.

## Open Questions

1. How does a genuinely newer, materially stronger model perform when a useful
   quantization fits the available RAM/VRAM without severe swap?
2. If project policy permits, what quality ceiling does the unchanged reference
   prompt achieve with a commercial frontier model?
3. What is the measured reviewer time and correction distribution once the
   direct Qwen3.6 draft path is exercised in an end-to-end MVP workflow?
4. Which non-protocol product benefits justify revisiting diarization later?

No further Qwen3.6 prompt variants, anonymous-diarization protocol runs or old
70B Q3 scale tests are recommended.

## Recommended Next Product Step

Build the practical end-to-end MVP around direct Qwen3.6 generation and an
explicit human review handoff. Measure reviewer time and correction categories
in real use. Keep the experiment artifacts and semantic Gold work as validation
evidence, but do not block the first product loop on broader research stages.