Evals

How we know Quorum can be trusted: three things that could go wrong (false alarms, missed conflicts, invented quotes), each measured separately below against a human-labeled gold set — run it yourself, no API key, no network call.

Read the full eval methodology ↓

What could go wrong

  • False alarms. Quorum tells you two people disagree when they don't — the fastest way to lose trust.
  • Missed conflicts. Two people genuinely disagree and Quorum stays silent.
  • Invented quotes. Quorum shows you words nobody actually wrote.

How each is measured

  • False alarm rate — how often a scenario that should never be flagged gets flagged anyway. Target: zero.
  • Conflicts caught — of the genuine disagreements in the test set, how many Quorum actually surfaces.
  • Quote accuracy — every highlighted phrase is checked character-for-character against the real message; a highlight that doesn't match the source is a failure, not a rounding error.

Two separate graders

Extraction (did we correctly read what someone said) and judgment (did we correctly decide whether two readings conflict) are scored independently, on their own scenarios. A system that extracts perfectly but judges badly, or vice versa, cannot hide behind a single blended score — see the per-scenario tables below, never averaged into one number.

What the frozen baseline shows

Run it yourself above and the section below fills in with a live diff against the committed baseline — including the one regression currently on record, kept visible rather than reset, because that is what catching a regression is supposed to look like.

Guardrailed vs Open

Guardrailed (binary): the model answers exactly one question per topic — “do these live positions genuinely disagree?” — everything else (updates, corrections, who overrides whom) is decided by code before the model is ever called. Open (full7): the model chooses freely from the entire verdict vocabulary itself. Both run on the same free-tier model — this measures how much the guardrails change the outcome, not a cheap-vs-expensive model difference.

Deterministic rules (R1–R8)

Applied to every topic before any model is called. If a rule fully settles the question, the model is never invoked for that topic at all. These are hand-authored, studio-specific business rules (src/core/prerules.ts) — code, not model output, and not user-configurable in this build.

R1Someone relaying what another person said doesn't count as their own claim — excluded from the disagreement.
R1bA hedge, a proposal, or a question isn't a statement of fact — excluded.
R2The same person said something different later — their earlier message is retired in favour of the later one.
R3Someone explicitly said what something is NOT — checked against their own other positions rather than read as a fact itself.
R4Someone corrected themselves to match what someone else already said — read as a correction, not an open conflict.
R5The most senior person in the thread made the final call — it overrides the earlier disagreement.
R6Only one current position remains — nothing left to disagree about.
R6bMore than one current position, but they all agree — not a conflict.
R7No current position remains at all — nothing to disagree about.
R8Genuinely arguable both ways — kept in its own "contested" bucket rather than scored as right or wrong.
The actual prompts sent to the model

Exactly what's sent to the model — no scenario-specific hints, no few-shot examples encoding the right answer. “System” is the fixed instructions sent on every call. “User” below is not a description of what would be sent — it is a worked example: the literal, real output of renderUser() for M-001, the flagship bucket's opening message (“Kicking off planning for the Independence Day event...”), exactly as the model receives it.

Extraction (reads a message, emits claims) — system prompt:

view prompt
System
You extract factual claims made by the message author about project state.

Emit one object per distinct claim the author makes. Emit an empty array if the message makes no claim.

For each claim:
- "referent": a short noun phrase naming what the claim is about.
- "value": the value or position the author asserts.
- "source_span": copied character-for-character from the message text. Do not paraphrase, do not fix typos, do not add ellipses. This must be an exact substring of the message.
- "modality": one of "assertion", "hedge", "proposal", "question", "reported".
  - "assertion": the author states it as fact.
  - "hedge": the author softens or qualifies it.
  - "proposal": the author is suggesting a change, not stating current fact.
  - "question": the author is asking, not asserting.
  - "reported": the author is relaying what someone else said or decided, not asserting it themselves.
- "polarity": "positive" if the author affirms the value, "negative" if the author denies or rejects it.
- "attributed_to": when modality is "reported", the name or handle of the person credited with the statement. Otherwise null.

Do not infer claims from any context messages provided; they are shown only to help you resolve pronouns and are not themselves a source of claims.

Output JSON only, matching this schema exactly:
{"claims": [{"referent": string, "value": string, "source_span": string, "modality": "assertion"|"hedge"|"proposal"|"question"|"reported", "polarity": "positive"|"negative", "attributed_to": string|null}]}
User
Message from Meera Iyer (Product Manager):
Kicking off planning for the Independence Day event. Working assumption is we go live 12 August, config frozen by the 5th so QA gets a clean week.

Judgment, Guardrailed (binary) scope — system prompt, and a worked example built from M-001's claim plus its real gold contradiction partner (M-002, the C1 scenario):

view prompt
System
Decide whether a set of live claims are mutually incompatible statements about the same thing.

Two claims are incompatible if they cannot both be true at the same time. Differences in emphasis, scope, or wording are not incompatibility.

You are told which claims are live. Claims that were superseded, withdrawn, or reported have already been removed from what you see — do not reason about them or try to reconstruct them.

Output "CONTRADICTION" if the live claims conflict, "COMPATIBLE" if they do not.

Output JSON only, matching this schema exactly:
{"verdict": "CONTRADICTION"|"COMPATIBLE", "rationale": string, "conflicting_claim_ids": string[]}
"conflicting_claim_ids" should list the claim ids that are in conflict, or an empty array if COMPATIBLE.
User
Referent: indep_event.launch_date

Live claims:
- M-001#0 | meera.iyer | 6 Jul 2026, 10:12 AM IST | value: "2026-08-12" | span: "we go live 12 August"
- M-002#0 | priya.raghunathan | 15 Jul 2026, 6:22 PM IST | value: "2026-08-15" | span: "Go-live is 15 August"

Judgment, Open (full7) scope — same worked example, different system prompt:

view prompt
System
Decide the relationship between a set of live claims about the same referent.

You are told which claims are live. Claims that were superseded, withdrawn, or reported have already been removed from what you see.

Choose exactly one verdict:
- "CONTRADICTION": two or more live claims from different people conflict and neither has been resolved.
- "UPDATE": a single person's claim has simply been revised over time.
- "RESOLVED_BY_SUPERSESSION": a higher-authority person's claim has settled an earlier conflict.
- "RESOLVED_BY_CORRECTION": someone withdrew their own claim in favor of someone else's.
- "AMBIGUOUS_REFERENT": the claims only appear to conflict because they are actually about different things.
- "COMPATIBLE": the live claims do not conflict.
- "CONTESTED": reasonable people could disagree about whether this is a conflict at all.

Output JSON only, matching this schema exactly:
{"verdict": "CONTRADICTION"|"UPDATE"|"RESOLVED_BY_SUPERSESSION"|"RESOLVED_BY_CORRECTION"|"AMBIGUOUS_REFERENT"|"COMPATIBLE"|"CONTESTED", "rationale": string, "conflicting_claim_ids": string[]}
User
Referent: indep_event.launch_date

Live claims:
- M-001#0 | meera.iyer | 6 Jul 2026, 10:12 AM IST | value: "2026-08-12" | span: "we go live 12 August"
- M-002#0 | priya.raghunathan | 15 Jul 2026, 6:22 PM IST | value: "2026-08-15" | span: "Go-live is 15 August"
How this page works (for reviewers)

npm run eval -- --print-hash against the same committed recordings prints the same report hash offline — that is the actual reproducibility guarantee, not just the on-screen claim above. The regression protocol treats any single-scenario regression as a failure even if the average improves; the diff panel above implements exactly that rule, not an aggregate pass/fail. See src/core/eval/diff.ts and src/core/eval/run-eval.ts for the scoring implementation, and src/core/eval/scenarios.ts for the full scenario registry (C1-C9, N1-N18) this page's tables are driven from.

Full detail in the README: /architecture#evals