How this was built

Five dimensions this assignment is actually evaluated on, each answered with a real decision I made building Quorum, not vocabulary. I built it with Claude Code (Claude Sonnet 5 as the coding assistant) across iterative sessions; every commit in this repo carries a Co-Authored-By trailer.

1. Problem structuring

User: a PM at a mobile games studio running live ops. Trigger: a message asserts something about a referent (a launch date, a success metric, a scope decision) where a live claim from a different person already conflicts with it. Outcome: the conflict surfaces with both source messages and the reasoning, before it reaches execution.

I pushed for real research before writing any code: producer pain-point threads, a former EA PM's recorded talk on misaligned definitions of success and a “data truth trap,” and the Tufts mental-model-discrepancy literature. I used that as a frame, not as evidence. I didn't cite their lab-study numbers on a two-person task as if they transferred to this domain.

Claude's first framing was a plain misalignment detector. I pushed back on it: was contradiction detection even the right unit, or was there a bigger idea underneath it? That's where I landed on “shared-truth ledger” instead: contradiction detection as the first thing built on top of a persistent belief model, not the whole idea.

Explicit assumption

No games-studio PM was interviewed to validate this. What I'd validate first, given the chance: whether contradiction (what this system detects) or omission (what it deliberately does not) is the failure mode PMs actually notice first in practice.

Success metricMeasurable from this corpus?
Detection lead timeYes
Caught before vs. after executionYes
Time to reconcilePartially
Rework avoidedNo. Needs studio-supplied revenue/cost inputs.

2. Scoping in stages

Stage 1: LedgerClaim history per referent. Useful with zero contradictions detected.
Stage 2: AdjudicationEarned by Stage 1. Runs only on buckets already holding 2+ live claims.
Stage 3: DeferredMinutes, owner/ETA tracking, gated reconciliation drafting, omission detection.

There was a bigger, flashier idea early on: an AI that sits in meetings, sends minutes, tracks owners, a full personal-assistant pitch. I dropped it deliberately, not because it was uninteresting, but because the same research that grounded the problem statement showed “too many meetings” was the #1 complaint from actual game devs. Shipping something that generates more meeting artifacts would have been tone-deaf to my own sourcing. I moved the ambition to the roadmap instead of the MVP; the ledger stayed the thing that made the roadmap trustworthy later, rather than being cut along with it.

What else got cut, and why

CutReason
Fine-tuningTwo-tier prompting is enough at this volume and eval surface
Chunk-and-embed RAGWrong unit of meaning. The unit is a claim, not a 512-token window.
Vector memory storeSame reasoning: claims are structured records, not embeddings to search
Multi-agent debate / planningWould make regression testing impossible; eval is what this project optimises for
Real OAuth, real-time streamingOut of scope for this build's surface

/settings demonstrates the Stage 3 configuration surface as a real form with real local state. Every control is traceable to an actual field in the codebase, but it stays inert by design: wiring it live would break the reproducibility guarantee the Evals tab depends on.

3. Applied AI methodology

Loop diagram: hands-on product testing surfaces a result that looks wrong or unclear, which gets traced to root cause in source rather than guessed at, then fixed, then verified against the committed eval baseline via npm run eval. If the eval shows a regression, it loops back to root-cause tracing. If it shows zero regressions, the change ships and the loop returns to more hands-on testing.

The loop I actually worked in, distinct from the running system's data-flow pipeline on Architecture.

The escalation-router incident

The most surprising thing that happened wasn't in planning. It was after deployment. A feature I added (a confidence-gated escalation router) silently regressed one eval scenario while improving another: net zero on correctness, zero buckets ever actually escalated across the full recorded set, and it sat live on the deployed site until I caught it by closely reading the eval diff output, not by trusting a headline number. I demanded a full investigation before any fix, specifically to know whether this was a baseline-freezing artifact or a real logic bug before deciding what to do.

Given the investigation, I reverted the router entirely rather than keep the one scenario it had “improved.” Carrying a known, unresolved regression into a submission wasn't acceptable just because the net was zero. I fixed a separate, genuinely diagnosable bug found in the same pass (a referent-resolution issue behind one missed contradiction). I left a second failing case reported as an honest, documented limitation rather than forcing a fix. One real, understood gap beats something that looks artificially perfect. See commit b69ca2d.

Concrete choices, not buzzwords

  • Model selection, sequenced and reversible. Free tier first, deliberately, to prove the whole orchestration before spending on a paid model. This shaped the architecture, not just the budget: every model call goes through one ModelClient interface, and every implementation (live, replay, stub, fallback) satisfies it identically. I confirmed this by grepping for any instanceof check on a concrete client anywhere in the pipeline and finding none. A strong config (Claude Sonnet 5 for adjudication) is already fully plumbed and selectable via --config=strong. Swapping the free model for a stronger one later is a config change, not a rewrite, and that interface boundary is the receipt, not just the claim.
  • Context loading is bounded, on purpose. Extraction sends the prior 3 messages in the same thread as context, not the whole thread. That's enough for a single-claim extraction task, without paying for or diluting the call with unbounded history.
  • No hand-offs between sub-agents. There is no planner and no sub-agent delegation. The pipeline order (gate, extract, resolve, pre-rule, adjudicate, persist) never varies, and nothing decides what runs next at runtime. A planner agent choosing the pipeline dynamically would make regression testing impossible, and eval quality is explicitly what this project optimises for over agentic flexibility.
  • No git hooks. Not used in this build. The closer equivalent is a manual discipline: every change to the scored pipeline is gated by reading npm run eval's baseline diff before it's considered done. Same check a pre-commit hook would automate, done by habit instead of tooling here.
  • MCP as the real external-tool boundary. mcp-server/ exposes the fixture corpus over stdio as four tools (Slack search/thread, Gmail search/thread), both backed by one shared adapter module. Said plainly: the web app doesn't yet import that same adapter. It reads the identical corpus through a separate in-process implementation of the same interface. One interface, two call-sites, not yet consolidated. A known next step, not hidden.

The free tier has a real rate limit, and it's enforced, not just documented

Live mode is capped at 10 calls per session per 10 minutes: an in-memory counter keyed to a session cookie, not a suggestion. Go over it and the API returns a 429 with “Rate limit reached for this session (10 live calls per 10 minutes). Try replay mode, or wait.” Sandbox's retry-live button hits the exact same counter as a normal run. It asks for more output room per call; it doesn't get more calls.

This limit exists for fairness, not because recording the eval fixtures ran into it. The free-tier model itself already rate-limits per request regardless of anything in this app, so the counter here just stops one person's live exploration from eating the whole shared quota. Replay being the default mode is a separate decision, made for reproducibility: byte-identical results, offline, no API key needed. The two facts sit next to each other, but one didn't cause the other.

Gold claims (the labels the adjudication grader is scored against) are hand-labeled by one person: me, with no measured agreement from anyone else checking my work. I say that plainly rather than smoothing it over. They live in evals/gold-claims.json, and I put the full set on the Architecture page, with the real source message behind each one, so a reviewer doesn't have to take “gold” on faith.

Scope discipline near the end

I considered live-configurable temperature/thresholds and a Zapier/Make-style drag-and-drop orchestration builder. I dropped the node-builder once it was clear a flashy graph UI would visually misrepresent a system that's mostly deterministic code with only two narrow model calls in it. /settings stayed, but scoped explicitly as a non-functional demonstration surface, protecting the reproducibility guarantee the Evals tab depends on rather than trading it for a more impressive-looking page.

Shipping it

This is a plain Next.js App Router project, so deploying it is zero-config: no vercel.json, no custom build step. I connected the GitHub repo in the Vercel dashboard once, and every push to main auto-deploys from there. Live mode and the AI Gateway key aren't committed anywhere: AI_GATEWAY_API_KEY and LIVE_MODE_ENABLED are set as Vercel project environment variables, same as any secret. One thing that caught me: Vercel doesn't pick up a new environment variable on an already-running deployment. Adding one means triggering a fresh deploy, not just saving the setting.

4. Eval design

Two graders, never merged. The extraction grader scores the pipeline's real predicted claims. The adjudication grader is fed gold claims only, never the extractor's own output, so a bad extraction run can neither mask nor manufacture an adjudication error. Per-scenario results, never averaged: a weak spot must not hide behind a healthy mean.

False positive rate0/18 = 0
Contradiction recall6/8
Span validity65/65 = 1

Reproducible via npm run eval, or live in the browser on the Evals tab. No API key required either way.

One scenario (a staged rollout with a stale client cache, where both readings can genuinely be true at once) is labeled CONTESTED and excluded from headline scoring entirely, rather than folded into precision or recall where it would misrepresent what the system actually got right or wrong.

The single best evidence the ledger is stateful, not a pairwise text diff: the flagship launch-date bucket evaluated at two points in time. A live contradiction between two people at one snapshot, resolved by deterministic authority-based supersession once the studio head's message lands at the next. Never by asking the model to be clever about who outranks whom.

The reproducibility push itself came from a product judgment, not a technical default. I assumed a reviewer wouldn't run terminal commands, so the eval suite needed to be runnable from the UI, not just documented. That decision is why Evals is a first-class in-browser tab today, not a script with a results file attached.

5. System design accuracy

npm ci && npm run eval reproduces the same per-scenario numbers on any machine, offline, with no API key. Replay is the only mode the eval harness uses: every model call is cached to a committed recording, keyed on model, prompt version, and semantic input. The replay boundary is narrow. Only the raw model HTTP response is cached; everything else (noise gate, schema/span validation, referent resolution, pre-rules, both graders) runs live in every mode. A replay miss is a hard, visible error, never a silent fallback. Nothing in the scored pipeline calls Date.now(), enforced by a repo-wide test, so evaluation time is a frozen, injected parameter everywhere.

Every model-call implementation shares one interface. Live, replay, stub, and fallback all implement the same ModelClient, and nothing branches on which one it's holding. The pipeline only ever calls model.call(...). Grepping for an instanceof check on any concrete client, anywhere a model gets called, turns up zero. That's why swapping the free model for the already-plumbed strong config is a config change, not a rewrite.

Persistence: an env var (LEDGER_STORE) picks between a file-backed store, which survives a restart, and an in-memory store that matches a warm serverless instance and resets on cold start. Both exist because the deployment target determines which guarantee is available, not because one is more “finished” than the other.

Failure modes, named rather than hidden

  • A JSON parse failure falls back to COMPATIBLE, deliberately. This can only ever hurt contradiction recall, never inflate the false-positive rate. A conservative failure mode I chose on purpose, not an accident.
  • The flagship launch-date transition reproduces on the gold-claims eval path but not on the live-app extractor path. One specific message's live extraction call truncates before finishing, and is correctly rejected rather than guessed at. I root-caused it exactly, disclosed it on the live page itself, and deliberately didn't patch it by re-prompting or raising the token cap after seeing the result. That would be exactly the kind of post-hoc tuning this project refuses to do.
  • The noise gate's short-message rule also gates some legitimate short acknowledgements along with genuine chatter. An accepted precision/recall tradeoff in the gate itself, reported via a visible gated-message count rather than hidden.