Algolia-Central (Second-Brain)
v2/wiki/decisions/2026-06-28-content-source-routing-spike-verdict.md
ADR: Content-Source Multi-Agent Routing — Spike Verdict + Data-Bound Depth Ceiling
Date: 2026-06-28 Status: Accepted (supersedes-in-part 2026-06-18-content-source-multi-agent)
Context
We had never tested with data whether routing a question to a content-source-honed specialist (support→support agent, academy→academy agent, …) actually produces better answers than ONE all-source neural agent. The 2026-06-18 ADR assumed content-source specialists were the multi-agent design. This spike tested the assumption.
Decision
- Content-source SCOPING alone gives NO answer-quality lift over a single all-source neural agent → as a pure "cordon each agent to a source" play, multi-agent is overhead. KILLED in that form.
- The real depth ceiling is the DATA, not the architecture or the prompt. The corpus is a search index of titles + short summaries + facets — only Support (median desc 626 chars, 70% deep) and "Other" (median 6,503) carry deep text. Documentation is 96% one-line stubs; Academy/Blog/Developers are catalog-grade.
- Next phase (in progress): "hone within the data's limits" (Path 1). Purpose-built specialist prompts (depth doctrine per source) + a warm baton (RC2-style context handover) — tested against an all-source baseline. Expectation: Support (and Marketer-on-"Other") may show real lift; Technical/Academy tie (their ceiling is data).
Rationale
- A/B/C spike, 4-dim judge, 32 v3 Qs + 6 stress, 3 rounds. First run (baseline = 6-source incumbent Maverick) → +0.76 "multi-agent wins." Re-run with a FAIR all-source baseline (
ac2-allsource-neural, same MULTI_NEURAL index, NO source filter) → +0.25, within ±0.3 noise → KILL. The +0.51 swing was pure baseline-blindness confound (incumbent couldn't see Academy/Support). Lesson: baseline and treatment must share the same information set. (Claude memory note:feedback-ab-baseline-same-information-set) - Data-hardening (n=1,000/source, not n=2) confirmed the ceiling and corrected two of my own n=2 errors: Customer Stories are 85% usable (not "corrupted" — only ~15% Oh Polly dup); "Other" is deep (not 1-line). Facets (
facet1industry,facet6features) ARE returned to the model by the agent's search tool (verified by dumping a live tool-result) — so facet-based proof is valid where facets exist (Customer Stories, Developers, Resources, Blog). - Honing mostly reshapes the answer, not retrieval; so where the data is thin, no prompt creates depth. Only Support/"Other" have raw material for genuine depth.
Alternatives Considered
| Option | Why rejected (for now) |
|---|---|
| Ship content-source multi-agent as quality play | No measured lift on a fair baseline; +2× token cost + router-error mode |
| Enrich index with full doc/lesson bodies, then test (Path 2) | The real unlock for deep Technical/Academy answers, but bigger effort — deferred until honed-prompt run shows the data ceiling bites |
| Stop entirely (Path 3) | Arijit chose Path 1 — test honed prompts + warm baton honestly within data limits first |
Consequences
- New agent on CENTRAL
0EXRPAXB56:ac2-allsource-neural(id26712546-8a6b-4a17-bd7d-18f7a7621746) — fair all-source baseline (same index, no filter). Created byscripts/setup/create_allsource_agent.mjs(idempotent). - Honed specialist prompts drafted (data-realistic v3):
scripts/setup/honed/instructions_{support,technical,academy,marketer}.md+_shared_grounding.md(grounding + warm-baton block). NOT yet pushed live. - Spike harness:
lab/server/src/experiments/sourceRoutingAb.ts+sourceRouting/(labels, classifier, routingAgg) +npm run expt:sourcerouting; 18 unit tests; verdict docdocs/experiment/2026-06-28-content-source-routing-verdict.md. - Agent Studio list API paginates at 10 — must use
?limit=100(21 agents live). - Data quality bug to fix: Customer Stories ~15% duplicate descriptions (Oh Polly text on 12 records).
- NEXT: push Support honed prompt live → 4-question adherence smoke → if good, push other 3 + run warm-baton multi-turn A/B vs
ac2-allsource-neural.