Algolia-Central (Second-Brain)
v2/wiki/decisions/2026-06-18-pivot-to-2x2-four-panel-lab.md
ADR: Pivot to a 2×2 four-panel lab
Date: 2026-06-18 Status: Accepted
Context
The lab compared incumbent-vs-us (algolia.com website search · Ask AI floor · our single agent). That framing answers "do we beat Ask AI" but doesn't isolate which of our own choices drives answer quality, and it's not the comparison a customer cares about.
Decision
Replace the three incumbent-vs-us panels with a 2×2 experiment over our own system: two independent variables — retrieval (keyword vs neural) × architecture (single-agent vs multi-agent) — across four panels, all on identical data, each scored by the LLM judge.
| Single agent | Multi-agent | |
|---|---|---|
| Keyword | P1 | P2 |
| Neural | P3 | P4 |
(Numbering row-major: P1 single+keyword · P2 multi+keyword · P3 single+neural · P4 multi+neural.)
Rationale
- Cleanly isolates the two things worth proving: does neural beat keyword, does multi-agent beat single, and do they compound.
- Every architectural choice is data-proven by answer quality, never assumed (project's standing rule).
- A sharper, more defensible story for a flagship demo than "we beat Ask AI."
Alternatives Considered
| Option | Why rejected |
|---|---|
| Keep incumbent-vs-us 3-panel | Doesn't isolate our own variables; weaker narrative |
| 3 panels (drop one cell) | Loses the clean 2×2; can't separate the two variables |
Consequences
- Remove the algolia.com website panel and the Ask AI panel (and their wiring).
- UI goes from 3 lanes to 4 lanes.
- Requires four indices and a multi-agent flow (see related ADRs).
- Judge now does intra-system comparison across 4 panels.