Dashboard/Algolia-Central (Second-Brain)/v2/wiki/open-questions.md
Algolia-Central (Second-Brain)

v2/wiki/open-questions.md

Open Questions — Answer-Quality Lab

Resolved

  • [x] What does "multi-agent" mean here? Resolution (2026-06-18): content-source specialists (not sales-discovery), coordinated by Maverick. See 2026-06-18-content-source-multi-agent.
  • [x] Port rc2 or rc3 wholesale? Resolution (2026-06-18): Neither — borrow the orchestrator→specialist pattern only; build purpose-built specialists scoped by source.
  • [x] Specialist partition. Resolution (2026-06-18): Technical {Documentation, Developers, Customer Stories} · Support · Website {Website, Blog, Resources, Other} · Academy.
  • [x] Language scope. Resolution (2026-06-18): English only for now.
  • [x] Seed snapshot. Resolution (2026-06-18): latest available at build time, deduped by url.
  • [x] allOptional on Panel 1. Resolution (2026-06-18): keep (faithful Case-3 reproduction); loop revisits with data.
  • [x] Judge model. Resolution (2026-06-18): keep the 3-dimension composite per panel + cross-panel ranking.
  • [x] Agent names. Resolution (2026-06-18): Maverick (orchestrator) · Technical · Marketer · Academy · Support. Data-driven, not legacy — rc2's SE/SA persona structure explicitly dropped (rc2/3 = legacy POC; A-C2 = v2).
  • [x] Duplication / seed scope. Resolution (2026-06-18): profiled via scripts/setup/profile_source_dups.py. Source has distinct:url (15,179 physical → 12,064 unique; en 11,063 → 8,103 unique). Seed = language_code:en, dedup by url (keep latest lastUpdated), ≈8,100 recs; do NOT filter by environment (real content = nonprod20260220).
  • [x] Question set. Resolution (2026-06-18): RECREATE — build a new set; must include items where neural/multi-agent can differentiate.
  • [x] Old panel code. Resolution (2026-06-18): DELETE — salvage only genuinely reusable parts; clean, lean folder, no fat.
  • [x] Lab UX layout. Resolution (2026-06-18): 2×2 panel grid = scoreboard (rows retrieval, cols architecture); per-panel score+time sandwich; dynamic source pills; judge = score-click slide-in drawer, pinnable+foldable. See 2026-06-18-ux-and-judge-design.
  • [x] Judge panel content. Resolution (2026-06-18): lean — 3 always-on (composite+verdict / dimensions / comparison-deltas) + 2 expanders (how-it-answered / per-judge); grounding flagged-cards only on violation.
  • [x] Judges count + live/batch. Resolution (2026-06-18): LIVE = 1 judge (flash, thin sources → indicative); BATCH = 3 judges (pro, full sources, supermajority gate → authoritative). Per-judge layer shows in batch only.
  • [x] 3 judges vs 1 + how to show the layer. Resolution (2026-06-18): keep 3 ONLY as DIVERSE perspectives — 🔍 Skeptic (grounding) · 🛠 Practitioner (confidence/usefulness) · 🎓 Expert (breadth/depth); same rubric, synthesized (mean + supermajority). UI = one drawer template + judge selector [Synthesis] 🔍🛠🎓 and inline consensus marks on each dimension bar; click a judge → same template, that judge's scores. (Identical-sample 3 judges would collapse to 1 — diversity is the justification.)
  • [x] Blind judge. Resolution (2026-06-18): judge is blind, scored vs each panel's own retrieved sources.
  • [x] Grounding weight. Resolution (2026-06-18, CONFIRMED): ×1 (equal weight) + hard gate. ×2 would let one parameter (passed by all) dominate the composite and flatten the differentiators. Composite = gate × mean(grounding, confidence, breadth/depth).
  • [x] Leaderboard view. Resolution (2026-06-18): 4-layer batch aggregate — ① 2×2 means + verdict + grounding headline · ② dimension attribution (why each lift) · ③ per-question table → click row reuses the Arena 4-panel + judge drawer · ④ expanders (win-rate, distribution, flagged/capped audit). "Compare two runs" = Phase 2. See ux-design-v1.md v1.4.
  • [x] Does brain.expandedQuery (pre-retrieval rephrase) earn its keep? Resolution (2026-06-28): No — DROPPED from retrieval (send raw turn). No scored lift (+0.24, within noise) + a grounding hazard on bait queries (strips skeptical framing → confirms unsupported stat). NeuralSearch fires on raw NL. See 2026-06-28-drop-expandedquery-from-retrieval.
  • [x] Can Agent Studio native memory replace manual messages[] replay / a memory store? Resolution (2026-06-28): No — native memory does NOT carry context via the completions API (Backlog B probe negative across all variants). AC2 keeps stateless replay + dossier; no Redis (rc2 used it, AC2 never did). See 2026-06-28-stateless-replay-vs-redis.
  • [x] Does content-source SCOPING (cordon each agent to a source, same prompt) beat one all-source neural agent? Resolution (2026-06-28): NO — A/B/C spike vs a fair all-source baseline = +0.25 (noise). KILL as a pure-scoping play. See 2026-06-28-content-source-routing-spike-verdict.
  • [x] What is the actual per-source data depth (does the index even hold deep content)? Resolution (2026-06-28): Measured n=1,000/source: corpus = titles+summaries+facets, NO body field. Deep only in Support (median 626, 70% deep) + "Other" (median 6,503). Docs 96% stubs; Academy/Blog/Developers catalog-grade. Facets reach the model. So depth is DATA-bound, not prompt-bound.

Open

  • [ ] Port the ACS suggestions-based A2A handoff + source-pills UI + judge harness to this project ("algolia-contentsearch") — Arijit's instruction (2026-07-09): reuse the orchestration/handoff/UI logic verbatim from Algolia-Central-Spectrum, change only the prompts (content-specific) and add an analytics/real-query review (Algolia-specific, no ACS equivalent). Full replicable playbook, 17 gotchas, exact env vars, deploy checklist: ACS Porting Playbook (mirrored to repo docs/porting-playbook-from-acs.md). This target already has a live page — treat as merge-forward against this project's own multi-agent history above (Maverick→3-agent rationalization), not a blind overwrite. [Added 2026-07-09]
  • [ ] ⭐ Honed-prompt adherence + lift (ACTIVE NEXT) — push ac2-support-neural honed prompt live (snapshot first) → 4-Q adherence smoke → if good, push other 3 + run warm-baton multi-turn A/B vs ac2-allsource-neural. Does a purpose-built prompt + warm baton beat all-source WHERE data allows (Support, "Other")? [Added 2026-06-28]
  • [ ] Path 2: enrich the index with full bodies? — the real unlock for deep Technical/Academy answers is chunked full doc/lesson text, not prompt-craft. Decide after the honed-prompt run shows whether the data ceiling bites. [Added 2026-06-28]
  • [ ] Data bug: Customer Stories ~15% duplicate descriptions (Oh Polly text on 12 of 79 records) — fix at seed. [Added 2026-06-28]

  • [ ] Warm-baton engineering — RC2-style context handover (history + dossier preamble) so a routed specialist never makes the user repeat; baton = framing/retrieval context, NOT a grounding source. To build into the spike harness (and later the live coordinator if multi-agent ships). [Added 2026-06-28]

  • [ ] Delta-sync mechanics — change detection (objectID + lastUpdated/hash), scheduling (cron/end-of-day), English-only filter applied per sync. [Added 2026-06-18]

  • [ ] Delta-sync mechanics (build-time) — change detection (objectID + lastUpdated/hash), scheduling, en-only + url-dedup per sync. Designed in the plan, built in UltraCode. [Added 2026-06-18]
  • [ ] ⭐ WRITE THE PLAN (ACTIVE NEXT) — design is locked. Fuse architecture + UX + mechanics into a single GOAL + ordered build steps + acceptance criteria; hand to a FRESH UltraCode session (this session = architect, next = builder). [Added 2026-06-18]
  • [ ] Leaderboard — DESIGNED (ux-design-v1.md v1.4); 3 minor choices pending confirm (headline = mean composite · keep dimension-attribution · per-question row reuses Arena). Not blocking the plan. [Added 2026-06-18]
  • [ ] Multi-agent on keyword (Panel 3) — specialists' search needs a keyword query path; confirm vs neural (Panel 4). [Added 2026-06-18]
  • [ ] Dataset enrichment (Phase 2) — attributes/config to give specialists richer charters. [Added 2026-06-18]
  • [ ] Multi-agent on keyword (Panel 3) — specialists' search needs a keyword query path; confirm behavior vs neural (Panel 4). [Added 2026-06-18]
  • [ ] Dataset enrichment (Phase 2) — what attributes/config to add so specialists get richer "knowledge charters." [Added 2026-06-18]