Dev Log — Answer-Quality Lab

2026-06-28 — Content-source routing spike → KILL (scoping), data ceiling found, honed-prompt phase next

Status: Done (spike + data-hardening); honed-prompt run IN PROGRESS (next: push Support + smoke) What happened: Built an A/B/C spike to test the 2026-06-18 assumption that content-source specialists beat one all-source agent. First run (baseline = 6-source incumbent Maverick) = +0.76 "multi-agent wins." Smelled the confound; re-ran with a FAIR all-source baseline (ac2-allsource-neural, same MULTI_NEURAL index, no source filter) → +0.25 (noise) = KILL. The swing was 100% baseline-blindness (incumbent couldn't see Academy/Support). Then data-hardened (n=1,000/source): corpus is titles+summaries+facets, NOT full bodies — only Support (median 626 chars, 70% deep) and "Other" (median 6,503) carry depth; Documentation is 96% stubs; Academy/Blog/Developers catalog-grade. Corrected two of my own n=2 errors (Customer Stories 85% usable not corrupted; "Other" deep not 1-line). Verified facets reach the model via a live tool-result dump. Wrote data-realistic honed specialist prompts (Path 1: hone within data limits) + a warm-baton (RC2-style context handover) design. Key decisions: 2026-06-28-content-source-routing-spike-verdict (supersedes-in-part 2026-06-18-content-source-multi-agent). Tests written: 18 unit tests (sourceRouting labels/classifier/routingAgg); full suite 127 green; tsc clean. Next: push ac2-support-neural honed prompt live (snapshot first) → 4-question adherence smoke → if good, push other 3 + run warm-baton multi-turn A/B vs ac2-allsource-neural. Then decide Path 2 (enrich index with full bodies) if the data ceiling bites.

2026-06-18 — Major refactor: design & intake

Status: Pivoted (design locked; build not started) What happened: Worked through a ground-up redesign. Decided the 2×2 four-panel lab, the move to FLAGSHIP_Accelerator_Program_APP with four source-identical indices, and the content-source multi-agent design. Validated the source facet against the live Visibility index (12,064 records; en/fr/de) — English slice sums to exactly the screenshot's 8,019. Read rc2-algolia and rc3-phoenix in full to inform the multi-agent decision: both are sales-discovery orchestrators on a two-index Atlas+Ledger schema; rc3 is a fragile hybrid (~2,355 lines backend, P0 defects). Conclusion: don't port wholesale; reuse the orchestrator→specialist pattern only. Key decisions: Five ADRs — 2026-06-18-pivot-to-2x2-four-panel-lab, 2026-06-18-flagship-app-and-four-indices, 2026-06-18-content-source-multi-agent, 2026-06-18-judge-3dim-per-panel, 2026-06-18-daily-delta-sync. Tests written: none (design phase). Next: Resolve open questions (Technical-agent name, dupe profiling, sync mechanics, question set), finalize the goal, then execute the build via UltraCode orchestration.

2026-06-18 — Dedup profiling + process roadmap locked

Status: Done (analysis); design phase next What happened: Profiled the live source for duplication (scripts/setup/profile_source_dups.py, browse via the write key, read-only). Found distinct:url masking dupes: 15,179 physical → 12,064 unique (en: 11,063 → 8,103). Locked the seed rule (en + url-dedup, keep latest, ≈8,100; no environment filter). Confirmed the build process flow with Arijit: Brief → Experience/UX design → Mechanics → Plan (goal + steps) → UltraCode in fresh context (this session = architect, next = builder). Key decisions: Question set → recreate. Old code → delete (lean, no fat). Roadmap captured in repo brief §10. Next: Experience/UX design pass (multi-layered presentation) — the last design piece before writing the plan.

2026-06-18 — Lab UX + judge panel designed

Status: Done (design); Scorecard view remaining What happened: Designed the full lab experience (repo docs/refactor/ux-design-v1.md; ADR 2026-06-18-ux-and-judge-design). 2×2 panel grid doubles as the scoreboard; clicking a panel's score slides in a pinnable/foldable judge drawer for that panel; dynamic source pills per panel. Judge panel made lean (3 always-on blocks + 2 expanders; grounding flagged-claim cards only on violation). Judging tiers decided: 1 judge LIVE (flash, indicative) / 3 judges BATCH (pro, full sources, supermajority grounding gate, authoritative) — the per-judge layer surfaces only in batch. Recommended grounding as a GATE at ×1 (not ×2) to avoid muting the breadth/confidence signal that proves neural/multi — pending Arijit's confirm. Key decisions: ADR 2026-06-18-ux-and-judge-design. Blind judge vs each panel's own sources (adopted). Next: design the Scorecard view (batch aggregate drill-down), confirm grounding ×1, then write the plan → UltraCode.

2026-06-19 (later) — Senior-UX redesign + perf/stream diagnosis + Fix-and-Learn loop

What happened: Ran a senior-UX pass over the 2×2 lab across the three personas (developers/execs/merchandisers) and shipped polish fixes, verified live (desktop + mobile): (1) unified question bar — one top bar to ask AND see the question, Sample-questions beside it; (2) removed the winner-verdict headline strip → compact "Best answer" tag on the winning tile; (3) viewport-fit layout — 2×2 fills the screen, each tile scrolls internally, page never scrolls; (4) alive waiting state — ticking timer + per-panel hint; (5) neural badge honesty — "enabling" only when the backend explicitly reports not-live, else plain "Neural"; (6) mobile fixes (stacked tiles size to content, question bar wraps). tsc clean; browser-verified every state. Diagnoses (backend, blocked on SSH/redeploy): "No streaming" = answer.ts awaits the full Agent Studio completion then emits one SSE block (firstTokenMs===totalMs); upstream streams, the runner buffers — fix = forward tokens as SSE deltas. "Suddenly slow" = backend (multi-agent chains + neural + likely Gemini throttling); frontend cannot affect generation time (network trace shows all calls hit the VPS). New mechanism — Fix-and-Learn loop (always-on): every resolved issue → Symptom/Root-cause/Fix/Evidence/Prevention, recorded to memory + vault. SOP: Standards/FixAndLearnLoopSOP.md + ~/.claude/docs/fix-and-learn-loop.md (wired into CLAUDE.md Cardinal Rule 6). Project arm: repo docs/sop/lessons-log.md (seeded with ~12 issues from this 2×2 effort). ADR: 2026-06-19-ux-redesign-and-fix-learn-loop. Working-dir note: Dropbox renamed the project folder; canonical copy is now ~/AI-Development-OLD/RAG/Algolia-Central2. Committed + pushed: origin/refactor/2x2-answer-quality-lab @ 0b1e872 (commits eb5d912 UI, 0b1e872 docs). Next: VPS backend redeploy (SSH allow-rule) → token streaming + /health neural field + source facets; flip neural; authoritative batch cli pipeline.

2026-06-19 (~11am) — Discovery rebuild brainstormed + locked; VPS redeployed

Brainstorm (locked): the one-shot 2×2 lab becomes a two-directional peel-the-onion discovery engine (learning from RC2/RC3). Spine: purpose=BOTH (prove the lift + demo the experience); discovery on ALL 4 panels (intent→entity→expand→answer→follow-up); ONE shared conversation thread, each panel proposes its own follow-up; multi-agent merit = the inline generic→specialist HANDOFF on P2/P4 (measured, not assumed); build = LAYER onto the existing engine (port RC2/3 logic), PHASED — P1 discovery → P2 handoff+verification → P3 discovery-aware scoring. NEXT = write the Phase 1 spec (docs/superpowers/specs/2026-06-19-discovery-on-all-4-panels-design.md) → writing-plans. RC2/RC3 research done; honest finding: NO A/B proof multi-agent beats single on quality (value = experience + verification). Shipped + DEPLOYED: senior-UX pass (viewport-fit, single Best-answer winner, 3 judge personalities shown, traffic-light dots, draggable docked marker, synced timers, multi-category clickable source pills); scoring rework — grounding floor fires only on CONTRADICTION not un-verifiability (fixes all-3.0). VPS redeployed ac2-lab-backend@0db66dc; /health neural field live; neural ON both indices. Pipeline measured: bottleneck = synchronous follow-up LLM call (8–15s), not retrieval. Open: OpenAI blocked on billing (429 insufficient_quota); token streaming unimplemented (agentRunner.ts:173 await res.text()); Vercel frontend not yet redeployed.

2026-06-28 — Backlog A (expandedQuery drop) + Backlog B (native memory) — both closed

Status: Done — shipped to main. What happened: - A — dropped brain.expandedQuery from retrieval (shipped). Built a scored A/B harness (lab/server/src/experiments/expandedQueryAb.ts + pure, unit-tested expandedQueryAgg.ts). Gate 1: raw NL queries with getRankingInfo fire NeuralSearch's semantic layer (non-zero neural/semanticScore) — no rewrite needed. Gate 2 (32 Q): mean Δ +0.24 (noise) AND a grounding hazard — the rephrase strips skeptical framing on bait queries (Q7.5: raw refuses 10.0 → rewrite confirms unsupported stat, gate-tripped 1.67). Fix: orchestrate.ts sends the raw user turn (both agent calls) + TDD test. ADR 2026-06-28-drop-expandedquery-from-retrieval. - B — native memory does NOT carry (NEGATIVE). Probe nativeMemoryProbe.ts: same conversation id, "what was my previous question?", no replay → "you have not asked me a previous question." Negative across client-id, +5s delay + messages[].id, and server-returned id (only alg_msg_* returned, no alg_cnv_*). Corrected the earlier findings-doc claim (its pronoun test was a false positive). AC2 keeps manual messages[] replay; no Redis. ADR 2026-06-28-stateless-replay-vs-redis (repo ADR-002). Key decisions: ADRs 2026-06-28-drop-expandedquery-from-retrieval, 2026-06-28-stateless-replay-vs-redis. Tests written: orchestrate.test.ts raw-turn assertion; expandedQueryAgg.test.ts (7). Suite 109 green, tsc clean. Lessons (Fix-and-Learn, repo docs/sop/lessons-log.md): (1) a pre-retrieval rephrase rewrites intent not just vocabulary → defeats refusal on bait queries; test the bait class + read rows when an A/B mean is within noise. (2) To test memory, ask something impossible to answer without the prior turn (recall the previous question) — a pronoun the model can guess from domain priors is a false positive; re-run a "resolved" conflict's own flagged open test before building on it. Committed + pushed: main dc03bc7 (A) + 03e9184 (B + ADR-002). Next: only P2b remains (human-rank judge calibration — needs Arijit, ~20 min). P3 (RC2 gym) skipped.