Dashboard/Algolia-Central (Second-Brain)/v2/wiki/decisions/2026-07-01-adr-entity-boost-native-not-atlas.md
Algolia-Central (Second-Brain)

v2/wiki/decisions/2026-07-01-adr-entity-boost-native-not-atlas.md

ADR: Conversation-entity retrieval biasing — native optionalFilters, NOT an Atlas-style second index

Status: Accepted (Arijit + Claude, 2026-07-01) Context axis: retrieval quality / coordinator → specialist retrieval. Related Claude memory notes: project-content-source-routing-killed, feedback-query-rephrase-strips-skeptical-framing, feedback-ab-baseline-same-information-set.

Context

RC2 used an Atlas pattern: the coordinator (Maverick) extracted entities from the customer conversation → queried a separate Atlas Algolia index → got a curated "map" of critical/related terms → used them as boosts + query expansion for the main ledger retrieval. Two retrievals + custom orchestration + a curated index to maintain.

Why Atlas existed in RC2: the ledger was keyword/vector — it did not understand language. Atlas hand-fed the retriever the semantic expansion + entity disambiguation it couldn't do itself. Atlas was a crutch for a non-neural retriever.

Decision

Adopt the insight (bias retrieval toward the conversation's critical entities); reject rebuilding Atlas. Implement natively, fused into the single NeuralSearch call:

RC2 Atlas step AC2 native Runtime cost
extract entities from convo coordinator already does this (baton dossier) already paid
Atlas lookup → curated map not needed — the taxonomy already lives on records as facets (industry_tag/product_tag/feature_tag/solution_tag, facet1 industry, facet6 features) 0
boost ledger query w/ map optionalFilters on those facets (ranking boost, not a filter) inside the ONE neural query 0 extra hops
curated expansion / disambiguation Query Rules + AI Synonyms (curated once, native) 0 runtime hops

Rationale

  1. NeuralSearch already does semantic expansion — Atlas re-implements a native capability. (Same lesson that killed content-source routing: pre-retrieval intervention gave ~0 lift over the all-source neural agent.)
  2. The map is redundant — the entity→concept taxonomy is already encoded as facets on every record; no separate index to build/keep fresh.
  3. Least hops / lowest latency — optionalFilters are applied server-side in the same request. Atlas = +1 round-trip + orchestration code + index maintenance. Native is faster AND simpler.
  4. Grounding preserved — optionalFilters boost RANKING, not the corpus and not the query text. Every claim still traces to a retrieved hit (110% grounded).
  5. Avoids the query-rewrite hazard — injecting boost terms into query TEXT rewrites intent and already defeated refusal on bait queries. Facet boosts don't touch intent → safer.
  6. No tech debt — nothing custom to own; all levers are native Algolia (NeuralSearch + optionalFilters + Rules + Synonyms).

Consequences

  • Coordinator passes extracted entities as optionalFilters (mapped to facet values) into the specialist's neural search.
  • Curate a small Rules + AI Synonyms set for domain expansion/disambiguation (once, in-dashboard or via API).
  • No Atlas index, no second retrieval, no boost-term query rewrite.

Validation (measure before trusting — do NOT assume lift)

A/B on the specialist rubric: Panel A = plain NeuralSearch vs Panel B = NeuralSearch + entity optionalFilters, same corpus, same questions (fair-baseline invariant). Prior: small-but-real lift on ambiguous / persona / cross-entity queries (where boosting the customer's industry/product disambiguates), ~nil on clear single-intent queries. Trust the delta only past judge noise; judge calibration (P2b) still gates trust-mode conclusions.

Backlog item (eval)

AB-ENTITY-BOOST: neural-alone vs neural+optionalFilters(from extracted entities), 4-dim rubric, rounds≥3, split clean-vs-ambiguous. Runs after crawl-enrichment lands (richer bodies may change the picture) and after P2b.