Algolia-Central

Documentation/06-Retrieval-Architecture.md

06 — Retrieval Architecture

Last updated: 2026-04-17. Verified against lib/search/retrieval_orchestrator.ts.

Two-Index Model

Index	Purpose	How it's queried
Atlas	Classification map / Golden Map	Loaded once per request via `fetchGoldenMap()`, cached 60s in-memory (`GLOBAL_MAP_CACHE`). Used to bias retrieval, not return content directly.
NeuralSearch	Knowledge store (blogs, docs, customer stories, pricing, support)	Searched via `algoliaClient.searchSingleIndex()` with signal-derived filters. This is the primary content source.

Both indexes stay. Neither is vector-based locally (Algolia's own NeuralSearch handles semantic matching remotely).

RetrievalOptions (input contract)

From retrieval_orchestrator.ts:21:

interface RetrievalOptions {
  query: string;                  // raw user query
  searchQuery?: string;           // LLM-expanded semantic query (from signal_extractor)
  keywordQuery?: string;          // LLM-derived keyword fallback
  filters?: {
    industry?, product?, brand?,
    feature?, solution?,
    stack?, scale?, role?, pain?
  };
  newsBoost?: boolean;
  matchCount?: number;            // default 30
  mapData?: any[];                // the Atlas output
  sourceTypeFilters?: string[];   // ['blog', 'doc', 'customer_story', ...]
  persona?: string;
  tracer?: any;
}

RetrievalResult (output contract)

interface RetrievalResult {
  chunks: any[];                  // array of mapped hits
  totalRetrieved: number;
  strategy: 'filtered' | 'relaxed' | 'fallback';
  atlasMatch?: any;               // the Atlas entry that biased the search
  queryID?: string;               // Algolia queryID for Insights
}

Strategy Selection (3-tier fallback)

Strategy	When used	Observed timing
filtered	User signals give a clean filter profile (industry + product + etc.)	~700ms
relaxed	Filtered returns too few chunks — drop some filters, retry	1100–1600ms
fallback	Both above return nothing — broad search	Rare

The selection logic lives in orchestrateRetrieval() (line 113). [EXPAND — document the exact fallback triggers on a later pass.]

Atlas Integration

`fetchGoldenMap()` (line 570)

Runs searchSingleIndex() against the Atlas index with empty query (returns all entries, paginated).
Cached in GLOBAL_MAP_CACHE for 60 seconds — same container reuses across requests.
Returns any[] — each entry contains vertical info, display_name, product_key, etc.
Called by: igniteMaverick (step 8), plus classifyLink() uses mapData to validate customer URL claims.

Atlas match ("golden")

After retrieval, a specific Atlas entry may match the user's query. This becomes result.atlasMatch — seen in trace as e.g. atlasMatch="AI Relevance" or atlasMatch="Ecommerce". Used by: - Maverick's dossier (to show "AE context: Ecommerce") - Frontend for vertical badge display

The Algolia Filter String

Built by buildAlgoliaFilters(filters, sourceTypeFilters) (internal, line 59). Pattern:

(source_type:"blog" OR source_type:"doc" OR source_type:"customer_story" OR ...) AND NOT is404:true

Note the comment on line 73: industry/product hard-gate filters were REMOVED to support multi-intent queries. Only source_type + 404 filter remain.

Source Type Filters (defaults)

From orchestrator.ts:204, when signalResult.filters isn't set:

['marketing', 'blog', 'customer_story', 'guide', 'news', 'doc', 'changelog', 'video']

Notably excluded from Maverick (AE-facing): - 'api-reference' / 'support' / 'code-snippet' — Maverick is a value-seller, not a technical doc-reader. Per Knowledge Charter: "Disallowed: Deep Technical APIs, Support Tickets, Code Snippets" (see comment line 201).

Known issue (F-027 cleanup target): this default array is hardcoded in orchestrator.ts line 204. Should move to config/system/PERSONAS.md as per-persona config. Step 5.

Chunk Mapping: `mapAlgoliaHitToChunk(hit)` (line 88)

Algolia hit → chunk interface:

Chunk field	Source in hit
`chunk_id`	`hit.objectID`
`chunk_text`	`hit.content \\|\\| hit.summary \\|\\| hit.abstract \\|\\| hit.description \\|\\| '...'`
`doc_title`	`hit.title \\|\\| 'Untitled'`
`doc_url`	`hit.url`
`doc_summary`	`hit.summary \\|\\| hit.abstract \\|\\| hit.description`
`source_type`	`hit.source_type \\|\\| hit.source \\|\\| 'doc'`
`industry_tag`	`hit.facets?.facet1`
`product_tag`	`hit.category`
`feature_tag`	`hit.tags?.[0]`
`vector_score`, `keyword_score`	always 0 (NeuralSearch doesn't expose these directly)
`final_score`	`hit._rankingInfo?.userScore \\|\\| 0`

Note: No publishedAt / date field currently mapped — F-039 blocker. Step 4 must verify whether Algolia records have such an attribute; if yes, map it here. If no, backfill via scraping blog HTML <time> tags.

Post-Retrieval Processing

`injectPricingIfNeeded(query, concepts, chunks)` (line 532)

Conditional: if query or architecture_concepts mention pricing/scale/tier, pricing chunks are appended to the result set. Catches queries the semantic search might have missed.

In orchestrator: deduplication

After retrieval, orchestrator.ts:237–242 maps chunks into a sources-event payload, truncating chunk_text to 200 chars for the wire.

Insights Tracking (post-retrieval)

orchestrator.ts:246–264 — fires trackView() with: - index — live index name from getIndexName() - eventName — 'Maverick Result View' - userToken — sessionId (or 'anonymous') - queryId — from RetrievalResult.queryID - objectIds — top 20 chunk IDs

Non-blocking. Feeds Algolia's Insights dashboard for relevance tuning.

Known Dead Code in This Area (Step 4 cleanup targets)

Item	Location	Why dead
`atlas_guided` strategy	line 47 union	Declared but never returned
`optionalFilters` boost logic	line 268–312	Uses `<score=N>` syntax Algolia v5 ignores (comment line 82 confirms)
`VERTICAL_SYNONYM_MAP`	line 273–308	2× duplicate boosts; no trace evidence
Reverse Atlas lookup	line 393–427	Rarely fires; would log "Reverse Atlas lookup succeeded"
Unused `getCache, setCache` imports	line 17 (removed in Step 1)	✅ done

Observed Strategy Distribution (Runs 001–006, 6 turns)

Strategy	Occurrences	Avg chunks	Avg time
`filtered`	3 turns	29–34	~900ms
`relaxed`	3 turns	18–25	~1050ms
`fallback`	0	—	—

P-3 target: cap chunks at 10. Currently returns 16–34. Downstream LLM only uses ~10 of them anyway.

Retrieval-Level Known Issues

F#	Issue	Fix
F-039	No recency signal → 2022–2024 blog posts surfaced in 2026	Step 4 — verify + add publishedAt ranking, OR 18-month filter
P-2	Meta/conversational queries trigger full retrieval (~1500ms waste)	Step 4a — pushback/meta classifier before retrieval
P-3	Returns 16–34 chunks when only ~10 used	Step 4d — cap at 10
~~F-044~~	~~Short chip turns lost accumulated context → off-topic retrieval~~	✅ SHIPPED 2026-04-17 — see Context Inheritance section

Context Inheritance (F-044 — SHIPPED 2026-04-17)

Short chip turns (≤20 chars, e.g. "developer") carried no problem content into retrieval. The Brain echoed the chip into search_query, Algolia retrieved whatever the chip matched (often unrelated doc pages), and Maverick's value-sell collapsed into a doc-bot explanation.

Fix lives in lib/search/retrieval_context.ts: - shouldInheritContext(currentQuery, history) — inherits when trimmed query ≤ 20 chars AND history length ≥ 2. - buildAccumulatedQuery(history, sessionState) — concatenates first user turn + every locked signal (stack/scale/role/pain/brand/industry/product + architecture_concepts) into one keyword-rich query.

Wired in orchestrator.igniteMaverick before orchestrateRetrieval. When inheritance fires, logs F-044 retrieval context inheritance active with the synthesized query.

Related fixes in the same step: - signal_extractor.ts — removed .slice(-6) truncation. Brain now sees full history. - prompts/brain.ts — STEP 2 SHORT-TURN RULE added (explicit instruction to build search_query from accumulated context on short inputs).

Source Filter Charter (F-044 Layer 2 — SHIPPED 2026-04-17)

Previously, orchestrator.ts:231 let signalResult.filters (from the Brain) override Maverick's default AE source mix. On intent=technical this flipped Maverick to doc-only, killing customer stories and blogs.

Per Q3-Brain-Filter-Investigation.md, the override was architecture-drift: signalResult.filters was designed when Brain output drove a single answer path across all three personas. After specialists moved to Agent Studio (which uses its own tools), the field only fires against Maverick — the wrong target.

Fix: Maverick always receives the fixed AE source mix ['marketing', 'blog', 'customer_story', 'guide', 'news', 'doc', 'changelog', 'video']. signalResult.filters remains in the Brain output and in metadata_manager.filters_applied telemetry for observability.

Cross-references

02-Data-Flow — retrieval step in pipeline
04-Functions-Reference — orchestrateRetrieval, fetchGoldenMap
10-Known-Issues — F-039 + P-2 + P-3