Dashboard/Algolia-Central/Documentation/02-Data-Flow.md
Algolia-Central

Documentation/02-Data-Flow.md

02 — Data Flow

Last updated: 2026-04-17. Verified against lib/search/orchestrator.ts lines 45–833, plus live traces Runs 001–006.

Overview

Every request enters through one HTTP endpoint and takes one of two code paths:

  • Maverick path — igniteMaverick() — full discovery pipeline, 10+ pipeline steps
  • Specialist path — igniteSpecialist() — thin Agent Studio wrapper, ~7 pipeline steps

Both return a ReadableStream that writes Server-Sent Events (SSE) to the client. The client hooks parse events by type.

Maverick Path — Step-by-Step

Entry: api-src/search.ts:284 → igniteMaverick({query, history}, sessionId, requestId) at lib/search/orchestrator.ts:45.

Return value: ReadableStream — the streaming SSE body.

Step 1 — Session load

  • File: lib/search/orchestrator.ts:71–89
  • Reads: getSessionState(sessionId) from Redis (lib/search/redis.ts)
  • If input.history.length === 0 → resetSessionState(sessionId) then reload
  • Observed timing: 45–223ms (cold), 41–50ms (warm)
  • Emits: pipeline_step with durationMs, turn, askedSignals

Step 2 — Signal extract

  • File: lib/search/orchestrator.ts:91–104 → calls extractSignals() in signal_extractor.ts
  • What happens: single Gemini call that returns (all in one response):
  • intent (e.g., "ae")
  • onion_signals — stack, scale, role, pain
  • entities — brand, industry, product, architecture_concepts
  • search_query / keyword_query — for retrieval
  • discovery_question — next best pivot, pre-written by Gemini
  • filters — source-type filters (optional)
  • Why one call: consolidated from an earlier 2-call design. The discovery question generation and signal extraction used to be separate — now merged.
  • Observed timing: ~1800–2000ms (CRITICAL path; 27% of Maverick turn time)
  • Emits: pipeline_step signal_extract with intent, hasDiscoveryQ, entities count
  • Emits: pulse SIGNAL_EXTRACTION
  • Performance target: P-1 parallelize with retrieval (Step 4 of audit)

Step 2.5 — Intent guardrail

  • File: orchestrator.ts:106–113 → guardIntent(query, intent) in domain_guardrails.ts
  • Purpose: if the LLM returned a weird intent, guardrail overrides to a safe classification.

Step 2.7 — Merge signals into session state

  • File: orchestrator.ts:115–128
  • Pattern: brand: signalResult.entities.brand || sessionState.brand (the || operator)
  • BUG (F-026): this merge never accepts correction — a locked signal stays locked even if user contradicted it. The || always prefers old value if new is null. Fix = Step 3 pushback detector.
  • Writes merged state back to Redis.

Step 3 — Discovery analysis

  • File: orchestrator.ts:130–151 → analyzeDiscoveryState() in discovery_analyzer.ts
  • Returns: {isQualified, isStrictDiscovery, isFatigueReached, availableFields, showDiscoveryUI}
  • Qualification rule: requires lockedSignals >= 4 AND other conditions. F-035 observed: at turn 4 with 8 locked signals, only then isQualified=true. Demo perception = too slow.
  • Observed timing: ~130–150ms
  • Emits: pipeline_step discovery_analysis + event: signals (8 signals + lockedSignals count)

Step 4 — Retrieval

  • File: orchestrator.ts:172–244 → calls fetchGoldenMap() then orchestrateRetrieval() in retrieval_orchestrator.ts
  • Sequence: 1. fetchGoldenMap() — loads the Atlas index (classification map). Cached 60s. 2. orchestrateRetrieval() — runs search on NeuralSearch index with:
    • searchQuery (semantic)
    • keywordQuery (lexical fallback)
    • filters (signal-derived — industry, product, etc.)
    • sourceTypeFilters — defaults to marketing/blog/customer_story/guide/news/doc/changelog/video if not set. Hardcoded at line 204 (F-027 cleanup target — should move to persona config).
    • matchCount: 30 3. injectPricingIfNeeded() — adds pricing chunks if query mentions pricing/scale concepts
  • Returns chunks[], atlasMatch, strategy (filtered/relaxed/fallback), queryID
  • Observed timing:
  • filtered strategy: ~700ms
  • relaxed strategy: ~1100–1600ms
  • Emits: event: sources with all chunks (truncated to 200 chars each), queryID, indexName
  • Emits: pipeline_step retrieval with chunks, strategy, atlasMatch, sourceTypes
  • Performance targets: P-2 (skip retrieval on meta queries), P-3 (cap at 10 chunks)

Step 4b — Track view (Algolia Insights)

  • File: orchestrator.ts:246–264 → trackView() in insights.ts
  • Fires Insights event with queryID + objectIds[] — tells Algolia which results were served.
  • Non-blocking (fire-and-forget).

Step 5 — Build prompt

  • File: orchestrator.ts:266–298
  • Sequence: 1. generateDossierSummary() → terse bullets of current signals (213–271 char range observed) 2. Emits event: maverick_header with dossier (instant UI update) 3. formatHitsForLLM(chunks) — formats retrieved chunks for prompt injection 4. If not qualified + has discovery_question → build discoveryInstructions = End your response with: <discovery_pivot signal="X">question</discovery_pivot> 5. buildMaverickPromptWithPersona(dossier, sources, discoveryInstructions, turnCount, strictMode) — assembles the full system prompt from prompts/maverick.ts
  • Observed timing: ~5ms (fast — all in-memory)
  • Observed prompt size: 16KB–157KB. F-006 BUG: non-monotonic spike (went 16K→157K→43K across turns). Root cause unknown. Fix = Step 4c.
  • Emits: pipeline_step prompt_build

Step 6 — LLM stream

  • File: orchestrator.ts:320–380
  • Calls: provider.startChat() → streamMaverickResponse() in stream_processor.ts:60
  • Critical side effect: stream_processor.streamMaverickResponse creates an InlineStreamAuditor(sources, products) at line 82 and runs EVERY paragraph through auditor.auditParagraph() before emitting to client. This is what strips hallucinated content from Maverick. See 08-Audit-Pipeline.
  • SSE emission pattern: per-chunk from Gemini → buffer paragraphs → audit → flush to client as event: chunk with {content: "..."}
  • Observed timing: 2400–3300ms (40% of turn time)
  • Emits: pipeline_step llm_stream_start, pipeline_step llm_stream_end, event: chunk per paragraph (many)
  • Performance target: P-4 trim prompt 16KB → ≤8KB

Step 6.5 — Turn snapshot

  • File: orchestrator.ts:385–391
  • pipe.turnSnapshot({persona:'maverick', question, answer, history, linkAuditResults})
  • Purpose: developer-visible complete turn record in console + SSE. 15KB payload size observed — P-7 cleanup target.
  • Emits: event: turn_snapshot

Step 7 — Quality + Audit events

  • File: orchestrator.ts:393–415
  • Computes word count, citation count, depth score, and emits:
  • event: quality — {score, passed, metrics, regenerationAttempts}
  • event: audit — {confidenceScore, links, tone, grounding} (derived from inline auditor result)
  • Note: tone.compliant and grounding are placeholders after Step 2 — those theater methods were deleted. Values are pass-through estimates.

Step 8 — Discovery + Handoff emission

  • File: orchestrator.ts:418–503
  • Discovery question emit:
  • Uses signalResult.discovery_question directly (NOT extracted from Maverick's output — that was an earlier design)
  • Calls addAskedSignal(sessionId, signal) to track asked signals
  • Emits: event: discovery with {signal, question, lockedSignals, showUI}
  • Handoff detection:
  • Regex match on <specialist_handoff specialist="ELENA|BRUNO"> in Maverick's full response
  • Keyword override: STRONG_BRUNO_SIGNALS = ['architect', 'architecture', 'infrastructure', 'scale planning', 'governance'] forces Bruno routing regardless of LLM pick
  • Persists pendingHandoff: {expandedQuestion, target} to session state
  • Emits: event: handoff_proposal with {target, expandedQuestion, transition_sentence, collectedSignals, requiresConsent:true}
  • Emits: pipeline_step handoff_detected
  • Three-layer routing is a known Frankenstein pattern (F-027 cleanup target — Step 5).

Step 9 — Post-generation metadata

  • File: orchestrator.ts:506–530 → processPostGenerationMetadata() in metadata_manager.ts
  • Produces: uniqueSources[], wordCount

Step 10 — Increment turn count

  • updateSessionState(sessionId, {turn_count: +1})

Step 11 — Final event

  • File: orchestrator.ts:537–563 → constructFinalEvent() in metadata_manager.ts
  • Emits: event: final — the complete turn payload (content, sources, signals, discoveryResult, searchResults, queryID, indexName)
  • Emits: pipeline_step final_event_emitted
  • Emits: pipe.finish() (closes the pipeline_total trace)
  • Observed timing: this step takes ~300–390ms (Maverick). Compare with Elena handshake: 44ms. This gap is P-6 target.

Step 12 — Telemetry

  • File: orchestrator.ts:566–592 → persistSessionTelemetry() in telemetry.ts
  • Writes TelemetryTurn to Redis with status handoff_success or in_progress

Step 13 — Close

  • controller.close() — stream ends. Client sees done: true.

Specialist Path — Step-by-Step

Entry: api-src/search.ts:274 OR api-src/search.ts:203 (consent branch) → igniteSpecialist({query, history, persona, trigger, expandedQuestion}, sessionId, requestId) at orchestrator.ts:611.

Two triggers: - trigger='handshake' — the post-handoff summary-back-to-me turn (short, ~3-5s) - trigger='execute' — the deep dive after user consents (~33s — F-022)

Step 1 — Persona validate

  • validatePersona(input.persona) — guardrail against injection.

Step 2 — Session load

  • getSessionState(sessionId) — same Redis read as Maverick.

Step 3 — Build Agent Studio message

  • File: orchestrator.ts:789–833 → buildAgentStudioMessage(params)
  • Handshake trigger (line 800–803): constructs the canned protocol: "EXECUTE HANDSHAKE PROTOCOL: You are ${specialistName}, Algolia's ${role}. Start by thanking Maverick by name... Then introduce yourself... Then summarize what you understand... Finally ask if there's anything else to factor in before you go deep. Do not answer any technical questions yet."
  • Execute/consent trigger (line 804–805): uses expandedQuestion || pendingHandoff.expandedQuestion — the question carried over from Maverick's qualification.
  • Default fallback (line 806–807): raw query.
  • Appends <extracted_entities> block listing signal values (industry, brand, product, stack, scale, role, pain, architecture_concepts)
  • Output: string wrapped in <user_question> + <extracted_entities> tags

Step 4 — Agent Studio call

  • File: orchestrator.ts:663–676 → callAgentStudio() in lib/agent-studio/client.ts
  • Resolves agentId via getAgentId(persona):
  • elena → f029acbb-a7a0-43b1-9a85-a14ef3907cd3
  • bruno → facb549e-8f27-47e9-9e42-e20032b0f1a1
  • POSTs to Agent Studio with {agentId, message, history}
  • Returns {status, ok, body} where body is a ReadableStream (remote SSE)
  • Observed timing: ~1500–1800ms for initial response (status 200)
  • Emits: pipeline_step agent_studio_call_start, pipeline_step agent_studio_response

Step 5 — Adapt Agent Studio stream

  • File: orchestrator.ts:678–695 → adaptAgentStudioStream() in lib/agent-studio/stream-adapter.ts
  • Parses Agent Studio's XML-tagged markdown SSE, emits:
  • event: chunk per parsed chunk {content: "..."}
  • event: sources at end with the tools/citations Agent Studio used
  • Uses StatefulStreamStripper internally to remove Agent Studio's XML control tags
  • Returns {fullContent, sources}
  • 🚨 NO PARAGRAPH AUDIT HAPPENS HERE. This is F-038. The Agent Studio stream is adapted but not audited for hallucination, fabricated metrics, or dangling attributions. Only HEAD checks happen — see Step 6.
  • Observed timing:
  • Handshake: ~3-5s
  • Execute: ~30-35s (F-022 — Agent Studio-side)
  • Emits: pipeline_step specialist_stream_end
  • File: orchestrator.ts:697–712
  • Extracts all markdown links via regex /\[([^\]]*)\]\((https?:\/\/[^)]+)\)/g
  • Parallel HEAD requests with 4s timeout
  • Returns linkAuditResults = [{url, status:'verified', httpStatus, healthy}]
  • Does NOT strip dead links. Just records status. Client displays whatever was streamed.
  • Crucially: does not verify that the linked title matches content, does not verify that cited customer metrics match sources. Only verifies "the URL returns HTTP 2xx."

Step 7 — Turn snapshot + Quality

  • Same pattern as Maverick (pipe.turnSnapshot + event: quality)

Step 8 — Telemetry

  • persistSessionTelemetry() — records the specialist turn.

Step 9 — Final event

  • File: orchestrator.ts:746–757
  • Payload: {status:'complete', persona, content, specialist:{artifact:{type:'MARKDOWN', content}}, sources, usage}
  • Emits: event: final
  • Emits: pipeline_step final_event_emitted, pipe.finish()
  • Observed timing: 44ms for Elena handshake (the gold standard to aim for in P-6)

Step 10 — Close

  • controller.close() — stream ends.

SSE Event Catalog (both paths)

Event When emitted Consumer Payload
pulse At phase transitions Frontend status bar {status, message}
pipeline_step End of each backend step Browser console trace {step, durationMs, details}
signals After discovery_analysis Dossier component All 8 signals + lockedSignals + isQualified
sources After retrieval Sources panel {sources, queryID, indexName}
maverick_header Before LLM stream Yellow header {header, requestId}
chunk Per paragraph (streaming) Message body {content: "..."}
discovery After Maverick turn Discovery pivot UI {signal, question, lockedSignals, showUI}
handoff_proposal Handoff detected "Bring in expert" button {target, expandedQuestion, collectedSignals, requiresConsent}
quality After generation Quality badges {score, passed, metrics}
audit After generation Audit panel {confidenceScore, links, tone, grounding}
turn_snapshot End of turn Console trace Full turn record (15KB)
final End of stream Triggers finalize {content, sources, signals, ...}
error On caught exception Error toast {error: message}

Timings Summary (from Runs 001–006)

Maverick (averaged)

Step Time %
session_load 41–50ms (warm) <1%
signal_extract 1800–2000ms 27%
discovery_analysis 140ms 2%
retrieval 700–1600ms 10–24%
prompt_build 5ms <1%
llm_stream 2400–3300ms 40%
handoff_detected 300ms (qualification turn) 4%
final_event_emitted 300–390ms 6%
TOTAL ~6500ms target ≤4000ms

Specialist

Step Time Notes
session_load 44–53ms
agent_studio_call_start ~55ms just initiating
agent_studio_response 1500–1800ms remote first byte
specialist_stream_end (handshake) ~3500ms total Agent Studio stream
specialist_stream_end (execute) ~34500ms F-022 — out of scope
final_event_emitted 44ms (handshake) / 4186ms (execute)

Cross-references

  • 03-Files-Reference — which file owns each step
  • 04-Functions-Reference — function-by-function signatures
  • 08-Audit-Pipeline — what the InlineStreamAuditor does (and where F-038 lives)
  • 10-Known-Issues — F-026, F-035, F-038 all show up in this flow