Dashboard/Algolia-Central/Documentation/08-Audit-Pipeline.md
Algolia-Central

Documentation/08-Audit-Pipeline.md

08 — Audit Pipeline

Last updated: 2026-04-17. Verified against lib/search/audit.ts + content_auditor.ts post-Step-2.

The audit pipeline is the zero-hallucination guardrail. This chapter documents what fires, what was deleted as theater in Step 2, and the F-038 gap that this pipeline currently does NOT cover.

Two auditor classes

Both live in lib/search/content_auditor.ts:

Class Fires when Scope
InlineStreamAuditor During Gemini streaming, per paragraph Full hallucination guard — links, grounding, customer claims
ContentAuditor After stream completes (post-stream) Post-Step-2: link HEAD checks only

After Step 2, ContentAuditor is a much leaner class — tone/persona/grounding/confidence methods were deleted as theater (logged but never enforced).

InlineStreamAuditor.auditParagraph(text): string

File: content_auditor.ts:426

Called by: streamMaverickResponse in stream_processor.ts:90 — once per paragraph as Gemini streams.

Pipeline inside:

htmlAnchorsToMarkdown(text)
  ↓
auditLinks(text)               ← uses classifyLink per link
  ↓
upgradeGenericLinkLabels(text) ← F-031 lives here
  ↓
stripDanglingAttributions(text)
  ↓
auditGrounding(text)           ← hallucinated customer sentences
  ↓
return cleaned text

Side effects: console.log + buffered trace lines ([AUDIT] ══ paragraph in/out ══, [AUDIT] KEEP|STRIP|REPLACE | ...) visible in live trace.

Trace draining: the stream processor calls auditor.drainTrace() periodically to collect lines and emit them via event: audit_trace.

What each sub-step does

htmlAnchorsToMarkdown(content) — audit.ts:245

Converts any <a href="...">label</a> in the response to markdown [label](url). Gemini occasionally emits HTML anchors despite markdown instructions.

For each markdown link [label](url) found in the paragraph: 1. canonicalizeUrl(url) — normalize 2. classifyLink(label, url, ctx) → returns LinkDecision 3. applyDecision(label, fullMatch, decision) — mutates the text based on the decision 4. formatTraceLine(...) — produces the console log

classifyLink(label, url, ctx) — audit.ts:159

THE core decision function. Possible outcomes:

Decision When Effect
keep Link passes all checks no change
replace Generic label + matching source found label upgraded from generic → real doc title
strip Hostname not allowed remove link
strip URL is technical doc + path not whitelisted remove link (per Knowledge Charter — AE shouldn't cite API refs)
strip URL matches HALLUCINATED_CUSTOMER_SENTINEL remove entire sentence containing this link

ClassifyContext (line 110):

{
  sources: any[],       // retrieval sources
  mapData: any[],       // Atlas map
  allowedHostnames: string[],
  isMaverick: boolean,
  enableHealthChecks: boolean,  // post-Step-2: now always ON
}

upgradeGenericLinkLabels(content, sources) — audit.ts:256

If a link has a generic label like "Algolia blog" or "this article", tries to match the URL to a source and replace the label with the source's doc_title.

F-031 lives here: the upgrade appends | Algolia suffix even when the source title already ends with | Algolia. Step 3 fix = strip trailing suffix before appending OR normalize to single canonical.

stripDanglingAttributions(content) — audit.ts:269

When a link gets stripped, there's often an orphaned "— Source 3" or "according to Algolia" fragment. This removes them.

auditGrounding(text) (internal in InlineStreamAuditor)

Scans for sentences that mention known customer entities + makes claims. Uses knownEntities Set (built in constructor from mapData + sources). If a sentence mentions a customer but the customer isn't in knownEntities, or the claim isn't grounded in source text, the sentence gets stripped via stripHallucinatedCustomerSentences.

Current gap (F-038): - Works for: "Company X said Y" where X isn't in knownEntities → strip - Fails for: "Company X achieved +206% revenue" where X might not be checked rigorously, OR the metric isn't verified against any source text, OR the sentence lacks a case study link - Fails entirely for specialist responses (InlineStreamAuditor is never instantiated for Elena/Bruno)

stripHallucinatedCustomerSentences(content) — audit.ts:300

Removes entire sentences flagged with HALLUCINATED_CUSTOMER_SENTINEL. Sentence-level removal (not just the link).

ContentAuditor.auditContent() (post-stream, post-Step-2)

File: content_auditor.ts:113

Fires after streaming completes, in stream_processor.ts.

Post-Step-2 scope (what's left): - Link HEAD checks — fetches every link with HEAD request, records httpStatus, healthy - Enabled since Step 2 — enableHealthChecks: true at stream_processor.ts:244

Deleted in Step 2 (theater): - checkPersonaCompliance — scored tone compliance, never enforced (F-029 → Step 6) - checkSourceGrounding — duplicate of InlineStreamAuditor's work - computeConfidence — weighted avg score, never gated response - persistAuditResult — fire-and-forget Redis write, never read back - Persona forbidden-phrase constants (UNIVERSAL_FORBIDDEN, MAVERICK_RULES, ELENA_RULES, BRUNO_RULES)

Specialist path HEAD check (note)

Although specialist stream is NOT audited for hallucination, it DOES get a HEAD check on every link in its final response:

Code: orchestrator.ts:697–712

const specialistLinkRegex = /\[([^\]]*)\]\((https?:\/\/[^)]+)\)/g;
// extract URLs, Promise.all HEAD fetch with 4s timeout
// record {url, httpStatus, healthy}

What this covers: dead links — a broken URL gets surfaced in linkAuditResults. What this does NOT cover: - Whether the label matches the content (no title verification) - Whether attributed metrics (+206%) are grounded in sources - Whether customer names mentioned are real Algolia customers - Whether a customer claim has a case study link attached

This is the F-038 gap.

SSE events emitted by the audit pipeline

Event Source Payload
audit_trace Inline auditor trace lines Array of [AUDIT] ... strings
audit After stream end {confidenceScore, links, tone, grounding} — post-Step-2 these are partially placeholder values

Testing

Active tests

  • __tests__/auditor-links.test.ts — link classification edge cases [12 failing — F-030 pre-existing drift, not Step-1/2 caused]
  • __tests__/content-auditor-links.test.ts — integration for link audit

Deleted in Step 2

  • __tests__/auditor-persona.test.ts (150 lines) — tested removed checkPersonaCompliance
  • __tests__/auditor-confidence.test.ts (120 lines) — tested removed computeConfidence

Planned additions in Step 3 (F-038 fix)

1. Wire InlineStreamAuditor into specialist path

Location: lib/agent-studio/stream-adapter.ts in adaptAgentStudioStream() — aggregate chunks into paragraphs, call auditParagraph() before passing to the client's event: chunk emitter.

Complication: Agent Studio chunks don't align to paragraph boundaries. Need a buffering layer (could reuse StatefulStreamStripper's state machine pattern).

2. Add validateCustomerClaim(sentence, sources, mapData): { valid, reason? }

Location: lib/search/audit.ts Rule (ALL must be true for claim to pass): 1. Company name appears in at least one retrieved source or Atlas mapData (case-insensitive match) 2. Metric value (regex \+?\d+%|\$\d+[MBK]?|\d+x) appears in at least one source attributed to the same company 3. Sentence contains a markdown link to algolia.com/customers/<slug> 4. That link passes HEAD check (already wired — post-Step-2 enableHealthChecks=true)

On fail: strip the entire sentence via stripHallucinatedCustomerSentences pattern. No rewriting (rewriting = new hallucination).

3. Gate tests

  • Al-Futtaim fabrication must strip
  • PcComponentes unlinked claim must strip
  • Real customer WITH verified case study link must pass

Known issues in the audit area (post-Step-3)

F# Issue Status
F-028 Need holistic answer auditor (not just link) Open — Step 5
F-030 Pre-existing test drift (expanded to ~60 failures during Step 2 refactor) Separate cleanup pass needed
F-031 \| Algolia \| Algolia double-suffix ✅ FIXED Step 3 — normalizeLinkLabelSuffix() in audit.ts
F-032 "Namedrop:" prompt leak ✅ FIXED Step 3 — PERSONAS.md renamed + stripPromptInstructionLeaks() scrubber
F-038 Specialist has no hallucination audit ✅ FIXED Step 3 — validateCustomerClaim() + InlineStreamAuditor wired into adaptAgentStudioStream

New audit primitives shipped in Step 3

normalizeLinkLabelSuffix(content): string (audit.ts, F-031)

Collapses [... | Algolia | Algolia](url) → [... | Algolia](url). Also handles triple/quad repeats. Case-insensitive. Runs after upgradeGenericLinkLabels in InlineStreamAuditor.auditParagraph.

stripPromptInstructionLeaks(content): string (audit.ts, F-032)

Defence-in-depth strip of prompt-template headings that the LLM occasionally echoes verbatim (Namedrop:, VALUE SELL:, Customer Example:, SIGNAL EXTRACTION:). Strips the whole heading line — preserves surrounding paragraphs. Runs at the start of InlineStreamAuditor.auditParagraph.

validateCustomerClaim(sentence, sources, mapData): { valid, reason? } (audit.ts, F-038)

The hallucination guard for specialist output. Returns valid=false with a reason when a sentence contains BOTH a non-Algolia capitalized name AND a quantitative token (%, $, Nx) but fails ANY of: 1. Company name appears in sources[] or mapData[] 2. Metric value appears in sources/map entries attributed to that company 3. Sentence contains a algolia.com/customers/<slug> link matching the company

Called from InlineStreamAuditor.auditGrounding via a non-destructive sentence-level regex replace. On fail → strip sentence (preserves surrounding whitespace/newlines).

Specialist audit wiring (F-038 Part B)

Previously, adaptAgentStudioStream in lib/agent-studio/stream-adapter.ts emitted raw Agent Studio chunks straight to the client. Now it:

  1. Accepts optional auditContext: { sources, mapData } in StreamAdapterOptions
  2. Instantiates new InlineStreamAuditor(auditSources, auditMapData) at entry
  3. Buffers text deltas into paragraphs (double-newline boundaries)
  4. Calls inlineAuditor.auditParagraph(paragraph) on each completed paragraph BEFORE emitting event: chunk to client
  5. Grows auditSources live from Agent Studio's a: tool-result events so sentences citing those sources can ground correctly

orchestrator.igniteSpecialist now passes auditContext: { sources: [], mapData: fetchGoldenMap() } to the adapter. Specialist stream is no longer a blind passthrough — same guardrails as Maverick.

Cross-references

  • 02-Data-Flow — where the audit fires in the pipeline
  • 04-Functions-Reference — classifyLink, auditParagraph, InlineStreamAuditor
  • 07-Personas-Reference — specialist audit gap (F-038)
  • 10-Known-Issues — full finding list