Algolia-Search-Audit

Refactor-Architecture.md

Algolia Search Audit — Refactor Architecture

The target architecture for the refactored audit pipeline. Based on Anthropic's agent harness patterns (AgentHarnessPatterns).

Status: Planning (2026-04-08) Decision: Python CLI harness + Pydantic schemas + codified deterministic logic

Design Principles

Context isolation — each module runs in its own claude -p subprocess (fresh 200K context)
Schema-first — every module output validated by Pydantic before the harness marks it complete
Codify what you can — if a step is deterministic math/rules, it's Python, not LLM
Files are the API — modules communicate only through .json files in the workspace
The harness is not an LLM — orchestration, gating, retry, and progress tracking are pure Python
Resume-from-failure — JSONL progress log allows --resume to skip completed modules

Target Architecture

audit-harness.py  ← Python CLI entry point (no LLM)
│
├── lib/
│   ├── schemas/              ← Pydantic models (one per module output)
│   │   ├── __init__.py       ← SKILL_SCHEMA_MAP registry
│   │   ├── company_context.py
│   │   ├── tech_stack.py
│   │   ├── traffic_data.py
│   │   ├── competitors.py
│   │   ├── financial_profile.py
│   │   ├── investor_intel.py
│   │   ├── hiring_signals.py
│   │   ├── social_signals.py
│   │   ├── news_signals.py
│   │   ├── partner_intel.py     ← NEW (doesn't exist today)
│   │   ├── industry_intel.py
│   │   ├── test_queries.py
│   │   ├── browser_findings.py
│   │   ├── business_case.py
│   │   ├── sales_plays.py
│   │   ├── audit_data.py        ← wraps existing JSON Schema
│   │   ├── abx_campaign.py
│   │   └── factcheck.py
│   │
│   ├── scripts/              ← Python data collection (no LLM)
│   │   ├── collect-company.py        (existing)
│   │   ├── collect-techstack.py      (existing)
│   │   ├── collect-traffic.py        (existing)
│   │   ├── collect-competitors.py    (existing)
│   │   ├── collect-financials.py     (existing)
│   │   ├── collect-hiring.py         (existing)
│   │   ├── collect-social.py         (existing)
│   │   ├── collect-news.py           (existing)
│   │   ├── collect-industry.py       (existing)
│   │   ├── collect-investor.py       (existing)
│   │   ├── collect-sec-data.py       ← NEW (shared by 1E + 1G)
│   │   ├── collect-partners.py       ← NEW (cross-ref techstack vs partner list)
│   │   ├── calculate-roi.py          (existing — wire into pipeline)
│   │   ├── calculate-score.py        (existing)
│   │   ├── factcheck-mechanical.py   ← NEW (dims 1-12 + 18-20)
│   │   ├── classify-roles.py         ← NEW (title → buyer tier keyword matching)
│   │   └── generate-audit-data.py    (existing)
│   │
│   ├── utils/
│   │   ├── source_labeler.py         ← NEW (shared [FACT — source, date] formatting)
│   │   └── progress.py               ← NEW (AuditProgress class — JSONL read/write)
│   │
│   └── data/                 ← Static datasets (version-controlled, refreshed periodically)
│       ├── algolia_customers.json    ← NEW (domain/vertical/metric/URL)
│       ├── algolia_partners.json     ← NEW (tech partner list for cross-ref)
│       ├── vertical_benchmarks.json  ← NEW (Baymard/Forrester stats by vertical)
│       └── buyer_tier_keywords.json  ← NEW (title keywords → tier mapping)
│
├── prompts/                  ← LLM task prompts (one per module that needs LLM)
│   ├── company-enrich.txt        (executives, vertical, portfolio)
│   ├── competitors-enrich.txt    (golden angle analysis, scenario classification)
│   ├── financial-public-enrich.txt  (SEC 10-K reading, earnings quotes)
│   ├── financial-private-enrich.txt (6-source waterfall narrative)
│   ├── investor-enrich.txt       (quote extraction from transcripts)
│   ├── industry-enrich.txt       (benchmark synthesis, algolia_angle)
│   ├── queries.txt               (full query generation)
│   ├── browser.txt               (Playwright test execution + findings)
│   ├── business-case-narrative.txt  (narrative framing — numbers come from Python)
│   ├── sales-plays.txt           (playbook generation)
│   ├── report-assembly.txt       (narrative sections only — data from JSON)
│   ├── abx-campaign.txt          (email/LinkedIn/Loom generation)
│   └── factcheck-narrative.txt   (dims 13-17 only — quality checks)
│
├── requirements.txt          ← pydantic, rich, anthropic (for API calls if needed)
└── install.sh

Module Execution Model (New)

Each module now has a clear execution model:

Type 1: Pure Python (no LLM)

Script runs, produces .json, harness validates against Pydantic, done.

Module	Script	Schema
1B techstack	`collect-techstack.py` + deterministic status classification	TechStack
1C traffic	`collect-traffic.py`	TrafficData
1H hiring	`collect-hiring.py` + `classify-roles.py`	HiringSignals
1I social	`collect-social.py` + keyword relevance scoring	SocialSignals
1J news	`collect-news.py` + keyword categorization	NewsSignals
3A business case (numbers)	`calculate-roi.py`	BusinessCase
4A factcheck (mechanical)	`factcheck-mechanical.py`	FactcheckMechanical

Type 2: Python Script + LLM Enrichment

Script runs first (API calls, data collection), then claude -p enriches the output.

Module	Script	LLM Prompt	Schema
1A company	`collect-company.py`	`company-enrich.txt`	CompanyContext
1D competitors	`collect-competitors.py`	`competitors-enrich.txt`	Competitors
1E financial (public)	`collect-financials.py` + `collect-sec-data.py`	`financial-public-enrich.txt`	FinancialProfile
1F financial (private)	`collect-financials.py --private`	`financial-private-enrich.txt`	FinancialProfile
1G investor	`collect-investor.py` + `collect-sec-data.py` (shared)	`investor-enrich.txt`	InvestorIntel
1K partner	`collect-partners.py` (NEW)	partner enrichment (if needed)	PartnerIntel
1L industry	`collect-industry.py`	`industry-enrich.txt`	IndustryIntel

Type 3: Pure LLM (irreducibly creative)

claude -p with full prompt, reads specific input files, writes output.

Module	LLM Prompt	Schema
2 queries	`queries.txt`	TestQueries
L2 browser	`browser.txt`	BrowserFindings
3A business case (narrative)	`business-case-narrative.txt`	(merged with calc output)
3B sales plays	`sales-plays.txt`	SalesPlays
3C report	`report-assembly.txt`	AuditData
3D ABX campaign	`abx-campaign.txt`	ABXCampaign
4B factcheck (narrative)	`factcheck-narrative.txt`	FactcheckNarrative

Harness Flow

async def run_audit(domain: str, workspace: Path, resume: bool = False):
    harness = AuditHarness(domain, workspace, resume)

    # Phase 0: Determine public vs private
    is_public, ticker = await harness.determine_company_type(domain)

    # Phase 1: Wave 1 — all parallel
    wave1_tasks = [
        harness.run_module("company", type="script+llm"),
        harness.run_module("techstack", type="script"),
        harness.run_module("traffic", type="script"),
        harness.run_module("competitors", type="script+llm"),
        harness.run_module("financial-public" if is_public else "financial-private", type="script+llm"),
        harness.run_module("investor", type="script+llm"),
        harness.run_module("hiring", type="script"),
        harness.run_module("social", type="script"),
        harness.run_module("news", type="script"),
        harness.run_module("partner", type="script+llm"),
        harness.run_module("industry", type="script+llm"),
    ]
    results = await asyncio.gather(*wave1_tasks, return_exceptions=True)
    harness.gate("wave1", min_files=11, min_size=500)

    # Phase 2: Query generation
    await harness.run_module("queries", type="llm", reads=["01-company-context.json", "02-tech-stack.json"])
    harness.gate("wave2", required_files=["05-test-queries.json"])

    # Phase 3: Browser audit
    await harness.run_module("browser", type="llm", reads=["05-test-queries.json", "02-tech-stack.json"])
    harness.gate("browser", min_screenshots=10)

    # Phase 4: Synthesis chain (sequential)
    await harness.run_module("business-case", type="script+llm")  # calculate-roi.py → narrative
    await harness.run_module("sales-plays", type="llm")
    await harness.run_module("report", type="script+llm")  # generate-audit-data.py → narrative sections
    await harness.run_module("abx-campaign", type="llm")
    harness.gate("synthesis", required_files=["audit-data.json", "index.html"])

    # Phase 5: Factcheck
    await harness.run_module("factcheck-mechanical", type="script")
    await harness.run_module("factcheck-narrative", type="llm")

    # Phase 6: Stage for review
    harness.stage_for_review()

Key Differences from Current Architecture

Aspect	Current	Refactored
Orchestration	Claude Code Skill tool (inline, shared context)	Python `audit-harness.py` (subprocess, isolated contexts)
Schema validation	None	Pydantic model per module, validated before marking complete
ROI calculation	LLM inference	`calculate-roi.py` (Python)
Report data source	`.md` prose files	`.json` structured files only
Partner intel output	`.md` only	`.md` + `.json`
ABX data storage	Mutates `audit-data.json`	Separate `abx-data.json`
SEC/earnings fetch	Duplicated (1E + 1G)	Shared `collect-sec-data.py`
Factcheck	All 20 dims via LLM	15 dims Python + 5 dims LLM
Algolia customers	Scraped per audit	Static `algolia_customers.json`
Progress tracking	Inconsistent JSONL	Harness-managed, per-module cost/timing
Retry logic	None (manual re-run)	3 attempts per module with backoff
Resume	Start over	`--resume` reads JSONL, skips completed
Observability	None	Rich terminal dashboard + JSONL audit trail

Distributable Package

For sharing with others (AEs, SEs, partners):

# Install
pip install -r requirements.txt
# Requires: claude CLI installed and authenticated

# Run
python3 audit-harness.py dsw.com --company "DSW" --ticker DBI

# Resume after failure
python3 audit-harness.py dsw.com --resume

# Run single module
python3 audit-harness.py dsw.com --module techstack

# Status dashboard
python3 audit-harness.py dsw.com --status

No Claude Code skills needed. No Skill tool. Just Python + claude CLI.

Sprint Plan

Sprint 1: Foundation (harness + schemas + codification)

audit-harness.py core (workspace, progress, args, resume)
All 20 Pydantic schemas
Wire calculate-roi.py, create factcheck-mechanical.py
Create collect-sec-data.py (shared 1E + 1G)
Create collect-partners.py (tech stack cross-ref)
Static datasets (algolia_customers.json, buyer_tier_keywords.json)

Sprint 2: Skill prompts + enrichment

Write all prompts/*.txt files
Review + fix each skill's LLM enrichment instructions
Add JSON output where missing (partner, queries, browser findings, sales plays)
Fix field name inconsistencies across all skills

Sprint 3: Report pipeline

Switch report to read .json exclusively
Wire generate-audit-data.py + calculate-score.py
Fix capability matrix dynamic competitor mapping
Separate ABX output from audit-data.json

Sprint 4: Terminal UI + testing

Rich progress dashboard
Per-module cost/timing tracking
End-to-end smoke test on a known domain
Documentation

AgentHarnessPatterns — Anthropic's harness patterns (source of key patterns)
Index — project overview
Module-Catalog — per-module detail
Known-Issues — problems this refactor fixes