Dashboard/PRISM/Architecture/intel-company-3track-pipeline.md
PRISM

Architecture/intel-company-3track-pipeline.md

intel-company: 3-Track Pipeline Architecture

Designed and implemented: 2026-04-14 | Commit: a78f0cb

Overview

intel-company is the SEED module — every spoke module reads from its output. It runs a 3-track pipeline to maximize data accuracy and source coverage.

Track Architecture

Track 1: WebFetch (Primary — always wins conflicts)

  • Uses shared BrowserClient (prism_platform/browser/)
  • Fetches: leadership/about page, IR page, newsroom page
  • Smart link discovery: fetches homepage, extracts about/leadership hrefs
  • Handles JS rendering via Jina Reader API fallback
  • Ground truth — live data from company's own website today

Track 2: Perplexity sonar-pro (External lens)

  • Playbook v2.2.0 — knows it's Track 2, knows what Track 1 already has
  • Focused on what WebFetch cannot get: competitors, industry classification, private company financials, MEDDPICC role mapping
  • Explicit LinkedIn search per exec (search: "{name}" "{company}" site:linkedin.com/in)
  • Single call — made sharper, not split into two

Track 3: Synthesis LLM (Reconciliation)

  • Model: gemini-3.1-flash-lite-preview
  • Takes Track 1 + Track 2 outputs
  • Resolution rule: WebFetch wins conflicts; Perplexity fills gaps
  • Perplexity-only fields: competitors, industry, sub_vertical, MEDDPICC roles
  • WebFetch-priority fields: exec names/titles, employee count from 10-K, social URLs

Pipeline Health

  • PipelineHealthLog captures every failure, fallback, tier escalation, data quality warning
  • Overall status: HEALTHY / DEGRADED / FAILED
  • Surfaces in: markdown report header, JSON output, console

What WebFetch provides vs Perplexity

Field WebFetch Perplexity
Exec names/titles ✅ primary fallback
Employee count ✅ public cos (10-K) private cos
Revenue ✅ public cos (10-K) ✅ private cos only source
Company social URLs ✅ primary fallback
Competitors ❌ never ✅ only source
Industry/sub_vertical ❌ ✅ only source
MEDDPICC roles ❌ ✅ only source
Recent headlines press releases only ✅ analyst/journalist
Exec LinkedIn URLs sometimes ✅ better coverage

File Map

intel_company/
  config.py                 — module config
  schemas.py                — CompanySeedOutput (final contract)
  fetcher.py                — Track 1 (BrowserClient)
  playbook.md               — Track 2 (Perplexity v2.2.0)
  playbook_synthesis.md     — Track 3 (Gemini prompt)
  hooks.py                  — post_execute (writes to accounts table)

Shared Infrastructure

prism_platform/browser/
  client.py       — BrowserClient (unified, tiered)
  types.py        — FetchResult, FetchOptions, FetchTier
  detector.py     — bot-block detection
  tier1_direct.py — httpx + Jina Reader (implemented)
  tier2_stealth.py — Playwright stealth (stub)
  tier3_service.py — Browserless/ScrapingBee (stub)

Caching

  • module_executions table: keyed on (module_name, domain)
  • Cache-first: zero API calls on repeat runs
  • --refresh flag to force re-run
  • 180-day TTL for intel-company

What's Still Missing

  • Wire Track 3 into activities.py (Temporal production path)
  • Unit tests + integration tests
  • UI: company identity card view, pipeline health badge, cache freshness
  • Tier 2 Playwright stealth (shared with search audit)