Dashboard/Scout/wiki/architecture/scout-platform-architecture.md
Scout

wiki/architecture/scout-platform-architecture.md

Scout Platform Architecture

Last Updated: 2026-05-16

Scout is a provider-agnostic web-to-record platform. Its job is to acquire source evidence from the best available channel, normalize that evidence, process it through a use-case-specific schema, validate the result, and write portable artifacts.

System Diagram

flowchart LR
  User["User or Agent"] --> Interfaces["Interfaces
CLI | HTTP | Claude/Codex Skill | Scheduler"]
  Interfaces --> Run["Run Orchestrator
scout.core.platform.run"]
  Run --> Registry["Use Case Registry
products | jobs | prism | investor | research | docs | news | social | locations"]
  Run --> Providers["Provider Layer"]

  Providers --> Crawl4AI["Crawl4AI
standalone default"]
  Providers --> HostFetch["Host WebFetch/WebSearch
skill-only when available"]
  Providers --> Browser["Browser/CDP/In-App Session
secondary fallback"]
  Providers --> Files["Saved HTML / DOM / PDFs / Feeds"]
  Providers --> Adapters["Future Adapters
ATS | social | ecommerce APIs"]

  Crawl4AI --> Evidence["Normalized Source Evidence"]
  HostFetch --> Evidence
  Browser --> Evidence
  Files --> Evidence
  Adapters --> Evidence

  Evidence --> Processor["Vertical Processor"]
  Registry --> Processor

  Processor --> Products["Product Records
Algolia-ready PDP/search"]
  Processor --> Jobs["Job Hunter Records
target companies + job postings"]
  Processor --> Prism["PRISM Records
company, execs, socials, investor, careers"]
  Processor --> Research["Research Records
about pages, docs, news, locations"]

  Products --> Validation["Validation + Provenance"]
  Jobs --> Validation
  Prism --> Validation
  Research --> Validation

  Validation --> Artifacts["Standard Artifact Writer"]
  Artifacts --> Manifest["manifest.json"]
  Artifacts --> Records["records.json / records.jsonl"]
  Artifacts --> Sources["source_pages.json"]
  Artifacts --> Blocked["blocked_pages.json"]
  Artifacts --> Findings["validation.json"]
  Artifacts --> Report["extraction_report.md"]

  Records --> Consumers["Consumers
Algolia | PRISM | job monitor | analyst review | downstream apps"]

Core Principle

Scout is not just a crawler. A crawler fetches pages. Scout produces reusable records with provenance.

That means every workflow has five stages:

  1. Intent: parse the user goal or workflow profile.
  2. Acquisition: choose providers and gather source evidence.
  3. Normalization: convert HTML, markdown, browser DOM, PDFs, feeds, or host tool output into a common evidence shape.
  4. Extraction: apply a use-case-specific schema and processor.
  5. Packaging: write records, source evidence, blocked pages, validation findings, and a human-readable report.

Provider Strategy

Default order for standalone CLI:

  1. Crawl4AI scrape/map/crawl.
  2. Saved HTML/DOM/PDF input when supplied.
  3. Optional CDP/profile browser attach when explicitly configured.
  4. Future direct adapters such as ATS feeds or ecommerce APIs.

Default order for Claude/Codex skill usage:

  1. Crawl4AI through local Scout when appropriate.
  2. Host WebSearch/WebFetch when the host can fetch pages more reliably.
  3. In-app browser session only as a fallback for blocked or session-dependent pages.
  4. Saved evidence replay into Scout processing.

Browser execution is not the default. It is a fallback channel when regular acquisition is blocked, incomplete, or needs session context.

Artifact Contract

Every high-level scout run workflow writes:

manifest.json
records.json
records.jsonl
source_pages.json
blocked_pages.json
validation.json
extraction_report.md

This is the portability layer. The same folder can be inspected by a human, indexed into Algolia, attached to a PRISM run, or used by a scheduler to detect changes over time.

Current Build State

  • Platform foundation: implemented.
  • Product contracts and extraction path: implemented, with hard-site limitations documented.
  • Browser fallback: implemented as a secondary channel for product scraping.
  • Job Hunter V1: implemented for profile loading and target company record artifacts.
  • Job Hunter live extraction: next slice.
  • PRISM vertical processor: planned next after Job Hunter acquisition/matching patterns stabilize.