Dashboard/Scout/wiki/decisions/2026-05-20-scout-full-intelligence-platform.md
Scout

wiki/decisions/2026-05-20-scout-full-intelligence-platform.md

ADR: Scout Is A Full Web Intelligence Platform

Date: 2026-05-20 Status: Accepted

Decision

Scout is no longer framed as only a product scraper or only a job hunter. Scout is a reusable, provider-agnostic web intelligence platform that can run as:

  • standalone CLI,
  • standalone HTTP app,
  • Claude/Codex skill,
  • reusable extraction engine behind other systems such as PRISM and Algolia demos.

The platform supports these first-class use cases:

  • company intelligence,
  • PRISM prospect research,
  • investor intelligence,
  • careers and hiring intelligence,
  • job extraction and scoring,
  • product catalog extraction for Algolia,
  • news and signal monitoring,
  • generic research/document/site extraction.

Why

The common problem across these workflows is not just crawling. It is turning messy web evidence into reusable, provenance-backed records. A normal WebFetch can retrieve a page, but Scout adds:

  • repeatable execution modes,
  • provider fallback,
  • vertical extraction contracts,
  • validation and blocked-page reporting,
  • durable artifact folders,
  • stable records suitable for downstream indexing and research.

Consequences

  • Browser fallback is not default. It is secondary and should trigger only when regular acquisition is blocked, sparse, JS-heavy, or explicitly being verified.
  • scout run <use-case> is the stable high-level user surface.
  • CLI and HTTP must accept the same execution mode language.
  • Skill docs must teach when to use local Scout, host WebFetch/WebSearch, or browser fallback.
  • Every run must preserve provenance and write the standard artifact contract.
  • Private profile data remains outside the public repo.

Current Implementation Slice

The current build slice adds the platform foundation:

  • execution mode enum and provider ladder,
  • typed V1/V2 record contracts for broad verticals,
  • deterministic V1 high-level run outputs,
  • CLI and HTTP /run/{use_case} integration,
  • updated skill playbook and repo docs.

Deeper live acquisition for each vertical remains an implementation layer behind the now-stable surface.