Dashboard/Scout/wiki/syntheses/build-plan-2026-05-15.md
Scout

wiki/syntheses/build-plan-2026-05-15.md

Scout Build Plan

Date: 2026-05-15

Decision

All identified Scout use cases are required. The build should not pick only product extraction. It should build a shared platform foundation first, then implement use-case modules on top.

Required Use Cases

  • Product catalog extraction.
  • Job hunter and scheduled career monitoring.
  • PRISM company/prospect intelligence.
  • Investor and financial document intelligence.
  • Generic web research and opportunity discovery.
  • Website quality and competitive gap analysis.
  • Documentation and knowledge-base extraction.
  • Newsroom and signal monitoring.
  • Social signal normalization through explicit providers.
  • Store/location extraction.

Architecture Direction

Scout will be built around this pipeline:

Intent -> Discovery -> Provider Fetch -> Extraction -> Enrichment -> Validation -> Records -> Artifacts

Provider acquisition is separate from extraction.

Core providers:

  • WebSearch host provider.
  • WebFetch host provider.
  • Crawl4AI provider.
  • Browser/session provider.
  • CDP/profile provider.
  • Saved HTML/DOM provider.
  • PDF/document provider.
  • ATS provider adapters.
  • Social provider import adapters.

UX Direction

High-level command family:

scout run <use-case> [options]
scout plan <use-case> [options]
scout validate <run-dir>
scout replay <run-dir-or-fixture>
scout init-profile <profile-type>

Existing low-level commands such as scrape, crawl, map, and products can remain.

Use-Case Invocation Examples

Products:

scout run products --site esteelauder.com --query "top categories plus best sellers" --limit-per-category 10

Jobs:

scout run jobs --profile job-profile.yaml --output-dir scout-runs/job-hunter-ai-pm

PRISM:

scout run prism --company nike --output-dir scout-runs/prism-nike

Investor:

scout run investor --company salesforce --ticker CRM --output-dir scout-runs/investor-salesforce

Generic research:

scout run research --query "companies with poor websites in local dental clinics in Austin"

Website quality:

scout run website-quality --urls urls.txt --rubric website-rubric.yaml

Docs:

scout run docs --site https://www.algolia.com/doc/api-reference

News:

scout run news --company nike --days 60

Social:

scout run social --provider-input linkedin-export.json --company nike

Locations:

scout run locations --site nike.com --query "stores in New York"

Input And Output Contracts

Every high-level run normalizes input into:

  • use case
  • query
  • target URLs/companies/sites
  • provider preferences
  • output directory
  • limits
  • optional schedule
  • optional profile path

Every high-level run writes:

  • manifest.json
  • records.json
  • records.jsonl
  • source_pages.json
  • blocked_pages.json
  • validation.json
  • extraction_report.md

Test Strategy

Each use case requires:

  • schema unit tests
  • parser/normalizer tests
  • fixture tests using saved HTML/markdown/DOM/PDF/provider JSON
  • artifact contract tests
  • live smoke tests
  • blocked/partial-success tests

Scheduling-oriented use cases also need delta tests.

Repository Artifacts

Detailed design:

docs/workspace/scout-core/07-platform-build-design.md

Foundation implementation plan:

docs/superpowers/plans/2026-05-15-scout-platform-foundation.md

Execute the foundation plan first. Then write separate vertical implementation plans for:

  1. Product catalog extraction v2.
  2. Job hunter intake, company discovery, ATS detection, and daily deltas.
  3. PRISM company intelligence source discovery and evidence bundle.
  4. Investor/PDF parsing.
  5. Generic research and website quality.
  6. Docs/news/social/location modules.