Dashboard/Scout/wiki/requirements.md
Scout

wiki/requirements.md

Scout - Requirements

Last Updated: 2026-05-14

Product Requirement

Scout must convert web-accessible content into structured, provenance-backed records for downstream use cases such as Algolia indexing, company intelligence, investor intelligence, career/job discovery, and research workflows.

Core Requirements

R1 - Provider-Agnostic Fetch Input

Scout must accept content from multiple providers:

  • Hosted WebFetch/WebSearch in skill mode.
  • Crawl4AI in standalone mode.
  • Browser/session DOM or HTML.
  • CDP/profile browser attachment.
  • Saved HTML/DOM files.
  • PDF/document parser.
  • ATS and social providers where relevant.

Acceptance criteria:

  • Extractors do not depend directly on Crawl4AI response objects.
  • Every fetched page records provider name, source URL, final URL, fetched timestamp, and failure/block status.
  • A provider can fail without killing the whole run when another provider can be attempted.

R2 - Domain Record Schemas

Scout must define versioned record schemas for each supported domain.

Initial record types:

  • ProductRecord
  • ExecutiveRecord
  • CompanyPageRecord
  • InvestorDocumentRecord
  • FinancialMetricRecord
  • JobPostingRecord
  • NewsArticleRecord
  • SocialSignalRecord
  • DocumentationPageRecord
  • LocationRecord
  • GenericPageFactRecord

Acceptance criteria:

  • Each record includes objectID or equivalent stable ID.
  • Each record includes source provenance.
  • Each record includes confidence/completeness metadata.
  • Records can be written as JSON and JSONL.

R3 - Discovery Strategies

Scout must discover URLs using the best available route for the use case.

Discovery routes:

  • Explicit URLs.
  • Known site seed configs.
  • WebSearch.
  • Sitemap/map.
  • Crawl.
  • Provider-specific APIs.
  • Page-internal links.

Acceptance criteria:

  • User can provide explicit URLs to skip discovery.
  • Known seed config can override generic guessing.
  • Discovery outputs are persisted in source_pages.json or equivalent.
  • Failed discovery is reported with a reason, not as empty success.

R4 - Extraction And Enrichment

Scout must support both first-pass extraction and enrichment.

Examples:

  • Product listing card + PDP enrichment.
  • Company about page + leadership page + LinkedIn/provider enrichment.
  • Investor index page + PDF report extraction.
  • Career landing page + ATS job detail enrichment.

Acceptance criteria:

  • A record can be merged from multiple source pages.
  • Merge logic prefers higher-confidence fields.
  • Field provenance can identify which page supplied a field when useful.

R5 - Artifact Discipline

Every run should write predictable artifacts.

Required artifacts:

  • records.json
  • records.jsonl
  • manifest.json
  • source_pages.json
  • blocked_pages.json
  • extraction_report.md

Acceptance criteria:

  • User can choose output directory.
  • Non-interactive mode has a deterministic default output directory.
  • Blocked pages and partial results are visible.
  • Artifacts are stable enough to test with snapshots.

R6 - Standalone And Skill Distribution

Scout must work as:

  • Python package.
  • CLI.
  • Importable library.
  • Local API service.
  • Claude/Codex skill.

Acceptance criteria:

  • Skill mode can use host tools like WebSearch/WebFetch/browser when available.
  • CLI mode can run without host tools using Crawl4AI, saved HTML, CDP/profile, or other standalone providers.
  • Documentation clearly marks provider availability by environment.

R7 - Evaluation Harness

Each use case must have tests.

Test layers:

  • Unit tests for parsers, normalizers, merge logic.
  • Fixture tests from saved HTML/markdown/DOM/PDF samples.
  • Contract tests for schemas and artifacts.
  • Live smoke tests for selected sites, marked separately from deterministic tests.

Acceptance criteria:

  • No use case is "supported" without at least one fixture test and one documented live smoke scenario.
  • Hard-site failures produce expected blocked/provider-required artifacts.

Non-Goals For V1

  • Fully automated login or authenticated scraping.
  • Applying to jobs.
  • Bypassing platform controls as a primary design goal.
  • Replacing official APIs where official APIs are available and appropriate.
  • Full social scraping of LinkedIn/X with raw Crawl4AI.
  • Final strategic synthesis without source-backed evidence.