Dashboard/Scout/wiki/syntheses/scout-platform-strategy.md
Scout

wiki/syntheses/scout-platform-strategy.md

Scout Platform Strategy

Last Updated: 2026-05-14

Strategic Thesis

Scout can become a multi-use-case intelligence substrate if it is built around one durable abstraction:

source content -> structured records -> downstream workflows

The mistake would be to build "a crawler that also does some extraction." The stronger product is "an extraction engine that can use crawling when crawling is the right acquisition method."

Where Scout Fits

Scout should sit below PRISM, job applicator tools, Algolia indexing workflows, and research agents.

Scout produces:

  • Evidence.
  • Records.
  • Provenance.
  • Artifacts.
  • Quality scores.

Scout should not own every final analysis narrative. Analysis layers can consume Scout output.

What You Are Thinking About Correctly

1. The use cases share a common pipeline

Products, company pages, investor docs, jobs, docs pages, newsrooms, and social outputs all need the same skeleton:

discover -> fetch -> extract -> enrich -> normalize -> score -> write

This shared skeleton is the product opportunity.

2. WebFetch should be a first-class provider

If hosted WebFetch can read a page, use it. There is no moral victory in making Crawl4AI struggle when WebFetch works.

3. Crawl4AI still matters

Crawl4AI matters for standalone CLI/API/Docker, site crawling, screenshots, JS rendering, sitemap traversal, and environments without hosted tools.

4. Scout should support both skill mode and standalone mode

Skill mode can use host capabilities. Standalone mode cannot assume them. The docs and architecture must be honest about that difference.

What Else You May Not Be Thinking About

1. Evaluation will become the product moat

Everyone can call a fetcher. The hard part is proving that extraction is correct across many websites.

Scout needs a benchmark set:

  • 5 product sites.
  • 5 leadership/about sites.
  • 5 investor sites.
  • 5 careers/ATS sites.
  • 5 docs sites.
  • 5 newsroom sites.
  • 5 hard/blocked sites.

Each needs saved fixtures, expected records, and live smoke tests.

2. Schema governance matters early

If records change shape every week, Scout cannot power downstream tools reliably.

Recommendation:

  • Version every schema.
  • Include schema_version in every record.
  • Include source.provider, source.url, source.fetched_at, and source.confidence.
  • Keep raw provider output in artifacts for debugging.

3. Provider capability matrix must be explicit

Users will get confused unless Scout tells them what each provider can and cannot do.

Example:

Provider Skill mode CLI mode Good for Bad for
WebFetch Yes No unless host-backed Known URLs, readable pages Deep crawling, local-only use
Crawl4AI Yes Yes Normal sites, maps, JS pages Some protected retail/social sites
Browser/session Yes With CDP/profile Hard dynamic sites Scale, speed
PDF parser Yes Yes Investor docs Live dynamic pages
ATS provider Yes Yes Jobs Non-ATS custom pages without adapters

4. Legal/compliance posture cannot be afterthought

Scout should record:

  • Source URL.
  • Access method.
  • Timestamp.
  • Whether auth was used.
  • Whether data is public, user-provided, or provider-derived.

Scout should avoid:

  • Automated account actions.
  • Application submission.
  • Circumventing login walls.
  • Scraping personal data without a clear user-provided reason.

5. Social platforms need special treatment

LinkedIn and X should be provider-driven, not Crawl4AI-driven.

Scout can normalize social outputs, but it should not advertise raw scrape support unless a provider is validated.

6. "All attributes" needs a contract

Users will ask for "everything." Scout needs to translate that into:

  • Required fields.
  • Optional fields.
  • Enrichment fields.
  • Missing field report.

For products, "everything" might mean variants, colors, sizes, images, price, sale price, ratings, reviews, ingredients, benefits, availability, category breadcrumbs, and badges.

For investor pages, "everything" might mean reports, presentations, SEC filings, key metrics, risk factors, guidance, executive quotes, and event dates.

7. Partial success should be a first-class outcome

Most real web extraction is partial.

Scout should say:

50 records produced
43 complete
7 partial
3 PDP enrichment failures
2 blocked pages
1 provider unavailable

This is better than pretending success or failure is binary.

8. Output directories and run history matter

Scout should make every run findable.

Recommended structure:

scout-runs/
  2026-05-14_esteelauder_products/
    manifest.json
    records.json
    records.jsonl
    source_pages.json
    blocked_pages.json
    extraction_report.md
    raw/

9. The package should be modular from day one

Potential package extras:

  • scout-core
  • scout[browser]
  • scout[pdf]
  • scout[ats]
  • scout[social]
  • scout[api]
  • scout[all]

10. The skill should be a thin orchestrator

The Claude/Codex skill should not contain the business logic. It should:

  • Parse user intent.
  • Use host tools when useful.
  • Pass content into Scout core.
  • Return paths to artifacts.

The Python package should own schemas, extraction, enrichment, scoring, and artifact writing.

  1. Freeze the provider/result contract.
  2. Freeze base source/provenance schema.
  3. Implement content-ingest extraction API: saved markdown/html/DOM in, records out.
  4. Build product use case first because Estee gives a hard validation target.
  5. Add company/about leadership use case.
  6. Add investor/PDF use case.
  7. Add careers/ATS use case.
  8. Add evaluation harness and golden fixtures.
  9. Update docs/distribution around skill mode versus CLI mode.

Strategic North Star

Scout should answer:

Give me useful structured records from the web, using the best available acquisition channel, with provenance I can trust.