Dashboard/Scout/wiki/overview.md
Scout

wiki/overview.md

Scout - Overview

Last Updated: 2026-05-14

One-Sentence Product Definition

Scout is a provider-agnostic web-to-record intelligence engine that discovers, fetches, extracts, enriches, normalizes, scores, and writes structured records from web content.

The Important Pivot

The old definition was:

Scout is a self-hosted web intelligence platform built on Crawl4AI.

That is too narrow.

The better definition is:

Scout is an extraction and normalization engine with multiple acquisition providers.

Crawl4AI remains useful, but it is only one provider. WebFetch, WebSearch, current browser sessions, CDP-attached Chrome, saved HTML, PDF parsers, ATS APIs, and social providers can all feed Scout.

Why This Matters

The Estee Lauder test exposed the core architectural truth:

  • Local Crawl4AI can be blocked by protected retail pages.
  • Hosted WebFetch/WebSearch can sometimes read those same pages.
  • The in-app browser can sometimes access those same pages through a trusted session.
  • The useful output is not the fetched page itself. The useful output is a structured record ready for Algolia, PRISM, a job tool, a research report, or another downstream system.

Therefore, Scout should not compete with WebFetch as "the better fetcher." Scout should sit after any fetcher and turn messy content into reusable data.

Product Layers

User intent
  |
  v
Intent parser
  |
  v
Discovery layer
  - known seed URLs
  - WebSearch
  - sitemap
  - crawl
  - provider-specific discovery
  |
  v
Fetch provider layer
  - WebFetch host provider
  - Crawl4AI
  - in-app browser/session
  - CDP/profile
  - saved HTML/DOM
  - PDF/document parser
  - ATS provider
  - social provider
  |
  v
Extractor layer
  - product extractor
  - company page extractor
  - executive extractor
  - investor document extractor
  - job posting extractor
  - docs/content extractor
  - social signal extractor
  |
  v
Enrichment and merge layer
  - listing + PDP merge
  - page + PDF merge
  - company + investor + hiring merge
  - deduplication
  - confidence scoring
  |
  v
Record layer
  - Algolia product record
  - executive record
  - investor document record
  - financial metric record
  - job posting record
  - social post record
  - generic page fact record
  |
  v
Artifacts
  - records.json
  - records.jsonl
  - manifest.json
  - source_pages.json
  - blocked_pages.json
  - extraction_report.md

Personas

Arijit / Builder

Needs Scout to run locally, as a skill, and as a reusable package. Wants fast experiments, clear artifacts, and no silent failures.

PRISM / Algolia Prospect Research

Needs Scout to collect evidence from prospect websites: company pages, leadership pages, investor pages, careers pages, product catalogs, newsrooms, and social/provider outputs.

Search / Merchandising / Ecommerce User

Needs Scout to extract product records from retailer sites and prepare them for Algolia indexing.

Job Applicator / Career Agent

Needs Scout to discover job openings, normalize job posts, track apply URLs, and match roles to user criteria.

Future App Developer

Needs Scout as a pip-installable library and CLI with stable schemas, testable providers, and predictable artifacts.

What Scout Does Extra Beyond WebFetch

WebFetch reads a page. Scout turns that page into data.

Scout adds:

  • Field extraction: names, titles, prices, reviews, categories, locations, metrics, documents.
  • Normalization: currency, numbers, dates, URLs, categories, schema names.
  • Enrichment: listing + detail page, company + investor + hiring, HTML + PDF.
  • Deduplication: one best record per product, person, job, document, or page fact.
  • Quality scoring: completeness, confidence, provenance coverage.
  • Artifacts: outputs that can be indexed, audited, rerun, and debugged.
  • Repeatability: the same request produces consistent record shapes regardless of provider.

What Scout Should Not Become

Scout should not become:

  • A brittle one-off scraper pile.
  • A browser automation toy that clicks randomly through sites.
  • A replacement for official APIs where official APIs are the right provider.
  • A social-media scraper that pretends LinkedIn/X can be reliably crawled without provider support.
  • A final analyst that invents conclusions without evidence.

Scout should be the evidence engine. Synthesis can sit on top.