wiki/decisions/2026-05-14-scout-web-to-record-engine.md
ADR: Scout as Web-to-Record Engine
Date: 2026-05-14 Status: Accepted
Context
Scout started as a Crawl4AI-based crawler for company intelligence. During Estee Lauder product extraction testing, local Crawl4AI was blocked by category and PDP pages even in headed, stealth, persistent-profile, and realistic user-agent configurations. At the same time, hosted WebFetch/WebSearch and the in-app browser could read the same pages and expose usable product data.
This proved that "better crawler" is not the correct abstraction. Different acquisition channels work in different environments. The durable value is the processing layer after content acquisition.
Decision
Scout will be codified as a provider-agnostic web-to-record engine.
The core product will separate acquisition providers from extraction, enrichment, normalization, scoring, and artifact writing.
Provider Model
Scout should support this conceptual provider ladder:
| Provider | Where it fits | Notes |
|---|---|---|
| WebSearch | Skill-host discovery | Finds candidate URLs when URL is unknown. Not available inside plain pip package unless host provides it. |
| WebFetch | Skill-host fetch | Best for known URLs and pages that hosted retrieval can access. |
| Crawl4AI | Standalone default | Good for CLI/API/Docker crawling, sitemaps, normal websites, screenshots, JS pages. |
| Browser/session | Skill-host fallback | Uses current trusted browser session when a host app exposes it. |
| CDP/profile | Standalone trusted-browser mode | Attaches to a user-launched Chrome/Chromium session or profile. |
| Saved HTML/DOM | Offline/replay mode | Allows deterministic tests and user-provided pages. |
| PDF/document parser | Investor docs, reports, decks | Needed for annual reports, 10-K/10-Q, presentations, PDFs. |
| ATS provider | Careers/job extraction | Handles Greenhouse, Lever, Workday, Ashby, SmartRecruiters, custom job boards. |
| Social provider | LinkedIn/X/social signals | Should use official APIs, hosted connectors, Apify-like providers, or browser/session where appropriate. |
Rationale
- Web access is environment-dependent. A page may be blocked locally but accessible through hosted fetch or trusted browser.
- Most valuable use cases need structured records, not raw page reads.
- Scout must run both as a skill and as a standalone pip package.
- Provider abstraction prevents Crawl4AI-specific behavior from leaking into product extraction, investor extraction, job extraction, or company intelligence.
- A stable record/artifact layer makes Scout useful for Algolia indexing, PRISM evidence collection, job tooling, and future products.
Alternatives Considered
| Option | Why rejected |
|---|---|
| Keep Scout as Crawl4AI-first crawler | Fails on hard sites like Estee Lauder and makes the product too dependent on one transport. |
| Use only WebFetch/WebSearch | Works in hosted skill mode but not standalone CLI/API/Docker. Also lacks artifact discipline, schemas, and repeatable pipelines by itself. |
| Use only browser automation | Heavy, fragile, slower, and unnecessary for simple known-URL or hosted-fetch scenarios. |
| Build separate tools per use case | Faster short-term, but creates duplicate provider logic, inconsistent artifacts, and unshared test infrastructure. |
Consequences
- Scout core needs a
FetchProviderabstraction. - Extractors must accept normalized fetched content, not only Crawl4AI responses.
- Each use case needs schema contracts and golden tests.
- The CLI and skill can expose different providers while sharing the same core processing engine.
- Documentation must clearly distinguish skill-host capabilities from standalone package capabilities.