wiki/decisions/2026-05-16-provider-agnostic-run-architecture.md
ADR: Provider-Agnostic Run Architecture
Date: 2026-05-16 Status: Accepted
Context
Scout started as a Crawl4AI-backed scraper and Claude/Codex skill. The product direction expanded: it must remain independently installable by pip install, continue to work as an agent skill, support blocked-site fallback channels, and serve multiple use cases beyond product scraping: Job Hunter, PRISM company intelligence, investor research, careers monitoring, website quality research, documentation extraction, news/social/location normalization, and general web-to-record extraction.
A crawler-only design is too narrow. Some sites work through Crawl4AI; some work better through host WebFetch/WebSearch; some require an authenticated or in-app browser session; some sources are documents, PDFs, ATS feeds, saved HTML, or social/provider imports. The common value of Scout is not only fetching pages. It is converting messy source evidence into clean, typed, reusable records with provenance and portable artifacts.
Decision
Scout will use a provider-agnostic run architecture: high-level use cases call a shared run orchestration layer, providers return normalized source evidence, vertical processors convert evidence into typed Pydantic records, and every run writes the same artifact contract.
Rationale
- Keeps Scout standalone and skill-compatible: the CLI can use Crawl4AI, files, CDP/browser attach, and future adapters; the skill can add host WebFetch/WebSearch or in-app browser sessions when available.
- Avoids making browser automation the default. Browser fallback is a secondary channel after regular acquisition is blocked or insufficient.
- Gives every vertical one output shape: records, source pages, blocked pages, validation findings, report, and manifest.
- Makes outputs easy to index into Algolia, inspect manually, or feed into PRISM and future automation jobs.
- Lets each use case own its schema without contaminating the raw crawler layer.
Architecture Implications
scout.core.platformowns run contracts, providers, artifacts, content normalization, registry, and high-level run dispatch.scout.core.use_casesowns domain contracts and processors such as products, jobs, and PRISM.scout run <use-case>is the stable CLI surface for multi-step workflows.- All run outputs must be written to a user-selected or explicit output directory, never hidden in
/tmpby default. - Records must be Pydantic models at boundaries; raw dicts should remain internal only at serialization edges.
Alternatives Considered
| Option | Why Rejected |
|---|---|
| Crawl4AI-only crawler | Too brittle for blocked sites and too narrow for PDFs, ATS feeds, host webfetch, browser sessions, and future adapters. |
| Agent-skill-only Scout | Would make Scout dependent on Claude/Codex host tools and break standalone distribution. |
| Browser-first scraping | Too heavy, slower, harder to run headlessly, and not appropriate as default behavior. Browser remains a fallback or explicit provider. |
| Separate tool per use case | Duplicates provider handling, artifact writing, validation, and provenance logic. |
Consequences
- Build order should prioritize reusable platform contracts before vertical depth.
- Each vertical should add tests at three levels: contract, processing, and CLI/run artifact behavior.
- Product extraction, Job Hunter, and PRISM should converge on the same run artifact contract.
- Live extraction for hard sites must be treated as provider selection and evidence quality, not only crawler tuning.