Scout Full Intelligence Platform Architecture

Date: 2026-05-20

Architecture Summary

Scout has three front doors and one core engine:

• CLI: scout run <use-case> --mode <mode>
• HTTP: POST /run/{use_case}
• Skill: Claude/Codex playbook that can use local Scout plus host tools

The core engine receives a RunRequest, chooses an execution mode/provider ladder, normalizes evidence, runs a vertical processor, and writes artifacts.

flowchart TD
    User["User / Agent / Scheduled Job"]
    CLI["CLI: scout run"]
    HTTP["HTTP: /run/{use_case}"]
    Skill["Claude/Codex Skill"]
    Request["RunRequest: use_case, mode, query, targets, output_dir"]
    Ladder["Execution Mode Router"]
    Crawl4AI["crawl4ai"]
    API["api adapters"]
    Saved["saved replay"]
    WebFetch["host webfetch"]
    WebSearch["host websearch"]
    Browser["browser fallback"]
    Evidence["Source Evidence + Provenance"]
    Processors["Vertical Processors"]
    Records["Typed Records"]
    Artifacts["manifest, records, sources, blocked, validation, report"]
    Consumers["Algolia, PRISM, job monitor, reports, agents"]

    User --> CLI
    User --> HTTP
    User --> Skill
    CLI --> Request
    HTTP --> Request
    Skill --> Request
    Request --> Ladder
    Ladder --> Crawl4AI
    Ladder --> API
    Ladder --> Saved
    Ladder --> WebFetch
    Ladder --> WebSearch
    Ladder --> Browser
    Crawl4AI --> Evidence
    API --> Evidence
    Saved --> Evidence
    WebFetch --> Evidence
    WebSearch --> Evidence
    Browser --> Evidence
    Evidence --> Processors
    Processors --> Records
    Records --> Artifacts
    Artifacts --> Consumers

Execution Mode Ladder

auto means:

1. crawl4ai
2. api
3. saved
4. webfetch
5. websearch
6. browser

Browser is intentionally last because it is host-dependent and expensive, but it is necessary for hard sites and JS-shell pages.

Artifact Contract

Every high-level run writes:

• manifest.json
• records.json
• records.jsonl
• source_pages.json
• blocked_pages.json
• validation.json
• extraction_report.md

Diagram Prompt For Image Generation

Create a polished system architecture diagram for Scout, a provider-agnostic web intelligence platform. Show three front doors on the left: CLI, HTTP app, Claude/Codex skill. They flow into a RunRequest and Execution Mode Router. The router connects to provider modes: crawl4ai, webfetch, websearch, browser fallback, saved replay, api adapters. These flow into normalized source evidence with provenance, then vertical processors: company, PRISM, investor, careers, jobs, products, news, research. Then typed records and artifact outputs: manifest.json, records.json, records.jsonl, source_pages.json, blocked_pages.json, validation.json, extraction_report.md. Downstream consumers on the right: Algolia, PRISM, job monitor, reports, agents. Style: clean technical architecture, white background, subtle blue and green accents, legible labels, professional SaaS diagram.