Scout — Architecture Overview

What Scout Is

A self-hosted universal web acquisition and evidence engine built on Crawl4AI. Current product surfaces:

1. HTTP API — FastAPI at localhost:8421. Endpoints include /scrape, /extract, /crawl, /products, /structure, /harvest, /map, /screenshot, /run/{use_case}, hosted beta routes, and launch website routes.
2. Python package and CLI — installable package with scout command.
3. Python library — from scout.core import ScoutCrawler. Direct import.
4. Docker service — self-hosted API with volume-backed artifacts.
5. Claude/Codex skill backend — agents can use the local Scout server/package as an acquisition tool.
6. Launch website — served by scout serve at /, with /quickstart, /guide, /examples, /pricing, /status, /beta, /legal, /terms, and /privacy.

Scout's private-beta promise is evidence-first acquisition and record preparation. It is not a generic "crawler beats every crawler" claim and it is not a certified app UI product.

Current Launch State

• Private beta: ready_with_limits.
• Public launch: blocked.
• Main blockers: license approval, final license files/metadata, public pricing and hosted limits, registry publishing policy, Crawl4AI/lxml risk decision, real Stripe test-mode smoke, final terms/privacy.
• Launch evidence: repo: docs/product/launch-evidence-index-2026-06-29.md.

Component Map

scout/
├── core/                    # Engine layer (no web framework)
│   ├── crawler.py           # ScoutCrawler — main entry point
│   ├── types.py             # ALL Pydantic contracts
│   ├── acquisition.py       # Escalation spine (T0→T1→T2→human)
│   ├── blocking.py          # Block detection (P1)
│   ├── capture_extract.py   # structure_capture() — raw:// scheme
│   ├── cdp_acquire.py       # acquire_open_page() — CDP-attach
│   ├── human_assisted.py    # scout unblock CLI
│   ├── crawl_from_here.py   # Multi-page crawl from cleared session
│   └── live_browser.py      # CDP screencast engine
│
├── api/                     # HTTP layer (FastAPI)
│   ├── main.py              # App entry + router mounting
│   ├── deps.py              # get_crawler() DI
│   ├── config.py            # Settings
│   ├── middleware/auth.py    # API key auth
│   ├── frontend.py          # Main /app UI
│   ├── live_browser_page.py # /app/live-browser console UI
│   └── routers/
│       ├── scrape.py        # POST /scrape
│       ├── extract.py       # POST /extract
│       ├── crawl.py         # POST /crawl
│       ├── products.py      # POST /products
│       ├── structure.py     # POST /structure (PRISM)
│       ├── harvest.py       # POST /harvest (PRISM)
│       ├── map.py           # POST /map
│       ├── screenshot.py    # POST /screenshot
│       ├── run.py           # POST /run/{use_case}
│       ├── app_browser.py   # Native browser open/capture
│       ├── live_browser.py  # WebSocket bridge for screencast
│       └── ...
│
├── cli/                     # CLI entry points
│   └── ...                  # scout scrape, scout unblock, scout serve
│
└── skill/
    └── scout.md             # Claude Code skill definition

Launch Website

scout serve now serves the private-beta website:

• / — homepage.
• /quickstart — local install, Docker, workdir, first run.
• /guide — human developer guide for choosing local, hosted, HTTP, CLI, and skill usage.
• /examples — beta-safe workflow examples and artifact expectations.
• /pricing — local-free and hosted-metered posture.
• /status — private beta and public launch readiness.
• /beta — local-vs-hosted beta path and checkout form.
• /legal, /terms, /privacy, /third-party-notices — beta legal boundaries and attribution.

Swagger remains at /docs; the human guide is /guide.

Acquisition Escalation

T0 (vanilla fetch) → T1 (stealth: proxy/UA/nav-patch/delay) → T2 (headless browser) → Human-assisted (native Chrome + human solve)

Block detection (blocking.py) identifies the wall vendor and escalates. Human-assisted is the terminal rung for behavioral walls (PerimeterX, DataDome).

Key Contracts

All in scout/core/types.py: - ScrapeResult — standard scrape response - CrawlResult — recursive crawl - CaptureExtraction — structured output from structure/harvest (markdown + records) - BlockInfo — vendor + challenge detection

Related

• index