Scout — Platform Spec

What It Is

Scout is a self-hosted web intelligence platform built on Crawl4AI. It is the data acquisition layer for PRISM — replacing Apify — and a general-purpose company intelligence crawler embeddable in any tool (Claude via skill, CLI, N8N).

Scout is NOT a general-purpose scraper. It is purpose-built for specific company intelligence use cases: About page, executive team, hiring openings, investor relations, and PDF reports.

Why It Exists

• Replace Apify (SaaS billing, TypeScript FFI, vendor data access, no Python integration)
• Firecrawl rejected: AGPL-3.0 core, loses Fire-engine on self-host, TypeScript stack
• Crawl4AI: Apache 2.0, Python-native, stealth Playwright, LLM extraction built-in
• PRISM intel modules need a consistent, schema-validated crawling interface

License note: Crawl4AI is Apache 2.0 (NOT MIT). Attribution required per project custom requirement: "This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project, licensed under the Apache License 2.0." Attribution appears in README.md and CLI --help.

Endpoints

API Contracts (source of truth: scout/core/types.py)

All request/response shapes are Pydantic v2 models. No raw dicts cross boundaries.

Authentication: X-API-Key header. /health exempt. Missing/wrong key → 403.

ExtractRequest — dual-mode extraction: - LLM mode: schema (dict) + instruction (str) + llm_provider. Requires LLM_API_KEY env var. - CSS mode: css_schema (dict: {name, baseSelector, fields: [{name, selector, type}]}). No API key needed, fast. - If no API key AND no css_schema → returns success=False with clear error message.

MapResponse: urls: list[str] — always plain strings. aseed_urls normalisation applied server-side.

Architecture

scout/
  core/           ← Python library (PRISM imports directly — no HTTP)
    types.py      ← All Pydantic contracts
    version.py    ← Crawl4AI version pin (currently 0.7.7)
    crawler.py    ← ScoutCrawler: unified async entry point
    modes/        ← scrape, crawl, extract, map, screenshot
  api/            ← FastAPI service (external callers: Claude skill, CLI, N8N)
    main.py       ← App + lifespan (ScoutCrawler singleton)
    routers/      ← 5 endpoint routers + health
    middleware/   ← API key auth
    config.py     ← Pydantic Settings
  cli.py          ← Typer CLI (scout scrape/crawl/map/extract/screenshot)
  skill/          ← Claude Code skill (scout.md)
    scout.md      ← Company intelligence playbook (map→scrape pattern, 5 use cases)
  docker/         ← Dockerfile + docker-compose.yml

Key design decision: PRISM uses from scout.core import ScoutCrawler directly (zero HTTP overhead). FastAPI is for external callers only.

Critical Crawl4AI API Facts (verified against v0.7.7)

• aseed_urls(domain, config=SeedingConfig) → takes BARE domain string (not full URL); returns list[str] OR list[dict] depending on domain — normalise output unconditionally (extract u["url"] from dicts)
• crawler.arun(url, config=run_cfg) with deep_crawl_strategy=bfs in CrawlerRunConfig → returns AsyncGenerator[CrawlResult, None] in stream mode
• BFSDeepCrawlStrategy entry point: attach to CrawlerRunConfig(deep_crawl_strategy=bfs, stream=True) then call crawler.arun(url, config) — DeepCrawlDecorator intercepts arun() and dispatches BFS internally. Do NOT call arun_many() from outside — it bypasses the decorator.
• result.markdown → use str(result.markdown or "") to safely handle None
• result.links → {"internal": [{"href": "...", "text": "..."}], "external": [...]}
• FilterChain default: FilterChain() not None

Known Limitations

Claude Code Skill

scout/skill/scout.md — installed at ~/.claude/commands/scout.md.

The skill is a company intelligence playbook, not an API reference. It teaches: 1. Full Company Profile pattern: map once → scrape all 5 sections (About, Exec team, Hiring, Investor, PDFs) 2. formats: ["raw_html"] technique for JS-rendered team pages that return sparse markdown 3. ATS detection guidance (Workday, Greenhouse, Lever) 4. Map → Filter → Scrape pattern for single intelligence types

Phase Status

Repo

/Users/arijitchowdhury/AI-Development/Scout/ Branch: main Latest commits: - 4cbd952 fix(map): normalise aseed_urls output to list[str] for all domains - 2aee8a9 feat(cli): add Scout CLI with scrape/crawl/map/extract/screenshot commands - d14c248 docs: add README with Crawl4AI attribution and usage guide - 0d9aaa7 feat(extract): add CSS fallback when no LLM API key configured - e8b06bd refactor(core): use native Crawl4AI for map + crawl modes