Scout Leadership Intel — Research + Technical Context

Created: 2026-05-05

Sources

• Scout skill/command file: ~/.claude/commands/scout.md
• Scout API source: /Users/arijitchowdhury/AI-Development/Scout/scout/core/types.py
• Scout project memory: ~/.claude/projects/-Users-arijitchowdhury-AI-Development-PIP/memory/project_scout_crawler.md
• Scout eval data: ~/.claude/scout-skill-workspace/iteration-2/eval-1-exec-team/

Scout Platform — What It CAN Do

Scout exposes 5 POST endpoints at http://localhost:8421, all authenticated via X-API-Key: dev-key:

Core integration pattern (Map → Filter → Scrape): 1. POST /map {"url": "https://company.com", "max_pages": 200} → discover all URLs 2. Filter urls[] for leadership patterns 3. POST /scrape {"url": "<best_match>", "use_js": true} → markdown content 4. If sparse: retry with {"formats": ["raw_html"]} → parse HTML card patterns

ScrapeRequest shape:

{
  "url": "https://apple.com/leadership",
  "formats": ["markdown"],
  "use_js": true,
  "wait_for": ".team-grid",
  "timeout_ms": 30000
}

ScrapeResponse fields relevant to this feature: - markdown: str — primary extraction target - raw_html: str — fallback for JS-card pages - links: list[str] — for LinkedIn URL harvesting - metadata.crawled_at: str — source timestamp for attribution - success: bool + error: str — for failure reporting - duration_ms: int — for latency tracking

ExtractRequest shape (LLM or CSS):

{
  "url": "https://apple.com/leadership",
  "css_schema": {
    "baseSelector": ".leadership-card",
    "fields": [
      {"name": "name", "selector": "h3", "type": "text"},
      {"name": "title", "selector": ".role", "type": "text"}
    ]
  }
}

Scout Platform — What It CANNOT Do

• Not a general-purpose scraper — optimised for 5 specific intel types only
• No parallel crawls — single-instance; sequential only
• No access to authenticated or login-gated pages
• ATS SPA job boards (Workday, Greenhouse React): JS XHR interception gap
• /extract with LLM requires LLM_API_KEY in Scout .env; CSS fallback works without it
• localhost URLs cannot be fetched via WebFetch (sandbox restriction — use curl/bash)

Real-World Performance (Eval Data)

From eval-1-exec-team on algolia.com (iteration 2): - 75% pass rate on exec extraction (3/4 assertions passed) - Map-first is mandatory — standard leadership URL paths can 404 - formats: ["raw_html"] is the reliable path for JS-heavy card layouts - Exec data from press releases (fallback) may be stale vs. current leadership page - Leadership URL patterns vary: /about/leadership, /about/team, /company/management — no single correct path

Scout Status

• Feature complete as of 2026-05-04
• 78/78 unit tests passing
• Source: /Users/arijitchowdhury/AI-Development/Scout/
• Start command: cd /Users/arijitchowdhury/AI-Development/Scout && python3 -m uvicorn scout.api.main:app --host 0.0.0.0 --port 8421
• Built on Crawl4AI (Apache 2.0) + Playwright