Dashboard/scout-leadership-intel/02-requirements.md
scout-leadership-intel

02-requirements.md

Scout Leadership Intel — Spec

Generated: 2026-05-05

Problem Statement

Account executives and BDRs doing pre-call research must manually navigate 2–4 pages of a company's website to find decision-makers — a repetitive, time-consuming task that interrupts sales workflow. There is no single command to ask "who leads this company?" and get a structured, sourced answer. The consequence is either skipped research (entering calls underprepared) or wasted time (5–10 minutes per company across dozens of accounts).

Research Sources + Technical Context

Source: Scout skill file at ~/.claude/commands/scout.md and Scout API source at /Users/arijitchowdhury/AI-Development/Scout/

Full research notes: docs/workspace/scout-leadership-intel/00-research.md

Key capabilities leveraged

Scout exposes 5 POST endpoints at http://localhost:8421, all authenticated via X-API-Key: dev-key:

Endpoint Use in this feature
/health Pre-flight check before any call
/map Discover all URLs on a company domain (returns urls: list[str])
/scrape Fetch a single page as markdown or raw HTML
/extract Structured extraction via CSS schema or LLM instruction

Core integration pattern (Map → Filter → Scrape): 1. POST /map with {"url": "https://company.com", "max_pages": 200} → get full URL list 2. Filter urls[] for leadership patterns: /team, /leadership, /management, /executives, /about/team, /about/leadership, /company/leadership 3. POST /scrape with {"url": "<best_match>", "use_js": true} → get markdown content 4. If markdown is sparse: retry with {"formats": ["raw_html"]} and parse name/title card patterns from HTML

Key constraints discovered in research: - Scout is single-instance — no parallel crawls - Cannot access authenticated or login-gated pages - ATS SPA job boards (Workday, Greenhouse) fail — not relevant here, but same JS-interception gap could affect some team pages - /extract with LLM requires LLM_API_KEY in Scout .env; fallback is css_schema mode - Real-world eval (Algolia.com): 75% pass rate on exec extraction; raw_html fallback is the reliable path for JS-heavy card layouts - Leadership page URLs vary wildly across companies — map-first is mandatory - Timeout: /scrape default 30s, configurable via timeout_ms; /map up to 60s

Data model for exec output (from Scout skill):

## Executive Team — [Company]
| Name | Title |
|---|---|
| Jane Smith | CEO |

Response contract: - ScrapeResponse.markdown: str — primary extraction target - ScrapeResponse.raw_html: str — fallback for JS-card pages - ScrapeResponse.links: list[str] — for LinkedIn URL harvesting - ScrapeResponse.metadata.crawled_at — source timestamp for attribution - ScrapeResponse.success: bool + error: str — for failure reporting - ScrapeResponse.duration_ms: int — for latency tracking

Personas

Persona A — Account Executive (AE) - Job-to-be-done: Before a discovery call, identify the economic buyer, champion, and technical decision-maker at a target account - Pain: Spends 5–10 minutes per account manually navigating websites; results are inconsistent quality - Critical need: Title precision — "VP of Engineering" vs "SVP Engineering" matters for call strategy

Persona B — Business Development Representative (BDR) - Job-to-be-done: Build prospecting lists quickly; find any named executive to cold-reach at scale across many companies per day - Pain: Manual research doesn't scale across a large account list - Critical need: Volume — more names across more companies per hour, even with lower precision on titles

Value Props per Persona (Features-for-Whom)

Persona A — AE: | Feature | Value delivered | |---|---| | Map → Scrape → structured table | Get decision-maker names and titles from the authoritative source (the company's own website) in one command, not 5 clicks | | Source attribution on every result | Know whether to trust the data — "sourced from apple.com/leadership at 2026-05-05" vs "fallback: press release" changes call prep | | Partial results with fallback note | Still get something useful even when the primary page fails; AE can judge whether to dig further | | Error message with URLs tried | When it genuinely fails, know what was attempted so manual investigation has a starting point |

Persona B — BDR: | Feature | Value delivered | |---|---| | Single-command exec list | Research 10+ companies per hour instead of 3; scales prospecting velocity | | Name + title table (any source) | Any executive name beats zero names for cold outreach; source label tells BDR if they should verify first | | Written output file | Leave result files in workspace; reference later without re-running |

Functional Requirements

Must-Have (MVP)

  • The skill shall accept a company domain (e.g., apple.com) as input and return a leadership table with at minimum: name, title, and source attribution
  • When a domain is provided, the skill shall call Scout /health to verify the service is running before any other API call
  • When Scout is unreachable, the skill shall return a clear error message: "Scout is not running at localhost:8421. Start it with: cd /Users/arijitchowdhury/AI-Development/Scout && python3 -m uvicorn scout.api.main:app --host 0.0.0.0 --port 8421"
  • The skill shall call /map on the domain first (max_pages: 200) to discover the URL structure — it shall not guess or hardcode leadership URL patterns without mapping first
  • When /map returns a URL matching leadership patterns (/team, /leadership, /management, /executives, /about/team, /about/leadership, /company/team, /company/leadership, /company/management), the skill shall scrape that URL
  • When the primary scrape returns sparse markdown (fewer than 100 words), the skill shall retry with formats: ["raw_html"] and parse name/title card data from the HTML
  • The skill shall output a markdown table with columns: Name, Title, Source
  • The skill shall label every row's source as either the page URL (if scraped from leadership page) or "Fallback: [source]" (if extracted from press release or other non-authoritative page)
  • When no leadership page is found after mapping, the skill shall return: "No leadership page found for [domain]. URLs tried: [list]" — it shall not return an empty result silently
  • When a partial result is available (execs found from a non-primary source), the skill shall return those results with a visible warning: "WARNING: data sourced from [fallback source], not a dedicated leadership page — may be incomplete or stale"
  • The skill shall write output to {company-slug}-leadership.md in the current workspace directory
  • The skill shall print elapsed time and source URL in the output footer
  • The total skill execution time shall target under 60 seconds for p90 of runs

Should-Have (fast-follow)

  • When a LinkedIn URL is found in ScrapeResponse.links matching the executive's name context, the skill shall include it as an optional fourth column
  • The skill shall support a company name as input (e.g., "Apple") in addition to a domain, resolving it to a domain via a simple search heuristic
  • The skill shall support a --raw flag to output JSON instead of markdown table (for programmatic consumption)

Nice-to-Have (future)

  • Batch mode: accept a list of domains and run sequentially, producing one file per company
  • Short bio extraction (1–2 sentences per executive)
  • Confidence score per executive row (leadership page = high, press release = medium, inferred = low)
  • Cron/scheduled batch runs against a pipeline list

UX Flow

Entry Point

Two invocation paths: 1. Slash command: /scout-leadership apple.com 2. Natural language in session: "get me the leadership team for Apple" → Claude routes to this skill

Primary Flow (happy path)

User: /scout-leadership apple.com
  │
  ▼
[1] Health check — GET /health
  → if DOWN: print startup command, exit
  │
  ▼
[2] Map domain — POST /map {"url": "https://apple.com", "max_pages": 200}
  → get urls[] list (may take 10–45s)
  │
  ▼
[3] Filter urls[] for leadership patterns
  → rank: /leadership > /executives > /team > /management > /about/* variants
  → if no match: log "no leadership URL found", proceed to fallback
  │
  ▼
[4a] Scrape best-match URL — POST /scrape {"url": "...", "use_js": true}
  → if markdown word_count < 100: go to [4b]
  → if scrape fails (success=false): go to fallback
  │
[4b] Retry with raw_html — POST /scrape {"url": "...", "use_js": true, "formats": ["raw_html"]}
  → parse HTML for card patterns (h3/h4 names, role/title spans)
  │
  ▼
[5] Structure output
  → Build markdown table: | Name | Title | Source |
  → If LinkedIn links found in ScrapeResponse.links: add | LinkedIn |
  → Attach source label per row
  │
  ▼
[6] Write file: ./{company-slug}-leadership.md
  │
  ▼
[7] Print to chat:
  ## Leadership Team — Apple (apple.com)
  | Name | Title | Source |
  |------|-------|--------|
  | Tim Cook | CEO | apple.com/leadership |
  ...
  Source: https://www.apple.com/leadership/
  Scraped: 2026-05-05T14:32:00Z | Duration: 38s

Fallback Flow (no leadership page found)

[3] Filter returns no match
  │
  ▼
[F1] Attempt common fallbacks: /about, /company, /about-us
     → scrape and scan for executive mentions
  │
  ▼
[F2] If execs found in fallback:
     → return table with WARNING banner: "Data from [fallback URL], not a dedicated leadership page"
  │
  ▼
[F3] If nothing found:
     → return: "No leadership page found for apple.com. URLs checked: [list]. Try: apple.com/leadership manually."

Back Navigation / Dead Ends

  • If Scout is DOWN: exit immediately with startup instructions — no retry loop
  • If /map times out (>60s): return error, suggest --max-pages 50 for faster mapping
  • If scrape returns 403: note bot detection, suggest manual verification
  • No infinite retry loops — maximum 2 attempts per URL (primary + raw_html fallback)

Acceptance Criteria

Health check: - When the skill is invoked, the system shall call GET http://localhost:8421/health before any other Scout API call - If Scout returns a non-200 response or connection refused, then the system shall display the startup command and exit without making further API calls

Map-first enforcement: - The system shall call /map on the target domain before attempting any /scrape call - If /map returns success: false, then the system shall log the error and attempt scraping fallback URLs in priority order

Source attribution: - The system shall label every executive row's source with the URL it was scraped from and the crawled_at timestamp - If data is sourced from a non-leadership page, then the system shall display a WARNING label distinguishing it from authoritative leadership page data

Failure transparency: - If no leadership page is found after mapping and all fallback attempts, then the system shall return a list of all URLs attempted — it shall not return an empty result or generic error - While partial results are available (some execs found, leadership page incomplete), the system shall return those partial results with an explicit completeness warning

Output format: - The system shall always write results to {company-slug}-leadership.md in the current working directory - The system shall always print results to the chat session (markdown table format)

Performance: - The system shall complete the full Map → Scrape → Extract pipeline in under 60 seconds for p90 of runs against Fortune 500 company domains - Where Scout returns duration_ms in the response, the system shall display total elapsed time in the output footer

JavaScript rendering: - When the initial /scrape returns markdown with fewer than 100 words, the system shall retry using formats: ["raw_html"] and parse the HTML for executive card patterns - The system shall always include "use_js": true in all /scrape calls to handle JS-rendered pages

Data minimalism: - The system shall only source data from the target company's own public website — it shall not call LinkedIn, third-party APIs, or any authenticated resource

Success Metrics

Framework: HEART

Metric Definition Target
Task Success Rate % of queries against Fortune 500 companies that return ≥3 verified executives from a leadership/team page ≥80%
Time-on-Task (p90) Elapsed seconds from command invocation to result printed in chat ≤60s
Adoption Number of /scout-leadership invocations per week across AEs + BDRs Baseline in week 1; target 10+/week by week 4
Error transparency % of failed runs that return an explicit "here's what I tried" message (vs silent empty) 100%

North Star: An AE can walk into any discovery call having queried the skill for the company, confident they know the top 3 decision-makers' names and titles, in under 60 seconds.

MVP Scope

The following ships in v1:

  1. Scout health check pre-flight
  2. /map domain → URL discovery
  3. Leadership URL pattern matching and priority ranking
  4. /scrape with use_js: true primary attempt
  5. Raw HTML fallback when markdown is sparse
  6. Markdown table output: Name, Title, Source
  7. Source attribution label per row (authoritative vs fallback)
  8. Partial results with WARNING banner
  9. Explicit failure message with URLs-tried list
  10. File write to {company-slug}-leadership.md
  11. Elapsed time + source URL in output footer

Out of Scope

  1. LinkedIn scraping — ToS violation risk; explicitly excluded
  2. Batch / scheduled mode — sequential multi-company runs deferred to fast-follow
  3. Executive bios — text extraction beyond name + title is fast-follow
  4. Photos or avatars — not requested, excluded from MVP and fast-follow
  5. Authenticated pages — Scout cannot access login-gated leadership portals; behavior is to log and skip
  6. LLM-based /extract endpoint — CSS schema fallback only for MVP to avoid LLM_API_KEY dependency; LLM extraction is fast-follow
  7. BDR-specific output mode — different output density for BDR vs AE is post-MVP; both personas get the same table in v1
  8. Company name resolution — accepting "Apple" and resolving to "apple.com" automatically is fast-follow; MVP requires a domain as input

Risks + Assumptions

  • [Assumption — confirm: apple.com and similar large-company sites render their team pages via JavaScript and will require use_js: true to return meaningful content]
  • [Assumption — confirm: Scout is running and accessible at localhost:8421 when the skill is invoked; skill does NOT auto-start Scout]
  • [Risk — mitigate: Leadership page URLs vary widely across companies; the map-first pattern is mandatory — hardcoding URL guesses leads to 404s (observed in eval data)]
  • [Risk — mitigate: Some companies (including Algolia, from eval data) do not have a dedicated /leadership or /team page — press release fallback may be the only source, and it may be stale]
  • [Assumption — confirm: X-API-Key: dev-key is the correct API key for Scout in this environment; production may require a different key]
  • [Risk — mitigate: Scout's single-instance constraint means if a previous long /map or /crawl call is in progress, this skill may timeout or return incomplete data]
  • [Assumption — confirm: The current working directory when the skill runs is the correct destination for {company-slug}-leadership.md]

Open Questions

  1. Should the skill auto-start Scout if the health check fails, or only print the startup command?
  2. For the LinkedIn URL bonus feature (fast-follow): should it only include LinkedIn if it appears in the scraped page's own links, or is a LinkedIn search heuristic acceptable?
  3. Is dev-key the correct API key for all environments (local dev, CI, team members)? Should this be an env var?
  4. When the map returns 100+ URLs and multiple match leadership patterns (e.g., /about/team AND /about/leadership AND /company/management), what ranking heuristic should determine the primary scrape target?
  5. Should the output file be written to the current working directory, a fixed workspace path, or a configurable path?

Dependencies

  • Scout at http://localhost:8421 — must be running before skill invocation
  • Scout API Key — X-API-Key: dev-key (confirm env var strategy)
  • Scout source at /Users/arijitchowdhury/AI-Development/Scout/
  • Python/uvicorn — required to start Scout if not running
  • Crawl4AI + Playwright — Scout's underlying stack; Playwright handles JS rendering and stealth mode
  • Claude Code skill system — skill will be installed at ~/.claude/skills/scout-leadership/ or as a command at ~/.claude/commands/