Scout — Feature Complete Session (2026-05-04)

What Shipped This Session

1. CSS Extraction Fallback (feat(extract))

Problem: /extract required an LLM API key. User challenged: "why does extract need an LLM API key? This is plain Crawl4AI extraction."

Decision: Keep LLM extraction as primary (correct for arbitrary pages with unknown DOM). Add JsonCssExtractionStrategy as fallback when: - No LLM_API_KEY configured AND - css_schema dict provided in the request

If neither is present → success=False with clear error message.

ExtractRequest schema updated: - extraction_schema now optional (default {}) - instruction now optional (default "") - css_schema: dict | None = None added

2. Scout CLI (feat(cli))

Typer-based CLI: scout scrape/crawl/map/extract/screenshot.

Why: e2e testing without curl, developer experience, and standalone use.

Entry point wired via pyproject.toml [project.scripts]. After pip install -e .:

scout scrape https://company.com/about
scout map https://company.com --pages 200
scout crawl https://company.com/careers --depth 2 --pages 30
scout extract https://company.com/team --llm-key $GEMINI_API_KEY

Crawl4AI attribution appears in CLI --help output (Apache 2.0 requirement).

3. README + License Compliance (docs)

Crawl4AI is Apache 2.0 (NOT MIT — a common mistake). Custom attribution clause requires:

"This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai), licensed under the Apache License 2.0."

Attribution placed in README.md and CLI --help.

4. Scout Skill Redesign

Old skill: API reference guide (which endpoint does what).

New skill: Company intelligence playbook. The user's core challenge: "Scout is NOT a random crawler. It is purpose-built for specific types of crawling."

Five intelligence use cases: 1. About / Company Overview 2. Executive / Leadership Team — incl. formats: ["raw_html"] for JS card pages 3. Hiring / Open Roles — incl. ATS detection (Workday, Greenhouse, Lever) 4. Investor Relations 5. Reports and PDF Documents

Full Company Profile pattern: map once (one /map call, max_pages=300) → scrape all 5 sections. Replaced the old "call one endpoint" pattern.

Installed at: ~/.claude/commands/scout.md Repo copy at: scout/skill/scout.md

5. /map Dict Normalisation Bug Fix (fix(map))

Bug: aseed_urls() returns list[dict] on some domains (shopify.com, algolia.com) instead of list[str]. The cast(list[str], ...) call was a no-op — dicts flowed into MapResponse(urls=dicts) and Pydantic rejected them.

Fix: Normalise immediately after aseed_urls() call:

raw_urls = [
    u["url"] if isinstance(u, dict) and "url" in u else str(u)
    for u in cast(list, raw)
    if u
]

Two new unit tests cover: dict-only list, mixed list.

Commit: 4cbd952

Eval Process (Skill Creator)

Ran skill-creator eval loop to validate the new Scout skill:

Iteration 1 (example.com): Wrong — tested generic scraping, not intelligence use cases. Abandoned.

Iteration 2 (algolia + adobe + shopify, single intelligence type per eval): User challenge: "I asked to do the same validation for Adobe.com and Microsoft.com. And you have only got the leadership data — what about investor, what about the career?"

Iteration 3 (Adobe + Microsoft, full 5-section profile per eval): - Adobe: 4/5 sections complete. Careers gap = Workday SPA (architecture limit, not Scout bug). - Microsoft: 4/5 sections complete. Same careers gap (React SPA). Everything else excellent.

Key finding from evals: Scraping exec team pages that are JS SPAs works best with formats: ["raw_html"] — bypasses sparse markdown conversion, returns full HTML for regex parsing. Added to skill.

Known Limitation Confirmed

ATS SPA job boards (Workday, Greenhouse React) load listings via authenticated XHR after page mount. Scout's use_js: true renders the page but cannot intercept the post-mount API call. Skill documents this: detect the ATS platform, note it, move on.

What Was Deliberately Not Done

• Docker smoke test end-to-end — not a feature-complete blocker. docker compose up + curl /health is straightforward, deferred.
• MCP server — Phase 3, future.
• intel-company Scout migration — intel-company still uses httpx/Jina Track 1.
• Temporal timeout update — intel-hiring activity timeout still 120s, should be 180s.