Scout Design Log — 2026-05-03

What Was Built

Phase 1 core library is complete. Five modes (scrape, crawl, extract, map, screenshot), ScoutCrawler unified entry point, all Pydantic contracts, and all three test layers.

Key Decisions Made This Session

1. Crawl4AI version is 0.7.7, not 0.8.x

The plan file said crawl4ai>=0.8.0 but the installed package is 0.7.7. Version corrected in version.py and pyproject.toml. All API surface verified against 0.7.7.

2. cast() for arun() return type

arun() returns CrawlResultContainer[CrawlResult]. pyright can't see attribute access through __getattr__ delegation. Solution: cast(CrawlResult, await crawler.arun(...)). No-op at runtime (CrawlResultContainer delegates all attributes to _results[0]). Preserves test compatibility (tests mock arun() to return a plain MagicMock — container[0] would have broken this).

3. arun_many() with stream=True

Without stream=True, arun_many() returns CrawlResultContainer (not async-iterable). Adding stream=True to CrawlerRunConfig returns AsyncGenerator[CrawlResult, None] which supports async for.

4. fit_markdown deprecated — use result.markdown.fit_markdown

result.fit_markdown on CrawlResult raises: "Please use 'markdown.fit_markdown' instead." Fixed across all modes using getattr(_md, "fit_markdown", None) or str(_md or "") or "" — works on real StringCompatibleMarkdown AND plain str mocks in tests.

5. FilterChain default is FilterChain(), not None

BFSDeepCrawlStrategy(filter_chain=None) fails type check. Always pass FilterChain() as default.

6. structlog approved over stdlib logging

structlog is strictly better (structured key=value, piped JSON). SOPs updated in mind: stdlib logging SOP targets the spirit (no print(), module logger, leveled). structlog satisfies all of this. No file logging needed for Phase 1 — stdout/docker logs sufficient.

7. Pydantic version field skipped on Scout models

CodingSOPs §7 says every model needs a version field. Skipped for Scout because: Scout is infrastructure, not a versioned public API. The version isolation mechanism is core/version.py (Crawl4AI pin), not Pydantic model fields. Deviation noted.

8. Methodology failure corrected

Code was built before Phase 1 thinking docs — corrected in this session. Workspace docs written (docs/workspace/scout-core/), SOP validation pass run, three test layers built. New feedback memory written: feedback_methodology_order.md.

Test State at Session End

Unit tests:       41  (mocked Crawl4AI, fast)
Contract tests:   19  (Pydantic shape validation)
Integration:       6  (real Crawl4AI, real network — ~15s total)
Total:            66  all passing

pyright:           0 errors
ruff:              clean

Integration tests at tests/integration/test_modes_live.py. Run with:

python3 -m pytest tests/integration/ -v  # ~15 seconds
python3 -m pytest tests/unit/ -v          # ~1.4 seconds

Open Questions Carried Forward

1. Integration test for /extract: Skipped — requires real LLM API key (Gemini). Manual verification needed during Docker smoke test.
2. Docker Playwright shm: Research Crawl4AI's own Dockerfile before writing ours. Need --shm-size=1g.
3. ScrapingBee fallback: Plan says 3-tier (httpx → Playwright → ScrapingBee). Crawl4AI may not natively support ScrapingBee as a fallback. Not needed for Phase 1 but will surface in PRISM integration.

Next Session Entry Point

Build scout/api/ — FastAPI service. Full spec at docs/workspace/scout-core/04-prd.md. Critical constraint: ScoutCrawler must be injected via Depends(), not constructed per-request (pre-mortem T5).