Dashboard/PRISM/Decisions/2026-05-07-scout-subrouter-pattern.md
PRISM

Decisions/2026-05-07-scout-subrouter-pattern.md

ADR: Scout as Pipeline Sub-Agent Pattern

Date: 2026-05-07 Status: Partially Accepted (1A implemented; others pending design)

Context

Scout (crawl4ai + Playwright stealth) was originally built as a standalone intelligence tool. After adding stealth mode (2026-05-07), it reliably bypasses Cloudflare and Akamai — the two most common WAF vendors encountered in the audit pipeline. This opens Scout as a general-purpose crawl layer inside PRISM intelligence modules, not just a standalone tool.

The question is: should Scout be wired into individual modules as a sub-agent, and if so, how?

Decision

Scout will be integrated as a sub-agent inside select PRISM intelligence modules using the following pattern:

Module script (collect-*.py)
  → calls scout_company.run(url, output_dir) or scout_company.scrape_url(url)
  → Scout returns structured JSON (not raw markdown)
  → Module's pre-filter layer extracts signal keywords/fields
  → Only the filtered signal (not the full crawl) is passed to LLM enrichment

Scout is NOT a replacement for WebSearch or WebFetch. It is the crawl layer for pages that block those methods (Akamai, Cloudflare, JS-rendered pages, ATS portals).

Rationale

  1. The WAF problem is real and pervasive. SimilarWeb API expired, BuiltWith returns JS loading state, basic WebFetch fails on ATS portals, IAG's investor page was Cloudflare-blocked. Scout stealth resolves all of these.

  2. The data volume problem requires a pre-filter layer. Raw crawl output cannot go directly to LLM — context burns. The filter layer (keyword matching, CSS schema extraction, token budgets) must be defined per module before implementation.

  3. Module 1A is the proof of concept. Scout Step 4 in collect-company.py is live, tested, and working. It fills null fields without overwriting existing data. This pattern is the template for all other modules.

Alternatives Considered

Option Why rejected
Replace WebFetch entirely with Scout Over-engineering. WebFetch is faster and sufficient for non-WAF pages. Scout adds latency (2-5s stealth mode). Use Scout only where needed.
Use Scout for everything in parallel Would burn Crawl4AI compute and slow pipelines. Sequential, targeted calls are better.
Apify for all crawling Expensive per-run costs. Scout is self-hosted, unlimited.

Consequences

  • scout_company.py is the shared Scout adapter. Modules import it rather than calling Scout API directly.
  • Each module that adopts Scout must define: (a) which URLs to crawl, (b) token budget for crawl content, (c) pre-filter keywords/schema, (d) fallback behavior when Scout returns sparse content.
  • The pre-filter layer must be implemented BEFORE any new Scout integration ships. Raw markdown → LLM is explicitly prohibited.
  • Cache TTL per content type must be defined before 1H or 1D integration.

Next Design Steps (before implementing 1H or 1D)

  1. Define the ScoutSignal canonical output schema — compact JSON, not raw markdown
  2. Define per-module token budgets
  3. Define the pre-filter layer (keyword regex vs CSS schema vs embedding similarity)
  4. Define cache TTLs per content type (JDs: 7 days, newsrooms: 1 day, IR pages: 30 days)
  5. Define tiered model routing (Haiku for filtering, Sonnet for synthesis)