PRISM

Research/Scout-Pipeline-Integration-Strategy.md

Scout Pipeline Integration Strategy

Date: 2026-05-07 Status: Discovery — Not yet implemented beyond Module 1A Context: Strategic analysis from audit session (BA + LBP). Scout stealth mode confirmed working. Integration into 1A (company context) shipped. All other modules still pending evaluation.

What We Proved This Session

Scout (crawl4ai + Playwright stealth) now bypasses Cloudflare and Akamai after adding enable_stealth=True, simulate_user=True, magic=True to Crawl4AI's BrowserConfig and CrawlerRunConfig.

Confirmed live: - IAG investor page (Cloudflare): 0 words without stealth → 714 words with stealth ✓ - LBP homepage (Akamai): 0 URLs without stealth → 220 links with stealth ✓ - LBP executive board page: personal LinkedIn URLs extracted directly ✓ - LBP annual results: 2 PDF download links surfaced ✓

Where Scout Is Already Wired In

Module	Integration	What It Adds
1A Company Context	Step 4 of collect-company.py	website field, linkedin_url, twitter_handle, careers_url confirmation
1G Investor Intel	Ad hoc (IAG Cloudflare bypass)	Annual report pages, IR PDFs

Where Scout Should Be Added (Prioritised)

1. 1H Hiring Signals — Highest ROI

Current problem: v3.0 removed Apify and uses basic WebFetch + WebSearch. Most ATS portals (Workday, Greenhouse, Lever, Taleo) block basic requests. Job titles are found but full JD text is missed.

Scout's value: Full job description text mining. A single JD saying "experience migrating from Apache Solr to a modern NLP search platform" is a stronger signal than 20 job titles. Scout with stealth can crawl ATS portals and return full JD content.

Implementation pattern: Scout → full JD markdown → keyword extraction (pre-LLM filter) → signal classification.

⚠️ Context risk: 50 JDs × 2,500 words = 125K words → needs pre-filtering before LLM. See Open Questions.

2. 1D Competitor Intelligence — Strategic Value

Current problem: SimilarWeb API expired (401), BuiltWith returns JS loading state. Competitor search vendors are tagged "Unknown / Phase 2 needed."

Scout's value — three distinct angles:

A. Competitor careers pages → what technology are they hiring for? - If Lufthansa has 12 "Elasticsearch Engineer" roles: displacement signal for BA pitch - If Delta is hiring "Algolia Platform Engineer": golden angle (existing Algolia customer)

B. Competitor newsrooms/press → what are they announcing? - "Lufthansa launches AI-powered flight search" = competitive pressure signal for BA - Conference talks, product launches, technology partnerships

C. Vendor case study cross-reference → when BuiltWith says "competitor uses Coveo", Scout crawls Coveo's website for a case study on that competitor, extracts the claimed metrics, feeds into competitive gap analysis

⚠️ Context risk: Competitor newsrooms can have 100+ press releases. Need URL-level filtering before full crawl. See Open Questions.

3. 1J News Signals — Moderate Improvement

Current approach: Google News RSS (3 queries, 60-day lookback). Works but misses newsroom-direct items and trade press that isn't yet indexed.

Scout's value: Direct crawl of company newsrooms (/press, /newsroom, /news). Faster than Google indexing. Also: trade press sites (The Register, PhocusWire, Retail Dive) that soft-paywall certain articles — Scout with stealth can often retrieve them.

⚠️ Context risk: Lower than 1H/1D. Newsroom pages are typically 500-2000 words each. Manageable.

4. 1E/1F Financial — Targeted PDF Mining

Current approach: yfinance for public companies (works well), 6-source waterfall for private (struggles with ecdb/PitchBook failures).

Scout's value: PDF mining from IR pages. Scout already demonstrated this — LBP annual results page returned 2 PDF links. Private companies that publish official results as PDFs (common in French corporate reporting) are well-served by Scout.

⚠️ Context risk: Annual report PDFs can be 100+ pages. Need page-range targeting. Pass only the MD&A and digital revenue sections to LLM.

5. Executive Profile Chaining — New Angle

Not currently in any module.

When the company page links to executive LinkedIn profiles (as LBP's governance page does), Scout can follow those links and scrape the executive bio — previous companies, conference speaking history, quoted statements. This feeds 1G (investor intel) and 1A (company context) with executive language that isn't in press releases.

⚠️ Context risk: LinkedIn profiles are short (500-800 words). Low risk.

6. Regulatory Filing Mining — Financial Services Specific

Not currently in any module.

For banks and financial services companies, regulatory bodies publish supervisory review documents. France's ACPR (acpr.banque-france.fr) publishes resolution plans and supervisory decisions mentioning technology investment commitments. These are not in Google News and not in annual reports but are public.

Where Scout Won't Help

Module	Why
1C Traffic	SimilarWeb is an aggregated proprietary dataset — not scrapeable
1I Social Signals	LinkedIn and Twitter/X block crawlers even with stealth. Apify actors have special session/cookie handling
Phase 2 Browser Audit	Scout is a passive crawler. The audit needs interactive behavior (typing, SAYT observation, network monitoring) — that's Playwright's domain
Yahoo Finance data	Structured API is faster and more reliable than scraping

✅ Corrected Architecture (2026-05-07 — supersedes table above)

Scout /scrape already returns clean markdown (via PruningContentFilter threshold 0.4). It strips navigation, footers, ads, boilerplate. A 45KB raw HTML job listing page becomes ~800 words of clean prose markdown. This is already Haiku-ready — no regex content cleaning needed.

The pre-filter is at the URL-selection layer, not the content layer:

Scout /map → discovers N URLs (e.g. 200 career listings)
    ↓
URL regex filter  ← operates on URL STRINGS only, no LLM, near-zero cost
  keep: /job/, /position/, /opening/, /role/
  discard: /about/, /culture/, /benefits/, /team/
    ↓
Scout /scrape on filtered subset (e.g. 40 URLs) → clean markdown per page
    ↓
Haiku directly — no further content cleaning needed
  narrow prompt per page: "extract: title, tech_mentions, role_tier, seniority"
  output: compact ScoutSignal JSON (~200 tokens)
    ↓
Sonnet synthesis across all ScoutSignal objects
    ↓
Module scratchpad entry

For single-page targets (one exec bio, one IR page, one newsroom article):

Scout /scrape(url, stealth=true) → clean markdown → Haiku → ScoutSignal → Sonnet

No URL filter needed. Direct pipeline.

The token budget concern is about number of pages crawled, not content quality per page. Scout's markdown is already at Haiku-consumable quality. The URL filter controls how many pages enter the pipeline, keeping total Haiku calls bounded.

Open Questions — Practical Implementation Challenges

The user raised a critical concern on 2026-05-07: "Practically, this is quite challenging. We will be dealing with significantly large amounts of data, given to an LLM to parse. It's going to burn context like crazy, and the value we get back might be very less unless we carefully think about how it should be implemented and what the data structure should be."

These are unresolved. Do not implement any of the above without answering them first:

[ ] Pre-filtering architecture: How do we extract signal from large crawl payloads WITHOUT passing everything to the LLM? What filtering layer sits between Scout and the LLM? Options: regex keyword extraction, CSS schema targeting, embedding similarity, tiered Haiku/Sonnet routing.
[ ] Data structure for Scout outputs: What is the canonical JSON schema for a Scout result that downstream modules consume? It needs to be compact (not raw markdown) and structured (not a blob).
[ ] Token budget per module: What is the maximum token budget allowed for Scout-fetched content per module run? 1H hiring might get 5K tokens total for JD content. 1D competitors might get 3K per competitor. Define these limits before implementation.
[ ] Chunked processing pattern: For large content (PDFs, long pages), what's the chunking strategy? Process N tokens at a time with a cheap model (Haiku), only escalate relevant chunks to a larger model?
[ ] Tiered crawl strategy: Tier 1 = title/URL scan (no LLM). Tier 2 = summary extraction (Haiku). Tier 3 = full synthesis (Sonnet). Define trigger conditions for each tier.
[ ] Which modules to implement first? User confirmed 1H (hiring) is the highest priority. But which specific Scout integration within 1H — ATS portal crawl, full JD text mining, or both?
[ ] Cache strategy for Scout results: Scout crawls are slow (2-5 seconds per page with stealth). What's the cache TTL for competitor pages, newsroom pages, JD pages? Different TTLs for different content types?
[ ] Failure modes: What happens when Scout returns sparse content (<200 words) from a page? What's the fallback? When does the module proceed without Scout data vs abort?

Projects/Scout/ — Scout codebase and design docs
Projects/PRISM/Modules/Intel-Hiring.md — Hiring module spec
Projects/PRISM/Decisions/2026-05-07-scout-subrouter-pattern.md — ADR on Scout as pipeline sub-agent

✅ Signal Scoring + Hiring Module Decision (2026-05-07)

Two hard corrections on hiring intelligence

1. T1 buyers never appear on public job boards. C-level and VP roles are hired via executive search firms (Spencer Stuart, Korn Ferry, board networks). Job boards only contain T2 (Lead/Principal Engineer) and T3 (PM, Manager). The hiring module cannot identify Economic Buyers from job postings.

2. Search technology is never named in job descriptions. No JD says "must know Elasticsearch" or "Algolia experience required" — search is one task among many in a large platform role. Haiku extraction of tech mentions from JD text is the wrong approach and will return empty or generic results.

Conclusion: Hiring module = supporting qualification evidence only. Confirms a team exists. Does not find buyers or search tech signals.

Signal scorecard (scored: urgency × qualification × contact ID × hook)

Signal type	Score	Scout incremental value
Strategic trigger events	8.5	Trade press bypass (The Register, PhocusWire)
Partner intel	8.0	SI case study verification
Exec/investor quotes	7.5	IR PDF mining — proven (IAG Cloudflare bypass)
Competitor newsrooms	6.8	Highest incremental ROI — not yet implemented
Tech stack	6.5	Done
News signals	6.5	Done
Hiring (generic)	3.8	Low — last priority

scout_hiring.py — DEPRIORITISED

Plan written (plans/2026-05-07-scout-hiring-intel.md) but wrong approach. JD text mining for tech mentions is invalid. If hiring module is built: title-only classification, T2/T3 vacancy count only, no JD text scraping.

Next Scout engineering priority

Competitor newsroom enrichment (1D augmentation). Simple pipeline: Scout scrapes 3-5 competitor newsroom pages → Haiku extracts strategic signals → appended to 04-competitors.md. "EasyJet launches AI search" = direct urgency for BA AE pitch.