Dashboard/Competitive Intelligence/plans/2026-06-27-scout-enhancement-plan.md
Competitive Intelligence

plans/2026-06-27-scout-enhancement-plan.md

Scout CI Acquisition Profile V1

Date: 2026-06-27

Status: Released as Scout 0.1.1. A separate /ci/scrape endpoint was implemented during exploration, rejected as bespoke endpoint debt, and removed. The live direction is now enforced in code: enhance Scout's existing /scrape path with optional acquisition metadata only through shared request/response types, not a separate route.

Capability audit status: Crawl4AI capability and version audit completed on 2026-06-27.

Summary

Scout should expose a focused acquisition metadata profile for Competitive Intelligence pages where browser-grade extraction, clean markdown, JavaScript rendering, screenshots, crawling, or page mapping matter.

The goal is not to force Scout onto every source, and it is not to make Scout a full CI product. The goal is to make Scout a measurable acquisition quality engine that can tell the CI runner whether a captured page is trustworthy enough to use.

Architecture boundary:

/scrape = general-purpose Scout scrape
/scrape with CI acquisition profile/options = scrape + CI cleanup + quality scoring + collector recommendation

There must be no separate CI scrape route. CI behavior must reuse Scout's core acquisition types and Crawl4AI integration inside the existing scrape architecture. As of the 2026-06-27 implementation, POST /ci/scrape returns 404.

Backward compatibility rule:

Existing /scrape callers must not break. Enhanced acquisition behavior must be opt-in through request flags or compatible nullable fields.

Evidence From Smoke Test

Scout performed better than direct HTTP on Google AI blog and Constructor product pages, and provided richer extraction for Coveo/Bloomreach, but it showed three improvement needs:

  • API clients must send X-API-Key.
  • Consent/cookie boilerplate can pollute markdown.
  • RSS feeds should not be processed through generic page scraping.
  • Simple static pages may not need Scout because direct HTTP is faster.

Crawl4AI Feasibility Check

Scout is using Crawl4AI as its acquisition engine. The live Scout container now reports:

  • Scout version: 0.1.1
  • Crawl4AI package version: 0.9.0
  • PyPI latest checked from Chowmes on 2026-06-27: 0.9.0

Important correction:

Scout previously reported Crawl4AI 0.7.7 from a hardcoded version constant while the container actually had 0.9.0 installed. This made operational truth unreliable. Scout now reads the installed package version from Python package metadata, and pyproject.toml pins crawl4ai==0.9.0 so future rebuilds do not silently upgrade Crawl4AI without tests.

What Crawl4AI Already Supports

Crawl4AI supports the core primitives needed for this plan:

  • Browser rendering through AsyncWebCrawler.
  • Browser configuration through BrowserConfig.
  • Run-level configuration through CrawlerRunConfig.
  • Clean markdown through DefaultMarkdownGenerator.
  • Heuristic filtering through PruningContentFilter.
  • Structured extraction through JsonCssExtractionStrategy.
  • LLM extraction through LLMExtractionStrategy.
  • Deep crawling through BFSDeepCrawlStrategy.
  • Sitemap/URL discovery through aseed_urls and SeedingConfig.
  • Screenshots and PDFs through run config media flags.
  • JS wait conditions through wait_for, wait_until, and page timeouts.
  • Stealth/user simulation knobs including simulate_user, magic, override_navigator, user agent, proxy, and persistent/session options.
  • Content cleanup knobs including excluded_tags, excluded_selector, remove_overlay_elements, and remove_consent_popups.
  • Advanced page handling including full-page scan, iframe processing, shadow DOM flattening, network request capture, console capture, virtual scroll config, retries, and fallback fetch functions.

Are We Using Crawl4AI Optimally?

Not yet.

Scout already uses several good Crawl4AI primitives: AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, PruningContentFilter, DefaultMarkdownGenerator, JsonCssExtractionStrategy, LLMExtractionStrategy, BFSDeepCrawlStrategy, screenshot mode, sitemap mapping, and stealth options.

The CI runner currently calls Scout's existing basic scrape path. Scout does not yet fully use Crawl4AI's best available controls for CI:

  • It does not pass remove_consent_popups=True.
  • It does not pass remove_overlay_elements=True.
  • It does not use source-specific css_selector, target_elements, excluded_tags, or excluded_selector.
  • It does not use BM25ContentFilter or source-query-focused filtering for noisy pages.
  • It does not use capture_network_requests or capture_console_messages for diagnosing broken dynamic pages.
  • It does not expose screenshot/PDF/MHTML capture as optional CI evidence.
  • It does not use virtual scroll or full-page scan for dynamic/news index pages.
  • It does not use persistent sessions for sources that benefit from cookie/session continuity.
  • It does not use Crawl4AI's fallback fetch hook as part of a collector ladder.

Conclusion: the Scout enhancement plan is feasible, but the first implementation is only a starting point. The next Scout work should move CI acquisition closer to Crawl4AI-native configuration instead of adding too much custom post-processing.

Implemented Slice On 2026-06-27

Implemented on the live Chowmes Scout service under /opt/prism/scout:

  • Released Scout source/runtime as 0.1.1.
  • Updated Scout pyproject.toml, scout/core/version.py, README.md, CHANGELOG.md, and scout/skill/scout.md.
  • Removed the exploratory /ci/scrape endpoint and its tests.
  • Kept /scrape as the single shared acquisition primitive.
  • Added backward-compatible opt-in request flags:
  • quality_analysis
  • cleanup
  • expected_markers
  • recommend_collector
  • source_id
  • Added explicit evidence fields to ScrapeResponse:
  • raw_markdown
  • clean_markdown
  • acquisition
  • Added AcquisitionMetadata under acquisition with schema alias acquisition_metadata.v1.
  • Added explainable quality scoring with quality_score and quality_reasons.
  • Added marker validation with markers_found, markers_missing, and marker_score.
  • Added collector recommendation metadata with recommended_collector and recommended_collector_reason.
  • Added feed-like URL detection that recommends rss_feed without parsing feeds inside Scout.
  • Added a benchmark CLI:
scout benchmark URL --output-dir DIR [--js/--no-js] [--expected-marker ...]

Benchmark runs write:

benchmark.json
comparison.md
direct_http.txt
scout_raw.md
scout_clean.md
samples/

Important implementation decision:

Crawl4AI-native remove_consent_popups and remove_overlay_elements were not enabled by default in this slice. During live exploration, enabling native cleanup created unacceptable latency on a simple page. The first slice therefore uses explicit post-processing cleanup and preserves raw evidence. Native cleanup must be reintroduced only after benchmarked source-by-source validation.

Enhancements To Add

1. Lock ci_acquisition.v1 response schema

Implementation status: implemented first slice.

Add CI-oriented profile behavior to the existing /scrape architecture with defaults optimized for competitive monitoring:

  • raw markdown
  • clean markdown
  • link extraction
  • word count and token estimate
  • content hash
  • boilerplate/consent cleanup
  • cleanup rules applied
  • source quality metadata
  • quality score reasons
  • extraction method metadata
  • collector recommendation reasons
  • duration and error metadata

Contract model:

scrape_response.v1 + acquisition_metadata.v1

The endpoint remains /scrape; only the optional response metadata is versioned.

Proposed opt-in request fields:

{
  "quality_analysis": true,
  "cleanup": true,
  "expected_markers": ["AI Search", "Product updates"],
  "recommend_collector": true
}

Endpoint:

POST /scrape

No separate CI endpoint is allowed.

Proposed response shape:

{
  "success": true,
  "url": "https://example.com",
  "markdown": "current/default markdown",
  "raw_markdown": "raw markdown when requested",
  "clean_markdown": "clean markdown when requested",
  "links": [],
  "metadata": {},
  "acquisition": {
    "schema": "acquisition_metadata.v1",
    "source_id": "optional-source-id",
    "final_url": "https://example.com",
    "status_code": 200,
    "fetched_at": "2026-06-27T00:00:00Z",
    "provider": "crawl4ai",
    "content_hash": "...",
    "boilerplate_removed": 0,
    "cleanup_rules_applied": [],
    "blocked": false,
    "markers_found": [],
    "markers_missing": [],
    "marker_score": 1.0,
    "quality_score": 0.82,
    "quality_reasons": [],
    "recommended_collector": "scout_scrape",
    "recommended_collector_reason": "browser_rendered_product_page"
  }
}

Field inventory:

  • success
  • source_id
  • url
  • final_url
  • canonical_url
  • title
  • raw_markdown
  • clean_markdown
  • links
  • status_code
  • fetched_at
  • provider
  • content_hash
  • word_count
  • token_estimate
  • duration_ms
  • boilerplate_removed
  • cleanup_rules_applied
  • blocked
  • markers_found
  • markers_missing
  • marker_score
  • quality_score
  • quality_reasons
  • recommended_collector
  • recommended_collector_reason
  • error

Compatibility note:

The current implementation returns markdown. The schema lock should either rename this to clean_markdown or keep markdown as a backwards-compatible alias while adding explicit raw_markdown and clean_markdown.

Do not overload markdown fields confusingly:

  • markdown remains the existing/default field.
  • raw_markdown is raw evidence when requested.
  • clean_markdown is cleaned CI-ready content when requested.
  • acquisition-only fields live under acquisition.

Implementation status: initial pattern-based cleanup now exists inside the shared /scrape path/profile. Broader benchmark against real competitor pages is still required.

Crawl4AI implication: test native remove_consent_popups, remove_overlay_elements, excluded_tags, and excluded_selector source-by-source before enabling. Live testing showed native popup/overlay cleanup can add too much latency when applied broadly.

Add reusable text filters for:

  • cookie banners
  • privacy preference text
  • "accept all cookies"
  • login/signup nav duplication
  • repeated footer/nav blocks
  • newsletter boilerplate

The cleanup must be opt-in or mode-specific at first so it does not damage product extraction use cases.

Auditability rule:

Cleanup must never destroy the only evidence. Scout should preserve raw markdown or a raw artifact, then return cleaned markdown with boilerplate_removed and cleanup_rules_applied.

3. Feed detection and handoff

Implementation status: feed-like URLs are detected and handed off with recommended_collector: rss_feed; true feed parsing remains to be built in the CI collector layer.

Scout should detect RSS/Atom/XML feeds and either:

  • return recommended_collector: rss_feed

This prevents massive markdown blobs like the OpenAI RSS result.

Boundary rule:

RSS/feed parsing should stay outside Scout for now. Scout can detect feeds and recommend rss_feed, but the CI runner owns feed parsing unless a future benchmark proves centralizing feed adapters inside Scout is better.

4. Extraction quality score

Implementation status: initial quality score implemented using text length, marker presence, title presence, blocked-page detection, and link extraction.

Crawl4AI implication: improve scoring with native diagnostics such as cleaned HTML length, fit-vs-raw markdown ratio, network/console capture on failures, screenshot evidence, and source-specific expected selectors.

Add a quality score to each scrape:

  • enough text extracted
  • expected markers present
  • page title present
  • blocked-page patterns absent
  • consent ratio below threshold
  • links extracted

This lets CI decide whether Scout output is reportable or needs fallback.

Quality score must be explainable. Add quality_reasons, for example:

[
  "title_present",
  "expected_markers_found",
  "low_boilerplate_ratio",
  "links_extracted"
]

The score should never be a black box.

5. Expected marker validation

Implementation status: initial marker validation implemented.

Allow callers to pass expected markers:

{
  "url": "https://example.com",
  "expected_markers": ["Product updates", "AI Search"]
}

Scout returns:

  • markers_found
  • markers_missing
  • marker_score

6. Collector benchmark harness

Implementation status: implemented first CLI artifact harness. Full six-source benchmark remains pending.

Crawl4AI implication: benchmark should compare direct HTTP, Scout basic scrape, Scout CI scrape with native cleanup, Scout screenshot evidence, and RSS/feed parsing.

Added a command, not an endpoint, to compare acquisition methods:

scout benchmark URL --output-dir DIR [--js/--no-js] [--expected-marker ...]

Minimum comparison dimensions:

  • success
  • duration
  • character count
  • word count
  • marker score
  • link count
  • blocked status
  • boilerplate ratio
  • output sample

Benchmark output must be artifacted. Each run should write:

benchmark.json
comparison.md
direct_http.txt
scout_raw.md
scout_clean.md
samples/

Benchmark minimum source set:

  • Google AI blog
  • Constructor product page
  • Coveo page
  • Bloomreach page
  • OpenAI RSS
  • simple static page

7. CI source strategy export

Implementation status: not implemented. CI source strategy metadata has started in the CI runner, but Scout does not yet export source strategy patches.

Scout can recommend collector strategy:

  • rss_feed for feeds
  • direct_http for simple static sources
  • scout_scrape for product/positioning/blog pages
  • manual_only for gated sources

But Scout should not own source configuration mutation. Patching sources.yaml belongs to the CI runner. Scout should output recommendation fields such as:

{
  "recommended_collector": "rss_feed",
  "recommended_collector_reason": "content_type_xml_feed"
}

The CI system decides whether to update source config.

Rollout Order

  1. Build benchmark harness. Done for CLI artifact harness.
  2. Run benchmark against the six target sources. Pending.
  3. Capture failures and gaps as Scout improvement tasks. Pending.
  4. Lock CI acquisition fields as a backward-compatible extension of the existing /scrape contract. Done for first slice.
  5. Add Crawl4AI-native cleanup/options to /scrape. Deferred until benchmarked because broad native cleanup caused latency in live testing.
  6. Add raw and cleaned markdown separation. Done for opt-in profile.
  7. Add quality score reasons and collector recommendation reasons under acquisition. Done for first slice.
  8. Integrate CI runner client. Pending.
  9. Add source health records showing collector used and quality score. Pending.
  10. Rerun benchmark and compare before/after. Pending.
  11. Let the CI runner propose or patch source strategy config. Pending.

Verification Log

2026-06-27 release verification

Live Scout release:

  • docker compose -f docker/docker-compose.yml up -d --build scout built a scout-0.1.1 wheel and restarted the scout container.
  • docker ps showed scout healthy on 127.0.0.1:8421.
  • /health returned status: ok, Scout 0.1.1, Crawl4AI 0.9.0.

Release contract checks:

  • Default POST /scrape against https://example.com succeeded in 1348ms.
  • Default response stayed backward-compatible:
  • acquisition: null
  • raw_markdown: ""
  • clean_markdown: ""
  • Opt-in POST /scrape succeeded in 1189ms.
  • Opt-in response included:
  • schema: acquisition_metadata.v1
  • quality_score: 1.0
  • markers_found: ["Example Domain"]
  • recommended_collector: direct_http
  • recommended_collector_reason: simple_static_page
  • POST /ci/scrape returned 404.

Release benchmark:

  • scout benchmark https://example.com --output-dir /tmp/scout-release-011-benchmark --no-js --expected-marker "Example Domain" succeeded.
  • Artifacts created:
  • benchmark.json
  • comparison.md
  • direct_http.txt
  • scout_raw.md
  • scout_clean.md
  • samples/
  • Benchmark result:
  • direct HTTP succeeded in 30ms
  • Scout scrape succeeded in 1239ms
  • recommendation: direct_http
  • reason: faster_static_page

Release test verification:

66 passed, 1 warning in 3.02s

The warning is a Starlette/httpx deprecation warning from FastAPI test client dependencies, not a Scout behavior failure.

Conclusion: Scout source-level enhancement is complete for this slice. The Competitive Intelligence workspace should now consume Scout 0.1.1 through /scrape and scout benchmark, not continue adding Scout features in this thread.

2026-06-27 implementation verification

Live Scout container:

  • docker ps showed scout healthy on 127.0.0.1:8421.
  • /health returned status: ok, Scout 0.1.0, Crawl4AI 0.9.0.

Endpoint behavior:

  • Default POST /scrape against https://example.com succeeded in 1332ms.
  • Default response kept compatibility:
  • acquisition: null
  • raw_markdown: ""
  • clean_markdown: ""
  • Opt-in POST /scrape with quality_analysis, cleanup, expected_markers, recommend_collector, and source_id succeeded in 1177ms.
  • Opt-in response included:
  • acquisition.schema: acquisition_metadata.v1
  • source_id: smoke-example
  • content_hash
  • markers_found: ["Example Domain"]
  • marker_score: 1.0
  • quality_score: 1.0
  • recommended_collector: direct_http
  • recommended_collector_reason: simple_static_page
  • POST /ci/scrape returned 404, confirming no bespoke CI endpoint is live.

Benchmark artifact verification:

  • scout benchmark https://example.com --output-dir /tmp/scout-benchmark-smoke --no-js --expected-marker "Example Domain" succeeded.
  • Artifacts created:
  • /tmp/scout-benchmark-smoke/benchmark.json
  • /tmp/scout-benchmark-smoke/comparison.md
  • /tmp/scout-benchmark-smoke/direct_http.txt
  • /tmp/scout-benchmark-smoke/scout_raw.md
  • /tmp/scout-benchmark-smoke/scout_clean.md
  • /tmp/scout-benchmark-smoke/samples/
  • Benchmark result:
  • direct HTTP succeeded in 39ms
  • Scout scrape succeeded in 1313ms
  • recommendation: direct_http
  • reason: faster_static_page

Test verification:

  • Focused unit suite passed in the rebuilt container:
66 passed, 1 warning in 2.83s

The warning is a Starlette/httpx deprecation warning from FastAPI test client dependencies, not a Scout behavior failure.

Earlier exploration verification

Smoke-tested after rebuilding the live scout container on Chowmes:

  • /health returned status: ok, Scout 0.1.0, Crawl4AI 0.7.7.
  • A separate POST /ci/scrape route was tested during exploration, then rejected as endpoint debt and removed from the accepted architecture.
  • Legacy POST /scrape still returned a successful scrape for https://example.com.
  • After the Crawl4AI audit, /health now reports Crawl4AI 0.9.0, matching the installed package reported by pip show.
  • pyproject.toml now pins crawl4ai==0.9.0 rather than floating on crawl4ai>=0.7.7.

Host-side Python compile passed for:

  • scout/core/types.py
  • scout/api/routers/ci.py
  • scout/api/main.py
  • tests/unit/api/test_ci_scrape.py

Limitation: Scout unit tests could not be run on the VPS host because pytest is not installed outside the container. The endpoint was verified through live API smoke tests instead.

Acceptance Criteria

  • Scout can be called securely from the CI runner.
  • /scrape rejects missing X-API-Key.
  • Enhanced acquisition fields are opt-in or backward compatible.
  • Response shape is documented as scrape_response.v1 + acquisition_metadata.v1.
  • Scout output includes enough metadata to compare against direct HTTP.
  • Acquisition metadata includes content_hash, source_id, final_url, fetched_at, provider, and collector metadata.
  • Acquisition metadata is nested under acquisition instead of cluttering the top-level scrape response.
  • quality_score includes explainable quality_reasons.
  • Collector recommendations include recommended_collector_reason.
  • Cleanup never destroys raw evidence.
  • Consent boilerplate is reduced on Constructor-like pages.
  • RSS feeds are not treated as normal scrape pages.
  • RSS/feed parsing stays in the CI runner unless later evidence argues for a Scout-owned feed adapter.
  • CI can choose collector strategy by source type.
  • Source health records which collector was used.
  • Scout failures are visible as Scout improvement tasks, not hidden by fallback.
  • Benchmark compares Scout and direct HTTP on the minimum benchmark source set.
  • Benchmark outputs are saved as artifacts with raw, clean, direct, JSON, and markdown comparison files.

Relationship To Broader Scout

This plan is limited to Competitive Intelligence acquisition. It does not solve the broader Scout product/browser workbench.

Separate Scout work remains:

User Browser Capture -> DOM evidence -> product records -> Algolia preview

The CI acquisition profile should run in parallel conceptually, but should not sprawl into the larger Scout app/browser workflow.

Strategic Principle

Scout is Chowmes-owned infrastructure. Competitive Intelligence should use it aggressively where it creates advantage and should expose its weaknesses so Scout improves.