Dashboard/Competitive Intelligence/logs/2026-06-27-scout-fetch-comparison.md
Competitive Intelligence

logs/2026-06-27-scout-fetch-comparison.md

Scout Fetch Comparison

Date: 2026-06-27

Purpose

Test whether Scout should be treated as a first-class CI data acquisition path instead of a secondary fallback.

Environment

Scout is running on Chowmes:

  • Host binding: 127.0.0.1:8421
  • Status: healthy
  • Scout version: 0.1.0
  • Crawl4AI version: 0.7.7
  • API auth: protected endpoints require X-API-Key

Test Set

Five representative Competitive Intelligence sources were tested through:

  1. Existing direct HTTP fetch.
  2. Scout /scrape with authenticated API access.

Results

Source Direct HTTP Scout scrape Assessment
Coveo press 1,951 chars, 132 ms 6,268 chars, 10.1 sec Direct is faster and enough for index monitoring; Scout captures richer article links.
Bloomreach updates 14,115 chars, 89 ms 17,560 chars, 8.5 sec Direct is fast; Scout output appears cleaner but slower.
Constructor product 12,156 chars, 169 ms 5,491 chars, 3.8 sec Scout likely better because it produced cleaner markdown, but consent text needs cleanup.
Google AI blog 3,352 chars, 9.6 sec 3,740 chars, 1.9 sec Scout is better for this source.
OpenAI RSS 140,440 chars, 378 ms 643,968 chars, 16 sec Generic scraping is wrong for feeds; use feed parsing.

Findings

  1. Scout should not be fallback-only.
  2. Scout is especially useful for product, positioning, dynamic, and markdown-quality-sensitive pages.
  3. Direct HTTP remains useful for fast hash checks and simple static pages.
  4. RSS/feed sources need a dedicated feed parser.
  5. Scout currently includes consent/cookie text on some pages.
  6. Scout is slower than direct HTTP on simple sources, so routing must be source-specific.
  7. The first unauthenticated Scout test failed with 403 Unauthorized; this was an integration issue, not a scraping failure.

Decision Impact

The CI acquisition plan is updated to use a collector router:

source registry
  -> collector strategy
  -> direct_http | scout_scrape | rss_feed | parallel_search | parallel_monitor | manual_only
  -> source health
  -> snapshot
  -> signal
  -> semantic delta

Scout is now a first-class collection path for CI.

Next Work

  • Add Scout collector support to the CI skill.
  • Add source-level collector metadata.
  • Add a benchmark command that compares collectors across configured sources.
  • Add Scout enhancement plan for consent cleanup, feed handling, extraction quality scoring, and CI-specific acquisition mode.