Dashboard/Competitive Intelligence/logs/2026-06-27-scout-acquisition-implementation.md
Competitive Intelligence

logs/2026-06-27-scout-acquisition-implementation.md

Scout acquisition implementation log

Date: 2026-06-27

Status: Released first shared-acquisition slice as Scout 0.1.1 on live Chowmes Scout.

Decision

Do not create /ci/scrape.

Competitive Intelligence should consume Scout through the shared /scrape primitive. CI-specific needs are handled through opt-in request fields and nested acquisition metadata. This prevents endpoint sprawl such as /ci/scrape, /product/scrape, /jobs/scrape, and keeps Scout as infrastructure rather than a collection of bespoke consumer routes.

Implemented

On live Scout under /opt/prism/scout:

  • Bumped Scout from 0.1.0 to 0.1.1.
  • Updated Scout source documentation:
  • README.md
  • CHANGELOG.md
  • scout/skill/scout.md
  • Removed the exploratory /ci/scrape route.
  • Added optional /scrape request fields:
  • quality_analysis
  • cleanup
  • expected_markers
  • recommend_collector
  • source_id
  • Added response fields:
  • raw_markdown
  • clean_markdown
  • acquisition
  • Added AcquisitionMetadata with schema alias acquisition_metadata.v1.
  • Added explainable quality scoring.
  • Added expected marker validation.
  • Added collector recommendations.
  • Added feed-like URL detection that recommends rss_feed.
  • Added scout benchmark CLI artifact harness.

Verification

Release verification:

  • Rebuilt and restarted Scout with docker compose -f docker/docker-compose.yml up -d --build scout.
  • Build produced a scout-0.1.1 wheel.
  • /health returned status: ok, Scout 0.1.1, Crawl4AI 0.9.0.
  • Container stayed healthy on 127.0.0.1:8421.

Live endpoint checks:

  • Default /scrape succeeded and stayed backward-compatible:
  • acquisition: null
  • raw_markdown: ""
  • clean_markdown: ""
  • Opt-in /scrape returned acquisition metadata with:
  • schema: acquisition_metadata.v1
  • content_hash
  • quality_score
  • quality_reasons
  • markers_found
  • recommended_collector
  • recommended_collector_reason
  • /ci/scrape returned 404.

Benchmark harness check:

  • scout benchmark https://example.com --output-dir /tmp/scout-benchmark-smoke --no-js --expected-marker "Example Domain" created:
  • benchmark.json
  • comparison.md
  • direct_http.txt
  • scout_raw.md
  • scout_clean.md
  • samples/
  • The smoke benchmark recommended direct_http for example.com because direct HTTP was faster than Scout on a simple static page.
  • Release benchmark artifacts were also verified at /tmp/scout-release-011-benchmark.
  • Release benchmark result:
  • direct HTTP: 30ms
  • Scout scrape: 1239ms
  • recommendation: direct_http
  • reason: faster_static_page

Unit test verification:

66 passed, 1 warning in 3.02s

The warning is a FastAPI/Starlette test client deprecation warning, not a Scout behavior failure.

Deferred

  • Full six-source benchmark against Google AI blog, Constructor, Coveo, Bloomreach, OpenAI RSS, and a static page.
  • CI runner integration.
  • Source health records storing collector used and quality score.
  • Native Crawl4AI cleanup defaults.

Native Crawl4AI popup/overlay cleanup was deliberately deferred because live exploration showed broad native cleanup could create unacceptable latency. It should be reintroduced only after benchmarked source-by-source validation.

Closeout

Scout source work is closed for this slice. Do not continue adding Scout features merely because they are possible. The next Competitive Intelligence work should consume Scout 0.1.1 through the shared /scrape profile and benchmark harness, then decide source-by-source whether to use direct HTTP, RSS/feed parsing, Scout browser acquisition, or manual-only handling.