Dashboard/Scout/wiki/decisions/2026-06-27-shared-scrape-acquisition-profile.md
Scout

wiki/decisions/2026-06-27-shared-scrape-acquisition-profile.md

ADR: Shared Scrape Acquisition Profile

Date: 2026-06-27 Status: Accepted

Context

Competitive Intelligence needed better acquisition evidence from Scout: raw and cleaned markdown, source quality, marker validation, content hashes, and collector recommendations. During exploration, a consumer-specific /ci/scrape route was created, but that would create endpoint sprawl over time.

Scout is shared infrastructure. CI is one consumer, not the owner of the scrape contract.

Decision

Scout will expose acquisition quality metadata through the existing shared POST /scrape primitive, not through a consumer-specific /ci/scrape endpoint.

Rationale

  • Keeps Scout's core contract reusable across CI, PRISM, products, jobs, research, and future consumers.
  • Avoids endpoint debt such as /ci/scrape, /product/scrape, /jobs/scrape, and other narrow variants.
  • Lets each caller opt into richer acquisition metadata without breaking existing /scrape callers.
  • Preserves raw evidence while allowing cleaned content for downstream synthesis.
  • Makes collector selection measurable through benchmark artifacts instead of preference.

Implementation

Released as Scout 0.1.1.

Source updates:

  • pyproject.toml: version bumped to 0.1.1.
  • scout/core/version.py: runtime Scout version bumped to 0.1.1.
  • README.md: documented shared acquisition metadata and benchmark harness.
  • CHANGELOG.md: added release notes for 0.1.1.
  • scout/skill/scout.md: documented the shared scrape acquisition profile and no /ci/scrape rule.

Shared /scrape opt-in fields:

  • quality_analysis
  • cleanup
  • expected_markers
  • recommend_collector
  • source_id

Response additions:

  • raw_markdown
  • clean_markdown
  • nested acquisition metadata

Benchmark CLI:

scout benchmark URL --output-dir DIR [--js/--no-js] [--expected-marker ...]

Alternatives Considered

Option Why rejected
Add /ci/scrape Creates consumer-specific endpoint debt and forks Scout's acquisition contract.
Force all CI sources through Scout Adds noise and latency where direct HTTP or RSS/feed parsing is better.
Enable every Crawl4AI enhancement immediately Adds unproven complexity; native popup/overlay cleanup showed unacceptable latency during live testing.
Keep Scout unchanged and handle all quality logic in CI Duplicates acquisition quality concerns outside the infrastructure layer.

Verification

  • Rebuilt and restarted live Scout on Chowmes.
  • /health returned status: ok, Scout 0.1.1, Crawl4AI 0.9.0.
  • Default /scrape stayed backward-compatible:
  • acquisition: null
  • raw_markdown: ""
  • clean_markdown: ""
  • Opt-in /scrape returned acquisition_metadata.v1, quality_score, marker validation, and collector recommendation.
  • POST /ci/scrape returned 404.
  • scout benchmark https://example.com --output-dir /tmp/scout-release-011-benchmark --no-js --expected-marker "Example Domain" created benchmark artifacts.
  • Focused verification suite passed:
66 passed, 1 warning in 3.02s

The warning is a FastAPI/Starlette test client deprecation warning, not a Scout behavior failure.

Consequences

  • CI can now consume Scout 0.1.1 as a first-class acquisition primitive.
  • CI must still choose source collectors source-by-source: direct HTTP, Scout scrape, RSS/feed parsing, Parallel discovery, or manual-only.
  • Future Scout enhancements must be justified by benchmark gaps, not added because they are possible.
  • Native Crawl4AI cleanup should be reintroduced only after source-by-source latency and quality benchmarks.