Dashboard/Scout (Second-Brain)/wiki/dev-log.md
Scout (Second-Brain)

wiki/dev-log.md

Scout — Dev Log

Detailed development log. For the operation summary, see log.

2026-06-29 — Website Logo And Scroll-Aware Nav Polish

Status: Complete.

What changed: Added a real Scout SVG mark and wordmark, replaced the old block S brand treatment across static pages, tightened homepage density by about 10%, and added section-aware homepage nav with IntersectionObserver scrollspy.

Why: The site direction was strong but slightly cluttered, and the nav did not update while scrolling even though each major page section should feel mapped to navigation.

Verification: Website unit tests passed; launch-readiness stayed ready_with_limits; route smoke passed for homepage, logo assets, pricing, status, beta, and quickstart; Playwright desktop/mobile validation passed.

2026-06-29 — Unit Economics Pricing Gate

Status: Active launch gate.

Decision: $22 one-time and $9/month hosted pricing are rejected as arbitrary placeholders. Hosted pricing must be derived from unit economics.

Pricing direction: free local beta; invite-only hosted free allowance; pay-as-you-go or prepaid credits first; subscriptions only after usage telemetry supports them.

Required model: fixed monthly costs, variable run costs, payment fees, support/security/maintenance allocation, target margin, and break-even volume.

Repo source: docs/product/unit-economics-and-pricing-model-2026-06-29.md.

2026-06-29 — Private beta launch surface, website, and relationship map

Status: Done for checkpoint; public launch still blocked.

What happened: Scout's current launch state was recorded from the repo evidence. The product surface is local package/CLI, FastAPI HTTP service, Docker service, hosted private-beta API, Claude/Codex skill backend, and the launch website served by scout serve.

Key decisions: Added ADR 2026-06-29-private-beta-launch-surface: Scout is controlled private beta with explicit limits, not public launch. The legacy /app UI is not a launch surface.

Documentation captured: Launch evidence index, release checklist, website route verification, private beta invitation/handoff, product exports, hosted operating contract, scalability/security audit, competitor website research, and website pages (/, /quickstart, /guide, /examples, /pricing, /status, /beta, /legal, /terms, /privacy).

Relationships captured: Added downstream-consumers-and-project-relationships covering PRISM / Chowmes-PRISM, Competitive Intelligence, Algolia Search Audit skills, product/catalog workflows, job hunter workflows, and the Scout website.

Current verification evidence: Repo branch codex/scout-platform-foundation pushed to GitHub at latest observed commit 8aa8b09 Add beta developer guide page; launch evidence says private beta is ready_with_limits, public launch is blocked, and public launch has no Codex-actionable engineering tasks until founder/legal/payment/risk decisions close.

Next: Review the launched local website in-browser, make founder decisions on license/pricing/publishing/risk/Stripe, then close launch blockers in the release checklist.

2026-06-22 — SQLite run persistence (Plan Phase 1) + UI wiring + PRISM endpoints

SQLite Run Persistence (Phase 1 of code-complete plan)

  • New scout/api/db.py — RunDB class: async SQLite via aiosqlite, WAL journal mode, foreign keys ON.
  • Schema: runs table (run_id PK, use_case, query, status, mode, output_dir, artifacts_json, created_at, updated_at, finished_at) + run_events table (id AUTOINCREMENT, run_id FK CASCADE, stage, message, level, timestamp) + index on run_id.
  • Write-through pattern in run_store.py: in-memory dict for fast reads + SQLite for persistence. _db_ready() guard handles closed connections gracefully (falls back to in-memory only — needed for test isolation).
  • remember_run() and get_run() now async. All callers updated (run.py, runs.py).
  • Config: Settings.db_path (env: SCOUT_DB_PATH), defaults to {SCOUT_WORKDIR}/scout.db.
  • Lifespan: creates RunDB on startup, binds to run_store, closes on shutdown.
  • DI: get_run_db() in deps.py for router injection.
  • 10 new tests: init, save/get/list/delete runs, upsert, events, restart recovery.
  • Fix-and-learn: module-level _DB global can point to closed RunDB after TestClient lifespan teardown. Guard with _conn is not None check, not bare assertion.
  • 305 unit tests green (was 295 before new tests + 10 new), pyright 0 errors, ruff clean.
  • aiosqlite>=0.20.0 added to pyproject.toml dependencies.

Code-complete plan approved

  • 9-phase plan at ~/.claude/plans/hidden-jumping-crayon.md. ~13 sessions total.
  • User decision: ALL 9 intelligence verticals made real (not just core 4).
  • Vault wiki created at Projects/Scout/ with index, log, CLAUDE.md, overview, requirements, open-questions, dev-log, and 4 ADRs.

2026-06-22 — UI wiring + PRISM endpoints

UI wiring (commit 5409157)

  • Problem: API returned markdown+records from Crawl4AI but /app/live-browser still showed raw text blob.
  • Fix: Updated live_browser_page.py JS to use res.markdown and res.record_count. Added download buttons for .md and .json. Preview shows markdown + first 20 records.
  • Cleanup: Removed dead text/chars fields from NativeCaptureResult in app_browser.py. All lastCapture.text fallbacks removed — markdown is the only display path.

PRISM endpoints (commit 353e26e)

  • POST /structure — StructureRequest(html, source_url?, css_schema?, llm_schema?) → CaptureExtraction. Uses structure_capture() via raw:// internally.
  • POST /harvest — HarvestRequest(cdp_url, url, css_schema?, scan_full_page?) → CaptureExtraction. Uses acquire_open_page() via CDP-attach internally.
  • Both mounted in main.py, protected by existing API key auth middleware.
  • Tests: 8 unit (4 each) + 5 integration (3 structure + 2 harvest). Harvest integration test: marker injected into live Chromium DOM, marker survives extraction = proof of in-place read.

Branch pushed to GitHub

  • codex/scout-platform-foundation at commit 353e26e
  • 295 unit + 14 integration tests green, pyright 0 errors, ruff clean

2026-06-21 — Crawl4AI-over-CDP + raw:// structuring

See log for summary. Key technical details:

  • cdp_acquire.py::acquire_open_page(): BrowserConfig(browser_mode="cdp", cdp_url=...) + CrawlerRunConfig(js_only=True) → attach to existing tab by URL match (browser_manager.py:1078), no re-navigation.
  • capture_extract.py::structure_capture(): "raw://" + html → crawler.arun() → processes bytes in place, status 200, no network fetch.
  • Decisive proof: integration test injects DOM marker → acquire_open_page(js_only) → marker in output = in-place read (a goto would wipe it).