Scout — Wiki Index

Type: project Created: 2026-05-14 Last Updated: 2026-07-03

Read this file first. It is the canonical navigation map for the Scout project.

Current Thesis

Scout is a web-to-record intelligence engine. Crawl4AI is one acquisition provider; Scout's durable value is the layer after acquisition: discovery, provider routing, extraction, enrichment, normalization, quality scoring, provenance, artifact writing, and portable outputs for downstream tools.

Current Status

Latest checkpoint: Scout is checked into GitHub on branch codex/scout-saas-prod-ready and has a private-beta launch surface: local package/CLI, FastAPI HTTP service, Docker service, hosted private-beta API, Claude/Codex skill backend, product exports, launch website, quickstart, guide, examples, pricing/status/legal pages, and beta handoff docs.

Private beta remains controlled and usage-limited. The hosted service is live, but self-service key delivery and paid checkout are not production-complete until SMTP delivery, Stripe secret, Stripe price IDs, and the Stripe webhook secret are configured and smoke-tested. The legacy /app UI has been removed from the supported product surface.

Service-first product surface update: Scout is intentionally positioned as a utility/service, not a dashboard app. Supported surfaces are the launch website, CLI, local HTTP API, hosted /v1/hosted/* API, Docker-from-source, artifacts, and Claude/Codex skill usage. The old static /app UI and /api/config exposure were removed from the codebase and verified as 403 locally and on https://scout.chowmes.com.

License update: Scout local/core package and source distribution are now Apache-2.0. Hosted Scout access, paid service use, support, and enterprise terms remain separate. This closes the license implementation gate but does not approve public launch, PyPI, GHCR, Docker Hub, or public hosted signup.

Pricing update: arbitrary $22 one-time and $9/month hosted concepts are rejected as placeholders. The approved current beta hosted packages are pay-as-you-go/prepaid standard credits: $10 / 1,000, $25 / 3,000, and $100 / 15,000. The $10 / 1,000 package is currently modeled at $2.59 loaded cost, 74.1% gross margin, and 17 packs/month break-even against a $120/month fixed-cost assumption. Paid checkout is still not production-complete until Stripe and SMTP live smoke pass. See wiki/syntheses/unit-economics-pricing-2026-06-29.md.

Hosted access update: shared beta passwords are rejected. Beta access now uses the public /beta page as the self-service entrypoint. The current product posture is direct name/email beta registration: when HOSTED_BETA_SIGNUP_ENABLED and SMTP key delivery are configured, POST /v1/hosted/beta-key provisions a finite-credit hosted_beta_pass tenant and emails the one-time API key. If SMTP is not configured, /beta records a pending_delivery request without creating an account, key, or credits. Raw API keys are never shown in the browser.

Production readiness update: https://scout.chowmes.com is deployed on branch codex/scout-saas-prod-ready; metering and package metadata are live, and the hosted account DB is /data/hosted_accounts.sqlite. Operational beta readiness is now defined by ready_for_beta_key_delivery: beta signup enabled plus configured SMTP key delivery. Operators can later drain pending_delivery registrations through the protected admin endpoint POST /v1/billing/admin/deliver-pending-beta-keys after SMTP is configured and smoke-tested. Failed beta delivery attempts can be retried after SMTP is fixed through POST /v1/billing/admin/retry-failed-beta-keys or scripts/scout-hosted-admin retry-failed-signups; the retry path skips emails that already have a tenant and never returns raw API keys. Stripe settings are not configured in production yet, so paid checkout remains paused until Stripe and SMTP are wired. Admin usage monitoring includes scripts/scout-hosted-admin list-usage and scripts/scout-hosted-admin list-usage --summary for VPS ledger review without exposing raw API keys or stored key hashes.

Hosted billing portal update: Scout now has a Bearer-key protected Stripe Customer Portal endpoint at POST /v1/billing/stripe/customer-portal-session. It infers the Stripe customer from the authenticated tenant's stored checkout records and returns a short-lived Stripe portal URL only when a customer is linked. /v1/hosted/me advertises this link, and /account exposes a Manage billing action after account lookup. Live operation still requires STRIPE_SECRET_KEY, STRIPE_PORTAL_RETURN_URL, Stripe webhook configuration, price IDs, and SMTP key-delivery smoke.

Current beta-path correction: the current product stance is direct email registration, with pending-request capture when SMTP is not configured. The /beta page posts name/email to /v1/hosted/beta-key and never displays raw API keys. Production health is OK, but hosted beta self-service remains blocked by external SMTP configuration and smoke tests until email delivery can be completed live.

Stripe return URL update: beta and paid checkout now have separate return URL settings. Paid credit packages return to /pricing?checkout=...; $0 beta_trial setup returns to /beta?checkout=... when STRIPE_BETA_SUCCESS_URL and STRIPE_BETA_CANCEL_URL are configured. See wiki/decisions/2026-07-03-beta-specific-stripe-return-urls.md.

Stripe beta setup update: beta_trial Checkout Sessions now use mode=setup plus customer_creation=always, customer email, card collection, and package metadata. This makes the $0 beta path a real customer/payment-method setup flow before the signed webhook provisions and emails the hosted API key.

Signup monitoring update: operators can now run scripts/scout-hosted-admin list-signups to review self-service beta API-key requests from the hosted database, including delivered, failed, and duplicate outcomes. The command supports table/JSON output plus email/status filters and does not print raw API keys or stored key hashes.

Hosted config preflight update: scripts/scout-hosted-admin validate-config --require beta now requires SMTP host, port, from address, username, and password. This intentionally matches the hosted SaaS readiness gate so a local config cannot pass while production still reports missing SMTP credentials.

SMTP delivery smoke update: operators can now run scripts/scout-hosted-admin send-test-email after installing SMTP settings. The command sends a smoke-test message through the same hosted key-delivery path without creating an account, granting credits, issuing a real key, or printing SMTP secrets. This is the post-configuration proof step before opening self-service beta signup.

Hosted access lifecycle update: operators can now run scripts/scout-hosted-admin disable-access to disable a hosted tenant by email or tenant id, or disable one API key by key id. The command requires --yes, keeps audit rows in /data/hosted_accounts.sqlite, and never prints raw API keys or stored key hashes. This closes the immediate "what if a beta key leaks" operator gap, but it is still an admin CLI flow, not a self-service customer portal.

Hosted readiness checker update: scripts/scout-hosted-admin readiness now gives operators a repeatable non-secret preflight against https://scout.chowmes.com. It wraps the lower-level scripts/hosted_readiness_check.py checker from the main admin surface. The current live result is: health ready, packages ready, beta signup blocked, paid checkout blocked. Current blockers are hosted API-key email delivery not configured, Stripe Checkout not configured, Stripe webhook secret not configured, self-service beta key delivery not ready, and paid checkout/key delivery not ready. The public beta signup flag is enabled, but without SMTP it queues pending_delivery requests instead of issuing accounts or keys.

Stripe smoke update: scripts/scout-hosted-admin stripe-smoke now wraps scripts/stripe_test_mode_smoke.py from the main operator command surface. It checks non-secret readiness flags for beta_trial or paid packages and can create a real Stripe Checkout Session with --create-checkout once Stripe and SMTP are configured. Current live smoke against https://scout.chowmes.com fails correctly because Checkout, webhook, and key delivery are not configured.

Hosted economics command update: operators can now run scripts/scout-hosted-admin economics --format table to inspect the canonical package economics from the terminal. It prints the $10 / 1,000 standard credits candidate, all package rows, gross margin, break-even packages/month, and credit policy without reading or printing Stripe secrets, SMTP passwords, raw hosted API keys, or key hashes. The command supports Dockerized production by delegating into the running scout container when the VPS host Python cannot import Scout dependencies.

Hosted monitoring update: protected GET /v1/billing/admin/metrics now returns derived funnel and economics sections in addition to raw totals and recent rows. Operators can inspect beta registrations, delivered/pending/failed API key delivery, reissues, paid purchases, paid customers, conversion rates, revenue, estimated package loaded cost, estimated gross profit, estimated gross margin, credits used, and standard_1000 break-even pack count through scripts/scout-hosted-admin metrics --format table or JSON.

Paid checkout update: Stripe checkout provisioning now treats an existing hosted-account email as a credit top-up. Scout adds the purchased credits to the existing tenant, records the checkout against the existing tenant/key, and does not email or expose a second raw API key. First-time paid buyers still use the normal one-time key delivery path. Repeat $0 beta_trial checkouts for an existing hosted email are recorded but do not add more free credits; only paid packages can top up an existing balance.

Website checkout update: /pricing exposes readiness-gated paid hosted-credit checkout forms for standard packages. /beta is now a single registration surface that records name/email requests and, when SMTP is configured, emails a finite-credit beta API key. Stripe checkout remains the paid package path and is paused until Stripe checkout, signed webhook provisioning, and SMTP key delivery are configured and smoke-tested.

Public beta signup smoke update: operators can now run scripts/scout-hosted-admin beta-signup-smoke --email tester@example.com --name "Tester Name" after SMTP is configured. This exercises the public /v1/hosted/beta-key registration and status endpoints and verifies that an API-key email was delivered, without printing the raw key in terminal output.

Public copy alignment update: public website and README copy now matches the passwordless self-service model. Beta users register with name/email, Scout records the request in the hosted database, and API keys are delivered only by email. Scout does not display raw API keys in browser responses. The beta key email is signed by Arijit Chowdhury, Founder, Chowmes, and includes the key, credit limits, cURL test, account/usage links, and safety instructions.

Hosted ops update: Scout now has operator helpers, scripts/scout-hosted-admin validate-config and scripts/scout-hosted-admin configure-production-env, to validate then install allowlisted SMTP and Stripe values from an ignored local file into /opt/prism/scout/.env on the VPS without printing secret values. Current production still lacks SMTP and Stripe settings, so self-service email delivery and paid checkout remain blocked until those values are supplied and smoke-tested.

Hosted config template update: Scout now has scripts/scout-hosted-admin write-config-template, which writes a non-secret secrets/scout-production.env starter file for SMTP and Stripe configuration. It includes the passwordless beta signup flag, SMTP delivery fields, Stripe price/webhook fields, hosted success/cancel URLs, and the Arijit/Chowmes sender format, while refusing to overwrite an existing file unless --force is used.

Stripe price bootstrap update: Scout now has scripts/scout-hosted-admin bootstrap-stripe-prices, which reads the canonical hosted credit package model and creates one-time Stripe prices for the public paid packages standard_1000, standard_3000, and standard_15000. The helper excludes $0 beta trial setup and private browser-credit packages, can dry-run without secrets, can write returned price IDs back to the ignored secrets/scout-production.env, and refuses to print Stripe secret-looking values.

Email delivery update: The self-service beta API-key email is now a proper onboarding artifact. It is signed by Arijit Chowdhury, Founder, Chowmes, and includes the beta credit/30-day boundary, not-unlimited warning, quick curl test, account/balance, usage, purchase, docs/pricing links, secret-handling warning, and support reply instructions.

Website polish update: Scout now has a real SVG mark/wordmark, a tighter homepage density pass, scroll-aware homepage section navigation, and a simpler homepage story focused on demo, features, use cases, pricing, and beta access. The oversized Run Ledger/proof panels were replaced by a compact "Clean records with evidence attached" card so the first viewport is not swallowed by artifact labels. The public quickstart is hosted-first and uses https://scout.chowmes.com before local 127.0.0.1 examples.

Hosted VPS update: Scout is now exposed for private-beta consumers at https://scout.chowmes.com. The public route is Caddy -> localhost Scout; the container listens only on 127.0.0.1:8421. The launch website is served from the root path, while hosted API consumers use Authorization: Bearer scout_live_... against /v1/hosted/*. SCOUT_PUBLIC_HOSTED_ONLY=true disables local/admin routes publicly, including /app, /docs, and /api/config.

VPS security update: a 2026-06-29 audit found Postgres and Redis had been published on 0.0.0.0 by Docker. They are now bound to 127.0.0.1 only. External checks show only SSH, HTTP, and HTTPS are open.

The prior Scout 0.1.1 CI acquisition work remains valid: shared acquisition metadata lives on POST /scrape, not /ci/scrape. CI is a consumer of this shared primitive, not the owner of a bespoke endpoint.

Canonical Structure

Path	Purpose	Last Updated
wiki/overview.md	Product thesis, personas, and architecture direction	2026-05-14
wiki/requirements.md	Functional requirements and acceptance criteria	2026-05-14
wiki/architecture/	Architecture notes and diagrams	2026-05-21
wiki/ux/	App-first UX decisions, workflow specs, browser workbench notes, and mockup references	2026-05-22
wiki/use-cases/	Use-case catalog, target matrix, and validation roadmap	2026-05-21
wiki/decisions/	ADRs; one decision per file	2026-07-03
wiki/dev-log.md	Append-only build and discovery log	2026-07-03
wiki/open-questions.md	Living list of unresolved decisions	2026-05-22
wiki/sources/	Source notes, seed evidence, browser verification, and legacy design notes	2026-05-21
wiki/syntheses/	Strategy, build-plan syntheses, relationship map, and current status	2026-06-29
wiki/entities/	People, profiles, orgs, and private scenario notes	2026-05-17
raw/assets/	Immutable screenshots, generated images, and visual references	2026-05-21
log.md	Append-only wiki operation log	2026-07-03
AGENTS.md	Wiki schema and maintenance rules	2026-05-14

UX Workflow Specs

Workflow	File
Products and Algolia prep	wiki/ux/workflows/products-and-algolia.md
PRISM company intelligence	wiki/ux/workflows/prism-company-intelligence.md
Company intelligence	wiki/ux/workflows/company-intelligence.md
Investor intelligence	wiki/ux/workflows/investor-intelligence.md
Careers and jobs	wiki/ux/workflows/careers-and-jobs.md
News and blogs	wiki/ux/workflows/news-and-blogs.md
Research and documentation	wiki/ux/workflows/research-and-documentation.md
Website quality	wiki/ux/workflows/website-quality.md
History, presets, targets	wiki/ux/workflows/history-presets-targets.md
Data, integrations, settings, help	wiki/ux/workflows/data-integrations-settings-help.md
Browser workbench recovery	wiki/ux/browser-workbench-recovery-2026-05-22.md

Decision Records

ADR	Summary	Date	Status
wiki/decisions/2026-05-14-scout-web-to-record-engine.md	Scout is a provider-agnostic web-to-record engine	2026-05-14	Accepted
wiki/decisions/2026-05-16-provider-agnostic-run-architecture.md	Scout uses provider-agnostic run orchestration with standard artifacts	2026-05-16	Accepted
wiki/decisions/2026-05-20-scout-full-intelligence-platform.md	Scout is a multi-use-case intelligence platform	2026-05-20	Accepted
wiki/decisions/2026-05-20-citation-grade-source-evidence.md	Records need citation-grade source evidence	2026-05-20	Accepted
wiki/decisions/2026-05-20-balanced-validation-target-matrix.md	Validation targets must cover B2B, retail, public, hard-site, and travel cases	2026-05-20	Accepted
wiki/decisions/2026-05-21-ui-interaction-contract.md	Every visible enabled UI control must work or be visibly disabled	2026-05-21	Accepted
wiki/decisions/2026-05-21-app-first-validation-gate.md	App-first Scout work requires workflow specs, control inventory, live tests, and truthful reports before GitHub checkpoint	2026-05-21	Accepted
wiki/decisions/2026-05-22-browser-workbench-session-modes.md	Scout uses explicit crawler, Scout Browser, and User Browser session concepts	2026-05-22	Accepted
wiki/decisions/2026-06-27-shared-scrape-acquisition-profile.md	Scout uses shared /scrape acquisition metadata, not consumer-specific scrape endpoints	2026-06-27	Accepted
wiki/decisions/2026-06-29-private-beta-launch-surface.md	Scout is controlled private beta with explicit limits; public launch remains blocked	2026-06-29	Accepted
wiki/decisions/2026-06-29-apache-2-local-core-license.md	Scout local/core package and source distribution use Apache-2.0; hosted/service terms remain separate	2026-06-29	Accepted
wiki/decisions/2026-06-29-service-first-no-app-ui.md	Scout removes the broken /app UI and ships as a service/API/CLI/skill plus launch website	2026-06-29	Accepted
wiki/decisions/2026-07-03-self-service-beta-and-stripe-credit-packages.md	Scout uses self-service beta key registration plus Stripe checkout for beta-trial setup and paid credit packs	2026-07-03	Accepted
wiki/decisions/2026-07-03-service-key-billing-admin-metrics.md	Scout exposes non-secret hosted signup, usage, purchase, and revenue metrics behind the service API key	2026-07-03	Accepted
wiki/decisions/2026-07-03-direct-email-beta-key-registration.md	Scout beta registration collects name/email and emails the API key through /v1/hosted/beta-key; Stripe remains separate for paid/card-backed checkout	2026-07-03	Accepted
wiki/decisions/2026-07-03-email-first-beta-registration-supersedes-stripe-first.md	The deployed public beta path is email-first /v1/hosted/beta-key; older Stripe-first beta notes are superseded	2026-07-03	Accepted
wiki/decisions/2026-07-03-direct-beta-email-delivery-supersedes-card-backed-beta.md	Current operating truth: /beta supports primary direct email API-key registration plus optional $0 card-backed Stripe setup when billing is ready	2026-07-03	Accepted
wiki/decisions/2026-07-03-beta-readiness-follows-email-key-delivery.md	Hosted beta readiness follows ready_for_beta_key_delivery; Stripe checkout readiness is tracked separately	2026-07-03	Accepted

Product Syntheses

Synthesis	Summary
wiki/syntheses/unit-economics-pricing-2026-06-29.md	Current beta pricing posture: free local Scout plus metered hosted packages at $10 / 1,000, $25 / 3,000, and $100 / 15,000 standard credits; Stripe/SMTP live smoke still blocks production-complete paid self-service.

Status Pages

Page	Purpose
wiki/syntheses/current-status-2026-05-22.md	Latest built/pending/next-step summary
wiki/ux/browser-workbench-recovery-2026-05-22.md	Browser workbench UX and product workflow state
wiki/syntheses/downstream-consumers-and-project-relationships-2026-06-29.md	Who uses Scout, what feature they consume, and where interpretation belongs

Recent Activity

Date	Event
2026-07-03	Aligned repo docs, tests, and vault around approved current beta pay-as-you-go packages; full unit suite 819 passed, Pyright and Ruff clean
2026-07-03	Added scout-hosted-admin stripe-smoke wrapper for Stripe beta/paid smoke testing; live smoke fails cleanly until Stripe/SMTP secrets are configured
2026-07-03	Added failed beta delivery retry recovery via protected API and scripts/scout-hosted-admin retry-failed-signups; focused tests, Pyright, Ruff, and format checks passed
2026-07-03	Aligned pricing, quickstart, and launch docs to email-first beta registration; paid Stripe Checkout is now described as the separate hosted-credit package path
2026-07-03	Corrected /beta browser form to use ready_for_beta_key_delivery and submit email-first API-key registration instead of treating Stripe checkout as the primary beta gate
2026-07-03	Corrected hosted readiness semantics: scripts/scout-hosted-admin readiness --require-beta-signup now follows email key delivery readiness, not Stripe checkout readiness
2026-07-03	Updated beta self-service contract: /beta is now $0 card-backed beta setup when ready, with queued email registration fallback while checkout is paused; added superseding ADR 2026-07-03-card-backed-beta-with-email-queue-fallback.md
2026-07-03	Deployed commit 7900997; corrected repo docs and public /status copy so hosted beta is described as email-first registration with pending delivery while SMTP is missing
2026-07-03	Corrected current deployed beta truth: /beta is email-first registration through /v1/hosted/beta-key; Stripe-first beta notes are superseded by ADR 2026-07-03-email-first-beta-registration-supersedes-stripe-first.md; production at 858c7af is healthy but SMTP/Stripe remain unconfigured
2026-07-03	Deployed commit a1ed9ec; added protected admin endpoint to drain pending beta registrations into emailed API keys after SMTP is configured; production health OK, unauth drain 403, auth drain 503 until SMTP exists
2026-07-03	Deployed commit 3bf7bb1; health and billing packages OK, Stripe/SMTP readiness still false until production secrets are configured
2026-07-03	Closed beta-trial credit abuse path: repeat $0 beta checkout is recorded but does not add another 100 free credits
2026-07-03	Deployed commit dd5dfe6; /pricing has paid checkout form, /beta has direct key registration plus optional $0 beta checkout; Stripe/SMTP readiness still false
2026-07-03	Added VPS hosted env configuration helper; local gate clean, awaiting real SMTP and Stripe test-mode values
2026-07-03	Deployed commit bc58ff0; VPS repo now has configure-production-env helper and Scout container remains healthy
2026-07-03	Upgraded beta API-key email copy and tests; local focused delivery/webhook/signup tests passed
2026-07-03	Deployed commit 5b42a0b; production health OK, beta key email code is live, Stripe/SMTP readiness still false until secrets are configured
2026-07-03	Added hosted beta signup telemetry ledger; admin metrics now shows delivered, failed, and duplicate beta key request outcomes
2026-07-03	Hardened Stripe Checkout API to fail closed unless webhook signing and hosted key delivery are ready
2026-07-03	Deployed commit bac6eba to scout.chowmes.com; health OK, Stripe/SMTP readiness still false until secrets are configured
2026-07-03	Added paid checkout top-up semantics for existing hosted accounts; no duplicate tenant and no second raw API-key delivery
2026-07-03	Repointed /beta to direct name/email API-key registration via /v1/hosted/beta-key; Stripe remains the separate paid/card-backed checkout path
2026-07-03	Deployed commit ac80f29; /v1/billing/admin/metrics verified through scout.chowmes.com with service-key auth and missing-key 403
2026-07-03	Added service-key protected /v1/billing/admin/metrics for hosted beta signup, credit, usage, purchase, and revenue monitoring without exposing raw keys
2026-07-03	Tightened hosted beta self-service contract: no browser/raw-key fallback, no raw_api_key OpenAPI field, and missing SMTP fails closed before account creation
2026-07-03	Self-service beta registration and Stripe credit package decision recorded; /beta uses /v1/hosted/beta-key; /pricing exposes $0 beta_trial and paid packages
2026-07-03	Production verified at commit d6fc212; beta page patched to preflight key-delivery readiness and pause registration when SMTP is missing
2026-07-03	Added hosted admin usage-ledger monitoring script and verified production summary output
2026-06-30	Website correction: compact homepage proof card and hosted-first quickstart with domain examples before localhost
2026-06-29	Removed legacy /app UI and /api/config, simplified homepage, pushed commit d27b24e, and redeployed https://scout.chowmes.com; old app routes return 403
2026-06-29	Apache-2.0 local/core license implemented and verified; public launch remains blocked with 9 blockers
2026-06-29	Scout website launched at https://scout.chowmes.com; hosted API remains under /v1/hosted/*; local/admin routes blocked publicly
2026-06-29	VPS security checkpoint: Postgres and Redis moved from public Docker bindings to localhost-only bindings; stale Temporal Caddy host removed
2026-06-29	Private beta launch surface recorded; downstream relationship map linked to PRISM, Competitive Intelligence, Algolia Search Audit, product/catalog, and job workflows
2026-06-29	Hosted VPS route verified at https://judge.contentengagement.info/scout; hosted Bearer-key /v1/hosted/me and /v1/hosted/scrape smoke passed; local/admin routes disabled publicly
2026-06-27	Scout 0.1.1 released: shared /scrape acquisition metadata, scout benchmark, no /ci/scrape
2026-05-22	Browser Workbench recovery recorded: scout-browser captures screenshot/DOM/text/link evidence; User Browser bridge remains next
2026-05-22	Current status page added with built/pending/next-step summary
2026-05-21	App-first workflow specs created under wiki/ux/workflows/ and approved mockup saved under raw/assets/
2026-05-21	Loose legacy top-level design files moved into wiki/sources/legacy/
2026-05-21	UI interaction contract and validation gate recorded
2026-05-20	Full intelligence platform, citation evidence, and balanced target matrix recorded
2026-05-19	Job Hunter URL-seeded V1 build completed and verified