Dashboard/Crawler-Factory/Wiki/UserFlows.md
Crawler-Factory

Wiki/UserFlows.md

User Flows — Crawler Factory

Last updated: 2026-05-03 (S4) Source: Decisions/, Design-Language/01-design-language, Status.md

Flow Index

Flow Entry point Goal Demo-ready?
Create a New Factory /admin/factory → "Add new crawler" Scaffold N crawlers from a URL ⚠️ Partial — UI shipped S4; backend wiring pending S5 (UI-004)
Monitor a Live Crawl Factory workspace → Status page Watch real-time crawl progress ⚠️ Partial (LiveCrawlMonitor placeholder)
Review & Configure Domains Factory workspace → Setup → Review step Edit domain classification before crawlers are created ✅
Pause / Resume / Re-crawl Factory workspace → Run control cluster (top-right) Control long-running ingestion ⚠️ Partial (UI built; API support open OQ-4)
View Factory List /admin/factory See all factories; open a workspace ✅ Shipped S4 — factory entity model in place

Flow: Create a New Factory

Audience: Admin / developer setting up a new site ingestion Entry point: /admin/factory → "Add new crawler" button Goal: Scaffold N production Algolia crawlers (one per content domain) from a URL Approximate time: 3–10 min depending on site size

Steps

  1. User clicks "Add new crawler" → System creates a factory entity (e.g. algolia_com) → Factory workspace opens immediately (WorkspaceShell)
  2. Active page = Setup (first item under SETUP section in left nav). Outer Algolia chrome (top bar + left nav) stays visible throughout.
  3. Step 1 — Discover: User enters the target URL (e.g. https://www.algolia.com) → System starts streaming URL walk → RollingLog shows URLs arriving in real time → DomainBadge counts update per domain as they're detected
  4. Discovery completes → System shows detected content domains with record counts (e.g. Marketing: 847 URLs, Support: 2,341 URLs, Blog: 412 URLs)
  5. Step 2 — Review: User sees domain classification table → Can promote, demote, or merge domains → Clicks "Confirm domains"
  6. Step 3 — Configure: System samples HTML per domain → Generates recordExtractor JS → Runs sandbox test → Shows extraction preview per domain → User can edit extractors inline
  7. Test crawl: System calls crawl_urls real test against Algolia Crawler API → Shows sample records → User approves
  8. Step 4 — Done: User clicks "Create crawlers" → System creates N crawlers + N indices → Shows created crawlers with links → Workspace transitions to Overview page

Key UI states

  • Streaming discovery: RollingLog + DomainBadge updating live (SSE stream)
  • Domain review table: each domain = name + URL count + detected CMS + edit controls
  • Extractor preview: code editor (ConfigEditor) + sample record output (SamplePreview) side by side
  • Done state: summary card with N crawlers created, links to each in Algolia dashboard

Edge cases

  • WAF-blocked pages: Playwright stealth fetch kicks in automatically; user sees no difference
  • No domains detected: factory shows error with suggestion to check URL accessibility
  • Extractor sandbox failure: system flags the domain, allows manual extractor entry

Demo notes

See DemoScript for full narration.

Flow: Monitor a Live Crawl

Audience: Admin watching an in-progress crawl Entry point: Factory workspace → Status page (left nav) Goal: See real-time progress of running crawlers Approximate time: passive monitoring, no time limit

Steps

  1. User opens factory workspace → clicks "Status" in left nav
  2. LiveCrawlMonitor shows progress bars per domain (URLs crawled / total)
  3. Run control cluster (top-right): Pause / Resume / Stop buttons active during crawl
  4. User can click "Pause" → all sub-crawlers pause → button changes to "Resume"
  5. Crawl completes → status badges flip to ✅ → user can open blueprint or navigate to Algolia index

Key UI states

  • Running: progress bars, elapsed time, URLs/min rate
  • Paused: all progress frozen, "Resume" button highlighted
  • Complete: ✅ badges, index record counts, blueprint link

Demo notes

See DemoScript for full narration.

Flow: View Factory List

Audience: Admin returning to existing factories Entry point: /admin/factory (CrawlerListView) Goal: See all factories; open one to manage

Steps

  1. User navigates to /admin/factory → CrawlerListView shows one row per factory
  2. Each factory row shows: factory name, target domain, N sub-crawlers, last run status, Run control cluster (per-row)
  3. User clicks factory row → WorkspaceShell opens for that factory

Key UI states

  • Factory row (collapsed): name + domain + crawler count + status badge + controls
  • Sub-crawlers (inside workspace, not in list): only visible after opening a factory workspace

Demo notes

See DemoScript for full narration.