Crawler-Factory

Decisions/2026-05-03-Phase1ScreenInventory.md

Phase 1 Screen Inventory

Scope. Phase 1 = the web crawler module of Crawler Factory, embedded inside the Algolia dashboard at Data sources → Crawler. This document lists every screen, every state, every modal, every action surface that ships in Phase 1. It is the spec the React refactor reads from.

Out of scope. Phase 2 connectors (CMS, KB, video, code), Phase 3+ pipeline stages (Transformer, Enrichment, Indexing, Agent Studio). See **2026-05-03-EmbeddedArchitecture** Phased Roadmap.

Outer chrome assumption. Algolia's outer dashboard chrome (left product rail, Data sources sub-nav, top bar) wraps everything below. We do not draw outer chrome.

Screen inventory at a glance

#	Screen	Type	Source pattern	Status
1	Crawler list view	Page	Recipe 17 (toolbar) + Recipe 18 (table) + Recipe 23 (Run-State pill)	New (didn't exist in v1)
2	Crawler list — empty state	State	Recipe 13 — gated variant	New
3	"Add new crawler" choice modal	Modal	Recipe 9 with two CTA cards	New
4	Crawler Factory wizard — Step 1: Discover	Modal step	Recipe 9 + form rows	Refactor of `01-discover-screen.html`
5	Wizard — Step 2: Review (live discovery)	Modal step	Recipe 12 (KPIs) + tree component	Refactor of `02-live-discovery.html`
6	Wizard — Step 2: Review — empty/error states	States	Recipe 13	New
7	Wizard — Step 3: Configure	Modal step	Recipe 14 (tabs) + Recipe 1-7 (form rows)	Refactor of `03-configure.html`
8	Wizard — Step 4: Build summary	Modal step	Confirmation card	New (replaces `04-crawl.html`)
9	Wizard — wizard-level error state	State	Recipe 13 + retry CTA	New
10	Per-crawler focused workspace shell	Workspace	Recipe 15 (3-section nav) + Recipe 22 (RunControlCluster)	New (extends Algolia's existing)
11	Focused workspace — Overview page	Page	Existing Algolia pattern + KPI tiles	Light refactor
12	Focused workspace — Configuration page	Page	Existing Algolia setting cards	Use as-is
13	Focused workspace — Editor page	Page	Existing Algolia code editor	Use as-is
14	Focused workspace — Suggestions page	Page	Existing Algolia	Use as-is
15	Focused workspace — URL Inspector	Page	Existing Algolia	Use as-is
16	Focused workspace — Monitoring	Page	Existing Algolia	Use as-is
17	Focused workspace — Data Analysis	Page	Existing Algolia	Use as-is
18	Focused workspace — Path Explorer	Page	Existing Algolia + Crawler Factory metadata layer	Light extension
19	Focused workspace — Logs Explorer	Page	Existing Algolia	Use as-is
20	Focused workspace — Settings	Page	Existing Algolia + extra Crawler-Factory metadata	Light extension
21	Re-crawl confirm modal	Modal	Recipe 24	New
22	Stop crawl confirm modal	Modal	Lighter than Recipe 24 — single-line confirm	New
23	Delete crawler typed-confirm	Modal	Recipe 10	Use Algolia's
24	Bulk action confirm modal	Modal	Recipe 9	New

Total Phase 1 screens to design or refactor: ~14 net new + 10 reuse-as-is.

Detail per screen

1. Crawler list view

Location: Data sources → Crawler (replaces today's Crawler list page; both Factory-built and hand-built crawlers visible).

Anatomy: - Page H1: Crawler (matches Algolia today) - 4 tabs unchanged from Algolia today: Crawlers (active by default) / Domains / External Data / Settings - Info banner (Recipe 19): Crawler Factory · We can auto-discover your site and build N crawlers per content type. with Try it → link to the choice modal. This banner is dismissable but reappears for new users. - Toolbar (Recipe 17): - Left: search input + status filter dropdown (All Status, Idle, Running, Paused, Failed, Done) - Right: secondary Recrawl failed URLs (bulk action), primary Add new crawler (opens choice modal — Screen 3). - Table (Recipe 18): - Columns: Name (clickable link → focused workspace), Run state (Run-State pill — Recipe 23), Last indexed, Author, Source (badge: Factory if built via Crawler Factory, Manual if hand-built — new column), 3-dot row menu. - Row 3-dot menu: stateful per Run-Control Surface (Pause/Resume/Stop/Crawl now/Open/Settings/Delete). - Bulk-select (checkbox column): bulk toolbar appears at top of table when ≥1 row checked. Bulk actions: Pause all selected, Resume all selected, Stop all selected, Crawl all selected. Scoped to current page only (per design pack §2.5). - Pagination at bottom.

States: - Empty (no crawlers yet): see Screen 2. - Loading: skeleton rows. - Error: full-width error banner above the table with retry CTA.

2. Crawler list — empty state

Triggered when: the application has zero crawlers.

Anatomy: Recipe 13 gated variant. - Centered illustration (small Algolia-style line drawing of a globe + spider). - Heading: No crawlers yet. - Body: Crawler Factory can discover your site and build a set of crawlers in 4 steps. Or you can configure one by hand. - Primary CTA: Use Crawler Factory → opens wizard (Step 1 / Screen 4). - Secondary CTA: Build manually → opens Algolia's existing manual crawler creation flow.

Triggered when: user clicks the Add new crawler button on the list view.

Anatomy: Modal (Recipe 9 variant) with title How would you like to add a crawler? and two side-by-side CTA cards (each a clickable card, not a radio):

Card	Title	Description	Indicator
Left	Use Crawler Factory (recommended)	We discover your site, classify the content, and build one crawler per content type. ~4 minutes.	`RECOMMENDED` pill, top-right
Right	Build manually	Configure a single crawler by hand. For advanced or unusual sources.	none

Footer: Cancel (ghost) only. Clicking a card progresses to the chosen flow.

4. Wizard — Step 1: Discover

Triggered when: user picks Use Crawler Factory from the choice modal.

Anatomy: Modal with header (Add a new crawler title + Step 1 of 4 indicator + close X), body, footer.

Body: - Section title: Tell us your website - Single-line functional intro: We'll discover your content structure and create one crawler per content type. - Form row: Domain label + scheme dropdown (https://) + URL input + inline error slot. Live-validate; disable Continue until valid. - Suggestion pills below input: algolia.com, shopify.com, stripe.com (ghost-style; clicking fills the input). - Info banner under the form (Recipe 19): We only read public pages. No login, no scraping behind paywalls.

Footer: Cancel (ghost left) + Continue (primary right). Continue triggers the discovery API call.

States: - Continue button has a loading state (spinner replacing label, button disabled) while the discovery starts. - If the API returns an error (DNS fail, WAF block, malformed URL), inline error below the input.

5. Wizard — Step 2: Review (live discovery)

Triggered when: Step 1 successfully kicks off discovery.

Anatomy: Modal — same shell as Step 1, but the body switches to the live-discovery layout.

Body: - Section title: <domain> · Discovering content (becomes Review the categories once classifying completes). - Status line: <spinner> Walking sitemap… → <spinner> Grouping URLs… → <spinner> Classifying content… → Discovery complete — review the categories below. (single live-pulse dot, opacity-only per Recipe 23.) - Thin progress bar (8px, no shimmer, fills smoothly). - 4 KPI tiles per Recipe 12: Sitemaps found / Unique URLs / Categories found / Elapsed. Numbers update without ramping animation. - Tree component: - Root row: <globe icon> <domain> · <total URL count> - Branch rows (one per category): [checkbox] <grey icon tile> <category name> <confidence pill> <path glob> left-aligned + <URL count> right-aligned + <3-dot menu> for Rename / Merge with… / Split / Exclude. - Branches appear in place when classified — no entrance animation. - In-flight branch: row with <spinner> glyph and the literal text Classifying… and the path glob.

Footer: Back (ghost left) + Confirm categories (primary right). Disabled until all classification is complete and at least 1 category is selected.

States: - Loading: progress bar partial fill, KPIs ticking, branches appearing. - Empty (zero categories detected, e.g., site has no sitemap): Recipe 13 gated state inside the modal body. CTA: Try a different domain (returns to Step 1) or Build manually (drops the user out of Factory into Algolia's manual flow). - Error (discovery failed mid-way): inline error banner with Retry CTA. - WAF-detected: warning banner: This site appears to block our crawler. Phase 2 will support cookied/authenticated fetch. + CTA to switch to manual.

6. Wizard — Step 2 — empty / error states

Specced inline above. Captured here as a separate row in the inventory because they need explicit design.

7. Wizard — Step 3: Configure

Triggered when: user clicks Confirm categories on Step 2.

Anatomy: Modal — same shell.

Body: - Section title: Configure your crawlers · <N> categories - Honest readiness summary line: <X> of <N> ready where ready = (index name set + ≥3 fields + extractor test passed). Highlights the incomplete categories. - Two-column body: - Left rail (Recipe 15 nav style): one row per category. Active row has 2px blue left bar. Each row shows a small status pill (pill--success if ready, pill--warning if incomplete) and a per-category URL count. - Right pane: tabbed (Recipe 14 underline tabs): - Index name tab: input + description + index-name validation (regex, uniqueness check). - Fields tab: selectable card grid of detected fields with inline check toggles. Each field: name + type + sample value preview. + Add custom field opens Recipe 9 modal. Apply to all categories toggle per field. - Test extractor tab: 3-5 sample URLs from the category. Each shows an extracted record preview side-by-side with the source URL. Per-URL Re-run. Aggregate status: All passing / <n> failing with details.

Footer: Back (ghost) + Build crawlers (primary, disabled until all categories ready). Clicking Build crawlers does not start crawling — it creates the N crawler records in Algolia and dismisses to Step 4.

8. Wizard — Step 4: Build summary (replaces old `04-crawl.html`)

Triggered when: Step 3 successfully creates the crawlers.

Anatomy: Modal — final step.

Body: - Large success illustration (Algolia-style minimal). - Title: <N> crawlers created. - Body: A vertical list of the created crawlers. Each row: - Name (<domain>_products etc.) - Index name - URL count - Status: Idle (not yet crawled) - A per-row link Open → → focused workspace. - Bulk actions: - Crawl all now (primary) — fires Crawl now on all N crawlers and dismisses the modal back to the list view. - Crawl later (ghost) — dismisses the modal back to the list view; user starts crawls manually.

Footer: Same buttons as the body's bulk actions, repeated in standard footer position.

Why no "watch them crawl" inside the wizard: Per the Embedded Architecture decision, the per-crawler progress lives in each crawler's focused workspace's Overview page (Algolia native pattern). The list view's Run-State pill column shows progress at a glance. The wizard's job is creation; status is the workspace's job.

9. Wizard — wizard-level error state

If the wizard fails irrecoverably mid-flow (network down, server error), the modal body switches to a Recipe 13 error variant: title (Something went wrong), body (Your progress was saved. Resume to continue.), CTAs: Resume (re-attempts the failed step) + Cancel (closes the modal; user can resume later from the list view).

10. Per-crawler focused workspace shell

Triggered when: user clicks any crawler name in the list view, or a Open → link from the wizard's Step 4.

Anatomy: Recipe 15 + Recipe 22. - Top-left: CRAWLER wordmark (matches Algolia today; not CRAWLER FACTORY). - Center-top: scope indicator — small Crawler eyebrow + crawler avatar + crawler name dropdown (switch between crawlers). - Top-right: Run-Control Cluster per Recipe 22. Stateful primary button + 3-dot overflow menu. Plus close X to return to the list. - Left rail: 3 sections (SETUP / STATUS / CONFIGURATION) per Recipe 15. Items match Algolia's existing nav: Overview / Configuration / Editor / Suggestions; URL Inspector / Monitoring / Data Analysis / Path Explorer / Logs Explorer / Indices; External Data / Settings. - Main content area renders the active page (Screens 11-20).

States of the Run-Control Cluster: stateful per **2026-05-03-RunControlSurface**.

11. Focused workspace — Overview

Largely Algolia's existing pattern (2-overview.png capture). Phase 1 enhancements: - The KPI tile row at top now includes Crawler-Factory metadata: Source: Factory and Blueprint: <domain>/<category>. - The "Last complete crawl" card includes a Compare with previous link (Algolia native). - The Schedule card surfaces our per-crawler schedule edit (Algolia native form pattern).

12-19. Focused workspace pages 12-19

Use Algolia's existing patterns as-is. No Phase 1 redesign needed beyond making sure our React shell renders them via the existing components or close equivalents.

20. Focused workspace — Settings

Algolia's existing Settings page (22-settings.png capture) plus a Crawler Factory Metadata section above the danger zone: - Source: Factory v1.1 - Blueprint: <blueprint URL> (read-only, copyable). - Discovery session: <session-id> (read-only, copyable). - Re-discover from scratch (button): opens Recipe 24 confirm modal scoped at "this rebuilds the blueprint and may produce a different category split." - Danger zone: Delete this crawler per Recipe 10 (use Algolia's existing).

Recipe 24. Triggered from the Run-Control Cluster's "Re-crawl from scratch" overflow item, or from the Settings page's "Re-discover" button.

Lighter than Recipe 24 because Stop is reversible by Crawl-now-again. Single-line modal: - Title: Stop this crawl? - Body: Indexed records will be kept. The crawler will go back to Idle. - CTAs: Cancel (ghost) + Stop (primary blue, not red — non-destructive).

23. Delete crawler typed-confirm

Recipe 10 (Algolia native pattern). Use as-is.

When the user bulk-selects rows on the list view and clicks Stop all selected (or any other bulk action), a confirm modal appears: - Title: Stop <N> crawlers? - Body: Indexed records will be kept. All N crawlers will go to Idle. - CTAs: Cancel + Stop <N> crawlers.

For non-destructive bulk actions (Crawl all selected), no confirm modal — just fire and toast (wait — toasts are banned per design pack; replace with a quiet inline status row above the table for 3 seconds).

Cross-cutting requirements

Requirement	Where it shows up
Empty states	Screens 1 (empty list), 5 (no categories), 8 (no crawlers created — error path)
Error states	Screens 1, 5, 9, 21, 22
Loading states	Screens 1 (skeleton table), 4 (continue button), 5 (in-flight discovery), 7 (extractor test)
Keyboard accessibility	All screens. Tab order. Focus rings (`var(--accent-ring)`). Modal focus trap. Escape to close. Enter to submit.
Mobile	Out of scope for Phase 1 (Algolia dashboard is desktop-first; the Crawlers area is not mobile-optimized today).
Dark mode	Out of scope for Phase 1 (Algolia dashboard light-only; Editor is the only dark surface).
i18n	Use plain English copy in Phase 1; i18n-ready structure (no concatenated strings, no hardcoded text in components) but no translations shipped.
Telemetry	Every wizard step transition fires an event. Every Run-Control Cluster action fires an event. Every error state fires an event. Naming: `crawler_factory.<screen>.<action>`.

What this enables in Session 3

Reading this inventory + the gap analysis (**Session-2b-MockupGapAnalysis**) + the design pack should be enough to:

Refactor the existing React UI without further design questions.
Build the missing list view + Run-Control Cluster + Run-State Pill.
Drop the standalone outer chrome and prepare for embedded integration.
Ship the Phase 1 build as a self-contained module.

Open architectural questions remaining (deferred to Session 3+):

Question	Owner
Algolia Crawler API support for Pause/Resume — primitive or do we track state?	Backend spike
When Crawler Factory integrates into Algolia's actual codebase, what's the integration mechanism (PR to shared repo, federated module, micro-frontend)?	Algolia internal product alignment
Bulk action rate-limiting — what's the API ceiling?	Backend spike
When does Phase 2 start? Which connector ships next?	Roadmap discussion