Crawler-Factory

Sessions/Session-2-DesignLanguage.md

Session 2 — Design Language & UX Pivot

Session goal as stated by the user: "I'm going to stop here and pivot to first elaborating the user experience and locking down the screens and features/functionalities first. Functionality will match what UI says."

What actually happened: UX redesign, then Algolia dashboard reverse-engineering, then portable design language pack, then framing pause/resume/re-crawl as our value-add over Algolia's primitive set.


1. Where Session 1 left off

Session 1 (2026-04-30) shipped the v1 spike: 13 engineering specs, 897 tests passing, tsc clean. Backend modules built. Frontend components built. End-to-end Playwright-verified: discovery → category review → 3-tab configurator. Standalone scaffold complete.

Then Session 1 ended with a UI that didn't feel right. The 4-step wizard was functional but flat: "Discover content" hook copy was generic, the live log panel was empty real estate, the category review screen was confusing, the configurator UX didn't match what was in the user's head.

2. The pivot — UX before functionality

The user explicitly stopped functional work and said: lock down the screens and features first. Functionality will follow the UI.

This was the right call. The reasoning surfaced explicitly: - The first screen is "the system hook." It must be self-explanatory to every type of user, especially business users. - The top bar is not just a label — it should be a breadcrumb + progress indicator that shows where the operator is in the journey. - The live log isn't a text dump. It's a visual story — a tree that builds branch-by-branch as URLs are discovered.

3. What we built in Session 2

A. Clickable HTML mockup prototype — docs/workspace/ux-redesign/

A 4-screen end-to-end clickable prototype of the entire flow, navigable forward/back with state persistence between screens.

File Screen
00-index.html Designer's launcher — start prototype, jump to any screen, reset state
01-discover-screen.html Step 1 — punchy hook, URL input, "How it works" rail
02-live-discovery.html Step 2 — animated 4-stage progress bar + 4 live counters + branch-by-branch tree
03-configure.html Step 3 — per-category left rail + tab panel (Index name / Fields / Test extractor)
04-crawl.html Step 4 — staggered crawler launches, live record counters, indexing feed
_shared.css, _shared.js Shared design tokens + navigation helpers

Each screen functionally clickable. State persists via sessionStorage. Breadcrumb is real (locked future steps; clickable past steps).

B. Senior UX Designer review of the mockup

User asked for a critical review. Output: ~30 specific issues across 4 screens, ranked top 8 by impact. Summary verdict: "the bones are right, the polish isn't." Top issues: 1. Edit/merge/rename categories on Screen 2 — biggest functional gap. 2. Pause-all + error states on Screen 4 — production blocker. 3. Empty/error states everywhere — designed only the happy path. 4. Real readiness check on Screen 3 — currently lies. 5. Visible focus styles + keyboard tab order — a11y blocker. 6. Loading state on Screen 1 Start button. 7. Index name + field validation on Screen 3. 8. Honest time framing — "in minutes" promise will break.

C. Algolia dashboard reverse-engineering

Captured 32 screenshots of the live Algolia dashboard (Wellington crawler, app 0EXRPAXB56) covering: - Outer dashboard chrome (crawlers list) - Focused workspace chrome (overview, configuration with all 4 tabs, editor, suggestions, all 6 status sub-pages, external data, settings) - Edit-button interactions revealing input control library (toggle, conditional fieldset, modal-for-add, repeating row, day-picker pills, number stepper, conditional dropdown, safety-rail info banner) - Destructive flow (Delete crawler typed-confirm modal)

Findings distilled into a 319-line knowledge document: docs/design-language/00-findings.md.

D. The portable Design Language Pack v1.1

Five-file, ~1,940-line design system that travels with the project. Lives in: - Repo: docs/design-language/ (4 .md + tokens.css) - Vault (this folder): Projects/Crawler-Factory/Design-Language/ (mirror copy, see §5)

File Purpose
README.md Map of the pack
00-findings.md Knowledge from the 32 screenshots — every pattern, every input control, every state
01-design-language.md The portable design pack — tokens, components, voice, decision tree, what's banned
02-component-recipes.md 24 copy-paste-ready HTML/JSX skeletons
tokens.css Drop-in CSS variables + base classes

Design principles distilled in 7 lines: 1. Two chrome modes: outer dashboard for browsing, focused workspace for operating one asset. 2. Vertical left rail with three small-caps sections; subtle blue left-bar for active. 3. Page = H1 + optional description + optional underlined tabs. 4. Setting cards are the workhorse — icon + title + description + value + Edit; one per row; expand inline. 5. Edit existing → inline expansion. Create new → modal. Never mix. 6. Persistent top-right run-control cluster (CTA + per-crawler controls); close X exits. 7. Restraint: no gradients, no toasts, no animations, generous whitespace, blue is the only accent.

E. The value-add framing — pause/resume/re-crawl

Algolia's dashboard has no pause / resume / re-crawl primitive. Crawls are fire-and-forget. Pause is our value-add.

This was a key Session 2 decision. Captured in: - Decisions/2026-05-03-RunControlSurface.md — the durable record - Design Language Pack §2.5 — the spec - Component Recipes §22-24 — the implementation skeletons

The 5 run states with their stateful CTA cluster:

State Primary Overflow
idle Crawl now Edit configuration · Re-crawl from scratch · Settings
running Pause Stop crawl · View live log · Settings
paused Resume Stop crawl · Edit configuration · Re-crawl from scratch
stopped Crawl now Re-crawl from scratch · Edit configuration · Settings
failed Retry View error log · Edit configuration · Settings

F. The 500-error root cause investigation

Mid-session debug: pino-pretty not installed in node_modules but referenced as a transport target in lib/utils/logger.ts. Fix is a one-liner: npm install pino-pretty. Did not apply during Session 2 (we pivoted to UX before functionality). Logged in the repo's SESSION.md and in Status.md.

4. Decisions locked in Session 2

See Decisions/ folder for the canonical files. Summary:

# Decision File
D1 ALPHA badge stays in top bar (build/version tag, not session indicator) embedded in 01-design-language.md
D2 Live tree shows branches only, never URL leaves embedded in 01-design-language.md
D3 Two-chrome rule: outer dashboard for browsing, focused workspace for operating Decisions/2026-05-03-TwoChromeRule.md
D4 Pause / Resume / Re-crawl / Stop = our value-add over Algolia's fire-and-forget model Decisions/2026-05-03-RunControlSurface.md
D5 Match Algolia's existing dashboard design language; no parallel design system embedded in 01-design-language.md
D6 Re-crawl uses scoped confirm dialog (not typed-confirm); typed-confirm reserved for true irreversibility (Delete only) embedded in 01-design-language.md

5. The repo / vault relationship

The Design Language Pack lives in two places by design: - Repo docs/design-language/ — used during the build, imported by the project's CSS. - Vault Projects/Crawler-Factory/Design-Language/ — used cross-session and cross-project. Portable.

When the pack version changes, update both. The repo is the working copy; the vault is the cross-session canonical.

6. Session 2 deliverables checklist

  • [x] 4-screen clickable HTML prototype
  • [x] Senior UX review with prioritized issues
  • [x] 32 dashboard screenshots captured
  • [x] Findings document (319 lines)
  • [x] Design Language Pack v1.0 → v1.1 (added value-add run-control surface)
  • [x] 24 component recipes
  • [x] Algolia Central context document
  • [x] Bidirectional vault links
  • [x] Session log (this file)
  • [ ] Repo committed to GitHub (in progress at session end — see Status.md)

7. What Session 3 should pick up

Priority Item Source
P0 Fix the 500 on /api/factory/discover (npm install pino-pretty) SESSION.md in repo
P0 Initialize git, push to GitHub user requested 2026-05-03
P1 Refactor existing React UI to match Design Language Pack v1.1 this session
P1 Implement RunControlCluster + RunStatePill (Recipes §22-23) Decisions/2026-05-03-RunControlSurface.md
P2 Implement edit/merge/rename categories on Screen 2 (the biggest UX gap) senior UX review §4 issue #4
P2 Add empty / error / loading states everywhere senior UX review
P3 Vault: spec out future Algolia Central modules (Transformer, Enrichment, Indexing, Agent Studio) using the same design pack new