Dashboard/Crawler-Factory/Design-Language/00-findings.md
Crawler-Factory

Design-Language/00-findings.md

Algolia Crawler Dashboard — Findings

Source: 32 screenshots in docs/dashboard_screenshots/ captured 2026-05-03 from the live Algolia dashboard (dashboard.algolia.com) using the Wellington crawler in app 0EXRPAXB56.

Purpose: Distill what we observed so it can be re-used by any Claude session, by any teammate, without re-walking the dashboard. This is the durable knowledge.

1. The two-chrome rule

Algolia uses two distinct chrome modes depending on whether the user is browsing assets or operating one.

Mode When Chrome
Outer dashboard Browsing the application's product surface (Indices, Crawlers list, Connectors, etc.) Pink/red Algolia mark + far-left product rail with icons + secondary nav rail ("Data sources" → Events / Connectors / Indices / Crawler / Ask AI) + top bar with app picker, global search, Upgrade, AI Assist, Resources, user avatar
Focused workspace Operating a single asset (one crawler, in our case) Replaces all outer chrome. Wordmark "CRAWLER" with a green dot mark at top-left. Center-top: scope indicator ("Crawler" eyebrow + crawler name "Wellington" with avatar swatch). Top-right: persistent primary CTA ("Start Crawling" / "Crawl now") + close X to exit back to the outer dashboard. Left vertical sectioned nav.

Implication for our redesign: Crawler Factory needs both. The list view lives in outer chrome; opening any crawler drops the user into a focused workspace. Each future Algolia Central module (connectors, transformers, enhancers, agents) gets its own focused workspace with its own wordmark.

2. Left navigation — vertical, three sections, collapsible

The focused workspace has a single left rail with three sections, each with a small ALL-CAPS header and a collapse chevron.

SETUP ▼
  Overview
  Configuration  ← active
  Editor
  Suggestions

STATUS ▼
  URL inspector
  Monitoring
  Data Analysis
  Path Explorer
  Logs Explorer
  Indices

CONFIGURATION ▼
  External Data
  Settings

Pattern rules: - Section headers: 11px / 700 weight / letter-spacing 0.08em / muted ink color / a small leading icon (trend, wrench, gear). - Items: 14px / 400 weight / sentence case / muted ink at rest. - Active item: a thin 2px blue vertical bar on the left edge + blue text. No filled pill background. The active state is intentionally subtle. - Hover: text color brightens slightly. No pill, no underline. - Section collapse: arrow chevron next to header, click to collapse the items.

Note: The "Indices" entry under STATUS redirects out of the focused workspace to the global Indices view (it is not a crawler-scoped page). This is a leaky abstraction in their nav; we should not replicate.

3. Page header — the canonical pattern

Every focused-workspace page has the same header structure:

<Page title (24px / 700)>          <optional secondary control on the right>
<optional 1-2 line description (13px muted)>
<optional inline tabs (bottom-border underline, blue active)>

Examples: - Configuration: title + description + 4 tabs (Initial Set-up / Markdown for LLMs / Safety Checks & Performance / Security & Data Management). - Suggestions: title + intro + 2 tabs (Suggestions 2 / Dismissed Suggestions 0) — tabs include count badges. - Data Analysis: title + index dropdown (right-aligned) + Primary/Temporary index toggle pills. - Settings: title only, no description, no tabs.

Tab style: bottom-border underline only (~2px, accent blue). Inactive tabs are muted ink. No pills, no boxed tabs.

Tab badges: small rounded count chip next to label (e.g. "Suggestions 2"). Filled lighter when inactive, accent-blue when active.

4. Setting card — the most repeated atom

Every Configuration tab is a vertical stack of "setting cards" — the workhorse component of the dashboard. Anatomy:

┌──────────────────────────────────────────────────────────────────────┐
│ [icon]   Setting Name                              <value>  [Edit]   │
│          One-line description.                                       │
└──────────────────────────────────────────────────────────────────────┘

Properties: - Full width of content column. - Light grey 1px border, ~12px radius, white background. - ~28px internal padding. - Left: small grey square icon tile (32px) + bold title + muted single-line description. - Right: current value as muted text (e.g. "Enabled", "Disabled", "Enabled — 10%", "on the 28 day of the month") + secondary "Edit" button. - Cards stack with ~16px gap. - Cards never wrap to two columns. Always one per row.

When the card is in edit mode (clicked Edit), the card expands inline (does NOT open a modal). The right-side value + Edit disappear. Below the title/description, the controls render. A "Save" (primary, becomes blue when dirty) + "Cancel" (ghost) row sits at the bottom of the expanded card.

When edits are submitted, the card collapses back. No toasts observed — the value just updates.

5. Input control library

From the Edit interactions we captured, the control set is:

Control Where seen Notes
Toggle switch Schedule, Max URLs, Exclusion, JS Rendering, Record Loss, Website Auth, Index Backup, Crawler Logs Pill style. Blue dot/track when on, grey when off. Always paired with an "Enabled" / "Disabled" label to its right.
Day-picker pills Schedule (Su M Tu W Th F S) A row of small square buttons. Selected day fills blue.
Frequency dropdown Schedule (Week) Native-styled select.
Number stepper Max URLs ("100 URLs"), Record Loss ("10 %") Inline number with up/down chevrons + a unit label after it.
Text input Index Name, Crawler Name, Auth URL, Username, Password Standard 1px border, 12px radius, accent-blue focus ring.
Scheme + path duo Start URLs, Auth URL Small https:// dropdown on the left, full-width URL field on the right.
Repeating-row pattern Start URLs, URL Exclusion Rules Row of inputs + trash icon at right. "+ Start URL" / "+ URL Exclusion" ghost button below to append a new row.
Conditional fieldset Website Auth Toggle → reveals method dropdown → reveals a labelled fieldset containing the method's specific inputs. Three levels of progressive disclosure.
Read-only with copy Settings (Crawler ID, App ID) Greyed-out text input with a copy-to-clipboard icon button alongside.
Masked secret with reveal Settings (Algolia API Key) Mask •••••• + eye icon to reveal + copy icon.
Inline info banner Record Loss Policy ("won't update until you check and confirm") Light blue/grey background, info icon, single line, optional "Learn more" link. Used for safety rails on consequential actions.

6. The two creation patterns — modal vs inline

Algolia uses two distinct creation patterns:

Pattern When Example
Inline expansion Editing an existing setting on a card All Configuration card "Edit" buttons — schedule, max URLs, etc.
Modal dialog Creating a new sub-entity "+ Add Index" on Markdown for LLMs → opens "Add Markdown Index" modal with Name / Content Tag / Template fields.

Modal pattern: - Centered ~480px width, white card, 16px radius, soft shadow. - Title row: bold title left, close X right. - Form: stacked label + input pairs. Required fields marked with red asterisk. - Footer: right-aligned "Cancel" (ghost) + primary action ("Add Index", in blue). - Page behind dimmed.

Implication for our redesign: Edit existing → expand inline. Create new → modal.

7. Destructive action pattern — typed confirmation

The Settings page has a "Delete Crawler" danger card. Anatomy:

┌──────────────────────────────────────────────────────────────────────┐
│ Delete Crawler                                                       │
│ This crawler and all associated data will be lost                    │
│                                                                      │
│ [Delete this crawler]   ← red filled button                          │
└──────────────────────────────────────────────────────────────────────┘

Click the red button → modal opens with typed-confirmation pattern:

Delete crawler                                                  [×]

Are you sure you want to delete this crawler? This action is
irreversible. We will delete your crawler, the configuration,
all pending operations, all logs.

We will not remove anything from your Algolia account. If you
want to delete the associated Algolia indices, you will need to
do it by hand afterward.

Enter "DELETE" to confirm.

[ ___________________________________________________ ]

[ Delete (red, disabled until "DELETE" is typed exactly) ]

Why this matters: This is the only destructive primitive in the entire crawler workspace. Note the explicit scope statement ("we will not remove…") — sets expectations about what's not deleted.

8. Empty / locked / gated states

Three distinct empty patterns observed:

Type Example Treatment
Empty data External Data ("Empty / There is no data to display"), Dismissed Suggestions ("No dismissed suggestions at this time"), Markdown Indexing ("No markdown indices created yet. Click 'Add Index' to create your first one.") Light table or card with centered grey small text. No illustration.
Locked / requires-prereq Logs Explorer ("You need to enable Crawler Logs first") Centered illustration (light blue shield), bold heading, two-line explanation, primary blue CTA ("Go to Configuration") + a "Learn more" link. The full content area becomes the empty state — no other UI competes.
Edit-mode empty Markdown Indexing edit-card after toggle on Centered grey "No markdown indices created yet" text + a "+ Add Index" outlined button just below the description, before the Save/Cancel row.

Implication for our redesign: every page needs to design at least the empty and locked states alongside the populated one.

9. List / table pattern

The crawler list (outer dashboard) and several inner pages use a consistent table:

  • Top toolbar: search input (left, with magnifying-glass icon and grey "Search…" placeholder) + filter dropdown (e.g. "All Status", "Status") + right-aligned primary action ("Add new crawler") and/or secondary actions ("Recrawl failed URLs", "Crawl a URL").
  • Header row: muted small caps column labels.
  • Body rows: 56–64px tall, hover background subtle, primary cell often a clickable blue link (e.g., crawler name, URL).
  • Status column: pill badge — Canceled (orange tint), Created (grey tint), Success (green tint), Ignored (orange/grey tint).
  • Author column: avatar circle (initials, color from name hash) + email.
  • Optional action column on the right: chevron > to drill in.
  • Pagination centered at bottom: prev arrow / page numbers (current is filled) / next arrow.

The Path Explorer table is the most data-rich variant: 4 KPI cards above (URLs / Records / Errors / Skipped) + a table with Path / URLs / Records / Errors / Skipped / Size / Bandwidth columns. Each row has a folder icon — the row is expandable into deeper paths. This is the model for our "branches not leaves" tree.

10. Status / report patterns

Page Pattern
Overview Top-aligned info banner about last crawl status. Grid of 5 cards: Monitoring, Suggestions, Last complete crawl (start/cancelled/duration), Schedule, Indices — each with a "View in …" link as the action.
Monitoring Toolbar (Recrawl failed URLs / Compare with previous crawl / Filter). Table with Status / Category / Reason / Count columns + chevron drill-in.
Suggestions List of cards. Each card has a title, then a Description row + Solution row (label-style 2-column), and right-side action triplet: Snooze, Dismiss, Docs (with external-link icon). Badge on tab header counts open.
Data Analysis (post-Analyze Index) Banner with index name + "Last analyzed less than a minute ago" + "Restart analysis" button. Then warnings tabs. Warnings grouped by attribute (e.g., "description / Affects 3 records out of 13 records (23.08%)") — each group is a sub-table with Warning / Records affected / Preview / Dismiss Warning columns. "View URLs" + "Dismiss" inline actions per row.
URL Inspector Search by URL + Status filter + "Recrawl failed URLs" + "Crawl a URL" actions. Table of URL link + status pill. Pagination.

11. The Editor — power-user dark surface

Editor (under Setup) is the only dark-themed surface in the dashboard. Anatomy:

  • Top sub-header: pencil icon + version label ("v17 - Current Draft") + "Saved 3 days ago" timestamp + "Show Sensitive Information" toggle (right).
  • Left pane: code editor with line numbers, syntax highlighting, scheme appId / apiKey / indexPrefix / rateLimit / startUrls / sitemaps / actions / recordExtractor. Inline next-crawl annotation appears between fields.
  • Right pane: secondary tabbed pane — URL Tester (active), Versioning, Errors (disabled when none).
  • Right pane's URL Tester has its own input field and a centered illustration + explanation when empty.
  • Right pane's Versioning shows a vertical timeline of versions: each entry is a card with version number + relative date + current draft pill, edit + revert icons on the right, author email below.
  • Bottom status bar (full-width, dark): CONFIGURATION IS VALID (green ✓ icon), separator, Helpers, Suggestions, Give Feedback links, far-right "Review and Publish" button (disabled until dirty).

This is reserved for advanced users. The card-based Configuration screens cover the same settings for normal users. Think of the Editor as Algolia's "code view" mode — equivalent to switching from form to YAML in other products.

12. Persistent CTA model

Top-right of the focused workspace always shows one primary action that reflects the next obvious step for this crawler:

State CTA
Crawler created but not started "Start Crawling" (filled blue)
Crawler has run before, on Overview page "Crawl now" (filled blue)
Editor with unsaved changes "Review and Publish" (in the dark status bar, disabled until dirty)

Next to it: small close X to exit the focused workspace.

Critical observation: there is no Pause / Stop / Restart / Cancel button anywhere in the crawler workspace. Crawls run from a button or on a schedule. Once started, they finish or get cancelled by some implicit timeout/limit (the Wellington overview shows status "Canceled" with start/cancelled timestamps and a duration of 6 minutes).

Implication for our redesign: Algolia's current mental model is fire-and-forget runs, not long-lived processes. We are intentionally going further. Pause / Resume / Re-crawl are our value-add — the explicit enhancement that justifies Crawler Factory existing as a layer on top of the base crawler. Operators of large multi-domain ingestion pipelines need fine-grained control; Algolia hasn't built that yet, we will.

See 01-design-language.md §2.5 for the full per-crawler control surface and how it extends Algolia's primitives without breaking their visual language.

13. Color, type, spacing — the tokens you need

The exact dashboard CSS isn't accessible here, but these are the values inferred from screenshots.

Color - Surface: #FFFFFF (cards), #F8F8FA (canvas). - Ink: ~#1F2230 (primary), ~#5C6075 (secondary/description), ~#9498A8 (muted/labels). - Border: ~#E6E7EE (1px hairline). - Accent blue (action / link / active): Algolia blue ~#003DFF. Used for buttons, links, active nav, active tab underline, focus ring. - Status colors — only inside pill badges: - Success: ~#15803D text on #DCFCE7 fill. - Warning / In-progress / Cancelled: ~#B45309 text on #FEF3C7 fill. - Neutral / Created / Ignored: ~#52525B text on #F1F5F9 fill. - Danger / Delete: red #DC2626 filled button. - No gradients anywhere. No dark mode (except Editor, which is its own surface).

Type stack — appears to be a humanist sans-serif (likely the SystemUI / Inter family). Sizes observed: - Page H1: 24px / 700. - Card title: 15px / 600. - Card description: 13px / 400 / muted. - Small caps section header (left nav): 11px / 700 / letter-spacing 0.08em / muted. - Body / table cell: 14px / 400. - Tab label: 14px / 600 (active in accent color).

Spacing — generous. Card internal padding ~28px. Row gap between cards ~16px. Input-to-input gap inside an edit card ~20px. Side margin in main content area ~32px from nav rail.

Radii — 12px on cards and buttons, 8px on inputs, 999px (full pill) on status badges.

Shadows — barely any. Cards rely on borders. The few elevated surfaces (modals) have a soft 0 8px 24px rgba(0,0,0,0.06)-class shadow.

14. What's NOT in the dashboard (so we shouldn't add it)

  • Toasts on save (just inline collapse)
  • Animated transitions beyond simple expand/collapse
  • Drag-to-reorder
  • Keyboard shortcuts UI (cmd+K is the global search, but no ? shortcut overlay observed)
  • Undo / redo
  • Onboarding tour overlays
  • A "history" of changes per setting (only the global Editor versioning timeline)
  • Real-time live counters (the dashboard is request-response; nothing pulses or animates)

This restraint is itself a design choice. Our redesign should match it.

15. Direct mapping to our Crawler Factory redesign

Algolia pattern Our screen Replaces our current
Outer dashboard list view Crawler Factory list of crawlers (entry point) We don't have one yet; only a wizard.
Focused workspace + 3-section nav Each crawler's detail view Replaces the linear Discover→Review→Configure→Crawl breadcrumb post-creation.
Persistent top-right CTA "Crawl now" / "Start Crawling" Replaces our "Confirm categories" / "Build & start crawlers" footer pattern.
Setting cards stack All Configuration sub-screens Replaces our 3-tab "Configure" panel inside each branch.
Inline expand to edit Every setting Replaces our Tab 2 field selector.
Modal for new entity Create new crawler / add new markdown index Replaces our 4-step wizard's "Build" stage.
Path Explorer table Live tree on Screen 2 (Review) — branches with metrics Replaces our gradient "Live content tree" cards.
Suggestions card list Per-crawler "Suggestions" page NEW capability we should adopt.
Data Analysis grouped warnings Per-crawler "Quality" page NEW capability we should adopt.
Settings danger zone Per-crawler Settings Replaces no equivalent today (we had no delete path).
Empty/locked illustrations All states Replaces our missing edge cases.
No pause/resume primitive (Algolia) Add pause / resume / re-crawl as our value-add — see §2.5 of the design pack Differentiates Crawler Factory; justifies our layer existing.