Decisions/2026-05-03-RunControlSurface.md
Decision — Run-Control Surface (pause / resume / re-crawl)
Decision
Crawler Factory will add Pause / Resume / Stop / Re-crawl as a runtime control surface on top of every crawler. This is intentionally not how Algolia's base dashboard works — and that is the point. It is our value-add.
Context
Algolia's base crawler dashboard treats crawls as fire-and-forget. A crawl runs from a button click or on a schedule. There is no Pause, no Resume, no granular Stop. The full runtime control surface is just three primitives:
- Schedule (when to run)
- Crawl now (run immediately)
- Delete (remove the asset)
This is fine for a single-domain customer with one crawler. It breaks down for the Algolia Central / Content Engagement use case, where a single deployment ingests dozens to hundreds of domains and the operator needs fine-grained control: pause one while debugging, restart one without redoing site discovery, stop the noisy one without affecting the others.
What we add (and why each)
| Action | What it does | Reversible? | Why we add it |
|---|---|---|---|
| Pause | Stops fetching mid-run; preserves resumable state. Partial index stays. | Yes (Resume continues from last URL) | Operator-grade control. Necessary when a noisy site or a downstream issue requires a pause without losing progress. |
| Resume | Continues a paused crawl from the last URL. | Yes | Pair to Pause. |
| Stop | Ends current run cleanly, finalizes whatever was indexed, moves asset to stopped state. |
No (next start is fresh) | Necessary when "I just want this run to end." Different from Pause (which preserves resumable state). |
| Re-crawl from scratch | Wipes index records, re-fetches all category URLs. Crawler config + schedule + index name unchanged. | No (records lost; rebuilt next crawl) | Necessary when the crawler config changed and we want a clean rebuild without redoing site discovery. |
What we DO NOT add (deliberately)
- Re-discovery from scratch. Discovery (sitemap walking + classification) ran once at creation. Re-discovery is a separate, opt-in action — not part of the runtime control surface. It would invalidate the existing per-domain blueprint.
- Crawler-level versioning. Algolia's Editor has versioning (
v17 - Current Draft). We do not duplicate that. Rely on Algolia's version control of the crawler config.
How this maps to UI
Specified in Design-Language/01-design-language.md §2.5. Two surfaces:
Workspace top-right — <RunControlCluster />
Stateful primary button + overflow menu. Label depends on current state.
| State | Primary | Overflow |
|---|---|---|
idle (never run, or last run finished) |
Crawl now | Edit configuration · Re-crawl from scratch · Settings |
running |
Pause | Stop crawl · View live log · Settings |
paused |
Resume | Stop crawl · Edit configuration · Re-crawl from scratch |
stopped (user stopped) |
Crawl now | Re-crawl from scratch · Edit configuration · Settings |
failed |
Retry | View error log · Edit configuration · Settings |
List row — <RunStatePill /> + 3-dot overflow
Per-row stateful pill for state visualization (Idle / Running (pulsing dot) / Paused / Stopped / Failed / Done) + 3-dot menu exposing Pause / Resume / Stop / Open / Settings / Delete. Bulk multi-select for Pause-all / Resume-all / Crawl-all (scoped to current page only).
Confirmation patterns
| Action | Confirmation |
|---|---|
| Pause / Resume | None (immediate, reversible) |
| Stop | None (immediate, but irreversible for that run) |
| Re-crawl from scratch | Lightweight scoped-confirm dialog (Recipe 24): explains what's wiped (records) and what's preserved (config, schedule, index name) |
| Delete | Typed-confirm modal (Recipe 10): user must type DELETE to enable the destructive button |
The strict separation between scoped-confirm (for recoverable destructive actions) and typed-confirm (for true irreversibility) is intentional. Reserving typed-confirm for only truly irreversible actions is what keeps it credible.
Rationale — why this is our value-add, not a divergence
Crawler Factory is one module in Algolia Central (see **Algolia-Central-Context**). Central's promise is long-running, multi-domain, public-content ingestion at scale. The base Algolia crawler doesn't address that operator persona.
Our control surface (Pause / Resume / Stop / Re-crawl) is what makes Crawler Factory a layer on top of the base crawler, not a replacement for it.
Visually, the cluster matches Algolia's design language exactly (same button shapes, same blue accent, same overflow menu style). Functionally it goes further. This is the right shape: visual continuity, functional differentiation.
Open questions
| Question | Status |
|---|---|
| What backend implementation supports Pause/Resume? Does Algolia Crawler API expose a pause/resume primitive, or do we need to track state ourselves? | Not yet investigated. Spike needed in Session 3. |
| When a crawler is paused for >24h, do we auto-resume on a schedule? Or stay paused until manual Resume? | TBD. Default to "stay paused" until evidence says otherwise. |
| Bulk operations across multiple crawlers — how is this rate-limited at the Algolia API level? | Not yet investigated. |
Source materials
- 32 dashboard screenshots:
**00-findings**§12 - Design pack §2.5:
**01-design-language** - Implementation recipes:
**02-component-recipes**Recipes 22, 23, 24 - Conversation context:
**Session-2-DesignLanguage**