Dashboard/Crawler-Factory/Decisions/2026-05-03-FactoryEntityModel.md
Crawler-Factory

Decisions/2026-05-03-FactoryEntityModel.md

Decision: Factory Entity Model — Sub-Crawlers Live Inside Factory

Date: 2026-05-03 (Session 3) Status: Locked

Decision

Factory-created crawlers are grouped under one parent factory entity. They are NOT accessible from the main crawler list as standalone entries. All management of factory sub-crawlers happens through the factory workspace.

Context

During Session 3, the user reviewed the real Algolia crawler dashboard (5 screenshots). The existing "Add new crawler" flow creates one crawler per modal invocation, and each crawler appears as an independent row in the main list.

Crawler Factory creates MULTIPLE crawlers from a single domain (one per content type: docs, blog, community, products). If these appeared as individual rows in the same list as manual crawlers, the list would become confusing — users couldn't distinguish which crawlers are related, which belong to a factory, which can be modified independently.

The Model

Main crawler list: - One row per factory (e.g. algolia_com) — source badge = "Factory", shows sub-crawler count + aggregate run state - Individual rows for manual crawlers — source badge = "Manual" - Factory row opens the factory workspace on click

Factory workspace: - Left nav shows factory-level navigation including a "Setup" page - Sub-crawlers (docs, blog, community, products) are listed and managed INSIDE this workspace - Run/pause/stop/configure operations for sub-crawlers all go through factory workspace - Sub-crawlers do NOT appear in the main list — ever

Consequences

  • Cleaner main list: factories and manual crawlers are visually distinct
  • Factory context is preserved: all sub-crawlers for algolia.com are co-located
  • Operator cannot accidentally manage a factory sub-crawler as if it were independent
  • Wizard Done step returns one factory entity name, not N individual crawler names
  • D19: Wizard lives as "Setup" page inside WorkspaceShell (not a full-screen overlay)
  • D20: Factory creation flow: Add → factory entity created → workspace opens → Setup page runs wizard