Crawler-Factory
Wiki/Decisions.md
Decisions — Crawler Factory
Source: Status.md, Decisions/ Last updated: 2026-05-03
One row per locked decision. For decisions with a dedicated ADR file, link is in the Source column.
| # | Decision | Why | Rejected alternatives | Source |
|---|---|---|---|---|
| D1 | Persistence via Algolia meta-index algoliacentral_factory_sessions (sharded) |
Keeps all data in Algolia ecosystem; no separate DB to manage | Postgres, Redis, localStorage | S1 / 00-Plan.md |
| D2 | UI placement: /admin/factory sub-route |
Embedded in existing Algolia Central admin shell; no separate app | Standalone app, new window | S1 / 00-Plan.md |
| D3 | One index per content domain (marketing, support, education, etc.) | Per-domain indices enable per-domain customRanking, facets, searchableAttributes — impossible in a single mixed index | One mega-index, one index per site | S1 / 00-Plan.md |
| D4 | Sandbox-first test (new Function), then crawl_urls real test as final confirmation |
Fail fast without consuming Algolia API credits; sandbox catches obvious errors | Test directly against real API; manual review only | S1 / 00-Plan.md |
| D5 | Reuse global lib/llm/ (MiniMax via Algolia inference) |
Already in-repo, already configured; no new dependency | Direct Anthropic/OpenAI call, custom LLM wrapper | S1 / 00-Plan.md |
| D6 | recordExtractor canonical at runtime in Algolia Crawler dashboard; version-controlled copy at crawler-configs/{site}/{domain}-{group}.js |
Algolia Crawler API is the source of truth at runtime; version control for auditability | Vault-only, dashboard-only | S1 / 00-Plan.md |
| D7 | CMS-fingerprint-first cascade (NOT JSON-LD-first) | Web Almanac 2024: JSON-LD structured data only 41% global coverage, ~7% with useful @types; CMS fingerprinting is more reliable | JSON-LD first, LLM first | S1 / 04-Web-Reality-Catalog.md |
| D8 | No URL/depth caps; streaming walker; sharded persistence | Must handle sites from 5K to 10M+ URLs (Microsoft, AWS scale); depth caps break deep-nested edu sites | Max 10K URLs, max depth 3 | S1 / 00-Plan.md |
| D9 | Architectural posture: hub-and-spoke agent future; every crawler emits a blueprint with agent_slot |
Crawler Factory is Layer 1 of Algolia Central; blueprints are the API contract between ingestion and future agent factory | Generic crawlers without agent metadata | S1 / 00-Plan.md |
| D10 | Match Algolia's existing dashboard design language; no parallel design system | Crawler Factory will eventually merge into the Algolia dashboard; divergent design = rework | Custom design system, Material UI | S2 / Design-Language/01-design-language.md |
| D11 | Two-chrome rule: outer dashboard chrome vs. focused workspace chrome | Consistent with Algolia's own crawler workspace pattern; prevents context loss during long wizard flows | Single chrome, full-screen overlay | S2 / Decisions/2026-05-03-TwoChromeRule.md |
| D12 | Pause/Resume/Stop/Re-crawl = our value-add over Algolia's fire-and-forget model | Core product differentiator; Algolia dashboard has no long-run control primitives | Mirror Algolia exactly, no extensions | S2 / Decisions/2026-05-03-RunControlSurface.md |
| D13 | Live tree shows branches only, never URL leaves | Rendering 10M individual URLs would freeze the browser; branch-level gives sufficient signal | Full URL tree, count-only | S2 / Design-Language §3 |
| D14 | ALPHA badge stays in top bar | Honest build/version signaling; prevents confusion about production readiness | No badge, version in footer | S2 |
| D15 | Re-crawl uses scoped confirm; typed-confirm (type the name) reserved for Delete only | Typed-confirm creates friction proportional to irreversibility; re-crawl is reversible, delete is not | Typed-confirm for re-crawl, no confirm for delete | S2 |
| D16 | Factory is a container — sub-crawlers NOT in main CrawlerListView; only inside factory workspace | A factory with 9 sub-crawlers would produce 9 rows in the list, obscuring the factory-level view | Show all crawlers flat in the list | S3 / Decisions/2026-05-03-FactoryEntityModel.md |
| D17 | Wizard lives as "Setup" page inside WorkspaceShell — not a full-screen overlay | Algolia's own flow opens the crawler workspace immediately on creation; our wizard should follow the same pattern | Full-screen overlay, separate route | S3 / Decisions/2026-05-03-WizardInWorkspace.md |
| D18 | Factory creation flow: Add → factory entity created → workspace opens → Setup page runs wizard | Derived from observing real Algolia dashboard flow (Images 3-7, Session 3) | Modal wizard before workspace opens | S3 / Decisions/2026-05-03-WizardInWorkspace.md |
| D45 | Test extractor is the engine before field configuration, not a sidebar | Fields shown before extraction are DSS hardcoded defaults (wrong). Real extraction output → user curates → then names index. Confirmed S24. | Third tab (passive verify), fields from infer-schema SSE | S24 |
| D46 | Extract, Fields, and Launch are separate wizard screens (not sections in one screen) | Extraction is a prerequisite for informed field selection; field selection is a prerequisite for a meaningful launch summary. Sequential screens enforce this contract and give each step its own focus. | Single Configure screen with sequential panels | S25 |
| D47 | CrawlerClient uses { id } → { crawlerId } transform in zod schema |
Algolia Crawler REST API returns { id } on POST /crawlers; all callers expect { crawlerId }. Transform at the client boundary keeps callers clean. |
Rename callers to use id; accept both keys |
S25 |
| D48 | recordExtractor.__type must be 'function' (not 'custom') |
Algolia Crawler API schema validation rejects 'custom'; 'function' is the documented value. Browser error confirmed rejection. |
Keep 'custom' |
S25 |