Decision — Embedded Architecture

Decision

Crawler Factory will be embedded inside the Algolia dashboard, not shipped as a standalone application.

The Crawler Factory experience lives at Data sources → Crawler in the Algolia dashboard. It is not a separate product surface, not a separate URL, not a separate auth context. The user reaches it via the same nav rail they already use to reach Indices, Events, Connectors, and Ask AI today.

This decision was committed by Arijit on 2026-05-03 during Session 2.

Why this decision matters

It reverses the standalone-app posture we were carrying. The earlier mental model treated Crawler Factory as its own product surface — its own outer chrome, its own URL (/admin/factory), its own brand mark. The user pasted screenshots of the live Algolia dashboard, we reverse-engineered its design language, and once the language was on the table the implication was unavoidable: anything that sits next to Indices and Connectors in the Algolia dashboard must look like it was authored by the Algolia team.

This is not a small reskin. It changes:

1. Chrome — Crawler Factory inherits the Algolia outer dashboard chrome (left product rail, Data sources sub-nav, top bar with app picker, search, Upgrade, AI Assist, avatar). It does not author its own.
2. Entry point — Users do not navigate to /admin/factory. They click Data sources → Crawler like they already do for the existing Crawler page, and Crawler Factory is what they see.
3. Navigation — The two-chrome rule still applies inside Crawler Factory (list view → focused workspace), but the outermost chrome is Algolia's.
4. Branding — The wordmark in the focused-workspace top-left says CRAWLER (matching Algolia's existing crawler workspace), not CRAWLER FACTORY. Our differentiation is functional (the value-add controls), not visual. This supersedes the open question in **2026-05-03-TwoChromeRule**.
5. Screen inventory — We need a list view of crawlers (which we didn't have in the standalone wizard). See **2026-05-03-Phase1ScreenInventory**.

How Crawler Factory replaces / extends Algolia's existing Crawler page

Algolia today has a Data sources → Crawler page (we have 32 screenshots of it). Today's flow:

1. User lands on the Crawler list page.
2. Clicks Add new crawler.
3. Fills in a name, picks an Algolia App, gets an empty crawler with a placeholder config.
4. Goes to the Editor (advanced) and writes JavaScript by hand to define indices and extractors. Or fills out the cards in Configuration laboriously.
5. Clicks Start Crawling and waits.

The pain: steps 3–4 are completely hand-built. Every crawler is artisanal. There's no concept of "auto-discover the site, classify the content, and stamp out N crawlers per content domain."

Crawler Factory replaces step 3 (and most of step 4) with a guided flow:

1. User lands on the Crawler list page (same as today).
2. Clicks Add new crawler — but the modal that opens is now a choice: Build manually (advanced) or Use Crawler Factory (recommended).
3. If they pick Factory, they enter the 4-step wizard (Discover → Review → Configure → Crawl).
4. At the end of the wizard, N crawlers have been created (one per detected content domain), each pre-configured with a working extractor. These crawlers are real Algolia crawlers — visible in the same list, with the same config / editor / status / suggestions / logs sub-pages.
5. The user manages them ongoing using Algolia's existing crawler workspace UI, augmented with our value-add Run-Control Surface (**2026-05-03-RunControlSurface**).

In other words: Crawler Factory is the smart creation flow + the operator-grade control layer on top of crawlers Algolia already supports. The crawlers themselves are not a new entity; we just produce them in bulk and manage them with more granular controls.

The placement question, answered

The phased roadmap

Phase 1 (current build, this session and next): Web crawler only. - Embedded inside Data sources → Crawler in Algolia dashboard. - 4-screen wizard for creation (Discover → Review → Configure → Crawl). - Crawler list view (existing Algolia view, with our additions). - Per-crawler focused workspace (existing Algolia workspace, with our Run-Control Cluster).

Phase 2 (future): Additional Layer-1 connectors. - CMS connector (Contentful, WordPress, Drupal, headless CMS APIs). - KB connector (Zendesk, Salesforce KB, Confluence). - Video transcript connector (YouTube, Vimeo, Loom). - Code repository connector (GitHub, GitLab). - Each follows the same pattern: a smart creation wizard + per-asset focused workspace + run-control surface. - Most likely as sibling tabs inside a unified "Connectors" page.

Phase 3 (further future): The other Layer-1 stages, then Layers 2–3. - Transformer (UI for the homogenization stage). - Enrichment (UI for Algolia Fetch enrichment rules). - Indexing (consume the existing Indices page). - Agent Studio (already exists in some form; integration target). - These likely become top-level entries in the global product rail alongside Data sources, since they're full pipeline stages, not sub-modules.

Embedded vs. standalone — what we lose and what we gain

The trade is correct. Our value is not the chrome.

What this means for the existing repo

The current repo (/Users/arijitchowdhury/AI-Development/Crawler_Factory) was built as a standalone Vite app with its own /admin/factory route and its own chrome. Most of the React UI in src/components/admin/factory/ will need to be either rewritten or refactored to live inside the Algolia dashboard's React shell when we eventually integrate.

Until that integration window opens, the standalone repo continues to function as our prototype + reference implementation — running locally, demoing the wizard end-to-end, but rendering the chrome we will eventually drop. This is fine. The mockups, the React components, and the design pack all transfer; only the outer shell wrapper needs replacement.

What this changes about the Phase 1 build spec

See **2026-05-03-Phase1ScreenInventory** for the full screen list. Highlights:

1. We are no longer designing an outer dashboard chrome — assume Algolia's outer chrome wraps everything.
2. We need a crawler list view that we never had before.
3. The wizard becomes a modal flow over the list view instead of a 4-step page route.
4. Each crawler's focused workspace must extend Algolia's existing workspace IA (don't replace it).
5. The Run-Control Cluster slots into Algolia's existing "Start Crawling" CTA position. Same place, more behavior.
6. Empty / error / loading states must follow Algolia's patterns (Recipe 13 in component recipes), not invent new ones.

Open questions

Source materials

• 32 dashboard screenshots: **00-findings**
• Design pack: **01-design-language**
• Run-control surface: **2026-05-03-RunControlSurface**
• Two-chrome rule (revised by this decision): **2026-05-03-TwoChromeRule**
• Phase 1 screen inventory (consequence of this decision): **2026-05-03-Phase1ScreenInventory**
• Conversation: **Session-2-DesignLanguage**