Crawler-Factory-Pointer.md
Crawler Factory — Pointer
What this file is: the bidirectional anchor pointing from
**Algolia-Central**to**Crawler-Factory**. Crawler Factory is one of N modules in Algolia Central. From here you can jump to its docs; from inside Crawler Factory,**Algolia-Central-Context**jumps back to this side.
What Crawler Factory is
The first connector in Algolia Central's Layer 1 (Data Layer). Takes a website URL and produces N production Algolia crawlers — one per detected content domain — each writing to a purpose-built index.
Status of work: see [Status](/p/algolia-central/../Crawler-Factory/Status.md).
Where it sits in the stack
Layer 3 — Agent Studio
Layer 2 — AI Retrieval (NeuralSearch)
Layer 1 — Data Layer
├─ Connectors ← Crawler Factory (web connector; first of many)
├─ Transformer
├─ Enrichment
├─ Normalization
└─ Indexing
The web connector is shipping first. Future connectors (CMS, KB, video transcripts, code repos) follow the same pattern and will likely be authored by replicating the Crawler Factory architecture.
Quick-jump to Crawler Factory docs
| Doc | Purpose |
|---|---|
| README | Project overview |
| Algolia-Central-Context | How Crawler Factory fits into Algolia Central (this file's mirror, deeper) |
| Status | Live state |
| Session-2-DesignLanguage | Most recent session log (UX pivot, design language pack) |
| 2026-05-03-RunControlSurface | Pause/Resume/Re-crawl as our value-add |
| 2026-05-03-TwoChromeRule | UI architecture rule |
| 01-design-language | Portable design language pack v1.1 |
| Content-Engagement-Memo | Verbatim Google Doc transcript (Content Engagement memo) |
| Engineering plan | The implementation plan (~2000 lines) |
Decisions in Crawler Factory that affect Algolia Central as a whole
| Decision | Why it matters to Central |
|---|---|
| Match Algolia's existing dashboard design language | When future Central modules (Transformer UI, Enrichment UI, etc.) are built, they should reuse the same design pack. Cross-module visual coherence comes free. |
| Two-chrome rule (outer dashboard vs focused workspace) | Establishes the IA pattern for all future Central modules. Each module gets its own focused workspace with its own wordmark. |
| Pause/Resume/Stop/Re-crawl as our value-add | Sets a precedent: Central modules add operator-grade controls on top of the base Algolia primitives. This is what makes Central a layer, not a replacement. |
| Per-content-domain indices, blueprint registry | Becomes the contract between Layer 1 and Layers 2-3. The transformer / enrichment / agent layers consume Crawler Factory's blueprints. |
Open questions that span Algolia Central and Crawler Factory
| ID | Question | Where tracked |
|---|---|---|
| OQ-3 | When does Crawler Factory's UI merge into the main Algolia dashboard? | Status |
| OQ-4 | Backend support for Pause/Resume — does Algolia Crawler API expose primitives? | 2026-05-03-RunControlSurface |
| Q-AS4 | How is content sliced into N indices — by content type, vertical, source? | Brief Part 5 |
When this pointer is wrong
If Crawler Factory's role in Algolia Central changes (becomes obsolete, gets renamed, gets absorbed), update this pointer and **Algolia-Central-Context**. Do not let them drift.