README.md
Crawler Factory
A guided admin tool that takes a website URL and produces N production Algolia crawlers (one per detected content domain), each writing to a purpose-built index that becomes the data foundation for a future specialist agent.
Crawler Factory is the first connector / Layer 1 ingestion module of Algolia Central / Content Engagement. Read Algolia-Central-Context before making product decisions here.
Status: Session 4 (2026-05-03) — factory entity model + wizard-in-workspace shipped and browser-verified. See Status for live state.
Repo: github.com/arijitchowdhury80/Crawler_Factory (private, standalone, main branch)
Models: Sonnet 4.6 (build); Opus 4.7 1M context (planning + design).
Where this fits in Algolia Central
Algolia Central / Content Engagement
├─ Layer 3 — Agent Studio
├─ Layer 2 — AI Retrieval (NeuralSearch)
└─ Layer 1 — Data Layer
├─ Connectors ← Crawler Factory (THIS PROJECT) — first connector is web; future: CMS, KB, video, code
├─ Transformer
├─ Enrichment
├─ Normalization
└─ Indexing
See Algolia-Central-Context for the full vision and how this project plugs in.
Why this matters
Today every Algolia crawler is hand-built in the dashboard. We have one crawler writing to one index; Maverick/Elena/Bruno all share it. The factory is the path to:
- Programmatically scaffold N crawlers per site — based on detected content domains (marketing, support, education, technical, customer-stories, etc.)
- Per-domain indices with config tuned for that domain (different searchableAttributes, customRanking, facets)
- Forward-compatible with the hub-and-spoke agent architecture — the factory writes blueprints that a future specialist-agent factory will consume to scaffold purpose-built agents
Quick links
| Document | Purpose |
|---|---|
| 00-Plan | The full implementation plan (frozen 2026-04-30, ~2000 lines) |
| 01-Methodology | The operating protocol (extracted from §19) |
| 02-DSS | Domain Schema Standard reference (extracted from §4) |
| Core/03-SiteTypes/ | Per-CMS / per-site-type playbooks |
| 04-Web-Reality-Catalog | Empirical findings from Web Almanac 2024 + 60-site audit |
| 05-Architecture-Hub-Spoke | Future agent architecture vision |
| 06-Multi-Tenant | Multi-tenant orchestration for multi-brand corporates |
| 07-Update-Audit-Trail | What the validation research changed in the plan, and where |
| Algolia-Central-Context | How Crawler Factory fits into Algolia Central (read first when joining) |
| Research/ | Raw research findings from each validation round |
| Engineering-Specs/ | 13 cluster specs — generated from 00-Plan.md after SOPs applied |
| Sessions/ | Session logs (S1–S4) — durable record of what each session accomplished |
| Decisions/ | Locked product decisions, one ADR per file |
| Design-Language/ | Design language pack v1.1 — tokens, components, recipes |
| Reference/ | Source materials — context docs, transcripts, memos |
| Status | Live state — planned, in progress, shipped |
Decisions locked (summary)
| # | Decision | Choice |
|---|---|---|
| 1 | Persistence | Algolia meta-index algoliacentral_factory_sessions (sharded) |
| 2 | UI placement | Embedded at Data sources → Crawler (not a separate route) |
| 3 | Index strategy | One index per content domain |
| 4 | Test crawl | Sandbox first, then crawl_urls real test |
| 5 | LLM | Reuse global lib/llm/ (MiniMax via Algolia inference) |
| 6 | recordExtractor source | Canonical at runtime; version-controlled at crawler-configs/{site}/ |
| 7 | Content classification | CMS-fingerprint-first cascade (NOT JSON-LD-first) |
| 8 | URL/depth caps | None. Streaming walker, sharded persistence. |
| 9 | Architectural posture | Hub-and-spoke agent future; every crawler emits blueprint with agent_slot |
| 10–18 | UX decisions | See **** for all ADRs |
Validation coverage (60 sites, 14 verticals)
See 04-Web-Reality-Catalog for the full empirical record with citations.
Enterprise B2B, CMS/generalist, government, multi-brand corporates, massive scale (Microsoft 1M–10M URLs), SaaS B2B, education, healthcare, news, community, non-profit, manufacturing, e-commerce retail, DTC retail.