Dashboard/Crawler-Factory
Project

Crawler-Factory

Vault: myos   Path: Crawler-Factory

README.md

Crawler Factory

A guided admin tool that takes a website URL and produces N production Algolia crawlers (one per detected content domain), each writing to a purpose-built index that becomes the data foundation for a future specialist agent.

Crawler Factory is the first connector / Layer 1 ingestion module of Algolia Central / Content Engagement. Read Algolia-Central-Context before making product decisions here.

Status: Session 4 (2026-05-03) — factory entity model + wizard-in-workspace shipped and browser-verified. See Status for live state. Repo: github.com/arijitchowdhury80/Crawler_Factory (private, standalone, main branch) Models: Sonnet 4.6 (build); Opus 4.7 1M context (planning + design).

Where this fits in Algolia Central

Algolia Central / Content Engagement
├─ Layer 3 — Agent Studio
├─ Layer 2 — AI Retrieval (NeuralSearch)
└─ Layer 1 — Data Layer
   ├─ Connectors  ← Crawler Factory (THIS PROJECT) — first connector is web; future: CMS, KB, video, code
   ├─ Transformer
   ├─ Enrichment
   ├─ Normalization
   └─ Indexing

See Algolia-Central-Context for the full vision and how this project plugs in.

Why this matters

Today every Algolia crawler is hand-built in the dashboard. We have one crawler writing to one index; Maverick/Elena/Bruno all share it. The factory is the path to:

  1. Programmatically scaffold N crawlers per site — based on detected content domains (marketing, support, education, technical, customer-stories, etc.)
  2. Per-domain indices with config tuned for that domain (different searchableAttributes, customRanking, facets)
  3. Forward-compatible with the hub-and-spoke agent architecture — the factory writes blueprints that a future specialist-agent factory will consume to scaffold purpose-built agents
Document Purpose
00-Plan The full implementation plan (frozen 2026-04-30, ~2000 lines)
01-Methodology The operating protocol (extracted from §19)
02-DSS Domain Schema Standard reference (extracted from §4)
Core/03-SiteTypes/ Per-CMS / per-site-type playbooks
04-Web-Reality-Catalog Empirical findings from Web Almanac 2024 + 60-site audit
05-Architecture-Hub-Spoke Future agent architecture vision
06-Multi-Tenant Multi-tenant orchestration for multi-brand corporates
07-Update-Audit-Trail What the validation research changed in the plan, and where
Algolia-Central-Context How Crawler Factory fits into Algolia Central (read first when joining)
Research/ Raw research findings from each validation round
Engineering-Specs/ 13 cluster specs — generated from 00-Plan.md after SOPs applied
Sessions/ Session logs (S1–S4) — durable record of what each session accomplished
Decisions/ Locked product decisions, one ADR per file
Design-Language/ Design language pack v1.1 — tokens, components, recipes
Reference/ Source materials — context docs, transcripts, memos
Status Live state — planned, in progress, shipped

Decisions locked (summary)

# Decision Choice
1 Persistence Algolia meta-index algoliacentral_factory_sessions (sharded)
2 UI placement Embedded at Data sources → Crawler (not a separate route)
3 Index strategy One index per content domain
4 Test crawl Sandbox first, then crawl_urls real test
5 LLM Reuse global lib/llm/ (MiniMax via Algolia inference)
6 recordExtractor source Canonical at runtime; version-controlled at crawler-configs/{site}/
7 Content classification CMS-fingerprint-first cascade (NOT JSON-LD-first)
8 URL/depth caps None. Streaming walker, sharded persistence.
9 Architectural posture Hub-and-spoke agent future; every crawler emits blueprint with agent_slot
10–18 UX decisions See **** for all ADRs

Validation coverage (60 sites, 14 verticals)

See 04-Web-Reality-Catalog for the full empirical record with citations.

Enterprise B2B, CMS/generalist, government, multi-brand corporates, massive scale (Microsoft 1M–10M URLs), SaaS B2B, education, healthcare, news, community, non-profit, manufacturing, e-commerce retail, DTC retail.

All files (61)