Dashboard/Crawler-Factory/Wiki/README.md
Crawler-Factory

Wiki/README.md

Crawler Factory

A guided admin tool that takes a website URL and produces N production Algolia crawlers — one per detected content domain — each writing to a purpose-built index that becomes the data foundation for a future specialist agent.

Status: In build — Session 4 pending (UI entity model + wizard-in-workspace) Last updated: 2026-05-03 Parent platform: Algolia Central

What it does

Today, every Algolia crawler is hand-built in the dashboard. Crawler Factory automates this: paste in a URL, and the factory discovers the site's content domains (marketing, support, education, technical, customer-stories, etc.), samples each domain, generates per-domain recordExtractor configurations, and creates N production crawlers — each writing to a purpose-built, agent-ready Algolia index.

It is the first connector (Layer 1) in the Algolia Central / Content Engagement platform. Every crawler the factory creates emits a blueprint with an agent_slot field — forward-compatible with a future specialist-agent factory that will scaffold agents on top.

Key use cases

  1. Onboard a new site in minutes: User pastes https://www.algolia.com, factory discovers 9 content domains, creates 9 crawlers with tuned configs — no dashboard hand-editing.
  2. Multi-domain enterprise ingestion: Factory handles sites from 5K to 10M+ URLs with no depth caps; streaming walker + sharded persistence prevents timeouts.
  3. Control plane for long-running crawls: Pause / Resume / Re-crawl / Stop are first-class primitives — value-add over Algolia's fire-and-forget dashboard model.

What makes it different

Algolia's native dashboard treats crawl creation as a one-shot form. Crawler Factory is a control plane: it classifies domains before creating crawlers, sandboxes the recordExtractor before going live, and maintains resumable session state so a 10M-URL site doesn't have to start over.

Where it fits

Algolia Central / Content Engagement
├─ Layer 3 — Agent Studio
├─ Layer 2 — AI Retrieval (NeuralSearch)
└─ Layer 1 — Data Layer
   ├─ Connectors  ← Crawler Factory (THIS PROJECT)
   ├─ Transformer
   ├─ Enrichment
   ├─ Normalization
   └─ Indexing
Document Purpose
UserFlows.md What users can do, step by step
Architecture.md Components, modules, data flows
Decisions.md All locked decisions (D1–D18)
Brand.md Design language and visual identity
Constraints.md Non-goals and explicit limits
Status.md What's built, what's next
../Engineering-Specs/00-Index.md 13-cluster engineering spec index
../Design-Language/01-design-language.md Full design language pack
../Status.md Live project status (authoritative)

Repository

  • Local: /Users/arijitchowdhury/AI-Development/Crawler_Factory
  • GitHub: github.com/arijitchowdhury80/Crawler_Factory (private)