Crawler-Factory

Wiki/Architecture.md

Architecture — Crawler Factory

Last updated: 2026-05-03 (S4) Source: Engineering-Specs/00-Index, 00-Plan

Component map

Browser (React)
  └─ AdminFactory.tsx
       ├─ CrawlerListView         ← list of factories; "Add new crawler" entry
       └─ WorkspaceShell          ← per-factory workspace (outer Algolia chrome stays)
            ├─ left nav: SETUP / STATUS / CONFIGURATION
            ├─ Setup page          ← wizard lives here (4 steps inline)
            ├─ Overview page
            ├─ Configurator page
            └─ Live Monitor page

API Layer (Next.js API routes under /api/factory/)
  ├─ discover         ← streams URL walk + domain detection
  ├─ sample           ← samples HTML per domain
  ├─ generate-extractor ← produces recordExtractor JS per domain
  ├─ test-sandbox     ← runs extractor in sandbox (new Function)
  ├─ test-real        ← runs real crawl_urls against Algolia Crawler API
  ├─ create-crawler   ← creates N crawlers via Algolia Crawler API
  ├─ crawl-progress   ← SSE stream of live progress
  ├─ session          ← reads/writes session state from meta-index
  └─ blueprints       ← returns completed blueprints

Backend Modules
  ├─ sitemap.ts           ← streaming URL walker (no caps)
  ├─ path-grouper.ts      ← URL → path-group clusters
  ├─ waf-detector.ts      ← detects Cloudflare / Akamai / Imperva
  ├─ playwright-fetcher.ts← stealth fetch for WAF-protected pages
  ├─ sampler.ts           ← sample HTML per domain
  ├─ cms-detector.ts      ← CMS fingerprinting (WordPress, Drupal, etc.)
  ├─ classifier.ts        ← domain classification cascade
  ├─ structure-analyzer.ts← analyzes HTML structure for extraction hints
  ├─ extractor-generator.ts← generates recordExtractor per domain (LLM)
  ├─ sandbox-runner.ts    ← new Function sandbox for extractor testing
  ├─ crawler-client.ts    ← typed Algolia Crawler REST API client
  ├─ index-manager.ts     ← creates per-domain Algolia indices
  └─ brand-domain-discoverer.ts ← detects brand sub-domains

Storage
  ├─ Algolia meta-index `algoliacentral_factory_sessions` ← session state (sharded)
  └─ crawler-configs/{site}/{domain}-{group}.js ← version-controlled recordExtractors

Data flow

User pastes URL
  → sitemap.ts: streaming walk → path groups
  → waf-detector.ts: detect WAF → playwright-fetcher if blocked
  → sampler.ts: sample HTML per domain
  → cms-detector.ts + classifier.ts: fingerprint → classify domains
  → (User reviews domains at Gate 2)
  → structure-analyzer.ts: per-domain structure analysis
  → extractor-generator.ts: LLM → recordExtractor JS per domain
  → sandbox-runner.ts: new Function test → validate extraction
  → crawler-client.ts: create N crawlers via Algolia API
  → index-manager.ts: create N purpose-built indices
  → Session state → algoliacentral_factory_sessions (resumable)
  → Blueprint emitted per factory (includes agent_slot field)

13-cluster build map

#	Cluster	Key modules	Status
01	Foundations	DSS, types, session-store, blueprint-store	✅
02	Sitemap discovery	sitemap.ts, path-grouper.ts, scope-estimator.ts	✅
03	WAF + sampling	waf-detector.ts, playwright-fetcher.ts, sampler.ts	✅
04	CMS + classification	cms-detector.ts, classifier.ts, site-type-rubric.ts	✅
05	Extraction + sandbox	structure-analyzer.ts, extractor-generator.ts, sandbox-runner.ts	✅
06	Crawler client + index manager	crawler-client.ts, index-manager.ts	✅
07	Brand domain discovery	brand-domain-discoverer.ts	✅
08	Backend API endpoints	9 API routes	✅
09	Frontend foundations	App.tsx, AdminFactory.tsx, hooks/factory/*	✅
10	Frontend discovery flow	RollingLog, DomainBadge, DiscoveryPanel, CategoryReview	✅
11	Frontend configurator	CrawlerListView (factory entity), WorkspaceShell (factory mode + Setup page), FactoryWizard (inline), AdminFactory (persistent chrome)	✅ S4
12	Frontend live monitor + orchestrator	LiveCrawlMonitor, sub-crawler management view, backend wiring	⚠️ Partial — sub-crawler nav (UI-003) + backend wiring (UI-004) pending S5
13	Bundle + smoke + docs	Final verification	⚠️ Pending S5 backend wiring

Integration points

External system	How we connect	Purpose
Algolia Crawler API	REST (crawler-client.ts)	Create crawlers, test with crawl_urls, monitor progress
Algolia Search API	JavaScript client (index-manager.ts)	Create per-domain indices
Algolia meta-index	Algolia Search API	Session state persistence (sharded)
Playwright	npm package	Stealth fetch for WAF-protected pages
LLM (MiniMax via lib/llm/)	Global lib/llm/	recordExtractor generation

Known risks / fragile points

Vite SSE buffering: URLs arrive all at once in dev; real-time only on Vercel (KI-003)
No auth on factory endpoints: API endpoints unprotected (KI-004)
LLM dependency for extractor generation: MiniMax via global lib/llm/ — if LLM changes, extractors change
Algolia Crawler API limits: Domain verification (OQ-1) and Pause/Resume API support (OQ-4) are open questions