Crawler Factory — Live Status
As of: 2026-05-03 (Session 4 closed)
Parent project: Algolia Central (this is one of N modules in the Content Engagement platform)
Current phase: Factory entity model shipped — backend wiring next
Session 4 (2026-05-03) shipped UI-001 (factory entity model in list) and UI-002 (wizard embedded as workspace Setup page). The two-chrome rule is now correctly enforced — outer Algolia chrome stays visible throughout wizard flow. Full browser-tested end-to-end. TypeScript clean.
Session 5 = sub-crawler management in factory workspace (UI-003) + wire backend discovery API to the wizard Step 2 animation.
Sessions
| # |
Date |
Theme |
Output |
Log |
| 1 |
2026-04-30 |
v1 functional spike |
13 specs, 897 tests, end-to-end demo on algolia.com |
Session-1-Process-Documentation, Session-1-Lessons-Learned |
| 2 |
2026-05-03 |
UX pivot + design language |
4-screen clickable mockup, 32 dashboard screenshots, 5-file design language pack, 2 locked decisions |
Session-2-DesignLanguage |
| 3 |
2026-05-03 |
React UI refactor + wizard |
Algolia design tokens, CrawlerListView, WorkspaceShell, FactoryWizard (4-step), GitHub repo, decisions D16-D18 |
Session-3-ReactUIRefactor |
| 4 |
2026-05-03 |
Factory entity model + wizard-in-workspace |
UI-001 + UI-002 shipped. Factory row in list, wizard inline in workspace. Browser-verified. |
Session-4-FactoryEntityModel |
Decisions locked (running list)
| # |
Decision |
When |
File |
| 1 |
Persistence via Algolia meta-index algoliacentral_factory_sessions (sharded) |
S1 |
00-Plan.md |
| 2 |
UI placement: embedded at Data sources → Crawler |
S1→S2 |
2026-05-03-EmbeddedArchitecture |
| 3 |
One index per content domain |
S1 |
00-Plan.md |
| 4 |
Sandbox-first test, then crawl_urls real test |
S1 |
00-Plan.md |
| 5 |
Reuse global lib/llm/ (MiniMax) |
S1 |
00-Plan.md |
| 6 |
recordExtractor canonical at runtime; version-controlled at crawler-configs/{site}/ |
S1 |
00-Plan.md |
| 7 |
CMS-fingerprint-first cascade (NOT JSON-LD-first) |
S1 |
00-Plan.md |
| 8 |
No URL/depth caps; streaming walker; sharded persistence |
S1 |
00-Plan.md |
| 9 |
Hub-and-spoke agent posture; every crawler emits a blueprint with agent_slot |
S1 |
00-Plan.md |
| 10 |
Match Algolia's existing dashboard design language; no parallel design system |
S2 |
01-design-language |
| 11 |
Two-chrome rule: outer dashboard chrome vs focused workspace chrome |
S2 |
2026-05-03-TwoChromeRule |
| 12 |
Pause/Resume/Stop/Re-crawl = our value-add over Algolia's fire-and-forget model |
S2 |
2026-05-03-RunControlSurface |
| 13 |
Live tree shows branches only, never URL leaves |
S2 |
01-design-language §3 |
| 14 |
ALPHA badge stays in top bar (build/version tag) |
S2 |
01-design-language |
| 15 |
Re-crawl uses scoped confirm; typed-confirm reserved for Delete only |
S2 |
02-component-recipes §10, §24 |
| 16 |
Factory is a container — sub-crawlers NOT in main list, only inside factory workspace |
S3 |
2026-05-03-FactoryEntityModel |
| 17 |
Wizard lives as "Setup" page inside WorkspaceShell — not a full-screen overlay |
S3 |
2026-05-03-WizardInWorkspace |
| 18 |
Factory creation flow: Add → factory entity → workspace opens → Setup page runs wizard |
S3 |
2026-05-03-WizardInWorkspace |
Cluster status (engineering build from S1)
| Cluster |
Status |
Notes |
| Cluster 1 (Foundations) |
✅ |
DSS, types, session/blueprint stores |
| Cluster 2 (Sitemap walker) |
✅ |
streaming walker, no caps |
| Cluster 3 (Path grouper) |
✅ |
URL → path-group clustering |
| Cluster 4 (Detection cascade) |
✅ |
json-ld → cms → … → llm |
| Cluster 5 (Sampler + structure) |
✅ |
sample HTML, DSS-aware structure |
| Cluster 6 (Index manager + crawler client) |
✅ |
per-domain indices + TS crawler client |
| Cluster 7 (Extractor generator) |
✅ |
per-domain recordExtractor generation |
| Cluster 8 (Sandbox + real test) |
✅ |
new Function sandbox + crawl_urls real test |
| Cluster 9 (API endpoints) |
✅ |
discover, sample, …, blueprints |
| Cluster 10 (Frontend hooks) |
✅ |
useFactorySession, SSE consumers |
| Cluster 11 (UI shell + components) |
✅ S4 |
CrawlerListView (factory entity model), WorkspaceShell (factory mode, Setup page), FactoryWizard (inline), AdminFactory (persistent chrome). Sub-crawler management view (UI-003) still pending. |
| Cluster 12 (Configurator + monitor) |
⚠️ Partial |
WorkspaceShell placeholder pages. Sub-crawler list in factory nav (UI-003) + backend wiring (UI-004) pending. |
| Cluster 13 (Bundle + smoke + docs) |
⚠️ Pending |
Awaiting S5 backend wiring before final verification run. |
Known issues (open)
| # |
Issue |
Discovered |
Severity |
Status |
| ~~UI-001~~ |
~~Factory entity model not in UI~~ |
S3 |
~~P0~~ |
✅ Done S4 |
| ~~UI-002~~ |
~~Wizard is full-screen overlay~~ |
S3 |
~~P0~~ |
✅ Done S4 |
| UI-003 |
Factory workspace sub-crawler management view not built |
S3 |
P1 |
S5 |
| UI-004 |
Backend not wired — all wizard UI uses mock data |
S4 |
P1 |
S5 |
| KI-003 |
Vite SSE buffering — URLs arrive all at once in dev; real-time only on Vercel |
S1 |
P2 |
S5+ |
| KI-004 |
No authentication on factory endpoints |
S1 |
P2 |
S5+ |
Open product questions
| ID |
Question |
Status |
| OQ-1 |
Domain verification step for first-crawler creation — manual via Algolia dashboard, or API-able? |
Open since S1 |
| OQ-2 |
neuralSearch cold-start — when can newly-created indices enable it? |
Open since S1 |
| OQ-3 |
When does Crawler Factory's UI merge into the main Algolia dashboard? |
Open from S2 |
| OQ-4 |
Backend support for Pause/Resume — does Algolia Crawler API expose primitives, or do we track state ourselves? |
Open from S2 |
| OQ-5 |
Health DSS row — deferred until evidence emerges |
Open since S1 |
| OQ-6 |
Community DSS row — deferred until evidence emerges |
Open since S1 |
Repository / GitHub
- Local repo:
/Users/arijitchowdhury/AI-Development/Crawler_Factory
- Git: 4 commits on main as of S4
- GitHub repo:
github.com/arijitchowdhury80/Crawler_Factory (private) ✅ created S3
- Vault sync: the durable knowledge for this project lives here under
Projects/Crawler-Factory/. The repo's docs/design-language/ mirrors Design-Language/ in this folder.
Cross-project links
- Parent: Algolia-Central-Context — Crawler Factory's place in the Content Engagement platform
- Sibling: Brief — durable brief for the parent product
- Reference: Content-Engagement-Memo — verbatim Google Doc transcript
What Session 5 should pick up
- UI-003 — Sub-crawler list in factory workspace left nav (algolia_docs / algolia_blog / algolia_community with individual run states)
- UI-004 — Wire
/api/factory/discover SSE stream to wizard Step 2 animation (replace mock timers with real backend)
- Wire wizard Step 3 "Build crawlers" to
/api/factory/create-crawler and /api/factory/session