Crawler-Factory
Wiki/Status.md
Status — Crawler Factory (Wiki Summary)
As of: 2026-05-05 (Session 25 closed)
Authoritative status: SESSION.md in repo root — that file has the detailed resume action.
Current phase: S25 closed — 5-step wizard shipped + API format fixes committed. Branch feat/step3-redesign ready to merge to main.
What's done
| Area | Status | Notes |
|---|---|---|
| Backend (all 9 modules + 9 API endpoints) | ✅ Complete | 566/566 tests. tsc clean. |
| Frontend — full wizard flow (Steps 1-5) | ✅ Complete S25 | Discover, Review, Extract, Fields, Launch |
| Design language tokens | ✅ Complete | v1.1 frozen |
| Auth middleware | ✅ Done S18 | requireAuth() on all /api/factory/* endpoints |
| Mock data purge | ✅ Done S18 | No mock fallback for discovery; real error state shown |
| URL count integrity check | ✅ Done S18 | Amber banner if |
| extract-preview endpoint | ✅ Done S18 | POST /api/factory/extract-preview — WAF-aware fetch + cheerio extraction |
| FactoryWizard.tsx split | ✅ Done S20/S21 | 2,612 lines → 7 files, 85-line orchestrator (c6c3bd7) |
| Classifier improvement (Tiers 1-3) | ✅ Done S21 | Token dict + HTML light fetch + structural signals (eaa751a). |
| Enrichment pass endpoint | ✅ Done S22 | POST /api/factory/enrich-categories — batch classify per confirmed category, SSE progress |
| TestExtractorPanel rebuild | ✅ Done S22 | Simulated Algolia record field→value table (9 domains) |
| FactoryWizard enrichment wiring | ✅ Done S22 | "Analyzing categories…" blocking SSE modal between Step 2 and Step 3 |
| Build crawlers API format fix | ✅ Done S25 | pathsToMatch + recordExtractor inside actions[]; __type='function'; appId/apiKey/rateLimit at root; crawlerId response fix |
| 5-step wizard redesign | ✅ Done S25 | Step 3=Extract, Step 4=Fields, Step 5=Launch (was monolithic Configure). Branch feat/step3-redesign. |
What's next
Immediate: Commit KI-013 fix → browser end-to-end validation → merge feat/step3-redesign → main.
- Commit
api-src/create-crawler.tsKI-013 fix (uncommitted as of S25 persist) - Browser end-to-end: Discovery → Review → Extract (all categories auto-run) → Fields → Launch → StepDone
- Merge
feat/step3-redesign→ main - Track B: Save/load factory config (download JSON + upload on new run)
- Decide on Vercel deployment (KI-008 requireAuth in prod mode)
Known issues
| ID | Description | Priority |
|---|---|---|
| KI-003 | Vite SSE buffering in dev (works on Vercel) | Low |
| KI-005 | Flat-urlset sites get 50 samples — add regression test | Low |
| KI-006 | Depth-3+ sitemapindexes: only first grandchild urlset sampled | Low |
| KI-007 | Cross-section URL dedup: allSampledUrls may have duplicates | Low |
| KI-013 | ✅ FIXED S25 tail — write-only key created, ALGOLIA_CRAWLER_WRITE_KEY in .env.local, code updated (uncommitted) | — |