Dashboard/Crawler-Factory/Wiki/Constraints.md
Crawler-Factory

Wiki/Constraints.md

Constraints — Crawler Factory

Last updated: 2026-05-03

Non-goals (explicit)

These are things Crawler Factory will never do in Phase 1. Revisit only with an explicit decision.

  • Not a general-purpose crawler UI: Crawler Factory creates N crawlers per site, all domain-scoped. It does not replace Algolia's native single-crawler creation flow for arbitrary use cases.
  • Not a data transformation layer: The factory creates crawlers and indices. Normalization, enrichment, and transformation are handled by Layer 1 siblings (Transformer, Enrichment) — not this tool.
  • Not a user-facing search interface: The factory builds the data infrastructure. Algolia NeuralSearch + Agent Studio sit on top.
  • Not a standalone SaaS product: Crawler Factory is embedded in the Algolia Central admin shell at /admin/factory. It is not a separately deployable app.
  • No CMS connectors beyond web: Phase 1 is web-only. Future connectors (CMS, KB, video, code) are F1–F4 in [Architecture/Future Extensions].
  • No multi-tenant orchestration in Phase 1: Multi-brand corporates (LVMH pattern — multiple sites, shared factory) are 06-Multi-Tenant.md scope, not Phase 1.
  • No Heygen / video pipeline: That is Demo Factory (Video-Editing project).

Technical limits

  • SSE buffering in dev: Vite SSE buffering causes all URLs to arrive at once in development. Real-time streaming only works on Vercel (production). Known issue KI-003.
  • No auth on factory endpoints: API routes currently unprotected. Authentication is Phase 2. Known issue KI-004.
  • LLM for extractor generation: The extractor-generator.ts module requires an LLM call (MiniMax via lib/llm/). If the LLM is unavailable, extractor generation fails — no fallback template today.
  • Algolia Crawler API coverage: Pause/Resume primitives depend on Algolia Crawler API support (OQ-4 — open). If API doesn't expose these, we track state ourselves.
  • Domain verification: First-crawler creation may require manual domain verification via Algolia dashboard if the API doesn't expose this step (OQ-1 — open).

Known gaps (not bugs)

  • neuralSearch cold-start: Newly-created indices may not immediately support NeuralSearch enablement. Timing unclear (OQ-2 — open).
  • Health + Community DSS rows: Two content domain types (Health, Community) deferred until empirical evidence emerges (OQ-5, OQ-6).
  • Factory workspace sub-crawler management view: Factory workspace shows placeholder pages for Configurator and Monitor. Full management view pending (UI-003, S4).

Dependencies we cannot control

Dependency What we rely on Risk if it changes
Algolia Crawler REST API Create crawlers, test with crawl_urls, monitor progress If API changes, crawler-client.ts breaks
Algolia Search API Create per-domain indices If index-creation API changes, index-manager.ts breaks
MiniMax LLM via lib/llm/ recordExtractor generation If LLM provider changes or quota exceeded, extractor generation fails
Playwright WAF stealth fetch Playwright updates can break stealth fingerprinting
Algolia meta-index schema Session state persistence Schema changes require migration