Dashboard/Crawler-Factory/Core/00-Plan.md
Crawler-Factory

Core/00-Plan.md

Crawler Factory — Implementation Plan (Spike)

Branch: rc3-phoenix Mode: Spike (v1 = full end-to-end loop; polish deferred to v2) Planning model: Opus 4.7 (1M context). Build model: switch to Sonnet 4.6 before Phase 1 implementation begins. Document on the build branch. Plan file: /Users/arijitchowdhury/.claude/plans/joyful-strolling-lobster.md

1. Context

Problem

Today, every Algolia crawler is hand-built in the dashboard. We have one crawler (c12d1f0e-1772-4744-bdf2-985bd041b143) writing to one index. Adding a new domain or splitting an existing one into per-section crawlers means manual config in the dashboard, manual recordExtractor JS authoring, manual seed-list curation, and no version control of the result. SESSION.md says we want a JobController — parallel section-based crawling that creates N crawlers programmatically and all write to the same index. The Crawler Factory is the human-in-the-loop path that gets us there: it takes a website URL, discovers the sitemap, helps the user decide how to slice it into categories, generates a tailored recordExtractor per category, lets the user verify against real samples, and then commits the crawler to Algolia and kicks it off.

Outcome

A guided admin wizard at /admin/factory that takes a domain URL → produces N production crawlers in Algolia (one per content domain the user selects), each writing to its own purpose-built index with a domain-tailored schema and recordExtractor. Each index is the future home of a specialist agent (support-agent, marketing-agent, education-agent, etc.); the factory produces the data foundation those agents will read from. Every step is observable (rolling log), editable (user is in the loop), and resumable (session state lives in Algolia itself). Discovery has no URL cap, no depth cap.

Decisions locked this session

# Decision Choice
1 Persistence layer Algolia meta-index algoliacentral_factory_sessions (sharded — see §4)
2 UI placement New sub-route /admin/factory
3 Index target per crawler One index per content domain (e.g., algoliacentral_marketing, algoliacentral_support, algoliacentral_education). Factory creates the index if it doesn't exist. User can override the auto-suggested index at creation time. NOT a single shared index.
4 Test crawl mechanism Sandbox first (cheerio + new Function), real crawl_urls as final confirmation
5 LLM provider Reuse global lib/llm/ (whatever LLM_PROVIDER resolves to — currently MiniMax)
6 recordExtractor source of truth Algolia Crawler dashboard is canonical at runtime; we ALSO version-control a copy at crawler-configs/{domain}/{contentDomain}-{categoryId}.js
7 Content classification Schema.org-driven with detection cascade: JSON-LD → microdata/RDFa → OpenGraph → HTML heuristics → URL pattern → LLM classification. See §4 (Domain Schema Standard).
8 URL discovery limits No cap. Stream + cursor through whatever the site exposes. Visited-set handles cycle prevention; no arbitrary depth limit.
9 Architecture posture Designed for hub-and-spoke agent future (§15). Each crawler+index pair is a specialist agent's data foundation. Factory persists the metadata that makes future agent scaffolding trivial.

2. Architecture

Finite state machine

Tenant (e.g., "LVMH")
  ├─ FactorySession 1 → corporate (lvmh.com)
  ├─ FactorySession 2 → louisvuitton.com
  ├─ FactorySession 3 → dior.com
  ├─ FactorySession 4 → celine.com
  └─ ... (one per brand domain)

Component boundaries

Index: algoliacentral_factory_tenants
  objectID: <tenantId>
  name: "LVMH"
  description: "Luxury conglomerate; 75+ brands across fashion, wines, spirits, cosmetics, watches"
  parent_domain: "lvmh.com"
  brand_domains: ["louisvuitton.com", "dior.com", "celine.com", ...]
  session_ids: ["sess_001", "sess_002", ...]
  blueprint_ids: [...]
  created_at: ts

3. Tech additions

Dep Why Where
fast-xml-parser Parse sitemap.xml + sitemap indexes Backend bundle
(none other) robots.txt parser is trivial — roll our own —

cheerio, playwright, algoliasearch are already installed.

Add to scripts/bundle-api.mjs externals: fast-xml-parser.

3a. Seeder-list discovery: best practices we follow

The user asked: "What will you use to do this? Is it just web-fetch or some other tool?" Answer: a small standard pipeline, no external library beyond fast-xml-parser. Why this is the right approach:

  1. robots.txt first (RFC 9309). Honors site policy. Crawlers that skip this get blocked. We GET /robots.txt and parse Sitemap: directives (one or many).
  2. Sitemap protocol fallback (sitemaps.org spec). If robots.txt has no Sitemap: directive, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml in order. These are the four canonical locations across all major CMSes (WordPress, Shopify, Webflow, Hubspot, custom).
  3. Recurse sitemap indexes without a depth cap, with arbitrary nesting allowed. The sitemaps.org spec officially permits 1 level of nesting (<sitemapindex> → <urlset>), but real sites violate this. Empirical evidence (§14): - Microsoft.com has nested sitemap-indexes 2-3 levels deep (root index → product index → language urlsets). - learn.microsoft.com declares 1000+ child sitemaps including nested structures. - The walker treats EITHER <sitemapindex> or <urlset> as valid at any depth. Cycle prevention via a Set<string> of fetched URLs (correct + unbounded). No depth limit.
  4. Parallel sub-sitemap fetch with concurrency control. Microsoft Learn has 1000+ sub-sitemaps. Serial fetching would take hours. The walker fetches sub-sitemaps in parallel batches of 8 concurrent by default (configurable per-domain via Crawl-delay). This is the difference between "factory works for Algolia.com" and "factory works for learn.microsoft.com."
  5. No URL cap. The factory streams URLs as they're discovered — write batches of 1000 into Algolia url_shard records as they come in (§4d). Memory stays bounded; the wizard UI paginates. A 1M-URL site (Microsoft Learn scale) is supported by the same machinery as a 5K-URL site.
  6. Single-file size resilience. Empirical: dotnet_en-us_1.xml on Microsoft Learn exceeds 10MB. Walker uses streaming SAX-style parser when file > 5MB instead of loading the whole DOM (fast-xml-parser supports streaming via XMLParser with chunk handler). This avoids OOM on Vercel functions (1024MB ceiling).
  7. hreflang alternate extraction. Multi-language sitemap entries embed <xhtml:link rel="alternate" hreflang="..." href="..."/> inside each <url> block. Walker extracts these AND yields them as separate URLs (or yields a localeAlternates: {...} map per URL). The user picks which locales to crawl in the categorizer step.
  8. Native fetch is enough for sitemaps and robots.txt — they're plain XML/text. Cheerio comes in only when sampling individual pages for content classification.
  9. Respect Crawl-delay in robots.txt where present. Default rateLimit: 8 req/s. Inflate if robots.txt asks; never deflate below site policy. Drupal sites consistently use Crawl-delay: 10 — honor strictly.
  10. gzip support — sitemaps are sometimes served as .xml.gz. Native fetch + DecompressionStream('gzip') handles this. The walker also detects gzip via Content-Encoding response header.
  11. No WebFetch MCP tool — too coarse for recursive sitemap walking, can't honor robots.txt, can't stream, can't parallelize. Native fetch + fast-xml-parser is right-sized.

3a-bis. Pre-flight scope estimation

Before committing to a full discovery, the factory does a fast scope probe (1-2 seconds): fetch robots.txt, fetch the top-level sitemap, count <sitemap> children if it's an index, multiply by an average urlset size estimate. Surface the estimate to the user:

"This site exposes a sitemap index with 531 sub-sitemaps. Estimated total URLs: 100,000–500,000. Estimated discovery time at 8 concurrent: 12-30 minutes. Proceed?"

The user picks: (a) full crawl, (b) sub-tree only — let me enter a path prefix, (c) cancel. This handles the user's "I want to crawl just learn.microsoft.com/dotnet/" case without forcing a 10M-URL discovery.

This is the same pattern used by Algolia's own crawler, Google's Search Console fetcher, and sitemapper (npm). We don't use sitemapper itself because (a) rolling our own takes ~120 lines, (b) we want streaming/batched output not a single Promisewzxhzdk:19, (c) one fewer dep.

3b. Content-domain classification: REVISED based on empirical web reality

Critical correction (from Web Almanac 2024 + 14-site empirical audit, see §14): Schema.org adoption is far lower than the v0 plan assumed. Only 41% of pages have JSON-LD globally; on actual content pages of major publishers (TechCrunch, GOV.UK, USA.gov, Drupal.org), JSON-LD is mostly absent. Schema.org cannot be the primary classifier. It must be the enrichment layer, not the foundation.

The corrected cascade leads with CMS fingerprinting + URL patterns (which cover ~85% of the web reliably) and uses schema.org only when present:

  1. CMS fingerprint — most reliable single signal. ~51% of all sites use a recognizable CMS (WordPress 18% alone, plus Drupal/AEM/Shopify/Wix/Squarespace/Joomla/Hubspot/Webflow). Each leaves stable, hard-to-fake signatures: - WordPress: /wp-content/, /wp-json/, wp-block-*/entry-*/post-* classes, REST API at /wp-json/wp/v2/posts - Drupal: Crawl-delay: 10 in robots.txt, /themes/custom/, /sites/default/files/, /?q=search legacy clean URLs, region-*/node-*/field-* classes, data-drupal-* attrs - AEM (Adobe Experience Manager): /content/dam/, data-cmp-* attrs, /content/{site}/{lang}/... URL convention - Shopify: cdn.shopify.com, /products/, /collections/, Shopify.theme, JSON product API - Hubspot: /hubfs/, hs-cta-* classes, _hsmi=... URL params - Webflow: data-wf-* attrs, /wf-domain/... - Squarespace: static1.squarespace.com, sqs-block classes - Wix: _wix* cookies, wixstatic.com, data-mesh-id attrs - Salesforce Experience Cloud: data-aura-* attrs (LWC: lwc-*) - Custom enterprise: identified by elimination

When CMS is detected, we apply that CMS's known URL-pattern → content-domain mapping (see §16 for per-CMS playbooks). This is high signal because CMS owners themselves built these conventions.

  1. URL pattern + path tokens (universal — present on 100% of sites). Strong, well-known conventions across the web: - /blog/, /news/, /articles/, /insights/, /{YYYY}/{MM}/{DD}/{slug}/ → marketing - /customers/, /case-studies/, /success-stories/, /our-customers/ → customer-stories - /docs/, /documentation/, /developer/, /api/, /reference/, /guides/ → technical - /support/, /help/, /faq/, /troubleshooting/, /hc/ (Zendesk) → support - /courses/, /learn/, /academy/, /training/, /tutorials/ → education - /products/, /services/, /solutions/, /offerings/ → product-catalog - /events/, /webinars/, /conferences/, /calendar/ → events - /legal/, /terms/, /privacy/, /policies/ → legal - /community/, /forum/, /discuss/ → social/community - Localization prefixes (/en/, /us-en/, /de-de/, /es/) — preserved but not used for domain classification

  2. JSON-LD parsing — when present (41% of pages globally per Web Almanac 2024). Walk <script type="application/ld+json">, parse @type chain, match against DSS schema.org types. High signal when present (confirms or refines CMS+URL guess) but never assumed.

  3. OpenGraph + Twitter Card — 64% coverage. og:type is coarse (only article/website/video/product values) but reliable when CMS+URL are inconclusive.

  4. Microdata + RDFa — legacy. RDFa 66% (declining), microdata 26%. Mostly noise; check only if 1–4 produce nothing.

  5. Date + author text regex — 99% of articles have a visible date and author somewhere on the page, even if not in semantic markup. Universal fallback for marketing/support/education classification (any "dated content with byline" → article-family).

  6. Semantic HTML heuristics — <article> adoption is ~5%, <time> is ~8% (Web Almanac, low). These are weak signals. Use only when nothing else applies.

  7. LLM classification — last resort. Cheerio-extracted text (first 4KB) + DSS table → "which content domain best fits?" Fires only when steps 1–7 are inconclusive (combined confidence < 0.55).

Realistic per-layer hit rates (from Web Almanac 2024 + sample verification, §14):

Layer Coverage Confidence when hit
1. CMS fingerprint ~51% of sites 0.80–0.90
2. URL pattern ~100% (always present) 0.50–0.75
3. JSON-LD 41% of pages 0.85–0.95
4. OpenGraph 64% of pages 0.55–0.70
5. Microdata/RDFa 26%/66% 0.40–0.65
6. Date+author regex ~99% of articles 0.40–0.60 (article-family only)
7. Semantic HTML ~5% 0.45–0.60
8. LLM classification 100% (always available) 0.50–0.85

Aggregator: take max-confidence layer, but BOOST when multiple layers agree (CMS+URL+JSON-LD all pointing to "marketing" → 0.95). Show the user the contributing signals, not just a number.

3c. Algolia index design per content domain

Each DSS-defined content domain gets its own Algolia index. Index settings are derived from DSS, not invented per crawl:

  • searchableAttributes — chosen to match what users search for in that domain. Marketing puts headlines first, support puts the question/name first, technical puts code samples and method names alongside body text.
  • customRanking — recency for marketing/news; reviewed-date for support; version for technical; aggregateRating for products; startDate (ascending — upcoming first) for events.
  • attributesForFaceting — domain-relevant filters. Industry/companySize for customer-stories; programmingLanguage/version for technical; severity/productAffected for support.
  • neuralSearch mode — enabled by default (cold-start uses neuralSearchPreset: "default" once the index has Insights events; bootstrap from a sibling index if newly created and empty).
  • typo + plural settings — same defaults across domains; tuning happens in v2 after live traffic.

This DSS-derived approach means the index design IS the crawler design — they're two faces of the same DSS row. The factory commits both atomically.

3d. WAF / bot-detection reality

Empirical finding (§14): Salesforce, Adobe, Oracle, BMW, Mercedes, and most major enterprise sites WAF-block plain fetch from datacenter IPs (Vercel runs in datacenters). Symptoms: HTTP 403, 429, or timeout on robots.txt or sample fetches.

Vercel functions cannot easily route through residential proxies. The factory must treat WAF blocking as a first-class case:

  • Detect early. When the discovery step gets repeated 403/timeout/CAPTCHA from a domain, surface it in the UI: "This site appears to be WAF-protected. Options: (a) request Algolia Crawler dashboard to whitelist your crawler IP, (b) switch to Playwright stealth mode, (c) supply a manual seed list."
  • Playwright fallback (v1, not v2). Use playwright (already installed) with a stealth-style user-agent, realistic viewport, and small per-request jitter. We accept the latency cost. Only for domains flagged WAF-protected.
  • Algolia Crawler IP allowlist. Once the user creates a real crawler, Algolia Crawler runs from a known IP range that many enterprise sites already allowlist for SEO. The factory's pre-creation discovery is the painful part; once the real crawler is running, it usually works.

These are real operational concerns, not theoretical. The plan's task list (§7) reflects this: Playwright sampling is in v1.

4. Data model — Domain Schema Standard (DSS)

The fundamental shift: instead of a single enterprise_ledger record type writing to a single index, we have content domains. Each content domain: - Maps to one or more schema.org types (e.g., marketing → BlogPosting, NewsArticle, OpinionArticle) - Has its own record schema (different fields per domain — a recipe has cookTime, a course has learningResourceType, a support article has lastReviewed) - Writes to its own index with config tuned for that domain (different searchableAttributes, customRanking, attributesForFaceting) - Is the data foundation for a future specialist agent (support-agent on support index, etc.)

4a. The Domain Schema Standard (DSS) — the core taxonomy

DSS is our internal type system. Each row maps a content domain to: schema.org parent types, record schema, index name, and Algolia config tuning. DSS is data, not code — it lives in lib/factory/dss.ts as a typed registry, easy to extend.

Content domain Index name schema.org types in scope Distinctive record fields Algolia config tuning
marketing algoliacentral_marketing BlogPosting, NewsArticle, OpinionArticle, Article, AnnouncementPage, SpecialAnnouncement headline, articleBody, datePublished, author, articleSection, keywords searchable: [headline, articleBody, keywords]; customRanking: [desc(datePublished), desc(view_count)]; facets: [articleSection, author, year]
support algoliacentral_support TechArticle (troubleshooting), FAQPage, Question, Answer, HowTo (fix-it) name, articleBody, lastReviewed, acceptedAnswer, severity, productAffected, supportLevel* searchable: [name, articleBody, acceptedAnswer]; customRanking: [desc(lastReviewed), desc(helpful_count)]; facets: [productAffected, severity, status]
education algoliacentral_education Course, LearningResource, HowTo, Guide, VideoObject (educational) name, description, learningResourceType, educationalLevel, timeRequired, hasCourseInstance, transcript (video) searchable: [name, description, articleBody, transcript]; customRanking: [desc(datePublished), desc(enrollment_count)]; facets: [educationalLevel, learningResourceType, language, duration]
technical algoliacentral_technical TechArticle, APIReference*, SoftwareSourceCode, SoftwareApplication, WebAPI name, articleBody, programmingLanguage, version, applicationCategory, codeSampleType, apiEndpoint* searchable: [name, articleBody, programmingLanguage, codeSampleType]; customRanking: [desc(version), desc(view_count)]; facets: [programmingLanguage, framework, version, apiEndpoint]
customer-stories algoliacentral_customers Article (case study), Review, Person, Organization headline, articleBody, about (Organization name), industry, companySize, region, metrics, quotes searchable: [headline, articleBody, about.name, quotes]; customRanking: [desc(datePublished), desc(prominence)]; facets: [industry, companySize, region, useCase]
product-catalog algoliacentral_products Product, Offer, Service, Brand name, description, sku, offers, brand, category, aggregateRating, image searchable: [name, description, brand]; customRanking: [desc(aggregateRating), desc(popularity)]; facets: [brand, category, priceRange, rating]
events algoliacentral_events Event, EducationEvent, BusinessEvent, Conference name, startDate, endDate, location, organizer, eventType, recordingUrl searchable: [name, description]; customRanking: [asc(startDate)] (upcoming first); facets: [eventType, location, year]
legal algoliacentral_legal DigitalDocument, Legislation, TermsOfService, PrivacyPolicy name, text, version, datePublished, jurisdiction, documentType* searchable: [name, text]; customRanking: [desc(version), desc(datePublished)]; facets: [jurisdiction, documentType]
social algoliacentral_social VideoObject (YouTube), Comment, Conversation, SocialMediaPosting* text, transcript, uploadDate, channel, viewCount, commentCount, platform, sentiment searchable: [text, transcript]; customRanking: [desc(uploadDate), desc(engagement)]; facets: [platform, channel, sentiment]

Asterisked fields are not in schema.org — they're our extensions, defined in DSS as dss: namespace fields.

The DSS is extensible at runtime: a user (or future automation) can add a new content domain by adding a new entry to the registry. No code changes needed if the new domain reuses an existing schema.org type set.

4b. Detection cascade — REVISED based on Web Almanac 2024 + 14-site empirical audit

Cascade order is CMS-first, not schema.org-first. Schema.org is a confirmer/refiner, not the foundation. See §3b for the empirical justification and §14 for the source data.

# Method What it inspects Expected hit rate Confidence when hit
1 CMS fingerprint URL paths (/wp-content/, /content/dam/, /sites/default/files/), HTML class signatures (wp-block-*, region-*, data-cmp-*, data-aura-*), robots.txt patterns (Drupal's Crawl-delay: 10), known JSON endpoints (/wp-json/wp/v2/posts, Shopify product API) 51% of all sites 0.80–0.90
2 URL pattern /blog/, /customers/, /docs/, /help/, /support/, /courses/, /products/, /{YYYY}/{MM}/, etc. (full table in §3b) 100% (always available) 0.50–0.75
3 JSON-LD parser <script type="application/ld+json"> blocks. Walks @type chain. Matches against DSS schema.org types. 41% of pages globally; ~7% with content-classifying types 0.85–0.95
4 OpenGraph + Twitter Card og:type (article/website/video/product), og:article:section, twitter:card 64% of pages 0.55–0.70
5 Microdata + RDFa itemtype="https://schema.org/X"; RDFa typeof 26% / 66% — declining 0.40–0.65
6 Date + author regex "Published: YYYY-MM-DD", "by Author Name", <time datetime=...> 99% of articles 0.40–0.60 (article-family only)
7 Semantic HTML heuristics <article>, <main>, <header>, breadcrumbs, code-block density, table-of-contents shape ~5–8% (low) 0.45–0.60
8 LLM classification Cheerio-extracted text (first 4KB) + DSS table → "which content domain best fits?" Cheap; ~$0.001 per call. 100% (always available) 0.50–0.85

Aggregation rules: - Each layer outputs {contentDomain, confidence} or null. - Final confidence = max-confidence layer + boost when multiple layers AGREE on the same domain (each agreement adds 0.05, capped at 0.95). - LLM (layer 8) only fires when combined confidence from layers 1–7 is < 0.55. - UI shows the contributing layers per pathGroup so the user can see WHY a classification was made and override if needed.

Confidence thresholds: - ≥ 0.85: green — auto-accept, recommend "Configure" - 0.65–0.84: yellow — show confidence, recommend manual review - < 0.65: amber — flag for manual override; default action is LLM tiebreak + manual confirmation

4c. Record schemas — one zod schema per content domain

Each content domain has its own zod schema. Common base + domain-specific extension:

wzxhzdk:2

The legacy CrawlerRecord (packages/crawl/src/crawl/models.py) is kept as-is for the existing algolia-central_enterprise_ledger index — it's not deleted. New crawlers from the factory write to per-domain indices using the new schemas. Migration is opt-in.

4d. Sharded session storage (no record-size cap)

A single Algolia record has a 100KB limit. For sites with 100K+ URLs, one giant session record won't fit. Shard:

wzxhzdk:3

attributesForFaceting: [filterOnly(record_type), filterOnly(parent_id)]. Reads filter on parent_id:<sessionId> then split by record_type.

The factory streams URLs into shards as discovery proceeds — no waiting for full discovery, no memory pressure.

4e. Factory blueprints — agent-ready metadata

Every successfully created crawler also writes a blueprint to a separate index. This is the metadata a future orchestrator/specialist-agent scaffolder will read.

wzxhzdk:4

This index is the link between the crawl world and the agent world. When we build the orchestrator + specialist agents (future spike), it reads from algoliacentral_factory_blueprints to know what specialists exist and what domains they cover.

4f. Session main record (zod, slimmed for 100KB compliance)

wzxhzdk:5

4g. API contract (revised)

wzxhzdk:6

5. UI specs

5a. Visual conventions (existing /admin aesthetic — match exactly)

  • Background: bg-[#0a0a0f], text white, borders border-white/10
  • Headers: font-black uppercase tracking-tighter text-3xl, accent indigo
  • Section labels: text-sm font-bold text-white/40 uppercase tracking-widest
  • Mono font for IDs/URLs/timestamps
  • Badges: bg-{color}-500/20 text-{color}-400 border border-{color}-500/30 rounded-md px-2 py-1 text-[10px] font-bold uppercase tracking-widest
  • Animations: framer-motion fade-up on section mount

5b. Page layout

wzxhzdk:7

After discovery, Panel A switches to category review:

wzxhzdk:8

When user clicks Configure Selected, a per-category drawer opens (one at a time):

wzxhzdk:9

When user clicks Commit + Create Crawler:

wzxhzdk:10

6. Phasing — v1 vs deferred

v1 (this spike) — must ship

  • ✅ Streaming sitemap discovery (no URL cap, no depth cap, gzip support, visited-set cycle prevention)
  • ✅ Path-prefix grouping → per-pathGroup detection cascade (JSON-LD → microdata → OG → heuristics → URL → LLM fallback)
  • ✅ DSS registry covering 9 content domains shipped as defaults (marketing, support, education, technical, customer-stories, product-catalog, events, legal, social)
  • ✅ Cheerio-based sampling (5–10 pages per pathGroup); SPA detection flag, manual renderJavaScript toggle
  • ✅ DSS-aware recordExtractor generation (LLM fills in selectors against the domain template)
  • ✅ Editable extractor code box + "Regenerate" + free-form feedback
  • ✅ Sandbox test (cheerio + new Function + zod validation against the domain's record schema)
  • ✅ Real test crawl via Algolia crawl_urls
  • ✅ Per-domain index creation (algoliacentral_<domain>) with DSS-derived settings
  • ✅ Crawler creation + reindex + live progress SSE
  • ✅ Sharded session storage (session, url_shard, sample, log records — no 100KB ceiling)
  • ✅ Blueprint persistence in algoliacentral_factory_blueprints (agent-ready metadata)
  • ✅ recordExtractor copies version-controlled at crawler-configs/{site-domain}/{contentDomain}-{pathGroupId}.js

v2 (deferred, documented in §15 future)

  • HTML link-graph discovery for sites without sitemaps
  • Playwright fallback for SPA pages (auto-rendered)
  • Visual extraction overlay (rendered page with field highlights)
  • Per-index "Data Analysis" insights panel (port audit.py — field-coverage bars, type distribution, dupe scan)
  • Custom DSS rows: user-defined content domains beyond the default 9
  • Re-discovery / drift detection (alert when sitemap changes post-crawl)
  • Site-template presets per CMS (WordPress, Shopify, Webflow, Hubspot — pre-tuned selectors)
  • Specialist agent scaffolding — read blueprints index → create Agent Studio agent per blueprint → register with orchestrator (see §15)
  • Orchestrator agent — fan-out/fan-in router (see §15)
  • Multi-domain JobController (queue + parallel scheduler)

7. Implementation tasks (TDD, bite-sized)

Every task: write failing test → run → minimal impl → run → commit. Use superpowers:test-driven-development. Tasks are ordered to flow bottom-up: data layer → discovery → classification → configuration → orchestration → UI.

Task 0: Scaffold + dependency

Files: - Create: crawler-configs/.gitkeep - Modify: package.json (add fast-xml-parser) - Modify: scripts/bundle-api.mjs (add fast-xml-parser to externals)

  • [ ] Install fast-xml-parser: npm install fast-xml-parser
  • [ ] Add to bundle-api.mjs externals array: 'fast-xml-parser'
  • [ ] Commit: chore(factory): add fast-xml-parser dep

Task 1: Domain Schema Standard registry

Files: - Create: lib/factory/dss.ts - Test: lib/factory/__tests__/dss.test.ts

  • [ ] Write failing test asserting getDss('marketing') returns the marketing entry with schema.org types [BlogPosting, NewsArticle, OpinionArticle, Article, AnnouncementPage, SpecialAnnouncement], indexName algoliacentral_marketing, and the expected algolia config
  • [ ] Write failing test for each of the 9 default domains (marketing, support, education, technical, customer-stories, product-catalog, events, legal, social)
  • [ ] Write failing test: dssBySchemaOrgType('BlogPosting') returns 'marketing'
  • [ ] Run — expect FAIL
  • [ ] Implement dss.ts as a typed registry: Record<ContentDomain, DssEntry>. Exports DSS, getDss, dssBySchemaOrgType, listContentDomains.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): Domain Schema Standard registry

Task 2: Zod schemas — record union + session

Files: - Create: lib/factory/types.ts - Test: lib/factory/__tests__/types.test.ts

  • [ ] Write failing tests for each domain record schema (MarketingRecord, SupportRecord, EducationRecord, TechnicalRecord, CustomerStoryRecord, ProductRecord, EventRecord, LegalRecord, SocialRecord) — minimal valid + invalid cases
  • [ ] Write failing test for FactoryRecordSchema discriminated union — accepts a marketing record, rejects a marketing record with content_domain:'support'
  • [ ] Write failing test for FactorySessionSchema — slimmed (no inline URL list)
  • [ ] Write failing test for BlueprintSchema
  • [ ] Run — expect FAIL
  • [ ] Implement all schemas in types.ts
  • [ ] Tests pass
  • [ ] Commit: feat(factory): zod schemas for record union, session, blueprint

Task 3: Streaming sitemap walker (no caps)

Files: - Create: lib/factory/sitemap.ts - Test: lib/factory/__tests__/sitemap.test.ts (mocked fetch)

  • [ ] Write failing test: given mocked robots.txt with Sitemap: https://x.com/sitemap.xml, discoverSitemaps('https://x.com') returns ['https://x.com/sitemap.xml']
  • [ ] Write failing test: given mocked sitemap.xml with 3 <url><loc> entries, walkSitemap(url) yields 3 URLs via async iterator
  • [ ] Write failing test: given mocked sitemapindex with 2 sub-sitemaps, walker recurses and yields combined URLs across all leaves
  • [ ] Write failing test: given a sitemapindex that points to itself (cycle), walker visits each URL exactly once and exits
  • [ ] Write failing test: given a .xml.gz sitemap, walker decompresses and parses correctly
  • [ ] Write failing test: walker has NO depth or count limit (test with depth-5 nested index + 10K URLs — confirm all yielded)
  • [ ] Run — expect FAIL
  • [ ] Implement as async function* walkSitemap() — async iterator yielding URL batches of 1000. Uses visited-set for cycle prevention. Uses DecompressionStream('gzip') for .gz URLs.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): streaming sitemap walker, no caps

Task 4: Path grouper

Files: - Create: lib/factory/path-grouper.ts - Test: lib/factory/__tests__/path-grouper.test.ts

  • [ ] Write failing test: given URLs [/blog/a, /blog/b, /docs/x, /docs/y, /], groupByPathPrefix(urls) returns [{pattern:'/blog/*', count:2, samples:...}, {pattern:'/docs/*', count:2, ...}, {pattern:'/', count:1, ...}]
  • [ ] Write failing test: with 100K URLs, grouper streams (returns iterator); never holds all URLs in memory
  • [ ] Run — expect FAIL
  • [ ] Implement: use first non-empty path segment as group key; merge groups with <3 URLs into "other". Returns iterator of pathGroups.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): streaming path-prefix grouper

Task 5: Detection cascade (classifier)

Files: - Create: lib/factory/classifier.ts - Test: lib/factory/__tests__/classifier.test.ts

  • [ ] Write failing test: given HTML with <script type="application/ld+json">{"@type":"BlogPosting"}</script>, classifyJsonLd(html) returns {contentDomain:'marketing', schemaType:'BlogPosting', confidence:0.95}
  • [ ] Write failing test: given HTML with itemtype="https://schema.org/TechArticle", classifyMicrodata(html) returns {contentDomain:'technical', schemaType:'TechArticle', confidence:0.80}
  • [ ] Write failing test: given HTML with <meta property="og:type" content="article"> and <meta property="og:article:section" content="Engineering">, classifyOpenGraph(html) returns Article-family with confidence:0.70
  • [ ] Write failing test: given HTML with <article>...code blocks...<time>...</time></article> and no metadata, classifyHeuristic(html) returns technical or marketing with confidence:0.55
  • [ ] Write failing test: given URL https://x.com/help/troubleshooting, classifyByUrl(url) returns 'support' with confidence:0.50
  • [ ] Write failing test: classify(html, url) runs the full cascade and returns the highest-confidence result with detected_via set
  • [ ] Write failing test: when all cascade steps return confidence <0.6, classify falls back to classifyWithLLM(html, url, dss) which calls the LLM and returns its choice
  • [ ] Run — expect FAIL
  • [ ] Implement each cascade step as a pure function; orchestrator runs them and aggregates
  • [ ] Tests pass
  • [ ] Commit: feat(factory): content-domain detection cascade

Task 6: Sharded session store

Files: - Create: lib/factory/session-store.ts - Test: lib/factory/__tests__/session-store.test.ts

  • [ ] Write failing tests using mocked algoliasearch client:
  • saveSession(session) writes the main record_type:'session' record
  • appendUrls(sessionId, urls) writes record_type:'url_shard' records in batches of 1000
  • addSample(sessionId, pathGroupId, urlHash, sample) writes record_type:'sample'
  • appendLog(sessionId, log) writes record_type:'log'
  • getSession(id) reads the session record + tail of latest 50 log records (filter by parent_id)
  • streamUrls(sessionId) reads url_shard records in order, yields URL batches
  • [ ] Run — expect FAIL
  • [ ] Implement against algolia-central_factory_sessions index. Configure facets [filterOnly(record_type), filterOnly(parent_id)].
  • [ ] Tests pass
  • [ ] Commit: feat(factory): sharded session store

Task 7: Blueprint store

Files: - Create: lib/factory/blueprint-store.ts - Test: lib/factory/__tests__/blueprint-store.test.ts

  • [ ] Write failing tests: saveBlueprint(b), getBlueprint(crawlerId), listBlueprintsByContentDomain('support'), listAllBlueprints()
  • [ ] Implement against algoliacentral_factory_blueprints index
  • [ ] Tests pass
  • [ ] Commit: feat(factory): blueprint store (agent-ready metadata)

Task 8: Sampler

Files: - Create: lib/factory/sampler.ts - Test: lib/factory/__tests__/sampler.test.ts

  • [ ] Write failing test: sampleUrls(['url1','url2'], {timeout:5000}) returns [{url, html, status, contentLength, isSpa: boolean, headers}, ...]
  • [ ] Write failing test: SPA detection — given HTML with <div id="root"></div> and <noscript>You need to enable JavaScript</noscript> returns isSpa: true
  • [ ] Write failing test: response > 5MB is truncated and truncated: true flag set
  • [ ] Run — expect FAIL
  • [ ] Implement: parallel fetch with abort controllers, 5s timeout each, user-agent AlgoliaCentralFactory/1.0, response size cap, SPA heuristic
  • [ ] Tests pass
  • [ ] Commit: feat(factory): URL sampler with SPA detection

Task 9: Structure analyzer (DSS-aware)

Files: - Create: lib/factory/structure-analyzer.ts - Test: lib/factory/__tests__/structure-analyzer.test.ts

  • [ ] Write failing test: analyzeStructure(sample, contentDomain, mockLLM) returns {schemaOrgJsonLd, selectors: {[fieldName]: cssSelector}, confidence}. The DSS row for contentDomain dictates which fields to look for (e.g., for marketing: headline, articleBody, author, datePublished, articleSection).
  • [ ] Write failing test: when JSON-LD is present, selectors are partly bypassed — fields from JSON-LD take precedence
  • [ ] Write failing test: when fields are missing, returns missing: [fieldName] so the caller can show "best effort"
  • [ ] Run — expect FAIL
  • [ ] Implement: extract JSON-LD blocks first, then for each missing DSS field run a per-field LLM prompt asking "given this HTML and that we need fieldX, what selector?". Cache common patterns.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): DSS-aware structure analyzer

Task 10: Extractor generator (per-domain)

Files: - Create: lib/factory/extractor-generator.ts - Create: lib/factory/record-extractor-template.ts (per-domain scaffold + LLM prompts) - Test: lib/factory/__tests__/extractor-generator.test.ts

  • [ ] Write failing test: generateExtractor(pathGroup, structureAnalysis, dssEntry, mockLLM) returns a JS string. When wrapped in new Function, executed against sample HTML, the resulting records pass dssEntry.recordSchema validation.
  • [ ] Write failing test: the generated extractor includes JSON-LD parsing as fallback — if a future page has JSON-LD that wasn't in the sample, it's still extracted
  • [ ] Write failing test: 9 separate tests, one per content domain, that the generator produces a domain-correct extractor (marketing → uses headline + articleBody, support → uses name + acceptedAnswer, etc.)
  • [ ] Run — expect FAIL
  • [ ] Implement: per-domain templates in record-extractor-template.ts. Each template is a JS skeleton with {{selector}} placeholders. LLM fills placeholders given structureAnalysis. Validate output by new Function('return ' + src).
  • [ ] Tests pass
  • [ ] Commit: feat(factory): per-domain extractor generator

Task 11: Sandbox runner (DSS validation)

Files: - Create: lib/factory/sandbox-runner.ts - Test: lib/factory/__tests__/sandbox-runner.test.ts

  • [ ] Write failing test: runInSandbox(extractorSrc, sample, contentDomain) returns {records, errors}. Records pass the DSS-derived zod schema for that content domain.
  • [ ] Write failing test: when extractor produces a record missing required fields (e.g., marketing record without headline), errors includes the zod validation message and the record is excluded
  • [ ] Write failing test: when extractor throws (bad selector), error is caught and reported, no crash
  • [ ] Run — expect FAIL
  • [ ] Implement: typescript const fn = new Function('ctx', `return (${extractorSrc})(ctx)`); const $ = cheerio.load(sample.html); const helpers = { tagFromUrl, normalize, parseJsonLd, ... }; const result = fn({ $, helpers, url: sample.url, content: $.text(), headers: sample.headers }); const recordSchema = getDss(contentDomain).recordSchema; // validate each record; segregate passing vs failing
  • [ ] Tests pass
  • [ ] Commit: feat(factory): sandbox runner with DSS validation

Task 12: TS Crawler client

Files: - Create: lib/factory/crawler-client.ts - Test: lib/factory/__tests__/crawler-client.test.ts (mocked fetch)

  • [ ] Write failing test: client.createCrawler({name, config}) POSTs to /1/crawlers with Basic auth → {crawlerId}
  • [ ] Write failing test: client.patchConfig(id, patch) PATCHes /1/crawlers/{id}/config
  • [ ] Write failing test: client.crawlUrls(id, urls, save?) POSTs to /1/crawlers/{id}/urls/crawl → taskId
  • [ ] Write failing test: client.startReindex(id) POSTs to /1/crawlers/{id}/reindex → taskId
  • [ ] Write failing test: client.getStatus(id) GETs /1/crawlers/{id}
  • [ ] Write failing test: client.getStats(id) GETs /1/crawlers/{id}/stats/urls
  • [ ] Run — expect FAIL
  • [ ] Implement matching the Python CrawlerClient (packages/crawl/src/crawl/manage_seeder.py:34-102). Auth: Basic base64(userId:apiKey) from ALGOLIA_CRAWLER_USER_ID + ALGOLIA_CRAWLER_API_KEY.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): TS Algolia Crawler API client

Task 13: Index manager

Files: - Create: lib/factory/index-manager.ts - Test: lib/factory/__tests__/index-manager.test.ts

  • [ ] Write failing test: ensureIndex(contentDomain) creates the index if missing AND applies DSS-derived settings (searchableAttributes, customRanking, attributesForFaceting). Idempotent.
  • [ ] Write failing test: ensureIndex does NOT enable neuralSearch on a brand-new index (since the API rejects this without events). Logs a warning + returns {neuralPending: true}.
  • [ ] Write failing test: when index exists and settings differ from DSS, ensureIndex updates settings (PUT /settings).
  • [ ] Run — expect FAIL
  • [ ] Implement using algoliasearch client. Pulls config from DSS row.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): per-domain Algolia index manager

Task 14: Discover API endpoint (SSE, streaming)

Files: - Create: api-src/factory/discover.ts - Test: api-src/factory/__tests__/discover.test.ts

  • [ ] Write failing test: POST with {url} returns SSE stream with events:
  • {type:'log', level, message} — narrative
  • {type:'urls', count, batch} — every 1000 URLs found, persist as url_shard
  • {type:'pathGroup', group} — when path-grouper emits a group
  • {type:'classified', pathGroupId, contentDomain, confidence, method} — per pathGroup classification
  • {type:'done', sessionId} — final
  • [ ] Write failing test: discovery NEVER caps URLs — test with mocked sitemap returning 50K URLs
  • [ ] Write failing test: appendUrls is called per 1000-URL batch; appendLog per narrative event
  • [ ] Run — expect FAIL
  • [ ] Implement: validate {url} with zod, set SSE headers (mirror search.ts:220-247), call walkSitemap (async iterator), pipe to path-grouper, sample 5 URLs per group via sampler, classify, emit events, save session+url_shards+samples, emit done.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/discover SSE endpoint (streaming)

Task 15: Sample API endpoint

Files: - Create: api-src/factory/sample.ts - Test: api-src/factory/__tests__/sample.test.ts

  • [ ] Write failing test: POST {sessionId, pathGroupId, n:5} fetches 5 sample URLs from the pathGroup, persists record_type:'sample' records, returns them
  • [ ] Run — expect FAIL
  • [ ] Implement using sampler.sampleUrls. If samples already exist for the pathGroup, return cached unless ?refresh=true.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/sample endpoint

Task 16: Generate-extractor API endpoint

Files: - Create: api-src/factory/generate-extractor.ts - Test: api-src/factory/__tests__/generate-extractor.test.ts

  • [ ] Write failing test: POST {sessionId, pathGroupId, userFeedback?} reads samples + DSS row + structureAnalysis → returns {recordExtractor, crawlerConfig}
  • [ ] Write failing test: when userFeedback is provided, the extractor is regenerated with feedback prepended to the LLM prompt
  • [ ] Run — expect FAIL
  • [ ] Implement: load samples, call structure-analyzer then extractor-generator, persist to session.pathGroups[i].blueprint, return
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/generate-extractor endpoint

Task 17: Test-sandbox API endpoint

Files: - Create: api-src/factory/test-sandbox.ts - Test: api-src/factory/__tests__/test-sandbox.test.ts

  • [ ] Write failing test: POST {sessionId, pathGroupId} runs the sandbox against all samples for the pathGroup, returns {results: [{url, records, errors}]}
  • [ ] Implement using sandbox-runner against the dssEntry.recordSchema
  • [ ] Persist sandboxResults to session
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/test-sandbox endpoint

Task 18: Test-real API endpoint

Files: - Create: api-src/factory/test-real.ts - Test: api-src/factory/__tests__/test-real.test.ts

  • [ ] Write failing test: POST {sessionId, pathGroupId} creates (or reuses) a test crawler factory-test-{sessionId}-{pathGroupId}, PATCHes the extractor + a temp test index algoliacentral_factory_test, calls crawlUrls(sampleUrls), polls until done, returns extracted records
  • [ ] Implement using crawler-client. Decision: test crawler is reused per pathGroup (no reset between attempts) so the user can inspect history in the dashboard.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/test-real endpoint

Task 19: Create-crawler API endpoint

Files: - Create: api-src/factory/create-crawler.ts - Test: api-src/factory/__tests__/create-crawler.test.ts

  • [ ] Write failing test: POST {sessionId, pathGroupId} does: 1. index-manager.ensureIndex(contentDomain) 2. crawler-client.createCrawler({name: 'algoliacentral-{site-domain}-{contentDomain}-{pathGroupId}'}) 3. crawler-client.patchConfig(crawlerId, {startUrls, recordExtractor, indexName, renderJavaScript, rateLimit, pathsToMatch}) 4. crawler-client.startReindex(crawlerId) 5. blueprint-store.saveBlueprint(...) — agent-ready metadata 6. Write recordExtractor JS to crawler-configs/{site-domain}/{contentDomain}-{pathGroupId}.js 7. Update session.pathGroups[i].blueprint.crawlerId + reindexTaskId 8. Return {crawlerId, reindexTaskId, indexName, blueprintId}
  • [ ] Run — expect FAIL
  • [ ] Implement
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/create-crawler endpoint

Task 20: Crawl-progress API endpoint (SSE)

Files: - Create: api-src/factory/crawl-progress.ts - Test: api-src/factory/__tests__/crawl-progress.test.ts

  • [ ] Write failing test: GET ?crawlerId=... opens SSE; emits {type:'progress', stats} every 3s while reindex runs; emits {type:'done', stats} when reindexing===false
  • [ ] Implement: poll client.getStatus(id) + client.getStats(id). Time out after 30 minutes — emit {type:'timeout'}.
  • [ ] Tests pass
  • [ ] Commit: feat(factory): /api/factory/crawl-progress SSE

Task 21: Session + Blueprints API endpoints (CRUD)

Files: - Create: api-src/factory/session.ts, api-src/factory/blueprints.ts - Tests in matching __tests__

  • [ ] Session: GET (existing + 404), PUT (upsert)
  • [ ] Blueprints: GET list (optionally filtered by content_domain)
  • [ ] Implement
  • [ ] Commit: feat(factory): session + blueprints CRUD

Task 22: App route + page shell

Files: - Modify: src/App.tsx (add route) - Create: src/pages/AdminFactory.tsx

  • [ ] Add <Route path="/admin/factory" element={<AdminFactory />} /> above catch-all
  • [ ] Page shell: header + framer-motion fade-up + container. No logic yet.
  • [ ] Smoke: navigate to /admin/factory, see the shell
  • [ ] Commit: feat(factory): /admin/factory route + shell

Task 23: Frontend hooks

Files: - Create: src/hooks/factory/{useFactorySession.ts, useDiscoveryStream.ts, useCrawlProgress.ts} - Tests in matching __tests__

  • [ ] useFactorySession(id): TanStack Query GET /api/factory/session
  • [ ] useDiscoveryStream(): SSE consumer for /api/factory/discover. Parses log/urls/pathGroup/classified/done events, keeps separate state arrays. Pattern from useAgentStudioMaverick.ts:162-246.
  • [ ] useCrawlProgress(crawlerId): SSE consumer for /api/factory/crawl-progress
  • [ ] Tests pass
  • [ ] Commit: feat(factory): frontend hooks

Task 24: Reusable components — RollingLog, DomainBadge

Files: - Create: src/components/admin/factory/RollingLog.tsx - Create: src/components/admin/factory/DomainBadge.tsx - Tests in matching __tests__

  • [ ] RollingLog: shadcn ScrollArea, auto-scroll on new entry, level badges (info=indigo, warn=amber, error=red), monospace timestamps
  • [ ] DomainBadge: pill showing content domain + confidence percentage, color-coded per domain (marketing=indigo, support=red, education=emerald, technical=blue, customer-stories=purple, products=amber, events=pink, legal=slate, social=fuchsia)
  • [ ] Tests pass
  • [ ] Commit: feat(factory): RollingLog + DomainBadge

Task 25: DiscoveryPanel

Files: - Create: src/components/admin/factory/DiscoveryPanel.tsx

  • [ ] URL input form (react-hook-form + zod) + Submit
  • [ ] On submit: invoke useDiscoveryStream.discover(url)
  • [ ] Renders: form, status line ("Discovering... N URLs found, M categorized"), embedded RollingLog
  • [ ] Commit

Task 26: CategoryReview (path-group tree)

Files: - Create: src/components/admin/factory/CategoryReview.tsx

  • [ ] Tree layout: contentDomain (collapsible group) → pathGroups inside
  • [ ] Each pathGroup row: checkbox + path pattern + URL count + DomainBadge with confidence + 3 sample URLs (collapsed)
  • [ ] User can also re-classify a pathGroup manually (override detected domain via dropdown)
  • [ ] "Configure Selected (N)" CTA — enabled when ≥1 selected
  • [ ] Commit

Task 27: CategoryConfigurator (drawer)

Files: - Create: src/components/admin/factory/CategoryConfigurator.tsx - Create: src/components/admin/factory/SamplePreview.tsx - Create: src/components/admin/factory/ConfigEditor.tsx - Create: src/components/admin/factory/TestCrawlResults.tsx

  • [ ] Drawer (shadcn Sheet, 70% width). Title shows {contentDomain} | {pathGroup.pattern}.
  • [ ] On open: POST /sample then POST /generate-extractor
  • [ ] Layout sections (top to bottom):
  • SamplePreview: sample URLs with "view HTML" expand. Detected structure as selector ✓ field rows. Schema.org JSON-LD shown if found.
  • ConfigEditor:
    • Form: indexName (default algoliacentral_<contentDomain>, editable), renderJS toggle, rateLimit, pathsToMatch
    • Monospace textarea for recordExtractor JS
    • "Regenerate from samples" + "Free-form feedback to AI" input
  • TestCrawlResults: per-sample rows. Each row: ✓/✗ + record count + field coverage bars (per DSS-required field) + first record JSON expand. Sandbox tab and Real tab.
  • [ ] CTAs at bottom: "Run sandbox test" / "Run REAL test crawl" / "Commit + Create Crawler"
  • [ ] Commit each sub-component separately

Task 28: LiveCrawlMonitor

Files: - Create: src/components/admin/factory/LiveCrawlMonitor.tsx

  • [ ] One block per crawler created in this session, stacked vertically
  • [ ] Each block: name + crawlerId mono + DomainBadge + indexName + progress bar + URLs/sec + success/fail/ignored counters
  • [ ] Uses useCrawlProgress(crawlerId) per block
  • [ ] Commit

Task 29: CrawlerFactory orchestrator

Files: - Create: src/components/admin/factory/CrawlerFactory.tsx - Modify: src/pages/AdminFactory.tsx

  • [ ] Top-level FSM driver. Reads ?session=<id> from URL.
  • [ ] Renders DiscoveryPanel → CategoryReview → CategoryConfigurator (drawer overlay) → LiveCrawlMonitor based on session.status
  • [ ] Persistent RollingLog on the right column always visible
  • [ ] Commit

Task 30: Bundle + smoke test

  • [ ] Run node scripts/bundle-api.mjs
  • [ ] Confirm each api/factory/*.mjs bundle exists; commit api/ bundles
  • [ ] Start npm run dev:api (port 3005) + npm run dev (port 5173)
  • [ ] Manual smoke: see §9 verification
  • [ ] Commit: chore(factory): bundle + smoke test pass

Task 31: Documentation

Files: - Modify: SESSION.md - Create: docs/factory/README.md - Update vault: Projects/Algolia-Central/EngineeringPlans/CrawlerFactory.md

  • [ ] SESSION.md: mark crawler factory as live; remove old algolia-central-All_Algo_Stack priority (replaced by per-domain indices)
  • [ ] docs/factory/README.md: concept diagram, env vars, resume via ?session=<id>, troubleshooting, schema.org coverage tips
  • [ ] Vault: copy this plan + a one-page operator guide
  • [ ] Commit

8. Test strategy

8a. Unit (vitest)

  • Each lib/factory/*.ts has matching __tests__/*.test.ts
  • Each api-src/factory/*.ts has matching __tests__/*.test.ts
  • Mock external calls (fetch, algoliasearch, LLM) — no live network in unit tests
  • Target: 80%+ line coverage on lib/factory/

8b. Integration (vitest, gated by env)

  • lib/factory/__tests__/integration.test.ts: live test against Algolia Crawler API + meta-index. Skipped unless ALGOLIA_CRAWLER_USER_ID + ALGOLIA_CRAWLER_API_KEY present.
  • Tests: discover → categorize → save session → load session → real crawl_urls test against algolia.com

8c. E2E (manual smoke)

  • Task 29 is the smoke. Document in docs/factory/README.md.

8d. Regression baseline

  • Existing 407 tests must stay GREEN
  • tsc --noEmit must stay CLEAN
  • Run npm run test before each commit

9. Verification

End-to-end manual verification (the "did the spike work?" test):

  1. npm run dev:api (port 3005) + npm run dev (port 5173)
  2. Navigate to http://localhost:5173/admin/factory
  3. Type https://www.algolia.com, click Submit
  4. Watch rolling log fill — expect: - robots.txt OK → sitemap-index found → streaming sitemap walk - URL counter climbs continuously (no cap; stops only when sitemaps are exhausted) - path-grouper emits ~10–20 path groups - classifier fires per group, emits content domain + confidence - some groups go straight to a high-confidence content domain via JSON-LD; some fall through to heuristics or LLM fallback
  5. Confirm CategoryReview shows the tree: marketing, technical, customer-stories, education, etc., each with their pathGroups inside (e.g., marketing/blog, marketing/news, technical/docs, technical/integrations)
  6. Pick at least 3 (e.g., one marketing pathGroup, one technical pathGroup, one customer-stories pathGroup)
  7. Click "Configure Selected (3)" — drawer opens for the first
  8. Wait for samples + structure analysis (5–15s). Confirm: - JSON-LD detected if site exposes it (algolia.com does) - Generated extractor uses domain-correct fields (marketing → headline + articleBody; technical → name + articleBody + programmingLanguage where applicable)
  9. "Run sandbox test" — confirm records pass DSS validation; field-coverage bars show ≥80% for required fields
  10. "Run REAL test crawl" — wait ~10s, confirm records returned and visible in algoliacentral_factory_test index in dashboard
  11. "Commit + Create Crawler" — confirm:
    • Index algoliacentral_marketing (or _technical, etc.) created if missing, with DSS settings
    • Crawler appears in Algolia dashboard, name format algoliacentral-www.algolia.com-marketing-blog
    • algoliacentral_factory_blueprints has a new record
    • crawler-configs/www.algolia.com/marketing-blog.js exists in repo
  12. LiveCrawlMonitor shows reindex starting + URL counter ticking up for each crawler
  13. Repeat for the other selected pathGroups — each goes to its own domain index
  14. Hard reload → session resumes via ?session=<id>
  15. Algolia dashboard inspection:
    • algoliacentral_factory_sessions: 1 session record + N url_shard + N sample + N log records (filter by parent_id)
    • algoliacentral_factory_blueprints: 3 blueprints (one per crawler created)
    • 3 new content-domain indices, each receiving records as crawlers run
  16. Search test: query each domain index for a relevant term — confirm results match the DSS searchableAttributes

If any step fails, the spike has failed and we debug before declaring done.

10. Risks + mitigations

Risk Likelihood Impact Mitigation
new Function sandbox produces wrong records vs. real Algolia runtime Medium High Mandatory real crawl_urls test before commit catches drift
fast-xml-parser chokes on malformed sitemaps Low Medium Wrap each sitemap fetch in try/catch + emit warn-level log; continue with whatever was parsed
Vercel serverless 60s timeout for discovery on huge sites (1M+ URLs) High Medium Streaming walker + sharded persistence — discovery resumes from last shard if request times out. Re-invoking /discover with same sessionId continues where it left off (Vercel returns 504 → client retries).
LLM-generated extractor produces invalid JS Medium Medium Validate via new Function('return ' + src) before saving; "Regenerate" button on parse error
Algolia Crawler API rate limit on crawl_urls (500/24h) Low Medium Surface remaining quota in UI; require explicit "Run REAL test" click
Crawler creation succeeds but reindex queue stuck Low Medium crawl-progress endpoint times out after 30 min; user can re-trigger reindex from dashboard
Schema.org coverage low on a site (cascade falls to URL/LLM heuristics) High Medium Detection cascade has 6 layers; LLM fallback always available; UI shows confidence per pathGroup so user can override classification manually
Domain not verified in Algolia Crawler dashboard (first time crawling a new site) Medium High First createCrawler call surfaces this; we show error + deep link to dashboard's domain-verify flow; user does one-time manual step
Index proliferation (too many algoliacentral_* indices accumulate) Medium Low Listing UI in v2 (/admin/factory/indices); for v1, document the naming convention so it's auditable in dashboard
Same content classified into multiple domains (e.g., a tech blog post) Medium Low v1 picks one (highest confidence); v2 supports dual-indexing with different recordExtractors. User can manually override classification.
neuralSearch can't be enabled on freshly-created indices via API High Low Document as known limitation; dashboard step required after first crawl populates events. Index manager logs neuralPending: true.
Session storage shard count grows large for 1M URL sites (1000 url_shard records) Low Low Algolia handles this fine — all reads filter by parent_id. Old sessions can be deleted via /admin/factory/sessions (v2).

11. Build-time model handoff

⚠️ Switch model from Opus 4.7 to Sonnet 4.6 before starting Task 0.

Planning was done on Opus 4.7 (1M context). Build phase runs on Sonnet 4.6 for cost + speed. Document the switch in the first commit's session log. If a task requires deep architectural reasoning during build (e.g., resolving a TS type cycle), temporarily switch back to Opus.

12. Open trade-offs (call out at build start)

  • Wizard step persistence on hard reload: We persist sessionId in URL ?session=<id>. If the user closes the tab without that, session is unfindable from the UI. Mitigation: a /admin/factory/sessions index page listing recent sessions. Defer to v2 if not critical.
  • Concurrent users: All sessions go to the same meta-index. No user-scoped filtering yet. Acceptable since /admin is single-user during the spike.
  • Free-form feedback regeneration: When user types feedback ("the title selector is wrong, try .post-title"), we re-run generateExtractor with feedback prepended. No diff view in v1 — user just sees the new code. v2: side-by-side diff.
  • Domain verification: SESSION.md notes ADD/VERIFY domain is dashboard-only (no API). For new domains the factory cannot fully self-serve; first crawler creation for a domain may fail until the user verifies the domain in the Algolia dashboard. We surface this with a clear error message and a deep link to the dashboard.

13. Critical reference files (existing — read while building)

  • api-src/search.ts:220-339 — SSE writer pattern (mirror for discover.ts, crawl-progress.ts)
  • src/hooks/chat/useAgentStudioMaverick.ts:162-246 — SSE reader pattern (reuse parseSSELine + buffer logic)
  • lib/llm/config.ts + lib/llm/adapters/openai.ts — LLM provider factory (reuse for categorizer + extractor-generator)
  • packages/crawl/src/crawl/manage_seeder.py:34-102 — Python CrawlerClient reference for the TS port
  • packages/crawl/src/crawl/models.py — CrawlerRecord canonical schema (mirror in zod)
  • packages/crawl/config/seeder-list.json — DomainConfig structure to extend
  • src/pages/Admin.tsx — visual conventions to match
  • src/components/ui/{sheet,form,checkbox,scroll-area,progress,input}.tsx — shadcn components used
  • scripts/bundle-api.mjs — extend externals for fast-xml-parser
  • vitest.config.ts — test glob includes api-src/**/__tests__

14. Web Reality Catalog — empirical grounding

This section anchors the plan in real-world data, not assumptions. All numbers below are sourced from HTTP Archive's Web Almanac 2024 (a census of ~16M websites) and from a 14-site empirical audit conducted during planning. The cascade order, fallback strategies, and risk register all derive from these findings.

14a. Structured-data adoption (Web Almanac 2024)

Format Adoption (mobile pages) Trend
OpenGraph 64% Stable, ubiquitous on social-aware sites
RDFa 66% Declining; mostly legacy WordPress/Drupal
JSON-LD 41% Fastest growing (was 34% in 2022)
Microdata 26% Niche; mostly older WP/Drupal templates
None of the above ~37% Pure HTML, no structured data

Top JSON-LD @type values on mobile pages: - WebSite — 12.7% - Organization — 7.2% - BreadcrumbList — 5.7% - LocalBusiness — 4.0% - ItemList — 2.4% - BlogPosting — 1.4% - Product — 0.8% - Article — 0.18% - FAQPage, HowTo, Course, Recipe, Review — each <0.5%

Key implication: Even when JSON-LD is present, it usually only declares structural types (WebSite, Organization, BreadcrumbList) — not content-classifying types (BlogPosting, Article, Product). Content-class JSON-LD is on roughly 7% of pages globally. A factory that depends on JSON-LD for content-domain classification will fail on >90% of the web.

Source: https://almanac.httparchive.org/en/2024/structured-data

14b. Sitemap & robots.txt adoption (Web Almanac 2024)

  • robots.txt presence: 83.9% of sites (mobile)
  • Sitemap protocol limits: 50,000 URLs OR 50 MB per file (uncompressed). Gzip permitted.
  • Sitemap-index nesting: Per spec, only ONE level of nesting is allowed — a sitemap-index can point to urlsets, but not to other sitemap-indexes. (Some sites violate this; treat as best-effort with cycle prevention.)
  • Sitemap discoverability: Web Almanac 2024 doesn't publish %, but inferred to be < 60% based on robots.txt-vs-sitemap correlation.

Plan impact: The walker must handle gzipped sitemaps (.xml.gz) natively. The plan's /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml fallback list is correct. <lastmod> is reliable on enterprise/government sites; <changefreq> and <priority> are nearly useless (rarely populated).

Sources: https://almanac.httparchive.org/en/2024/seo, https://www.sitemaps.org/protocol.html

14c. Semantic HTML adoption (Web Almanac 2024)

Element Adoption
<h1> ~70%
<h2> ~71%
<h3> ~59%
<article> NOT REPORTED (estimated <10%; 14-site sample showed ~5%)
<main> NOT REPORTED (estimated <10%)
<time> with datetime NOT REPORTED (estimated <10%)

Plan impact: Semantic HTML5 elements (<article>, <main>, <time>) are unreliable signals. The plan's earlier reliance on these for heuristic classification is wrong. Strip them to layer 7 (low confidence). Use heading hierarchy (<h1>/<h2>/<h3>) which IS widely adopted.

Source: https://almanac.httparchive.org/en/2024/markup

14d. CMS market share (Web Almanac 2024)

  • 51% of all sites use a CMS (up from 46% in 2022)
  • WordPress: 35.6% of CMS sites = 18.2% of all sites
  • Wix: 2.8% / Squarespace: 1.5% / Joomla: 1.5% / Drupal: 1.2% / Shopify: ~1% / others: <1%

Plan impact: Detecting WordPress alone covers ~18% of the entire web. Adding Drupal, AEM, Shopify, Wix, Squarespace pushes coverage to ~30% via CMS fingerprint. This is more reliable than schema.org for these sites.

Per-CMS schema.org defaults: - WordPress: minimal — relies on plugins (Yoast, RankMath) for structured data. Plugin adoption ~35% of WP sites. - Shopify: ships Product + Organization + BreadcrumbList by default. - Wix/Squarespace: proprietary auto-generation, opaque, often non-standard. - AEM (Adobe): no defaults; structured data is bespoke per implementation. - Drupal: minimal defaults; site-specific.

Source: https://almanac.httparchive.org/en/2024/cms

14e. JS framework / SPA adoption (Web Almanac 2024)

  • jQuery: 74% of pages (entrenched, mostly WordPress legacy)
  • React: 10% (slight growth from 8% in 2023)
  • All modern frameworks combined: ~20% of pages

Plan impact: ~80% of pages are server-rendered or static. Cheerio handles them. ~20% need JS execution. SPA detection should err on the side of "static-first, escalate to Playwright on signals" — wasting Playwright cycles on every page is expensive.

Source: https://almanac.httparchive.org/en/2024/javascript

14f. Cross-vertical site coverage validated (~59 sites across 13 verticals)

This plan was validated against an extensive cross-section of real sites during the planning session. Each site was probed for robots.txt, sitemap structure, JSON-LD, OpenGraph, CMS signature, WAF posture, and URL conventions.

Vertical Sites validated WAF-block rate JSON-LD coverage §16 row
Enterprise B2B Salesforce, Adobe, Oracle, HP, BMW, Mercedes 6/6 unverifiable (blocked) 16c
CMS / generalist TechCrunch, WordPress.org, Shopify, Drupal.org, Wikipedia, Guardian 0/6 1/6 (Wikipedia partial) 16a, 16b, 16d, 16h
Government USA.gov, GOV.UK 0/2 0/2 16b, 16j
Multi-brand corporates LVMH, Diageo, Unilever, P&G 2/4 rare 16k
Massive scale tech Microsoft, AWS, Apple, Algolia, learn.microsoft.com 0/5 partial 16l
SaaS B2B / API-first commercetools, BigCommerce, OpenText 0/3 rare 16m
Education / online learning Coursera, Khan Academy, MIT OCW, edX 0/4 3/4 (Course schema) (sub-row of 16m)
SaaS dev tools Stripe, Notion, Atlassian, Datadog 0/4 1/4 (Datadog only) (sub-row of 16m)
Healthcare Mayo Clinic, WebMD, Cleveland Clinic 1/3 0/3 16h-extended
News / media Medium, NYT, BBC, Reuters 3/4 1/4 (Medium NewsArticle) 16a-extended
Community / social Stack Overflow, Reddit, Hacker News 2/3 0/3 16h-extended
Non-profit / docs Mozilla, Adobe Helpx 1/2 0/2 16h-extended
Manufacturing 3M, Caterpillar, Siemens 2/3 1/3 (Siemens) 16c-extended
E-commerce retail Walmart, Etsy, eBay, Wayfair, Best Buy 5/5 0/5 (verifiable) 16o
DTC retail commerce Nike, Restoration Hardware, Walgreens, Dell, LL Bean, Oriental Trading, SKIMS, Havertys 5/8 0/8 of accessible 16o
Furniture vertical (subset) Restoration Hardware, Havertys 2/2 (full WAF) unverifiable 16o (furniture sub-schema in v2)

Aggregate WAF rate: ~40% of sites tested fully WAF-block plain fetch from a Vercel-class datacenter IP. Higher concentrations in: enterprise B2B (100%), premium DTC retail (>50%), retail (Walmart/Etsy/Wayfair-class). Lower in: government (0%), open-source CMSes (0%), education with permissive Allow: / (MIT OCW), modern SaaS docs.

Aggregate JSON-LD usefulness for content classification: when content-class @types are needed (Article, Product, Course, MedicalCondition), real-world coverage is <25% across the verticals tested. Education (Course schema) is the single domain where JSON-LD is reliable enough to lead the cascade. Everywhere else: CMS-fingerprint + URL-pattern leads, JSON-LD is a confirmer/enricher.

Universal patterns (high reliability): - robots.txt: present on 95%+ - Sitemap discoverable (via robots.txt or /sitemap.xml fallback): ~70% (lower than expected; many sites WAF-block sitemaps too) - URL path conventions for content domain: ~95% (the universal floor for classification) - CMS signature detectability when CMS is present: ~90%

14g. 14-site initial audit findings (planning session — round 1)

Sites successfully audited (CMS / generalist)

TechCrunch, WordPress.org, Shopify, Drupal.org, USA.gov, GOV.UK, Wikipedia, The Guardian.

Findings: - 0 of 8 had JSON-LD on the listing/index pages tested. - Wikipedia had partial JSON-LD (WebPage, Article, ImageObject) only on actual article pages. - All 8 had robots.txt with at least one Sitemap directive. - All 8 had a discoverable sitemap (variants: top-level urlset, sitemap-index with sub-sitemaps; sub-sitemap counts ranged from 1 to 1,450). - Drupal-based sites (Drupal.org, USA.gov, GOV.UK partially) all enforced Crawl-delay: 10. - WordPress sites (TechCrunch, WordPress.org) had /wp-content/, /wp-json/, wp-block-* classes. Nearly identical signatures. - <article> tags appeared on roughly 1 in 8 sites (low). Date+author was always recoverable from text + visible page elements.

Sites blocked by WAF (enterprise B2B)

Salesforce, Adobe, Oracle, HP (partial), BMW, Mercedes-Benz.

Findings: - Plain fetch from Vercel-class IPs returned 403 / timeout / CAPTCHA on most attempts. - HP's homepage WAS reachable; revealed Adobe AEM signatures (/content/dam/, /digitnav/menu/). - Conclusion: enterprise B2B requires Playwright stealth mode for the discovery+sample phases. The Algolia Crawler itself runs from a different IP range that many enterprise WAFs already allowlist for SEO — so the production crawler usually works once configured.

Plan impact: Playwright fallback moved from v2 to v1. The factory must detect WAF blocks early (repeated 403/timeout patterns) and surface the option.

14g. Detection cascade hit rate (synthesized from above)

For an arbitrary content page in 2026:

Cumulative cascade through layer Expected coverage
1. CMS fingerprint 51%
1+2. CMS + URL pattern ~95% (URL is universal floor)
1+2+3. + JSON-LD when present ~95% (URL already covers; JSON-LD enriches confidence)
1+2+3+4. + OG ~95% with refined confidence
1+2+3+4+5+6. + microdata + date/author regex ~98%
All layers + LLM fallback 100%

Key insight: URL pattern matching is the universal floor. CMS fingerprint is the most-reliable single signal. Schema.org is a confidence booster, not a foundation. The cascade reaches >95% useful coverage WITHOUT relying on JSON-LD.

15. Future agent architecture (hub-and-spoke vision)

The Crawler Factory is the data-foundation layer for a multi-agent future. This section captures the vision so the factory's architecture (especially the algoliacentral_factory_blueprints index and the per-domain index naming) is forward-compatible.

15a. The vision

Today: Algolia Central has Maverick (sales), Elena (architect), Bruno (solutions engineer) — three agents sharing one index (algolia-central_enterprise_ledger). Routing between them is hand-coded in scripts/agent-studio/maverick-agent-config.ts.

Future: N specialist agents, each owning one content domain. They share a common orchestrator that fans questions out and synthesizes answers.

wzxhzdk:11

15b. How the factory enables this

Every successful crawler creation writes a blueprint record to algoliacentral_factory_blueprints (§4e). The blueprint contains: - The content domain (e.g., support) - The index name (e.g., algoliacentral_support) - The schema.org types in scope (so the agent knows what shapes of records to expect) - The URL paths that feed it (so the agent knows what the user's question MIGHT be about) - An agent_slot field, currently null

When we build the specialist-agent factory (a future spike, not v1), it reads algoliacentral_factory_blueprints, generates an Agent Studio config per blueprint (system prompt, tool list pointing at the right index, voice tuned to the domain), and writes the new agent's UUID back to agent_slot. The orchestrator then queries blueprints to know which specialists exist and what they cover.

15c. Orchestrator query flow (example)

User: "What's our latest product launch and what are people saying about it?"

  1. Orchestrator parses → identifies sub-questions: - "What's our latest product launch?" → marketing index - "What are people saying about it?" → social index + customer-stories index
  2. Orchestrator fans out: 3 parallel Agent Studio calls
  3. Each specialist runs its own retrieval against its tuned index
  4. Orchestrator collects responses, deduplicates evidence, synthesizes a unified answer with citations from all 3 specialists.

15d. What v1 of the factory delivers toward this future

  • Per-domain indices created and populated → ✅ data foundation ready
  • Blueprints index with agent_slot field → ✅ scaffolding hook ready
  • DSS-driven config per domain (different searchableAttributes, customRanking, facets) → ✅ each index pre-tuned for its agent's needs
  • Naming convention algoliacentral_<domain> → ✅ predictable for the future agent factory to discover

What v1 does NOT do (intentionally, future work): - Specialist agent scaffolding - Orchestrator routing logic - A2A communication protocol - Specialist agent prompt templates per domain

These are all future spikes that consume the v1 outputs.

16. Site-Type Validation Matrix — what the factory does for each site type

This is the validation matrix the user can challenge me with. For any site they name, the factory's behavior should map to one of these rows.

16a. WordPress sites (TechCrunch, WordPress.org, ~18% of the web)

Detection signal: robots.txt blocks /wp-admin/, URL paths contain /wp-content/, page HTML has wp-block-* classes, REST API at /wp-json/wp/v2/posts.

Factory behavior: 1. Walker pulls sitemap-index (TechCrunch has ~855 sub-sitemaps). 2. Path-grouper sees /{YYYY}/{MM}/{DD}/{slug}/, /category/{cat}/, /tag/{tag}/, /author/{author}/. 3. Date-based archives → marketing (NewsArticle/BlogPosting via DSS). 4. Sample fetch via cheerio (no Playwright needed; SSR). 5. Selector hints from CMS playbook: title=.entry-title, content=.entry-content, author=.byline, date=.entry-date. 6. JSON-LD MAY be present if Yoast plugin is installed (~35% of WP sites). Layer 3 confirms or skips. 7. Generated extractor uses CMS-tuned selectors as fallback when JSON-LD is missing. 8. Indexes: algoliacentral_marketing (for posts), optionally algoliacentral_support if /faq/ pathGroup detected.

Failure modes: Custom WP themes that strip entry-* classes — fall through to generic article heuristics + LLM.

16b. Drupal-based government / org sites (USA.gov, GOV.UK, Drupal.org)

Detection signal: robots.txt has Crawl-delay: 10, paths contain /themes/custom/, /sites/default/files/, /?q=search. Page HTML has region-*, node-*, field-* classes.

Factory behavior: 1. Walker honors Crawl-delay: 10 — paces sitemap fetches accordingly. 2. Path-grouper sees /agencies/, /government/, /guidance/, /news/, /forum/. 3. URL classification → marketing (news), support (faq), legal (guidance), education (training). 4. Sample fetch via cheerio (Drupal is SSR by default). 5. Selector hints: title=h1.page-title or <h1> in .region-content, content=.field--name-body. 6. JSON-LD usually absent — cascade falls through to URL+CMS+heuristic. Confidence stays high because all 3 layers agree. 7. Indexes: algoliacentral_marketing (news), algoliacentral_legal (guidance/policies), algoliacentral_support (FAQ).

Failure modes: Headless Drupal feeding a React frontend — escalate to Playwright. Detection: page returns near-empty <div id="root">.

16c. Adobe AEM enterprise sites (HP, Adobe.com, BMW, Mercedes — ~30% of large enterprise B2B)

Detection signal: URL paths /content/dam/, /content/{site}/{lang}/.... HTML has data-cmp-* attributes. Often WAF-protected.

Factory behavior: 1. Step 0 — WAF check. Plain fetch on robots.txt. If 403/timeout, escalate to Playwright stealth mode (v1 feature). If still blocked, surface UI: "This site requires WAF allowlisting. Options: (a) ask user to whitelist Algolia Crawler IPs in their WAF, (b) supply manual seed list, (c) defer to later when allowlisted." 2. Walker fetches sitemap (most AEM sites publish one). 3. Path-grouper sees /content/{site}/{lang}/{section}/... URLs. Localization prefix /us-en/, /de-de/ recognized and stripped for grouping; {section} is the meaningful segment. 4. URL classification: /products/, /solutions/ → product-catalog/marketing; /customers/, /case-studies/ → customer-stories; /support/, /help/ → support. 5. Sample fetch via Playwright (AEM is mixed SSR/CSR; some pages need JS for personalization). 6. Selector hints: title=h1 inside [data-cmp-is="title"], content=[data-cmp-is="text"] blocks. AEM uses Core Components consistently. 7. JSON-LD on enterprise AEM is hit-or-miss; usually Organization + BreadcrumbList only. Confidence boosted by CMS+URL agreement. 8. Indexes: domain-specific.

Failure modes: Personalization that varies content per visitor — capture multiple snapshots; warn user that crawled content may be a single visitor variant.

16d. Shopify e-commerce sites

Detection signal: cdn.shopify.com references, Shopify.theme global, /products/, /collections/ URL conventions, Shopify.shop cookies.

Factory behavior: 1. Walker pulls sitemap (Shopify auto-generates). 2. Path-grouper sees /products/{handle}, /collections/{handle}, /blogs/{name}/{post}. 3. URL classification: /products///collections/ → product-catalog; /blogs/ → marketing. 4. Sample fetch via cheerio (Shopify Liquid is SSR). 5. Shopify ships Product + Organization + BreadcrumbList JSON-LD by default — JSON-LD layer wins for product pages. 6. Selector hints from Shopify's standard theme conventions (Dawn theme is most common). 7. Indexes: algoliacentral_products (with Shopify-specific facets like vendor, productType, priceRange), algoliacentral_marketing (for blog).

Failure modes: Custom themes with non-standard markup → cascade falls to URL + JSON-LD only.

16e. Salesforce Experience Cloud / Lightning Web Components (Salesforce.com)

Detection signal: data-aura-*, lwc-* attrs, force.com references. Heavy SPA.

Factory behavior: 1. Step 0 — WAF + SPA check. Almost certainly both. Force Playwright path. 2. Walker fetches sitemap if available; Salesforce sites often have one for SEO. 3. Path-grouper sees /products/, /solutions/, /resources/, /customers/, /developer/. 4. Sample fetch via Playwright (mandatory). 5. Selector hints: complex; LWC components nest. Use post-render DOM (Playwright page.content()) and cheerio-parse the rendered HTML. 6. JSON-LD is hit-or-miss. URL + LLM fallback often needed. 7. Indexes: same domain mapping as AEM.

Failure modes: Authentication walls on /customers/ or /resources/ — surface in UI: "This section requires login. Skipping."

16f. Auto manufacturer sites (BMW, Mercedes — heavy multimedia + localization + AEM-like CMS)

Detection signal: Localized URLs (/en/, /de/, /us/). Heavy WebP imagery. WAF-protected. Often AEM or bespoke CMS.

Factory behavior: 1. WAF check → Playwright. 2. Walker pulls sitemap (mandatory for SEO on these sites). 3. Localization detection: walker normalizes /en-us/, /de-de/, etc. — user picks which locale to crawl (or all). 4. Path-grouper sees /vehicles/, /models/, /innovation/, /lifestyle/, /news/. 5. Vehicle pages → product-catalog (Vehicle/Product schema.org if present). 6. News/lifestyle → marketing. 7. Selector hints: AEM-style if AEM-detected; else generic semantic + class-based. 8. Indexes: algoliacentral_products (vehicles), algoliacentral_marketing (news), algoliacentral_education (innovation/technology articles).

Failure modes: Configurator/build-your-own pages — exclude entirely; not crawlable static content.

16g. Single-page applications without sitemaps (some SaaS, custom React apps)

Detection signal: No sitemap discoverable; near-empty <div id="root">; client-side routing.

Factory behavior: 1. Walker fails to find sitemap → surface in UI: "No sitemap found. Use one of: (a) manual seed list, (b) HTML link-graph crawl (v2), (c) ask user for known URL list." 2. v1 supports manual seed list — user pastes URLs into a textarea. 3. From there: Playwright sample fetch, normal classification cascade. 4. v2 will add link-graph crawling.

Failure modes: This is the worst case for v1. Documented limitation.

16h. Wikipedia / large structured-content sites

Detection signal: wikipedia.org host, /wiki/ URL pattern, mw-* HTML classes.

Factory behavior: 1. Walker discovers sitemap (Wikipedia provides Special:AllPages + sitemap aggregates). 2. Path-grouper: simple — /wiki/{title} → all educational/reference content. 3. Some pages have JSON-LD (Article, WebPage); cascade catches them. 4. Selector hints: content=#mw-content-text, title=#firstHeading. 5. Indexes: algoliacentral_education (most Wikipedia is reference/educational).

Failure modes: Edit-history pages, talk pages — exclude via pathsToMatch exclusions.

16i. Complex multi-domain corporate sites (Oracle.com)

Detection signal: Many subdomains (cloud.oracle.com, docs.oracle.com, blogs.oracle.com), deep URL hierarchy, mix of CMSes.

Factory behavior: 1. Discovery is per-host. The user enters one host at a time. 2. Each host discovered separately; the factory recognizes them as siblings via the user's session (multiple discoveries can be linked). 3. Each host's pathGroups classified independently. 4. Indexes still domain-named, not host-named — cloud.oracle.com/products/X and oracle.com/products/X both go to algoliacentral_products (the user can override).

Failure modes: Cross-subdomain content duplication — surface to user; let them choose which to crawl.

16j. Government sites with strict crawl rules (GOV.UK, .gov)

Detection signal: Crawl-delay directive, formal robots.txt, often Drupal or bespoke.

Factory behavior: 1. Walker honors Crawl-delay: 10 (or whatever specified). 2. Cheerio sampling with the same delay. 3. Per-government-section URL classification. 4. Indexes: typically algoliacentral_legal (policies, regulations), algoliacentral_marketing (announcements), algoliacentral_education (guidance docs).

Failure modes: Some sections require API keys or pagination tokens — out of scope.

16k. Multi-brand federated corporates (LVMH, Diageo, Unilever, P&G)

Detection signal: Corporate domain (lvmh.com, diageo.com) shows a "Our Brands" or "Portfolio" page; brand listings link OUT to separate domains (louisvuitton.com, johnnie-walker.com, etc.). Parent sitemap does NOT recursively include child brand URLs.

Empirical findings (§14): - Diageo's corporate sitemap has ~1,100 URLs covering corporate + flagship brand pages (Johnnie Walker, Guinness internal). Premium brands (Casamigos, Ketel One, Seedlip) link to external brand domains. - LVMH and Unilever WAF-block plain fetches but the federated pattern is industry-standard. - P&G uses a more centralized model (single us.pg.com).

Factory behavior: 1. Discovery on parent corporate domain (e.g., https://www.diageo.com): - Walker finds corporate sitemap → ~1K URLs covering corporate + flagship brand pages - Path-grouper sees /our-brands/, /news/, /careers/, /sustainability/ - Classification: marketing (news), customer-stories (case studies if present), legal (policies) 2. Brand-discovery sub-flow (NEW — v1 feature for multi-brand sites): - The factory probes the parent's brand-index page (typically at /our-brands or /portfolio or /companies) - Cheerio extracts outbound brand domain links - Surfaces them in UI: "Detected 12 external brand domains (louisvuitton.com, dior.com, ...). Each requires its own discovery session. Start one for each, or skip." - User picks; each gets a separate factory session, linked under a "tenant" parent record (new pattern, see §17 below) 3. Indexes: - Corporate parent → algoliacentral_marketing + algoliacentral_legal + algoliacentral_customers - Each brand domain → its own discovery → typically algoliacentral_products (luxury goods, alcohol products) + algoliacentral_marketing (campaigns) + sometimes algoliacentral_education (training/heritage content)

Failure modes: - WAF blocks (LVMH, Unilever) → escalate to Playwright stealth. - Brand-index page is JS-rendered → Playwright extraction. - Brand domains require auth (rare, but happens for reseller portals) → surface in UI, skip.

16l. Massive product+language silo sites (Microsoft, AWS, Apple, learn.microsoft.com)

Detection signal: Sitemap-index nesting ≥ 2 levels (root index → product index → language urlsets); URL counts in 100K–10M range; localization via /en-us/, /de-de/ paths; mature SEO infrastructure.

Empirical findings (§14): - Microsoft.com root: 20+ top-level sitemap-index references, including /store/sitemap-index.xml, /sitemaps/microsoft-365/sitemapindex.xml, etc. - learn.microsoft.com: 1000+ child sitemaps. dotnet alone has 180+ language/region combos. Total: 1M–10M URLs. - AWS: 20 regional sitemaps under /sitemaps/index/. ~50K-200K URLs. - Apple Autopush (iOS app metadata): 531 sub-sitemaps. - Microsoft 365: 64 language variants, ~100K-500K URLs. - Single urlset files exceed 10MB on these sites — streaming parser mandatory.

Factory behavior: 1. Pre-flight scope probe (§3a-bis) — surface the 10M-URL estimate IMMEDIATELY. User must pick: - Full crawl (multi-day discovery, but supported) - Sub-tree (e.g., learn.microsoft.com/dotnet/ only) — RECOMMENDED for most cases - Specific language (e.g., en-us only) 2. Walker: - Recursively follows nested sitemap-indexes (depth 2-3 confirmed) - Parallel sub-sitemap fetch (8 concurrent default; can be raised on permissive sites) - Streaming XML parser for files > 5MB - Honors any Crawl-delay 3. Path-grouper sees product-rooted paths: /azure/, /microsoft-365/, /dotnet/, /python/, /visualstudio/. Each is a candidate pathGroup. 4. Localization: walker yields localeAlternates per URL; UI shows "Detected 64 language variants. Crawl: [dropdown: All / en-us only / pick languages]" 5. Classification: /docs/, /learn/, /reference/ → technical; /blog/ → marketing; /customers/ → customer-stories. 6. Sample fetch: cheerio is sufficient for Microsoft Learn (pages are SSR). AWS and Apple may need Playwright for some pages. 7. Indexes: split by content domain. algoliacentral_technical for docs (most of the volume), algoliacentral_marketing for blog, etc. 8. Crawler creation: each pathGroup × locale combination COULD be its own crawler. v1 keeps it simple — one crawler per pathGroup writes to one index for ALL locales (with language_code as a facet). User can split per-locale in v2 if needed.

Failure modes: - 60-second Vercel function timeout on huge discoveries → resume from last shard (§10 risk row) - Single urlset > 50MB (rare, beyond spec) → emit warning, skip that one urlset, continue - Rate-limited mid-crawl → walker pauses, user notified

16m. SaaS B2B / API-first (commercetools, BigCommerce, OpenText)

Detection signal: Modern sitemap with content-type splits (e.g., BigCommerce has 14 separate content-type sitemaps). Often have public docs subdomain, blog subdomain, customer portal subdomain.

Empirical findings (§14): - commercetools: ~1K URLs in a flat urlset. /products/, /docs/, /blog/, /customers/, /why/. - BigCommerce: 14 sitemaps split by content type. Each is a flat urlset. ~3-5K URLs total.

Factory behavior: 1. Walker pulls all top-level sitemaps (sometimes one per content type — easy mapping to pathGroups). 2. For BigCommerce-like sites: each content-type sitemap = one pathGroup directly. Skip the heuristic grouping. 3. Sample fetch: cheerio (these are SSR). 4. Classification: usually high-confidence URL+CMS match (/blog/ → marketing, /docs/ → technical, /customers/ → customer-stories). 5. Indexes: standard domain-split.

Failure modes: Subdomain proliferation (docs.commercetools.com, blog.commercetools.com) — user runs separate discoveries per subdomain.

16o. DTC retail commerce (Nike, RH, Walgreens, Dell, LL Bean, Oriental Trading, SKIMS)

Detection signal: product-detail URL patterns (/products/, /p/, /t/, /itm/, /catalog/product/, /store/c/); commerce-specific paths (/cart, /checkout); image CDNs (imgix.net, custom CDNs); often premium WAF.

Empirical findings (§14): - 0 of 7 use off-the-shelf platforms. No Shopify, no Salesforce Commerce Cloud, no Adobe Commerce. All custom proprietary. - 0 of 7 expose JSON-LD Product schema on the actual product detail pages. This invalidates the schema.org-first product strategy that worked for Shopify-pattern sites. - 4 of 7 fully WAF-block from datacenter IPs (Nike, RH, Walgreens, Dell). WAF density on premium DTC matches enterprise B2B. - Variant strategies vary radically: SKIMS uses separate URL per color (/products/bra-clay, /products/bra-onyx); LL Bean keeps variants on one URL with AJAX; Oriental Trading encodes filter state in URL slug (a1-ID+filter+count.fltr). No canonical-tag consensus. - Reviews always lazy-loaded. Stock/availability rarely exposed in initial HTML. Prices baked in HTML on most, dynamic on some.

Factory behavior: 1. WAF check first. If plain fetch returns 403 → escalate to Playwright stealth (v1 capability per §3d). 4-of-7 DTC sites need this. 2. Sitemap walk — most expose sitemaps (LL Bean: 1,251 URLs, SKIMS: 500+ products + collections + blog). Some hide them behind WAF (Nike, RH); use Playwright to fetch. 3. Path-grouper sees: - /products/ or /t/ or /p/ → product-catalog - /collections/ or /c/ → collection (also product-catalog with different schema) - /blog/, /journal/, /magazine/, /style-guide/ → marketing - /customer-service/, /help/, /returns/, /shipping/ → support - /about/, /our-story/, /sustainability/ → marketing (corporate) 4. DOM-extraction first, schema.org as bonus. The DSS product-catalog row's recordExtractor template prioritizes: - Selectors-based: <h1> for name, [class*="price"] or microdata fallback for price, [class*="image"] img[src] for images, <select> or [class*="variant"] for variants - Strip variant suffix from URL for canonical (heuristic: trim trailing -{color} segments matching a known palette) - Defer reviews to XHR (Playwright page.waitForResponse or skip in v1) 5. Variant deduplication strategy: the factory's recordExtractor for product-catalog content domain SHOULD include logic to: - Detect canonical tag (<link rel="canonical">) and use it as the deduplication key - If absent, use a heuristic: strip last URL path segment if it matches a known color/size suffix pattern - Same name + brand + parent-canonical → same product object; variants stored as variants: [{color, size, sku, image, available}] array 6. Indexes: all 7 → algoliacentral_products, with rich product fields (specs, materials, dimensions, variants, reviews-count, price, image_count). Some also produce algoliacentral_marketing (blog/journal) and algoliacentral_support (returns/shipping policies).

Failure modes: - WAF blocks even Playwright (Cloudflare Bot Management, Akamai Bot Manager) — surface to user with "use Algolia Crawler IP allowlist or supply manual seeds" path. - Reviews/specs that are entirely XHR-driven — capture via Playwright page.waitForLoadState('networkidle') plus selector waits, but accept partial data on timeout. - Sites that geo-redirect — factory honors the user's stated locale or defaults to the IP-detected one. v2 supports per-locale crawler creation. - No canonical tag → variant explosion. v1 uses URL-suffix heuristic; v2 adds LLM-based canonical inference.

Furniture retail specifically (Restoration Hardware, Havertys, Crate & Barrel pattern): the user's mental model is "tell me about that sofa I saw." Empirical (planning session): both RH and Havertys 403-block plain fetches from datacenter IPs — Playwright stealth mandatory just to read the homepage. Once content is reachable, the recordExtractor must capture: - Dimensions (overall H×W×D, seat depth, seat height, arm height, leg height) — usually in a "Dimensions" or "Specs" tab/section - Materials — frame (kiln-dried hardwood vs particleboard), fill (down vs polyfill vs foam density), springs (8-way hand-tied vs sinuous) - Fabric / leather options — RH alone has 300+ fabric SKUs per sofa; capture as a variant array with name, color family, content (linen/cotton/blend), durability rating - Lead times — "ships in 4–6 weeks" or "made-to-order, 12–14 weeks" - Care instructions — washable, dry clean, leather conditioner, etc. - Price tiers — RH has membership pricing (Members vs non-Members) plus fabric upcharges - Configurability — sectional pieces snap together; capture configurator state space if extractable These are domain-specific extensions to the product-catalog schema. v1 captures whatever DSS+DOM finds; v2 adds a furniture sub-schema (extending product-catalog) with dimension parsing, material taxonomy, fabric variant arrays, lead-time normalization. The pattern of "specialized sub-schema per industry vertical" extends to other DTC categories: apparel (sizes/fits/wash/composition), electronics (specs sheets/compatibility), beauty (ingredients/skin types/concerns), home goods (room fits, assembly). DSS rows can be subclassed via composition.

16p. Validation challenge protocol

For any site the user names, before declaring the plan validated, walk through: 1. Match a row. Which §16 row does this site fit? If none — add a new row, don't paper over. 2. Sitemap structure. Run a scope probe: nesting depth, sub-sitemap count, URL estimate, language splits. 3. Brand federation. Is this a multi-brand corporate? If yes, plan brand-domain enumeration. 4. WAF check. Plain fetch on robots.txt — does it return 200, 403, or timeout? 5. CMS fingerprint. What CMS does this site run? If CMS detected, expect 0.85+ confidence on classification. 6. JSON-LD coverage on samples. What % of sample pages have JSON-LD with content-classifying types? 7. Per-pathGroup classification confidence. Walk through the cascade for 3 sample pathGroups. 8. Index count expected. How many algoliacentral_<domain> indices will this site populate?

If steps 1-8 all answer cleanly, the site is supported. If any step has "I don't know" — the plan needs work before it ships.

17. Multi-tenant + multi-domain orchestration (v1.5)

The user's vision includes scenarios where one logical entity (LVMH) has many brands (Louis Vuitton, Dior, Céline, ...), each on its own domain, each with its own crawl. The factory needs a "tenant" abstraction so these are managed together but crawled independently.

17a. Tenant model

wzxhzdk:12

Each FactorySession produces N crawlers. Each crawler writes to a domain-specific Algolia index. Indices CAN be shared across tenant brands (e.g., all LVMH brand-marketing → algoliacentral_marketing with tenant_id as a facet).

17b. Algolia meta-index for tenants (NEW)

wzxhzdk:13

Tenant creation is OPTIONAL in v1 — single-domain workflows skip it. The "Discover brands" CTA on a parent corporate session creates a tenant and links sub-sessions.

17c. v1 vs v2 split for multi-tenant

v1 (this spike): - Single-session, single-domain workflow works end-to-end - "Detect brand domains" sub-flow on multi-brand parent: scrapes brand-index page, lists external domains, lets user start ONE brand session at a time (manually triggered) - Tenant record created automatically when 2+ sessions share a parent

v2 (deferred): - Bulk brand crawl: "crawl all 12 brands in parallel" - Cross-tenant search routing for the orchestrator - Tenant-level dashboards (combined progress, combined record counts)

18. Data the factory exposes for downstream automation

Every artifact the factory produces is queryable via Algolia indices. This makes it consumable by future automation (the SOP-driven engineering spec generator, the agent factory, evaluation pipelines, etc.):

Index Purpose Consumed by
algoliacentral_factory_sessions Session state (sharded: session, url_shard, sample, log records) Factory wizard UI, audit trails
algoliacentral_factory_blueprints Crawler blueprints (agent-ready metadata) Future specialist-agent factory
algoliacentral_factory_tenants Multi-tenant grouping Future cross-brand orchestration
algoliacentral_<domain> (×N) Actual content indices, one per content domain Specialist agents, current chat agents

The factory writes; everything else reads. No tight coupling. Each downstream automation reads from a stable, queryable contract.

19. Methodology — the operating protocol

This section distills the validation research (§14, §16) into the deterministic protocol the factory follows on any new site. It is meant to be machine-executable: an engineer or SOP-bot reading this can implement each step without re-deriving the rationale.

19a. First-touch protocol (opening 30 seconds, any new site)

wzxhzdk:14

The first 30 seconds answer: can we crawl this site, how big is it, what is its language structure, and which transport (cheerio vs Playwright) do we use for sampling?

19b. WAF response ladder (5-level escalation)

Level Trigger Action Cost
0 Plain fetch works Cheerio sampling, sequential 8 RPS $
1 Sporadic 429 Add 0.5–2s jitter; back off on 429 $
2 Persistent 403 from datacenter IP Escalate to Playwright stealth (chromium, real viewport, Accept-Language) $$
3 Playwright also 403 Surface UI: "Site enforces Bot Manager. Options: (a) Algolia Crawler IPs likely allowlisted by site SEO team — proceed with crawler creation; pre-creation discovery WILL fail. (b) Manual seed list. (c) Cancel." manual
4 All above fail Document as "uncrawlable from factory infra"; user must add seeds manually OR escalate to enterprise procurement of residential proxy infra (out of scope for this spike) manual

Empirical gate: Levels 2–3 fire on ~40% of enterprise B2B sites and ~50% of premium DTC retail (per §14 audit). Levels 0–1 cover ~95% of CMS, government, education, generalist content sites.

19c. Detection cascade — execution rules

Per §3b's 8-layer cascade, with concrete halting and aggregation rules:

wzxhzdk:15

19d. Per-site-type recognition rubric (real-time)

When the factory probes a site, it tries to assign it to a §16 row within the first 30 seconds, so it can apply the right playbook. Recognition signals (in order checked):

wzxhzdk:16

This rubric is data, not code — it lives in lib/factory/site-type-rubric.ts as an ordered list of {predicate, siteType} rules. Easy to extend.

19e. Confidence scoring & user-in-the-loop policy

The factory does NOT auto-commit anything. The user always sees: 1. Detected site type + confidence (from §19d) 2. Per-pathGroup content domain + cascade contributors + confidence (from §19c) 3. Generated recordExtractor with editable code box 4. Sandbox test results (records the extractor would produce) 5. Real test crawl results (records Algolia's crawler actually produces)

Auto-accept thresholds are RECOMMENDATIONS, never silent commits. Every CTA ("Commit + Create Crawler") requires explicit user click. This is by design — the factory's value is human-in-the-loop quality, not full automation.

19f. Methodology principles (codified)

The protocol above derives from these principles, which the implementation must respect:

  1. Empirical over theoretical. Every default in the plan is grounded in §14's data. If a future change contradicts the data, it must include new data, not opinion.
  2. CMS+URL first; schema.org enriches, doesn't lead. Reversing this caused the v0 plan's 60% blind spot.
  3. No caps on URL count, sitemap depth, or page count. The walker is streaming + sharded; the system handles 10M as easily as 5K.
  4. WAF is a first-class concern, not an edge case. 40% of sites will hit it. Playwright is in v1.
  5. Every artifact is queryable from Algolia. Sessions, blueprints, tenants, and content indices are all Algolia indices. No external state.
  6. Per-content-domain indices. Multi-index hub-and-spoke is the architectural commitment.
  7. DSS is data, not code. Adding a new content domain or sub-schema is a configuration change, not a code release.
  8. Confidence is shown to the user, never hidden. The user sees why the factory thinks what it thinks.
  9. Resumability everywhere. Vercel function times out → session resumes from last shard. User closes tab → URL ?session=<id> resumes. No "lost work."
  10. Forward-compatible with the agent factory. Blueprints index has agent_slot field today; specialist-agent factory will populate it without schema migration.

20. Plan-update summary — what the validation research changed

Mapping each empirical finding to the specific plan section it modified. This is the audit trail for "why is the plan the way it is."

Finding Source Old assumption Plan section updated
JSON-LD adoption is 41%, not 70%; only ~7% useful @types Web Almanac 2024 §14a "JSON-LD first cascade" §3b cascade reordered to CMS-first; §4b table; §19c
Real content pages (TechCrunch, GOV.UK, Drupal.org) lack JSON-LD 26-site empirical §14g Same §3b, §4b, §19c
Sitemap nesting reaches 2-3 levels (Microsoft) §14 audit "1 level per spec" §3a §3, §16l, §19a
Sites with 1M-10M URLs exist (learn.microsoft.com) §14 audit "Cap at 5K v1" All caps removed; §3a §4-5; §10 risks; §19a
Single urlset files exceed 10MB §14 audit "Default fast-xml-parser dom" §3a §6 streaming SAX parser when >5MB
~40% of sites WAF-block plain fetch 60-site audit "Playwright in v2" §3d; §6 v1 includes Playwright; §19b ladder
Multi-brand corporates (LVMH, Diageo) are federated §14 audit "Single discovery per site" §16k brand-domain enumeration sub-flow; §17 multi-tenant
51% of sites use a CMS; WordPress alone = 18% Web Almanac 2024 §14d "CMS detection nice-to-have" Promoted to layer-1 of cascade; §19d rubric
Crawl-delay: 10 is a Drupal signature §14 audit Generic respect §19d explicit Drupal trigger
<article> tag adoption ~5% Web Almanac 2024 §14c "Use semantic HTML" Demoted to layer 7 (low confidence)
99% of articles have date+author in text §14 audit + heuristics Not in cascade Added as layer 6 (universal article-family fallback)
Education sites consistently use Course schema.org §14f DSS theoretical DSS row validated; §16 sub-row; education's the one vertical where schema-first works
DTC retail uses 100% custom platforms (no Shopify) DTC audit "Shopify for product schema" §16o DOM-extraction-first for product pages; furniture sub-schema in v2
DTC has high WAF (Nike, RH, Walgreens, Dell, Havertys all 403) DTC audit Same as enterprise §16o + §19b ladder
RH/Havertys-class furniture has rich domain data (dimensions, materials, fabrics) DTC audit Generic product schema §16o furniture sub-schema notes; v2 vertical sub-schemas
Reviews are universally lazy-loaded on DTC DTC audit Static HTML extraction §16o defer reviews; v2 add XHR capture
Color/size variants strategy varies (URL vs state) DTC audit One pattern §16o canonical inference + variant deduplication
News sites WAF-block but Medium accessible with NewsArticle schema §14 audit Generic article handling §16a-extended note
Healthcare sites don't expose MedicalCondition schema §14 audit Add health DSS row DEFERRED — health maps to education until evidence emerges
Q&A/community sites avoid structured metadata (Hacker News confirmed) §14 audit Q&A → support DSS §16h-extended; v2 may add community row
Microsoft sitemap structure has 64+ language variants §14 audit Single-locale crawler §16l + locale picker UI; v2 per-locale crawlers
Site-type detection should drive playbook selection Cross-cutting Implicit §19d rubric — explicit ordered predicates

20a. Tasks added or strengthened by validation

These task changes (relative to my v0 task list) were forced by the findings:

  • NEW Task A: lib/factory/cms-detector.ts — the CMS fingerprint library (WordPress, Drupal, AEM, Shopify, Hubspot, Webflow, Squarespace, Wix, Salesforce LWC, custom-by-elimination). Drives layer 1 of the cascade.
  • NEW Task B: lib/factory/waf-detector.ts + lib/factory/playwright-fetcher.ts — WAF probe and stealth-mode fallback. Used by sampler when plain fetch 403s.
  • NEW Task C: lib/factory/site-type-rubric.ts — the §19d rubric as ordered predicates.
  • NEW Task D: lib/factory/canonical-inferer.ts — for DTC retail variant deduplication. Heuristic + LLM fallback.
  • NEW Task E: lib/factory/scope-estimator.ts — pre-flight scope probe (§3a-bis).
  • NEW Task F: lib/factory/brand-domain-discoverer.ts — for multi-brand corporates. Scrapes brand-index page (/our-brands etc.), extracts outbound brand domains.
  • STRENGTHENED Task 5 (classifier): explicit 8-layer cascade with all halting and aggregation rules from §19c.
  • STRENGTHENED Task 9 (structure-analyzer): CMS-aware selector libraries per CMS detected, NOT pure LLM.
  • STRENGTHENED Task 8 (sampler): WAF detection + Playwright escalation built in.

These task additions will be integrated into §7 in the next plan revision (after this section is approved). For now the additions are documented here so nothing is lost during the freeze → SOPs → engineering specs flow.

20b. v1 scope adjustments forced by findings

  • ✅ Playwright sampling — moved from v2 to v1 (was: defer; now: required for 40% of sites)
  • ✅ CMS detector library — added to v1 (was: implicit in classifier; now: standalone module)
  • ✅ Canonical inferer — added to v1 (was: not in plan; now: required for DTC retail)
  • ✅ Brand-domain discoverer — added to v1 (was: v2; now: light v1 scope — read brand-index page, list domains, link sessions)
  • ✅ Scope estimator — added to v1 (was: not in plan; now: required for Microsoft-class sites)
  • ⏸ Per-locale crawlers — stays v2 (one crawler per pathGroup; locale as facet)
  • ⏸ Bulk multi-brand parallel crawl — stays v2 (one brand at a time in v1)
  • ⏸ Vertical sub-schemas (furniture, apparel, electronics) — stays v2 (v1 captures generic product-catalog only)
  • ⏸ Health DSS row — stays v2/deferred (no schema.org evidence to validate)
  • ⏸ Community DSS row — stays v2/deferred (no structured metadata in real samples)

20c. Plan freeze readiness checklist

Concern Status
Architecture documented end-to-end ✅ §2
Data model with zod schemas ✅ §4
UI specs with mockups ✅ §5
Tasks with TDD bite-sized steps ✅ §7 (with §20a additions to integrate)
Test strategy ✅ §8
Verification protocol ✅ §9
Risks + mitigations ✅ §10
Empirical grounding (Web Almanac + 60-site audit) ✅ §14
Future agent architecture ✅ §15
Per-site-type playbooks (16 rows) ✅ §16
Multi-tenant abstraction ✅ §17
Methodology / operating protocol ✅ §19
Plan-update audit trail ✅ §20
Decisions locked ✅ §1 (9 decisions); §6 phasing
Build-time model handoff ✅ §11
Critical reference files ✅ §13

The plan is ready to be frozen and committed.

20d. Vault commit destination — NEW project: "Crawler Factory"

User instruction (2026-04-30): create a NEW vault project called Crawler Factory and put all knowledge under it. Local disk is ephemeral; vault is the source of truth across machines and sessions.

Vault root: ~/Library/CloudStorage/GoogleDrive-arijit.chowdhury@algolia.com/My Drive/AI-Docs/Obsidian/ArijitOS-Brain/

New vault project structure to create:

wzxhzdk:17

Local mirror in repo (not authoritative — for local dev convenience):

wzxhzdk:18

Memory pointers to add (cross-session): - project_crawler_factory_plan.md — points to vault Projects/Crawler-Factory/00-Plan.md - Updates the existing project_web_reality_research_findings.md to link the vault Research/ folder

Workflow after vault commit: 1. Plan is frozen at vault Projects/Crawler-Factory/00-Plan.md 2. Apply CodingSOPs / TestingSOPs / WritingSOPs (logging guidance lives inside CodingSOPs) to generate engineering specs into Engineering-Specs/ 3. Switch model to Sonnet 4.6 4. Multi-agent execution reads Engineering-Specs and produces code

This is the explicit handoff to the next phase. Plan signed off → vault → SOPs → specs → agent factory.