Methodology — Crawler Factory Operating Protocol

Extracted from 00-Plan.md §19. This is the deterministic protocol the factory follows on any new site. Machine-executable: an SOP-bot or engineer reading this can implement each step without re-deriving the rationale.

Source of truth: 00-Plan.md §19 (this is a navigation extract; if they ever drift, the master plan wins).

19. Methodology — the operating protocol

This section distills the validation research (§14, §16) into the deterministic protocol the factory follows on any new site. It is meant to be machine-executable: an engineer or SOP-bot reading this can implement each step without re-deriving the rationale.

19a. First-touch protocol (opening 30 seconds, any new site)

1. Normalize input → root URL (https + apex or www).
2. fetch(/robots.txt) with timeout=8s, AlgoliaCentralFactory/1.0 user-agent.
   ├─ 200 → parse Sitemap: directives + Crawl-delay + bot-blocking patterns
   ├─ 403/429/timeout → mark site as WAF-suspected; jump to (5) Playwright probe
   └─ 404 → no robots.txt; try fallback sitemaps directly
3. fetch(top sitemap) with same constraints.
   ├─ Parse with fast-xml-parser; classify as urlset OR sitemapindex
   ├─ If sitemapindex → enumerate sub-sitemaps (do NOT fetch all yet)
   ├─ Compute scope estimate: sub_count × avg_urls_per_sub (default 5000)
   └─ Detect language splits via URL pattern + hreflang inspection on first leaf urlset
4. Pre-flight scope report to UI:
   "Site: <domain>. Sitemap depth: <N>. Sub-sitemaps: <count>.
    Estimated URLs: <range>. Languages detected: <list>.
    Estimated discovery time at <concurrency>: <range>.
    Proceed with full crawl / sub-tree only / cancel?"
5. WAF probe (if step 2 or 3 failed): Playwright stealth (headless=true) fetches /robots.txt.
   ├─ 200 with Playwright but 403 with fetch → set domain.transport = 'playwright'
   ├─ 403 even with Playwright → escalate user-side (manual seed list OR Algolia Crawler IP allowlist)
   └─ Document outcome in session log
6. Persist session record with status='discovering' BEFORE any crawl work — so a Vercel timeout doesn't lose context.

The first 30 seconds answer: can we crawl this site, how big is it, what is its language structure, and which transport (cheerio vs Playwright) do we use for sampling?

19b. WAF response ladder (5-level escalation)

Level	Trigger	Action	Cost
0	Plain fetch works	Cheerio sampling, sequential 8 RPS	$
1	Sporadic 429	Add 0.5–2s jitter; back off on 429	$
2	Persistent 403 from datacenter IP	Escalate to Playwright stealth (chromium, real viewport, Accept-Language)	$$
3	Playwright also 403	Surface UI: "Site enforces Bot Manager. Options: (a) Algolia Crawler IPs likely allowlisted by site SEO team — proceed with crawler creation; pre-creation discovery WILL fail. (b) Manual seed list. (c) Cancel."	manual
4	All above fail	Document as "uncrawlable from factory infra"; user must add seeds manually OR escalate to enterprise procurement of residential proxy infra (out of scope for this spike)	manual

Empirical gate: Levels 2–3 fire on ~40% of enterprise B2B sites and ~50% of premium DTC retail (per §14 audit). Levels 0–1 cover ~95% of CMS, government, education, generalist content sites.

19c. Detection cascade — execution rules

Per §3b's 8-layer cascade, with concrete halting and aggregation rules:

RUN every applicable layer concurrently on a 5-page stratified sample per pathGroup:
  Layer 1: CMS fingerprint   → returns {domain, conf} or null
  Layer 2: URL pattern        → ALWAYS returns {domain, conf} — universal floor
  Layer 3: JSON-LD            → returns {domain, conf} or null per page; aggregate by majority
  Layer 4: OpenGraph          → returns {domain, conf} or null per page
  Layer 5: Microdata + RDFa   → returns {domain, conf} or null per page
  Layer 6: Date + author rgx  → returns 'article-family' if matches, else null
  Layer 7: Semantic HTML      → returns hint or null
  Layer 8: LLM (cheap)        → only fired if combined ≤7 confidence < 0.55

AGGREGATE:
  candidates = group results by content_domain
  for each candidate:
    base_conf = max(layer.conf for layers voting this candidate)
    agreement_boost = 0.05 × (count_distinct_layers_agreeing − 1)
    final_conf = min(0.95, base_conf + agreement_boost)
  pick top candidate by final_conf

UI traffic light:
  ≥ 0.85: green — auto-accept, recommend Configure
  0.65–0.84: yellow — show contributing layers, recommend manual review
  < 0.65: amber — manual override required (LLM tiebreak shown for context)

19d. Per-site-type recognition rubric (real-time)

When the factory probes a site, it tries to assign it to a §16 row within the first 30 seconds, so it can apply the right playbook. Recognition signals (in order checked):

IF robots.txt has Crawl-delay: 10              → Drupal-class (§16b)
ELIF /wp-content/ in any sample URL             → WordPress (§16a)
ELIF /content/dam/ OR data-cmp-* attrs          → AEM enterprise (§16c)
ELIF cdn.shopify.com OR Shopify.shop cookie     → Shopify (§16d)
ELIF data-aura-* OR lwc-* attrs                 → Salesforce LWC (§16e)
ELIF mw-* classes + /wiki/ paths                 → Wikipedia/MediaWiki (§16h)
ELIF top-level /our-brands/ AND outbound brand
     domain links detected on brand-index page  → Multi-brand corporate (§16k)
ELIF sitemapindex depth ≥2 + ≥100 sub-sitemaps  → Massive product+language silo (§16l)
ELIF /listing/ OR /products/ on apex DTC retail  → DTC retail (§16o)
ELIF .gov host                                  → Government (§16j)
ELIF /docs/ subdomain + Markdown URL pattern    → SaaS B2B docs (§16m)
ELIF empty <div id="root"> + no SSR content     → SPA (§16g — manual seeds)
ELSE                                             → Custom; rely on universal cascade

This rubric is data, not code — it lives in lib/factory/site-type-rubric.ts as an ordered list of {predicate, siteType} rules. Easy to extend.

19e. Confidence scoring & user-in-the-loop policy

The factory does NOT auto-commit anything. The user always sees: 1. Detected site type + confidence (from §19d) 2. Per-pathGroup content domain + cascade contributors + confidence (from §19c) 3. Generated recordExtractor with editable code box 4. Sandbox test results (records the extractor would produce) 5. Real test crawl results (records Algolia's crawler actually produces)

Auto-accept thresholds are RECOMMENDATIONS, never silent commits. Every CTA ("Commit + Create Crawler") requires explicit user click. This is by design — the factory's value is human-in-the-loop quality, not full automation.

19f. Methodology principles (codified)

The protocol above derives from these principles, which the implementation must respect:

1. Empirical over theoretical. Every default in the plan is grounded in §14's data. If a future change contradicts the data, it must include new data, not opinion.
2. CMS+URL first; schema.org enriches, doesn't lead. Reversing this caused the v0 plan's 60% blind spot.
3. No caps on URL count, sitemap depth, or page count. The walker is streaming + sharded; the system handles 10M as easily as 5K.
4. WAF is a first-class concern, not an edge case. 40% of sites will hit it. Playwright is in v1.
5. Every artifact is queryable from Algolia. Sessions, blueprints, tenants, and content indices are all Algolia indices. No external state.
6. Per-content-domain indices. Multi-index hub-and-spoke is the architectural commitment.
7. DSS is data, not code. Adding a new content domain or sub-schema is a configuration change, not a code release.
8. Confidence is shown to the user, never hidden. The user sees why the factory thinks what it thinks.
9. Resumability everywhere. Vercel function times out → session resumes from last shard. User closes tab → URL ?session=<id> resumes. No "lost work."
10. Forward-compatible with the agent factory. Blueprints index has agent_slot field today; specialist-agent factory will populate it without schema migration.

---