Crawler-Factory

Research/Index.md

Research Record — Crawler Factory Validation

All empirical research conducted during planning, summarized round by round. The findings are baked into ../00-Plan.md §14 (Web Reality Catalog) and §16 (Site-Type Validation Matrix). This folder preserves the research chronology for audit purposes.

Source of truth: ../00-Plan.md §14 has the consolidated empirical record with citations. This folder is the per-round chronology.

Round 1 (2026-04-30) — Enterprise B2B + CMS/Generalist + Authoritative References

Goal: Establish baseline empirical reality (vs. theoretical assumptions).

Sites probed: - Enterprise B2B (mostly WAF-blocked): Salesforce, Adobe, Oracle, HP, BMW, Mercedes - CMS / generalist (mostly accessible): TechCrunch, WordPress.org, Shopify, Drupal.org, USA.gov, GOV.UK, Wikipedia, The Guardian - Authoritative refs: Web Almanac 2024 (structured-data, SEO, markup, JavaScript, CMS chapters), sitemap protocol spec, Google Search Central

Critical findings: - JSON-LD adoption: 41% of pages globally (Web Almanac), only ~7% with content-class @types. NOT 70% as the v0 plan assumed. - Real content pages I tested (TechCrunch, GOV.UK, USA.gov, Drupal.org, WordPress.org) had zero JSON-LD on listing/index pages. Wikipedia partial only. - 6 of 6 enterprise B2B sites WAF-blocked plain fetch from datacenter IPs. - Drupal signature: Crawl-delay: 10 is universal across Drupal sites. - WordPress = 18% of all sites (single most reliable CMS detection target).

Plan impact: Reordered detection cascade to CMS-first; demoted JSON-LD to layer 3.

Cited sources: - https://almanac.httparchive.org/en/2024/structured-data - https://almanac.httparchive.org/en/2024/seo - https://almanac.httparchive.org/en/2024/markup - https://almanac.httparchive.org/en/2024/javascript - https://almanac.httparchive.org/en/2024/cms - https://www.sitemaps.org/protocol.html

Round 2 (2026-04-30) — Multi-Brand + Massive Scale + SaaS B2B

Goal: Validate against the most complex websites — multi-brand corporates and massive-scale tech.

Sites probed: - Multi-brand: LVMH, Diageo, Unilever, P&G - Massive scale tech: Microsoft, learn.microsoft.com, AWS, Apple, Algolia (reference) - SaaS B2B: commercetools, BigCommerce, OpenText

Critical findings: - Sitemap nesting reaches depth 2-3 in reality (Microsoft) despite the spec saying 1 max. - learn.microsoft.com: 1M–10M URLs, 1000+ sitemaps. dotnet alone has 180+ language/region combos. - Multi-brand corporates use federated architecture: parent corporate site sitemap does NOT include child brand URLs. LVMH/Louis Vuitton, Diageo/Casamigos = separate domains, separate sitemaps. - Microsoft 365 sitemap: 64 language variants. - Single urlset files exceed 10MB (dotnet_en-us_1.xml) — streaming parser needed. - LVMH and Unilever WAF-block plain fetches. Algolia, Microsoft, AWS, Apple, Diageo, P&G accessible.

Plan impact: Removed all URL/depth caps; added streaming SAX parser for >5MB files; added §17 multi-tenant abstraction; parallel sub-sitemap fetching with 8 concurrent default.

Round 3 (2026-04-30) — Commerce + Manufacturing

Goal: Validate retail e-commerce and manufacturing patterns.

Sites probed: Walmart, Etsy, eBay, Wayfair, Best Buy, 3M, Caterpillar, Siemens.

Critical findings: - 6 of 8 sites WAF-blocked or timed out (Walmart CAPTCHA on sitemap, Etsy 403, Wayfair 429, eBay/3M/Caterpillar timeouts). - Etsy has no Sitemap directives at all in robots.txt — must use HTML link discovery. - Siemens sitemap accessible: sitemapindex with 38 regional variants (en-us, de-de, ja-jp, etc.). AEM-style content. - Walmart, Wayfair, Best Buy, 3M: Akamai/Cloudflare/Imperva-class WAF.

Plan impact: Confirmed WAF density on retail; reinforced Playwright fallback in v1; documented absence-of-sitemap fallback (manual seeds in v1, HTML link-graph in v2).

Round 4 (2026-04-30) — Education + SaaS Dev Tools

Goal: Validate education platforms (often Course-schema heavy) and SaaS docs sites.

Sites probed: Coursera, Khan Academy, MIT OCW, edX, Stripe, Notion, Atlassian, Datadog.

Critical findings: - All 8 accessible (no WAF blocks). - Education is the one vertical where JSON-LD is reliable: MIT OCW, Coursera, edX use Course + Person + Organization. Khan Academy unclear from samples. - MIT OCW most permissive (Allow: /); edX has Crawl-delay: 10; Coursera & Khan block GPTBot/AI bots. - SaaS docs subdomain proliferation: Stripe (8 subdomains), Atlassian (6+), Notion (5), Datadog (6). - Atlassian's confluence.atlassian.com enforces 60-second crawl-delay + 500+ Disallow paths — deliberate cost-rationing. - JSON-LD weak on SaaS marketing sites (Datadog has Org + Product schema; others have nothing).

Plan impact: DSS Course row validated; added subdomain-handling note to §16m; education flagged as "the schema-first vertical."

Round 5 (2026-04-30) — Healthcare + News + Community + Foundation

Goal: Validate verticals with potentially specialized schema needs.

Sites probed: Mayo Clinic, WebMD, Cleveland Clinic, NYT, BBC, Reuters, Stack Overflow, Reddit, Hacker News, Mozilla, Adobe Helpx (retry).

Critical findings: - Healthcare: 1/3 fully accessible (Cleveland Clinic). Zero exposed MedicalCondition schema.org on any accessible site. WebMD aggressively blocks AI bots (ClaudeBot, GPTBot, CCBot). - News: 3/4 WAF-blocked (NYT, BBC, Reuters). Medium accessible — uses NewsArticle JSON-LD. - Community: Stack Overflow + Reddit blocked. Hacker News accessible — no schema.org, deliberately omits structured metadata. - Mozilla: Bedrock CMS, minimal schema. - Adobe Helpx: still WAF-blocked (60s timeout) — docs subdomain not exempt from WAF.

Plan impact: Health DSS row deferred (no schema.org evidence). Community DSS row deferred. News maps to marketing. WAF density on premium news confirmed.

Round 6 (2026-04-30) — DTC Retail Commerce + Furniture

Goal: Validate direct-to-consumer commerce and furniture retail (the "tell me about that sofa" use case).

Sites probed: Nike, Restoration Hardware, Walgreens, Oriental Trading, Dell, LL Bean, SKIMS, Havertys.

Critical findings: - 5 of 8 fully WAF-blocked (Nike, RH, Walgreens, Dell, Havertys). - 3 accessible: Oriental Trading, LL Bean, SKIMS. - 0 of 8 use off-the-shelf platforms. No Shopify, no Salesforce Commerce Cloud, no Adobe Commerce — all custom proprietary. - 0 of 8 expose JSON-LD Product schema on accessible product pages. - SKIMS: Sanity CMS + Imgix. 500+ products. Color = separate URL. No canonical tag visible on samples. - LL Bean: 1,251 product category URLs in flat sitemap. Modern responsive (likely Jamstack). No schema visible. - Oriental Trading: complex faceted-search URL patterns (a1-ID+filter+count.fltr). Custom catalog engine. - Reviews universally lazy-loaded. - Furniture sites (RH, Havertys) have rich domain-specific data (dimensions, materials, fabrics, lead times) — but ALL WAF-blocked.

Plan impact: §16o DTC retail row added with DOM-extraction-first approach; §19b WAF ladder; furniture sub-schema deferred to v2; canonical URL inferer added as v1 task.

Aggregate validation coverage

Total sites probed: ~60 across 14 verticals. WAF-block rate aggregate: ~40% (concentrated in enterprise B2B and premium DTC). JSON-LD content-class @type usefulness: <25% across verticals tested (education is the outlier where it's reliable). Sitemap discoverability: ~70% (lower than expected — many sites WAF-block sitemap files even when robots.txt is open). CMS signature detectability when present: ~90%.

The plan is grounded in this empirical reality. See ../04-Web-Reality-Catalog.md for the full data layer with citations.