Dashboard/Crawler-Factory/Core/07-Update-Audit-Trail.md
Crawler-Factory

Core/07-Update-Audit-Trail.md

Plan Update Audit Trail

Extracted from 00-Plan.md §20. Maps each empirical finding to the specific plan section it modified.

20. Plan-update summary — what the validation research changed

Mapping each empirical finding to the specific plan section it modified. This is the audit trail for "why is the plan the way it is."

Finding Source Old assumption Plan section updated
JSON-LD adoption is 41%, not 70%; only ~7% useful @types Web Almanac 2024 §14a "JSON-LD first cascade" §3b cascade reordered to CMS-first; §4b table; §19c
Real content pages (TechCrunch, GOV.UK, Drupal.org) lack JSON-LD 26-site empirical §14g Same §3b, §4b, §19c
Sitemap nesting reaches 2-3 levels (Microsoft) §14 audit "1 level per spec" §3a §3, §16l, §19a
Sites with 1M-10M URLs exist (learn.microsoft.com) §14 audit "Cap at 5K v1" All caps removed; §3a §4-5; §10 risks; §19a
Single urlset files exceed 10MB §14 audit "Default fast-xml-parser dom" §3a §6 streaming SAX parser when >5MB
~40% of sites WAF-block plain fetch 60-site audit "Playwright in v2" §3d; §6 v1 includes Playwright; §19b ladder
Multi-brand corporates (LVMH, Diageo) are federated §14 audit "Single discovery per site" §16k brand-domain enumeration sub-flow; §17 multi-tenant
51% of sites use a CMS; WordPress alone = 18% Web Almanac 2024 §14d "CMS detection nice-to-have" Promoted to layer-1 of cascade; §19d rubric
Crawl-delay: 10 is a Drupal signature §14 audit Generic respect §19d explicit Drupal trigger
<article> tag adoption ~5% Web Almanac 2024 §14c "Use semantic HTML" Demoted to layer 7 (low confidence)
99% of articles have date+author in text §14 audit + heuristics Not in cascade Added as layer 6 (universal article-family fallback)
Education sites consistently use Course schema.org §14f DSS theoretical DSS row validated; §16 sub-row; education's the one vertical where schema-first works
DTC retail uses 100% custom platforms (no Shopify) DTC audit "Shopify for product schema" §16o DOM-extraction-first for product pages; furniture sub-schema in v2
DTC has high WAF (Nike, RH, Walgreens, Dell, Havertys all 403) DTC audit Same as enterprise §16o + §19b ladder
RH/Havertys-class furniture has rich domain data (dimensions, materials, fabrics) DTC audit Generic product schema §16o furniture sub-schema notes; v2 vertical sub-schemas
Reviews are universally lazy-loaded on DTC DTC audit Static HTML extraction §16o defer reviews; v2 add XHR capture
Color/size variants strategy varies (URL vs state) DTC audit One pattern §16o canonical inference + variant deduplication
News sites WAF-block but Medium accessible with NewsArticle schema §14 audit Generic article handling §16a-extended note
Healthcare sites don't expose MedicalCondition schema §14 audit Add health DSS row DEFERRED — health maps to education until evidence emerges
Q&A/community sites avoid structured metadata (Hacker News confirmed) §14 audit Q&A → support DSS §16h-extended; v2 may add community row
Microsoft sitemap structure has 64+ language variants §14 audit Single-locale crawler §16l + locale picker UI; v2 per-locale crawlers
Site-type detection should drive playbook selection Cross-cutting Implicit §19d rubric — explicit ordered predicates

20a. Tasks added or strengthened by validation

These task changes (relative to my v0 task list) were forced by the findings:

  • NEW Task A: lib/factory/cms-detector.ts — the CMS fingerprint library (WordPress, Drupal, AEM, Shopify, Hubspot, Webflow, Squarespace, Wix, Salesforce LWC, custom-by-elimination). Drives layer 1 of the cascade.
  • NEW Task B: lib/factory/waf-detector.ts + lib/factory/playwright-fetcher.ts — WAF probe and stealth-mode fallback. Used by sampler when plain fetch 403s.
  • NEW Task C: lib/factory/site-type-rubric.ts — the §19d rubric as ordered predicates.
  • NEW Task D: lib/factory/canonical-inferer.ts — for DTC retail variant deduplication. Heuristic + LLM fallback.
  • NEW Task E: lib/factory/scope-estimator.ts — pre-flight scope probe (§3a-bis).
  • NEW Task F: lib/factory/brand-domain-discoverer.ts — for multi-brand corporates. Scrapes brand-index page (/our-brands etc.), extracts outbound brand domains.
  • STRENGTHENED Task 5 (classifier): explicit 8-layer cascade with all halting and aggregation rules from §19c.
  • STRENGTHENED Task 9 (structure-analyzer): CMS-aware selector libraries per CMS detected, NOT pure LLM.
  • STRENGTHENED Task 8 (sampler): WAF detection + Playwright escalation built in.

These task additions will be integrated into §7 in the next plan revision (after this section is approved). For now the additions are documented here so nothing is lost during the freeze → SOPs → engineering specs flow.

20b. v1 scope adjustments forced by findings

  • ✅ Playwright sampling — moved from v2 to v1 (was: defer; now: required for 40% of sites)
  • ✅ CMS detector library — added to v1 (was: implicit in classifier; now: standalone module)
  • ✅ Canonical inferer — added to v1 (was: not in plan; now: required for DTC retail)
  • ✅ Brand-domain discoverer — added to v1 (was: v2; now: light v1 scope — read brand-index page, list domains, link sessions)
  • ✅ Scope estimator — added to v1 (was: not in plan; now: required for Microsoft-class sites)
  • ⏸ Per-locale crawlers — stays v2 (one crawler per pathGroup; locale as facet)
  • ⏸ Bulk multi-brand parallel crawl — stays v2 (one brand at a time in v1)
  • ⏸ Vertical sub-schemas (furniture, apparel, electronics) — stays v2 (v1 captures generic product-catalog only)
  • ⏸ Health DSS row — stays v2/deferred (no schema.org evidence to validate)
  • ⏸ Community DSS row — stays v2/deferred (no structured metadata in real samples)

20c. Plan freeze readiness checklist

Concern Status
Architecture documented end-to-end ✅ §2
Data model with zod schemas ✅ §4
UI specs with mockups ✅ §5
Tasks with TDD bite-sized steps ✅ §7 (with §20a additions to integrate)
Test strategy ✅ §8
Verification protocol ✅ §9
Risks + mitigations ✅ §10
Empirical grounding (Web Almanac + 60-site audit) ✅ §14
Future agent architecture ✅ §15
Per-site-type playbooks (16 rows) ✅ §16
Multi-tenant abstraction ✅ §17
Methodology / operating protocol ✅ §19
Plan-update audit trail ✅ §20
Decisions locked ✅ §1 (9 decisions); §6 phasing
Build-time model handoff ✅ §11
Critical reference files ✅ §13

The plan is ready to be frozen and committed.

20d. Vault commit destination — NEW project: "Crawler Factory"

User instruction (2026-04-30): create a NEW vault project called Crawler Factory and put all knowledge under it. Local disk is ephemeral; vault is the source of truth across machines and sessions.

Vault root: ~/Library/CloudStorage/GoogleDrive-arijit.chowdhury@algolia.com/My Drive/AI-Docs/Obsidian/ArijitOS-Brain/

New vault project structure to create:

Projects/Crawler-Factory/
├── README.md                               # Project landing — vision, status, links
├── 00-Plan.md                              # This entire plan, copied verbatim
├── 01-Methodology.md                       # §19 extracted as standalone (operating protocol)
├── 02-DSS.md                               # §4 DSS extracted (Domain Schema Standard reference)
├── 03-SiteTypes/                           # §16 each row as its own note (linkable)
│   ├── 16a-WordPress.md
│   ├── 16b-Drupal-Government.md
│   ├── 16c-AEM-Enterprise.md
│   ├── 16d-Shopify.md
│   ├── 16e-Salesforce-LWC.md
│   ├── 16f-Auto-Manufacturer.md
│   ├── 16g-SPA-NoSitemap.md
│   ├── 16h-Wikipedia-MediaWiki.md
│   ├── 16i-Multi-Domain-Corporate.md
│   ├── 16j-Government.md
│   ├── 16k-Multi-Brand-Federated.md
│   ├── 16l-Massive-Scale-Silo.md
│   ├── 16m-SaaS-B2B.md
│   ├── 16n-Education.md
│   └── 16o-DTC-Retail-Commerce.md
├── 04-Web-Reality-Catalog.md               # §14 — Web Almanac data + audit findings
├── 05-Architecture-Hub-Spoke.md            # §15 future agent architecture
├── 06-Multi-Tenant.md                      # §17 multi-tenant orchestration
├── 07-Update-Audit-Trail.md                # §20 — what changed vs why
├── Research/
│   ├── round1-enterprise-b2b.md            # Salesforce, Adobe, Oracle, HP, BMW, Mercedes raw findings
│   ├── round2-cms-generalist.md            # TechCrunch, WordPress.org, Drupal.org, etc.
│   ├── round3-web-almanac-2024.md          # Authoritative source extracts + citations
│   ├── round4-multi-brand-massive.md       # LVMH, Diageo, Microsoft, AWS, Apple, Algolia
│   ├── round5-commerce-manufacturing.md    # Walmart, Etsy, eBay, Wayfair, 3M, Caterpillar, Siemens
│   ├── round6-education-saas-devtools.md   # Coursera, Khan, MIT OCW, edX, Stripe, Notion, Atlassian, Datadog
│   ├── round7-healthcare-news-community.md # Mayo, WebMD, Cleveland Clinic, NYT, BBC, Reuters, Hacker News, Mozilla, Adobe Helpx
│   └── round8-dtc-commerce.md              # Nike, RH, Walgreens, Dell, LL Bean, Oriental Trading, SKIMS, Havertys
├── Engineering-Specs/                      # POPULATED LATER by SOP layer
│   └── (created when coding/writing/logging SOPs are applied to 00-Plan.md)
└── Status.md                               # Live state: what's planned, in progress, shipped

Local mirror in repo (not authoritative — for local dev convenience):

docs/planning/2026-04-30-crawler-factory-plan.md  # Copy of 00-Plan.md

Memory pointers to add (cross-session): - project_crawler_factory_plan.md — points to vault Projects/Crawler-Factory/00-Plan.md - Updates the existing project_web_reality_research_findings.md to link the vault Research/ folder

Workflow after vault commit: 1. Plan is frozen at vault Projects/Crawler-Factory/00-Plan.md 2. Apply CodingSOPs / TestingSOPs / WritingSOPs / LoggingSOPs (TODO: locate Logging SOPs path) to generate engineering specs into Engineering-Specs/ 3. Switch model to Sonnet 4.6 4. Multi-agent execution reads Engineering-Specs and produces code

This is the explicit handoff to the next phase. Plan signed off → vault → SOPs → specs → agent factory.