Crawler-Factory

Sessions/Session-1-Process-Documentation.md

Process Documentation — Session 1 (2026-04-30)

Phase-by-phase record of how the Crawler Factory plan + 13 specs + verification cycle were produced. Every skill, tool, and agent type is named so the proposed meta-skill (Idea-to-Build-Skill-Proposal.md) can codify the orchestration.

Session length: ~one Opus 4.6 1M context window (consumed ~64% before this reflection). Operator: Arijit Chowdhury. Output: master plan, 13 engineering specs (~22K lines), 3 verification reports, 4 SOP audit reports, lessons-learned, this documentation.

Phase 0 — Ideation + Dossier Handoff (Operator → Agent)

Operator inputs: - Idea: "build a Crawler Factory — give a website URL, factory auto-builds the crawler" - UX walk-through: enter URL → discover sitemap → categorize → pick categories → configure → test → commit → live progress - Functional walk-through: per-category crawler creation, sample-driven configuration, test-then-commit gate - Knowledge sources / dossier: - SESSION.md (rc3-phoenix repo) - MEMORY.md (auto-loaded) - vault Standards/ (CodingSOPs, TestingSOPs, WritingSOPs, FolderStructure) - vault Projects/Algolia-Central/ (prior work, decisions, releases) - existing crawler infrastructure: packages/crawl/ (Python), lib/utils/logger.ts, api-src/, lib/llm/ - Algolia Crawler API knowledge captured in SESSION.md

Skills used: none yet — pure context loading. Tools used: Read (initial SESSION.md scan), Bash (recon on packages/crawl, src/, api/).

Critical observation: the operator's dossier was rich AND structured. Not a vague brief. This made Phase 1 possible. The proposed skill should require a structured dossier, not a vague prompt.

Phase 1 — Initial Plan Sketch

Skill invoked: superpowers:writing-plans Tools used: Write (plan file at ~/.claude/plans/joyful-strolling-lobster.md), Read (codebase recon), Bash (file structure exploration), Agent of subagent_type Explore (3 parallel Explore agents: backend/SSE/Supabase patterns, crawler infra, admin UI patterns).

Output: initial plan with FSM, file structure, data model, UI specs, phasing v1 vs v2, 31 implementation tasks, test strategy, verification protocol, risks, model handoff note.

Operator review: caught caps and assumptions, listed below in Phase 2.

This is the most important phase. The operator caught five major architectural mistakes in the initial plan:

"Why are you using Supabase? There is no Supabase." → I corrected to Redis. → Operator: "no redis either." → I asked about persistence → Operator picked Algolia meta-index.
"Categories going to their own indices, not one index." Marketing index, support index, education index — multi-index, not single.
"Why a 4-level depth cap on sitemap?" → no cap.
"Why a 5000 URL cap? Even simple sites have more." → no cap.
"The data model needs fresh design — what about schema.org? Most sites won't have it. Do thorough research."

The operator also surfaced the hub-and-spoke vision: one index per content domain, one specialist agent per index, an orchestrator agent on top doing fan-out/synthesis. The factory's architecture had to be forward-compatible with this.

Skills used: none new — this was conversational push-back. Tools used: Write (plan edits), Edit (surgical plan revisions), AskUserQuestion (lock 4 blocking decisions: persistence, UI placement, index target, test crawl mechanism).

Critical observation: the operator's pushback drove the most important architectural pivots. Without it, the v0 plan would have shipped a single-index, capped-discovery, schema.org-first system that ignored 60% of the web. The proposed skill MUST include explicit pushback gates — the agent cannot self-detect its own assumptions.

Phase 3 — Empirical Research-Driven Validation

This phase is what the operator flagged as "not part of any skill, but there needs to be a skill which guides that to happen."

Operator instruction: "Did you do research on common schema.org patterns? Did you research websites — automobile, CMS, B2B SaaS, multi-brand corporates, e-commerce? Validate the plan against real sites or I'm not accepting it."

Skills used: none formal — this is the gap. Tools used: - WebFetch (direct, ~30 calls during agent research) - Agent subagents of subagent_type Explore for parallel research (8 batches, ~60 sites total)

Research topics dispatched to Explore agents: - Round 1: Salesforce, Adobe, Oracle, HP, BMW, Mercedes (enterprise B2B) - Round 2: TechCrunch, WordPress.org, Drupal.org, Wikipedia, Guardian, Shopify (CMS/generalist) - Round 3: Web Almanac 2024 + sitemaps.org spec + Google Search Central (authoritative refs) - Round 4: LVMH, Diageo, Unilever, P&G (multi-brand corporates) - Round 5: Microsoft, AWS, Apple, Algolia (massive scale) - Round 6: commercetools, BigCommerce, OpenText (SaaS B2B) - Round 7: Walmart, Etsy, eBay, Wayfair, Best Buy, 3M, Caterpillar, Siemens (commerce + manufacturing) - Round 8: Coursera, Khan, MIT OCW, edX, Stripe, Notion, Atlassian, Datadog (education + dev tools) - Round 9: Mayo, WebMD, Cleveland, NYT, BBC, Reuters, Stack Overflow, Reddit, Hacker News, Mozilla, Adobe Helpx (healthcare + news + community + foundation) - Round 10: Nike, Restoration Hardware, Walgreens, Dell, LL Bean, Oriental Trading, SKIMS (DTC) - Round 11: Havertys (additional DTC, operator request)

Findings that fundamentally changed the plan: - JSON-LD adoption is 41% globally (Web Almanac), only ~7% with content-class @types — not 70% as my initial plan assumed - ~40% of sites WAF-block plain fetch from datacenter IPs - Sitemap nesting reaches 2-3 levels in reality (Microsoft) despite spec saying 1 - 1M-10M URL sites exist (learn.microsoft.com) - Multi-brand corporates use federated architecture - DTC retail uses 0% off-the-shelf platforms (no Shopify schema.org reliance) - WordPress alone = 18% of all sites; CMS detection > schema.org

These findings drove: - Reordering the detection cascade (CMS-first, not JSON-LD-first) - Promoting Playwright to v1 - Removing all caps (URL count, sitemap depth) - Adding multi-tenant abstraction for federated brands - Adding 9 new tasks (CMS detector, WAF detector, site-type rubric, canonical inferer, scope estimator, brand-domain discoverer)

Critical observation: this phase ABSOLUTELY needs a skill. It happened because the operator told me to do it AND because I was on Opus 1M (budget for parallel research). A repeatable skill must: - Force "what assumptions in this plan need empirical validation?" as an explicit step - Dispatch research agents for each unverified assumption - Aggregate findings BACK into the plan (not as appendix) - Produce a "Reality Catalog" section anchored in citations

Phase 4 — Plan Finalization + Vault Persistence

Skills used: none new. Tools used: Write, Edit (extensive plan revisions: §14 Web Reality Catalog, §15 Future Agent Architecture, §16 Site-Type Validation Matrix with 16 rows, §17 Multi-Tenant, §18 Data Exposure, §19 Methodology, §20 Plan Update Audit Trail), Bash (vault directory creation, file copy).

Vault structure created:

Projects/Crawler-Factory/
├── 00-Plan.md (master plan, 2043 lines)
├── 01-Methodology.md (extracted §19)
├── 02-DSS.md (extracted §4)
├── 03-SiteTypes/Index.md
├── 04-Web-Reality-Catalog.md (extracted §14)
├── 05-Architecture-Hub-Spoke.md
├── 06-Multi-Tenant.md
├── 07-Update-Audit-Trail.md
├── README.md
├── Status.md
├── Research/Index.md
└── Engineering-Specs/ (empty — populated next phase)

Local repo mirror: docs/planning/2026-04-30-crawler-factory-plan.md (committed a0785b17).

Critical observation: the operator instructed me to commit to vault as a NEW project. This was correct — the plan is the source of truth across machines and sessions. The proposed skill MUST require vault commit before declaring "plan frozen."

Phase 5 — Engineering Spec Generation

Operator instruction: "What skill do you need or should you be using to create the engineering plan, because otherwise you're going to go all over the place and create weird plans based on assumptions not validated? I don't trust your skills."

Skill invoked: superpowers:writing-plans (re-invoked, this time for spec-level plans) Tools used: - Write (Spec 01 inline as exemplar — ~1500 lines) - Agent of subagent_type general-purpose x 12 in parallel — each invoking superpowers:writing-plans skill internally to generate one cluster spec - Each agent saved its spec directly to vault Engineering-Specs/

Output: 13 specs totaling ~22,040 lines, plus 00-Index.md.

Critical observation: dispatching parallel agents was efficient (12 specs in ~10 min wall-clock) BUT introduced the parallel-author drift problem (caught in Phase 6). The proposed skill MUST mandate a "Contract Lock" phase BEFORE parallel spec generation — generate a contracts.ts of cross-cluster types first, then the parallel agents consume it as a fixed contract.

Phase 6 — Multi-Axis Verification (V1)

Operator instruction: "What is the guarantee that the build will be successful? Have you verified your specs against the plan to make sure that the specs are actually compatible and are going to implement the plan? Have you verified that you have the skills to verify..."

This is the second "I don't trust you" gate.

Wrong skill choices I made (operator caught both): 1. superpowers:verification-before-completion — operator caught: "this is for verifying UI or something. Is this even the right skill?" It's for runtime claims, not document review. 2. superpowers:requesting-code-review — operator caught: "we haven't written any code, so what code review are we doing with this skill?"

Skills actually used (correct fit): - superpowers:requesting-code-review (overridden via custom prompt) → spawned superpowers:code-reviewer agent for cross-axis (plan coverage + interface drift) - standards-coding (for CodingSOPs audit on spec content) - standards-testing (for TestingSOPs audit on spec test code) - standards-writing (for WritingSOPs audit on spec narrative)

Tools used: - Agent x 4 in parallel (one code-reviewer for cross-axis, three general-purpose agents executing the standards-* skills)

Output reports (saved to vault Engineering-Specs/): - VERIFICATION-REPORT.md (V1 cross-axis): 7 Critical + 4 Important + 4 Minor - SOP-CODING-REPORT.md: 0 FAIL, 0 WARN - SOP-TESTING-REPORT.md: 0 FAIL, 1 cosmetic WARN - SOP-WRITING-REPORT.md: 14 FAIL on Hard Rule 1 (em-dashes); other axes clean

Critical observation: THIS IS THE LARGEST SKILL GAP. There's no canonical skill for "spec compliance review across multiple artifacts and SOPs with closed re-verification loop." I composed it from 4 separate skill invocations + 4 parallel agents, with manual orchestration. The proposed meta-skill should expose this as ONE invocation: verify-specs(plan, specs[], sops[]) → report.

Phase 7 — Fix Pass

Skills used: none formal — Edit and Bash for surgical fixes, Agent for parallel fix dispatch.

Tools used: - Agent x 3 in parallel (Spec 08 fixes B1-B6; Spec 11+12 fixes B7+B10+B11; Spec 04+05+07+13 fixes A1+B8+B9+C1+D7) - Then 4 inline edits for follow-up flags (Spec 01 getSamples, Spec 08 analysis return, Spec 02/03 doc drift) - Then 1 Agent (general-purpose) for em-dash mechanical sweep across 14 files via Python script

Critical observation: the fix pass introduced a NEW Critical (B14 — fetch payload contract drift). The operator was right to demand re-verification: without it, B14 would have surfaced at Spec 13 smoke time, requiring 6 cluster builds to be unwound.

The proposed skill MUST include a closed re-verify loop, not a one-shot fix-and-ship.

Phase 8 — Re-Verification (V2 + V3)

Skills used: same as Phase 6 (re-invoked). Tools used: Agent x 2 in parallel for V2 (cross-axis re-audit + writing re-audit); Agent x 1 for V3 (final cross-axis confirmation).

Output reports: - VERIFICATION-REPORT-V2.md: all 11 V1 issues FIXED + 1 NEW Critical (B14) - SOP-WRITING-REPORT-V2.md: 0 FAIL after em-dash sweep - VERIFICATION-REPORT-V3.md: B14 closed, no new drift, Ready to build

Plus inline polish edits for 4 chained-colon awkward constructions flagged by writing audit.

Phase 9 — Lessons Learned + Process Reflection

Operator instruction: "are there any moments, any learnings, any specific observations, or any thoughts that you have? Would it make sense to add this to the SOP, or would it make sense to maybe perhaps modify..."

Output: Lessons-Learned-Session-1.md with 7 patterns + concrete proposals for SOP/skill updates.

Then this current document set (Verbatim Transcript + Process Documentation + Idea-to-Build Skill Proposal).

Skills used in this session — full catalog

Phase	Skill	Why used	Worked?
1	`superpowers:writing-plans`	Initial plan creation	✅ Yes
4	`superpowers:writing-plans`	Plan revisions	✅ Yes
5	`superpowers:writing-plans`	Spec generation (12x via subagents)	✅ Yes (with Contract Lock gap)
6 (wrong)	`superpowers:verification-before-completion`	I picked it for "verify"; operator caught it's runtime-only	❌ Wrong skill
6 (wrong)	`superpowers:requesting-code-review`	Closer fit; operator caught "we have no code"	⚠️ Force-fit
6 (right)	`superpowers:requesting-code-review` (overridden prompt) → `superpowers:code-reviewer` agent	Cross-axis spec review	✅ Worked but unnamed
6 (right)	`standards-coding`	CodingSOPs audit	✅ Correct fit
6 (right)	`standards-testing`	TestingSOPs audit	✅ Correct fit
6 (right)	`standards-writing`	WritingSOPs audit	✅ Correct fit
All	`Skill`-tool primitives + parallel `Agent` dispatch	Orchestration	✅ But manual

Skills NOT used but should have been (or should exist): - A "guided exploratory research" skill — for Phase 3 (research-driven validation) - A "spec compliance review" skill — closes the gap between code-review (CODE) and standards-* (CONTENT); covers cross-artifact interface drift + plan coverage + SOP audit + closed re-verify loop - A "skill router" skill — would have prevented the verification-before-completion / requesting-code-review misfires - A "Contract Lock" sub-step in writing-plans for parallel spec generation - An "idea-to-build pipeline" meta-skill — orchestrates everything

Tools used in this session — full catalog

Tool	Phases	Notes
`Read`	All	Master plan, specs, SOPs, codebase exploration
`Write`	All writing phases	Plan files, spec files, audit reports, memory notes
`Edit`	Refinement phases	Surgical plan/spec corrections
`Bash`	All	Git, file ops, recon, mirror sync
`Skill`	Phases 1, 5, 6	writing-plans, standards-* invocations
`Agent` (Explore)	Phases 1, 3	Codebase + research subagents
`Agent` (general-purpose)	Phases 5, 6, 7, 8	Spec generation, SOP audits, fix passes, em-dash sweep
`Agent` (superpowers:code-reviewer)	Phases 6, 8	Cross-axis spec verification
`WebFetch`	Phase 3	Sample fetches against ~60 sites
`ToolSearch`	All	On-demand loading of deferred tools (AskUserQuestion, EnterPlanMode, ExitPlanMode, TaskCreate, etc.)
`EnterPlanMode` / `ExitPlanMode`	Phase 1	Formal plan mode for initial drafting
`AskUserQuestion`	Phase 2	4 blocking decisions locked
`TaskCreate` / `TaskUpdate` / `TaskList`	All execution phases	Progress tracking

Key memory feedback notes saved this session

feedback_invoke_skill_for_plans_specs.md — always invoke the canonical skill for planning/spec work
feedback_verify_before_claiming_done.md — never assume parallel work is compatible; explicit verification pass required
feedback_skill_gap_spec_vs_plan_review.md — there's no canonical skill for spec-vs-plan compliance; don't force-fit
feedback_plans_to_vault_then_sops_then_factory.md — workflow direction
project_persistence_redis_not_supabase.md — no Supabase, no Redis for new features
project_multi_index_hub_spoke_vision.md — multi-index per content domain + specialist agents
project_web_reality_research_findings.md — empirical numbers from this session's research
project_crawler_factory_plan.md — pointer to vault project

What this documentation enables

A future agent (Sonnet 4.6 in build phase, or me in a future session) can read this document and trace exactly: - What knowledge was provided - What questions were asked - What pushback occurred - What research was done - What skills failed to fit and why - What skills worked and where

Combined with Process-Reflection-Verbatim-Transcript.md (the operator's own framing) and Idea-to-Build-Skill-Proposal.md (the design), this is the complete record needed to codify the meta-skill the operator asked for.