Dashboard/Crawler-Factory/Sessions/Session-1-Lessons-Learned.md
Crawler-Factory

Sessions/Session-1-Lessons-Learned.md

Lessons Learned — Crawler Factory Planning Session (2026-04-30)

Observations from spending Opus-1M context on a 13-cluster engineering plan + spec generation + verification cycle. The user (Arijit) explicitly asked me to flag SOP/skill gaps as I work. This is that report.

TL;DR — five proposals worth acting on

  1. NEW SOP section: Language Equivalence Map (Python ↔ TypeScript ↔ Go ↔ Rust) — moves the manual translation table out of every spec preamble and into the SOP itself.
  2. NEW skill: superpowers:spec-compliance-review — closes the gap between requesting-code-review (CODE) and standards-* (CONTENT). Handles cross-artifact interface drift + plan coverage + SOP audit + closed re-verify loop.
  3. UPDATE superpowers:writing-plans — when generating multiple sub-plans in parallel, require an explicit "shared interface registry" up front so cluster N's exports are locked before cluster M consumes them. Prevents the parallel-author drift pattern.
  4. UPDATE WritingSOPs Hard Rule 1 — codify the em-dash replacement table to prevent the awkward X: Y: Z chained-colon artifact when two em-dashes flank a parenthetical.
  5. NEW pattern in CodingSOPs: Data-Driven Extensibility — document the "DSS-as-data, not code" pattern that worked spectacularly well for content domains and the site-type rubric.

Pattern 1 — Parallel-author drift is the dominant failure mode

What I observed: - 12 parallel agents wrote 13 spec files (~22K lines). - Each agent had: master plan + Spec 01 exemplar + their cluster brief. - Each agent self-reviewed and reported clean. - Post-hoc audit found 7 Critical interface mismatches and 4 Important issues. - The fix pass introduced a NEW Critical (B14) because the fix agents renamed types but didn't re-check fetch-payload contracts at the integration layer.

Why it happens: - Each agent works from a partial view. Cluster N defines exports; Cluster M consumes them. The agent for Cluster M doesn't read Cluster N's actual signatures — it makes a plausible guess from the brief and the master plan's high-level contract. - Self-review catches violations against the brief, not violations against other artifacts. - "I'm done" is reported on local correctness, not system-wide consistency.

Proposal — extend superpowers:writing-plans with an explicit Interface Registry step:

When the orchestrator decomposes a plan into N sub-plans, generate ONE pre-cluster artifact: a Contracts.md (or zod-only contracts.ts stub file) that locks every cross-cluster type, function signature, SSE event shape, and HTTP response shape. Each sub-plan agent imports from this contract at the top of its spec and may not deviate. The contract becomes the single source of truth that cluster authors must respect.

This is essentially what Cluster 01 (foundations) did organically — DSS + zod types — and every spec correctly based on it. The pattern needs to be made explicit and enforced for ALL boundary types, not just data shapes.

Skill update: add a "Phase 0: Contract Lock" to writing-plans when the spec set is multi-cluster. Lock fetch-payload contracts BEFORE any cluster spec is written.

Pattern 2 — The skill ecosystem has a gap between CODE review and CONTENT review

What I observed: - verification-before-completion is for runtime claims (build passes, tests pass) — not for static document review. - requesting-code-review dispatches a code-reviewer agent built around git diff of CODE. I had to override its template to use it for documents. - standards-coding/testing/writing validate content against SOPs but don't check cross-artifact interface drift or plan coverage. - No canonical skill for "do these N specs together implement plan X correctly?"

Proposal — new superpowers:spec-compliance-review skill:

Inputs: - A plan/spec document - N artifact documents (specs, requirement docs, design docs) - Set of applicable SOPs

Pipeline: 1. Plan coverage — for every plan requirement, point to the artifact that covers it. Gaps = Critical. 2. Cross-artifact interface drift — extract export/import graph from each artifact; verify exports match consumer expectations. Drift = Critical. 3. SOP compliance — invoke the relevant standards-* skills per artifact. 4. Hallucinated dependencies — check every referenced symbol resolves to a definition somewhere. 5. Re-verify loop — if fixes are applied, re-run automatically. Halt only when 0 Critical, 0 Important.

This is what I built manually with 4 parallel agents over 2 hours. It should be one skill invocation.

Pattern 3 — SOP language assumptions impose translation cost

What I observed: - CodingSOPs.md is Python-flavored: Pydantic v2, pyright --strict, Ruff, logging.getLogger(__name__), Optional[T]. - The Crawler Factory build is TypeScript: zod, tsc --noEmit, eslint+prettier, Pino via lib/utils/logger.ts, T | null. - Every spec had to include a Python→TypeScript translation table in its SOP rules section. - Every SOP-validation agent had to be told the translation rules in its prompt. - This is repetitive and error-prone.

Proposal — add a "Language Equivalence" section to CodingSOPs.md:

## Language Equivalence

The principles in this SOP are language-agnostic. The tooling is not. Use the canonical equivalent for your stack:

| Principle | Python | TypeScript | Go | Rust |
|-----------|--------|-----------|-----|------|
| Runtime data validation at boundaries | Pydantic v2 | zod | go-validator + struct tags | serde + validator |
| Strict static type checking | pyright --strict | tsc --noEmit (strict) | go build (always) | cargo check |
| Code formatter + linter | Ruff | eslint + prettier | gofmt + golangci-lint | rustfmt + clippy |
| Structured logging | structlog or logging | pino + createLogger wrapper | zerolog or zap | tracing |
| Test runner | pytest | vitest / jest | go test | cargo test |
| Nullable type idiom | `T | None` (3.10+) | `T | null` | `*T` or `Option[T]` (generic) | `Option<T>` |
| Constants naming | UPPER_SNAKE_CASE | UPPER_SNAKE_CASE | PascalCase exported, camelCase package-private | UPPER_SNAKE_CASE |

When you violate the principle in any language, you violate this SOP. The implementation tooling is irrelevant.

This eliminates the per-spec translation block.

Frontend pragma: also codify the React-error-handling pattern (state/toasts vs pino) that surfaced as a sanctioned departure in Specs 09-12. Either promote it or document why it's an exception.

Pattern 4 — Mechanical SOP enforcement creates new problems if the rule lacks context

What I observed: - WritingSOPs Hard Rule 1: No Em-Dashes. - 729 em-dashes flagged as FAIL across 14 specs. - Mechanical replacement (— → :) created new problems: X: Y: Z chained-colon clusters where two em-dashes flanked a parenthetical. - The sweep was SOP-correct but readability-degrading.

Proposal — extend WritingSOPs Hard Rule 1 with a replacement table:

## Hard Rule 1: No Em-Dashes

Em-dashes (`—`, U+2014) are banned in narrative prose. Use the appropriate replacement based on grammatical role:

| Pattern | Replace with |
|---------|--------------|
| `**Bold —** rest` | `**Bold:** rest` |
| `clause — start of new clause` | `clause: start of new clause` (most common) |
| `clause — end of sentence. New sentence` | `clause. New sentence` |
| `X — parenthetical aside — Y` | `X (parenthetical aside) Y` (NEVER `X: parenthetical: Y`) |
| `X — Y — Z` (parenthetical at end) | `X (Y), Z` or `X, Y, Z` |
| Numeric range `5—10` | Use en-dash `5–10` per Rule N (range expressions) |

Exempt: em-dashes inside fenced code blocks, regex strings, or string literals.

This is what the sweep agent was missing — Rule 4 (parenthetical em-dashes) needs an explicit "use parens or commas" instruction.

Pattern 5 — "Data-driven extensibility" worked spectacularly; codify it

What I observed: - DSS (Domain Schema Standard) is a typed registry — Record<ContentDomain, DssEntry>. Adding a 10th content domain is one config change, no code release. - Site-type rubric (§19d) is an ordered list of {predicate, siteType} rules — same pattern. - CMS detector signatures (Spec 04) — same pattern. - These were the parts of the plan that suffered ZERO drift across parallel agents. Everyone consumed them as data.

Proposal — add a "Data-Driven Extensibility" pattern to CodingSOPs:

## N. Data-Driven Extensibility

When the variation space is bounded, well-known, and likely to grow over time, prefer data over polymorphic code.

GOOD — registry pattern:
```typescript
const REGISTRY: Record<Key, Entry> = { ... };
function lookup(key: Key): Entry { ... }

BAD — class-per-variation:

class MarketingHandler extends BaseHandler { ... }
class SupportHandler extends BaseHandler { ... }
// ... 9 classes

The registry pattern has three properties polymorphic code lacks: 1. Adding a new variation is a config change, not a code change. 2. Cross-cutting changes (e.g., "all entries need a new field") are one diff, not N. 3. Parallel work consumes the registry as a contract; class hierarchies invite ad-hoc extension by parallel authors.

Use this pattern for: content type registries, route maps, feature-flag definitions, error-code → user-message mappings, schema → handler routing. ```

This pattern is also why Cluster 01 (foundations) had zero drift — every other spec consumed DSS as data.

Pattern 6 — Verification needs a re-verify loop, not a one-shot pass

What I observed: - V1 verification found 11 issues. - V1 fix pass (3 parallel agents) closed all 11 but introduced 1 NEW Critical (B14). - Without V2 verification, B14 would have surfaced at runtime in Spec 13 smoke — by which time 6 downstream cluster builds would have committed code that needs unwinding. - I'm now running a V2 fix + V3 verification.

Proposal — verification skills must include a closed re-verify loop:

Any verification skill (current standards-*, future spec-compliance-review) should: 1. Identify issues 2. (Optional) auto-dispatch fix agents 3. Mandatorily re-run verification after fixes 4. Halt only when verdict is clean OR a max-iterations limit is hit (currently I'd suggest 3 — beyond 3 fix-loops, escalate to human review)

Today this discipline lives in the operator (me). It should live in the skill.

Pattern 7 — Plan persistence pays off

What I observed: - The master plan + 13 specs + verification reports all live in vault Projects/Crawler-Factory/. - When V2 found B14, the V1 verification report told me exactly which file:line refs to inspect. - When the Spec 08 fix agent finished, its summary cited the original report's issue IDs. - The audit trail is queryable.

Proposal — formalize the plan persistence pattern in feedback_plans_to_vault_then_sops_then_factory.md:

The current memory note covers the workflow direction (plan → SOPs → specs → factory). It should also explicitly require: - Verification reports are first-class artifacts, NOT throwaway agent output. Save with version suffix (V1, V2, V3 …). - Each fix pass references the issue ID from the most recent verification report. - The final state of the project includes ALL verification reports as part of the audit trail (so a future operator can see "we found 11 issues, fixed them, re-verified, found 1 more, fixed it, re-verified clean").

What I'd add to my own behavior (memory feedback)

I've saved 5 feedback memories this session about my own skill misuse: - Always invoke the canonical skill for planning/spec/document work - Verify before claiming done — never assume parallel work is compatible - No canonical skill for spec-vs-plan compliance review - Plans go to vault + docs/planning, then SOPs, then agent factory - No Supabase, no Redis for new features

The pattern: the user keeps catching me defaulting to "the most plausible skill name" instead of reading what each skill actually does. The fix is in skill DISCOVERY, not in my memory. A superpowers:skill-router skill that takes a task description and recommends the right skill (or flags "no canonical skill — use these primitives") would reduce this load.

When to bring these up again

This document is the standing record. I'll surface specific items as they become relevant — e.g., next time I generate parallel specs, I'll proactively suggest the Contract Lock pattern from Section 1. Next time someone asks for a Python translation in CodingSOPs, I'll point at the Language Equivalence proposal in Section 3.

The user asked: "Is there anything that you have seen? Is there anything you want to talk to me or tell me about now or maybe in the future?" The honest answer is: yes, the seven patterns above. They're worth ~half a day of SOP/skill work. ROI scales with future parallel-build sessions.