Dashboard/Crawler-Factory/Engineering-Specs/04-cms-classification.md
Crawler-Factory

Engineering-Specs/04-cms-classification.md

Crawler Factory: Spec 04: CMS Detection + Content Classification Cascade

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Implement the CMS-fingerprint-first content-domain classification cluster (detectCms, the 8-layer detection cascade aggregator, and the site-type recognition rubric) so every pathGroup the factory finds gets a {contentDomain, confidence, evidence} verdict the user can trust.

Architecture: CMS+URL is the cascade FOUNDATION (~95% combined coverage per Web Almanac 2024 + the 14-site empirical audit). JSON-LD enriches but never leads (only 41% global, ~7% useful @type values). LLM is a last-resort tiebreaker. Each layer is a pure function returning {contentDomain, confidence} or null; the aggregator picks the max-confidence candidate, applies an agreement boost (each agreeing layer adds 0.05, capped at 0.95), and only fires the LLM layer when combined layers 1–7 confidence < 0.55. The site-type rubric is an ordered list of {predicate, siteType} rules (data, not code) that maps a probed site to a §16 playbook in the first 30 seconds.

Tech Stack: TypeScript (strict), zod, cheerio (already in repo), vitest with vi.mock, pino logger via lib/utils/logger.ts, existing lib/llm/ provider for layer 8.

Depends on: Spec 01 (DSS, dssBySchemaOrgType, ContentDomain, DetectedVia, types). Spec 03 (the Sample shape: {url, html, status, structureAnalysis?}: used as input to the cascade and the rubric). Consumed by: Spec 06 (configurator: each pathGroup carries the cascade verdict), Spec 07 (extractor generator: site-type rubric output drives selector hints), Spec 11 (frontend: traffic-light per pathGroup), Spec 12 (orchestrator).

Plan source: Projects/Crawler-Factory/00-Plan.md §3b (Content-domain classification: REVISED), §4b (Detection cascade: 8 layers), §7 Task 5 (classifier), §14d (CMS market share), §16 (per-CMS signatures), §19c (Cascade execution rules), §19d (Site-type recognition rubric), §19f (Methodology principles).

File structure

Path Responsibility
lib/factory/cms-detector.ts detectCms(samples, robotsTxt?): returns {cms, confidence, evidence} for the 12-CMS roster
lib/factory/classifier.ts 8-layer detection cascade: pure layer functions + classify(...) aggregator
lib/factory/site-type-rubric.ts Ordered {predicate, siteType} rules implementing the §19d rubric
lib/factory/__tests__/cms-detector.test.ts Unit tests, one happy-path per CMS + ambiguity + null cases
lib/factory/__tests__/classifier.test.ts Unit tests per layer + aggregator + LLM-fires-only-on-threshold + traffic-light bands
lib/factory/__tests__/site-type-rubric.test.ts Unit tests per predicate + ordering + first-match-wins

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (every public API + parsed JSON-LD + LLM JSON output); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful (a layer returning "no opinion" returns null).
  2. Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:cms-detector'); etc. No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
  3. Try/catch every fallible call. Every external call (LLM provider, JSON.parse on JSON-LD, cheerio parse on malformed HTML) wrapped. Catch specific error first, generic last. Always log before rethrow with full context.
  4. Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object. Cascade orchestrator may exceed 20 lines only if split into helpers (do the split).
  5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. Cascade verdicts are typed Verdict objects; CMS detector output is a typed CmsDetection.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock external services in unit tests. vi.mock the lib/llm/ provider for layer 8 tests. No real LLM calls in unit tests.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
  10. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.

Task 0: Type additions and module scaffolding

Files: - Modify: lib/factory/types.ts: add CmsId, CmsDetection, Verdict, Evidence, SiteType exports - Test: lib/factory/__tests__/types.test.ts: add tests for the new schemas

The cascade verdict and CMS-detection result both cross module boundaries (classifier ↔ cms-detector ↔ rubric ↔ Spec 06 configurator). Per CodingSOPs §6, they must be zod-typed.

  • [ ] Step 0.1: Write failing test for new schemas

Append to lib/factory/__tests__/types.test.ts:

import {
  CmsIdEnum,
  CmsDetectionSchema,
  VerdictSchema,
  EvidenceSchema,
  SiteTypeEnum,
} from '../types';

describe('CmsIdEnum', () => {
  it('accepts the 12 known CMS ids and "custom"', () => {
    const valid = [
      'wordpress', 'drupal', 'aem', 'shopify', 'hubspot', 'webflow',
      'squarespace', 'wix', 'salesforce-lwc', 'mediawiki', 'bedrock', 'custom',
    ];
    for (const id of valid) {
      expect(() => CmsIdEnum.parse(id)).not.toThrow();
    }
  });

  it('rejects an unknown CMS id', () => {
    expect(() => CmsIdEnum.parse('joomla')).toThrow();
  });
});

describe('EvidenceSchema', () => {
  it('accepts a minimal evidence record', () => {
    const valid = { kind: 'url-pattern', signal: '/wp-content/', sourceUrl: 'https://x.com/p/1' };
    expect(() => EvidenceSchema.parse(valid)).not.toThrow();
  });

  it('rejects empty signal', () => {
    expect(() => EvidenceSchema.parse({ kind: 'url-pattern', signal: '', sourceUrl: 'https://x.com' })).toThrow();
  });
});

describe('CmsDetectionSchema', () => {
  it('accepts a wordpress detection', () => {
    const valid = {
      cms: 'wordpress' as const,
      confidence: 0.9,
      evidence: [{ kind: 'url-pattern' as const, signal: '/wp-content/', sourceUrl: 'https://x.com/p/1' }],
    };
    expect(() => CmsDetectionSchema.parse(valid)).not.toThrow();
  });

  it('rejects confidence > 1', () => {
    const invalid = {
      cms: 'wordpress' as const,
      confidence: 1.5,
      evidence: [{ kind: 'url-pattern' as const, signal: '/wp-content/', sourceUrl: 'https://x.com' }],
    };
    expect(() => CmsDetectionSchema.parse(invalid)).toThrow();
  });
});

describe('VerdictSchema', () => {
  it('accepts a populated verdict', () => {
    const valid = {
      contentDomain: 'marketing' as const,
      confidence: 0.85,
      detectedVia: 'cms' as const,
      contributingLayers: ['cms', 'url'],
      schemaOrgType: 'BlogPosting',
    };
    expect(() => VerdictSchema.parse(valid)).not.toThrow();
  });
});

describe('SiteTypeEnum', () => {
  it('accepts the 13 documented site types', () => {
    const valid = [
      'wordpress', 'drupal-class', 'aem-enterprise', 'shopify',
      'salesforce-lwc', 'wikipedia-mediawiki', 'multi-brand-corporate',
      'massive-silo', 'dtc-retail', 'government', 'saas-b2b-docs', 'spa', 'custom',
    ];
    for (const id of valid) expect(() => SiteTypeEnum.parse(id)).not.toThrow();
  });
});
  • [ ] Step 0.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/types.test.ts -t CmsIdEnum

Expected: FAIL: exports CmsIdEnum, CmsDetectionSchema, VerdictSchema, EvidenceSchema, SiteTypeEnum don't exist.

  • [ ] Step 0.3: Add new schemas to lib/factory/types.ts

Append to lib/factory/types.ts (immediately after BlueprintSchema):

// ─── CMS detection (Spec 04) ────────────────────────────────────────────────

export const CmsIdEnum = z.enum([
  'wordpress',
  'drupal',
  'aem',
  'shopify',
  'hubspot',
  'webflow',
  'squarespace',
  'wix',
  'salesforce-lwc',
  'mediawiki',
  'bedrock',
  'custom',
]);
export type CmsId = z.infer<typeof CmsIdEnum>;

export const EvidenceSchema = z.object({
  kind: z.enum([
    'url-pattern',
    'html-class',
    'html-attr',
    'cookie',
    'asset-host',
    'robots-directive',
    'json-endpoint',
    'meta-tag',
    'json-ld-type',
    'microdata-itemtype',
    'og-type',
    'date-byline',
    'semantic-html',
    'llm-output',
  ]),
  signal: z.string().min(1),
  sourceUrl: z.string().url(),
});
export type Evidence = z.infer<typeof EvidenceSchema>;

export const CmsDetectionSchema = z.object({
  cms: CmsIdEnum,
  confidence: z.number().min(0).max(1),
  evidence: z.array(EvidenceSchema),
});
export type CmsDetection = z.infer<typeof CmsDetectionSchema>;

// ─── Cascade verdict (Spec 04) ──────────────────────────────────────────────

export const VerdictSchema = z.object({
  contentDomain: ContentDomainEnum,
  confidence: z.number().min(0).max(1),
  detectedVia: DetectedViaEnum,
  contributingLayers: z.array(z.string().min(1)),
  schemaOrgType: z.string().nullable().optional(),
});
export type Verdict = z.infer<typeof VerdictSchema>;

// ─── Site type rubric (Spec 04) ─────────────────────────────────────────────

export const SiteTypeEnum = z.enum([
  'wordpress',
  'drupal-class',
  'aem-enterprise',
  'shopify',
  'salesforce-lwc',
  'wikipedia-mediawiki',
  'multi-brand-corporate',
  'massive-silo',
  'dtc-retail',
  'government',
  'saas-b2b-docs',
  'spa',
  'custom',
]);
export type SiteType = z.infer<typeof SiteTypeEnum>;
  • [ ] Step 0.4: Run new schema tests
npx vitest run lib/factory/__tests__/types.test.ts -t 'CmsIdEnum|EvidenceSchema|CmsDetectionSchema|VerdictSchema|SiteTypeEnum'

Expected: all new tests pass.

  • [ ] Step 0.5: Run full project tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 0.6: Commit
git add lib/factory/types.ts lib/factory/__tests__/types.test.ts
git commit -m "feat(factory): add CmsDetection, Verdict, Evidence, SiteType zod schemas"

Task 1: CMS detector: detectCms(samples, robotsTxt?)

Files: - Create: lib/factory/cms-detector.ts - Create: lib/factory/__tests__/cms-detector.test.ts

The detector inspects up to 5 sample pages plus robots.txt and returns one CmsDetection. It checks signatures in priority order: most specific first: and returns on the first hit with confidence ≥ 0.80. If no signature hits, it returns { cms: 'custom', confidence: 0.0, evidence: [] }.

Per-CMS signatures (from §3b + §16):

CMS URL pattern HTML class/attr Asset host / cookie Other
wordpress /wp-content/, /wp-json/ wp-block-*, entry-*, post-* classes : :
drupal /themes/custom/, /sites/default/files/ region-*, node-*, field-*, data-drupal-* : Crawl-delay: 10 in robots.txt
aem /content/dam/, /content/{site}/{lang}/ data-cmp-* attrs : :
shopify /products/{handle}, /collections/ : cdn.shopify.com, Shopify cookies Shopify.theme global
hubspot /hubfs/ hs-cta-* classes : :
webflow : data-wf-* attrs : :
squarespace : sqs-block classes static1.squarespace.com :
wix : data-mesh-id attrs wixstatic.com, _wix* cookies :
salesforce-lwc : data-aura-*, lwc-* attrs : :
mediawiki /wiki/ mw-* classes : :
bedrock : : mozilla.org footer signature Mozilla-specific
  • [ ] Step 1.1: Write failing test for WordPress detection

Create lib/factory/__tests__/cms-detector.test.ts:

import { describe, it, expect } from 'vitest';
import { detectCms } from '../cms-detector';

const sample = (overrides: Partial<{ url: string; html: string; status: number }> = {}) => ({
  url: 'https://example.com/page',
  html: '<html><body></body></html>',
  status: 200,
  ...overrides,
});

describe('detectCms — WordPress', () => {
  it('detects WordPress via /wp-content/ path', async () => {
    const samples = [
      sample({ url: 'https://techcrunch.com/2026/04/30/post', html: '<img src="/wp-content/uploads/x.png">' }),
    ];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wordpress');
    expect(result.confidence).toBeGreaterThanOrEqual(0.8);
    expect(result.evidence.some((e) => e.signal.includes('/wp-content/'))).toBe(true);
  });

  it('detects WordPress via wp-block-* class', async () => {
    const samples = [
      sample({ html: '<div class="wp-block-heading">Hello</div>' }),
    ];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wordpress');
  });

  it('detects WordPress via /wp-json/ REST endpoint reference', async () => {
    const samples = [
      sample({ html: '<link rel="https://api.w.org/" href="https://x.com/wp-json/">' }),
    ];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wordpress');
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/cms-detector.test.ts

Expected: FAIL: Cannot find module '../cms-detector'.

  • [ ] Step 1.3: Implement lib/factory/cms-detector.ts: WordPress branch first

Create lib/factory/cms-detector.ts:

/**
 * CMS fingerprint detector — Layer 1 of the detection cascade (Spec 04).
 *
 * Why CMS-first: Web Almanac 2024 says 51% of sites use a CMS;
 * WordPress alone is 18% of the web. CMS signatures are stable,
 * hard to spoof, and tied to predictable URL→content-domain mappings
 * (see §16 playbooks). This is the most reliable single signal in the
 * cascade — see 00-Plan.md §3b + §14d.
 */

import { createLogger } from '../utils/logger';
import {
  CmsDetectionSchema,
  type CmsDetection,
  type CmsId,
  type Evidence,
} from './types';

const log = createLogger('factory:cms-detector');

export interface CmsDetectorSample {
  url: string;
  html: string;
  status: number;
}

interface SignatureHit {
  cms: CmsId;
  confidence: number;
  evidence: Evidence[];
}

/**
 * Detect the CMS powering a site by inspecting up to 5 sample pages
 * plus optional robots.txt body. Returns a single CmsDetection with
 * the highest-confidence hit, or `{cms:'custom', confidence:0}` on miss.
 */
export async function detectCms(
  samples: CmsDetectorSample[],
  robotsTxt?: string,
): Promise<CmsDetection> {
  const allHits: SignatureHit[] = [];
  for (const s of samples.slice(0, 5)) {
    allHits.push(...scanSample(s));
  }
  if (robotsTxt) allHits.push(...scanRobots(robotsTxt, samples[0]?.url ?? ''));

  if (allHits.length === 0) {
    return CmsDetectionSchema.parse({ cms: 'custom', confidence: 0, evidence: [] });
  }

  const best = pickBest(allHits);
  log.info(
    { event: 'detectCms', cms: best.cms, confidence: best.confidence, hits: allHits.length },
    'cms detected',
  );
  return CmsDetectionSchema.parse(best);
}

function pickBest(hits: SignatureHit[]): SignatureHit {
  const byCms = new Map<CmsId, SignatureHit>();
  for (const h of hits) {
    const existing = byCms.get(h.cms);
    if (!existing) {
      byCms.set(h.cms, { ...h, evidence: [...h.evidence] });
    } else {
      existing.confidence = Math.min(0.95, existing.confidence + 0.05);
      existing.evidence.push(...h.evidence);
    }
  }
  return [...byCms.values()].sort((a, b) => b.confidence - a.confidence)[0];
}

function scanSample(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  hits.push(...detectWordPress(s));
  hits.push(...detectDrupal(s));
  hits.push(...detectAem(s));
  hits.push(...detectShopify(s));
  hits.push(...detectHubspot(s));
  hits.push(...detectWebflow(s));
  hits.push(...detectSquarespace(s));
  hits.push(...detectWix(s));
  hits.push(...detectSalesforceLwc(s));
  hits.push(...detectMediaWiki(s));
  hits.push(...detectBedrock(s));
  return hits;
}

function scanRobots(body: string, sourceUrl: string): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/^\s*Crawl-delay:\s*10\s*$/im.test(body)) {
    hits.push({
      cms: 'drupal',
      confidence: 0.7,
      evidence: [{ kind: 'robots-directive', signal: 'Crawl-delay: 10', sourceUrl }],
    });
  }
  return hits;
}

function detectWordPress(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/\/wp-content\//.test(s.html) || /\/wp-content\//.test(s.url)) {
    hits.push(wpHit('url-pattern', '/wp-content/', s.url));
  }
  if (/\/wp-json\//.test(s.html)) {
    hits.push(wpHit('json-endpoint', '/wp-json/', s.url));
  }
  if (/class="[^"]*\bwp-block-[a-z0-9-]+/i.test(s.html)) {
    hits.push(wpHit('html-class', 'wp-block-*', s.url));
  }
  if (/class="[^"]*\bentry-(content|title|meta|date)\b/i.test(s.html)) {
    hits.push(wpHit('html-class', 'entry-*', s.url));
  }
  return hits;
}

function wpHit(kind: Evidence['kind'], signal: string, sourceUrl: string): SignatureHit {
  return { cms: 'wordpress', confidence: 0.85, evidence: [{ kind, signal, sourceUrl }] };
}

function detectDrupal(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/\/themes\/custom\//.test(s.html) || /\/sites\/default\/files\//.test(s.html)) {
    hits.push(drupalHit('url-pattern', '/sites/default/files/', s.url));
  }
  if (/class="[^"]*\b(region|node|field)-[a-z0-9-]+/i.test(s.html)) {
    hits.push(drupalHit('html-class', 'region-*/node-*/field-*', s.url));
  }
  if (/data-drupal-[a-z-]+="/i.test(s.html)) {
    hits.push(drupalHit('html-attr', 'data-drupal-*', s.url));
  }
  return hits;
}

function drupalHit(kind: Evidence['kind'], signal: string, sourceUrl: string): SignatureHit {
  return { cms: 'drupal', confidence: 0.85, evidence: [{ kind, signal, sourceUrl }] };
}

function detectAem(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/\/content\/dam\//.test(s.html) || /\/content\/dam\//.test(s.url)) {
    hits.push(aemHit('url-pattern', '/content/dam/', s.url));
  }
  if (/data-cmp-[a-z-]+="/i.test(s.html)) {
    hits.push(aemHit('html-attr', 'data-cmp-*', s.url));
  }
  if (/\/content\/[a-z0-9-]+\/[a-z]{2}(-[a-z]{2})?\//i.test(s.url)) {
    hits.push(aemHit('url-pattern', '/content/{site}/{lang}/', s.url));
  }
  return hits;
}

function aemHit(kind: Evidence['kind'], signal: string, sourceUrl: string): SignatureHit {
  return { cms: 'aem', confidence: 0.85, evidence: [{ kind, signal, sourceUrl }] };
}

function detectShopify(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/cdn\.shopify\.com/.test(s.html)) {
    hits.push(shopifyHit('asset-host', 'cdn.shopify.com', s.url));
  }
  if (/Shopify\.theme\b/.test(s.html) || /Shopify\.shop\b/.test(s.html)) {
    hits.push(shopifyHit('html-attr', 'Shopify.theme', s.url));
  }
  if (/\/products\/[a-z0-9-]+/.test(s.url) && /shopify/i.test(s.html)) {
    hits.push(shopifyHit('url-pattern', '/products/{handle}', s.url));
  }
  return hits;
}

function shopifyHit(kind: Evidence['kind'], signal: string, sourceUrl: string): SignatureHit {
  return { cms: 'shopify', confidence: 0.85, evidence: [{ kind, signal, sourceUrl }] };
}

function detectHubspot(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/\/hubfs\//.test(s.html)) {
    hits.push(hsHit('url-pattern', '/hubfs/', s.url));
  }
  if (/class="[^"]*\bhs-cta-[a-z0-9-]+/i.test(s.html)) {
    hits.push(hsHit('html-class', 'hs-cta-*', s.url));
  }
  return hits;
}

function hsHit(kind: Evidence['kind'], signal: string, sourceUrl: string): SignatureHit {
  return { cms: 'hubspot', confidence: 0.85, evidence: [{ kind, signal, sourceUrl }] };
}

function detectWebflow(s: CmsDetectorSample): SignatureHit[] {
  if (/data-wf-[a-z-]+="/i.test(s.html)) {
    return [{ cms: 'webflow', confidence: 0.85, evidence: [{ kind: 'html-attr', signal: 'data-wf-*', sourceUrl: s.url }] }];
  }
  return [];
}

function detectSquarespace(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/static1\.squarespace\.com/.test(s.html)) {
    hits.push({ cms: 'squarespace', confidence: 0.85, evidence: [{ kind: 'asset-host', signal: 'static1.squarespace.com', sourceUrl: s.url }] });
  }
  if (/class="[^"]*\bsqs-block\b/i.test(s.html)) {
    hits.push({ cms: 'squarespace', confidence: 0.85, evidence: [{ kind: 'html-class', signal: 'sqs-block', sourceUrl: s.url }] });
  }
  return hits;
}

function detectWix(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/wixstatic\.com/.test(s.html)) {
    hits.push({ cms: 'wix', confidence: 0.85, evidence: [{ kind: 'asset-host', signal: 'wixstatic.com', sourceUrl: s.url }] });
  }
  if (/data-mesh-id="/i.test(s.html)) {
    hits.push({ cms: 'wix', confidence: 0.85, evidence: [{ kind: 'html-attr', signal: 'data-mesh-id', sourceUrl: s.url }] });
  }
  return hits;
}

function detectSalesforceLwc(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/data-aura-[a-z-]+="/i.test(s.html)) {
    hits.push({ cms: 'salesforce-lwc', confidence: 0.85, evidence: [{ kind: 'html-attr', signal: 'data-aura-*', sourceUrl: s.url }] });
  }
  if (/\blwc-[a-z0-9-]+="/i.test(s.html)) {
    hits.push({ cms: 'salesforce-lwc', confidence: 0.85, evidence: [{ kind: 'html-attr', signal: 'lwc-*', sourceUrl: s.url }] });
  }
  return hits;
}

function detectMediaWiki(s: CmsDetectorSample): SignatureHit[] {
  const hits: SignatureHit[] = [];
  if (/class="[^"]*\bmw-[a-z0-9-]+/i.test(s.html)) {
    hits.push({ cms: 'mediawiki', confidence: 0.85, evidence: [{ kind: 'html-class', signal: 'mw-*', sourceUrl: s.url }] });
  }
  if (/\/wiki\//.test(s.url)) {
    hits.push({ cms: 'mediawiki', confidence: 0.6, evidence: [{ kind: 'url-pattern', signal: '/wiki/', sourceUrl: s.url }] });
  }
  return hits;
}

function detectBedrock(s: CmsDetectorSample): SignatureHit[] {
  if (/mozilla\.org/i.test(s.html) && /class="[^"]*\bmzp-/i.test(s.html)) {
    return [{ cms: 'bedrock', confidence: 0.85, evidence: [{ kind: 'html-class', signal: 'mzp-*', sourceUrl: s.url }] }];
  }
  return [];
}
  • [ ] Step 1.4: Run WordPress tests, expect PASS
npx vitest run lib/factory/__tests__/cms-detector.test.ts -t WordPress

Expected: 3 tests pass.

  • [ ] Step 1.5: Add tests for the remaining 10 CMS branches + the "custom" fallback + robots.txt drupal hit + ambiguity boost

Append to lib/factory/__tests__/cms-detector.test.ts:

describe('detectCms — Drupal', () => {
  it('detects Drupal via region-* + node-* classes', async () => {
    const samples = [sample({ html: '<div class="region-content"><article class="node-page">x</article></div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('drupal');
  });

  it('boosts Drupal confidence when robots.txt has Crawl-delay: 10', async () => {
    const samples = [sample({ html: '<div class="field-name-body">x</div>' })];
    const robots = 'User-agent: *\nCrawl-delay: 10\n';
    const result = await detectCms(samples, robots);
    expect(result.cms).toBe('drupal');
    expect(result.evidence.some((e) => e.kind === 'robots-directive')).toBe(true);
    expect(result.confidence).toBeGreaterThan(0.85);
  });
});

describe('detectCms — AEM', () => {
  it('detects AEM via /content/dam/', async () => {
    const samples = [sample({ html: '<img src="/content/dam/hp/products/laptop.png">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('aem');
  });

  it('detects AEM via data-cmp-* attrs', async () => {
    const samples = [sample({ html: '<div data-cmp-is="text">copy</div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('aem');
  });
});

describe('detectCms — Shopify', () => {
  it('detects Shopify via cdn.shopify.com', async () => {
    const samples = [sample({ html: '<img src="https://cdn.shopify.com/x.png">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('shopify');
  });

  it('detects Shopify via Shopify.theme global', async () => {
    const samples = [sample({ html: '<script>window.Shopify = {theme:"dawn"};</script>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('shopify');
  });
});

describe('detectCms — Hubspot', () => {
  it('detects via /hubfs/', async () => {
    const samples = [sample({ html: '<img src="/hubfs/logo.png">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('hubspot');
  });
});

describe('detectCms — Webflow', () => {
  it('detects via data-wf-* attrs', async () => {
    const samples = [sample({ html: '<div data-wf-page="abc">x</div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('webflow');
  });
});

describe('detectCms — Squarespace', () => {
  it('detects via static1.squarespace.com asset host', async () => {
    const samples = [sample({ html: '<img src="https://static1.squarespace.com/x.jpg">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('squarespace');
  });
});

describe('detectCms — Wix', () => {
  it('detects via wixstatic.com', async () => {
    const samples = [sample({ html: '<img src="https://x.wixstatic.com/y.png">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wix');
  });

  it('detects via data-mesh-id', async () => {
    const samples = [sample({ html: '<section data-mesh-id="comp-1">x</section>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wix');
  });
});

describe('detectCms — Salesforce LWC', () => {
  it('detects via data-aura-*', async () => {
    const samples = [sample({ html: '<div data-aura-rendered-by="123">x</div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('salesforce-lwc');
  });

  it('detects via lwc-* attrs', async () => {
    const samples = [sample({ html: '<input lwc-2g3rk="">' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('salesforce-lwc');
  });
});

describe('detectCms — MediaWiki', () => {
  it('detects Wikipedia via mw-* + /wiki/', async () => {
    const samples = [sample({ url: 'https://en.wikipedia.org/wiki/Algolia', html: '<div class="mw-parser-output">x</div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('mediawiki');
  });
});

describe('detectCms — Bedrock (Mozilla)', () => {
  it('detects via mozilla.org + mzp-* class', async () => {
    const samples = [sample({ url: 'https://www.mozilla.org/en-US/firefox/', html: '<div class="mzp-c-hero"><a href="https://mozilla.org">x</a></div>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('bedrock');
  });
});

describe('detectCms — custom fallback', () => {
  it('returns cms=custom and confidence=0 when nothing matches', async () => {
    const samples = [sample({ html: '<html><body><h1>plain</h1></body></html>' })];
    const result = await detectCms(samples);
    expect(result.cms).toBe('custom');
    expect(result.confidence).toBe(0);
    expect(result.evidence).toEqual([]);
  });
});

describe('detectCms — multi-signal boost', () => {
  it('boosts confidence when multiple WordPress signals appear across samples', async () => {
    const samples = [
      sample({ url: 'https://x.com/p/1', html: '<div class="wp-block-group">x</div>' }),
      sample({ url: 'https://x.com/p/2', html: '<img src="/wp-content/uploads/y.png">' }),
      sample({ url: 'https://x.com/p/3', html: '<link href="/wp-json/" rel="https://api.w.org/">' }),
    ];
    const result = await detectCms(samples);
    expect(result.cms).toBe('wordpress');
    expect(result.confidence).toBeGreaterThan(0.85);
    expect(result.evidence.length).toBeGreaterThanOrEqual(3);
  });
});
  • [ ] Step 1.6: Run all CMS detector tests
npx vitest run lib/factory/__tests__/cms-detector.test.ts

Expected: all tests pass.

  • [ ] Step 1.7: Run tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 1.8: Commit
git add lib/factory/cms-detector.ts lib/factory/__tests__/cms-detector.test.ts
git commit -m "feat(factory): CMS fingerprint detector for 11 known platforms + custom fallback"

Task 2: Classifier: pure layer functions (Layers 1–7)

Files: - Create: lib/factory/classifier.ts - Create: lib/factory/__tests__/classifier.test.ts

Each layer is a pure, deterministic function. Layers 1–7 do NOT call the LLM. Layer 8 lives in Task 3 (separated for testability: Task 2 can be fully validated without mocking the LLM provider).

Layer signatures (all are pure functions):

Layer Function Input Output
1 classifyByCms(cms, url) CmsId, string Verdict \| null
2 classifyByUrl(url) string Verdict \| null
3 classifyByJsonLd(html, url) string, string Verdict \| null
4 classifyByOpenGraph(html, url) string, string Verdict \| null
5 classifyByMicrodata(html, url) string, string Verdict \| null
6 classifyByDateAuthor(html, url) string, string Verdict \| null
7 classifyBySemanticHtml(html, url) string, string Verdict \| null

URL-pattern table (Layer 2): full set per §3b:

URL token Domain Confidence
/blog/, /news/, /articles/, /insights/, /{YYYY}/{MM}/ marketing 0.70
/customers/, /case-studies/, /success-stories/, /our-customers/ customer-stories 0.75
/docs/, /documentation/, /developer/, /api/, /reference/, /guides/ technical 0.75
/support/, /help/, /faq/, /troubleshooting/, /hc/ support 0.75
/courses/, /learn/, /academy/, /training/, /tutorials/ education 0.70
/products/, /services/, /solutions/, /offerings/, /p/, /t/ product-catalog 0.65
/events/, /webinars/, /conferences/, /calendar/ events 0.70
/legal/, /terms/, /privacy/, /policies/ legal 0.75
/community/, /forum/, /discuss/ social 0.60
  • [ ] Step 2.1: Write failing test for classifyByJsonLd

Create lib/factory/__tests__/classifier.test.ts:

import { describe, it, expect } from 'vitest';
import {
  classifyByCms,
  classifyByUrl,
  classifyByJsonLd,
  classifyByOpenGraph,
  classifyByMicrodata,
  classifyByDateAuthor,
  classifyBySemanticHtml,
} from '../classifier';

const URL = 'https://example.com/page';

describe('classifyByJsonLd', () => {
  it('returns marketing for BlogPosting', () => {
    const html = '<script type="application/ld+json">{"@type":"BlogPosting","headline":"x"}</script>';
    const v = classifyByJsonLd(html, URL);
    expect(v).not.toBeNull();
    expect(v!.contentDomain).toBe('marketing');
    expect(v!.detectedVia).toBe('json-ld');
    expect(v!.confidence).toBeGreaterThanOrEqual(0.85);
    expect(v!.schemaOrgType).toBe('BlogPosting');
  });

  it('returns technical for SoftwareSourceCode', () => {
    const html = '<script type="application/ld+json">{"@type":"SoftwareSourceCode"}</script>';
    const v = classifyByJsonLd(html, URL);
    expect(v!.contentDomain).toBe('technical');
  });

  it('returns null when no JSON-LD present', () => {
    const html = '<html><body>no schema</body></html>';
    expect(classifyByJsonLd(html, URL)).toBeNull();
  });

  it('returns null when JSON-LD @type is unmapped', () => {
    const html = '<script type="application/ld+json">{"@type":"WebPage"}</script>';
    expect(classifyByJsonLd(html, URL)).toBeNull();
  });

  it('handles malformed JSON-LD gracefully (returns null)', () => {
    const html = '<script type="application/ld+json">{not valid json}</script>';
    expect(classifyByJsonLd(html, URL)).toBeNull();
  });
});
  • [ ] Step 2.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/classifier.test.ts

Expected: FAIL: Cannot find module '../classifier'.

  • [ ] Step 2.3: Implement lib/factory/classifier.ts: Layers 1–7 + skeleton for orchestrator

Create lib/factory/classifier.ts:

/**
 * Detection cascade — 8-layer content-domain classifier.
 *
 * Per 00-Plan.md §3b + §4b + §19c:
 *   CMS+URL is the foundation (~95% combined coverage).
 *   JSON-LD enriches but never leads (Web Almanac 2024: 41% global,
 *   ~7% useful @types).
 *   LLM is the last-resort tiebreaker, fired only when combined
 *   layers 1–7 confidence < 0.55.
 */

import * as cheerio from 'cheerio';
import { z } from 'zod';
import { createLogger } from '../utils/logger';
import { dssBySchemaOrgType, listContentDomains, DSS, type ContentDomain } from './dss';
import { getLLMProvider } from '../llm';
import { type CmsId, type Verdict } from './types';

const log = createLogger('factory:classifier');

// ─── Layer 1: CMS-aware URL hint ────────────────────────────────────────────

/**
 * Layer 1 — Given a detected CMS and a URL, return the typical content-domain
 * hint that CMS's URL conventions imply. Confidence is moderate (0.60–0.75)
 * because the CMS narrows the prior even when the URL alone is ambiguous.
 */
export function classifyByCms(cms: CmsId, url: string): Verdict | null {
  if (cms === 'custom' || cms === 'bedrock') return null;
  const urlVerdict = classifyByUrl(url);
  if (!urlVerdict) return null;
  return {
    contentDomain: urlVerdict.contentDomain,
    confidence: Math.min(0.85, urlVerdict.confidence + 0.10),
    detectedVia: 'cms',
    contributingLayers: ['cms', 'url'],
    schemaOrgType: null,
  };
}

// ─── Layer 2: URL pattern (universal floor) ─────────────────────────────────

interface UrlRule {
  patterns: RegExp[];
  domain: ContentDomain;
  confidence: number;
}

const URL_RULES: UrlRule[] = [
  { patterns: [/\/customers\//, /\/case-stud(y|ies)\//, /\/success-stor(y|ies)\//, /\/our-customers\//], domain: 'customer-stories', confidence: 0.75 },
  { patterns: [/\/docs\//, /\/documentation\//, /\/developer\//, /\/api\//, /\/reference\//, /\/guides\//], domain: 'technical', confidence: 0.75 },
  { patterns: [/\/support\//, /\/help\//, /\/faq\//, /\/troubleshooting\//, /\/hc\//], domain: 'support', confidence: 0.75 },
  { patterns: [/\/legal\//, /\/terms\//, /\/privacy\//, /\/policies\//], domain: 'legal', confidence: 0.75 },
  { patterns: [/\/courses\//, /\/learn\//, /\/academy\//, /\/training\//, /\/tutorials\//], domain: 'education', confidence: 0.70 },
  { patterns: [/\/blog\//, /\/news\//, /\/articles\//, /\/insights\//, /\/\d{4}\/\d{2}\//], domain: 'marketing', confidence: 0.70 },
  { patterns: [/\/events\//, /\/webinars\//, /\/conferences\//, /\/calendar\//], domain: 'events', confidence: 0.70 },
  { patterns: [/\/products\//, /\/services\//, /\/solutions\//, /\/offerings\//, /\/p\//, /\/t\//], domain: 'product-catalog', confidence: 0.65 },
  { patterns: [/\/community\//, /\/forum\//, /\/discuss\//], domain: 'social', confidence: 0.60 },
];

/**
 * Layer 2 — URL path pattern match. Universal floor: present on 100% of pages.
 * Returns the FIRST matching rule (priority order = list order).
 */
export function classifyByUrl(url: string): Verdict | null {
  let path: string;
  try {
    path = new URL(url).pathname;
  } catch {
    return null;
  }
  for (const rule of URL_RULES) {
    if (rule.patterns.some((p) => p.test(path))) {
      return {
        contentDomain: rule.domain,
        confidence: rule.confidence,
        detectedVia: 'url',
        contributingLayers: ['url'],
        schemaOrgType: null,
      };
    }
  }
  return null;
}

// ─── Layer 3: JSON-LD ───────────────────────────────────────────────────────

/**
 * Layer 3 — Parse <script type="application/ld+json"> blocks, walk @type,
 * and look up via DSS. Returns null on malformed JSON or unmapped types.
 */
export function classifyByJsonLd(html: string, url: string): Verdict | null {
  const types = extractJsonLdTypes(html);
  for (const t of types) {
    const domain = dssBySchemaOrgType(t);
    if (domain) {
      return {
        contentDomain: domain,
        confidence: 0.90,
        detectedVia: 'json-ld',
        contributingLayers: ['json-ld'],
        schemaOrgType: t,
      };
    }
  }
  return null;
}

function extractJsonLdTypes(html: string): string[] {
  const types: string[] = [];
  try {
    const $ = cheerio.load(html);
    $('script[type="application/ld+json"]').each((_, el) => {
      const raw = $(el).text().trim();
      if (!raw) return;
      try {
        const parsed = JSON.parse(raw) as unknown;
        types.push(...collectTypes(parsed));
      } catch (err) {
        log.warn({ event: 'jsonld_parse_failed', url, err: String(err) }, 'malformed JSON-LD');
      }
    });
  } catch (err) {
    log.warn({ event: 'jsonld_cheerio_failed', url, err: String(err) }, 'cheerio failed on HTML');
  }
  return types;
}

function collectTypes(node: unknown): string[] {
  const out: string[] = [];
  if (!node) return out;
  if (Array.isArray(node)) {
    for (const item of node) out.push(...collectTypes(item));
    return out;
  }
  if (typeof node === 'object') {
    const obj = node as Record<string, unknown>;
    const t = obj['@type'];
    if (typeof t === 'string') out.push(t);
    if (Array.isArray(t)) for (const v of t) if (typeof v === 'string') out.push(v);
    if (Array.isArray(obj['@graph'])) out.push(...collectTypes(obj['@graph']));
  }
  return out;
}

// ─── Layer 4: OpenGraph ─────────────────────────────────────────────────────

/**
 * Layer 4 — og:type. Coarse but reliable: only article/website/video/product values.
 */
export function classifyByOpenGraph(html: string, url: string): Verdict | null {
  let ogType: string | null = null;
  try {
    const $ = cheerio.load(html);
    ogType = $('meta[property="og:type"]').attr('content') ?? null;
  } catch (err) {
    log.warn({ event: 'og_parse_failed', url, err: String(err) }, 'cheerio failed');
    return null;
  }
  if (!ogType) return null;
  const map: Record<string, ContentDomain> = {
    article: 'marketing',
    'article.news': 'marketing',
    'video.other': 'social',
    video: 'social',
    product: 'product-catalog',
    'product.item': 'product-catalog',
    book: 'education',
    course: 'education',
  };
  const domain = map[ogType.toLowerCase()];
  if (!domain) return null;
  return {
    contentDomain: domain,
    confidence: 0.65,
    detectedVia: 'opengraph',
    contributingLayers: ['opengraph'],
    schemaOrgType: null,
  };
}

// ─── Layer 5: Microdata + RDFa ──────────────────────────────────────────────

/**
 * Layer 5 — itemtype="https://schema.org/X". Legacy but still present
 * on 26% of pages (Web Almanac 2024). Lower confidence than JSON-LD.
 */
export function classifyByMicrodata(html: string, url: string): Verdict | null {
  let itemType: string | null = null;
  try {
    const $ = cheerio.load(html);
    const raw = $('[itemtype*="schema.org/"]').first().attr('itemtype');
    if (raw) {
      const match = raw.match(/schema\.org\/([A-Za-z]+)/);
      itemType = match ? match[1] : null;
    }
  } catch (err) {
    log.warn({ event: 'microdata_parse_failed', url, err: String(err) }, 'cheerio failed');
    return null;
  }
  if (!itemType) return null;
  const domain = dssBySchemaOrgType(itemType);
  if (!domain) return null;
  return {
    contentDomain: domain,
    confidence: 0.60,
    detectedVia: 'microdata',
    contributingLayers: ['microdata'],
    schemaOrgType: itemType,
  };
}

// ─── Layer 6: Date + author regex (article-family fallback) ─────────────────

const DATE_RE = /(?:Published|Posted|Updated)[:\s]+\d{4}-\d{2}-\d{2}|\b\d{4}-\d{2}-\d{2}\b|<time\b[^>]*datetime=/i;
const AUTHOR_RE = /\bby\s+[A-Z][a-z]+\s+[A-Z][a-z]+|class="[^"]*\b(byline|author|entry-author)\b/i;

/**
 * Layer 6 — "dated content with byline" → article-family. Universal: present
 * on ~99% of articles. Confidence 0.55: enough to confirm "marketing"
 * when combined with another layer but not enough to stand alone.
 */
export function classifyByDateAuthor(html: string, url: string): Verdict | null {
  if (!DATE_RE.test(html) || !AUTHOR_RE.test(html)) return null;
  return {
    contentDomain: 'marketing',
    confidence: 0.55,
    detectedVia: 'heuristic',
    contributingLayers: ['date-author'],
    schemaOrgType: null,
  };
}

// ─── Layer 7: Semantic HTML heuristics ──────────────────────────────────────

/**
 * Layer 7 — <article>+breadcrumb+code-density. Web Almanac 2024 says
 * <article> adoption is ~5%, so this is a weak signal. Confidence 0.50.
 */
export function classifyBySemanticHtml(html: string, url: string): Verdict | null {
  let domain: ContentDomain | null = null;
  try {
    const $ = cheerio.load(html);
    const hasArticle = $('article').length > 0;
    const codeBlocks = $('pre code, pre.code, .codeblock, .highlight').length;
    const hasBreadcrumb = $('[class*="breadcrumb"], nav[aria-label*="breadcrumb" i]').length > 0;
    if (hasArticle && codeBlocks >= 2) domain = 'technical';
    else if (hasArticle && hasBreadcrumb) domain = 'marketing';
    else if (hasArticle) domain = 'marketing';
  } catch (err) {
    log.warn({ event: 'semantic_parse_failed', url, err: String(err) }, 'cheerio failed');
    return null;
  }
  if (!domain) return null;
  return {
    contentDomain: domain,
    confidence: 0.50,
    detectedVia: 'heuristic',
    contributingLayers: ['semantic-html'],
    schemaOrgType: null,
  };
}
  • [ ] Step 2.4: Run JSON-LD tests, expect PASS
npx vitest run lib/factory/__tests__/classifier.test.ts -t classifyByJsonLd

Expected: 5 tests pass.

  • [ ] Step 2.5: Add tests for Layers 1, 2, 4, 5, 6, 7

Append to lib/factory/__tests__/classifier.test.ts:

describe('classifyByCms', () => {
  it('returns null for cms=custom (cannot infer from CMS alone)', () => {
    expect(classifyByCms('custom', 'https://x.com/blog/post')).toBeNull();
  });

  it('returns null for cms=bedrock (Mozilla-specific, no URL convention)', () => {
    expect(classifyByCms('bedrock', 'https://www.mozilla.org/en-US/firefox/')).toBeNull();
  });

  it('returns boosted marketing verdict for wordpress + /blog/ URL', () => {
    const v = classifyByCms('wordpress', 'https://x.com/blog/hello');
    expect(v).not.toBeNull();
    expect(v!.contentDomain).toBe('marketing');
    expect(v!.detectedVia).toBe('cms');
    expect(v!.confidence).toBeGreaterThan(0.70);
  });

  it('returns null when CMS is set but URL is uninformative', () => {
    const v = classifyByCms('wordpress', 'https://x.com/');
    expect(v).toBeNull();
  });
});

describe('classifyByUrl', () => {
  it('matches /docs/ → technical', () => {
    const v = classifyByUrl('https://x.com/docs/api');
    expect(v!.contentDomain).toBe('technical');
    expect(v!.confidence).toBe(0.75);
  });

  it('matches /customers/ → customer-stories', () => {
    expect(classifyByUrl('https://x.com/customers/acme')!.contentDomain).toBe('customer-stories');
  });

  it('matches date-shaped /YYYY/MM/ → marketing', () => {
    expect(classifyByUrl('https://x.com/2026/04/30/post')!.contentDomain).toBe('marketing');
  });

  it('matches /help/ → support', () => {
    expect(classifyByUrl('https://x.com/help/topic')!.contentDomain).toBe('support');
  });

  it('matches /products/ → product-catalog', () => {
    expect(classifyByUrl('https://x.com/products/widget')!.contentDomain).toBe('product-catalog');
  });

  it('returns null for non-matching URL', () => {
    expect(classifyByUrl('https://x.com/about-us')).toBeNull();
  });

  it('returns null for malformed URL', () => {
    expect(classifyByUrl('not a url')).toBeNull();
  });

  it('priority: /customers/ wins over /products/ when both appear (customer-stories first)', () => {
    expect(classifyByUrl('https://x.com/customers/acme/products/x')!.contentDomain).toBe('customer-stories');
  });
});

describe('classifyByOpenGraph', () => {
  it('maps og:type=article → marketing', () => {
    const html = '<meta property="og:type" content="article">';
    expect(classifyByOpenGraph(html, URL)!.contentDomain).toBe('marketing');
  });

  it('maps og:type=product → product-catalog', () => {
    const html = '<meta property="og:type" content="product">';
    expect(classifyByOpenGraph(html, URL)!.contentDomain).toBe('product-catalog');
  });

  it('returns null for og:type=website (uninformative)', () => {
    const html = '<meta property="og:type" content="website">';
    expect(classifyByOpenGraph(html, URL)).toBeNull();
  });

  it('returns null when og:type is missing', () => {
    expect(classifyByOpenGraph('<html></html>', URL)).toBeNull();
  });
});

describe('classifyByMicrodata', () => {
  it('maps itemtype=schema.org/TechArticle → technical or support per DSS', () => {
    const html = '<div itemscope itemtype="https://schema.org/TechArticle"></div>';
    const v = classifyByMicrodata(html, URL);
    expect(v).not.toBeNull();
    expect(v!.detectedVia).toBe('microdata');
  });

  it('maps itemtype=schema.org/Course → education', () => {
    const html = '<div itemscope itemtype="https://schema.org/Course"></div>';
    expect(classifyByMicrodata(html, URL)!.contentDomain).toBe('education');
  });

  it('returns null when itemtype is unmapped', () => {
    const html = '<div itemscope itemtype="https://schema.org/WebPage"></div>';
    expect(classifyByMicrodata(html, URL)).toBeNull();
  });

  it('returns null when no microdata present', () => {
    expect(classifyByMicrodata('<html></html>', URL)).toBeNull();
  });
});

describe('classifyByDateAuthor', () => {
  it('matches "Published: 2026-04-30" + byline → marketing@0.55', () => {
    const html = '<p>Published: 2026-04-30 by Jane Doe</p>';
    const v = classifyByDateAuthor(html, URL);
    expect(v!.contentDomain).toBe('marketing');
    expect(v!.confidence).toBeCloseTo(0.55);
  });

  it('matches <time datetime=...> + class="byline"', () => {
    const html = '<time datetime="2026-04-30">x</time><span class="byline">Jane</span>';
    expect(classifyByDateAuthor(html, URL)).not.toBeNull();
  });

  it('returns null when only date present (no byline)', () => {
    expect(classifyByDateAuthor('<p>Posted: 2026-04-30</p>', URL)).toBeNull();
  });

  it('returns null when only byline present (no date)', () => {
    expect(classifyByDateAuthor('<span class="byline">Jane</span>', URL)).toBeNull();
  });
});

describe('classifyBySemanticHtml', () => {
  it('returns technical when <article> + multiple <pre><code> blocks', () => {
    const html = '<article><pre><code>x</code></pre><pre><code>y</code></pre></article>';
    expect(classifyBySemanticHtml(html, URL)!.contentDomain).toBe('technical');
  });

  it('returns marketing when <article> + breadcrumb', () => {
    const html = '<article>x</article><nav aria-label="Breadcrumb">y</nav>';
    expect(classifyBySemanticHtml(html, URL)!.contentDomain).toBe('marketing');
  });

  it('returns marketing for plain <article>', () => {
    expect(classifyBySemanticHtml('<article>x</article>', URL)!.contentDomain).toBe('marketing');
  });

  it('returns null when no <article>', () => {
    expect(classifyBySemanticHtml('<div>x</div>', URL)).toBeNull();
  });
});
  • [ ] Step 2.6: Run all classifier layer tests
npx vitest run lib/factory/__tests__/classifier.test.ts

Expected: all Layer 1–7 tests pass.

  • [ ] Step 2.7: Run tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 2.8: Commit
git add lib/factory/classifier.ts lib/factory/__tests__/classifier.test.ts
git commit -m "feat(factory): cascade layers 1-7 (CMS, URL, JSON-LD, OG, microdata, date+author, semantic HTML)"

Task 3: Layer 8: LLM tiebreaker + cascade aggregator

Files: - Modify: lib/factory/classifier.ts: add classifyWithLlm + classify orchestrator - Modify: lib/factory/__tests__/classifier.test.ts: add LLM + aggregator tests with mocked provider

The LLM layer is in a separate task because it (a) requires mocking and (b) MUST only fire when combined layers 1–7 confidence < 0.55. That gating logic lives in the aggregator.

Aggregator algorithm (from §19c):

candidates = group {layer→Verdict|null} results by contentDomain
for each candidate:
  base_conf      = max(layer.conf for layers voting this candidate)
  agreement_boost = 0.05 × (count_distinct_layers_agreeing − 1)
  final_conf      = min(0.95, base_conf + agreement_boost)
pick top candidate by final_conf

if final_conf < 0.55 across all candidates:
  fire layer 8 (LLM) — winner becomes its result

Traffic-light bands (from §4b):

Band Range UI behavior
green ≥ 0.85 auto-accept, recommend Configure
yellow 0.65 – 0.84 manual review
amber < 0.65 manual override required
  • [ ] Step 3.1: Write failing test for classifyWithLlm and classify aggregator

Append to lib/factory/__tests__/classifier.test.ts:

import { vi, beforeEach } from 'vitest';

// Mock the LLM provider singleton
const mockGenerate = vi.fn();
vi.mock('../../llm', () => ({
  getLLMProvider: () => ({
    name: 'mock',
    capabilities: {
      supportsJsonMode: true,
      supportsJsonSchema: true,
      supportsSystemInstruction: true,
      supportsStreaming: false,
    },
    generateContent: mockGenerate,
    startChat: () => { throw new Error('not used'); },
  }),
  resetLLMProvider: () => {},
}));

import { classifyWithLlm, classify } from '../classifier';

describe('classifyWithLlm', () => {
  beforeEach(() => mockGenerate.mockReset());

  it('returns the LLM-picked domain at confidence 0.75', async () => {
    mockGenerate.mockResolvedValue({
      text: () => '',
      json: () => ({ contentDomain: 'support', confidence: 0.75 }),
    });
    const v = await classifyWithLlm('<html>some help text</html>', 'https://x.com/q');
    expect(v).not.toBeNull();
    expect(v!.contentDomain).toBe('support');
    expect(v!.detectedVia).toBe('llm');
    expect(v!.confidence).toBe(0.75);
  });

  it('returns null when LLM returns invalid JSON shape', async () => {
    mockGenerate.mockResolvedValue({
      text: () => '',
      json: () => ({ wrongKey: 'oops' }),
    });
    const v = await classifyWithLlm('<html></html>', 'https://x.com');
    expect(v).toBeNull();
  });

  it('returns null when LLM throws', async () => {
    mockGenerate.mockRejectedValue(new Error('rate limited'));
    const v = await classifyWithLlm('<html></html>', 'https://x.com');
    expect(v).toBeNull();
  });
});

describe('classify (aggregator)', () => {
  beforeEach(() => mockGenerate.mockReset());

  it('green band: 3 layers agreeing on marketing → confidence ≥ 0.85', async () => {
    const html = `
      <script type="application/ld+json">{"@type":"BlogPosting"}</script>
      <meta property="og:type" content="article">
      <article><time datetime="2026-04-30">x</time><span class="byline">By Jane Doe</span></article>
    `;
    const v = await classify(html, 'https://x.com/blog/post', { cms: 'wordpress' });
    expect(v.contentDomain).toBe('marketing');
    expect(v.confidence).toBeGreaterThanOrEqual(0.85);
    expect(v.contributingLayers.length).toBeGreaterThanOrEqual(3);
    expect(mockGenerate).not.toHaveBeenCalled();
  });

  it('yellow band: 1 layer alone → confidence in 0.65–0.84', async () => {
    const html = '<html><body><a href="/help/">help</a></body></html>';
    const v = await classify(html, 'https://x.com/help/topic', { cms: 'custom' });
    expect(v.contentDomain).toBe('support');
    expect(v.confidence).toBeGreaterThanOrEqual(0.65);
    expect(v.confidence).toBeLessThan(0.85);
    expect(mockGenerate).not.toHaveBeenCalled();
  });

  it('LLM fires only when combined layers 1-7 confidence < 0.55', async () => {
    mockGenerate.mockResolvedValue({
      text: () => '',
      json: () => ({ contentDomain: 'support', confidence: 0.70 }),
    });
    // No URL signal, no JSON-LD, no OG, no microdata, no date+author, no <article>, custom CMS
    const v = await classify('<html><body>plain text</body></html>', 'https://x.com/about-us', { cms: 'custom' });
    expect(mockGenerate).toHaveBeenCalledTimes(1);
    expect(v.contentDomain).toBe('support');
    expect(v.detectedVia).toBe('llm');
  });

  it('LLM does NOT fire when layer 1-7 already produced ≥ 0.55', async () => {
    const html = '<script type="application/ld+json">{"@type":"Course"}</script>';
    const v = await classify(html, 'https://x.com/learn/x', { cms: 'custom' });
    expect(v.contentDomain).toBe('education');
    expect(mockGenerate).not.toHaveBeenCalled();
  });

  it('amber band: LLM declined and no signals → returns lowest-confidence verdict (< 0.65)', async () => {
    mockGenerate.mockResolvedValue({ text: () => '', json: () => ({ wrongShape: true }) });
    const v = await classify('<html></html>', 'https://x.com/about', { cms: 'custom' });
    expect(v.confidence).toBeLessThan(0.65);
  });

  it('agreement boost: each agreeing layer adds 0.05, capped at 0.95', async () => {
    // 4 layers all pointing at marketing
    const html = `
      <script type="application/ld+json">{"@type":"NewsArticle"}</script>
      <meta property="og:type" content="article">
      <div itemscope itemtype="https://schema.org/Article"></div>
      <article><time datetime="2026-04-30">x</time><span class="byline">By Jane</span></article>
    `;
    const v = await classify(html, 'https://x.com/news/post', { cms: 'wordpress' });
    expect(v.confidence).toBeLessThanOrEqual(0.95);
    expect(v.confidence).toBeGreaterThanOrEqual(0.90);
    expect(v.contributingLayers.length).toBeGreaterThanOrEqual(4);
  });
});
  • [ ] Step 3.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/classifier.test.ts -t 'classifyWithLlm|classify \(aggregator\)'

Expected: FAIL: classifyWithLlm and classify not exported.

  • [ ] Step 3.3: Add Layer 8 + aggregator to lib/factory/classifier.ts

Append to lib/factory/classifier.ts:

// ─── Layer 8: LLM tiebreaker ────────────────────────────────────────────────
// (imports already hoisted to top of file at Step 2.3 — z, getLLMProvider,
//  listContentDomains, DSS are all in scope. CodingSOPs §9: imports at top.)

const LLM_RESPONSE_SCHEMA = z.object({
  contentDomain: z.enum([
    'marketing', 'support', 'education', 'technical',
    'customer-stories', 'product-catalog', 'events', 'legal', 'social',
  ]),
  confidence: z.number().min(0).max(1),
});

function buildLlmPrompt(): string {
  const lines = listContentDomains().map((d) => `- ${d}: ${DSS[d].description}`).join('\n');
  return [
    'You are a content classifier. Given a URL and the first 4KB of HTML body text, pick the single best content domain.',
    'Return ONLY valid JSON: {"contentDomain": "<one of below>", "confidence": <0-1>}.',
    'Domains:',
    lines,
  ].join('\n');
}

/**
 * Layer 8 — LLM fallback. Fires only when combined layers 1–7 confidence < 0.55.
 * Uses the existing lib/llm/ provider in JSON mode. Returns null on any failure.
 */
export async function classifyWithLlm(html: string, url: string): Promise<Verdict | null> {
  let bodyText: string;
  try {
    const $ = cheerio.load(html);
    bodyText = $('body').text().slice(0, 4000);
  } catch {
    bodyText = html.slice(0, 4000);
  }
  try {
    const provider = getLLMProvider();
    const resp = await provider.generateContent({
      systemPrompt: buildLlmPrompt(),
      messages: [{ role: 'user', content: `URL: ${url}\n\nText:\n${bodyText}` }],
      jsonMode: true,
      temperature: 0,
    });
    const parsed = LLM_RESPONSE_SCHEMA.safeParse(resp.json());
    if (!parsed.success) {
      log.warn({ event: 'llm_invalid_shape', url, err: parsed.error.message }, 'llm returned bad shape');
      return null;
    }
    return {
      contentDomain: parsed.data.contentDomain,
      confidence: parsed.data.confidence,
      detectedVia: 'llm',
      contributingLayers: ['llm'],
      schemaOrgType: null,
    };
  } catch (err) {
    log.warn({ event: 'llm_call_failed', url, err: String(err) }, 'llm fallback failed');
    return null;
  }
}

// ─── Aggregator ─────────────────────────────────────────────────────────────

export interface ClassifyOptions {
  cms: CmsId;
}

const AMBER_FLOOR: Verdict = {
  contentDomain: 'marketing',
  confidence: 0.30,
  detectedVia: 'heuristic',
  contributingLayers: ['fallback'],
  schemaOrgType: null,
};

/**
 * Run the full 8-layer cascade and return a single aggregated Verdict.
 * Layers 1–7 are pure; Layer 8 fires only when combined 1–7 confidence < 0.55.
 */
export async function classify(
  html: string,
  url: string,
  opts: ClassifyOptions,
): Promise<Verdict> {
  const layers: Array<Verdict | null> = [
    classifyByCms(opts.cms, url),
    classifyByUrl(url),
    classifyByJsonLd(html, url),
    classifyByOpenGraph(html, url),
    classifyByMicrodata(html, url),
    classifyByDateAuthor(html, url),
    classifyBySemanticHtml(html, url),
  ];
  const aggregated = aggregateVerdicts(layers);
  if (aggregated && aggregated.confidence >= 0.55) {
    log.info({ event: 'classify_no_llm', url, domain: aggregated.contentDomain, conf: aggregated.confidence });
    return aggregated;
  }
  const llmVerdict = await classifyWithLlm(html, url);
  if (llmVerdict) return llmVerdict;
  return aggregated ?? AMBER_FLOOR;
}

function aggregateVerdicts(verdicts: Array<Verdict | null>): Verdict | null {
  const present = verdicts.filter((v): v is Verdict => v !== null);
  if (present.length === 0) return null;
  const byDomain = new Map<ContentDomain, Verdict[]>();
  for (const v of present) {
    const list = byDomain.get(v.contentDomain) ?? [];
    list.push(v);
    byDomain.set(v.contentDomain, list);
  }
  let best: Verdict | null = null;
  for (const [domain, list] of byDomain) {
    const base = Math.max(...list.map((v) => v.confidence));
    const boost = 0.05 * (list.length - 1);
    const finalConf = Math.min(0.95, base + boost);
    const layers = uniq(list.flatMap((v) => v.contributingLayers));
    const schemaOrgType = list.find((v) => v.schemaOrgType)?.schemaOrgType ?? null;
    const candidate: Verdict = {
      contentDomain: domain,
      confidence: finalConf,
      detectedVia: list[0].detectedVia,
      contributingLayers: layers,
      schemaOrgType,
    };
    if (!best || candidate.confidence > best.confidence) best = candidate;
  }
  return best;
}

function uniq<T>(arr: T[]): T[] {
  return [...new Set(arr)];
}
  • [ ] Step 3.4: Run LLM + aggregator tests
npx vitest run lib/factory/__tests__/classifier.test.ts -t 'classifyWithLlm|classify \(aggregator\)'

Expected: all pass.

  • [ ] Step 3.5: Run all classifier tests
npx vitest run lib/factory/__tests__/classifier.test.ts

Expected: all pass (Layers 1–7 + Layer 8 + aggregator).

  • [ ] Step 3.6: Run tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 3.7: Commit
git add lib/factory/classifier.ts lib/factory/__tests__/classifier.test.ts
git commit -m "feat(factory): cascade aggregator + LLM tiebreaker (layer 8) with agreement boost"

Task 4: Site-type recognition rubric

Files: - Create: lib/factory/site-type-rubric.ts - Create: lib/factory/__tests__/site-type-rubric.test.ts

This implements the §19d rubric: an ordered list of {predicate, siteType} rules. The predicates inspect samples + robotsTxt and return the first match. Order matters: the rubric is data, not a switch statement, so adding a new site type means adding one entry to the list.

Rule order (per §19d, priority top-down):

# Site type Predicate
1 drupal-class robots.txt contains Crawl-delay: 10
2 wordpress any sample HTML or URL contains /wp-content/
3 aem-enterprise any sample contains /content/dam/ OR data-cmp-* attr
4 shopify any sample contains cdn.shopify.com OR Shopify.shop cookie/global
5 salesforce-lwc any sample contains data-aura-* OR lwc-* attrs
6 wikipedia-mediawiki any sample HTML contains mw-* class AND URL contains /wiki/
7 multi-brand-corporate any sample URL path matches /our-brands/, /portfolio/, or /companies/ AND ≥3 outbound brand domains detected
8 massive-silo sitemap-index nesting depth ≥ 2 AND ≥100 sub-sitemaps reported
9 dtc-retail any URL path matches /products/ OR /p/ OR /t/ AND no Shopify hit
10 government host ends in .gov
11 saas-b2b-docs host has docs. subdomain OR ≥1 sample URL on a *.docs.* host
12 spa every sampled HTML body contains an empty <div id="root"> AND no SSR text
13 custom always-true fallback

The rubric receives optional probe metadata (subSitemapCount, nestingDepth, outboundBrandDomains) that is supplied by Spec 02 (sitemap walker) and Spec 03 (brand-link probe). When metadata is missing, those rules cannot fire: the rubric falls through.

  • [ ] Step 4.1: Write failing tests for the rubric

Create lib/factory/__tests__/site-type-rubric.test.ts:

import { describe, it, expect } from 'vitest';
import { detectSiteType } from '../site-type-rubric';

const sample = (overrides: Partial<{ url: string; html: string; status: number }> = {}) => ({
  url: 'https://example.com/page',
  html: '<html><body></body></html>',
  status: 200,
  ...overrides,
});

describe('detectSiteType', () => {
  it('returns drupal-class for robots.txt with Crawl-delay: 10', () => {
    const result = detectSiteType({
      samples: [sample()],
      robotsTxt: 'User-agent: *\nCrawl-delay: 10\n',
    });
    expect(result).toBe('drupal-class');
  });

  it('returns wordpress for /wp-content/ in sample html', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<img src="/wp-content/x.png">' })],
    });
    expect(result).toBe('wordpress');
  });

  it('returns aem-enterprise for /content/dam/', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<img src="/content/dam/hp/x.png">' })],
    });
    expect(result).toBe('aem-enterprise');
  });

  it('returns aem-enterprise for data-cmp-*', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<div data-cmp-is="text">x</div>' })],
    });
    expect(result).toBe('aem-enterprise');
  });

  it('returns shopify for cdn.shopify.com', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<img src="https://cdn.shopify.com/x.png">' })],
    });
    expect(result).toBe('shopify');
  });

  it('returns salesforce-lwc for data-aura-*', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<div data-aura-rendered-by="123">x</div>' })],
    });
    expect(result).toBe('salesforce-lwc');
  });

  it('returns wikipedia-mediawiki for mw-* + /wiki/', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://en.wikipedia.org/wiki/Algolia', html: '<div class="mw-parser-output">x</div>' })],
    });
    expect(result).toBe('wikipedia-mediawiki');
  });

  it('returns multi-brand-corporate when /our-brands/ + ≥3 outbound brand domains', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://lvmh.com/our-brands/', html: '<a href="https://louisvuitton.com">LV</a><a href="https://dior.com">Dior</a><a href="https://celine.com">Celine</a>' })],
      probe: { outboundBrandDomains: ['louisvuitton.com', 'dior.com', 'celine.com'] },
    });
    expect(result).toBe('multi-brand-corporate');
  });

  it('does NOT return multi-brand-corporate when only 2 brand domains found', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://x.com/our-brands/', html: '<a href="https://a.com">A</a><a href="https://b.com">B</a>' })],
      probe: { outboundBrandDomains: ['a.com', 'b.com'] },
    });
    expect(result).not.toBe('multi-brand-corporate');
  });

  it('returns massive-silo for nestingDepth ≥ 2 AND ≥100 sub-sitemaps', () => {
    const result = detectSiteType({
      samples: [sample()],
      probe: { sitemapNestingDepth: 3, subSitemapCount: 1000 },
    });
    expect(result).toBe('massive-silo');
  });

  it('does NOT return massive-silo when only 50 sub-sitemaps', () => {
    const result = detectSiteType({
      samples: [sample()],
      probe: { sitemapNestingDepth: 2, subSitemapCount: 50 },
    });
    expect(result).not.toBe('massive-silo');
  });

  it('returns dtc-retail for /products/ without shopify signature', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://nike.com/products/air-max', html: '<h1>Air Max</h1>' })],
    });
    expect(result).toBe('dtc-retail');
  });

  it('shopify wins over dtc-retail when both signals present (priority order)', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://store.com/products/x', html: '<img src="https://cdn.shopify.com/y.png">' })],
    });
    expect(result).toBe('shopify');
  });

  it('returns government for .gov host', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://usa.gov/agencies' })],
    });
    expect(result).toBe('government');
  });

  it('returns saas-b2b-docs when docs.* subdomain detected', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://docs.commercetools.com/api' })],
    });
    expect(result).toBe('saas-b2b-docs');
  });

  it('returns spa when body contains empty <div id="root"> and no SSR text', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<html><body><div id="root"></div></body></html>' })],
    });
    expect(result).toBe('spa');
  });

  it('returns custom when nothing matches', () => {
    const result = detectSiteType({
      samples: [sample({ url: 'https://acme.com/about', html: '<html><body><h1>About Us</h1><p>company copy</p></body></html>' })],
    });
    expect(result).toBe('custom');
  });

  it('drupal-class wins over wordpress when both signals present (priority order)', () => {
    const result = detectSiteType({
      samples: [sample({ html: '<img src="/wp-content/x.png">' })],
      robotsTxt: 'User-agent: *\nCrawl-delay: 10\n',
    });
    expect(result).toBe('drupal-class');
  });

  it('first-match-wins: empty input falls through to custom', () => {
    const result = detectSiteType({ samples: [] });
    expect(result).toBe('custom');
  });
});
  • [ ] Step 4.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/site-type-rubric.test.ts

Expected: FAIL: module not found.

  • [ ] Step 4.3: Implement lib/factory/site-type-rubric.ts

Create lib/factory/site-type-rubric.ts:

/**
 * Site-type recognition rubric — implements 00-Plan.md §19d.
 *
 * Ordered list of {predicate, siteType}. First match wins. Data, not code:
 * adding a new site type means adding one entry. Each predicate is a pure
 * function of (samples, robotsTxt, probe).
 */

import { createLogger } from '../utils/logger';
import { type SiteType } from './types';

const log = createLogger('factory:site-type-rubric');

export interface RubricSample {
  url: string;
  html: string;
  status: number;
}

export interface RubricProbe {
  sitemapNestingDepth?: number;
  subSitemapCount?: number;
  outboundBrandDomains?: string[];
}

export interface RubricInput {
  samples: RubricSample[];
  robotsTxt?: string;
  probe?: RubricProbe;
}

interface Rule {
  siteType: SiteType;
  predicate: (input: RubricInput) => boolean;
}

const RULES: Rule[] = [
  { siteType: 'drupal-class', predicate: hasDrupalCrawlDelay },
  { siteType: 'wordpress', predicate: hasWordPressSignal },
  { siteType: 'aem-enterprise', predicate: hasAemSignal },
  { siteType: 'shopify', predicate: hasShopifySignal },
  { siteType: 'salesforce-lwc', predicate: hasSalesforceLwcSignal },
  { siteType: 'wikipedia-mediawiki', predicate: hasMediaWikiSignal },
  { siteType: 'multi-brand-corporate', predicate: hasMultiBrandSignal },
  { siteType: 'massive-silo', predicate: hasMassiveSiloSignal },
  { siteType: 'dtc-retail', predicate: hasDtcRetailSignal },
  { siteType: 'government', predicate: hasGovHostSignal },
  { siteType: 'saas-b2b-docs', predicate: hasSaasDocsSignal },
  { siteType: 'spa', predicate: hasSpaSignal },
  { siteType: 'custom', predicate: () => true },
];

/**
 * Run the rubric and return the first matching SiteType.
 * Always returns a SiteType — falls through to 'custom'.
 */
export function detectSiteType(input: RubricInput): SiteType {
  for (const rule of RULES) {
    if (rule.predicate(input)) {
      log.info({ event: 'siteType', siteType: rule.siteType }, 'site type detected');
      return rule.siteType;
    }
  }
  return 'custom';
}

function hasDrupalCrawlDelay(input: RubricInput): boolean {
  if (!input.robotsTxt) return false;
  return /^\s*Crawl-delay:\s*10\s*$/im.test(input.robotsTxt);
}

function hasWordPressSignal(input: RubricInput): boolean {
  return input.samples.some((s) => /\/wp-content\//.test(s.html) || /\/wp-content\//.test(s.url));
}

function hasAemSignal(input: RubricInput): boolean {
  return input.samples.some(
    (s) => /\/content\/dam\//.test(s.html) || /\/content\/dam\//.test(s.url) || /data-cmp-[a-z-]+="/i.test(s.html),
  );
}

function hasShopifySignal(input: RubricInput): boolean {
  return input.samples.some(
    (s) => /cdn\.shopify\.com/.test(s.html) || /\bShopify\.(shop|theme)\b/.test(s.html),
  );
}

function hasSalesforceLwcSignal(input: RubricInput): boolean {
  return input.samples.some(
    (s) => /data-aura-[a-z-]+="/i.test(s.html) || /\blwc-[a-z0-9-]+="/i.test(s.html),
  );
}

function hasMediaWikiSignal(input: RubricInput): boolean {
  return input.samples.some(
    (s) => /class="[^"]*\bmw-[a-z0-9-]+/i.test(s.html) && /\/wiki\//.test(s.url),
  );
}

function hasMultiBrandSignal(input: RubricInput): boolean {
  const hasBrandIndexUrl = input.samples.some((s) =>
    /\/(our-brands|portfolio|companies)\b/.test(safePath(s.url)),
  );
  const hasOutbounds = (input.probe?.outboundBrandDomains ?? []).length >= 3;
  return hasBrandIndexUrl && hasOutbounds;
}

function hasMassiveSiloSignal(input: RubricInput): boolean {
  const depth = input.probe?.sitemapNestingDepth ?? 0;
  const count = input.probe?.subSitemapCount ?? 0;
  return depth >= 2 && count >= 100;
}

function hasDtcRetailSignal(input: RubricInput): boolean {
  const hasProductPath = input.samples.some((s) =>
    /\/(products|p|t)\//.test(safePath(s.url)),
  );
  if (!hasProductPath) return false;
  const looksShopify = input.samples.some((s) => /cdn\.shopify\.com/.test(s.html));
  return !looksShopify;
}

function hasGovHostSignal(input: RubricInput): boolean {
  return input.samples.some((s) => {
    try {
      return new URL(s.url).hostname.endsWith('.gov');
    } catch {
      return false;
    }
  });
}

function hasSaasDocsSignal(input: RubricInput): boolean {
  return input.samples.some((s) => {
    try {
      return new URL(s.url).hostname.startsWith('docs.');
    } catch {
      return false;
    }
  });
}

function hasSpaSignal(input: RubricInput): boolean {
  if (input.samples.length === 0) return false;
  return input.samples.every((s) => {
    if (!/\<div\s+id="root"[^>]*>\s*<\/div\>/i.test(s.html)) return false;
    const text = s.html.replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim();
    return text.length < 100;
  });
}

function safePath(url: string): string {
  try {
    return new URL(url).pathname;
  } catch {
    return '';
  }
}
  • [ ] Step 4.4: Run all rubric tests
npx vitest run lib/factory/__tests__/site-type-rubric.test.ts

Expected: all tests pass.

  • [ ] Step 4.5: Run tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 4.6: Commit
git add lib/factory/site-type-rubric.ts lib/factory/__tests__/site-type-rubric.test.ts
git commit -m "feat(factory): site-type rubric with 13 ordered predicates per §19d"

Task 5: Cluster validation: full regression + integration sanity check

Files: - (Read-only verification + cluster-level smoke)

  • [ ] Step 5.1: Run all factory tests
npx vitest run lib/factory/

Expected: all factory tests pass: Spec 01 (~23) + Spec 04 (~70+ new). No regressions.

  • [ ] Step 5.2: Run the full project test suite
npx vitest run

Expected: 407 baseline + Spec 01 + Spec 04, all GREEN.

  • [ ] Step 5.3: Run full tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 5.4: Cluster-level integration sanity check (vitest, no network)

Create lib/factory/__tests__/spec-04-integration.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { detectCms } from '../cms-detector';
import { classify } from '../classifier';
import { detectSiteType } from '../site-type-rubric';

vi.mock('../../llm', () => ({
  getLLMProvider: () => ({
    name: 'mock',
    capabilities: { supportsJsonMode: true, supportsJsonSchema: true, supportsSystemInstruction: true, supportsStreaming: false },
    generateContent: async () => ({ text: () => '', json: () => ({ contentDomain: 'marketing', confidence: 0.7 }) }),
    startChat: () => { throw new Error('not used'); },
  }),
  resetLLMProvider: () => {},
}));

describe('Spec 04 — end-to-end pipeline (no network)', () => {
  it('TechCrunch-shaped sample: rubric=wordpress, cms=wordpress, classify=marketing@green', async () => {
    const samples = [
      {
        url: 'https://techcrunch.com/2026/04/30/some-post/',
        html: '<html><body><article class="post entry-content"><img src="/wp-content/uploads/x.png"><time datetime="2026-04-30">x</time><span class="byline">By Jane Doe</span><p>Body</p></article></body></html>',
        status: 200,
      },
    ];
    const siteType = detectSiteType({ samples });
    const cms = await detectCms(samples);
    const verdict = await classify(samples[0].html, samples[0].url, { cms: cms.cms });

    expect(siteType).toBe('wordpress');
    expect(cms.cms).toBe('wordpress');
    expect(verdict.contentDomain).toBe('marketing');
    expect(verdict.confidence).toBeGreaterThanOrEqual(0.85);
  });

  it('GOV.UK-shaped sample: rubric=drupal-class, cms=drupal, classify=legal/marketing@high', async () => {
    const samples = [
      {
        url: 'https://www.gov.uk/guidance/foo',
        html: '<html><body><div class="region-content"><div class="field-name-body"><h1>Guidance</h1><p>policy text</p></div></div></body></html>',
        status: 200,
      },
    ];
    const robotsTxt = 'User-agent: *\nCrawl-delay: 10\n';
    const siteType = detectSiteType({ samples, robotsTxt });
    const cms = await detectCms(samples, robotsTxt);
    const verdict = await classify(samples[0].html, samples[0].url, { cms: cms.cms });

    expect(siteType).toBe('drupal-class');
    expect(cms.cms).toBe('drupal');
    expect(['legal', 'marketing']).toContain(verdict.contentDomain);
  });

  it('plain custom site with no signals → rubric=custom, cms=custom, classify falls to LLM', async () => {
    const samples = [
      { url: 'https://acme.com/about', html: '<html><body><h1>About Us</h1><p>company.</p></body></html>', status: 200 },
    ];
    const siteType = detectSiteType({ samples });
    const cms = await detectCms(samples);
    const verdict = await classify(samples[0].html, samples[0].url, { cms: cms.cms });

    expect(siteType).toBe('custom');
    expect(cms.cms).toBe('custom');
    expect(verdict.detectedVia).toBe('llm');
  });
});
  • [ ] Step 5.5: Run integration test
npx vitest run lib/factory/__tests__/spec-04-integration.test.ts

Expected: 3 tests pass.

  • [ ] Step 5.6: Commit + mark Spec 04 done
git add lib/factory/__tests__/spec-04-integration.test.ts
git commit -m "test(factory): Spec 04 cluster-level integration check (no network)"

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 4 (CMS + Classification) | ⏸ | cms-detector, classifier, site-type-rubric |
+ | Cluster 4 (CMS + Classification) | ✅ | cms-detector, classifier, site-type-rubric |

Acceptance criteria: Spec 04 done means:

  1. ✅ lib/factory/types.ts exports CmsIdEnum, CmsDetectionSchema, EvidenceSchema, VerdictSchema, SiteTypeEnum and their inferred types
  2. ✅ lib/factory/cms-detector.ts exports detectCms(samples, robotsTxt?) returning CmsDetection for the 11 known CMSes + custom fallback
  3. ✅ lib/factory/classifier.ts exports the 7 layer functions (classifyByCms, classifyByUrl, classifyByJsonLd, classifyByOpenGraph, classifyByMicrodata, classifyByDateAuthor, classifyBySemanticHtml), classifyWithLlm, and the classify aggregator
  4. ✅ Aggregator implements §19c: max-confidence layer wins; agreement boost = 0.05 per agreeing layer, capped at 0.95; LLM fires only when combined layers 1–7 confidence < 0.55
  5. ✅ Traffic-light bands respected: ≥0.85 green, 0.65–0.84 yellow, <0.65 amber
  6. ✅ lib/factory/site-type-rubric.ts exports detectSiteType(input) with the 13 ordered rules from §19d, first-match-wins, always returns a SiteType
  7. ✅ All new tests pass: ~70+ tests across cms-detector, classifier, site-type-rubric, and integration
  8. ✅ Full project test suite still GREEN (no regressions)
  9. ✅ tsc --noEmit clean
  10. ✅ Per CodingSOPs: every module has logger, every public function has docstring, every fallible call has try/catch, every boundary uses zod, no any, functions ≤20 lines and ≤3 params (config object when more)
  11. ✅ LLM layer mocked in unit tests; real LLM calls are integration-only (Spec 13 cluster)

Out of scope (handled by other specs)

  • Sample collection (HTTP fetch + Playwright fallback) → Spec 03 (sampler)
  • recordExtractor generation per content domain → Spec 05 (extractor generator)
  • Index creation + facet configuration → Spec 06 (index-manager.ts)
  • Frontend traffic-light rendering per pathGroup → Spec 11 (frontend)
  • Real LLM integration smoke test → Spec 13 (smoke test cluster)
  • Streaming sitemap walker that supplies subSitemapCount/sitemapNestingDepth → Spec 02
  • Brand-index probe that supplies outboundBrandDomains → Spec 03 sub-flow (multi-brand corporates, §16k)