Crawler-Factory

Engineering-Specs/05-extraction-sandbox.md

Crawler Factory: Spec 05: Structure Analysis + Extractor Generation + Sandbox + Canonical Inference Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Given a sample HTML page + a detected content domain, generate a working recordExtractor JS string and validate it against the DSS-derived zod schema in a sandbox; additionally, infer canonical URLs so DTC variant explosions deduplicate before they reach the crawler.

Architecture: Pure functional pipeline: no I/O beyond the LLM provider (and even that is dependency-injected so unit tests use a mock). structure-analyzer inspects sample HTML with cheerio + JSON-LD parsing and emits {schemaOrgJsonLd, selectors, missing, confidence}. record-extractor-template exports per-content-domain JS-string scaffolds with {{placeholders}}. extractor-generator fills placeholders via the LLM (using structureAnalysis + DSS row + optional user feedback) and returns a {recordExtractor, crawlerConfig} blueprint. sandbox-runner instantiates the extractor via new Function, runs it against the sample, and segregates DSS-zod-valid records from failures. canonical-inferer resolves variant URLs to a canonical (<link rel="canonical"> → URL-suffix heuristic → LLM fallback) so the create-crawler step (Spec 06) can dedupe SKIMS-pattern variant explosions.

Tech Stack: TypeScript (strict), zod 3.25, cheerio 1.x (already in repo), vitest, the existing lib/llm/ provider (mocked in unit tests), pino logger via lib/utils/logger.ts.

Depends on: Spec 01 (DSS, types, zod schemas), Spec 03 (Sample type: lib/factory/sampler.ts), Spec 04 (classification result feeds analyzeStructure's contentDomain argument), existing lib/llm/ for LLM calls. Consumed by: Spec 06 (create-crawler step calls canonicalInferer), Spec 08 (/api/factory/generate-extractor and /api/factory/test-sandbox endpoints), Spec 11 (frontend CategoryConfigurator shows sandbox results).

Plan source: Projects/Crawler-Factory/00-Plan.md §7 Tasks 9–11, §16o (DTC retail: DOM-extraction-first; canonical inference), §16o-furniture sub-section, §19f principle 7 (DSS as data, not code), §20a Tasks D + extractor enhancements.

File structure

Path	Responsibility
`lib/factory/structure-analyzer.ts`	Inspect sample HTML; emit `{schemaOrgJsonLd, selectors, missing, confidence}` for the DSS-required fields of a content domain. JSON-LD wins; LLM fills ambiguous selectors; CMS-pattern cache short-circuits common cases.
`lib/factory/record-extractor-template.ts`	Per-content-domain JS-string templates. Each template has the signature `({ $, helpers, url, content, headers }) => Record[]`, hardcodes `is_chunk:false` / `status:'indexed'` (no `record_type`: that field is reserved for sharded session storage per Spec 01), embeds JSON-LD-first / selector-fallback logic, and exposes `{{placeholder}}` slots filled by the generator.
`lib/factory/extractor-generator.ts`	`generateExtractor(...)`: pulls the per-domain template, asks the LLM (mockable) to fill placeholders given `structureAnalysis` + DSS row + optional `userFeedback`, validates the output via `new Function('return ' + src)`, returns `{recordExtractor, crawlerConfig}`.
`lib/factory/sandbox-runner.ts`	`runInSandbox(extractorSrc, sample, contentDomain)`: instantiates extractor via `new Function`, runs against `cheerio.load(sample.html)`, validates each emitted record against `getRecordSchema(contentDomain)`, segregates passing from failing. Catches extractor exceptions cleanly.
`lib/factory/canonical-inferer.ts`	`inferCanonical(url, html, domain, llm?)`: `<link rel="canonical">` → URL-suffix heuristic for product-catalog → LLM fallback. Returns the canonical URL string.
`lib/factory/__tests__/structure-analyzer.test.ts`	Tests: JSON-LD-present vs absent; missing-field reporting; LLM fallback invoked only for ambiguous fields; CMS pattern cache hit short-circuits.
`lib/factory/__tests__/record-extractor-template.test.ts`	Tests: each of 9 content domains has a registered template; each template parses as JS via `new Function`; each declares the correct hardcoded base fields.
`lib/factory/__tests__/extractor-generator.test.ts`	Tests: per-domain happy path (mock LLM); `userFeedback` is prepended to LLM prompt; syntax-error in LLM output throws; `crawlerConfig.pathsToMatch` derived from `pathGroup.pattern`.
`lib/factory/__tests__/sandbox-runner.test.ts`	Tests: missing-required-field record lands in `errors`; thrown extractor caught and reported; passing record returns clean `records[]`; multiple records per page handled.
`lib/factory/__tests__/canonical-inferer.test.ts`	Tests: `<link rel="canonical">` present → returns it; absent + `product-catalog` + SKIMS-style `-clay`/`-onyx` suffix → strips suffix; LLM fallback invoked when neither rule fires; non-product-catalog domain skips heuristic.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

Two-layer type safety. zod for runtime boundary validation (every public API + LLM-output edge); tsc --noEmit strict for write-time. No any. Use T | null (not T | undefined) when absence is meaningful.
Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:<module>');. No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
Try/catch every fallible call. Every LLM call, every new Function() instantiation, every cheerio-on-malformed-HTML path. Catch specific error first, generic last. Always log before rethrow with full context (sessionId, pathGroupId, urlHash where available).
Functions ≤20 lines, ≤3 params. More than 3 params → typed config object.
Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
No raw dicts crossing module boundaries. Every public input/output is a zod-validated TypeScript type.
TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
Mock external services in unit tests. The LLM provider is dependency-injected and mocked in every test file via vitest's vi.fn(). Real LLM calls live in integration tests gated by env vars (out of scope here).
Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
Conventional commits. feat(factory):, test(factory):, chore(factory):, etc. One logical change per commit.
DSS-as-data principle (§19f.7). Per-domain logic lives in DSS rows + per-domain templates; adding a new domain or sub-schema is a configuration change, not a code release. No if (domain === 'marketing') ... branching outside the registries.

Task 0: Helper: DSS record-schema accessor

Files: - Modify: lib/factory/dss.ts: add getRecordSchema(domain) helper that returns the matching zod record schema from lib/factory/types.ts. - Modify: lib/factory/__tests__/dss.test.ts: add tests for the new helper.

This helper is consumed by sandbox-runner (Task 4) and extractor-generator (Task 3). Spec 01 deliberately kept dss.ts schema-free; this spec is the first consumer that needs schema lookup so we add it here.

[ ] Step 0.1: Write the failing test

Append to lib/factory/__tests__/dss.test.ts:

import {
  MarketingRecordSchema,
  SupportRecordSchema,
  ProductRecordSchema,
} from '../types';
import { getRecordSchema } from '../dss';

describe('getRecordSchema', () => {
  it('returns MarketingRecordSchema for marketing', () => {
    expect(getRecordSchema('marketing')).toBe(MarketingRecordSchema);
  });

  it('returns SupportRecordSchema for support', () => {
    expect(getRecordSchema('support')).toBe(SupportRecordSchema);
  });

  it('returns ProductRecordSchema for product-catalog', () => {
    expect(getRecordSchema('product-catalog')).toBe(ProductRecordSchema);
  });

  it('throws for unknown domain', () => {
    // @ts-expect-error — runtime check
    expect(() => getRecordSchema('made-up')).toThrow();
  });
});

[ ] Step 0.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/dss.test.ts -t getRecordSchema

Expected: FAIL: getRecordSchema is not a function / not exported.

[ ] Step 0.3: Implement getRecordSchema in lib/factory/dss.ts

Append to lib/factory/dss.ts (below listContentDomains):

import {
  MarketingRecordSchema,
  SupportRecordSchema,
  EducationRecordSchema,
  TechnicalRecordSchema,
  CustomerStoryRecordSchema,
  ProductRecordSchema,
  EventRecordSchema,
  LegalRecordSchema,
  SocialRecordSchema,
} from './types';
import type { ZodTypeAny } from 'zod';

const RECORD_SCHEMA_BY_DOMAIN: Readonly<Record<ContentDomain, ZodTypeAny>> = Object.freeze({
  marketing: MarketingRecordSchema,
  support: SupportRecordSchema,
  education: EducationRecordSchema,
  technical: TechnicalRecordSchema,
  'customer-stories': CustomerStoryRecordSchema,
  'product-catalog': ProductRecordSchema,
  events: EventRecordSchema,
  legal: LegalRecordSchema,
  social: SocialRecordSchema,
});

/**
 * Return the zod schema for records of the given content domain.
 * Used by sandbox-runner to validate extractor output against DSS.
 */
export function getRecordSchema(domain: ContentDomain): ZodTypeAny {
  const schema = RECORD_SCHEMA_BY_DOMAIN[domain];
  if (!schema) {
    log.error({ event: 'getRecordSchema_unknown_domain', domain }, 'Unknown content domain');
    throw new Error(`Unknown content domain: ${domain}`);
  }
  return schema;
}

[ ] Step 0.4: Run test, expect PASS

npx vitest run lib/factory/__tests__/dss.test.ts -t getRecordSchema

Expected: 4 tests pass.

[ ] Step 0.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 0.6: Commit

git add lib/factory/dss.ts lib/factory/__tests__/dss.test.ts
git commit -m "feat(factory): add getRecordSchema helper to DSS registry"

Task 1: Structure analyzer: JSON-LD-first, selector-fallback, LLM-on-ambiguity

Files: - Create: lib/factory/structure-analyzer.ts - Create: lib/factory/__tests__/structure-analyzer.test.ts

The analyzer's job: given a sample page + content domain, emit {schemaOrgJsonLd, selectors, missing, confidence} so the extractor generator knows what selectors to wire. Per §19f.2 (CMS+URL first; schema.org enriches, doesn't lead) and §16o (DOM-first for product-catalog), the analyzer prioritizes JSON-LD when present but falls back to cheerio inspection + LLM for ambiguous fields.

The Sample type is defined in Spec 03 (lib/factory/sampler.ts exports SampleSchema). For this spec, we only need the shape: { url: string; html: string; status: number; cmsHint?: string | null; headers?: Record<string, string> }.

[ ] Step 1.1: Write failing test for the JSON-LD-present case

Create lib/factory/__tests__/structure-analyzer.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import { analyzeStructure } from '../structure-analyzer';
import type { ContentDomain } from '../types';

const mkSample = (html: string, url = 'https://x.com/blog/p1') => ({
  url,
  html,
  status: 200,
  cmsHint: null,
  headers: {},
});

describe('analyzeStructure', () => {
  it('extracts fields from JSON-LD when present (marketing)', async () => {
    const html = `<html><head>
      <script type="application/ld+json">{
        "@context": "https://schema.org",
        "@type": "BlogPosting",
        "headline": "Hello world",
        "articleBody": "${'A'.repeat(300)}",
        "datePublished": "2026-04-30",
        "author": [{ "@type": "Person", "name": "Jane" }]
      }</script>
    </head><body><h1>Hello world</h1></body></html>`;

    const llm = vi.fn();
    const result = await analyzeStructure(mkSample(html), 'marketing' as ContentDomain, { llm });

    expect(result.schemaOrgJsonLd).toBeTruthy();
    expect(result.schemaOrgJsonLd?.['@type']).toBe('BlogPosting');
    expect(result.confidence).toBeGreaterThanOrEqual(0.8);
    expect(result.missing).not.toContain('headline');
    expect(result.missing).not.toContain('articleBody');
    expect(llm).not.toHaveBeenCalled(); // JSON-LD covered everything required
  });
});

[ ] Step 1.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/structure-analyzer.test.ts

Expected: FAIL: Cannot find module '../structure-analyzer'.

[ ] Step 1.3: Implement lib/factory/structure-analyzer.ts skeleton + JSON-LD path

Create lib/factory/structure-analyzer.ts:

/**
 * DSS-aware structure analyzer.
 *
 * Pipeline:
 *   1. Parse <script type="application/ld+json"> blocks (most authoritative).
 *   2. For each DSS-required field of the content domain, find a CSS selector
 *      via cheerio inspection (heuristic: tags/classes that match field semantics).
 *   3. For ambiguous cases, call the injected LLM with HTML snippet + DSS row +
 *      "what selector for fieldX?" prompt.
 *   4. Cache common patterns per CMS (cmsHint) so repeated samples short-circuit.
 *
 * Per §19f.2 + §16o: JSON-LD enriches but does not lead. Selectors are the
 * fallback that handles the 60% of real content pages without useful @types.
 */

import * as cheerio from 'cheerio';
import { z } from 'zod';
import { createLogger } from '../utils/logger';
import { getDss } from './dss';
import type { ContentDomain } from './types';

const log = createLogger('factory:structure-analyzer');

// ─── Public types ───────────────────────────────────────────────────────────

export const StructureAnalysisSchema = z.object({
  schemaOrgJsonLd: z.record(z.unknown()).nullable(),
  selectors: z.record(z.string()),
  missing: z.array(z.string()),
  confidence: z.number().min(0).max(1),
});
export type StructureAnalysis = z.infer<typeof StructureAnalysisSchema>;

export interface StructureSample {
  url: string;
  html: string;
  status: number;
  cmsHint?: string | null;
  headers?: Record<string, string>;
}

export type LlmFn = (prompt: string) => Promise<string>;

export interface AnalyzeOpts {
  llm?: LlmFn;
}

// ─── Per-domain required-field map (drives missing[] computation) ───────────

const REQUIRED_FIELDS_BY_DOMAIN: Readonly<Record<ContentDomain, readonly string[]>> = Object.freeze({
  marketing: ['headline', 'articleBody'],
  support: ['name', 'articleBody'],
  education: ['name', 'description'],
  technical: ['name', 'articleBody'],
  'customer-stories': ['headline', 'articleBody'],
  'product-catalog': ['name', 'description'],
  events: ['name', 'description', 'startDate'],
  legal: ['name', 'text'],
  social: ['text'],
});

// ─── In-memory CMS pattern cache (process-lifetime, not persisted) ──────────

const cmsPatternCache = new Map<string, Record<string, string>>();

function cacheKey(cmsHint: string | null | undefined, domain: ContentDomain): string {
  return `${cmsHint ?? 'unknown'}::${domain}`;
}

// ─── JSON-LD extraction ─────────────────────────────────────────────────────

function extractJsonLd($: cheerio.CheerioAPI): Record<string, unknown> | null {
  const blocks: Record<string, unknown>[] = [];
  $('script[type="application/ld+json"]').each((_, el) => {
    try {
      const raw = $(el).html();
      if (!raw) return;
      const parsed = JSON.parse(raw);
      if (Array.isArray(parsed)) blocks.push(...parsed);
      else blocks.push(parsed);
    } catch (err) {
      log.warn({ event: 'jsonld_parse_failed', err: String(err) }, 'malformed JSON-LD block');
    }
  });
  if (blocks.length === 0) return null;
  // Prefer blocks with a recognizable @type
  const typed = blocks.find((b) => typeof (b as { '@type'?: unknown })['@type'] === 'string');
  return typed ?? blocks[0] ?? null;
}

// ─── Heuristic selector for a single field ──────────────────────────────────

function heuristicSelector($: cheerio.CheerioAPI, field: string): string | null {
  const candidates: Record<string, string[]> = {
    headline: ['h1', '[itemprop="headline"]', 'article h1', '.entry-title'],
    name: ['h1', '[itemprop="name"]', '.product-title', '.product-name'],
    articleBody: ['article', '[itemprop="articleBody"]', '.entry-content', '.post-content', 'main'],
    description: ['[itemprop="description"]', '.product-description', '.description', 'meta[name="description"]'],
    text: ['article', 'main', 'body'],
    author: ['[itemprop="author"]', '.author', '.byline'],
    datePublished: ['time[datetime]', '[itemprop="datePublished"]', '.published'],
    startDate: ['time[datetime]', '[itemprop="startDate"]', '.event-date'],
    acceptedAnswer: ['.accepted-answer', '[itemprop="acceptedAnswer"]'],
    transcript: ['.transcript', '[itemprop="transcript"]'],
    programmingLanguage: ['code[class*="language-"]', '[itemprop="programmingLanguage"]'],
  };
  const list = candidates[field];
  if (!list) return null;
  for (const sel of list) {
    if ($(sel).length > 0) return sel;
  }
  return null;
}

// ─── Public entrypoint ──────────────────────────────────────────────────────

/**
 * Analyze a sampled page to discover how the DSS-required fields are encoded.
 * Returns JSON-LD blob (if any), per-field CSS selectors, missing fields, confidence.
 */
export async function analyzeStructure(
  sample: StructureSample,
  contentDomain: ContentDomain,
  opts: AnalyzeOpts = {}
): Promise<StructureAnalysis> {
  const $ = cheerio.load(sample.html);
  const dss = getDss(contentDomain);
  const required = REQUIRED_FIELDS_BY_DOMAIN[contentDomain];

  const cached = cmsPatternCache.get(cacheKey(sample.cmsHint, contentDomain));
  const jsonLd = extractJsonLd($);
  const selectors: Record<string, string> = { ...(cached ?? {}) };

  // Determine which fields JSON-LD already covers
  const coveredByJsonLd = new Set<string>();
  if (jsonLd) {
    for (const f of required) {
      if (typeof jsonLd[f] === 'string' && (jsonLd[f] as string).length > 0) {
        coveredByJsonLd.add(f);
      }
    }
  }

  // For uncovered required fields, try heuristic, then LLM
  const missing: string[] = [];
  for (const field of required) {
    if (coveredByJsonLd.has(field)) continue;
    if (selectors[field]) continue;
    const heur = heuristicSelector($, field);
    if (heur) {
      selectors[field] = heur;
      continue;
    }
    if (opts.llm) {
      try {
        const snippet = sample.html.slice(0, 4000);
        const prompt = `Given this HTML snippet from a ${dss.description} page and that we need a CSS selector to extract the "${field}" field, return only the selector string (no explanation).\n\nHTML:\n${snippet}`;
        const reply = (await opts.llm(prompt)).trim();
        if (reply && reply.length < 200 && $(reply).length > 0) {
          selectors[field] = reply;
          continue;
        }
      } catch (err) {
        log.warn(
          { event: 'llm_selector_failed', field, err: String(err) },
          'LLM selector resolution failed; field marked missing'
        );
      }
    }
    missing.push(field);
  }

  // Cache the pattern under cmsHint+domain (only if we resolved at least one field)
  if (sample.cmsHint && Object.keys(selectors).length > 0) {
    cmsPatternCache.set(cacheKey(sample.cmsHint, contentDomain), selectors);
  }

  // Confidence: 1.0 if JSON-LD covered everything; scale down by missing fraction
  const totalRequired = required.length;
  const found = totalRequired - missing.length;
  const baseConfidence = totalRequired === 0 ? 1 : found / totalRequired;
  const jsonLdBoost = jsonLd ? 0.1 : 0;
  const confidence = Math.min(1, baseConfidence + jsonLdBoost);

  log.info(
    { event: 'analyze_done', url: sample.url, domain: contentDomain, missing, confidence },
    'structure analysis complete'
  );

  return StructureAnalysisSchema.parse({
    schemaOrgJsonLd: jsonLd,
    selectors,
    missing,
    confidence,
  });
}

[ ] Step 1.4: Run JSON-LD test, expect PASS

npx vitest run lib/factory/__tests__/structure-analyzer.test.ts

Expected: 1 test passes.

[ ] Step 1.5: Add the JSON-LD-absent + missing-fields + LLM-fallback tests

Append to lib/factory/__tests__/structure-analyzer.test.ts:

describe('analyzeStructure (no JSON-LD)', () => {
  beforeEach(() => vi.restoreAllMocks());

  it('falls back to heuristic selectors when no JSON-LD', async () => {
    const html = `<html><body>
      <article><h1>Plain HTML title</h1>
      <div class="entry-content">${'word '.repeat(80)}</div></article>
    </body></html>`;

    const llm = vi.fn();
    const result = await analyzeStructure(mkSample(html), 'marketing' as ContentDomain, { llm });

    expect(result.schemaOrgJsonLd).toBeNull();
    expect(result.selectors.headline).toBe('h1');
    expect(result.selectors.articleBody).toBeTruthy();
    expect(result.missing).toEqual([]);
    expect(llm).not.toHaveBeenCalled();
  });

  it('reports missing fields when neither JSON-LD nor heuristic resolves', async () => {
    const html = '<html><body><div>nothing structured</div></body></html>';
    const llm = vi.fn().mockResolvedValue(''); // LLM returns empty → unhelpful
    const result = await analyzeStructure(mkSample(html), 'support' as ContentDomain, { llm });

    expect(result.missing).toContain('articleBody');
    // 'name' may resolve via h1-fallback; assert articleBody-class fields are missing
    expect(result.confidence).toBeLessThan(1);
  });

  it('invokes LLM only for fields not resolved by heuristic', async () => {
    const html = `<html><body>
      <h1>I have a name</h1>
      <div class="custom-weird-tag">${'word '.repeat(80)}</div>
    </body></html>`;
    const llm = vi.fn().mockResolvedValue('div.custom-weird-tag');
    const result = await analyzeStructure(mkSample(html), 'support' as ContentDomain, { llm });

    expect(llm).toHaveBeenCalledTimes(1); // only for articleBody
    expect(result.selectors.articleBody).toBe('div.custom-weird-tag');
  });

  it('uses CMS pattern cache on second call with same cmsHint+domain', async () => {
    const html1 = `<html><body><h1>One</h1><div class="entry-content">${'a '.repeat(80)}</div></body></html>`;
    const html2 = '<html><body><h1>Two</h1><div>different layout</div></body></html>';

    const llm = vi.fn();
    await analyzeStructure({ ...mkSample(html1), cmsHint: 'wordpress' }, 'marketing' as ContentDomain, { llm });
    const result2 = await analyzeStructure(
      { ...mkSample(html2), cmsHint: 'wordpress' },
      'marketing' as ContentDomain,
      { llm }
    );

    // Cache contains selectors from call #1; missing[] should be small thanks to cache
    expect(Object.keys(result2.selectors).length).toBeGreaterThan(0);
  });
});

[ ] Step 1.6: Run, expect PASS

npx vitest run lib/factory/__tests__/structure-analyzer.test.ts

Expected: 5 tests pass.

[ ] Step 1.7: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 1.8: Commit

git add lib/factory/structure-analyzer.ts lib/factory/__tests__/structure-analyzer.test.ts
git commit -m "feat(factory): DSS-aware structure analyzer (JSON-LD + heuristic + LLM)"

Task 2: Per-domain record-extractor templates

Files: - Create: lib/factory/record-extractor-template.ts - Create: lib/factory/__tests__/record-extractor-template.test.ts

Each template is a JS source string with the signature ({ $, helpers, url, content, headers }) => Record[]. Templates: - Hardcode is_chunk: false, status: 'indexed' (Algolia Central conventions). Do NOT emit record_type: that field is reserved for the sharded session-storage index (Spec 01 RecordBaseSchema); per-domain content records ride on the index name (algoliacentral_marketing, etc.). - Try JSON-LD first (using helpers.parseJsonLd), fall back to selectors filled by the generator. - Expose {{titleSelector}}, {{contentSelector}}, etc. as placeholders.

Per §19f.7 (DSS as data, not code), the templates live in a registry keyed by ContentDomain, never in branchy logic.

[ ] Step 2.1: Write failing test for template registry shape

Create lib/factory/__tests__/record-extractor-template.test.ts:

import { describe, it, expect } from 'vitest';
import {
  RECORD_EXTRACTOR_TEMPLATES,
  getExtractorTemplate,
  listTemplatePlaceholders,
} from '../record-extractor-template';
import { listContentDomains } from '../dss';

describe('RECORD_EXTRACTOR_TEMPLATES registry', () => {
  it('has a template for every content domain in the DSS', () => {
    for (const domain of listContentDomains()) {
      const tpl = RECORD_EXTRACTOR_TEMPLATES[domain];
      expect(tpl).toBeDefined();
      expect(typeof tpl).toBe('string');
      expect(tpl.length).toBeGreaterThan(100);
    }
  });

  it('every template omits record_type (reserved for session storage) and hardcodes is_chunk=false / status=indexed', () => {
    for (const domain of listContentDomains()) {
      const tpl = RECORD_EXTRACTOR_TEMPLATES[domain];
      expect(tpl).not.toContain('enterprise_ledger');
      expect(tpl).toContain('is_chunk: false');
      expect(tpl).toContain("status: 'indexed'");
    }
  });

  it('every template has the canonical signature ({ $, helpers, url, content, headers })', () => {
    for (const domain of listContentDomains()) {
      const tpl = RECORD_EXTRACTOR_TEMPLATES[domain];
      expect(tpl).toMatch(/\(\s*\{\s*\$,\s*helpers,\s*url,\s*content,\s*headers\s*\}\s*\)/);
    }
  });

  it('marketing template references headline + articleBody selectors', () => {
    const tpl = RECORD_EXTRACTOR_TEMPLATES.marketing;
    expect(tpl).toContain('{{headlineSelector}}');
    expect(tpl).toContain('{{articleBodySelector}}');
  });

  it('product-catalog template is DOM-extraction-first per §16o', () => {
    const tpl = RECORD_EXTRACTOR_TEMPLATES['product-catalog'];
    expect(tpl).toContain('{{nameSelector}}');
    expect(tpl).toContain('{{descriptionSelector}}');
    expect(tpl).toContain('variants'); // §16o: variants array
    // §16o: DOM selectors come BEFORE JSON-LD attempt for product-catalog
    const jsonLdIndex = tpl.indexOf('parseJsonLd');
    const domSelectorIndex = tpl.indexOf('{{nameSelector}}');
    expect(domSelectorIndex).toBeLessThan(jsonLdIndex);
  });
});

describe('getExtractorTemplate', () => {
  it('throws on unknown domain', () => {
    // @ts-expect-error — runtime check
    expect(() => getExtractorTemplate('made-up')).toThrow();
  });

  it('returns the marketing template', () => {
    expect(getExtractorTemplate('marketing')).toBe(RECORD_EXTRACTOR_TEMPLATES.marketing);
  });
});

describe('listTemplatePlaceholders', () => {
  it('extracts {{...}} placeholders from a template', () => {
    const placeholders = listTemplatePlaceholders(RECORD_EXTRACTOR_TEMPLATES.marketing);
    expect(placeholders).toContain('headlineSelector');
    expect(placeholders).toContain('articleBodySelector');
  });
});

[ ] Step 2.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/record-extractor-template.test.ts

Expected: FAIL: module not found.

[ ] Step 2.3: Implement lib/factory/record-extractor-template.ts

Create lib/factory/record-extractor-template.ts:

/**
 * Per-content-domain recordExtractor templates.
 *
 * Each template is a JS source string with the signature:
 *   ({ $, helpers, url, content, headers }) => Record[]
 *
 * Hardcoded base fields per Algolia Central convention:
 *   is_chunk:    false
 *   status:      'indexed'
 *
 * NOTE: per-domain content records do NOT carry `record_type` — that field is
 * reserved for the sharded session-storage index (see Spec 01 RecordBaseSchema).
 * Factory output flows into per-domain indices (`algoliacentral_marketing`,
 * `algoliacentral_support`, etc.) where the index name disambiguates.
 *
 * Placeholders ({{...}}) are filled by extractor-generator.ts using the
 * structureAnalysis output. Per §19f.7 (DSS as data, not code), adding a new
 * domain is a registry edit — no branching in consumers.
 *
 * For product-catalog (§16o): DOM-first, JSON-LD as bonus. For all other
 * domains: JSON-LD first, selectors as fallback (per Web Almanac §14a — JSON-LD
 * is reliable for the ~25% of pages where it covers content fields).
 */

import type { ContentDomain } from './types';

// ─── Marketing ──────────────────────────────────────────────────────────────

const MARKETING_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const headline = jsonLd?.headline || $('{{headlineSelector}}').first().text().trim();
  const articleBody = jsonLd?.articleBody || $('{{articleBodySelector}}').text().trim();
  if (!headline || !articleBody) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'marketing',
    schema_org_type: jsonLd?.['@type'] || 'Article',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.95 : 0.75,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    headline,
    articleBody,
    datePublished: jsonLd?.datePublished || $('{{datePublishedSelector}}').attr('datetime') || undefined,
    author: jsonLd?.author ? helpers.normalizeAuthor(jsonLd.author) : undefined,
    keywords: jsonLd?.keywords ? helpers.normalizeKeywords(jsonLd.keywords) : undefined,
  }];
}`;

// ─── Support ────────────────────────────────────────────────────────────────

const SUPPORT_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const name = jsonLd?.name || $('{{nameSelector}}').first().text().trim();
  const articleBody = jsonLd?.articleBody || $('{{articleBodySelector}}').text().trim();
  const acceptedAnswer = jsonLd?.acceptedAnswer?.text || $('{{acceptedAnswerSelector}}').text().trim() || undefined;
  if (!name || !articleBody) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'support',
    schema_org_type: jsonLd?.['@type'] || 'TechArticle',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.9 : 0.7,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    articleBody,
    acceptedAnswer,
    lastReviewed: jsonLd?.lastReviewed || undefined,
  }];
}`;

// ─── Education ──────────────────────────────────────────────────────────────

const EDUCATION_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const name = jsonLd?.name || $('{{nameSelector}}').first().text().trim();
  const description = jsonLd?.description || $('{{descriptionSelector}}').text().trim();
  if (!name || !description) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'education',
    schema_org_type: jsonLd?.['@type'] || 'Course',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.95 : 0.7,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    description,
    transcript: $('{{transcriptSelector}}').text().trim() || undefined,
    learningResourceType: jsonLd?.learningResourceType || undefined,
    educationalLevel: jsonLd?.educationalLevel || undefined,
  }];
}`;

// ─── Technical ──────────────────────────────────────────────────────────────

const TECHNICAL_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const name = jsonLd?.name || $('{{nameSelector}}').first().text().trim();
  const articleBody = jsonLd?.articleBody || $('{{articleBodySelector}}').text().trim();
  const programmingLanguage = $('{{programmingLanguageSelector}}').first().attr('class')?.match(/language-(\\w+)/)?.[1] || undefined;
  if (!name || !articleBody) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'technical',
    schema_org_type: jsonLd?.['@type'] || 'TechArticle',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.9 : 0.7,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    articleBody,
    programmingLanguage,
    version: jsonLd?.version || undefined,
  }];
}`;

// ─── Customer stories ───────────────────────────────────────────────────────

const CUSTOMER_STORIES_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const headline = jsonLd?.headline || $('{{headlineSelector}}').first().text().trim();
  const articleBody = jsonLd?.articleBody || $('{{articleBodySelector}}').text().trim();
  const aboutName = jsonLd?.about?.name || $('{{aboutNameSelector}}').first().text().trim() || undefined;
  const quotes = $('{{quotesSelector}}').map((_, el) => $(el).text().trim()).get().filter(Boolean);
  if (!headline || !articleBody) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'customer-stories',
    schema_org_type: jsonLd?.['@type'] || 'Article',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.85 : 0.7,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    headline,
    articleBody,
    about: aboutName ? { name: aboutName } : undefined,
    quotes: quotes.length > 0 ? quotes : undefined,
  }];
}`;

// ─── Product catalog (DOM-FIRST per §16o) ───────────────────────────────────

const PRODUCT_CATALOG_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  // §16o: DOM-extraction-first; JSON-LD is a bonus enrichment, not the lead.
  const name = $('{{nameSelector}}').first().text().trim();
  const description = $('{{descriptionSelector}}').first().text().trim();
  const priceRaw = $('{{priceSelector}}').first().text().trim();
  const images = $('{{imageSelector}}').map((_, el) => $(el).attr('src')).get().filter(Boolean);
  const variants = $('{{variantSelector}}').map((_, el) => ({
    color: $(el).attr('data-color') || $(el).attr('aria-label') || undefined,
    size: $(el).attr('data-size') || undefined,
    sku: $(el).attr('data-sku') || undefined,
    available: $(el).attr('data-available') !== 'false',
  })).get();
  // JSON-LD enriches whatever DOM gave us
  const jsonLd = helpers.parseJsonLd($);
  if (!name || !description) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'product-catalog',
    schema_org_type: jsonLd?.['@type'] || 'Product',
    detected_via: jsonLd ? 'json-ld' : 'heuristic',
    detection_confidence: jsonLd ? 0.85 : 0.65,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    description,
    sku: jsonLd?.sku || undefined,
    brand: jsonLd?.brand?.name || jsonLd?.brand || undefined,
    image: images.length > 0 ? images : undefined,
    variants: variants.length > 0 ? variants : undefined,
    offers: priceRaw ? { price: helpers.parsePrice(priceRaw), priceCurrency: helpers.parseCurrency(priceRaw) } : undefined,
  }];
}`;

// ─── Events ─────────────────────────────────────────────────────────────────

const EVENTS_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const name = jsonLd?.name || $('{{nameSelector}}').first().text().trim();
  const description = jsonLd?.description || $('{{descriptionSelector}}').text().trim();
  const startDate = jsonLd?.startDate || $('{{startDateSelector}}').attr('datetime') || $('{{startDateSelector}}').text().trim();
  if (!name || !description || !startDate) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'events',
    schema_org_type: jsonLd?.['@type'] || 'Event',
    detected_via: jsonLd ? 'json-ld' : 'cms',
    detection_confidence: jsonLd ? 0.95 : 0.7,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    description,
    startDate,
    endDate: jsonLd?.endDate || undefined,
    location: jsonLd?.location?.name || $('{{locationSelector}}').text().trim() || undefined,
    organizer: jsonLd?.organizer?.name || undefined,
  }];
}`;

// ─── Legal ──────────────────────────────────────────────────────────────────

const LEGAL_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const name = jsonLd?.name || $('{{nameSelector}}').first().text().trim();
  const text = $('{{textSelector}}').text().trim();
  if (!name || !text) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'legal',
    schema_org_type: jsonLd?.['@type'] || 'DigitalDocument',
    detected_via: jsonLd ? 'json-ld' : 'url',
    detection_confidence: jsonLd ? 0.9 : 0.75,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    name,
    text,
    version: jsonLd?.version || undefined,
    datePublished: jsonLd?.datePublished || undefined,
  }];
}`;

// ─── Social ─────────────────────────────────────────────────────────────────

const SOCIAL_TEMPLATE = `({ $, helpers, url, content, headers }) => {
  const jsonLd = helpers.parseJsonLd($);
  const text = $('{{textSelector}}').text().trim();
  if (!text || text.length < 50) return [];
  return [{
    objectID: helpers.urlHash(url),
    url,
    content_domain: 'social',
    schema_org_type: jsonLd?.['@type'] || 'VideoObject',
    detected_via: jsonLd ? 'json-ld' : 'url',
    detection_confidence: jsonLd ? 0.85 : 0.65,
    language_code: helpers.detectLanguage($) || 'en',
    source_url_root: helpers.urlRoot(url),
    crawled_at: Date.now(),
    is_chunk: false,
    status: 'indexed',
    text,
    transcript: $('{{transcriptSelector}}').text().trim() || undefined,
    uploadDate: jsonLd?.uploadDate || undefined,
    platform: helpers.detectPlatform(url) || undefined,
  }];
}`;

// ─── Registry ───────────────────────────────────────────────────────────────

export const RECORD_EXTRACTOR_TEMPLATES: Readonly<Record<ContentDomain, string>> = Object.freeze({
  marketing: MARKETING_TEMPLATE,
  support: SUPPORT_TEMPLATE,
  education: EDUCATION_TEMPLATE,
  technical: TECHNICAL_TEMPLATE,
  'customer-stories': CUSTOMER_STORIES_TEMPLATE,
  'product-catalog': PRODUCT_CATALOG_TEMPLATE,
  events: EVENTS_TEMPLATE,
  legal: LEGAL_TEMPLATE,
  social: SOCIAL_TEMPLATE,
});

/**
 * Get the template string for a given content domain. Throws on unknown domain.
 */
export function getExtractorTemplate(domain: ContentDomain): string {
  const tpl = RECORD_EXTRACTOR_TEMPLATES[domain];
  if (!tpl) throw new Error(`No extractor template for domain: ${domain}`);
  return tpl;
}

/**
 * Extract every {{name}} placeholder from a template (deduplicated, ordered by first appearance).
 */
export function listTemplatePlaceholders(tpl: string): string[] {
  const seen = new Set<string>();
  const out: string[] = [];
  const re = /\{\{(\w+)\}\}/g;
  let match: RegExpExecArray | null;
  while ((match = re.exec(tpl)) !== null) {
    if (!seen.has(match[1])) {
      seen.add(match[1]);
      out.push(match[1]);
    }
  }
  return out;
}

[ ] Step 2.4: Run tests, expect PASS

npx vitest run lib/factory/__tests__/record-extractor-template.test.ts

Expected: 7 tests pass (including the per-domain registry-completeness check).

[ ] Step 2.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 2.6: Commit

git add lib/factory/record-extractor-template.ts lib/factory/__tests__/record-extractor-template.test.ts
git commit -m "feat(factory): per-domain recordExtractor templates (DOM-first for products)"

Task 3: Extractor generator

Files: - Create: lib/factory/extractor-generator.ts - Create: lib/factory/__tests__/extractor-generator.test.ts

generateExtractor pulls the per-domain template, asks the LLM to fill {{placeholders}} given the structureAnalysis + DSS row + optional userFeedback, validates the result via new Function('return ' + src), and returns {recordExtractor, crawlerConfig}.

The crawlerConfig derives pathsToMatch from the pathGroup pattern, renderJavaScript: false for v1 (Spec 03's Playwright handles WAF; the crawler sees pre-rendered HTML), and rateLimit: 8 as the conservative default.

[ ] Step 3.1: Write failing test for happy-path generation

Create lib/factory/__tests__/extractor-generator.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { generateExtractor } from '../extractor-generator';
import { getDss } from '../dss';
import type { StructureAnalysis } from '../structure-analyzer';

const mkPathGroup = (overrides: Partial<{ id: string; pattern: string; sampleUrls: string[] }> = {}) => ({
  id: 'pg_blog',
  pattern: '/blog/*',
  sampleUrls: ['https://x.com/blog/p1'],
  urlCount: 100,
  detectedDomain: null,
  detectionConfidence: null,
  detectionMethod: null,
  selected: false,
  ...overrides,
});

const mkAnalysis = (overrides: Partial<StructureAnalysis> = {}): StructureAnalysis => ({
  schemaOrgJsonLd: null,
  selectors: { headline: 'h1', articleBody: 'article' },
  missing: [],
  confidence: 0.85,
  ...overrides,
});

describe('generateExtractor', () => {
  it('returns a recordExtractor JS string + crawlerConfig for marketing', async () => {
    const llm = vi.fn().mockResolvedValue(JSON.stringify({
      headlineSelector: 'h1',
      articleBodySelector: 'article',
      datePublishedSelector: 'time[datetime]',
    }));

    const result = await generateExtractor({
      pathGroup: mkPathGroup(),
      structureAnalysis: mkAnalysis(),
      dssEntry: getDss('marketing'),
      contentDomain: 'marketing',
      llm,
    });

    expect(result.recordExtractor).toContain('content_domain: \'marketing\'');
    expect(result.recordExtractor).not.toContain('{{'); // no unfilled placeholders
    expect(result.crawlerConfig.pathsToMatch).toEqual(['/blog/*']);
    expect(result.crawlerConfig.renderJavaScript).toBe(false);
    expect(result.crawlerConfig.rateLimit).toBeGreaterThan(0);
  });

  it('throws when generated extractor is not valid JS', async () => {
    const llm = vi.fn().mockResolvedValue('this is not valid JSON for placeholder filling');

    await expect(
      generateExtractor({
        pathGroup: mkPathGroup(),
        structureAnalysis: mkAnalysis(),
        dssEntry: getDss('marketing'),
        contentDomain: 'marketing',
        llm,
      })
    ).rejects.toThrow();
  });

  it('prepends userFeedback to the LLM prompt when provided', async () => {
    const llm = vi.fn().mockResolvedValue(JSON.stringify({
      headlineSelector: 'h2.custom',
      articleBodySelector: '.content',
      datePublishedSelector: 'time',
    }));

    await generateExtractor({
      pathGroup: mkPathGroup(),
      structureAnalysis: mkAnalysis(),
      dssEntry: getDss('marketing'),
      contentDomain: 'marketing',
      llm,
      userFeedback: 'Use h2.custom for the title, not h1.',
    });

    const promptArg = llm.mock.calls[0][0] as string;
    expect(promptArg).toContain('Use h2.custom for the title, not h1.');
    expect(promptArg.indexOf('Use h2.custom')).toBeLessThan(promptArg.indexOf('selectors'));
  });

  it('passes through structureAnalysis selectors when LLM omits them', async () => {
    const llm = vi.fn().mockResolvedValue(JSON.stringify({})); // LLM returns empty

    const analysis = mkAnalysis({ selectors: { headline: 'h1.special', articleBody: '.body' } });
    const result = await generateExtractor({
      pathGroup: mkPathGroup(),
      structureAnalysis: analysis,
      dssEntry: getDss('marketing'),
      contentDomain: 'marketing',
      llm,
    });

    expect(result.recordExtractor).toContain('h1.special');
    expect(result.recordExtractor).toContain('.body');
  });

  it('product-catalog generates without throwing (DOM-first template)', async () => {
    const llm = vi.fn().mockResolvedValue(JSON.stringify({
      nameSelector: 'h1.product-name',
      descriptionSelector: '.product-description',
      priceSelector: '.price',
      imageSelector: '.product-image img',
      variantSelector: '.variant-option',
    }));

    const result = await generateExtractor({
      pathGroup: mkPathGroup({ id: 'pg_products', pattern: '/products/*' }),
      structureAnalysis: mkAnalysis({ selectors: { name: 'h1.product-name', description: '.product-description' } }),
      dssEntry: getDss('product-catalog'),
      contentDomain: 'product-catalog',
      llm,
    });

    expect(result.recordExtractor).toContain('content_domain: \'product-catalog\'');
    expect(result.recordExtractor).toContain('variants');
    expect(result.crawlerConfig.pathsToMatch).toEqual(['/products/*']);
  });
});

[ ] Step 3.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/extractor-generator.test.ts

Expected: FAIL: module not found.

[ ] Step 3.3: Implement lib/factory/extractor-generator.ts

Create lib/factory/extractor-generator.ts:

/**
 * Per-domain extractor generator.
 *
 * Pipeline:
 *   1. Look up the per-domain template from RECORD_EXTRACTOR_TEMPLATES.
 *   2. Build an LLM prompt: DSS row + structureAnalysis + optional userFeedback +
 *      list of {{placeholders}} the LLM must fill (as JSON).
 *   3. Parse LLM output as JSON; merge with structureAnalysis.selectors as fallback.
 *   4. String-replace placeholders in the template.
 *   5. Validate via `new Function('return ' + src)` — throws if syntax error.
 *   6. Build crawlerConfig from pathGroup.pattern + DSS conventions.
 */

import { z } from 'zod';
import { createLogger } from '../utils/logger';
import { getExtractorTemplate, listTemplatePlaceholders } from './record-extractor-template';
import type { StructureAnalysis } from './structure-analyzer';
import type { ContentDomain, PathGroup } from './types';
import type { DssEntry } from './dss';

const log = createLogger('factory:extractor-generator');

const DEFAULT_RATE_LIMIT = 8;

// ─── Public types ───────────────────────────────────────────────────────────

export const CrawlerConfigSchema = z.object({
  renderJavaScript: z.boolean(),
  rateLimit: z.number().int().positive(),
  pathsToMatch: z.array(z.string()),
});
export type CrawlerConfig = z.infer<typeof CrawlerConfigSchema>;

export const GenerateExtractorResultSchema = z.object({
  recordExtractor: z.string().min(1),
  crawlerConfig: CrawlerConfigSchema,
});
export type GenerateExtractorResult = z.infer<typeof GenerateExtractorResultSchema>;

export type LlmFn = (prompt: string) => Promise<string>;

export interface GenerateExtractorOpts {
  pathGroup: Pick<PathGroup, 'id' | 'pattern' | 'sampleUrls'>;
  structureAnalysis: StructureAnalysis;
  dssEntry: DssEntry;
  contentDomain: ContentDomain;
  llm: LlmFn;
  userFeedback?: string;
}

// ─── Prompt construction ────────────────────────────────────────────────────

function buildPrompt(opts: GenerateExtractorOpts, placeholders: string[]): string {
  const { dssEntry, structureAnalysis, contentDomain, userFeedback } = opts;
  const feedback = userFeedback ? `USER FEEDBACK (apply this first):\n${userFeedback}\n\n` : '';
  return [
    feedback,
    `You are filling placeholders in a recordExtractor template for the "${contentDomain}" content domain.\n`,
    `DSS description: ${dssEntry.description}\n`,
    `Existing selectors discovered by structure analyzer: ${JSON.stringify(structureAnalysis.selectors)}\n`,
    `JSON-LD detected: ${structureAnalysis.schemaOrgJsonLd ? 'yes' : 'no'}\n`,
    `Placeholders to fill: ${JSON.stringify(placeholders)}\n`,
    `Return ONLY a JSON object mapping each placeholder to a CSS selector string. No prose, no markdown.`,
  ].join('');
}

// ─── LLM-output parsing (best-effort) ───────────────────────────────────────

function parseLlmFillings(raw: string): Record<string, string> {
  try {
    const cleaned = raw.trim().replace(/^```(?:json)?/i, '').replace(/```$/, '').trim();
    const parsed = JSON.parse(cleaned);
    if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return {};
    const out: Record<string, string> = {};
    for (const [k, v] of Object.entries(parsed)) {
      if (typeof v === 'string' && v.length > 0 && v.length < 200) out[k] = v;
    }
    return out;
  } catch (err) {
    log.warn({ event: 'parseLlm_failed', err: String(err) }, 'LLM output not valid JSON');
    return {};
  }
}

// ─── Placeholder fill — LLM first, structureAnalysis as fallback ────────────

function resolveFillings(
  placeholders: string[],
  llmFillings: Record<string, string>,
  analysisSelectors: Record<string, string>
): Record<string, string> {
  const out: Record<string, string> = {};
  for (const p of placeholders) {
    if (llmFillings[p]) {
      out[p] = llmFillings[p];
      continue;
    }
    // Strip "Selector" suffix to map placeholder → analysis field name
    const fieldName = p.endsWith('Selector') ? p.slice(0, -'Selector'.length) : p;
    if (analysisSelectors[fieldName]) {
      out[p] = analysisSelectors[fieldName];
      continue;
    }
    // Last-resort: use a permissive selector that won't match anything (so the
    // record-validator filters it out) rather than leaving an unfilled placeholder
    out[p] = `[data-missing-${p}]`;
  }
  return out;
}

// ─── Template substitution ──────────────────────────────────────────────────

function fillTemplate(tpl: string, fillings: Record<string, string>): string {
  return tpl.replace(/\{\{(\w+)\}\}/g, (_match, name) => {
    const v = fillings[name];
    if (!v) return `[data-missing-${name}]`;
    return v;
  });
}

// ─── Crawler config ─────────────────────────────────────────────────────────

function buildCrawlerConfig(pattern: string): CrawlerConfig {
  return {
    renderJavaScript: false,
    rateLimit: DEFAULT_RATE_LIMIT,
    pathsToMatch: [pattern],
  };
}

// ─── Public entrypoint ──────────────────────────────────────────────────────

/**
 * Generate a recordExtractor JS string + crawlerConfig for one path group.
 */
export async function generateExtractor(opts: GenerateExtractorOpts): Promise<GenerateExtractorResult> {
  const tpl = getExtractorTemplate(opts.contentDomain);
  const placeholders = listTemplatePlaceholders(tpl);

  let llmRaw = '';
  try {
    llmRaw = await opts.llm(buildPrompt(opts, placeholders));
  } catch (err) {
    log.error(
      { event: 'llm_call_failed', pathGroupId: opts.pathGroup.id, err: String(err) },
      'LLM call for extractor generation failed'
    );
    throw err;
  }

  const llmFillings = parseLlmFillings(llmRaw);
  const fillings = resolveFillings(placeholders, llmFillings, opts.structureAnalysis.selectors);
  const recordExtractor = fillTemplate(tpl, fillings);

  // Validate via new Function — throws if there's a syntax error
  try {
    // eslint-disable-next-line no-new-func
    new Function(`return (${recordExtractor})`);
  } catch (err) {
    log.error(
      { event: 'extractor_syntax_invalid', pathGroupId: opts.pathGroup.id, err: String(err) },
      'generated extractor failed syntactic validation'
    );
    throw new Error(`Generated extractor is not valid JS: ${String(err)}`);
  }

  const crawlerConfig = buildCrawlerConfig(opts.pathGroup.pattern);
  log.info(
    {
      event: 'extractor_generated',
      pathGroupId: opts.pathGroup.id,
      domain: opts.contentDomain,
      placeholdersFilled: placeholders.length,
    },
    'extractor generated'
  );

  return GenerateExtractorResultSchema.parse({ recordExtractor, crawlerConfig });
}

[ ] Step 3.4: Run tests, expect PASS

npx vitest run lib/factory/__tests__/extractor-generator.test.ts

Expected: 5 tests pass.

[ ] Step 3.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 3.6: Commit

git add lib/factory/extractor-generator.ts lib/factory/__tests__/extractor-generator.test.ts
git commit -m "feat(factory): per-domain extractor generator with LLM placeholder fill"

Task 4: Sandbox runner

Files: - Create: lib/factory/sandbox-runner.ts - Create: lib/factory/__tests__/sandbox-runner.test.ts

The sandbox runner instantiates the extractor JS via new Function, runs it against the sample HTML loaded with cheerio, and validates each emitted record against the DSS-derived zod schema (getRecordSchema(contentDomain)). Records that fail validation land in errors[] with the zod message; records that pass land in records[]. If the extractor itself throws (bad selector, runtime error), that's caught and reported as a single error entry: no crash.

helpers exposes the small set of utilities the templates expect: parseJsonLd, urlHash, urlRoot, detectLanguage, normalizeAuthor, normalizeKeywords, parsePrice, parseCurrency, detectPlatform. These are deliberately simple: the factory's value is in correctness, not extraction ML.

[ ] Step 4.1: Write failing test

Create lib/factory/__tests__/sandbox-runner.test.ts:

import { describe, it, expect } from 'vitest';
import { runInSandbox } from '../sandbox-runner';

const mkSample = (html: string, url = 'https://x.com/blog/p1') => ({
  url,
  html,
  status: 200,
});

describe('runInSandbox', () => {
  it('returns a passing marketing record from a valid extractor + sample', async () => {
    const extractor = `({ $, helpers, url }) => {
      const headline = $('h1').first().text().trim();
      const articleBody = $('article').text().trim();
      if (!headline || !articleBody) return [];
      return [{
        objectID: helpers.urlHash(url),
        url,
        content_domain: 'marketing',
        schema_org_type: 'BlogPosting',
        detected_via: 'cms',
        detection_confidence: 0.8,
        language_code: 'en',
        source_url_root: helpers.urlRoot(url),
        crawled_at: Date.now(),
        is_chunk: false,
        status: 'indexed',
        headline,
        articleBody,
      }];
    }`;

    const html = `<html><body>
      <h1>Hello world</h1>
      <article>${'word '.repeat(80)}</article>
    </body></html>`;

    const result = await runInSandbox(extractor, mkSample(html), 'marketing');
    expect(result.records).toHaveLength(1);
    expect(result.errors).toHaveLength(0);
    expect(result.records[0]).toMatchObject({ headline: 'Hello world', content_domain: 'marketing' });
  });

  it('records with missing required fields land in errors, not records', async () => {
    // articleBody too short — fails MarketingRecordSchema's min(200)
    const extractor = `({ $, helpers, url }) => {
      return [{
        objectID: helpers.urlHash(url),
        url,
        content_domain: 'marketing',
        schema_org_type: 'BlogPosting',
        detected_via: 'cms',
        detection_confidence: 0.8,
        language_code: 'en',
        source_url_root: helpers.urlRoot(url),
        crawled_at: Date.now(),
        is_chunk: false,
        status: 'indexed',
        headline: 'Hello',
        articleBody: 'too short',
      }];
    }`;

    const result = await runInSandbox(extractor, mkSample('<html></html>'), 'marketing');
    expect(result.records).toHaveLength(0);
    expect(result.errors.length).toBeGreaterThan(0);
    expect(result.errors[0].message).toMatch(/articleBody/i);
  });

  it('catches a thrown extractor without crashing the sandbox', async () => {
    const extractor = `({ $ }) => { throw new Error('selector typo'); }`;
    const result = await runInSandbox(extractor, mkSample('<html></html>'), 'marketing');
    expect(result.records).toHaveLength(0);
    expect(result.errors).toHaveLength(1);
    expect(result.errors[0].message).toMatch(/selector typo/);
  });

  it('handles multiple records emitted from one page', async () => {
    const extractor = `({ $, helpers, url }) => {
      return $('article').map((i, el) => ({
        objectID: helpers.urlHash(url) + '_' + i,
        url,
        content_domain: 'marketing',
        schema_org_type: 'BlogPosting',
        detected_via: 'cms',
        detection_confidence: 0.8,
        language_code: 'en',
        source_url_root: helpers.urlRoot(url),
        crawled_at: Date.now(),
        is_chunk: false,
        status: 'indexed',
        headline: 'Item ' + i,
        articleBody: $(el).text().trim(),
      })).get();
    }`;
    const html = `<html><body>
      <article>${'a '.repeat(150)}</article>
      <article>${'b '.repeat(150)}</article>
    </body></html>`;
    const result = await runInSandbox(extractor, mkSample(html), 'marketing');
    expect(result.records).toHaveLength(2);
    expect(result.records[0].headline).toBe('Item 0');
    expect(result.records[1].headline).toBe('Item 1');
  });

  it('rejects extractor with a JS syntax error', async () => {
    const extractor = `({ $ }) => { return [ { unclosed: `;
    const result = await runInSandbox(extractor, mkSample('<html></html>'), 'marketing');
    expect(result.records).toHaveLength(0);
    expect(result.errors).toHaveLength(1);
    expect(result.errors[0].message).toMatch(/syntax|unexpected|invalid/i);
  });
});

[ ] Step 4.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/sandbox-runner.test.ts

Expected: FAIL: module not found.

[ ] Step 4.3: Implement lib/factory/sandbox-runner.ts

Create lib/factory/sandbox-runner.ts:

/**
 * Sandbox runner — execute a generated recordExtractor against a sample page
 * and validate every emitted record against the DSS-derived zod schema.
 *
 * Two failure modes captured:
 *   1. Extractor throws (bad selector, runtime error) — single error entry
 *   2. Emitted record fails zod validation — per-record error entry
 *
 * No real I/O; safe to run inline. Out of scope: VM isolation (the extractor
 * runs in the same process; we only crawler-trust extractors we generated).
 */

import * as cheerio from 'cheerio';
import { createLogger } from '../utils/logger';
import { getRecordSchema } from './dss';
import type { ContentDomain } from './types';

const log = createLogger('factory:sandbox-runner');

// ─── Public types ───────────────────────────────────────────────────────────

export interface SandboxSample {
  url: string;
  html: string;
  status: number;
  headers?: Record<string, string>;
}

export interface SandboxError {
  type: 'extractor_threw' | 'syntax_error' | 'validation_failed';
  message: string;
  recordIndex?: number;
}

export interface SandboxResult {
  records: Record<string, unknown>[];
  errors: SandboxError[];
}

// ─── Helpers exposed to extractor templates ─────────────────────────────────

function urlHash(url: string): string {
  // Deterministic short hash — fine for sandbox use; production uses crypto
  let h = 0;
  for (let i = 0; i < url.length; i += 1) {
    h = (h * 31 + url.charCodeAt(i)) | 0;
  }
  return Math.abs(h).toString(36);
}

function urlRoot(url: string): string {
  try {
    const u = new URL(url);
    return `${u.protocol}//${u.host}`;
  } catch {
    return url;
  }
}

function parseJsonLd($: cheerio.CheerioAPI): Record<string, unknown> | null {
  const blocks: Record<string, unknown>[] = [];
  $('script[type="application/ld+json"]').each((_, el) => {
    try {
      const raw = $(el).html();
      if (!raw) return;
      const parsed = JSON.parse(raw);
      if (Array.isArray(parsed)) blocks.push(...parsed);
      else blocks.push(parsed);
    } catch {
      // ignore malformed blocks
    }
  });
  const typed = blocks.find((b) => typeof (b as { '@type'?: unknown })['@type'] === 'string');
  return typed ?? blocks[0] ?? null;
}

function detectLanguage($: cheerio.CheerioAPI): string | null {
  const lang = $('html').attr('lang');
  return lang ? lang.split('-')[0] : null;
}

function normalizeAuthor(input: unknown): string[] | undefined {
  if (Array.isArray(input)) {
    return input.map((a) => (typeof a === 'string' ? a : (a as { name?: string }).name)).filter(Boolean) as string[];
  }
  if (typeof input === 'string') return [input];
  if (input && typeof input === 'object' && 'name' in input) return [(input as { name: string }).name];
  return undefined;
}

function normalizeKeywords(input: unknown): string[] | undefined {
  if (Array.isArray(input)) return input.filter((k): k is string => typeof k === 'string');
  if (typeof input === 'string') return input.split(',').map((s) => s.trim()).filter(Boolean);
  return undefined;
}

function parsePrice(raw: string): number | undefined {
  const m = raw.match(/[\d,]+\.?\d*/);
  if (!m) return undefined;
  const n = parseFloat(m[0].replace(/,/g, ''));
  return Number.isFinite(n) ? n : undefined;
}

function parseCurrency(raw: string): string | undefined {
  if (raw.includes('$')) return 'USD';
  if (raw.includes('€')) return 'EUR';
  if (raw.includes('£')) return 'GBP';
  if (raw.includes('¥')) return 'JPY';
  return undefined;
}

function detectPlatform(url: string): string | undefined {
  if (url.includes('youtube.com')) return 'youtube';
  if (url.includes('twitter.com') || url.includes('x.com')) return 'twitter';
  if (url.includes('linkedin.com')) return 'linkedin';
  if (url.includes('facebook.com')) return 'facebook';
  return undefined;
}

const HELPERS = Object.freeze({
  urlHash,
  urlRoot,
  parseJsonLd,
  detectLanguage,
  normalizeAuthor,
  normalizeKeywords,
  parsePrice,
  parseCurrency,
  detectPlatform,
});

// ─── Public entrypoint ──────────────────────────────────────────────────────

/**
 * Run a generated extractor against a sample page; validate against DSS schema.
 */
export async function runInSandbox(
  extractorSrc: string,
  sample: SandboxSample,
  contentDomain: ContentDomain
): Promise<SandboxResult> {
  const records: Record<string, unknown>[] = [];
  const errors: SandboxError[] = [];

  let fn: (ctx: unknown) => unknown;
  try {
    // eslint-disable-next-line no-new-func
    fn = new Function(`return (${extractorSrc})`)() as (ctx: unknown) => unknown;
  } catch (err) {
    log.error({ event: 'sandbox_syntax_error', err: String(err) }, 'extractor failed to instantiate');
    return { records: [], errors: [{ type: 'syntax_error', message: String(err) }] };
  }

  let raw: unknown;
  try {
    const $ = cheerio.load(sample.html);
    raw = fn({
      $,
      helpers: HELPERS,
      url: sample.url,
      content: $.root().text(),
      headers: sample.headers ?? {},
    });
  } catch (err) {
    log.warn({ event: 'sandbox_extractor_threw', url: sample.url, err: String(err) }, 'extractor threw');
    return { records: [], errors: [{ type: 'extractor_threw', message: String(err) }] };
  }

  const list = Array.isArray(raw) ? raw : [];
  const schema = getRecordSchema(contentDomain);
  list.forEach((rec, idx) => {
    const parsed = schema.safeParse(rec);
    if (parsed.success) {
      records.push(parsed.data as Record<string, unknown>);
    } else {
      errors.push({
        type: 'validation_failed',
        message: parsed.error.issues.map((i) => `${i.path.join('.')}: ${i.message}`).join('; '),
        recordIndex: idx,
      });
    }
  });

  log.info(
    { event: 'sandbox_done', url: sample.url, domain: contentDomain, passing: records.length, failing: errors.length },
    'sandbox run complete'
  );
  return { records, errors };
}

[ ] Step 4.4: Run tests, expect PASS

npx vitest run lib/factory/__tests__/sandbox-runner.test.ts

Expected: 5 tests pass.

[ ] Step 4.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 4.6: Commit

git add lib/factory/sandbox-runner.ts lib/factory/__tests__/sandbox-runner.test.ts
git commit -m "feat(factory): sandbox runner with DSS zod validation + extractor isolation"

Task 5: Canonical inferer (DTC variant deduplication)

Files: - Create: lib/factory/canonical-inferer.ts - Create: lib/factory/__tests__/canonical-inferer.test.ts

Per §16o: SKIMS-style sites encode color/size variants as separate URLs (/products/bra-clay, /products/bra-onyx) without canonical tags, leading to variant explosion at crawl time. The inferer's job is to resolve a variant URL to its canonical so create-crawler (Spec 06) can dedupe before it indexes.

Strategy (in order): 1. Parse <link rel="canonical"> from HTML: wins if present. 2. Heuristic for product-catalog domain: strip trailing URL-suffix matching known color/size patterns. 3. LLM fallback: pass N variant URLs and ask "what's the canonical?".

Non-product domains skip the heuristic (no variant explosion problem).

[ ] Step 5.1: Write failing tests

Create lib/factory/__tests__/canonical-inferer.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { inferCanonical, stripKnownVariantSuffix } from '../canonical-inferer';

describe('inferCanonical', () => {
  it('returns the value of <link rel="canonical"> when present', async () => {
    const html = `<html><head>
      <link rel="canonical" href="https://x.com/products/bra" />
    </head></html>`;
    const result = await inferCanonical({
      url: 'https://x.com/products/bra-clay',
      html,
      domain: 'product-catalog',
    });
    expect(result).toBe('https://x.com/products/bra');
  });

  it('strips a known color suffix for product-catalog when no canonical link', async () => {
    const html = '<html></html>';
    const result = await inferCanonical({
      url: 'https://x.com/products/bra-onyx',
      html,
      domain: 'product-catalog',
    });
    expect(result).toBe('https://x.com/products/bra');
  });

  it('strips a size suffix for product-catalog', async () => {
    const html = '<html></html>';
    const result = await inferCanonical({
      url: 'https://x.com/products/sweater-large',
      html,
      domain: 'product-catalog',
    });
    expect(result).toBe('https://x.com/products/sweater');
  });

  it('does NOT strip suffix for non-product-catalog domains', async () => {
    const html = '<html></html>';
    const result = await inferCanonical({
      url: 'https://x.com/blog/post-clay',
      html,
      domain: 'marketing',
    });
    expect(result).toBe('https://x.com/blog/post-clay');
  });

  it('falls back to LLM when neither canonical link nor heuristic resolves', async () => {
    const html = '<html></html>';
    const llm = vi.fn().mockResolvedValue('https://x.com/products/bra');

    const result = await inferCanonical({
      url: 'https://x.com/products/bra-special-fabric-edition',
      html,
      domain: 'product-catalog',
      llm,
      siblingUrls: [
        'https://x.com/products/bra-special-fabric-edition',
        'https://x.com/products/bra-other-edition',
      ],
    });
    expect(result).toBe('https://x.com/products/bra');
    expect(llm).toHaveBeenCalledTimes(1);
  });

  it('returns the original URL when LLM is absent and heuristic does not fire', async () => {
    const html = '<html></html>';
    const result = await inferCanonical({
      url: 'https://x.com/products/sku-abc-123-no-pattern',
      html,
      domain: 'product-catalog',
    });
    expect(result).toBe('https://x.com/products/sku-abc-123-no-pattern');
  });
});

describe('stripKnownVariantSuffix', () => {
  it('strips -clay (color)', () => {
    expect(stripKnownVariantSuffix('https://x.com/products/bra-clay')).toBe('https://x.com/products/bra');
  });

  it('strips -large (size)', () => {
    expect(stripKnownVariantSuffix('https://x.com/products/shirt-large')).toBe('https://x.com/products/shirt');
  });

  it('returns null when no known suffix matches', () => {
    expect(stripKnownVariantSuffix('https://x.com/products/random-thing')).toBeNull();
  });

  it('strips trailing -black (color)', () => {
    expect(stripKnownVariantSuffix('https://x.com/products/jacket-black')).toBe('https://x.com/products/jacket');
  });
});

[ ] Step 5.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/canonical-inferer.test.ts

Expected: FAIL: module not found.

[ ] Step 5.3: Implement lib/factory/canonical-inferer.ts

Create lib/factory/canonical-inferer.ts:

/**
 * Canonical URL inferer — used to deduplicate DTC variant URLs (§16o).
 *
 * Strategy ladder:
 *   1. <link rel="canonical"> — wins if present.
 *   2. URL-suffix heuristic (product-catalog only) — strip trailing -<color>
 *      or -<size> patterns matching a known palette.
 *   3. LLM fallback — given N sibling variant URLs, ask "what's the canonical?".
 *
 * Returns the input URL unchanged when no rule fires + no LLM provided.
 */

import * as cheerio from 'cheerio';
import { createLogger } from '../utils/logger';
import type { ContentDomain } from './types';

const log = createLogger('factory:canonical-inferer');

// ─── Known variant tokens (kept conservative; expand from §16o evidence) ────

const KNOWN_COLORS = new Set([
  'black', 'white', 'red', 'blue', 'green', 'yellow', 'purple', 'pink',
  'orange', 'brown', 'gray', 'grey', 'beige', 'navy', 'cream', 'ivory',
  'gold', 'silver', 'bronze', 'tan', 'olive', 'mint', 'coral', 'teal',
  'clay', 'onyx', 'sand', 'stone', 'sage', 'rose', 'mauve', 'charcoal',
  'mocha', 'cocoa', 'rust', 'plum', 'wine', 'forest', 'sky', 'cobalt',
]);

const KNOWN_SIZES = new Set([
  'xs', 's', 'm', 'l', 'xl', 'xxl', 'xxxl',
  'small', 'medium', 'large', 'xlarge', 'xxlarge',
  'mini', 'regular', 'tall', 'short', 'plus',
]);

// ─── Public types ───────────────────────────────────────────────────────────

export type LlmFn = (prompt: string) => Promise<string>;

export interface InferCanonicalOpts {
  url: string;
  html: string;
  domain: ContentDomain;
  llm?: LlmFn;
  siblingUrls?: string[];
}

// ─── Stage 1: <link rel="canonical"> ────────────────────────────────────────

function readCanonicalLink(html: string): string | null {
  try {
    const $ = cheerio.load(html);
    const href = $('link[rel="canonical"]').attr('href');
    return href && href.length > 0 ? href : null;
  } catch (err) {
    log.warn({ event: 'canonical_link_parse_failed', err: String(err) }, 'failed to parse <link rel=canonical>');
    return null;
  }
}

// ─── Stage 2: URL-suffix heuristic ──────────────────────────────────────────

/**
 * If the URL's last path segment ends with -<color> or -<size>, strip that
 * trailing token. Returns null if no match.
 */
export function stripKnownVariantSuffix(url: string): string | null {
  let parsed: URL;
  try {
    parsed = new URL(url);
  } catch {
    return null;
  }
  const segments = parsed.pathname.split('/');
  const last = segments[segments.length - 1];
  if (!last || !last.includes('-')) return null;
  const tokens = last.split('-');
  const trailing = tokens[tokens.length - 1].toLowerCase();
  if (!KNOWN_COLORS.has(trailing) && !KNOWN_SIZES.has(trailing)) return null;
  if (tokens.length < 2) return null;
  segments[segments.length - 1] = tokens.slice(0, -1).join('-');
  parsed.pathname = segments.join('/');
  return parsed.toString().replace(/\/$/, '');
}

// ─── Stage 3: LLM fallback ──────────────────────────────────────────────────

async function llmFallback(opts: InferCanonicalOpts): Promise<string | null> {
  if (!opts.llm) return null;
  try {
    const siblings = opts.siblingUrls && opts.siblingUrls.length > 0
      ? opts.siblingUrls.slice(0, 10)
      : [opts.url];
    const prompt = [
      'Given these variant URLs from the same product catalog, return the single canonical URL (no prose, just the URL):',
      siblings.map((u) => `- ${u}`).join('\n'),
    ].join('\n');
    const reply = (await opts.llm(prompt)).trim();
    if (reply.startsWith('http')) return reply;
    return null;
  } catch (err) {
    log.warn({ event: 'llm_canonical_failed', url: opts.url, err: String(err) }, 'LLM canonical fallback failed');
    return null;
  }
}

// ─── Public entrypoint ──────────────────────────────────────────────────────

/**
 * Resolve a variant URL to its canonical. Returns the input URL unchanged when
 * no rule fires.
 */
export async function inferCanonical(opts: InferCanonicalOpts): Promise<string> {
  // Stage 1: <link rel="canonical">
  const fromLink = readCanonicalLink(opts.html);
  if (fromLink) {
    log.info({ event: 'canonical_from_link', url: opts.url, canonical: fromLink });
    return fromLink;
  }

  // Stage 2: URL-suffix heuristic — only for product-catalog
  if (opts.domain === 'product-catalog') {
    const stripped = stripKnownVariantSuffix(opts.url);
    if (stripped) {
      log.info({ event: 'canonical_from_heuristic', url: opts.url, canonical: stripped });
      return stripped;
    }
  }

  // Stage 3: LLM fallback
  const fromLlm = await llmFallback(opts);
  if (fromLlm) {
    log.info({ event: 'canonical_from_llm', url: opts.url, canonical: fromLlm });
    return fromLlm;
  }

  log.info({ event: 'canonical_unchanged', url: opts.url }, 'no rule fired; returning input URL');
  return opts.url;
}

[ ] Step 5.4: Run tests, expect PASS

npx vitest run lib/factory/__tests__/canonical-inferer.test.ts

Expected: 10 tests pass.

[ ] Step 5.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 5.6: Commit

git add lib/factory/canonical-inferer.ts lib/factory/__tests__/canonical-inferer.test.ts
git commit -m "feat(factory): canonical inferer (canonical-link + heuristic + LLM) for §16o variants"

Task 6: Cluster validation: full regression + smoke run across 9 domains

Files: - (Read-only verification + smoke test for cross-domain coverage) - Create: lib/factory/__tests__/extractor-end-to-end.smoke.test.ts

The smoke test wires generator → sandbox-runner across all 9 content domains using fixture HTML to confirm the pipeline produces a passing record for each. This is a defensive regression net: any future template edit breaks here first.

[ ] Step 6.1: Write the cross-domain smoke test

Create lib/factory/__tests__/extractor-end-to-end.smoke.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { generateExtractor } from '../extractor-generator';
import { runInSandbox } from '../sandbox-runner';
import { getDss, listContentDomains } from '../dss';
import type { StructureAnalysis } from '../structure-analyzer';
import type { ContentDomain } from '../types';

interface Fixture {
  domain: ContentDomain;
  html: string;
  selectors: Record<string, string>;
  llmFillings: Record<string, string>;
  expectsRecord: boolean;
}

const longText = (n: number) => 'word '.repeat(n);

const FIXTURES: Fixture[] = [
  {
    domain: 'marketing',
    html: `<html lang="en"><body><h1>Title</h1><article>${longText(80)}</article></body></html>`,
    selectors: { headline: 'h1', articleBody: 'article' },
    llmFillings: { headlineSelector: 'h1', articleBodySelector: 'article', datePublishedSelector: 'time' },
    expectsRecord: true,
  },
  {
    domain: 'support',
    html: `<html lang="en"><body><h1>Support article</h1><article>${longText(80)}</article></body></html>`,
    selectors: { name: 'h1', articleBody: 'article' },
    llmFillings: { nameSelector: 'h1', articleBodySelector: 'article', acceptedAnswerSelector: '.answer' },
    expectsRecord: true,
  },
  {
    domain: 'education',
    html: `<html lang="en"><body><h1>Course X</h1><div class="d">${longText(40)}</div></body></html>`,
    selectors: { name: 'h1', description: '.d' },
    llmFillings: { nameSelector: 'h1', descriptionSelector: '.d', transcriptSelector: '.t' },
    expectsRecord: true,
  },
  {
    domain: 'technical',
    html: `<html lang="en"><body><h1>API</h1><article>${longText(80)}</article><code class="language-js">x</code></body></html>`,
    selectors: { name: 'h1', articleBody: 'article' },
    llmFillings: { nameSelector: 'h1', articleBodySelector: 'article', programmingLanguageSelector: 'code' },
    expectsRecord: true,
  },
  {
    domain: 'customer-stories',
    html: `<html lang="en"><body><h1>Story</h1><article>${longText(80)}</article><div class="company">Acme</div></body></html>`,
    selectors: { headline: 'h1', articleBody: 'article' },
    llmFillings: {
      headlineSelector: 'h1',
      articleBodySelector: 'article',
      aboutNameSelector: '.company',
      quotesSelector: '.quote',
    },
    expectsRecord: true,
  },
  {
    domain: 'product-catalog',
    html: `<html lang="en"><body><h1>Product</h1><div class="desc">${longText(20)}</div><span class="p">$99</span><img class="i" src="x.jpg" /></body></html>`,
    selectors: { name: 'h1', description: '.desc' },
    llmFillings: {
      nameSelector: 'h1',
      descriptionSelector: '.desc',
      priceSelector: '.p',
      imageSelector: '.i',
      variantSelector: '.v',
    },
    expectsRecord: true,
  },
  {
    domain: 'events',
    html: `<html lang="en"><body><h1>Event</h1><div class="desc">${longText(20)}</div><time datetime="2026-06-01">Jun 1</time></body></html>`,
    selectors: { name: 'h1', description: '.desc', startDate: 'time' },
    llmFillings: {
      nameSelector: 'h1',
      descriptionSelector: '.desc',
      startDateSelector: 'time',
      locationSelector: '.loc',
    },
    expectsRecord: true,
  },
  {
    domain: 'legal',
    html: `<html lang="en"><body><h1>Terms</h1><article>${longText(80)}</article></body></html>`,
    selectors: { name: 'h1', text: 'article' },
    llmFillings: { nameSelector: 'h1', textSelector: 'article' },
    expectsRecord: true,
  },
  {
    domain: 'social',
    html: `<html lang="en"><body><div class="post">${longText(20)}</div></body></html>`,
    selectors: { text: '.post' },
    llmFillings: { textSelector: '.post', transcriptSelector: '.transcript' },
    expectsRecord: true,
  },
];

describe('extractor end-to-end — generator → sandbox per domain', () => {
  it('lists 9 domains in the smoke fixture', () => {
    expect(FIXTURES.map((f) => f.domain).sort()).toEqual([...listContentDomains()].sort());
  });

  it.each(FIXTURES.map((f) => [f.domain, f] as const))(
    'domain %s: generator + sandbox produces a passing record',
    async (_domain, fix) => {
      const llm = vi.fn().mockResolvedValue(JSON.stringify(fix.llmFillings));
      const analysis: StructureAnalysis = {
        schemaOrgJsonLd: null,
        selectors: fix.selectors,
        missing: [],
        confidence: 0.8,
      };

      const gen = await generateExtractor({
        pathGroup: { id: `pg_${fix.domain}`, pattern: '/x/*', sampleUrls: ['https://x.com/x/1'] },
        structureAnalysis: analysis,
        dssEntry: getDss(fix.domain),
        contentDomain: fix.domain,
        llm,
      });

      const result = await runInSandbox(
        gen.recordExtractor,
        { url: 'https://x.com/x/1', html: fix.html, status: 200 },
        fix.domain
      );

      if (fix.expectsRecord) {
        expect(result.errors).toHaveLength(0);
        expect(result.records.length).toBeGreaterThan(0);
        expect(result.records[0]).toMatchObject({ content_domain: fix.domain });
      }
    }
  );
});

[ ] Step 6.2: Run smoke test, expect PASS

npx vitest run lib/factory/__tests__/extractor-end-to-end.smoke.test.ts

Expected: 10 tests pass (1 listing check + 9 per-domain).

[ ] Step 6.3: Run the full project test suite

npx vitest run

Expected: prior factory tests + new spec-05 tests, all GREEN. (Spec 01 baseline + Spec 02 + Spec 03 + Spec 04 tests must still pass.)

[ ] Step 6.4: Run full tsc

npx tsc --noEmit

Expected: clean.

[ ] Step 6.5: Commit smoke test

git add lib/factory/__tests__/extractor-end-to-end.smoke.test.ts
git commit -m "test(factory): cross-domain smoke — generator+sandbox per content domain"

[ ] Step 6.6: Mark Spec 05 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 5 (Extraction + Sandbox + Canonical) | ⏸ | structure-analyzer, extractor-generator, record-extractor-template, sandbox-runner, canonical-inferer |
+ | Cluster 5 (Extraction + Sandbox + Canonical) | ✅ | structure-analyzer, extractor-generator, record-extractor-template, sandbox-runner, canonical-inferer |

Acceptance criteria: Spec 05 done means:

✅ lib/factory/dss.ts exports getRecordSchema(domain) returning the matching zod schema.
✅ lib/factory/structure-analyzer.ts exports analyzeStructure(sample, contentDomain, opts?) → {schemaOrgJsonLd, selectors, missing, confidence}. JSON-LD wins; heuristic next; LLM only for ambiguous fields; CMS pattern cached per cmsHint+domain.
✅ lib/factory/record-extractor-template.ts exports a registry with one template per content domain (9 total). Each template hardcodes is_chunk:false / status:'indexed' (no record_type: reserved for session storage), has ({ $, helpers, url, content, headers }) signature, embeds JSON-LD-fallback logic (DOM-first only for product-catalog per §16o), and has discoverable {{placeholders}}.
✅ lib/factory/extractor-generator.ts exports generateExtractor(opts) → {recordExtractor, crawlerConfig}. Validates output via new Function('return ' + src). Prepends userFeedback to LLM prompt when provided. Falls back to structureAnalysis.selectors when LLM omits a placeholder.
✅ lib/factory/sandbox-runner.ts exports runInSandbox(extractorSrc, sample, contentDomain) → {records, errors}. Validates each record against getRecordSchema(contentDomain). Catches extractor exceptions and syntax errors; never crashes the caller.
✅ lib/factory/canonical-inferer.ts exports inferCanonical({url, html, domain, llm?, siblingUrls?}) and stripKnownVariantSuffix(url). Strategy: <link rel="canonical"> → product-catalog suffix heuristic → LLM fallback → input URL.
✅ All 5 unit-test files pass (~30+ tests across the cluster).
✅ Cross-domain smoke test passes (10 tests covering all 9 domains).
✅ Full project test suite still GREEN.
✅ tsc --noEmit clean.
✅ Per CodingSOPs: every module has logger, every public function has docstring, every fallible call has try/catch, every boundary uses zod.
✅ Per §19f.7 (DSS as data, not code): no if (domain === '...') branching outside the registries (RECORD_EXTRACTOR_TEMPLATES, REQUIRED_FIELDS_BY_DOMAIN, RECORD_SCHEMA_BY_DOMAIN).

Out of scope (handled by other specs)

Real LLM provider wiring → Spec 08 (API endpoints inject the production lib/llm/ provider; this spec mocks it).
Real Algolia round-trip → Spec 13 (smoke cluster).
/api/factory/generate-extractor and /api/factory/test-sandbox HTTP endpoints → Spec 08.
Sample fetching with WAF detection → Spec 03 (sampler.ts + playwright-fetcher.ts).
Content classification (which content domain a pathGroup is) → Spec 04 (classifier.ts).
Crawler-config push to Algolia (pathsToMatch + extractor delivery) → Spec 06 (crawler-client.ts + index-manager.ts).
Canonical-driven dedup in the actual crawler config → Spec 06's create-crawler step calls inferCanonical from this spec.
Furniture/apparel/electronics vertical sub-schemas → v2 (deferred per §16o-furniture). v1 captures whatever generic product-catalog finds; sub-schemas are configuration adds in DSS, no code release.
VM-isolated extractor execution → out of scope. The factory only runs extractors it generated; same-process new Function is acceptable.