Dashboard/Crawler-Factory/Engineering-Specs/03-waf-sampling.md
Crawler-Factory

Engineering-Specs/03-waf-sampling.md

Crawler Factory: Spec 03: WAF Detection + Page Sampling Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Detect when a site WAF-blocks plain fetch from Vercel-class datacenter IPs, escalate to Playwright stealth, and sample HTML pages so downstream classification (Spec 04) and structure analysis (Spec 05) can run. Implements the §19b 5-level WAF response ladder up to Level 3 (escalate, then surface to UI).

Architecture: Three modules. waf-detector.ts performs a fast probe (HEAD/GET on robots.txt) and decides per-host whether transport should be 'fetch' | 'playwright' | 'blocked'. playwright-fetcher.ts lazy-loads a single chromium instance, applies stealth defaults (real Chrome UA, viewport 1920×1080, navigator.webdriver disabled), and reuses the browser across calls. sampler.ts composes the two: for each URL it consults the per-host transport decision (cached), fetches with the chosen transport, applies a 5MB response cap and SPA heuristic, and emits a Sample for downstream specs. NO classification logic here: that belongs to Spec 04. NO DOM structure analysis: that's Spec 05.

Tech Stack: TypeScript (strict), playwright 1.57+ (already a devDependency: promoted to runtime in Task 0), cheerio 1.2 (already a runtime dependency: used by the SPA heuristic to read <noscript> and root markers without parsing the full DOM tree), zod 3.25 for the Sample shape, vitest, pino logger via lib/utils/logger.ts.

Depends on: Spec 01 (logger conventions, factory module folder, zod patterns). Consumed by: Spec 04 (classifier reads Sample[] per pathGroup), Spec 05 (structure analyzer reads sample.html), Spec 02's discover endpoint (calls probeWaf once per session to surface WAF status to the UI).

Plan source: Projects/Crawler-Factory/00-Plan.md §3d (WAF/bot-detection reality), §7 Task 8 (sampler), §14f (60-site WAF rate findings: ~40% block plain fetch), §16c (AEM enterprise WAF check), §16o (DTC retail: 5/8 WAF-block), §19b (5-level WAF response ladder).

File structure

Path Responsibility
package.json Move playwright from devDependencies to dependencies (factory needs it at runtime in Vercel functions)
scripts/bundle-api.mjs Add 'playwright' and 'playwright-core' to externals so esbuild does not try to bundle the chromium binary path
lib/factory/waf-detector.ts probeWaf(url, opts): robots.txt probe → returns transport decision per host; implements §19b ladder L0–L3
lib/factory/playwright-fetcher.ts fetchWithPlaywright(url, opts): singleton chromium instance with stealth defaults; closes on process exit
lib/factory/sampler.ts sampleUrls(urls, opts): parallel fetch with abort controllers, 5MB cap, SPA heuristic, per-host transport cache, individual-403 fallback to Playwright
lib/factory/__tests__/waf-detector.test.ts WAF detector tests: 200 → fetch; 403/429/timeout → playwright; both blocked → blocked
lib/factory/__tests__/playwright-fetcher.test.ts Playwright fetcher tests: stealth defaults applied; browser reused across calls; close on shutdown
lib/factory/__tests__/sampler.test.ts Sampler tests: SPA detection on near-empty <div id="root">; 5MB truncation; 5s timeout; per-URL fallback to Playwright on 403

Why no Sample type in types.ts: Sample only crosses the boundary between sampler.ts and Spec 04/05 consumers in the same lib/factory/ folder. Per the user directive, we keep SampleSchema local to sampler.ts to avoid bloating types.ts. The type is exported and reused, the schema is not promoted.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:waf-detector'); (or appropriate context). No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
  2. Try/catch every fallible call. Every external (fetch, chromium.launch, browser.newContext, page.goto, page.content) wrapped. Catch specific error first, generic last. Always log before rethrow with full context.
  3. Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object.
  4. No any. Use T | null (not T | undefined) where absence is meaningful. zod for boundary types.
  5. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  6. Mock external services in unit tests. vi.mock('playwright') and vi.spyOn(globalThis, 'fetch'). Real Playwright launch lives only in Spec 13 (smoke tests).
  7. Comments explain WHY, not WHAT. e.g. why 5MB cap (Algolia record limit downstream + 99th-percentile HTML payload), why this UA (matches a real Chrome 120 build to dodge UA-only WAF rules), why singleton browser (chromium launch is ~1.2s on Vercel cold start; one launch per session).
  8. Naming. TypeScript: camelCase vars/functions, PascalCase types, UPPER_SNAKE_CASE constants. Files: kebab-case.ts. Tests: <name>.test.ts.
  9. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.

Task 0: Promote playwright to runtime + bundler externals

Files: - Modify: package.json: move playwright from devDependencies to dependencies - Modify: scripts/bundle-api.mjs: add 'playwright' and 'playwright-core' to externals

Why: Spec 03's factory code runs inside Vercel API routes (Spec 02's discover endpoint and Spec 06's sampling endpoints will import from lib/factory/). Vercel functions execute production builds: devDependencies are pruned. Playwright must be a runtime dependency. Adding it to bundler externals prevents esbuild from attempting to inline the chromium binary path resolver, which fails at build time.

  • [ ] Step 0.1: Move playwright to runtime dependencies
npm uninstall playwright && npm install playwright@^1.57.0

Expected: package.json shows playwright under dependencies, removed from devDependencies. package-lock.json updated.

  • [ ] Step 0.2: Verify install
node -e "console.log(require('playwright/package.json').version)"

Expected output: 1.57.x or higher.

  • [ ] Step 0.3: Add Playwright to bundler externals

Open scripts/bundle-api.mjs. Find the externals array (the same one Spec 01 modified). Add 'playwright' and 'playwright-core' alphabetically.

// Locate the existing externals array and add the two playwright entries
const externals = [
  '@google/generative-ai',
  'algoliasearch',
  'cheerio',
  'fast-xml-parser',
  'pino',
  'playwright',          // ← ADD
  'playwright-core',     // ← ADD (transitive — chromium driver loader)
  'uuid',
];
  • [ ] Step 0.4: Verify bundler still runs
node scripts/bundle-api.mjs

Expected: completes without error; api/*.mjs regenerate.

  • [ ] Step 0.5: Commit
git add package.json package-lock.json scripts/bundle-api.mjs
git commit -m "chore(factory): promote playwright to runtime dep + bundler externals"

Task 1: WAF detector: probeWaf(url, opts)

Files: - Create: lib/factory/waf-detector.ts - Create: lib/factory/__tests__/waf-detector.test.ts

The detector implements the §19b ladder, L0–L3:

Level Condition Returned transport
L0 (pass) robots.txt returns 2xx via plain fetch 'fetch'
L1 (jitter) 200 with Retry-After or 429 single hit 'fetch' (sampler applies jitter; not the detector's job)
L2 (escalate) 403 / 429 (persistent) / timeout (>8s) / TCP reset try Playwright probe
L3 (block) Playwright probe also fails (403/timeout) 'blocked'

opts shape: { timeoutMs?: number; userAgent?: string }. Defaults: timeoutMs=8000, userAgent='AlgoliaCentralFactory/1.0 (+https://algoliacentral.com/factory)'.

The probe target is always the hostname's /robots.txt: universal (95%+ adoption per §14f), small payload (rarely >50KB), and sites that block /robots.txt from datacenters always block content too.

  • [ ] Step 1.1: Write failing test for the 200 happy path

Create lib/factory/__tests__/waf-detector.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

// Mock Playwright BEFORE importing the module under test
const mockPlaywrightFetch = vi.fn();
vi.mock('../playwright-fetcher', () => ({
  fetchWithPlaywright: (...args: unknown[]) => mockPlaywrightFetch(...args),
}));

import { probeWaf } from '../waf-detector';

describe('probeWaf — Level 0 (plain fetch works)', () => {
  beforeEach(() => {
    mockPlaywrightFetch.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('returns transport=fetch when robots.txt returns 200', async () => {
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(
      new Response('User-agent: *\nAllow: /', { status: 200 })
    );
    const result = await probeWaf('https://www.algolia.com');
    expect(result.transport).toBe('fetch');
    expect(result.evidence.fetchStatus).toBe(200);
    expect(mockPlaywrightFetch).not.toHaveBeenCalled();
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/waf-detector.test.ts

Expected: FAIL: Cannot find module '../waf-detector'.

  • [ ] Step 1.3: Implement minimal waf-detector.ts for the 200 path

Create lib/factory/waf-detector.ts:

/**
 * WAF detector — decides per-host transport for the sampler.
 *
 * Implements §19b L0–L3 of the WAF response ladder:
 *   L0  fetch works               → 'fetch'
 *   L2  fetch blocked             → try Playwright probe
 *   L3  Playwright also blocked   → 'blocked' (UI must surface user options)
 *
 * Why probe robots.txt: 95%+ universal adoption (§14f). Tiny payload. If a
 * site blocks robots.txt from datacenter IPs it WILL block content fetches
 * too — the empirical floor of WAF posture.
 */

import { createLogger } from '../utils/logger';
import { fetchWithPlaywright } from './playwright-fetcher';

const log = createLogger('factory:waf-detector');

const DEFAULT_TIMEOUT_MS = 8_000;
const DEFAULT_USER_AGENT =
  'AlgoliaCentralFactory/1.0 (+https://algoliacentral.com/factory)';

export type Transport = 'fetch' | 'playwright' | 'blocked';

export interface WafProbeEvidence {
  fetchStatus: number | null;
  fetchError: string | null;
  playwrightStatus: number | null;
  playwrightError: string | null;
  probedUrl: string;
  durationMs: number;
}

export interface WafProbeResult {
  transport: Transport;
  evidence: WafProbeEvidence;
}

export interface ProbeWafOpts {
  timeoutMs?: number;
  userAgent?: string;
}

/**
 * Decide which transport to use for a given site by probing robots.txt.
 * One probe per host is sufficient — the sampler caches the decision.
 */
export async function probeWaf(
  siteUrl: string,
  opts: ProbeWafOpts = {}
): Promise<WafProbeResult> {
  const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
  const userAgent = opts.userAgent ?? DEFAULT_USER_AGENT;
  const probedUrl = buildRobotsUrl(siteUrl);
  const started = Date.now();

  const fetchOutcome = await tryPlainFetch(probedUrl, timeoutMs, userAgent);

  if (fetchOutcome.ok) {
    log.info(
      { event: 'probeWaf_l0', url: probedUrl, status: fetchOutcome.status },
      'plain fetch succeeded — using transport=fetch'
    );
    return {
      transport: 'fetch',
      evidence: {
        fetchStatus: fetchOutcome.status,
        fetchError: null,
        playwrightStatus: null,
        playwrightError: null,
        probedUrl,
        durationMs: Date.now() - started,
      },
    };
  }

  // L2 — escalate to Playwright
  log.warn(
    { event: 'probeWaf_l2_escalate', url: probedUrl, fetchStatus: fetchOutcome.status, err: fetchOutcome.error },
    'plain fetch blocked — escalating to Playwright'
  );

  const pwOutcome = await tryPlaywrightProbe(probedUrl, timeoutMs, userAgent);
  return finalizeProbe(probedUrl, started, fetchOutcome, pwOutcome);
}

function buildRobotsUrl(siteUrl: string): string {
  const u = new URL(siteUrl);
  return `${u.protocol}//${u.host}/robots.txt`;
}

interface FetchOutcome {
  ok: boolean;
  status: number | null;
  error: string | null;
}

async function tryPlainFetch(
  url: string,
  timeoutMs: number,
  userAgent: string
): Promise<FetchOutcome> {
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), timeoutMs);
  try {
    const res = await fetch(url, {
      method: 'GET',
      headers: { 'User-Agent': userAgent, Accept: 'text/plain,*/*' },
      signal: ac.signal,
      redirect: 'follow',
    });
    return { ok: res.ok, status: res.status, error: null };
  } catch (err) {
    return { ok: false, status: null, error: String(err) };
  } finally {
    clearTimeout(timer);
  }
}

interface PlaywrightOutcome {
  ok: boolean;
  status: number | null;
  error: string | null;
}

async function tryPlaywrightProbe(
  url: string,
  timeoutMs: number,
  userAgent: string
): Promise<PlaywrightOutcome> {
  try {
    const result = await fetchWithPlaywright(url, { timeoutMs, userAgent });
    const ok = result.status >= 200 && result.status < 400;
    return { ok, status: result.status, error: null };
  } catch (err) {
    return { ok: false, status: null, error: String(err) };
  }
}

function finalizeProbe(
  probedUrl: string,
  started: number,
  fetchOutcome: FetchOutcome,
  pwOutcome: PlaywrightOutcome
): WafProbeResult {
  const transport: Transport = pwOutcome.ok ? 'playwright' : 'blocked';
  if (transport === 'blocked') {
    log.error(
      { event: 'probeWaf_l3_blocked', url: probedUrl, pwStatus: pwOutcome.status, pwErr: pwOutcome.error },
      'both transports blocked — site uncrawlable from factory infra'
    );
  } else {
    log.info(
      { event: 'probeWaf_l2_pw_ok', url: probedUrl, status: pwOutcome.status },
      'Playwright probe succeeded — using transport=playwright'
    );
  }
  return {
    transport,
    evidence: {
      fetchStatus: fetchOutcome.status,
      fetchError: fetchOutcome.error,
      playwrightStatus: pwOutcome.status,
      playwrightError: pwOutcome.error,
      probedUrl,
      durationMs: Date.now() - started,
    },
  };
}

Note on the Playwright fetcher import: playwright-fetcher.ts doesn't exist yet. The 200 happy-path test mocks '../playwright-fetcher', so this is fine for now. We'll write playwright-fetcher.ts in Task 2 and the dependency will resolve.

  • [ ] Step 1.4: Run test, expect PASS
npx vitest run lib/factory/__tests__/waf-detector.test.ts

Expected: 1 test passes (returns transport=fetch when robots.txt returns 200).

  • [ ] Step 1.5: Add tests for the 403 escalation paths

Append to lib/factory/__tests__/waf-detector.test.ts:

describe('probeWaf — Level 2 (escalate to Playwright)', () => {
  beforeEach(() => {
    mockPlaywrightFetch.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('returns transport=playwright when fetch is 403 but Playwright works', async () => {
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('forbidden', { status: 403 }));
    mockPlaywrightFetch.mockResolvedValue({
      html: 'User-agent: *',
      status: 200,
      headers: {},
      finalUrl: 'https://www.adobe.com/robots.txt',
    });
    const result = await probeWaf('https://www.adobe.com');
    expect(result.transport).toBe('playwright');
    expect(result.evidence.fetchStatus).toBe(403);
    expect(result.evidence.playwrightStatus).toBe(200);
  });

  it('returns transport=playwright when fetch is 429 but Playwright works', async () => {
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('rate limited', { status: 429 }));
    mockPlaywrightFetch.mockResolvedValue({
      html: '...',
      status: 200,
      headers: {},
      finalUrl: 'https://x.com/robots.txt',
    });
    const result = await probeWaf('https://x.com');
    expect(result.transport).toBe('playwright');
  });

  it('returns transport=playwright when fetch times out but Playwright works', async () => {
    vi.spyOn(globalThis, 'fetch').mockImplementation(() => {
      return new Promise((_resolve, reject) => {
        setTimeout(() => reject(new Error('AbortError')), 50);
      });
    });
    mockPlaywrightFetch.mockResolvedValue({
      html: '...',
      status: 200,
      headers: {},
      finalUrl: 'https://x.com/robots.txt',
    });
    const result = await probeWaf('https://x.com', { timeoutMs: 30 });
    expect(result.transport).toBe('playwright');
    expect(result.evidence.fetchError).toBeTruthy();
  });
});

describe('probeWaf — Level 3 (both blocked)', () => {
  beforeEach(() => {
    mockPlaywrightFetch.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('returns transport=blocked when fetch and Playwright both fail', async () => {
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('forbidden', { status: 403 }));
    mockPlaywrightFetch.mockRejectedValue(new Error('navigation timeout'));
    const result = await probeWaf('https://www.nike.com');
    expect(result.transport).toBe('blocked');
    expect(result.evidence.fetchStatus).toBe(403);
    expect(result.evidence.playwrightError).toMatch(/timeout/i);
  });

  it('returns transport=blocked when Playwright returns 403', async () => {
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('forbidden', { status: 403 }));
    mockPlaywrightFetch.mockResolvedValue({
      html: '',
      status: 403,
      headers: {},
      finalUrl: 'https://www.rh.com/robots.txt',
    });
    const result = await probeWaf('https://www.rh.com');
    expect(result.transport).toBe('blocked');
    expect(result.evidence.playwrightStatus).toBe(403);
  });
});

describe('probeWaf — opts', () => {
  afterEach(() => vi.restoreAllMocks());

  it('honors a custom timeoutMs', async () => {
    const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(async (_url, init) => {
      // Verify abort signal was provided
      expect((init as RequestInit).signal).toBeDefined();
      return new Response('', { status: 200 });
    });
    await probeWaf('https://x.com', { timeoutMs: 100 });
    expect(fetchSpy).toHaveBeenCalled();
  });

  it('builds a robots.txt URL from any path-bearing input URL', async () => {
    const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('', { status: 200 }));
    await probeWaf('https://www.example.com/some/deep/page');
    expect(fetchSpy).toHaveBeenCalledWith(
      'https://www.example.com/robots.txt',
      expect.any(Object)
    );
  });
});
  • [ ] Step 1.6: Run all WAF detector tests
npx vitest run lib/factory/__tests__/waf-detector.test.ts

Expected: all tests pass (8 total: 1 L0 + 3 L2 + 2 L3 + 2 opts).

  • [ ] Step 1.7: Run tsc --noEmit
npx tsc --noEmit

Expected: clean. (Note: the import of ./playwright-fetcher will resolve because the mock alias intercepts it at test time, but tsc resolves the real path. If tsc errors with Cannot find module './playwright-fetcher', that's fine: Task 2 creates the file. If it errors and you need a clean tsc gate now, create a stub lib/factory/playwright-fetcher.ts exporting export async function fetchWithPlaywright(_url: string, _opts: { timeoutMs: number; userAgent: string }): Promise<{ html: string; status: number; headers: Record<string, string>; finalUrl: string }> { throw new Error('not implemented'); } and overwrite it in Task 2.)

  • [ ] Step 1.8: Commit
git add lib/factory/waf-detector.ts lib/factory/__tests__/waf-detector.test.ts
git commit -m "feat(factory): WAF detector with §19b ladder L0–L3 (fetch/playwright/blocked)"

Task 2: Playwright fetcher: singleton chromium with stealth defaults

Files: - Create (or replace stub from Task 1): lib/factory/playwright-fetcher.ts - Create: lib/factory/__tests__/playwright-fetcher.test.ts

fetchWithPlaywright(url, opts) returns { html, status, headers, finalUrl }. Stealth defaults applied to every context: - User agent: a real Chrome 120 build string (the AlgoliaCentralFactory/1.0 UA is ONLY for the L0 plain-fetch probe: once we're on Playwright we want to look like a normal user) - Accept-Language: en-US,en;q=0.9 - Viewport: { width: 1920, height: 1080 } (typical desktop, not the Playwright default 1280×720 which is a fingerprint giveaway) - navigator.webdriver overridden to undefined via init script - One chromium browser per process. Lazily launched on first call. process.on('exit') closes it.

A new BrowserContext per call (NOT a new browser: context creation is ~50ms vs browser launch ~1.2s) gives per-call isolation.

  • [ ] Step 2.1: Write failing test for the basic fetch shape

Create lib/factory/__tests__/playwright-fetcher.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

// Build mock harness for chromium
const mockPage = {
  goto: vi.fn(),
  content: vi.fn(),
  url: vi.fn(),
  close: vi.fn(),
  addInitScript: vi.fn(),
};

const mockContext = {
  newPage: vi.fn(() => Promise.resolve(mockPage)),
  close: vi.fn(),
};

const mockBrowser = {
  newContext: vi.fn(() => Promise.resolve(mockContext)),
  close: vi.fn(),
  isConnected: vi.fn(() => true),
};

const mockChromiumLaunch = vi.fn(() => Promise.resolve(mockBrowser));

vi.mock('playwright', () => ({
  chromium: { launch: mockChromiumLaunch },
}));

// Reset module state between tests so the singleton browser re-launches
beforeEach(async () => {
  vi.resetModules();
  mockChromiumLaunch.mockClear();
  mockBrowser.newContext.mockClear();
  mockContext.newPage.mockClear();
  mockPage.goto.mockReset();
  mockPage.content.mockReset();
  mockPage.url.mockReset();
  mockPage.close.mockReset();
  mockPage.addInitScript.mockReset();
  mockContext.close.mockReset();
});

describe('fetchWithPlaywright — basic shape', () => {
  it('returns html, status, headers, finalUrl', async () => {
    mockPage.goto.mockResolvedValue({
      status: () => 200,
      headers: () => ({ 'content-type': 'text/html' }),
    });
    mockPage.content.mockResolvedValue('<html><body>hi</body></html>');
    mockPage.url.mockReturnValue('https://x.com/robots.txt');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    const result = await fetchWithPlaywright('https://x.com/robots.txt', {
      timeoutMs: 5000,
      userAgent: 'test-ua',
    });

    expect(result).toEqual({
      html: '<html><body>hi</body></html>',
      status: 200,
      headers: { 'content-type': 'text/html' },
      finalUrl: 'https://x.com/robots.txt',
    });
  });
});
  • [ ] Step 2.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/playwright-fetcher.test.ts

Expected: FAIL: module missing or stub throw new Error('not implemented') from Task 1.

  • [ ] Step 2.3: Implement playwright-fetcher.ts

Create (or overwrite) lib/factory/playwright-fetcher.ts:

/**
 * Playwright fetcher — single chromium instance per process, reused
 * across calls. Stealth defaults dodge UA-only and webdriver-flag WAF rules.
 *
 * Why singleton: chromium.launch() is ~1.2s on Vercel cold start. Per-call
 * BrowserContext (~50ms) gives isolation without paying launch cost again.
 *
 * Why this UA: matches a current Chrome 120 build on macOS. The factory's own
 * `AlgoliaCentralFactory/1.0` UA is only for the L0 plain-fetch probe — once
 * we're on Playwright we present as a normal user. Per §3d.
 */

import { chromium, type Browser, type BrowserContext, type Page } from 'playwright';
import { createLogger } from '../utils/logger';

const log = createLogger('factory:playwright-fetcher');

// Real Chrome 120 build (refresh quarterly — cheap maintenance to stay current)
const STEALTH_USER_AGENT =
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36';

const STEALTH_VIEWPORT = { width: 1920, height: 1080 } as const;

let browserSingleton: Browser | null = null;
let shutdownHandlerRegistered = false;

export interface PlaywrightFetchOpts {
  timeoutMs: number;
  userAgent?: string;
}

export interface PlaywrightFetchResult {
  html: string;
  status: number;
  headers: Record<string, string>;
  finalUrl: string;
}

/**
 * Fetch a URL via the singleton chromium instance with stealth defaults.
 * Caller is responsible for catching errors — we throw on launch / nav failures.
 */
export async function fetchWithPlaywright(
  url: string,
  opts: PlaywrightFetchOpts
): Promise<PlaywrightFetchResult> {
  const browser = await getBrowser();
  const context = await openContext(browser, opts.userAgent ?? STEALTH_USER_AGENT);
  const page = await context.newPage();
  try {
    return await navigateAndCapture(page, url, opts.timeoutMs);
  } finally {
    await safeClose(page, context);
  }
}

async function getBrowser(): Promise<Browser> {
  if (browserSingleton && browserSingleton.isConnected()) return browserSingleton;
  try {
    browserSingleton = await chromium.launch({
      headless: true,
      // --disable-blink-features=AutomationControlled drops the navigator.webdriver tell.
      args: ['--disable-blink-features=AutomationControlled'],
    });
    log.info({ event: 'pw_browser_launched' }, 'chromium launched');
    registerShutdownHandler();
    return browserSingleton;
  } catch (err) {
    log.error({ event: 'pw_launch_failed', err: String(err) }, 'chromium.launch failed');
    throw err;
  }
}

async function openContext(browser: Browser, userAgent: string): Promise<BrowserContext> {
  try {
    const context = await browser.newContext({
      userAgent,
      viewport: { ...STEALTH_VIEWPORT },
      locale: 'en-US',
      extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },
    });
    // Belt-and-suspenders: even with the launch arg, some pages probe
    // navigator.webdriver from page scripts. This init script removes it.
    await context.addInitScript(() => {
      Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
    });
    return context;
  } catch (err) {
    log.error({ event: 'pw_context_failed', err: String(err) }, 'newContext failed');
    throw err;
  }
}

async function navigateAndCapture(
  page: Page,
  url: string,
  timeoutMs: number
): Promise<PlaywrightFetchResult> {
  try {
    const response = await page.goto(url, { timeout: timeoutMs, waitUntil: 'domcontentloaded' });
    if (!response) {
      throw new Error('page.goto returned null response');
    }
    const html = await page.content();
    return {
      html,
      status: response.status(),
      headers: response.headers(),
      finalUrl: page.url(),
    };
  } catch (err) {
    log.warn({ event: 'pw_goto_failed', url, err: String(err) }, 'page.goto failed');
    throw err;
  }
}

async function safeClose(page: Page, context: BrowserContext): Promise<void> {
  try {
    await page.close();
  } catch (err) {
    log.warn({ event: 'pw_page_close_failed', err: String(err) }, 'page.close failed (continuing)');
  }
  try {
    await context.close();
  } catch (err) {
    log.warn({ event: 'pw_ctx_close_failed', err: String(err) }, 'context.close failed (continuing)');
  }
}

function registerShutdownHandler(): void {
  if (shutdownHandlerRegistered) return;
  shutdownHandlerRegistered = true;
  const close = async (): Promise<void> => {
    if (browserSingleton) {
      try {
        await browserSingleton.close();
        log.info({ event: 'pw_browser_closed' }, 'chromium closed on shutdown');
      } catch (err) {
        log.warn({ event: 'pw_close_failed', err: String(err) }, 'chromium close failed');
      }
      browserSingleton = null;
    }
  };
  process.on('exit', () => void close());
  process.on('SIGINT', () => void close());
  process.on('SIGTERM', () => void close());
}

/**
 * Test-only export — force-close the singleton browser. Used by tests
 * to reset state between cases. Do NOT call from production code paths.
 */
export async function __resetPlaywrightForTests(): Promise<void> {
  if (browserSingleton) {
    try {
      await browserSingleton.close();
    } catch {
      /* ignore */
    }
    browserSingleton = null;
  }
}
  • [ ] Step 2.4: Run the basic shape test, expect PASS
npx vitest run lib/factory/__tests__/playwright-fetcher.test.ts -t "basic shape"

Expected: 1 test passes.

  • [ ] Step 2.5: Add tests for stealth defaults + browser reuse + close-on-exit

Append to lib/factory/__tests__/playwright-fetcher.test.ts:

describe('fetchWithPlaywright — stealth defaults', () => {
  it('applies real Chrome UA, en-US locale, 1920x1080 viewport, AcceptLanguage header', async () => {
    mockPage.goto.mockResolvedValue({ status: () => 200, headers: () => ({}) });
    mockPage.content.mockResolvedValue('<html></html>');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 });

    expect(mockBrowser.newContext).toHaveBeenCalledWith(
      expect.objectContaining({
        userAgent: expect.stringMatching(/Chrome\/\d+/),
        viewport: { width: 1920, height: 1080 },
        locale: 'en-US',
        extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },
      })
    );
    // The init script that removes navigator.webdriver
    expect(mockPage.addInitScript).not.toHaveBeenCalled(); // it's on context, not page
  });

  it('honors a custom userAgent override (used by waf-detector probe)', async () => {
    mockPage.goto.mockResolvedValue({ status: () => 200, headers: () => ({}) });
    mockPage.content.mockResolvedValue('<html></html>');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await fetchWithPlaywright('https://x.com/', {
      timeoutMs: 5000,
      userAgent: 'custom-ua/9.9',
    });

    expect(mockBrowser.newContext).toHaveBeenCalledWith(
      expect.objectContaining({ userAgent: 'custom-ua/9.9' })
    );
  });

  it('launches chromium with --disable-blink-features=AutomationControlled', async () => {
    mockPage.goto.mockResolvedValue({ status: () => 200, headers: () => ({}) });
    mockPage.content.mockResolvedValue('<html></html>');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 });

    expect(mockChromiumLaunch).toHaveBeenCalledWith(
      expect.objectContaining({
        headless: true,
        args: expect.arrayContaining(['--disable-blink-features=AutomationControlled']),
      })
    );
  });
});

describe('fetchWithPlaywright — singleton reuse', () => {
  it('launches the browser once across multiple calls', async () => {
    mockPage.goto.mockResolvedValue({ status: () => 200, headers: () => ({}) });
    mockPage.content.mockResolvedValue('<html></html>');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await fetchWithPlaywright('https://x.com/a', { timeoutMs: 5000 });
    await fetchWithPlaywright('https://x.com/b', { timeoutMs: 5000 });
    await fetchWithPlaywright('https://x.com/c', { timeoutMs: 5000 });

    expect(mockChromiumLaunch).toHaveBeenCalledTimes(1);
    expect(mockBrowser.newContext).toHaveBeenCalledTimes(3);
  });

  it('closes context and page after every call (cleanup)', async () => {
    mockPage.goto.mockResolvedValue({ status: () => 200, headers: () => ({}) });
    mockPage.content.mockResolvedValue('<html></html>');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 });

    expect(mockPage.close).toHaveBeenCalledTimes(1);
    expect(mockContext.close).toHaveBeenCalledTimes(1);
  });

  it('still closes context and page when goto throws', async () => {
    mockPage.goto.mockRejectedValue(new Error('navigation timeout'));
    mockPage.content.mockResolvedValue('');
    mockPage.url.mockReturnValue('https://x.com/');

    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await expect(
      fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 })
    ).rejects.toThrow(/navigation timeout/);

    expect(mockPage.close).toHaveBeenCalledTimes(1);
    expect(mockContext.close).toHaveBeenCalledTimes(1);
  });
});

describe('fetchWithPlaywright — failure surfaces', () => {
  it('throws when chromium.launch fails', async () => {
    mockChromiumLaunch.mockRejectedValueOnce(new Error('chromium binary missing'));
    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await expect(
      fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 })
    ).rejects.toThrow(/chromium binary missing/);
  });

  it('throws when goto returns null', async () => {
    mockPage.goto.mockResolvedValue(null);
    const { fetchWithPlaywright } = await import('../playwright-fetcher');
    await expect(
      fetchWithPlaywright('https://x.com/', { timeoutMs: 5000 })
    ).rejects.toThrow(/null response/);
  });
});
  • [ ] Step 2.6: Run all Playwright fetcher tests
npx vitest run lib/factory/__tests__/playwright-fetcher.test.ts

Expected: all 9+ tests pass.

  • [ ] Step 2.7: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 2.8: Commit
git add lib/factory/playwright-fetcher.ts lib/factory/__tests__/playwright-fetcher.test.ts
git commit -m "feat(factory): singleton Playwright fetcher with stealth defaults"

Task 3: Sampler: sampleUrls(urls, opts)

Files: - Create: lib/factory/sampler.ts - Create: lib/factory/__tests__/sampler.test.ts

sampleUrls(urls, opts) returns Sample[]:

interface Sample {
  url: string;
  html: string;
  status: number;
  contentLength: number;
  isSpa: boolean;
  headers: Record<string, string>;
  transport: 'fetch' | 'playwright';
  truncated: boolean;
}

Behavior: 1. Group input URLs by host. For each host, call probeWaf(host) ONCE and cache the decision (Map<host, Transport>). Hosts with transport='blocked' produce empty Sample[] rows for their URLs (caller decides how to surface this; sampler logs and returns empty). 2. Per-URL fetch with the cached transport. 5s timeout per URL via AbortController. Parallelism cap = 5 concurrent (avoid hammering small sites; matches §19b L0 sequential-8-RPS spirit). 3. Response cap = 5MB. If Content-Length header > 5MB OR streamed body > 5MB, truncate at the 5MB boundary, set truncated: true. Why 5MB: Algolia record limit is 100KB and we down-shard before indexing, but holding >5MB of HTML in a Vercel function would blow the 1024MB memory floor for moderate parallelism. 99th-percentile real HTML is ~2MB. 4. Per-URL fallback: if a URL configured for 'fetch' returns 403/429, retry that single URL via Playwright once (don't change the host's cached decision: could be a per-URL block). 5. SPA heuristic: response body < 50KB AND contains <noscript> AND has <div id="root"> or <div id="app"> whose inner text is empty/whitespace. We use cheerio to parse: cheap, already a runtime dep.

opts shape: { perUrlTimeoutMs?: number; maxBytes?: number; concurrency?: number; userAgent?: string }. Defaults: 5000, 5_242_880 (5MB), 5, 'AlgoliaCentralFactory/1.0'.

  • [ ] Step 3.1: Write failing test for the basic happy path

Create lib/factory/__tests__/sampler.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

const mockProbeWaf = vi.fn();
const mockFetchWithPlaywright = vi.fn();

vi.mock('../waf-detector', () => ({
  probeWaf: (...args: unknown[]) => mockProbeWaf(...args),
}));

vi.mock('../playwright-fetcher', () => ({
  fetchWithPlaywright: (...args: unknown[]) => mockFetchWithPlaywright(...args),
}));

import { sampleUrls } from '../sampler';

describe('sampleUrls — basic happy path', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
    mockFetchWithPlaywright.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('uses transport=fetch when probeWaf says fetch', async () => {
    mockProbeWaf.mockResolvedValue({
      transport: 'fetch',
      evidence: { fetchStatus: 200 },
    });
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(
      new Response('<html><body><h1>hi</h1></body></html>', {
        status: 200,
        headers: { 'content-type': 'text/html' },
      })
    );

    const samples = await sampleUrls(['https://x.com/page1', 'https://x.com/page2']);
    expect(samples).toHaveLength(2);
    expect(samples[0].transport).toBe('fetch');
    expect(samples[0].status).toBe(200);
    expect(samples[0].html).toContain('<h1>hi</h1>');
    expect(samples[0].isSpa).toBe(false);
    expect(samples[0].truncated).toBe(false);
    // One probe per host
    expect(mockProbeWaf).toHaveBeenCalledTimes(1);
  });
});
  • [ ] Step 3.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/sampler.test.ts

Expected: FAIL: module missing.

  • [ ] Step 3.3: Implement minimal sampler.ts

Create lib/factory/sampler.ts:

/**
 * Sampler — fetches HTML pages for downstream classification + structure analysis.
 *
 * Composes waf-detector (per-host transport decision, cached) with playwright-fetcher
 * (escalation path). Implements §3d, §7 Task 8, §19b L0–L3.
 *
 * Out of scope: classification (Spec 04), DOM structure analysis (Spec 05).
 */

import { z } from 'zod';
import * as cheerio from 'cheerio';
import { createLogger } from '../utils/logger';
import { probeWaf, type Transport } from './waf-detector';
import { fetchWithPlaywright } from './playwright-fetcher';

const log = createLogger('factory:sampler');

const DEFAULT_PER_URL_TIMEOUT_MS = 5_000;
const DEFAULT_MAX_BYTES = 5 * 1024 * 1024; // 5MB — see header comment
const DEFAULT_CONCURRENCY = 5;
const DEFAULT_USER_AGENT = 'AlgoliaCentralFactory/1.0';

const SPA_BODY_THRESHOLD_BYTES = 50 * 1024; // <50KB body + empty root → SPA

export const SampleSchema = z.object({
  url: z.string().url(),
  html: z.string(),
  status: z.number().int(),
  contentLength: z.number().int().nonnegative(),
  isSpa: z.boolean(),
  headers: z.record(z.string()),
  transport: z.enum(['fetch', 'playwright']),
  truncated: z.boolean(),
});
export type Sample = z.infer<typeof SampleSchema>;

export interface SampleUrlsOpts {
  perUrlTimeoutMs?: number;
  maxBytes?: number;
  concurrency?: number;
  userAgent?: string;
}

interface ResolvedOpts {
  perUrlTimeoutMs: number;
  maxBytes: number;
  concurrency: number;
  userAgent: string;
}

/**
 * Fetch HTML for each URL using the right transport per host.
 * Returns one Sample per URL whose host is reachable; URLs on hosts
 * marked 'blocked' are dropped and logged.
 */
export async function sampleUrls(
  urls: string[],
  opts: SampleUrlsOpts = {}
): Promise<Sample[]> {
  const resolved: ResolvedOpts = {
    perUrlTimeoutMs: opts.perUrlTimeoutMs ?? DEFAULT_PER_URL_TIMEOUT_MS,
    maxBytes: opts.maxBytes ?? DEFAULT_MAX_BYTES,
    concurrency: opts.concurrency ?? DEFAULT_CONCURRENCY,
    userAgent: opts.userAgent ?? DEFAULT_USER_AGENT,
  };
  const transportByHost = await resolveTransports(urls, resolved);
  return runWithConcurrency(urls, resolved.concurrency, (url) =>
    sampleOne(url, transportByHost, resolved)
  );
}

async function resolveTransports(
  urls: string[],
  opts: ResolvedOpts
): Promise<Map<string, Transport>> {
  const hosts = new Set<string>();
  for (const u of urls) hosts.add(new URL(u).host);
  const decisions = new Map<string, Transport>();
  for (const host of hosts) {
    const probe = await probeWaf(`https://${host}/`, {
      timeoutMs: opts.perUrlTimeoutMs * 2, // probe budget can be a bit higher
      userAgent: opts.userAgent,
    });
    decisions.set(host, probe.transport);
    if (probe.transport === 'blocked') {
      log.error({ event: 'sampler_host_blocked', host }, 'host fully blocked — URLs will be dropped');
    }
  }
  return decisions;
}

async function sampleOne(
  url: string,
  transportByHost: Map<string, Transport>,
  opts: ResolvedOpts
): Promise<Sample | null> {
  const host = new URL(url).host;
  const transport = transportByHost.get(host);
  if (!transport || transport === 'blocked') return null;

  if (transport === 'fetch') {
    const result = await tryFetchSample(url, opts);
    if (result.shouldFallback) {
      log.warn({ event: 'sampler_per_url_fallback', url, status: result.status }, 'per-URL fallback to playwright');
      return await tryPlaywrightSample(url, opts);
    }
    return result.sample;
  }

  return await tryPlaywrightSample(url, opts);
}

interface FetchSampleOutcome {
  sample: Sample | null;
  shouldFallback: boolean;
  status: number | null;
}

async function tryFetchSample(url: string, opts: ResolvedOpts): Promise<FetchSampleOutcome> {
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), opts.perUrlTimeoutMs);
  try {
    const res = await fetch(url, {
      method: 'GET',
      headers: { 'User-Agent': opts.userAgent, Accept: 'text/html,*/*' },
      signal: ac.signal,
      redirect: 'follow',
    });
    if (res.status === 403 || res.status === 429) {
      return { sample: null, shouldFallback: true, status: res.status };
    }
    const { html, truncated, contentLength } = await readCappedBody(res, opts.maxBytes);
    const headers = headersToRecord(res.headers);
    return {
      sample: buildSample(url, html, res.status, contentLength, headers, 'fetch', truncated),
      shouldFallback: false,
      status: res.status,
    };
  } catch (err) {
    log.warn({ event: 'sampler_fetch_failed', url, err: String(err) }, 'fetch failed; falling back to playwright');
    return { sample: null, shouldFallback: true, status: null };
  } finally {
    clearTimeout(timer);
  }
}

async function tryPlaywrightSample(url: string, opts: ResolvedOpts): Promise<Sample | null> {
  try {
    const r = await fetchWithPlaywright(url, {
      timeoutMs: opts.perUrlTimeoutMs,
      userAgent: opts.userAgent,
    });
    const { html, truncated, contentLength } = applyByteCap(r.html, opts.maxBytes);
    return buildSample(url, html, r.status, contentLength, r.headers, 'playwright', truncated);
  } catch (err) {
    log.warn({ event: 'sampler_pw_failed', url, err: String(err) }, 'playwright fetch failed');
    return null;
  }
}

interface CappedBody {
  html: string;
  truncated: boolean;
  contentLength: number;
}

async function readCappedBody(res: Response, maxBytes: number): Promise<CappedBody> {
  // Short-circuit on Content-Length header to avoid streaming huge bodies at all
  const cl = Number(res.headers.get('content-length') ?? '0');
  if (cl > maxBytes) {
    const text = (await res.text()).slice(0, maxBytes);
    return { html: text, truncated: true, contentLength: cl };
  }
  const text = await res.text();
  return applyByteCap(text, maxBytes);
}

function applyByteCap(text: string, maxBytes: number): CappedBody {
  // JS strings are UTF-16; use a Buffer-ish length for byte budget
  const byteLen = Buffer.byteLength(text, 'utf8');
  if (byteLen <= maxBytes) {
    return { html: text, truncated: false, contentLength: byteLen };
  }
  // Truncate at the maxBytes boundary (UTF-8 may end mid-char; acceptable)
  const buf = Buffer.from(text, 'utf8').subarray(0, maxBytes);
  return { html: buf.toString('utf8'), truncated: true, contentLength: byteLen };
}

function headersToRecord(headers: Headers): Record<string, string> {
  const out: Record<string, string> = {};
  headers.forEach((v, k) => {
    out[k] = v;
  });
  return out;
}

function buildSample(
  url: string,
  html: string,
  status: number,
  contentLength: number,
  headers: Record<string, string>,
  transport: Transport,
  truncated: boolean
): Sample {
  // Type narrowing: 'blocked' never reaches buildSample (filtered upstream)
  const t = transport === 'blocked' ? 'fetch' : transport;
  return {
    url,
    html,
    status,
    contentLength,
    isSpa: detectSpa(html, contentLength),
    headers,
    transport: t,
    truncated,
  };
}

/**
 * SPA heuristic: tiny body + <noscript> + empty root container.
 * Why these three: real CSR React/Vue apps emit ~10–40KB shells with a single
 * empty root div. Pre-rendered SSR pages are 100KB+ with content baked in.
 * <noscript> is a tell — frameworks generate one as a "JS required" fallback.
 */
function detectSpa(html: string, contentLength: number): boolean {
  if (contentLength >= SPA_BODY_THRESHOLD_BYTES) return false;
  if (!html.includes('<noscript')) return false;
  const $ = cheerio.load(html);
  const candidates = ['#root', '#app'];
  for (const sel of candidates) {
    const el = $(sel).first();
    if (el.length === 0) continue;
    const inner = el.text().trim();
    if (inner.length === 0) return true;
  }
  return false;
}

async function runWithConcurrency<T>(
  items: string[],
  concurrency: number,
  worker: (item: string) => Promise<T | null>
): Promise<T[]> {
  const out: T[] = [];
  let i = 0;
  const runners = Array.from({ length: Math.min(concurrency, items.length) }, async () => {
    while (i < items.length) {
      const idx = i++;
      const r = await worker(items[idx]);
      if (r !== null) out.push(r);
    }
  });
  await Promise.all(runners);
  return out;
}
  • [ ] Step 3.4: Run the basic happy path test, expect PASS
npx vitest run lib/factory/__tests__/sampler.test.ts -t "basic happy path"

Expected: 1 test passes.

  • [ ] Step 3.5: Add SPA detection tests

Append to lib/factory/__tests__/sampler.test.ts:

describe('sampleUrls — SPA detection', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
    mockFetchWithPlaywright.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('flags isSpa=true for <50KB body with <noscript> and empty <div id="root">', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const spaShell = `<!DOCTYPE html>
      <html><head><title>App</title></head>
      <body>
        <noscript>You need to enable JavaScript to run this app.</noscript>
        <div id="root"></div>
        <script src="/main.js"></script>
      </body></html>`;
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(
      new Response(spaShell, { status: 200 })
    );

    const [sample] = await sampleUrls(['https://spa.example.com/']);
    expect(sample.isSpa).toBe(true);
  });

  it('flags isSpa=true for empty <div id="app"> too', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const vueShell = `<html><body><noscript>JS required</noscript><div id="app"></div></body></html>`;
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(vueShell, { status: 200 }));

    const [sample] = await sampleUrls(['https://vue.example.com/']);
    expect(sample.isSpa).toBe(true);
  });

  it('flags isSpa=false when <div id="root"> has content (SSR-rendered)', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const ssrPage = `<html><body><noscript>JS required</noscript>
      <div id="root"><h1>Real content</h1><p>Lots of text here, pre-rendered.</p></div>
      </body></html>`;
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(ssrPage, { status: 200 }));

    const [sample] = await sampleUrls(['https://ssr.example.com/']);
    expect(sample.isSpa).toBe(false);
  });

  it('flags isSpa=false when body is large (>=50KB), even with empty root', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const padding = 'x'.repeat(60_000);
    const heavy = `<html><body><noscript>JS</noscript><div id="root"></div>${padding}</body></html>`;
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(heavy, { status: 200 }));

    const [sample] = await sampleUrls(['https://heavy.example.com/']);
    expect(sample.isSpa).toBe(false);
  });

  it('flags isSpa=false when no <noscript> tag is present', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const minimal = `<html><body><div id="root"></div></body></html>`;
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(minimal, { status: 200 }));

    const [sample] = await sampleUrls(['https://min.example.com/']);
    expect(sample.isSpa).toBe(false);
  });
});
  • [ ] Step 3.6: Run SPA tests, expect PASS
npx vitest run lib/factory/__tests__/sampler.test.ts -t "SPA detection"

Expected: 5 tests pass.

  • [ ] Step 3.7: Add 5MB truncation tests

Append to lib/factory/__tests__/sampler.test.ts:

describe('sampleUrls — 5MB truncation', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('truncates response body and sets truncated=true when body exceeds maxBytes', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    // Build an 8MB body — well over the 5MB default cap
    const huge = 'a'.repeat(8 * 1024 * 1024);
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(huge, { status: 200 }));

    const [sample] = await sampleUrls(['https://big.example.com/']);
    expect(sample.truncated).toBe(true);
    // html should be capped at 5MB
    expect(Buffer.byteLength(sample.html, 'utf8')).toBeLessThanOrEqual(5 * 1024 * 1024);
    // contentLength reflects the original size
    expect(sample.contentLength).toBeGreaterThanOrEqual(5 * 1024 * 1024);
  });

  it('does not truncate when body is under maxBytes', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const small = '<html><body>tiny</body></html>';
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(small, { status: 200 }));

    const [sample] = await sampleUrls(['https://small.example.com/']);
    expect(sample.truncated).toBe(false);
  });

  it('honors a custom maxBytes opt', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    const body = 'a'.repeat(2000);
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(body, { status: 200 }));

    const [sample] = await sampleUrls(['https://x.com/'], { maxBytes: 500 });
    expect(sample.truncated).toBe(true);
    expect(Buffer.byteLength(sample.html, 'utf8')).toBeLessThanOrEqual(500);
  });
});
  • [ ] Step 3.8: Run truncation tests, expect PASS
npx vitest run lib/factory/__tests__/sampler.test.ts -t "5MB truncation"

Expected: 3 tests pass.

  • [ ] Step 3.9: Add per-URL Playwright fallback + timeout + blocked-host tests

Append to lib/factory/__tests__/sampler.test.ts:

describe('sampleUrls — per-URL fallback', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
    mockFetchWithPlaywright.mockReset();
  });

  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('falls back to Playwright when an individual URL returns 403, even though host is fetch-OK', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('forbidden', { status: 403 }));
    mockFetchWithPlaywright.mockResolvedValue({
      html: '<html><body>recovered</body></html>',
      status: 200,
      headers: {},
      finalUrl: 'https://x.com/blocked-page',
    });

    const [sample] = await sampleUrls(['https://x.com/blocked-page']);
    expect(sample.transport).toBe('playwright');
    expect(sample.html).toContain('recovered');
    expect(mockFetchWithPlaywright).toHaveBeenCalledTimes(1);
  });

  it('returns no sample when both fetch and Playwright fail for a URL', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('forbidden', { status: 403 }));
    mockFetchWithPlaywright.mockRejectedValue(new Error('navigation timeout'));

    const samples = await sampleUrls(['https://x.com/dead']);
    expect(samples).toHaveLength(0);
  });
});

describe('sampleUrls — host blocked', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
  });

  afterEach(() => vi.restoreAllMocks());

  it('drops all URLs from a host that probeWaf marks as blocked', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'blocked', evidence: {} });
    const fetchSpy = vi.spyOn(globalThis, 'fetch');

    const samples = await sampleUrls(['https://blocked.example.com/a', 'https://blocked.example.com/b']);
    expect(samples).toHaveLength(0);
    expect(fetchSpy).not.toHaveBeenCalled();
  });
});

describe('sampleUrls — timeout', () => {
  beforeEach(() => mockProbeWaf.mockReset());

  afterEach(() => vi.restoreAllMocks());

  it('aborts a URL fetch that exceeds perUrlTimeoutMs and falls back to Playwright', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    vi.spyOn(globalThis, 'fetch').mockImplementation(
      (_url, init) =>
        new Promise((_resolve, reject) => {
          const signal = (init as RequestInit).signal as AbortSignal | undefined;
          signal?.addEventListener('abort', () => reject(new DOMException('aborted', 'AbortError')));
        })
    );
    mockFetchWithPlaywright.mockResolvedValue({
      html: '<html>recovered via playwright</html>',
      status: 200,
      headers: {},
      finalUrl: 'https://slow.example.com/',
    });

    const [sample] = await sampleUrls(['https://slow.example.com/'], { perUrlTimeoutMs: 30 });
    expect(sample.transport).toBe('playwright');
  });
});

describe('sampleUrls — host probe is per-host', () => {
  beforeEach(() => {
    mockProbeWaf.mockReset();
  });

  afterEach(() => vi.restoreAllMocks());

  it('calls probeWaf once per unique host, not per URL', async () => {
    mockProbeWaf.mockResolvedValue({ transport: 'fetch', evidence: {} });
    vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('<html></html>', { status: 200 }));

    await sampleUrls([
      'https://a.example.com/1',
      'https://a.example.com/2',
      'https://a.example.com/3',
      'https://b.example.com/1',
    ]);
    expect(mockProbeWaf).toHaveBeenCalledTimes(2); // once per host
  });
});
  • [ ] Step 3.10: Run all sampler tests
npx vitest run lib/factory/__tests__/sampler.test.ts

Expected: all tests pass (~14 total: 1 happy + 5 SPA + 3 truncation + 2 fallback + 1 blocked + 1 timeout + 1 per-host).

  • [ ] Step 3.11: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 3.12: Commit
git add lib/factory/sampler.ts lib/factory/__tests__/sampler.test.ts
git commit -m "feat(factory): URL sampler with WAF-aware transport + SPA detection + 5MB cap"

Task 4: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 4.1: Run the full project test suite
npx vitest run

Expected: Spec 01 baseline + Spec 02 baseline + ~31 new Spec 03 tests, all GREEN.

  • [ ] Step 4.2: Run full tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 4.3: Verify bundler still produces working bundles
node scripts/bundle-api.mjs

Expected: completes without error; api/*.mjs regenerate. (The factory modules are not yet imported by any API route: that wiring happens in Spec 02's discover endpoint and Spec 06's sampling endpoints.)

  • [ ] Step 4.4: Mark Spec 03 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 3 (WAF + Sampling) | ⏸ | waf-detector, playwright-fetcher, sampler |
+ | Cluster 3 (WAF + Sampling) | ✅ | waf-detector, playwright-fetcher, sampler |

Acceptance criteria: Spec 03 done means:

  1. ✅ playwright is a runtime dependency (not just devDependency) and externalized in bundler
  2. ✅ lib/factory/waf-detector.ts exports probeWaf(url, opts) returning {transport, evidence} per the §19b L0–L3 ladder
  3. ✅ lib/factory/playwright-fetcher.ts exports fetchWithPlaywright(url, opts) returning {html, status, headers, finalUrl}, with one chromium browser per process, contexts per call, real Chrome 120 UA + 1920×1080 viewport + en-US locale + --disable-blink-features=AutomationControlled + navigator.webdriver removed by init script, and shutdown hook closing the browser on exit/SIGINT/SIGTERM
  4. ✅ lib/factory/sampler.ts exports sampleUrls(urls, opts) returning Sample[] with per-host transport caching, 5s per-URL timeout, 5MB cap with truncated flag, SPA heuristic (<noscript> + empty #root/#app + body <50KB), and per-URL Playwright fallback on 403/429
  5. ✅ Sample type exported from sampler.ts (NOT promoted to types.ts: local to the module)
  6. ✅ All new tests pass (~31 tests total: 8 WAF + 9 fetcher + 14 sampler)
  7. ✅ Full project test suite still GREEN
  8. ✅ tsc --noEmit clean
  9. ✅ Per CodingSOPs: every module has logger, every public function has docstring, every fallible call has try/catch, every function ≤20 lines + ≤3 params, no any, comments explain WHY (5MB cap, singleton browser, real-Chrome UA)

Out of scope (handled by other specs)

  • Content classification of sampled HTML → Spec 04 (classifier.ts)
  • DOM structure analysis (selectors, repeated patterns, hero blocks) → Spec 05 (structure-analyzer.ts)
  • Persisting samples into the sharded session store → Spec 08 wires sampleUrls results into SessionStore.addSample from Spec 01
  • Surfacing the 'blocked' transport to the UI ("WAF-protected: choose option a/b/c") → Spec 11 (frontend) reads the discover endpoint's session log and renders the §19b L3 UI prompt
  • Per-locale crawling, residential proxies, Algolia Crawler IP allowlist UX → §15 future / out of v1
  • L4 of the §19b ladder (manual residential-proxy procurement) → explicitly out of scope for this spike per §19b
  • Real Playwright launch in tests (we mock playwright in unit tests) → Spec 13 (smoke test cluster) runs one real-browser smoke against a known-good public site