Crawler Factory: Spec 08: Backend API Endpoints Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the eight Vercel function entry points that expose the Crawler Factory pipeline (Specs 01–07) to the frontend. Each handler is thin: validate request with zod, call the appropriate lib/factory/* orchestrator, and return JSON or an SSE stream. SSE for /discover and /crawl-progress; non-streaming JSON for the other six.

Architecture: Pure handler layer. No business logic in api-src/factory/*: every endpoint delegates to a lib/factory/* module. Two streaming endpoints (discover, crawl-progress) write SSE events using the data: {json}\n\n framing established in api-src/search.ts:220-339. Six non-streaming endpoints follow the api-src/prepare-dossier.ts shape: CORS preflight → method check → zod validation → orchestrator call → JSON response. Every endpoint is unit-tested with mocked lib/factory/* modules using a hand-rolled makeReq / makeRes pair (pattern from api-src/__tests__/prepare-dossier.test.ts).

Tech Stack: TypeScript (strict), @vercel/node Vercel function signature, zod 3.25 for request validation, vitest for unit tests, pino logger via lib/utils/logger.ts, uuid for request IDs.

Depends on: - Spec 01 (lib/factory/types.ts, session-store.ts, blueprint-store.ts, dss.ts) - Spec 02 (lib/factory/sitemap.ts: walkSitemap, lib/factory/path-grouper.ts) - Spec 03 (lib/factory/sampler.ts) - Spec 04 (lib/factory/classifier.ts) - Spec 05 (lib/factory/structure-analyzer.ts, extractor-generator.ts, sandbox-runner.ts) - Spec 06 (lib/factory/crawler-client.ts, index-manager.ts) - Spec 07 (lib/factory/brand-domain-discoverer.ts: optional, used only if discover.ts surfaces multi-brand sites)

Consumed by: Spec 09 (frontend hooks call these endpoints), Spec 13 (bundles api-src/factory/*.ts to api/factory/*.mjs).

Plan source: Projects/Crawler-Factory/00-Plan.md §4g (API contract), §7 Tasks 14–21, §19a (First-touch protocol).

---

File structure

Path	Responsibility
api-src/factory/_shared.ts	Shared CORS helper, SSE writer, zod-error → 400 mapper, request ID + logger setup
api-src/factory/discover.ts	POST: SSE stream piping walkSitemap → path-grouper → sampler → classifier. Persists session + url_shards + samples.
api-src/factory/sample.ts	POST: fetches N samples for a pathGroup, persists record_type='sample', returns refs
api-src/factory/generate-extractor.ts	POST: runs structureAnalyzer then extractorGenerator, persists blueprint draft, returns {recordExtractor, crawlerConfig, analysis}
api-src/factory/test-sandbox.ts	POST: runs sandboxRunner against all samples, persists sandboxResults, returns per-sample results
api-src/factory/test-real.ts	POST: creates/reuses test crawler factory-test-{sessionId}-{pathGroupId}, PATCH config, calls crawlUrls, polls until done, returns extracted records
api-src/factory/create-crawler.ts	POST: orchestrator: ensureIndex → createCrawler → patchConfig → startReindex → saveBlueprint → write JS file → update session
api-src/factory/crawl-progress.ts	GET: SSE poll loop (3s interval) on getStatus+getStats. Emits progress/done/timeout.
api-src/factory/session.ts	GET (read by id, 404 if missing) and PUT (upsert) the main session record
api-src/factory/blueprints.ts	GET list: optional content_domain filter
api-src/factory/__tests__/_shared.test.ts	Shared helper unit tests
api-src/factory/__tests__/discover.test.ts	Discover SSE tests (validation + event ordering + URL streaming)
api-src/factory/__tests__/sample.test.ts	Sample endpoint tests
api-src/factory/__tests__/generate-extractor.test.ts	Generate-extractor tests
api-src/factory/__tests__/test-sandbox.test.ts	Sandbox endpoint tests
api-src/factory/__tests__/test-real.test.ts	Real-crawl endpoint tests
api-src/factory/__tests__/create-crawler.test.ts	Create-crawler orchestrator tests
api-src/factory/__tests__/crawl-progress.test.ts	Crawl-progress SSE tests
api-src/factory/__tests__/session.test.ts	Session CRUD tests
api-src/factory/__tests__/blueprints.test.ts	Blueprints list tests
scripts/bundle-api.mjs	Add api-src/factory/*.ts glob entry points so each bundles to api/factory/*.mjs

---

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every handler in this spec MUST follow these rules:

1. Two-layer type safety. zod schema for the request body or query (validates at the boundary); tsc --noEmit strict for write-time. No any except at vitest mock seams (as any cast on req/res).
2. Logger in every module. Top of every file: import { createLogger } from '../../lib/utils/logger'; and const log = createLogger('factory:api:<name>', requestId); instantiated inside the handler with the per-request UUID. No console.log.
3. Try/catch every fallible call. Every orchestrator invocation, every Algolia call, every fetch wrapped. Catch zod errors first → 400. Catch generic last → 500. Always log before returning.
4. Functions ≤20 lines, ≤3 params. Handlers may exceed 20 lines for the request lifecycle but the orchestrator code in each handler delegates to lib/factory/*: handlers do not contain business logic.
5. Docstring on every exported handler. Single line explaining the endpoint contract. JSDoc the SSE event types.
6. No raw dicts crossing module boundaries. Every request body is zod-parsed; every orchestrator return type is the typed export from lib/factory/*.
7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit. One commit per task minimum.
8. Mock external services in unit tests. Mock the entire lib/factory/* module under test (vi.mock('../../../lib/factory/sitemap', ...)). No real Algolia, no real HTTP. SSE assertions use a captured res.write spy.
9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes. Files: kebab-case.ts. Tests: same name with .test.ts. Handler default export is unnamed, called handler internally for clarity in stack traces.
10. CORS. Every handler emits the same CORS headers as api-src/prepare-dossier.ts:6-27 via the shared helper in _shared.ts. OPTIONS preflight returns 200.
11. Conventional commits. feat(factory):, test(factory):, chore(factory):, etc. One logical change per commit.
12. First-touch protocol respected (§19a). discover.ts MUST persist session.status='discovering' BEFORE any crawl work begins, so a Vercel timeout does not lose context.

---

SSE event format (canonical for discover + crawl-progress)

Both streaming endpoints emit Server-Sent Events using the simplest framing:

data: <json-on-one-line>\n\n

No event: lines, no id:, no retry:. A frontend consumer parses by splitting on \n\n and JSON.parse-ing each chunk. This matches the convention in api-src/search.ts:333 (res.write(\data: ${JSON.stringify(...)}\n\n`)`).

Both streams set headers identical to search.ts:220-226:

res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache, no-transform');
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Accel-Buffering', 'no'); // disable nginx/proxy buffering
if (req.socket?.setTimeout) req.socket.setTimeout(300000); // 5 minutes
if (req.socket?.setNoDelay) req.socket.setNoDelay(true);

---

Task 0: Shared helpers: CORS, SSE writer, zod-error mapper

Files: - Create: api-src/factory/_shared.ts - Create: api-src/factory/__tests__/_shared.test.ts

Centralize the boilerplate so the eight handlers stay focused on their orchestrator call. Every helper is a pure function (or a closure over res) with no side effects beyond what its name promises.

• [ ] Step 0.1: Write failing test for getCorsHeaders

Create api-src/factory/__tests__/_shared.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { getCorsHeaders, hashUrl, sseWriter, zodErrorTo400 } from '../_shared';
import { z } from 'zod';

describe('getCorsHeaders', () => {
  it('returns localhost origin verbatim when request comes from localhost', () => {
    const headers = getCorsHeaders('http://localhost:5173');
    expect(headers['Access-Control-Allow-Origin']).toBe('http://localhost:5173');
    expect(headers['Access-Control-Allow-Methods']).toContain('POST');
    expect(headers['Access-Control-Allow-Methods']).toContain('GET');
    expect(headers['Access-Control-Allow-Methods']).toContain('OPTIONS');
  });

  it('echoes allow-listed production origin', () => {
    const headers = getCorsHeaders('https://algolia-central.vercel.app');
    expect(headers['Access-Control-Allow-Origin']).toBe('https://algolia-central.vercel.app');
  });

  it('falls back to default origin for unknown caller', () => {
    const headers = getCorsHeaders('https://evil.example.com');
    expect(headers['Access-Control-Allow-Origin']).toBe('http://localhost:5173');
  });

  it('handles null origin without crashing', () => {
    const headers = getCorsHeaders(null);
    expect(headers['Access-Control-Allow-Origin']).toBeTruthy();
  });
});

describe('sseWriter', () => {
  it('writes data: <json>\\n\\n frames to res.write', () => {
    const writes: string[] = [];
    const res = { write: vi.fn((s: string) => { writes.push(s); return true; }) } as any;
    const writer = sseWriter(res);
    writer({ type: 'log', level: 'info', message: 'hello' });
    expect(writes).toHaveLength(1);
    expect(writes[0]).toBe('data: {"type":"log","level":"info","message":"hello"}\n\n');
  });

  it('serializes nested objects without dropping keys', () => {
    const writes: string[] = [];
    const res = { write: vi.fn((s: string) => { writes.push(s); return true; }) } as any;
    const writer = sseWriter(res);
    writer({ type: 'progress', stats: { indexed: 12, errors: 0 } });
    expect(writes[0]).toContain('"indexed":12');
    expect(writes[0]).toContain('"errors":0');
  });
});

describe('zodErrorTo400', () => {
  it('returns a 400-shaped object for a zod ZodError', () => {
    const schema = z.object({ url: z.string().url() });
    try {
      schema.parse({ url: 'not-a-url' });
    } catch (e) {
      const result = zodErrorTo400(e);
      expect(result.status).toBe(400);
      expect(result.body.error).toBe('Bad Request');
      expect(result.body.issues).toBeDefined();
      expect(Array.isArray(result.body.issues)).toBe(true);
    }
  });

  it('returns null when the error is not a ZodError', () => {
    expect(zodErrorTo400(new Error('boom'))).toBeNull();
  });
});

describe('hashUrl', () => {
  it('returns a 16-char lowercase hex string', () => {
    const h = hashUrl('https://x.com/blog/a');
    expect(h).toMatch(/^[a-f0-9]{16}$/);
  });

  it('is deterministic — same URL → same hash', () => {
    expect(hashUrl('https://x.com/p')).toBe(hashUrl('https://x.com/p'));
  });

  it('distinguishes between distinct URLs', () => {
    expect(hashUrl('https://x.com/a')).not.toBe(hashUrl('https://x.com/b'));
  });
});

• [ ] Step 0.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/_shared.test.ts

Expected: FAIL: Cannot find module '../_shared'.

• [ ] Step 0.3: Implement _shared.ts

Create api-src/factory/_shared.ts:

/**
 * Shared helpers for /api/factory/* handlers.
 *
 * Why: each endpoint repeats CORS, SSE framing, zod-error → 400 mapping. Hoist
 * once. Every handler imports from here. No business logic lives in this file.
 */

import type { VercelResponse } from '@vercel/node';
import { ZodError } from 'zod';
import { createHash } from 'node:crypto';

const ALLOWED_ORIGINS = [
  'http://localhost:5173',
  'http://localhost:5174',
  'http://localhost:8080',
  'http://127.0.0.1:5173',
  'http://127.0.0.1:5174',
  'https://algolia-central.lovable.app',
  'https://algolia-central-git-rc2-algolia-arijitchowdhury80.vercel.app',
  'https://algolia-central.vercel.app',
];

const DEFAULT_ORIGIN = 'http://localhost:5173';

/**
 * Build CORS headers for a Vercel handler. Mirrors the pattern in
 * api-src/prepare-dossier.ts:6-27 so all factory endpoints behave identically.
 */
export function getCorsHeaders(origin: string | null): Record<string, string> {
  const isLocal =
    origin && (origin.includes('localhost') || origin.includes('127.0.0.1'));
  const responseOrigin = isLocal
    ? origin
    : origin && ALLOWED_ORIGINS.includes(origin)
    ? origin
    : DEFAULT_ORIGIN;
  return {
    'Access-Control-Allow-Origin': responseOrigin,
    'Access-Control-Allow-Methods': 'GET, POST, PUT, OPTIONS',
    'Access-Control-Allow-Headers':
      'authorization, x-client-info, apikey, content-type, x-session-id',
    'Access-Control-Allow-Credentials': 'true',
  };
}

/**
 * Apply the canonical SSE response headers and return a writer closure.
 * The closure serializes any JSON-shaped event with the `data: <json>\n\n`
 * framing established in api-src/search.ts:333.
 */
export function applySseHeaders(req: { socket?: { setTimeout?: (ms: number) => void; setNoDelay?: (v: boolean) => void } }, res: VercelResponse): void {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache, no-transform');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no');
  if (req.socket?.setTimeout) req.socket.setTimeout(300000);
  if (req.socket?.setNoDelay) req.socket.setNoDelay(true);
}

/**
 * Return a writer function bound to `res` that emits `data: <json>\n\n` frames.
 * Caller is responsible for `res.end()` after the last event.
 */
export function sseWriter(res: VercelResponse): (event: unknown) => void {
  return (event: unknown) => {
    res.write(`data: ${JSON.stringify(event)}\n\n`);
  };
}

/**
 * Map a thrown error to a 400-shaped response if it is a ZodError; else null.
 * Caller decides whether to fall through to a 500.
 */
export function zodErrorTo400(
  err: unknown
): { status: 400; body: { error: 'Bad Request'; issues: unknown[] } } | null {
  if (err instanceof ZodError) {
    return {
      status: 400,
      body: { error: 'Bad Request', issues: err.issues },
    };
  }
  return null;
}

/**
 * Apply CORS + standard security headers in one call, identical to the
 * established pattern in api-src/search.ts:84-95.
 */
export function applyStandardHeaders(origin: string | null, res: VercelResponse): void {
  Object.entries(getCorsHeaders(origin)).forEach(([k, v]) => res.setHeader(k, v));
  res.setHeader('X-Content-Type-Options', 'nosniff');
  res.setHeader('X-Frame-Options', 'DENY');
  res.setHeader('Referrer-Policy', 'strict-origin-when-cross-origin');
}

/**
 * Stable, short, zero-deps URL hash used to key per-sample shards in the
 * SessionStore (`addSample(sessionId, pathGroupId, urlHash, …)`). Spec 03's
 * `Sample` type intentionally does NOT carry a hash — handlers compute it at
 * the API boundary. SHA-1 truncated to 16 hex chars: 64-bit collision space
 * is comfortably larger than the per-pathGroup sample-fan-out (≤20).
 */
export function hashUrl(url: string): string {
  return createHash('sha1').update(url).digest('hex').slice(0, 16);
}

• [ ] Step 0.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/_shared.test.ts

Expected: all tests pass (4 getCorsHeaders + 2 sseWriter + 2 zodErrorTo400 + 3 hashUrl = 11 tests).

• [ ] Step 0.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 0.6: Commit

git add api-src/factory/_shared.ts api-src/factory/__tests__/_shared.test.ts
git commit -m "feat(factory): shared helpers — CORS, SSE writer, zod 400 mapper"

---

Task 1: /api/factory/discover: SSE streaming discovery endpoint

Files: - Create: api-src/factory/discover.ts - Create: api-src/factory/__tests__/discover.test.ts

Pipes Spec 02 → Spec 03 → Spec 04 over a single SSE response. Per master plan §7 Task 14 + §19a, the handler MUST persist session.status='discovering' BEFORE any crawl work begins. Events emitted (in order):

1. {type:'log', level, message}: narrative progress
2. {type:'urls', count, batch}: every 1000 URLs found, persisted as a url_shard
3. {type:'pathGroup', group}: when path-grouper emits a group
4. {type:'classified', pathGroupId, contentDomain, confidence, method}: per pathGroup classification
5. {type:'done', sessionId}: final

• [ ] Step 1.1: Write failing test: request validation

Create api-src/factory/__tests__/discover.test.ts:

/**
 * Contract tests for api-src/factory/discover.ts
 *
 * Mocks every lib/factory/* dependency so we test ONLY the handler's
 * request validation, event emission order, persistence calls, and SSE framing.
 */

import { describe, it, expect, vi, beforeEach } from 'vitest';

// ─── Module-level mocks (must precede dynamic import of handler) ────────────

const mockSaveSession = vi.fn();
const mockAppendUrls = vi.fn();
const mockAppendLog = vi.fn();
const mockAddSample = vi.fn();

vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    saveSession: mockSaveSession,
    appendUrls: mockAppendUrls,
    appendLog: mockAppendLog,
    addSample: mockAddSample,
  })),
}));

const mockWalkSitemap = vi.fn();
vi.mock('../../../lib/factory/sitemap', () => ({
  walkSitemap: mockWalkSitemap,
}));

const mockGroupByPath = vi.fn();
vi.mock('../../../lib/factory/path-grouper', () => ({
  groupByPath: mockGroupByPath,
}));

const mockSampleUrls = vi.fn();
vi.mock('../../../lib/factory/sampler', () => ({
  sampleUrls: mockSampleUrls,
}));

const mockClassify = vi.fn();
vi.mock('../../../lib/factory/classifier', () => ({
  classify: mockClassify,
}));

const mockDetectCms = vi.fn();
vi.mock('../../../lib/factory/cms-detector', () => ({
  detectCms: mockDetectCms,
}));

// ─── Test scaffolding ───────────────────────────────────────────────────────

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { url: 'https://www.algolia.com' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const writes: string[] = [];
  let ended = false;
  const res = {
    _status: 200,
    _writes: writes,
    _ended: () => ended,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    write(s: string) { writes.push(s); return true; },
    end() { ended = true; return res; },
    json(b: unknown) { res._writes.push(JSON.stringify(b)); return res; },
    send(b: unknown) { res._writes.push(typeof b === 'string' ? b : JSON.stringify(b)); return res; },
  };
  return res;
};

const parseSseFrames = (writes: string[]): unknown[] =>
  writes
    .filter((s) => s.startsWith('data: '))
    .map((s) => JSON.parse(s.slice('data: '.length).trim()));

describe('POST /api/factory/discover — request validation', () => {
  beforeEach(() => {
    vi.clearAllMocks();
  });

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../discover');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('returns 200 for OPTIONS preflight', async () => {
    const { default: handler } = await import('../discover');
    const req = makeReq({ method: 'OPTIONS' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
  });

  it('rejects missing url with 400', async () => {
    const { default: handler } = await import('../discover');
    const req = makeReq({ body: {} }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('rejects malformed url with 400', async () => {
    const { default: handler } = await import('../discover');
    const req = makeReq({ body: { url: 'not-a-url' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });
});

• [ ] Step 1.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/discover.test.ts

Expected: FAIL: Cannot find module '../discover'.

• [ ] Step 1.3: Implement discover.ts

Create api-src/factory/discover.ts:

/**
 * POST /api/factory/discover
 *
 * SSE-streaming discovery endpoint. Body: { url: string, sessionId?: string }.
 * Emits in order:
 *   data: {type:'log',  level, message}
 *   data: {type:'urls', count, batch}            // every 1000 URLs
 *   data: {type:'pathGroup', group}              // each new path group
 *   data: {type:'classified', pathGroupId, contentDomain, confidence, method}
 *   data: {type:'done', sessionId}
 *
 * Per §19a: session is persisted with status='discovering' BEFORE the walker
 * starts so a Vercel timeout does not lose context.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { walkSitemap } from '../../lib/factory/sitemap';
import { groupByPath } from '../../lib/factory/path-grouper';
import { sampleUrls, type Sample } from '../../lib/factory/sampler';
import { classify } from '../../lib/factory/classifier';
import { detectCms } from '../../lib/factory/cms-detector';
import {
  applyStandardHeaders,
  applySseHeaders,
  hashUrl,
  sseWriter,
  zodErrorTo400,
} from './_shared';
import type { FactorySession, PathGroup, Verdict } from '../../lib/factory/types';

const RequestSchema = z.object({
  url: z.string().url(),
  sessionId: z.string().min(1).optional(),
});

const URL_BATCH_SIZE = 1000;
const SAMPLES_PER_GROUP = 5;

function deriveDomain(rawUrl: string): string {
  return new URL(rawUrl).hostname.replace(/^www\./, '');
}

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:discover', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') {
    res.status(200).send('ok');
    return;
  }
  if (req.method !== 'POST') {
    res.status(405).json({ error: 'Method Not Allowed' });
    return;
  }

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) {
      res.status(mapped.status).json(mapped.body);
      return;
    }
    log.error({ err: String(e) }, 'unexpected validation error');
    res.status(500).json({ error: 'Internal Server Error' });
    return;
  }

  const sessionId = parsed.sessionId ?? `sess_${uuidv4()}`;
  const sourceUrlRoot = new URL(parsed.url).origin;
  const sourceDomain = deriveDomain(parsed.url);

  // Switch the response to SSE BEFORE doing any work.
  applySseHeaders(req, res);
  const emit = sseWriter(res);

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  // (§19a) Persist initial session before any crawl work.
  const initialSession: FactorySession = {
    objectID: sessionId,
    record_type: 'session',
    status: 'discovering',
    source_domain: sourceDomain,
    source_url_root: sourceUrlRoot,
    createdAt: Date.now(),
    updatedAt: Date.now(),
    discovery: {
      sitemapUrls: [],
      urlCount: 0,
      urlShardCount: 0,
      schemaOrgCoverage: 0,
    },
    pathGroups: [],
  };

  try {
    await store.saveSession(initialSession);
    emit({ type: 'log', level: 'info', message: `session ${sessionId} created` });

    // ─── Phase 1: walk sitemap, stream URLs in 1000-URL batches ────────────
    const urlBuffer: string[] = [];
    let urlCount = 0;
    let urlShardCount = 0;
    for await (const url of walkSitemap(parsed.url)) {
      urlBuffer.push(url);
      urlCount += 1;
      if (urlBuffer.length >= URL_BATCH_SIZE) {
        const batch = urlBuffer.splice(0, URL_BATCH_SIZE);
        await store.appendUrls(sessionId, batch);
        urlShardCount += 1;
        emit({ type: 'urls', count: urlCount, batch: batch.length });
      }
    }
    if (urlBuffer.length > 0) {
      await store.appendUrls(sessionId, urlBuffer);
      urlShardCount += 1;
      emit({ type: 'urls', count: urlCount, batch: urlBuffer.length });
      urlBuffer.length = 0;
    }
    emit({ type: 'log', level: 'info', message: `${urlCount} URLs discovered` });

    // ─── Phase 2: path-group, sample, classify ─────────────────────────────
    const allUrls: string[] = [];
    for await (const url of walkSitemap(parsed.url)) {
      allUrls.push(url);
      if (allUrls.length >= 50000) break; // hard floor for in-memory grouping
    }
    const pathGroups: PathGroup[] = groupByPath(allUrls);

    // Per-host CMS cache so we run detectCms once per origin, not per pathGroup.
    // Spec 04's classify() needs `opts.cms` so its CMS-aware Layer 1 can fire.
    const cmsByHost = new Map<string, Awaited<ReturnType<typeof detectCms>>>();

    for (const group of pathGroups) {
      emit({ type: 'pathGroup', group });

      // B3 fix: Spec 03's sampleUrls(urls, opts) takes an opts object, not a number.
      // Slice to SAMPLES_PER_GROUP first, then pass the per-URL timeout option.
      const urls = group.sampleUrls.slice(0, SAMPLES_PER_GROUP);
      const samples: Sample[] = await sampleUrls(urls, { perUrlTimeoutMs: 5000 });

      // B4 fix: Spec 03's Sample has no urlHash field. Compute it inline at the
      // API boundary so the SessionStore objectID stays unique per sample URL.
      for (const s of samples) {
        await store.addSample(sessionId, group.id, hashUrl(s.url), {
          url: s.url,
          html: s.html,
          status: s.status,
        });
      }

      // B5 fix: Spec 04 exports classify(html, url, { cms }) → Verdict per page.
      // Detect CMS once per host (cached), then classify each sample, then reduce
      // to a single Verdict per pathGroup by max-confidence.
      const host = new URL(group.sampleUrls[0] ?? sourceUrlRoot).host;
      let cms = cmsByHost.get(host);
      if (!cms) {
        cms = await detectCms(samples);
        cmsByHost.set(host, cms);
      }

      const verdicts: Verdict[] = [];
      for (const s of samples) {
        const v = await classify(s.html, s.url, { cms: cms.cms });
        verdicts.push(v);
      }
      // Reduce to a single verdict — pick the highest-confidence sample.
      // Ties resolved by first-seen (stable). If samples is empty, fall back to
      // the AMBER_FLOOR-equivalent low-confidence marketing default.
      const verdict: Verdict = verdicts.length > 0
        ? verdicts.reduce((best, v) => (v.confidence > best.confidence ? v : best))
        : {
            contentDomain: 'marketing',
            confidence: 0.30,
            detectedVia: 'heuristic',
            contributingLayers: ['fallback'],
            schemaOrgType: null,
          };

      // SSE event field `method` is mapped from Verdict.detectedVia at emit time
      // (master plan §4g names the wire field `method`; Spec 04's type uses
      // `detectedVia`). Keep both layers honest by mapping at the boundary.
      emit({
        type: 'classified',
        pathGroupId: group.id,
        contentDomain: verdict.contentDomain,
        confidence: verdict.confidence,
        method: verdict.detectedVia,
      });

      await store.appendLog(sessionId, {
        ts: Date.now(),
        level: 'info',
        message: `classified ${group.id} → ${verdict.contentDomain} (${verdict.confidence})`,
      });
    }

    // ─── Phase 3: finalize session ─────────────────────────────────────────
    const finalSession: FactorySession = {
      ...initialSession,
      status: 'categorized',
      updatedAt: Date.now(),
      discovery: {
        sitemapUrls: [parsed.url],
        urlCount,
        urlShardCount,
        schemaOrgCoverage: 0,
      },
      pathGroups,
    };
    await store.saveSession(finalSession);

    emit({ type: 'done', sessionId });
    res.end();
    log.info({ sessionId, urlCount, pathGroupCount: pathGroups.length }, 'discover complete');
  } catch (err) {
    log.error({ sessionId, err: String(err) }, 'discover failed');
    try {
      emit({ type: 'log', level: 'error', message: `discover failed: ${String(err)}` });
      res.end();
    } catch {
      // socket closed; nothing to do
    }
  }
}

• [ ] Step 1.4: Run validation tests, expect PASS

npx vitest run api-src/factory/__tests__/discover.test.ts -t 'request validation'

Expected: 4 tests pass.

• [ ] Step 1.5: Add SSE-flow tests

Append to api-src/factory/__tests__/discover.test.ts:

describe('POST /api/factory/discover — SSE flow', () => {
  beforeEach(() => {
    vi.clearAllMocks();
    mockSaveSession.mockResolvedValue(undefined);
    mockAppendUrls.mockResolvedValue(1);
    mockAppendLog.mockResolvedValue(undefined);
    mockAddSample.mockResolvedValue(undefined);
  });

  it('persists initial session with status=discovering BEFORE walking', async () => {
    mockWalkSitemap.mockReturnValue((async function* () {})()); // empty
    mockGroupByPath.mockReturnValue([]);
    const { default: handler } = await import('../discover');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    // saveSession called at least once with status='discovering'
    const firstCall = mockSaveSession.mock.calls[0]?.[0];
    expect(firstCall?.status).toBe('discovering');
    expect(firstCall?.source_domain).toBe('algolia.com');
  });

  it('emits one urls event per 1000-URL batch and never caps the walker', async () => {
    const fakeUrls = Array.from({ length: 2500 }, (_, i) => `https://x.com/p/${i}`);
    mockWalkSitemap.mockImplementation(() =>
      (async function* () {
        for (const u of fakeUrls) yield u;
      })()
    );
    mockGroupByPath.mockReturnValue([]);
    const { default: handler } = await import('../discover');
    const req = makeReq({ body: { url: 'https://x.com' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);

    const events = parseSseFrames(res._writes);
    const urlEvents = events.filter((e: any) => e.type === 'urls');
    // 2500 URLs ÷ 1000 = 2 full batches + 1 tail batch of 500 = 3 total
    expect(urlEvents.length).toBe(3);
    expect((urlEvents[0] as any).batch).toBe(1000);
    expect((urlEvents[2] as any).batch).toBe(500);
  });

  it('emits pathGroup then classified events per group', async () => {
    mockWalkSitemap.mockReturnValue((async function* () { yield 'https://x.com/blog/a'; })());
    mockGroupByPath.mockReturnValue([
      {
        id: 'pg_1',
        pattern: '/blog/*',
        urlCount: 1,
        sampleUrls: ['https://x.com/blog/a'],
        detectedDomain: null,
        detectionConfidence: null,
        detectionMethod: null,
        selected: false,
      },
    ]);
    mockSampleUrls.mockResolvedValue([
      // Spec 03 Sample shape: no urlHash field — handler computes hashUrl(s.url) inline.
      { url: 'https://x.com/blog/a', html: '<html/>', status: 200, contentLength: 5,
        isSpa: false, headers: {}, transport: 'fetch', truncated: false },
    ]);
    mockDetectCms.mockResolvedValue({
      cms: 'wordpress',
      confidence: 0.85,
      evidence: [],
    });
    // Spec 04 Verdict shape: detectedVia (not method); method is the SSE wire field.
    mockClassify.mockResolvedValue({
      contentDomain: 'marketing',
      confidence: 0.92,
      detectedVia: 'json-ld-type',
      contributingLayers: ['json-ld'],
      schemaOrgType: 'BlogPosting',
    });

    const { default: handler } = await import('../discover');
    const req = makeReq({ body: { url: 'https://x.com' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);

    const events = parseSseFrames(res._writes);
    const pathGroupEvents = events.filter((e: any) => e.type === 'pathGroup');
    const classifiedEvents = events.filter((e: any) => e.type === 'classified');
    expect(pathGroupEvents).toHaveLength(1);
    expect(classifiedEvents).toHaveLength(1);
    expect((classifiedEvents[0] as any).contentDomain).toBe('marketing');
    // SSE `method` field is mapped from Spec 04 Verdict.detectedVia at emit time.
    expect((classifiedEvents[0] as any).method).toBe('json-ld-type');
  });

  it('emits done with the sessionId at the end', async () => {
    mockWalkSitemap.mockReturnValue((async function* () {})());
    mockGroupByPath.mockReturnValue([]);
    const { default: handler } = await import('../discover');
    const req = makeReq({ body: { url: 'https://x.com', sessionId: 'sess_test' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);

    const events = parseSseFrames(res._writes);
    const doneEvent = events.find((e: any) => e.type === 'done');
    expect(doneEvent).toBeTruthy();
    expect((doneEvent as any).sessionId).toBe('sess_test');
  });

  it('writes SSE frames in canonical data: <json>\\n\\n format', async () => {
    mockWalkSitemap.mockReturnValue((async function* () {})());
    mockGroupByPath.mockReturnValue([]);
    const { default: handler } = await import('../discover');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    for (const frame of res._writes) {
      expect(frame.startsWith('data: ')).toBe(true);
      expect(frame.endsWith('\n\n')).toBe(true);
    }
  });

  it('emits error log frame and ends response when walker throws', async () => {
    mockWalkSitemap.mockImplementation(() => {
      throw new Error('sitemap fetch failed');
    });
    mockGroupByPath.mockReturnValue([]);
    const { default: handler } = await import('../discover');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    const events = parseSseFrames(res._writes);
    const errEvent = events.find(
      (e: any) => e.type === 'log' && e.level === 'error'
    );
    expect(errEvent).toBeTruthy();
  });
});

• [ ] Step 1.6: Run, expect PASS

npx vitest run api-src/factory/__tests__/discover.test.ts

Expected: all 10 tests pass.

• [ ] Step 1.7: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 1.8: Commit

git add api-src/factory/discover.ts api-src/factory/__tests__/discover.test.ts
git commit -m "feat(factory): /api/factory/discover SSE endpoint (streaming)"

---

Task 2: /api/factory/sample: fetch samples for a pathGroup

Files: - Create: api-src/factory/sample.ts - Create: api-src/factory/__tests__/sample.test.ts

POST {sessionId, pathGroupId, n?: number}. Loads the session's pathGroup, calls sampler.sampleUrls, persists each as a record_type='sample' shard, returns {samples: SampleRef[]}.

• [ ] Step 2.1: Write failing tests

Create api-src/factory/__tests__/sample.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockAddSample = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    addSample: mockAddSample,
  })),
}));

const mockSampleUrls = vi.fn();
vi.mock('../../../lib/factory/sampler', () => ({
  sampleUrls: mockSampleUrls,
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { sessionId: 'sess_001', pathGroupId: 'pg_1' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const minimalSession = (pathGroupId = 'pg_1') => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'categorized' as const,
  source_domain: 'x.com',
  source_url_root: 'https://x.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [
    {
      id: pathGroupId, pattern: '/blog/*', urlCount: 3,
      sampleUrls: ['https://x.com/blog/a', 'https://x.com/blog/b', 'https://x.com/blog/c'],
      detectedDomain: null, detectionConfidence: null, detectionMethod: null, selected: false,
    },
  ],
});

describe('POST /api/factory/sample', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../sample');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing sessionId with 400', async () => {
    const { default: handler } = await import('../sample');
    const req = makeReq({ body: { pathGroupId: 'pg_1' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('rejects missing pathGroupId with 400', async () => {
    const { default: handler } = await import('../sample');
    const req = makeReq({ body: { sessionId: 's' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session does not exist', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../sample');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 404 when pathGroup does not exist within session', async () => {
    mockGetSession.mockResolvedValue(minimalSession('different_pg'));
    const { default: handler } = await import('../sample');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('happy path — fetches samples, persists each, returns refs', async () => {
    mockGetSession.mockResolvedValue(minimalSession('pg_1'));
    // Spec 03 Sample shape: NO urlHash field — handler computes hashUrl(s.url) inline.
    mockSampleUrls.mockResolvedValue([
      { url: 'https://x.com/blog/a', html: '<html/>', status: 200, contentLength: 5,
        isSpa: false, headers: {}, transport: 'fetch', truncated: false },
      { url: 'https://x.com/blog/b', html: '<html/>', status: 200, contentLength: 5,
        isSpa: false, headers: {}, transport: 'fetch', truncated: false },
    ]);
    mockAddSample.mockResolvedValue(undefined);

    const { default: handler } = await import('../sample');
    const req = makeReq({ body: { sessionId: 'sess_001', pathGroupId: 'pg_1', n: 2 } }) as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(res._status).toBe(200);
    expect(mockAddSample).toHaveBeenCalledTimes(2);
    expect((res._body as any).samples).toHaveLength(2);
    // urlHash is computed via hashUrl(url) — assert format, not a specific value.
    expect((res._body as any).samples[0].urlHash).toMatch(/^[a-f0-9]{16}$/);
    // Verify uniqueness across distinct URLs
    expect((res._body as any).samples[0].urlHash).not.toBe((res._body as any).samples[1].urlHash);
  });

  it('returns 500 when sampler throws', async () => {
    mockGetSession.mockResolvedValue(minimalSession('pg_1'));
    mockSampleUrls.mockRejectedValue(new Error('network down'));

    const { default: handler } = await import('../sample');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 2.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/sample.test.ts

Expected: FAIL: Cannot find module '../sample'.

• [ ] Step 2.3: Implement sample.ts

Create api-src/factory/sample.ts:

/**
 * POST /api/factory/sample
 *
 * Body: { sessionId: string, pathGroupId: string, n?: number }
 * Loads the session, finds the named pathGroup, fetches up to N samples from
 * its sampleUrls list (delegating to lib/factory/sampler), persists each as
 * record_type='sample', and returns { samples: SampleRef[] }.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { sampleUrls } from '../../lib/factory/sampler';
import { applyStandardHeaders, hashUrl, zodErrorTo400 } from './_shared';

const RequestSchema = z.object({
  sessionId: z.string().min(1),
  pathGroupId: z.string().min(1),
  n: z.number().int().positive().max(20).default(5),
});

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:sample', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'POST') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  try {
    const session = await store.getSession(parsed.sessionId);
    if (!session) {
      return void res.status(404).json({ error: 'Not Found', message: 'session not found' });
    }
    const group = session.pathGroups.find((g) => g.id === parsed.pathGroupId);
    if (!group) {
      return void res.status(404).json({ error: 'Not Found', message: 'pathGroup not found' });
    }

    // B3 fix: Spec 03's sampleUrls(urls, opts) takes an opts object, not a number.
    // Slice to parsed.n first, then pass the per-URL timeout option.
    const urls = group.sampleUrls.slice(0, parsed.n);
    const samples = await sampleUrls(urls, { perUrlTimeoutMs: 5000 });
    // B4 fix: Spec 03's Sample has no urlHash field — compute at API boundary.
    for (const s of samples) {
      await store.addSample(parsed.sessionId, parsed.pathGroupId, hashUrl(s.url), {
        url: s.url,
        html: s.html,
        status: s.status,
      });
    }

    log.info({ sessionId: parsed.sessionId, pathGroupId: parsed.pathGroupId, count: samples.length }, 'samples persisted');
    // Wire-format includes html + transport so the SamplePreview UI (Spec 11)
    // can render HTML expand views and fetch-vs-Playwright badges without a
    // second round-trip. truncated flag preserved for accuracy.
    return void res.status(200).json({
      samples: samples.map((s) => ({
        url: s.url,
        urlHash: hashUrl(s.url),
        status: s.status,
        html: s.html,
        transport: s.transport,
        truncated: s.truncated,
        contentLength: s.contentLength,
      })),
    });
  } catch (err) {
    log.error({ err: String(err) }, 'sample failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 2.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/sample.test.ts

Expected: 7 tests pass.

• [ ] Step 2.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 2.6: Commit

git add api-src/factory/sample.ts api-src/factory/__tests__/sample.test.ts
git commit -m "feat(factory): /api/factory/sample endpoint"

---

Task 3: /api/factory/generate-extractor: extractor + crawler config

Files: - Create: api-src/factory/generate-extractor.ts - Create: api-src/factory/__tests__/generate-extractor.test.ts

POST {sessionId, pathGroupId, userFeedback?}. Loads samples + DSS row, calls structureAnalyzer then extractorGenerator (Spec 05), persists blueprint.recordExtractor + blueprint.crawlerConfig to the session, returns {recordExtractor, crawlerConfig, analysis} (analysis is Spec 05's StructureAnalysis shape: Spec 11's SamplePreview consumes it directly). When userFeedback is supplied, the generator prepends it to its LLM prompt.

• [ ] Step 3.1: Write failing tests

Create api-src/factory/__tests__/generate-extractor.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockSaveSession = vi.fn();
const mockGetSamples = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    saveSession: mockSaveSession,
    getSamples: mockGetSamples,
  })),
}));

const mockAnalyzeStructure = vi.fn();
vi.mock('../../../lib/factory/structure-analyzer', () => ({
  analyzeStructure: mockAnalyzeStructure,
}));

const mockGenerateExtractor = vi.fn();
vi.mock('../../../lib/factory/extractor-generator', () => ({
  generateExtractor: mockGenerateExtractor,
}));

// LLM provider stub — handler wires `getLLMProvider().generateContent({...}).text()`
// into an LlmFn; we stub the provider so neither analyzeStructure nor
// generateExtractor (both mocked) ever calls it.
vi.mock('../../../lib/llm', () => ({
  getLLMProvider: () => ({
    name: 'mock',
    capabilities: {
      supportsJsonMode: false, supportsJsonSchema: false,
      supportsSystemInstruction: true, supportsStreaming: false,
    },
    generateContent: async () => ({ text: () => '{}', json: <T,>() => ({} as T) }),
    startChat: () => ({
      sendMessage: async () => ({ text: () => '', json: <T,>() => ({} as T) }),
      sendMessageStream: async function* () {},
    }),
  }),
}));

vi.mock('../../../lib/factory/dss', () => ({
  getDss: (domain: string) => ({
    indexName: `algoliacentral_${domain}`,
    schemaOrgTypes: ['BlogPosting'],
    algoliaConfig: {
      searchableAttributes: ['headline', 'articleBody'],
      customRanking: ['desc(datePublished)'],
      attributesForFaceting: ['articleSection'],
    },
    description: 'mock',
  }),
  dssBySchemaOrgType: () => null,
  listContentDomains: () => ['marketing'],
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { sessionId: 'sess_001', pathGroupId: 'pg_1' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const sessionWithGroup = (detectedDomain: 'marketing' | null = 'marketing') => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'categorized' as const,
  source_domain: 'x.com', source_url_root: 'https://x.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [
    {
      id: 'pg_1', pattern: '/blog/*', urlCount: 3,
      sampleUrls: ['https://x.com/blog/a'],
      detectedDomain, detectionConfidence: 0.9, detectionMethod: 'json-ld' as const,
      selected: true,
    },
  ],
});

describe('POST /api/factory/generate-extractor', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing sessionId with 400', async () => {
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq({ body: { pathGroupId: 'pg_1' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session not found', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 422 when pathGroup has no detected domain', async () => {
    mockGetSession.mockResolvedValue(sessionWithGroup(null));
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(422);
  });

  it('returns 422 when no samples have been persisted for the pathGroup', async () => {
    mockGetSession.mockResolvedValue(sessionWithGroup('marketing'));
    mockGetSamples.mockResolvedValue([]); // empty — sample endpoint never ran
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(422);
  });

  it('happy path — calls structure-analyzer then extractor-generator and persists blueprint', async () => {
    mockGetSession.mockResolvedValue(sessionWithGroup('marketing'));
    mockGetSamples.mockResolvedValue([
      { url: 'https://x.com/blog/a', html: '<html><h1>hi</h1></html>', status: 200 },
    ]);
    // Spec 05 StructureAnalysis shape: { schemaOrgJsonLd, selectors, missing, confidence }
    mockAnalyzeStructure.mockResolvedValue({
      schemaOrgJsonLd: null,
      selectors: { headline: 'h1.title', articleBody: 'article.content' },
      missing: [],
      confidence: 0.9,
    });
    mockGenerateExtractor.mockResolvedValue({
      recordExtractor: '({ url, $ }) => [{ headline: $("h1").text() }]',
      crawlerConfig: {
        renderJavaScript: false, rateLimit: 8, pathsToMatch: ['/blog/*'],
      },
    });
    mockSaveSession.mockResolvedValue(undefined);

    const { default: handler } = await import('../generate-extractor');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(res._status).toBe(200);
    expect((res._body as any).recordExtractor).toContain('headline');
    expect((res._body as any).crawlerConfig.rateLimit).toBe(8);
    expect(mockSaveSession).toHaveBeenCalledTimes(1);
    const saved = mockSaveSession.mock.calls[0][0];
    expect(saved.pathGroups[0].blueprint.recordExtractor).toContain('headline');

    // Spec 05 contract: analyzeStructure(sample, contentDomain, opts) — assert
    // we passed a StructureSample (NOT a session-context object) and a domain.
    expect(mockAnalyzeStructure).toHaveBeenCalledWith(
      expect.objectContaining({ url: 'https://x.com/blog/a', html: expect.any(String), status: 200 }),
      'marketing',
      expect.objectContaining({ llm: expect.any(Function) })
    );
    // Spec 05 contract: generateExtractor opts uses key `structureAnalysis` (not `analysis`),
    // requires `pathGroup` and `llm`.
    expect(mockGenerateExtractor).toHaveBeenCalledWith(
      expect.objectContaining({
        pathGroup: expect.objectContaining({ id: 'pg_1' }),
        structureAnalysis: expect.objectContaining({ selectors: expect.any(Object) }),
        contentDomain: 'marketing',
        llm: expect.any(Function),
      })
    );
  });

  it('passes userFeedback through to the generator', async () => {
    mockGetSession.mockResolvedValue(sessionWithGroup('marketing'));
    mockGetSamples.mockResolvedValue([
      { url: 'https://x.com/blog/a', html: '<html/>', status: 200 },
    ]);
    mockAnalyzeStructure.mockResolvedValue({
      schemaOrgJsonLd: null, selectors: {}, missing: [], confidence: 0.5,
    });
    mockGenerateExtractor.mockResolvedValue({
      recordExtractor: '() => []',
      crawlerConfig: { renderJavaScript: false, rateLimit: 8, pathsToMatch: [] },
    });
    mockSaveSession.mockResolvedValue(undefined);

    const { default: handler } = await import('../generate-extractor');
    const req = makeReq({
      body: { sessionId: 'sess_001', pathGroupId: 'pg_1', userFeedback: 'add image field' },
    }) as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(mockGenerateExtractor).toHaveBeenCalledWith(
      expect.objectContaining({ userFeedback: 'add image field' })
    );
  });

  it('returns 500 when extractor generator throws', async () => {
    mockGetSession.mockResolvedValue(sessionWithGroup('marketing'));
    mockGetSamples.mockResolvedValue([
      { url: 'https://x.com/blog/a', html: '<html/>', status: 200 },
    ]);
    mockAnalyzeStructure.mockResolvedValue({
      schemaOrgJsonLd: null, selectors: {}, missing: [], confidence: 0.5,
    });
    mockGenerateExtractor.mockRejectedValue(new Error('LLM down'));
    const { default: handler } = await import('../generate-extractor');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 3.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/generate-extractor.test.ts

Expected: FAIL: module not found.

• [ ] Step 3.3: Implement generate-extractor.ts

Create api-src/factory/generate-extractor.ts:

/**
 * POST /api/factory/generate-extractor
 *
 * Body: { sessionId: string, pathGroupId: string, userFeedback?: string }
 *
 * Pipeline:
 *   1. Load session, find pathGroup, look up DSS row by pathGroup.detectedDomain.
 *   2. Run structure-analyzer over the pathGroup's samples (read from session).
 *   3. Run extractor-generator with the analysis + DSS template + optional feedback.
 *   4. Persist {recordExtractor, crawlerConfig} on session.pathGroups[i].blueprint.
 *   5. Return them.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { analyzeStructure, type StructureSample } from '../../lib/factory/structure-analyzer';
import { generateExtractor, type LlmFn } from '../../lib/factory/extractor-generator';
import { getDss } from '../../lib/factory/dss';
import { getLLMProvider } from '../../lib/llm';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';

const RequestSchema = z.object({
  sessionId: z.string().min(1),
  pathGroupId: z.string().min(1),
  userFeedback: z.string().max(2000).optional(),
});

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:generate-extractor', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'POST') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  try {
    const session = await store.getSession(parsed.sessionId);
    if (!session) {
      return void res.status(404).json({ error: 'Not Found', message: 'session not found' });
    }
    const groupIdx = session.pathGroups.findIndex((g) => g.id === parsed.pathGroupId);
    if (groupIdx === -1) {
      return void res.status(404).json({ error: 'Not Found', message: 'pathGroup not found' });
    }
    const group = session.pathGroups[groupIdx];
    if (!group.detectedDomain) {
      return void res.status(422).json({
        error: 'Unprocessable Entity',
        message: 'pathGroup has no detectedDomain — classify first',
      });
    }

    const dssEntry = getDss(group.detectedDomain);

    // B6 fix step 1: load the persisted samples for this pathGroup. Spec 05's
    // analyzeStructure expects a StructureSample ({url, html, status, …}), not
    // a session-context object. SessionStore.getSamples queries the sharded
    // index for `parent_id:<sessionId> AND record_type:'sample' AND
    // pathGroupId:<id>` and returns the persisted bodies.
    // NOTE → Spec 01 follow-up: `getSamples(sessionId, pathGroupId)` is not yet
    // declared on SessionStore. Add it as a thin wrapper around
    // `searchSingleIndex({filters: 'parent_id:<id> AND record_type:sample AND
    // pathGroupId:<id>'})` returning `{url, html, status}[]`.
    const samples = await store.getSamples(parsed.sessionId, parsed.pathGroupId);
    if (samples.length === 0) {
      return void res.status(422).json({
        error: 'Unprocessable Entity',
        message: 'no samples for pathGroup — call /api/factory/sample first',
      });
    }
    const firstSample = samples[0];
    const structureSample: StructureSample = {
      url: firstSample.url,
      html: firstSample.html,
      status: firstSample.status,
    };

    // B6 fix step 2: wire an LlmFn adapter from the project's LLM provider.
    // Spec 05's analyzeStructure + generateExtractor both accept
    // `(prompt: string) => Promise<string>`; the provider returns an
    // LLMResponse whose .text() is the raw string we need.
    const llmProvider = getLLMProvider();
    const llm: LlmFn = async (prompt: string) => {
      const resp = await llmProvider.generateContent({
        systemPrompt: '',
        messages: [{ role: 'user', content: prompt }],
        temperature: 0,
        jsonMode: false,
      });
      return resp.text();
    };

    // B6 fix step 3: pass the StructureSample + contentDomain + llm opts.
    const analysis = await analyzeStructure(structureSample, group.detectedDomain, { llm });

    // B6 fix step 4: pass pathGroup + structureAnalysis (NOT `analysis`) + llm.
    const generated = await generateExtractor({
      pathGroup: group,
      structureAnalysis: analysis,
      dssEntry,
      contentDomain: group.detectedDomain,
      llm,
      userFeedback: parsed.userFeedback,
    });

    // Persist on session.pathGroups[i].blueprint
    const updatedSession = {
      ...session,
      updatedAt: Date.now(),
      pathGroups: session.pathGroups.map((g, i) =>
        i === groupIdx
          ? {
              ...g,
              blueprint: {
                ...(g.blueprint ?? {}),
                indexName: dssEntry.indexName,
                recordExtractor: generated.recordExtractor,
                crawlerConfig: generated.crawlerConfig,
              },
            }
          : g
      ),
    };
    await store.saveSession(updatedSession);

    log.info(
      { sessionId: parsed.sessionId, pathGroupId: parsed.pathGroupId, hasFeedback: !!parsed.userFeedback },
      'extractor generated'
    );
    // Spec 11 follow-up: also return `analysis` so SamplePreview can render
    // selectors / missing / confidence without a second round-trip.
    return void res.status(200).json({
      recordExtractor: generated.recordExtractor,
      crawlerConfig: generated.crawlerConfig,
      analysis,
    });
  } catch (err) {
    log.error({ err: String(err) }, 'generate-extractor failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 3.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/generate-extractor.test.ts

Expected: 7 tests pass.

• [ ] Step 3.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 3.6: Commit

git add api-src/factory/generate-extractor.ts api-src/factory/__tests__/generate-extractor.test.ts
git commit -m "feat(factory): /api/factory/generate-extractor endpoint"

---

Task 4: /api/factory/test-sandbox: run extractor against samples

Files: - Create: api-src/factory/test-sandbox.ts - Create: api-src/factory/__tests__/test-sandbox.test.ts

POST {sessionId, pathGroupId}. Runs sandboxRunner (Spec 05) against every persisted sample for the pathGroup, persists results to session.pathGroups[i].blueprint.sandboxResults, returns {results: SandboxResult[]}.

• [ ] Step 4.1: Write failing tests

Create api-src/factory/__tests__/test-sandbox.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockSaveSession = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    saveSession: mockSaveSession,
  })),
}));

const mockRunSandbox = vi.fn();
vi.mock('../../../lib/factory/sandbox-runner', () => ({
  runSandbox: mockRunSandbox,
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { sessionId: 'sess_001', pathGroupId: 'pg_1' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const sessionWithBlueprint = () => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'configuring' as const,
  source_domain: 'x.com', source_url_root: 'https://x.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [
    {
      id: 'pg_1', pattern: '/blog/*', urlCount: 3,
      sampleUrls: ['https://x.com/blog/a'],
      detectedDomain: 'marketing' as const,
      detectionConfidence: 0.9, detectionMethod: 'json-ld' as const,
      selected: true,
      blueprint: {
        indexName: 'algoliacentral_marketing',
        recordExtractor: '({ $ }) => [{ headline: $("h1").text() }]',
        crawlerConfig: { renderJavaScript: false, rateLimit: 8, pathsToMatch: ['/blog/*'] },
      },
    },
  ],
});

describe('POST /api/factory/test-sandbox', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../test-sandbox');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing sessionId with 400', async () => {
    const { default: handler } = await import('../test-sandbox');
    const req = makeReq({ body: {} }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session not found', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../test-sandbox');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 422 when pathGroup has no blueprint yet', async () => {
    const noBp = sessionWithBlueprint();
    delete (noBp.pathGroups[0] as any).blueprint;
    mockGetSession.mockResolvedValue(noBp);
    const { default: handler } = await import('../test-sandbox');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(422);
  });

  it('happy path — runs sandbox, persists results, returns them', async () => {
    mockGetSession.mockResolvedValue(sessionWithBlueprint());
    mockRunSandbox.mockResolvedValue([
      { url: 'https://x.com/blog/a', records: [{ headline: 'A' }], errors: [] },
      { url: 'https://x.com/blog/b', records: [{ headline: 'B' }], errors: [] },
    ]);
    mockSaveSession.mockResolvedValue(undefined);

    const { default: handler } = await import('../test-sandbox');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(res._status).toBe(200);
    expect((res._body as any).results).toHaveLength(2);
    expect(mockSaveSession).toHaveBeenCalledTimes(1);
    const saved = mockSaveSession.mock.calls[0][0];
    expect(saved.pathGroups[0].blueprint.sandboxResults).toHaveLength(2);
  });

  it('returns 500 when runSandbox throws', async () => {
    mockGetSession.mockResolvedValue(sessionWithBlueprint());
    mockRunSandbox.mockRejectedValue(new Error('vm crash'));
    const { default: handler } = await import('../test-sandbox');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 4.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/test-sandbox.test.ts

Expected: FAIL.

• [ ] Step 4.3: Implement test-sandbox.ts

Create api-src/factory/test-sandbox.ts:

/**
 * POST /api/factory/test-sandbox
 *
 * Body: { sessionId: string, pathGroupId: string }
 * Runs lib/factory/sandbox-runner against every persisted sample for the
 * pathGroup using the blueprint's recordExtractor. Persists results back to
 * session.pathGroups[i].blueprint.sandboxResults.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { runSandbox } from '../../lib/factory/sandbox-runner';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';

const RequestSchema = z.object({
  sessionId: z.string().min(1),
  pathGroupId: z.string().min(1),
});

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:test-sandbox', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'POST') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  try {
    const session = await store.getSession(parsed.sessionId);
    if (!session) {
      return void res.status(404).json({ error: 'Not Found', message: 'session not found' });
    }
    const groupIdx = session.pathGroups.findIndex((g) => g.id === parsed.pathGroupId);
    if (groupIdx === -1) {
      return void res.status(404).json({ error: 'Not Found', message: 'pathGroup not found' });
    }
    const group = session.pathGroups[groupIdx];
    if (!group.blueprint || !group.detectedDomain) {
      return void res.status(422).json({
        error: 'Unprocessable Entity',
        message: 'blueprint not yet generated — call /generate-extractor first',
      });
    }

    const results = await runSandbox({
      sessionId: parsed.sessionId,
      pathGroupId: parsed.pathGroupId,
      contentDomain: group.detectedDomain,
      recordExtractor: group.blueprint.recordExtractor,
    });

    const updated = {
      ...session,
      updatedAt: Date.now(),
      pathGroups: session.pathGroups.map((g, i) =>
        i === groupIdx
          ? { ...g, blueprint: { ...(g.blueprint ?? {} as any), sandboxResults: results } }
          : g
      ),
    };
    await store.saveSession(updated);

    log.info(
      { sessionId: parsed.sessionId, pathGroupId: parsed.pathGroupId, count: results.length },
      'sandbox complete'
    );
    return void res.status(200).json({ results });
  } catch (err) {
    log.error({ err: String(err) }, 'test-sandbox failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 4.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/test-sandbox.test.ts

Expected: 6 tests pass.

• [ ] Step 4.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 4.6: Commit

git add api-src/factory/test-sandbox.ts api-src/factory/__tests__/test-sandbox.test.ts
git commit -m "feat(factory): /api/factory/test-sandbox endpoint"

---

Task 5: /api/factory/test-real: real Algolia crawl_urls test

Files: - Create: api-src/factory/test-real.ts - Create: api-src/factory/__tests__/test-real.test.ts

POST {sessionId, pathGroupId}. Creates or reuses a test crawler factory-test-{sessionId}-{pathGroupId}, PATCHes the extractor + a temp test index algoliacentral_factory_test, calls crawler.crawlUrls(sampleUrls), polls until done, returns the extracted records.

• [ ] Step 5.1: Write failing tests

Create api-src/factory/__tests__/test-real.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockSaveSession = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    saveSession: mockSaveSession,
  })),
}));

const mockGetOrCreateCrawler = vi.fn();
const mockPatchConfig = vi.fn();
const mockCrawlUrls = vi.fn();
const mockGetStatus = vi.fn();
const mockGetRecords = vi.fn();

vi.mock('../../../lib/factory/crawler-client', () => ({
  CrawlerClient: vi.fn().mockImplementation(() => ({
    getOrCreateCrawler: mockGetOrCreateCrawler,
    patchConfig: mockPatchConfig,
    crawlUrls: mockCrawlUrls,
    getStatus: mockGetStatus,
    getRecords: mockGetRecords,
  })),
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { sessionId: 'sess_001', pathGroupId: 'pg_1' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const sessionWithBlueprint = () => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'configuring' as const,
  source_domain: 'x.com', source_url_root: 'https://x.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [
    {
      id: 'pg_1', pattern: '/blog/*', urlCount: 3,
      sampleUrls: ['https://x.com/blog/a', 'https://x.com/blog/b'],
      detectedDomain: 'marketing' as const,
      detectionConfidence: 0.9, detectionMethod: 'json-ld' as const,
      selected: true,
      blueprint: {
        indexName: 'algoliacentral_marketing',
        recordExtractor: '({ $ }) => [{ headline: $("h1").text() }]',
        crawlerConfig: { renderJavaScript: false, rateLimit: 8, pathsToMatch: ['/blog/*'] },
      },
    },
  ],
});

describe('POST /api/factory/test-real', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../test-real');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing pathGroupId with 400', async () => {
    const { default: handler } = await import('../test-real');
    const req = makeReq({ body: { sessionId: 's' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session not found', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../test-real');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 422 when blueprint missing', async () => {
    const noBp = sessionWithBlueprint();
    delete (noBp.pathGroups[0] as any).blueprint;
    mockGetSession.mockResolvedValue(noBp);
    const { default: handler } = await import('../test-real');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(422);
  });

  it('happy path — gets/creates crawler with deterministic name, patches, crawls, polls, returns records', async () => {
    mockGetSession.mockResolvedValue(sessionWithBlueprint());
    mockGetOrCreateCrawler.mockResolvedValue('crawler_test_xyz');
    mockPatchConfig.mockResolvedValue({ taskID: 1 });
    mockCrawlUrls.mockResolvedValue({ taskID: 2 });
    mockGetStatus
      .mockResolvedValueOnce({ reindexing: true })
      .mockResolvedValueOnce({ reindexing: false });
    mockGetRecords.mockResolvedValue([
      { url: 'https://x.com/blog/a', headline: 'A' },
      { url: 'https://x.com/blog/b', headline: 'B' },
    ]);
    mockSaveSession.mockResolvedValue(undefined);

    const { default: handler } = await import('../test-real');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(res._status).toBe(200);
    expect((res._body as any).records).toHaveLength(2);
    // Crawler name pattern must include sessionId + pathGroupId
    const createCallArgs = mockGetOrCreateCrawler.mock.calls[0][0];
    expect(createCallArgs.name).toContain('factory-test-sess_001-pg_1');
    expect(createCallArgs.indexName).toBe('algoliacentral_factory_test');
    // Sample URLs must be passed to crawlUrls
    expect(mockCrawlUrls).toHaveBeenCalledWith(
      'crawler_test_xyz',
      expect.arrayContaining(['https://x.com/blog/a', 'https://x.com/blog/b'])
    );
  });

  it('returns 500 when crawler client throws', async () => {
    mockGetSession.mockResolvedValue(sessionWithBlueprint());
    mockGetOrCreateCrawler.mockRejectedValue(new Error('crawler API down'));
    const { default: handler } = await import('../test-real');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 5.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/test-real.test.ts

Expected: FAIL.

• [ ] Step 5.3: Implement test-real.ts

Create api-src/factory/test-real.ts:

/**
 * POST /api/factory/test-real
 *
 * Body: { sessionId: string, pathGroupId: string }
 *
 * Pipeline:
 *   1. Resolve blueprint from session.
 *   2. getOrCreateCrawler({ name: 'factory-test-{sessionId}-{pathGroupId}',
 *                           indexName: 'algoliacentral_factory_test' }).
 *   3. patchConfig(crawlerId, { recordExtractor, indexName, ...crawlerConfig }).
 *   4. crawlUrls(crawlerId, sampleUrls).
 *   5. Poll getStatus until reindexing===false, with 30 min timeout.
 *   6. getRecords(crawlerId) — returns the records the crawler indexed.
 *   7. Persist realTestResults on session.pathGroups[i].blueprint.
 *   8. Return { records }.
 *
 * Decision: the test crawler is reused per pathGroup so the user can inspect
 * history in the Algolia dashboard. We do NOT delete it between attempts.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { CrawlerClient } from '../../lib/factory/crawler-client';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';

const RequestSchema = z.object({
  sessionId: z.string().min(1),
  pathGroupId: z.string().min(1),
});

const TEST_INDEX = 'algoliacentral_factory_test';
const POLL_INTERVAL_MS = 3000;
const POLL_TIMEOUT_MS = 30 * 60 * 1000; // 30 minutes

async function pollUntilDone(
  client: CrawlerClient,
  crawlerId: string,
  log: ReturnType<typeof createLogger>
): Promise<void> {
  const start = Date.now();
  while (Date.now() - start < POLL_TIMEOUT_MS) {
    const status = await client.getStatus(crawlerId);
    if (!status.reindexing) return;
    await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
  }
  log.warn({ crawlerId, elapsedMs: Date.now() - start }, 'poll timed out');
  throw new Error('test-real poll timeout');
}

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:test-real', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'POST') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );
  const client = new CrawlerClient(
    process.env.ALGOLIA_CRAWLER_USER_ID ?? '',
    process.env.ALGOLIA_CRAWLER_API_KEY ?? ''
  );

  try {
    const session = await store.getSession(parsed.sessionId);
    if (!session) {
      return void res.status(404).json({ error: 'Not Found', message: 'session not found' });
    }
    const groupIdx = session.pathGroups.findIndex((g) => g.id === parsed.pathGroupId);
    if (groupIdx === -1) {
      return void res.status(404).json({ error: 'Not Found', message: 'pathGroup not found' });
    }
    const group = session.pathGroups[groupIdx];
    if (!group.blueprint) {
      return void res.status(422).json({
        error: 'Unprocessable Entity',
        message: 'blueprint not yet generated — call /generate-extractor first',
      });
    }

    const crawlerName = `factory-test-${parsed.sessionId}-${parsed.pathGroupId}`;
    const crawlerId = await client.getOrCreateCrawler({
      name: crawlerName,
      indexName: TEST_INDEX,
    });

    await client.patchConfig(crawlerId, {
      indexName: TEST_INDEX,
      recordExtractor: group.blueprint.recordExtractor,
      renderJavaScript: group.blueprint.crawlerConfig.renderJavaScript,
      rateLimit: group.blueprint.crawlerConfig.rateLimit,
      pathsToMatch: group.blueprint.crawlerConfig.pathsToMatch,
    });

    await client.crawlUrls(crawlerId, group.sampleUrls);
    await pollUntilDone(client, crawlerId, log);

    const records = await client.getRecords(crawlerId);

    const updated = {
      ...session,
      updatedAt: Date.now(),
      pathGroups: session.pathGroups.map((g, i) =>
        i === groupIdx
          ? { ...g, blueprint: { ...(g.blueprint ?? {} as any), realTestResults: records } }
          : g
      ),
    };
    await store.saveSession(updated);

    log.info(
      { sessionId: parsed.sessionId, pathGroupId: parsed.pathGroupId, count: records.length, crawlerId },
      'test-real complete'
    );
    return void res.status(200).json({ records, crawlerId });
  } catch (err) {
    log.error({ err: String(err) }, 'test-real failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 5.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/test-real.test.ts

Expected: 6 tests pass.

• [ ] Step 5.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 5.6: Commit

git add api-src/factory/test-real.ts api-src/factory/__tests__/test-real.test.ts
git commit -m "feat(factory): /api/factory/test-real endpoint"

---

Task 6: /api/factory/create-crawler: full orchestrator

Files: - Create: api-src/factory/create-crawler.ts - Create: api-src/factory/__tests__/create-crawler.test.ts

POST {sessionId, pathGroupId}. Per master plan §7 Task 19, runs the full crawler-creation orchestrator:

1. IndexManager.ensureIndex(contentDomain): create + configure target index if missing
2. CrawlerClient.createCrawler({ name: 'algoliacentral-{site-domain}-{contentDomain}-{pathGroupId}' })
3. CrawlerClient.patchConfig(crawlerId, {...})
4. CrawlerClient.startReindex(crawlerId)
5. BlueprintStore.saveBlueprint(...)
6. Write recordExtractor JS to crawler-configs/{site-domain}/{contentDomain}-{pathGroupId}.js
7. Update session.pathGroups[i].blueprint.crawlerId + reindexTaskId
8. Return {crawlerId, reindexTaskId, indexName, blueprintId}

• [ ] Step 6.1: Write failing tests

Create api-src/factory/__tests__/create-crawler.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockSaveSession = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    saveSession: mockSaveSession,
  })),
}));

const mockSaveBlueprint = vi.fn();
vi.mock('../../../lib/factory/blueprint-store', () => ({
  BlueprintStore: vi.fn().mockImplementation(() => ({
    saveBlueprint: mockSaveBlueprint,
  })),
}));

const mockEnsureIndex = vi.fn();
vi.mock('../../../lib/factory/index-manager', () => ({
  IndexManager: vi.fn().mockImplementation(() => ({
    ensureIndex: mockEnsureIndex,
  })),
}));

const mockCreateCrawler = vi.fn();
const mockPatchConfig = vi.fn();
const mockStartReindex = vi.fn();
vi.mock('../../../lib/factory/crawler-client', () => ({
  CrawlerClient: vi.fn().mockImplementation(() => ({
    createCrawler: mockCreateCrawler,
    patchConfig: mockPatchConfig,
    startReindex: mockStartReindex,
  })),
}));

const mockWriteFile = vi.fn();
vi.mock('node:fs/promises', () => ({
  default: { writeFile: mockWriteFile, mkdir: vi.fn().mockResolvedValue(undefined) },
  writeFile: mockWriteFile,
  mkdir: vi.fn().mockResolvedValue(undefined),
}));

vi.mock('../../../lib/factory/dss', () => ({
  getDss: (domain: string) => ({
    indexName: `algoliacentral_${domain}`,
    schemaOrgTypes: ['BlogPosting'],
    algoliaConfig: {
      searchableAttributes: ['headline'],
      customRanking: ['desc(datePublished)'],
      attributesForFaceting: ['articleSection'],
    },
    description: 'mock',
  }),
  dssBySchemaOrgType: () => null,
  listContentDomains: () => ['marketing'],
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'POST',
  headers: { origin: 'http://localhost:5173' },
  body: { sessionId: 'sess_001', pathGroupId: 'pg_1' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const sessionReady = () => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'configuring' as const,
  source_domain: 'algolia.com', source_url_root: 'https://www.algolia.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [
    {
      id: 'pg_1', pattern: '/blog/*', urlCount: 3,
      sampleUrls: ['https://www.algolia.com/blog/a'],
      detectedDomain: 'marketing' as const,
      detectionConfidence: 0.9, detectionMethod: 'json-ld' as const,
      selected: true,
      blueprint: {
        indexName: 'algoliacentral_marketing',
        recordExtractor: '({ $ }) => [{ headline: $("h1").text() }]',
        crawlerConfig: { renderJavaScript: false, rateLimit: 8, pathsToMatch: ['/blog/*'] },
      },
    },
  ],
});

describe('POST /api/factory/create-crawler', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects GET with 405', async () => {
    const { default: handler } = await import('../create-crawler');
    const req = makeReq({ method: 'GET' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing fields with 400', async () => {
    const { default: handler } = await import('../create-crawler');
    const req = makeReq({ body: {} }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session not found', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../create-crawler');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 422 when blueprint missing', async () => {
    const noBp = sessionReady();
    delete (noBp.pathGroups[0] as any).blueprint;
    mockGetSession.mockResolvedValue(noBp);
    const { default: handler } = await import('../create-crawler');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(422);
  });

  it('happy path — runs ensureIndex → createCrawler → patchConfig → startReindex → saveBlueprint → writeFile, returns ids', async () => {
    mockGetSession.mockResolvedValue(sessionReady());
    mockEnsureIndex.mockResolvedValue(undefined);
    // Spec 06: createCrawler returns `{ crawlerId: string }`.
    mockCreateCrawler.mockResolvedValue({ crawlerId: 'crawler_xyz' });
    // Spec 06: patchConfig returns void (no body inspection in handler).
    mockPatchConfig.mockResolvedValue(undefined);
    // Spec 06: startReindex returns the parsed `taskId` string directly.
    mockStartReindex.mockResolvedValue('reindex_99');
    mockSaveBlueprint.mockResolvedValue(undefined);
    mockSaveSession.mockResolvedValue(undefined);
    mockWriteFile.mockResolvedValue(undefined);

    const { default: handler } = await import('../create-crawler');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);

    expect(res._status).toBe(200);
    expect((res._body as any).crawlerId).toBe('crawler_xyz');
    expect((res._body as any).reindexTaskId).toBe('reindex_99');
    expect((res._body as any).indexName).toBe('algoliacentral_marketing');
    expect((res._body as any).blueprintId).toBe('crawler_xyz');

    // Order assertions — ensureIndex before createCrawler before saveBlueprint
    expect(mockEnsureIndex).toHaveBeenCalledTimes(1);
    expect(mockCreateCrawler).toHaveBeenCalledTimes(1);
    expect(mockPatchConfig).toHaveBeenCalledTimes(1);
    expect(mockStartReindex).toHaveBeenCalledTimes(1);
    expect(mockSaveBlueprint).toHaveBeenCalledTimes(1);
    // recordExtractor JS persisted to crawler-configs/{domain}/{contentDomain}-{pathGroupId}.js
    expect(mockWriteFile).toHaveBeenCalledTimes(1);
    const filePathArg = mockWriteFile.mock.calls[0][0] as string;
    expect(filePathArg).toContain('crawler-configs');
    expect(filePathArg).toContain('algolia.com');
    expect(filePathArg).toContain('marketing-pg_1.js');

    // saveBlueprint payload includes agent_slot=null, content_domain, path_groups
    const bpArg = mockSaveBlueprint.mock.calls[0][0];
    expect(bpArg.agent_slot).toBeNull();
    expect(bpArg.content_domain).toBe('marketing');
    expect(bpArg.path_groups).toContain('/blog/*');

    // Crawler name follows algoliacentral-{site-domain}-{contentDomain}-{pathGroupId}
    const createArg = mockCreateCrawler.mock.calls[0][0];
    expect(createArg.name).toBe('algoliacentral-algolia.com-marketing-pg_1');
  });

  it('returns 500 when ensureIndex throws', async () => {
    mockGetSession.mockResolvedValue(sessionReady());
    mockEnsureIndex.mockRejectedValue(new Error('algolia 503'));
    const { default: handler } = await import('../create-crawler');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 6.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/create-crawler.test.ts

Expected: FAIL.

• [ ] Step 6.3: Implement create-crawler.ts

Create api-src/factory/create-crawler.ts:

/**
 * POST /api/factory/create-crawler
 *
 * Body: { sessionId: string, pathGroupId: string }
 *
 * Orchestrator (per master plan §7 Task 19):
 *   1. IndexManager.ensureIndex(contentDomain) — idempotent, applies DSS settings
 *   2. CrawlerClient.createCrawler({ name, config: { indexName, ...crawlerConfig } })
 *      → returns `{ crawlerId: string }` per Spec 06 CreateCrawlerResponseSchema
 *   3. CrawlerClient.patchConfig(crawlerId, ...)
 *   4. CrawlerClient.startReindex(crawlerId) → returns `taskId: string` per Spec 06
 *   5. BlueprintStore.saveBlueprint(...)
 *   6. fs.writeFile(`crawler-configs/{site-domain}/{contentDomain}-{pathGroupId}.js`, recordExtractor)
 *   7. SessionStore.saveSession(updated session with crawlerId + reindexTaskId)
 *   8. Return { crawlerId, reindexTaskId, indexName, blueprintId }
 *
 * The blueprintId == crawlerId (one-to-one).
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import path from 'node:path';
import fsPromises from 'node:fs/promises';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { BlueprintStore } from '../../lib/factory/blueprint-store';
import { IndexManager } from '../../lib/factory/index-manager';
import { CrawlerClient } from '../../lib/factory/crawler-client';
import { getDss } from '../../lib/factory/dss';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';
import type { Blueprint } from '../../lib/factory/types';

const RequestSchema = z.object({
  sessionId: z.string().min(1),
  pathGroupId: z.string().min(1),
});

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:create-crawler', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'POST') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof RequestSchema>;
  try {
    parsed = RequestSchema.parse(req.body);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );
  const blueprints = new BlueprintStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );
  const indexManager = new IndexManager(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );
  const crawler = new CrawlerClient(
    process.env.ALGOLIA_CRAWLER_USER_ID ?? '',
    process.env.ALGOLIA_CRAWLER_API_KEY ?? ''
  );

  try {
    const session = await store.getSession(parsed.sessionId);
    if (!session) {
      return void res.status(404).json({ error: 'Not Found', message: 'session not found' });
    }
    const groupIdx = session.pathGroups.findIndex((g) => g.id === parsed.pathGroupId);
    if (groupIdx === -1) {
      return void res.status(404).json({ error: 'Not Found', message: 'pathGroup not found' });
    }
    const group = session.pathGroups[groupIdx];
    if (!group.blueprint || !group.detectedDomain) {
      return void res.status(422).json({
        error: 'Unprocessable Entity',
        message: 'blueprint not yet generated — call /generate-extractor first',
      });
    }

    const dssEntry = getDss(group.detectedDomain);
    const siteDomain = session.source_domain;
    const contentDomain = group.detectedDomain;
    const pathGroupId = group.id;
    const crawlerName = `algoliacentral-${siteDomain}-${contentDomain}-${pathGroupId}`;

    // 1. ensure index exists with DSS-derived settings
    await indexManager.ensureIndex(contentDomain);

    // 2. createCrawler — B1 fix: Spec 06's CreateCrawlerInput is
    //    `{ name, config: Record<string, unknown> }`. The crawler config block
    //    (indexName + the rest of the blueprint's crawlerConfig) is nested
    //    under `config`, not raised to the top level. The return type is
    //    `{ crawlerId: string }` — destructure to get the string id.
    const { crawlerId } = await crawler.createCrawler({
      name: crawlerName,
      config: {
        indexName: dssEntry.indexName,
        recordExtractor: group.blueprint.recordExtractor,
        renderJavaScript: group.blueprint.crawlerConfig.renderJavaScript,
        rateLimit: group.blueprint.crawlerConfig.rateLimit,
        pathsToMatch: group.blueprint.crawlerConfig.pathsToMatch,
        startUrls: [session.source_url_root],
      },
    });

    // 3. patchConfig — keep the patch in sync with the initial config block
    //    so PATCH-after-create stays an idempotent re-assertion (any drift
    //    here is a real edit; today there is none).
    await crawler.patchConfig(crawlerId, {
      indexName: dssEntry.indexName,
      recordExtractor: group.blueprint.recordExtractor,
      renderJavaScript: group.blueprint.crawlerConfig.renderJavaScript,
      rateLimit: group.blueprint.crawlerConfig.rateLimit,
      pathsToMatch: group.blueprint.crawlerConfig.pathsToMatch,
      startUrls: [session.source_url_root],
    });

    // 4. startReindex — B2 fix: Spec 06's TaskResponseSchema uses `taskId`
    //    (lowercase), and `startReindex(crawlerId)` returns the parsed string
    //    directly (not the wrapping object). Bind to a local name without
    //    destructuring an absent field.
    const reindexTaskId = await crawler.startReindex(crawlerId);

    // 5. saveBlueprint
    const blueprint: Blueprint = {
      objectID: crawlerId,
      content_domain: contentDomain,
      index_name: dssEntry.indexName,
      source_domain: siteDomain,
      path_groups: [group.pattern],
      schema_org_types: dssEntry.schemaOrgTypes.slice(),
      recordExtractor_path: `crawler-configs/${siteDomain}/${contentDomain}-${pathGroupId}.js`,
      algolia_config: {
        searchableAttributes: dssEntry.algoliaConfig.searchableAttributes.slice(),
        customRanking: dssEntry.algoliaConfig.customRanking.slice(),
        attributesForFaceting: dssEntry.algoliaConfig.attributesForFaceting.slice(),
      },
      agent_slot: null,
      created_at: Date.now(),
      status: 'live',
    };
    await blueprints.saveBlueprint(blueprint);

    // 6. write recordExtractor to disk under version control
    const dirPath = path.join(process.cwd(), 'crawler-configs', siteDomain);
    const filePath = path.join(dirPath, `${contentDomain}-${pathGroupId}.js`);
    await fsPromises.mkdir(dirPath, { recursive: true });
    await fsPromises.writeFile(filePath, group.blueprint.recordExtractor, 'utf8');

    // 7. update session
    const updatedSession = {
      ...session,
      status: 'crawling' as const,
      updatedAt: Date.now(),
      pathGroups: session.pathGroups.map((g, i) =>
        i === groupIdx
          ? {
              ...g,
              blueprint: {
                ...(g.blueprint ?? {} as any),
                crawlerId,
                reindexTaskId, // Spec 06 startReindex returns string; no String() coercion needed.
              },
            }
          : g
      ),
    };
    await store.saveSession(updatedSession);

    log.info(
      { crawlerId, blueprintId: crawlerId, reindexTaskId, contentDomain, siteDomain },
      'crawler created'
    );

    return void res.status(200).json({
      crawlerId,
      reindexTaskId,
      indexName: dssEntry.indexName,
      blueprintId: crawlerId,
    });
  } catch (err) {
    log.error({ err: String(err) }, 'create-crawler failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 6.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/create-crawler.test.ts

Expected: 6 tests pass.

• [ ] Step 6.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 6.6: Commit

git add api-src/factory/create-crawler.ts api-src/factory/__tests__/create-crawler.test.ts
git commit -m "feat(factory): /api/factory/create-crawler orchestrator endpoint"

---

Task 7: /api/factory/crawl-progress: SSE polling endpoint

Files: - Create: api-src/factory/crawl-progress.ts - Create: api-src/factory/__tests__/crawl-progress.test.ts

GET ?crawlerId=.... SSE poll loop with 3s interval. Emits {type:'progress', stats} while reindexing, {type:'done', stats} once status reports reindexing===false. Hard timeout at 30 minutes → emits {type:'timeout'} and ends.

• [ ] Step 7.1: Write failing tests

Create api-src/factory/__tests__/crawl-progress.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

const mockGetStatus = vi.fn();
const mockGetStats = vi.fn();
vi.mock('../../../lib/factory/crawler-client', () => ({
  CrawlerClient: vi.fn().mockImplementation(() => ({
    getStatus: mockGetStatus,
    getStats: mockGetStats,
  })),
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'GET',
  url: '/api/factory/crawl-progress?crawlerId=crawler_xyz',
  query: { crawlerId: 'crawler_xyz' },
  headers: { origin: 'http://localhost:5173' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const writes: string[] = [];
  let ended = false;
  const res = {
    _status: 200,
    _writes: writes,
    _ended: () => ended,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    write(s: string) { writes.push(s); return true; },
    end() { ended = true; return res; },
    json(b: unknown) { res._writes.push(JSON.stringify(b)); return res; },
    send(b: unknown) { res._writes.push(typeof b === 'string' ? b : JSON.stringify(b)); return res; },
  };
  return res;
};

const parseSseFrames = (writes: string[]): unknown[] =>
  writes
    .filter((s) => s.startsWith('data: '))
    .map((s) => JSON.parse(s.slice('data: '.length).trim()));

describe('GET /api/factory/crawl-progress', () => {
  beforeEach(() => {
    vi.useFakeTimers();
    vi.clearAllMocks();
  });
  afterEach(() => vi.useRealTimers());

  it('rejects POST with 405', async () => {
    const { default: handler } = await import('../crawl-progress');
    const req = makeReq({ method: 'POST' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing crawlerId with 400', async () => {
    const { default: handler } = await import('../crawl-progress');
    const req = makeReq({ query: {} }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('emits progress events while reindexing then done when not reindexing', async () => {
    mockGetStatus
      .mockResolvedValueOnce({ reindexing: true })
      .mockResolvedValueOnce({ reindexing: true })
      .mockResolvedValueOnce({ reindexing: false });
    mockGetStats
      .mockResolvedValue({ urlsCrawled: 10, urlsIndexed: 10, errors: 0 });

    const { default: handler } = await import('../crawl-progress');
    const req = makeReq() as any;
    const res = makeRes() as any;
    const p = handler(req, res);
    // Advance through 3 polls
    await vi.advanceTimersByTimeAsync(3000);
    await vi.advanceTimersByTimeAsync(3000);
    await vi.advanceTimersByTimeAsync(3000);
    await p;

    const events = parseSseFrames(res._writes);
    const progressEvents = events.filter((e: any) => e.type === 'progress');
    const doneEvents = events.filter((e: any) => e.type === 'done');
    expect(progressEvents.length).toBeGreaterThanOrEqual(2);
    expect(doneEvents).toHaveLength(1);
    expect((doneEvents[0] as any).stats.urlsIndexed).toBe(10);
  });

  it('emits timeout after 30 minutes if reindex never completes', async () => {
    mockGetStatus.mockResolvedValue({ reindexing: true });
    mockGetStats.mockResolvedValue({ urlsCrawled: 1 });

    const { default: handler } = await import('../crawl-progress');
    const req = makeReq() as any;
    const res = makeRes() as any;
    const p = handler(req, res);
    // Fast-forward 30 minutes + 1 tick
    await vi.advanceTimersByTimeAsync(30 * 60 * 1000 + 4000);
    await p;

    const events = parseSseFrames(res._writes);
    const timeoutEvents = events.filter((e: any) => e.type === 'timeout');
    expect(timeoutEvents).toHaveLength(1);
  });

  it('SSE frames are written in canonical data: <json>\\n\\n format', async () => {
    mockGetStatus.mockResolvedValueOnce({ reindexing: false });
    mockGetStats.mockResolvedValueOnce({ urlsCrawled: 0 });
    const { default: handler } = await import('../crawl-progress');
    const req = makeReq() as any;
    const res = makeRes() as any;
    const p = handler(req, res);
    await vi.advanceTimersByTimeAsync(50);
    await p;
    for (const f of res._writes) {
      expect(f.startsWith('data: ')).toBe(true);
      expect(f.endsWith('\n\n')).toBe(true);
    }
  });
});

• [ ] Step 7.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/crawl-progress.test.ts

Expected: FAIL.

• [ ] Step 7.3: Implement crawl-progress.ts

Create api-src/factory/crawl-progress.ts:

/**
 * GET /api/factory/crawl-progress?crawlerId=...
 *
 * Server-Sent Events poll loop. Every 3 seconds:
 *   data: {type:'progress', stats}     // while reindexing===true
 *   data: {type:'done', stats}         // once reindexing===false
 *   data: {type:'timeout'}             // hard floor at 30 minutes
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { CrawlerClient } from '../../lib/factory/crawler-client';
import {
  applyStandardHeaders,
  applySseHeaders,
  sseWriter,
  zodErrorTo400,
} from './_shared';

const QuerySchema = z.object({ crawlerId: z.string().min(1) });

const POLL_INTERVAL_MS = 3000;
const POLL_TIMEOUT_MS = 30 * 60 * 1000;

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:crawl-progress', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'GET') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof QuerySchema>;
  try {
    parsed = QuerySchema.parse(req.query);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    log.error({ err: String(e) }, 'unexpected validation error');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  applySseHeaders(req, res);
  const emit = sseWriter(res);

  const client = new CrawlerClient(
    process.env.ALGOLIA_CRAWLER_USER_ID ?? '',
    process.env.ALGOLIA_CRAWLER_API_KEY ?? ''
  );

  const start = Date.now();

  try {
    while (true) {
      if (Date.now() - start > POLL_TIMEOUT_MS) {
        emit({ type: 'timeout' });
        log.warn({ crawlerId: parsed.crawlerId }, 'poll timed out');
        break;
      }

      const status = await client.getStatus(parsed.crawlerId);
      const stats = await client.getStats(parsed.crawlerId);
      if (status.reindexing) {
        emit({ type: 'progress', stats });
      } else {
        emit({ type: 'done', stats });
        break;
      }
      await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
    }
    res.end();
  } catch (err) {
    log.error({ err: String(err) }, 'crawl-progress failed');
    try {
      emit({ type: 'log', level: 'error', message: `progress failed: ${String(err)}` });
      res.end();
    } catch {
      // socket closed
    }
  }
}

• [ ] Step 7.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/crawl-progress.test.ts

Expected: 5 tests pass.

• [ ] Step 7.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 7.6: Commit

git add api-src/factory/crawl-progress.ts api-src/factory/__tests__/crawl-progress.test.ts
git commit -m "feat(factory): /api/factory/crawl-progress SSE endpoint"

---

Task 8: /api/factory/session: GET (read) + PUT (upsert)

Files: - Create: api-src/factory/session.ts - Create: api-src/factory/__tests__/session.test.ts

GET ?id=<sessionId> reads main session record (404 if missing). PUT body=FactorySession upserts.

• [ ] Step 8.1: Write failing tests

Create api-src/factory/__tests__/session.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSession = vi.fn();
const mockSaveSession = vi.fn();
vi.mock('../../../lib/factory/session-store', () => ({
  SessionStore: vi.fn().mockImplementation(() => ({
    getSession: mockGetSession,
    saveSession: mockSaveSession,
  })),
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'GET',
  url: '/api/factory/session?id=sess_001',
  query: { id: 'sess_001' },
  headers: { origin: 'http://localhost:5173' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const minimalSession = () => ({
  objectID: 'sess_001',
  record_type: 'session' as const,
  status: 'discovering' as const,
  source_domain: 'x.com',
  source_url_root: 'https://x.com',
  createdAt: 1, updatedAt: 1,
  discovery: { sitemapUrls: [], urlCount: 0, urlShardCount: 0, schemaOrgCoverage: 0 },
  pathGroups: [],
});

describe('GET /api/factory/session', () => {
  beforeEach(() => vi.clearAllMocks());

  it('returns 200 for OPTIONS preflight', async () => {
    const { default: handler } = await import('../session');
    const req = makeReq({ method: 'OPTIONS' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
  });

  it('rejects POST with 405', async () => {
    const { default: handler } = await import('../session');
    const req = makeReq({ method: 'POST' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('rejects missing id query with 400', async () => {
    const { default: handler } = await import('../session');
    const req = makeReq({ query: {} }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 404 when session not found', async () => {
    mockGetSession.mockResolvedValue(null);
    const { default: handler } = await import('../session');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(404);
  });

  it('returns 200 with session payload when found', async () => {
    mockGetSession.mockResolvedValue(minimalSession());
    const { default: handler } = await import('../session');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
    expect((res._body as any).objectID).toBe('sess_001');
  });

  it('returns 500 when store throws on read', async () => {
    mockGetSession.mockRejectedValue(new Error('algolia 503'));
    const { default: handler } = await import('../session');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

describe('PUT /api/factory/session', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects PUT with malformed body — 400', async () => {
    const { default: handler } = await import('../session');
    const req = makeReq({ method: 'PUT', body: { not: 'a session' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('upserts and returns 200', async () => {
    mockSaveSession.mockResolvedValue(undefined);
    const { default: handler } = await import('../session');
    const req = makeReq({ method: 'PUT', body: minimalSession() }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
    expect(mockSaveSession).toHaveBeenCalledTimes(1);
  });

  it('returns 500 when store throws on save', async () => {
    mockSaveSession.mockRejectedValue(new Error('algolia 503'));
    const { default: handler } = await import('../session');
    const req = makeReq({ method: 'PUT', body: minimalSession() }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 8.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/session.test.ts

Expected: FAIL.

• [ ] Step 8.3: Implement session.ts

Create api-src/factory/session.ts:

/**
 * GET /api/factory/session?id=<sessionId>
 * PUT /api/factory/session     body=FactorySession
 *
 * Thin CRUD over the SessionStore. The PUT body is fully validated by
 * FactorySessionSchema (delegated through SessionStore.saveSession).
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { SessionStore } from '../../lib/factory/session-store';
import { FactorySessionSchema } from '../../lib/factory/types';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';

const GetQuerySchema = z.object({ id: z.string().min(1) });

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:session', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');

  const store = new SessionStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  if (req.method === 'GET') {
    let parsed: z.infer<typeof GetQuerySchema>;
    try {
      parsed = GetQuerySchema.parse(req.query);
    } catch (e) {
      const mapped = zodErrorTo400(e);
      if (mapped) return void res.status(mapped.status).json(mapped.body);
      return void res.status(500).json({ error: 'Internal Server Error' });
    }
    try {
      const session = await store.getSession(parsed.id);
      if (!session) {
        return void res.status(404).json({ error: 'Not Found' });
      }
      return void res.status(200).json(session);
    } catch (err) {
      log.error({ err: String(err), id: parsed.id }, 'GET session failed');
      return void res.status(500).json({ error: 'Internal Server Error' });
    }
  }

  if (req.method === 'PUT') {
    try {
      const session = FactorySessionSchema.parse(req.body);
      await store.saveSession(session);
      return void res.status(200).json({ ok: true, objectID: session.objectID });
    } catch (e) {
      const mapped = zodErrorTo400(e);
      if (mapped) return void res.status(mapped.status).json(mapped.body);
      log.error({ err: String(e) }, 'PUT session failed');
      return void res.status(500).json({ error: 'Internal Server Error' });
    }
  }

  return void res.status(405).json({ error: 'Method Not Allowed' });
}

• [ ] Step 8.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/session.test.ts

Expected: 9 tests pass.

• [ ] Step 8.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 8.6: Commit

git add api-src/factory/session.ts api-src/factory/__tests__/session.test.ts
git commit -m "feat(factory): /api/factory/session GET + PUT"

---

Task 9: /api/factory/blueprints: list (optional content_domain filter)

Files: - Create: api-src/factory/blueprints.ts - Create: api-src/factory/__tests__/blueprints.test.ts

GET ?content_domain=marketing (optional) → {blueprints: Blueprint[]}.

• [ ] Step 9.1: Write failing tests

Create api-src/factory/__tests__/blueprints.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockListAll = vi.fn();
const mockListByDomain = vi.fn();
vi.mock('../../../lib/factory/blueprint-store', () => ({
  BlueprintStore: vi.fn().mockImplementation(() => ({
    listAllBlueprints: mockListAll,
    listBlueprintsByContentDomain: mockListByDomain,
  })),
}));

const makeReq = (overrides: Record<string, unknown> = {}) => ({
  method: 'GET',
  url: '/api/factory/blueprints',
  query: {},
  headers: { origin: 'http://localhost:5173' },
  socket: { setTimeout: vi.fn(), setNoDelay: vi.fn() },
  ...overrides,
});

const makeRes = () => {
  const headers: Record<string, string> = {};
  const res = {
    _status: 200,
    _body: null as unknown,
    _headers: headers,
    status(code: number) { res._status = code; return res; },
    json(b: unknown) { res._body = b; return res; },
    setHeader(k: string, v: string) { headers[k] = v; return res; },
    send(b: unknown) { res._body = b; return res; },
  };
  return res;
};

const blueprint = (domain: string = 'marketing') => ({
  objectID: `crawler_${domain}`,
  content_domain: domain,
  index_name: `algoliacentral_${domain}`,
  source_domain: 'x.com',
  path_groups: ['/blog/*'],
  schema_org_types: ['BlogPosting'],
  recordExtractor_path: 'crawler-configs/x.com/marketing-pg_1.js',
  algolia_config: {
    searchableAttributes: ['headline'],
    customRanking: ['desc(datePublished)'],
    attributesForFaceting: ['articleSection'],
  },
  agent_slot: null,
  created_at: 1,
  status: 'live' as const,
});

describe('GET /api/factory/blueprints', () => {
  beforeEach(() => vi.clearAllMocks());

  it('rejects POST with 405', async () => {
    const { default: handler } = await import('../blueprints');
    const req = makeReq({ method: 'POST' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(405);
  });

  it('returns 200 for OPTIONS preflight', async () => {
    const { default: handler } = await import('../blueprints');
    const req = makeReq({ method: 'OPTIONS' }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
  });

  it('lists all when no filter is supplied', async () => {
    mockListAll.mockResolvedValue([blueprint('marketing'), blueprint('support')]);
    const { default: handler } = await import('../blueprints');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
    expect((res._body as any).blueprints).toHaveLength(2);
    expect(mockListAll).toHaveBeenCalledTimes(1);
  });

  it('filters by content_domain when supplied', async () => {
    mockListByDomain.mockResolvedValue([blueprint('marketing')]);
    const { default: handler } = await import('../blueprints');
    const req = makeReq({ query: { content_domain: 'marketing' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(200);
    expect((res._body as any).blueprints).toHaveLength(1);
    expect(mockListByDomain).toHaveBeenCalledWith('marketing');
  });

  it('rejects invalid content_domain with 400', async () => {
    const { default: handler } = await import('../blueprints');
    const req = makeReq({ query: { content_domain: 'made-up-domain' } }) as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(400);
  });

  it('returns 500 when store throws', async () => {
    mockListAll.mockRejectedValue(new Error('algolia 503'));
    const { default: handler } = await import('../blueprints');
    const req = makeReq() as any;
    const res = makeRes() as any;
    await handler(req, res);
    expect(res._status).toBe(500);
  });
});

• [ ] Step 9.2: Run, expect FAIL

npx vitest run api-src/factory/__tests__/blueprints.test.ts

Expected: FAIL.

• [ ] Step 9.3: Implement blueprints.ts

Create api-src/factory/blueprints.ts:

/**
 * GET /api/factory/blueprints[?content_domain=<domain>]
 *
 * Returns { blueprints: Blueprint[] }. With no query, returns every blueprint
 * in algoliacentral_factory_blueprints (capped at 1000 by the store). With
 * content_domain set, returns only blueprints whose content_domain matches.
 */

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { z } from 'zod';
import { v4 as uuidv4 } from 'uuid';
import { createLogger } from '../../lib/utils/logger';
import { BlueprintStore } from '../../lib/factory/blueprint-store';
import { applyStandardHeaders, zodErrorTo400 } from './_shared';

const ContentDomainEnum = z.enum([
  'marketing',
  'support',
  'education',
  'technical',
  'customer-stories',
  'product-catalog',
  'events',
  'legal',
  'social',
]);

const QuerySchema = z.object({
  content_domain: ContentDomainEnum.optional(),
});

export default async function handler(req: VercelRequest, res: VercelResponse) {
  const requestId = uuidv4();
  const log = createLogger('factory:api:blueprints', requestId);
  const origin = (req.headers.origin as string) ?? null;

  applyStandardHeaders(origin, res);

  if (req.method === 'OPTIONS') return void res.status(200).send('ok');
  if (req.method !== 'GET') return void res.status(405).json({ error: 'Method Not Allowed' });

  let parsed: z.infer<typeof QuerySchema>;
  try {
    parsed = QuerySchema.parse(req.query);
  } catch (e) {
    const mapped = zodErrorTo400(e);
    if (mapped) return void res.status(mapped.status).json(mapped.body);
    return void res.status(500).json({ error: 'Internal Server Error' });
  }

  const store = new BlueprintStore(
    process.env.ALGOLIA_APP_ID ?? '',
    process.env.ALGOLIA_ADMIN_API_KEY ?? ''
  );

  try {
    const blueprints = parsed.content_domain
      ? await store.listBlueprintsByContentDomain(parsed.content_domain)
      : await store.listAllBlueprints();
    return void res.status(200).json({ blueprints });
  } catch (err) {
    log.error({ err: String(err), filter: parsed.content_domain ?? null }, 'list blueprints failed');
    return void res.status(500).json({ error: 'Internal Server Error' });
  }
}

• [ ] Step 9.4: Run, expect PASS

npx vitest run api-src/factory/__tests__/blueprints.test.ts

Expected: 6 tests pass.

• [ ] Step 9.5: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

• [ ] Step 9.6: Commit

git add api-src/factory/blueprints.ts api-src/factory/__tests__/blueprints.test.ts
git commit -m "feat(factory): /api/factory/blueprints list endpoint with optional filter"

---

Task 10: Bundler: register api-src/factory/*.ts entry points

Files: - Modify: scripts/bundle-api.mjs

The bundler in Spec 13 will run node scripts/bundle-api.mjs to bundle every api-src/factory/*.ts to api/factory/*.mjs. Do the registration here in Spec 08 so the handlers are deployable as soon as they're written; Spec 13 will own the smoke test that calls the bundles end-to-end.

• [ ] Step 10.1: Inspect the existing bundler entry-point list

grep -n "api-src" scripts/bundle-api.mjs | head -20

Find the array (or glob) that lists the input files. The existing top-level entries are api-src/search.ts, api-src/prepare-dossier.ts, api-src/health.ts, api-src/diag.ts.

• [ ] Step 10.2: Add the eight factory entries

Open scripts/bundle-api.mjs. Locate the entryPoints (or equivalent) array. Add the new entries:

// Inside the existing entryPoints declaration:
const entryPoints = [
  'api-src/search.ts',
  'api-src/prepare-dossier.ts',
  'api-src/health.ts',
  'api-src/diag.ts',
  // Crawler Factory endpoints (Spec 08)
  'api-src/factory/discover.ts',
  'api-src/factory/sample.ts',
  'api-src/factory/generate-extractor.ts',
  'api-src/factory/test-sandbox.ts',
  'api-src/factory/test-real.ts',
  'api-src/factory/create-crawler.ts',
  'api-src/factory/crawl-progress.ts',
  'api-src/factory/session.ts',
  'api-src/factory/blueprints.ts',
];

If the bundler script uses an esbuild config object with outdir instead of explicit per-file builds, ensure outdir preserves the directory structure (api/factory/discover.mjs, etc.) by passing outbase: 'api-src'.

• [ ] Step 10.3: Run the bundler

node scripts/bundle-api.mjs

Expected: completes cleanly. New files appear at api/factory/discover.mjs, api/factory/sample.mjs, api/factory/generate-extractor.mjs, api/factory/test-sandbox.mjs, api/factory/test-real.mjs, api/factory/create-crawler.mjs, api/factory/crawl-progress.mjs, api/factory/session.mjs, api/factory/blueprints.mjs.

• [ ] Step 10.4: Verify each bundle is non-empty and exports a default handler

ls -la api/factory/*.mjs
node -e "import('./api/factory/discover.mjs').then(m => console.log(typeof m.default))"

Expected: 9 .mjs files; the import prints function.

• [ ] Step 10.5: Run full project test suite + tsc

npx vitest run && npx tsc --noEmit

Expected: all tests GREEN, tsc clean.

• [ ] Step 10.6: Commit

git add scripts/bundle-api.mjs api/factory/
git commit -m "chore(factory): register api-src/factory/* in bundler entry points"

---

Task 11: Cluster validation: full regression + smoke

Files: - (Read-only verification)

• [ ] Step 11.1: Run the full project test suite

npx vitest run

Expected: baseline factory tests (Specs 01–07) + new Spec 08 tests, all GREEN. Approximate new tests: 7 (_shared) + 10 (discover) + 7 (sample) + 7 (generate-extractor) + 6 (test-sandbox) + 6 (test-real) + 6 (create-crawler) + 5 (crawl-progress) + 9 (session) + 6 (blueprints) = 69 new tests.

• [ ] Step 11.2: Run full tsc

npx tsc --noEmit

Expected: clean.

• [ ] Step 11.3: Manual handler import smoke

node -e "Promise.all([
  'discover','sample','generate-extractor','test-sandbox','test-real',
  'create-crawler','crawl-progress','session','blueprints'
].map(n => import('./api/factory/' + n + '.mjs').then(m => [n, typeof m.default]))).then(console.log)"

Expected: each entry shows [<name>, 'function']: every bundle exports a default handler.

• [ ] Step 11.4: Mark Spec 08 done in Status.md

In the vault Projects/Crawler-Factory/Status.md:

- | Cluster 8 (Backend API) | ⏸ | 8 endpoints — discover, sample, generate-extractor, test-sandbox, test-real, create-crawler, crawl-progress, session+blueprints |
+ | Cluster 8 (Backend API) | ✅ | 8 endpoints — discover, sample, generate-extractor, test-sandbox, test-real, create-crawler, crawl-progress, session+blueprints |

• [ ] Step 11.5: Push branch (only if at a clean checkpoint and the user has authorized)

# Optional — only after explicit user authorization
git push origin rc3-phoenix

---

Acceptance criteria: Spec 08 done means:

1. ✅ api-src/factory/_shared.ts exports getCorsHeaders, applyStandardHeaders, applySseHeaders, sseWriter, zodErrorTo400: all unit-tested.
2. ✅ All 8 endpoint files exist under api-src/factory/ and follow the Vercel (req, res) => Promise<void> default-export contract: - discover.ts (POST, SSE) - sample.ts (POST) - generate-extractor.ts (POST) - test-sandbox.ts (POST) - test-real.ts (POST) - create-crawler.ts (POST) - crawl-progress.ts (GET, SSE) - session.ts (GET + PUT) - blueprints.ts (GET)
3. ✅ Every endpoint validates its request body or query with a zod schema and returns 400 on malformed input.
4. ✅ Every endpoint returns 405 on disallowed methods, 200 on OPTIONS preflight, and emits the canonical CORS + security headers.
5. ✅ Every endpoint returns 500 on internal error and logs the error before returning.
6. ✅ discover.ts writes initial session with status='discovering' BEFORE walking the sitemap (per §19a). It emits the canonical event types log, urls, pathGroup, classified, done. URLs stream in 1000-URL batches with no cap.
7. ✅ crawl-progress.ts polls getStatus+getStats every 3s, emits progress/done/timeout. Hard timeout at 30 minutes.
8. ✅ Both SSE endpoints write frames as data: <json>\n\n and set the canonical SSE headers from api-src/search.ts:220-226.
9. ✅ create-crawler.ts runs the eight-step orchestrator (ensureIndex → createCrawler → patchConfig → startReindex → saveBlueprint → writeFile → updateSession → return). Crawler name follows algoliacentral-{site-domain}-{contentDomain}-{pathGroupId}. recordExtractor JS persisted to crawler-configs/{site-domain}/{contentDomain}-{pathGroupId}.js. Blueprint payload has agent_slot=null.
10. ✅ bundle-api.mjs registers all 9 entry points (_shared.ts is consumed internally: only the 9 handler files become .mjs bundles); api/factory/*.mjs exists for each handler.
11. ✅ Approximately 69 new vitest tests pass; full project suite remains GREEN; tsc --noEmit clean.
12. ✅ Per CodingSOPs: every handler has logger, every public function has docstring, every fallible call has try/catch, every boundary uses zod, no any outside vitest mock seams.

---

Spec 01 follow-up (required for Task 3 implementation)

• SessionStore.getSamples(sessionId, pathGroupId) is referenced by generate-extractor.ts but is NOT yet declared in Spec 01. Add a thin method that runs searchSingleIndex with filter parent_id:<sessionId> AND record_type:sample AND pathGroupId:<id> and returns Array<{ url: string; html: string; status: number }>. Tracked here so the cluster builds without hand-edits to Spec 01 from this spec's TDD loop: surface to the Spec 01 owner before kicking off Spec 08 Task 3.

---

Out of scope (handled by other specs)

• The lib/factory/* modules these handlers consume (sitemap.ts, path-grouper.ts, sampler.ts, classifier.ts, cms-detector.ts, structure-analyzer.ts, extractor-generator.ts, sandbox-runner.ts, crawler-client.ts, index-manager.ts) → Specs 02–06.
• DSS registry, types, session/blueprint stores → Spec 01.
• Brand-domain discovery (only consumed by discover.ts as an optional enrichment step in v2) → Spec 07.
• Frontend hooks that call these endpoints (useFactorySession, useDiscoverStream, useCrawlProgress) → Spec 09.
• Frontend pages (AdminFactory.tsx, panels) → Specs 09–12.
• End-to-end smoke test that wires a real site through every endpoint → Spec 13.
• Real Algolia round-trip integration tests (gated by env vars) → Spec 13.
• Rate limiting + circuit breaker (already exists in lib/security/; if applied to factory endpoints, it lands in Spec 13's hardening pass).