Dashboard/Crawler-Factory/Engineering-Specs/09-frontend-foundations.md
Crawler-Factory

Engineering-Specs/09-frontend-foundations.md

Crawler Factory: Spec 09: Frontend Foundations (Route, Page Shell, Hooks) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Land the frontend chassis for the Crawler Factory: the /admin/factory route, a page shell that matches the existing Admin aesthetic, and the three React hooks (useFactorySession, useDiscoveryStream, useCrawlProgress) that downstream component specs (10–12) will plug into. This spec ships only chassis + state plumbing; no UI components.

Architecture: Pure frontend wiring layer. The page shell reuses Admin.tsx visual conventions (dark #0a0a0f, framer-motion fade-up, font-black uppercase tracking-tighter headers, indigo accent). One TanStack-Query hook (useFactorySession) reads the session document by id. Two SSE-consumer hooks (useDiscoveryStream, useCrawlProgress) reuse the proven fetch + reader + decoder + line-buffer pattern from useAgentStudioMaverick.ts:162–246: the only working SSE consumer pattern in this codebase. Each event line is parsed by a type field and routed to a per-channel state slice. No EventSource (it can't send custom headers and we may need them later). Browser-native ReadableStream only.

Tech Stack: React 18, react-router-dom v6, @tanstack/react-query v5, framer-motion v12, lib/factory/types.ts (zod schemas from Spec 01), Tailwind, vitest + @testing-library/react.

Depends on: - Spec 01 (Foundations): imports FactorySessionSchema from lib/factory/types - Spec 08 (API endpoints): consumes /api/factory/session, /api/factory/discover, /api/factory/crawl-progress

Consumed by: - Spec 10 (DiscoveryPanel + CategoryReview): consumes useDiscoveryStream - Spec 11 (CategoryConfigurator drawer): consumes useFactorySession - Spec 12 (LiveCrawlMonitor + Orchestrator): consumes useCrawlProgress and useFactorySession

Plan source: Projects/Crawler-Factory/00-Plan.md §5 (UI specs: visual conventions, page layout), §7 Tasks 22 + 23, §13 critical reference files.

File structure

Path Responsibility
src/App.tsx Add /admin/factory route above the catch-all * route
src/pages/AdminFactory.tsx Page shell: header, back button, framer-motion fade-up, empty body container
src/hooks/factory/useFactorySession.ts TanStack Query GET /api/factory/session?id=…, zod-validates the response
src/hooks/factory/useDiscoveryStream.ts SSE consumer for /api/factory/discover, parses log/urls/pathGroup/classified/done event types
src/hooks/factory/useCrawlProgress.ts SSE consumer for /api/factory/crawl-progress, handles progress/done/timeout event types
src/hooks/factory/__tests__/useFactorySession.test.tsx Initial load + refetch + error path
src/hooks/factory/__tests__/useDiscoveryStream.test.tsx Event-by-event parsing for each of the 5 event types
src/hooks/factory/__tests__/useCrawlProgress.test.tsx Progress accumulation + closes stream on done
src/pages/__tests__/AdminFactory.test.tsx Smoke render + header text

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (every fetch response); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful.
  2. Logger-style trace at module top. No console.log in production paths; the SSE hooks may use console.warn for unparseable lines but nothing else. Errors flow through React state ({error}) so the UI can surface them.
  3. Try/catch every fallible call. Every fetch and reader.read() wrapped. Catch specific error first (AbortError), generic last. Always set state-side error before rethrowing.
  4. Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object. (The hooks themselves are larger by design: that's fine; the helpers inside must obey.)
  5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. Server responses are zod-validated against FactorySessionSchema (Spec 01). SSE event payloads are validated against discriminated-union schemas defined inside the hooks.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock external services in unit tests. Mock global.fetch for both query + stream cases. Provide a helper that builds a fake ReadableStream from a string array.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/components, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts for utilities, PascalCase.tsx for components, useFooBar.ts for hooks. Tests: same name with .test.tsx.
  10. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.
  11. Cleanup on unmount. Every SSE hook must abort the stream and release the reader on unmount. Failure mode if skipped: zombie streams hold sockets open across navigation, exhausting Vercel function quotas.
  12. No render of unbounded arrays without keys. Every state array in the SSE hooks ships with a stable id field on each entry so downstream components can key={entry.id} without warnings.

Task 22.0: Verify dependencies + add vitest jsdom env if missing

Files: - Read: package.json - Read: vitest.config.ts - Modify (only if missing): vitest.config.ts

These hooks render React components under test, so vitest must run with the jsdom environment for component tests. The repo already uses vitest 4 with @testing-library/react 16 (confirmed in package.json). This task verifies the test environment can render React.

  • [ ] Step 22.0.1: Read package.json and confirm deps
node -e "const p=require('./package.json'); console.log({rq: p.dependencies['@tanstack/react-query'], rrd: p.dependencies['react-router-dom'], fm: p.dependencies['framer-motion'], tlr: p.devDependencies['@testing-library/react'], jsd: p.devDependencies['jsdom'] || p.dependencies['jsdom']})"

Expected output includes @tanstack/react-query, react-router-dom, framer-motion, @testing-library/react. If jsdom is missing, install it.

  • [ ] Step 22.0.2: Install jsdom if missing
npm install --save-dev jsdom

Skip if the previous step already showed jsdom resolved.

  • [ ] Step 22.0.3: Verify vitest.config.ts enables jsdom for component tests

Read vitest.config.ts. If it does not already set test.environment: 'jsdom' (or use environmentMatchGlobs to apply jsdom to **/__tests__/*.test.tsx), edit it. Example minimal addition:

// vitest.config.ts — example merge if `test` block exists already
export default defineConfig({
  test: {
    environment: 'node',
    environmentMatchGlobs: [
      ['src/**/*.test.tsx', 'jsdom'],
      ['src/**/*.test.ts', 'jsdom'],
    ],
    setupFiles: ['./src/test/setup.ts'],
    globals: true,
  },
});

If setupFiles is new, create src/test/setup.ts with:

import '@testing-library/jest-dom';
import { afterEach } from 'vitest';
import { cleanup } from '@testing-library/react';

afterEach(() => {
  cleanup();
});
  • [ ] Step 22.0.4: Run baseline test suite
npx vitest run --reporter=basic

Expected: existing tests (407+ baseline + Specs 01–08 additions) all GREEN. If anything red here, stop: fix the env config before proceeding.

  • [ ] Step 22.0.5: Commit env config (only if files changed)
git add package.json package-lock.json vitest.config.ts src/test/setup.ts
git commit -m "chore(factory): enable jsdom env + testing-library cleanup for hook tests"

Skip the commit if nothing changed.

Task 22.1: Add /admin/factory route in src/App.tsx

Files: - Modify: src/App.tsx

The route must land above the catch-all * route and use the same <Route> pattern as the other admin routes (/admin, /debug, /devtools). Vite + react-router will lazy-resolve through ESM so a regular import is fine here.

  • [ ] Step 22.1.1: Write failing route smoke test

Create src/__tests__/App.routes.test.tsx:

import { describe, it, expect, vi } from 'vitest';
import { render, screen } from '@testing-library/react';
import { MemoryRouter, Routes, Route } from 'react-router-dom';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';

// Stub heavy children to keep the route resolution test fast and isolated
vi.mock('@/components/landing/NodeNetwork', () => ({ NodeNetwork: () => null }));
vi.mock('@/components/maverick-devtools', () => ({ MaverickDevTools: () => null }));
vi.mock('@/components/ui/toaster', () => ({ Toaster: () => null }));
vi.mock('@/components/ui/sonner', () => ({ Toaster: () => null }));

import AdminFactory from '../pages/AdminFactory';

describe('App routes — /admin/factory', () => {
  it('mounts AdminFactory at /admin/factory', () => {
    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    render(
      <QueryClientProvider client={qc}>
        <MemoryRouter initialEntries={['/admin/factory']}>
          <Routes>
            <Route path="/admin/factory" element={<AdminFactory />} />
          </Routes>
        </MemoryRouter>
      </QueryClientProvider>
    );
    expect(screen.getByText(/CRAWLER FACTORY/i)).toBeInTheDocument();
  });
});
  • [ ] Step 22.1.2: Run, expect FAIL
npx vitest run src/__tests__/App.routes.test.tsx

Expected: FAIL: Cannot find module '../pages/AdminFactory' (we haven't created it yet).

  • [ ] Step 22.1.3: Modify src/App.tsx: import + register route

Open src/App.tsx. After the existing imports (last one is import DevToolsPage from "./pages/DevToolsPage";), add:

import AdminFactory from "./pages/AdminFactory";

Inside <Routes> block, add the new route immediately above {/* ADD ALL CUSTOM ROUTES ABOVE THE CATCH-ALL "*" ROUTE */}:

<Route path="/admin/factory" element={<AdminFactory />} />

The full block, after the change, looks like:

<Routes>
  <Route path="/" element={<Landing />} />
  <Route path="/chat" element={<Index />} />
  <Route path="/docs" element={<Docs />} />
  <Route path="/admin" element={<Admin />} />
  <Route path="/admin/factory" element={<AdminFactory />} />
  <Route path="/debug" element={<Debug />} />
  <Route path="/devtools" element={<DevToolsPage />} />
  <Route path="/health" element={<HealthCheck />} />
  {/* ADD ALL CUSTOM ROUTES ABOVE THE CATCH-ALL "*" ROUTE */}
  <Route path="*" element={<NotFound />} />
</Routes>

(Don't run the test yet: AdminFactory page is created in Task 22.2.)

Task 22.2: Create src/pages/AdminFactory.tsx: page shell

Files: - Create: src/pages/AdminFactory.tsx - Create: src/pages/__tests__/AdminFactory.test.tsx

Visual conventions are dictated by 00-Plan.md §5a and mirror src/pages/Admin.tsx: - Background: bg-[#0a0a0f], text white, borders border-white/10 - Headers: font-black uppercase tracking-tighter text-3xl, accent indigo - Section labels: text-sm font-bold text-white/40 uppercase tracking-widest - Mono font for IDs/timestamps - Animations: framer-motion fade-up on section mount

The shell is intentionally empty inside its body container: Specs 10–12 fill it.

  • [ ] Step 22.2.1: Write failing test for header + back-button presence

Create src/pages/__tests__/AdminFactory.test.tsx:

import { describe, it, expect } from 'vitest';
import { render, screen } from '@testing-library/react';
import { MemoryRouter } from 'react-router-dom';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import AdminFactory from '../AdminFactory';

const renderShell = () => {
  const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
  return render(
    <QueryClientProvider client={qc}>
      <MemoryRouter>
        <AdminFactory />
      </MemoryRouter>
    </QueryClientProvider>
  );
};

describe('AdminFactory page shell', () => {
  it('renders the CRAWLER FACTORY header', () => {
    renderShell();
    expect(screen.getByText(/CRAWLER FACTORY/i)).toBeInTheDocument();
  });

  it('marks the page as a SPIKE', () => {
    renderShell();
    expect(screen.getByText(/SPIKE/i)).toBeInTheDocument();
  });

  it('renders the Build → Verify → Commit subtitle', () => {
    renderShell();
    expect(screen.getByText(/Build.*Verify.*Commit/i)).toBeInTheDocument();
  });

  it('renders a back-to-admin link', () => {
    renderShell();
    const back = screen.getByRole('link', { name: /back to mission control|admin|back/i });
    expect(back).toBeInTheDocument();
    expect(back).toHaveAttribute('href', '/admin');
  });

  it('renders an empty body container with a stable test id', () => {
    renderShell();
    const body = screen.getByTestId('factory-body');
    expect(body).toBeInTheDocument();
    expect(body.children.length).toBe(0);
  });
});
  • [ ] Step 22.2.2: Run, expect FAIL
npx vitest run src/pages/__tests__/AdminFactory.test.tsx

Expected: FAIL: Cannot find module '../AdminFactory'.

  • [ ] Step 22.2.3: Implement src/pages/AdminFactory.tsx

Create src/pages/AdminFactory.tsx:

/**
 * AdminFactory — page shell for the Crawler Factory.
 *
 * Visual conventions mirror src/pages/Admin.tsx (00-Plan.md §5a):
 * dark background, framer-motion fade-up, indigo accent, monospace
 * subtitle. The body container is empty by design; downstream specs
 * (10-12) populate it with DiscoveryPanel, CategoryReview,
 * CategoryConfigurator, and LiveCrawlMonitor.
 */

import { motion } from "framer-motion";
import { Link } from "react-router-dom";
import { ArrowLeft, Factory } from "lucide-react";
import { Button } from "@/components/ui/button";

const AdminFactory = () => {
  return (
    <div className="min-h-screen bg-[#0a0a0f] text-white selection:bg-indigo-500/30 font-sans p-6 pb-24">
      <div className="max-w-7xl mx-auto space-y-6">

        {/* Header — same shape as Admin.tsx */}
        <header className="flex items-center justify-between py-6 border-b border-white/10 mb-8">
          <div className="flex items-center gap-4">
            <Button
              asChild
              variant="ghost"
              size="icon"
              className="text-white/40 hover:text-white hover:bg-white/10 rounded-xl"
            >
              <Link to="/admin" aria-label="Back to Mission Control">
                <ArrowLeft className="w-5 h-5" />
              </Link>
            </Button>
            <div>
              <h1 className="text-3xl font-black text-white uppercase tracking-tighter flex items-center gap-3">
                Crawler Factory
                <span className="text-xs px-2 py-0.5 rounded border border-indigo-500/30 bg-indigo-500/10 text-indigo-400 tracking-widest font-mono">
                  SPIKE
                </span>
              </h1>
              <p className="text-sm text-white/40 font-mono tracking-wide mt-1">
                BUILD → VERIFY → COMMIT
              </p>
            </div>
          </div>
          <div className="hidden md:flex items-center gap-4">
            <div className="p-2 rounded-xl bg-indigo-500/20 border border-indigo-500/30">
              <Factory className="w-5 h-5 text-indigo-400" />
            </div>
          </div>
        </header>

        {/* Body — Specs 10-12 fill this in. Container deliberately empty. */}
        <motion.section
          initial={{ opacity: 0, y: 20 }}
          animate={{ opacity: 1, y: 0 }}
          transition={{ duration: 0.5 }}
          data-testid="factory-body"
        />

      </div>
    </div>
  );
};

export default AdminFactory;
  • [ ] Step 22.2.4: Run shell tests, expect PASS
npx vitest run src/pages/__tests__/AdminFactory.test.tsx

Expected: 5 tests pass.

  • [ ] Step 22.2.5: Run the App route smoke test (from Task 22.1)
npx vitest run src/__tests__/App.routes.test.tsx

Expected: 1 test passes.

  • [ ] Step 22.2.6: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 22.2.7: Commit route + shell
git add src/App.tsx src/pages/AdminFactory.tsx src/pages/__tests__/AdminFactory.test.tsx src/__tests__/App.routes.test.tsx
git commit -m "feat(factory): /admin/factory route + page shell"

Task 23.1: useFactorySession: TanStack Query GET hook

Files: - Create: src/hooks/factory/useFactorySession.ts - Create: src/hooks/factory/__tests__/useFactorySession.test.tsx

Reads /api/factory/session?id=<sessionId> and validates the response with FactorySessionSchema (from Spec 01). When sessionId is null, the query is disabled.

  • [ ] Step 23.1.1: Write failing test: initial load returns session

Create src/hooks/factory/__tests__/useFactorySession.test.tsx:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { renderHook, waitFor } from '@testing-library/react';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import type { ReactNode } from 'react';
import { useFactorySession } from '../useFactorySession';

const wrapper = (qc: QueryClient) =>
  ({ children }: { children: ReactNode }) => (
    <QueryClientProvider client={qc}>{children}</QueryClientProvider>
  );

const validSession = {
  objectID: 'sess_001',
  record_type: 'session',
  status: 'discovering',
  source_domain: 'algolia.com',
  source_url_root: 'https://www.algolia.com',
  createdAt: 1714432200000,
  updatedAt: 1714432200000,
  discovery: {
    sitemapUrls: ['https://www.algolia.com/sitemap.xml'],
    urlCount: 0,
    urlShardCount: 0,
    schemaOrgCoverage: 0,
  },
  pathGroups: [],
};

describe('useFactorySession', () => {
  beforeEach(() => {
    global.fetch = vi.fn();
  });
  afterEach(() => {
    vi.restoreAllMocks();
  });

  it('returns the session on a successful fetch', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce({
      ok: true,
      status: 200,
      json: async () => validSession,
    });

    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    const { result } = renderHook(() => useFactorySession('sess_001'), {
      wrapper: wrapper(qc),
    });

    await waitFor(() => expect(result.current.isLoading).toBe(false));
    expect(result.current.session?.objectID).toBe('sess_001');
    expect(result.current.error).toBeNull();
    expect(global.fetch).toHaveBeenCalledWith(
      '/api/factory/session?id=sess_001',
      expect.objectContaining({ method: 'GET' })
    );
  });

  it('does not fetch when sessionId is null', () => {
    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    renderHook(() => useFactorySession(null), { wrapper: wrapper(qc) });
    expect(global.fetch).not.toHaveBeenCalled();
  });

  it('surfaces a zod validation error when the server returns malformed data', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce({
      ok: true,
      status: 200,
      json: async () => ({ objectID: 'sess_001' }), // missing required fields
    });

    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    const { result } = renderHook(() => useFactorySession('sess_001'), {
      wrapper: wrapper(qc),
    });

    await waitFor(() => expect(result.current.isLoading).toBe(false));
    expect(result.current.session).toBeNull();
    expect(result.current.error).not.toBeNull();
  });

  it('surfaces an HTTP error when the server returns 500', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce({
      ok: false,
      status: 500,
      json: async () => ({ error: 'boom' }),
      text: async () => 'boom',
    });

    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    const { result } = renderHook(() => useFactorySession('sess_001'), {
      wrapper: wrapper(qc),
    });

    await waitFor(() => expect(result.current.isLoading).toBe(false));
    expect(result.current.error).not.toBeNull();
  });

  it('refetch() triggers a second fetch', async () => {
    (global.fetch as ReturnType<typeof vi.fn>)
      .mockResolvedValueOnce({ ok: true, status: 200, json: async () => validSession })
      .mockResolvedValueOnce({ ok: true, status: 200, json: async () => validSession });

    const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
    const { result } = renderHook(() => useFactorySession('sess_001'), {
      wrapper: wrapper(qc),
    });

    await waitFor(() => expect(result.current.isLoading).toBe(false));
    await result.current.refetch();
    await waitFor(() => expect((global.fetch as ReturnType<typeof vi.fn>).mock.calls.length).toBe(2));
  });
});
  • [ ] Step 23.1.2: Run, expect FAIL
npx vitest run src/hooks/factory/__tests__/useFactorySession.test.tsx

Expected: FAIL: Cannot find module '../useFactorySession'.

  • [ ] Step 23.1.3: Implement src/hooks/factory/useFactorySession.ts

Create src/hooks/factory/useFactorySession.ts:

/**
 * useFactorySession — TanStack Query hook for /api/factory/session.
 *
 * Validates the response against FactorySessionSchema (Spec 01) before
 * returning so downstream components never receive an unsafe shape.
 * When sessionId is null the query is disabled — no network call fires.
 */

import { useQuery } from '@tanstack/react-query';
import { FactorySessionSchema, type FactorySession } from '@/lib/factory/types';

interface UseFactorySessionResult {
  session: FactorySession | null;
  isLoading: boolean;
  error: Error | null;
  refetch: () => Promise<unknown>;
}

async function fetchFactorySession(sessionId: string): Promise<FactorySession> {
  const res = await fetch(`/api/factory/session?id=${encodeURIComponent(sessionId)}`, {
    method: 'GET',
    headers: { 'Content-Type': 'application/json' },
  });
  if (!res.ok) {
    const text = await res.text().catch(() => '');
    throw new Error(`session fetch failed: HTTP ${res.status}${text ? ` — ${text}` : ''}`);
  }
  const raw = await res.json();
  // Throws ZodError if the server returns a malformed shape — surfaced as
  // `error` in the hook return.
  return FactorySessionSchema.parse(raw);
}

/**
 * Read a factory session by id. Pass null to disable the query.
 */
export function useFactorySession(sessionId: string | null): UseFactorySessionResult {
  const query = useQuery({
    queryKey: ['factory', 'session', sessionId],
    queryFn: () => fetchFactorySession(sessionId as string),
    enabled: sessionId !== null,
    retry: false,
    staleTime: 5_000,
  });

  return {
    session: query.data ?? null,
    isLoading: query.isLoading || query.isFetching,
    error: (query.error as Error | null) ?? null,
    refetch: query.refetch,
  };
}
  • [ ] Step 23.1.4: Run hook tests, expect PASS
npx vitest run src/hooks/factory/__tests__/useFactorySession.test.tsx

Expected: 5 tests pass.

  • [ ] Step 23.1.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 23.1.6: Commit
git add src/hooks/factory/useFactorySession.ts src/hooks/factory/__tests__/useFactorySession.test.tsx
git commit -m "feat(factory): useFactorySession hook (TanStack Query + zod-validated)"

Task 23.2: useDiscoveryStream: SSE consumer for /api/factory/discover

Files: - Create: src/hooks/factory/useDiscoveryStream.ts - Create: src/hooks/factory/__tests__/useDiscoveryStream.test.tsx

Reuses the proven streaming pattern from useAgentStudioMaverick.ts:162–246 (fetch → reader → decoder → line buffer → trim → parse). Each event line is one JSON object terminated by \n. The shape:

{"type":"log","ts":1714432200000,"level":"info","message":"robots.txt OK"}
{"type":"urls","batch":["https://x.com/a","https://x.com/b"],"total":1247}
{"type":"pathGroup","group":{"id":"pg_1","pattern":"/blog/*","urlCount":412,"sampleUrls":["https://x.com/blog/a","https://x.com/blog/b"]}}
{"type":"classified","groupId":"pg_1","contentDomain":"marketing","confidence":0.92,"detectedVia":"json-ld"}
{"type":"done","sessionId":"sess_001"}

State channels: one slice per event type: - logs: LogEntry[] - urlBatches: string[][] - pathGroups: PathGroupSummary[] - classifications: Classification[] - status: 'idle' | 'streaming' | 'done' | 'error' - sessionId: string | null (set by the done event)

The discover(url: string) function kicks off the stream. Returns a cleanup abort() plus all state arrays.

  • [ ] Step 23.2.1: Write failing test: parses each event type

Create src/hooks/factory/__tests__/useDiscoveryStream.test.tsx:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { renderHook, act, waitFor } from '@testing-library/react';
import { useDiscoveryStream } from '../useDiscoveryStream';

/**
 * Build a fake Response whose body is a ReadableStream emitting `lines`
 * separated by '\n'. Each yielded chunk is a single line — exercises the
 * hook's line-buffering logic (some lines arrive split, some glued).
 */
function makeStreamResponse(lines: string[]): Response {
  const encoder = new TextEncoder();
  let i = 0;
  const stream = new ReadableStream({
    pull(controller) {
      if (i >= lines.length) {
        controller.close();
        return;
      }
      const chunk = i + 1 < lines.length && i % 2 === 0
        ? encoder.encode(lines[i] + '\n' + lines[i + 1] + '\n') // glue 2 lines
        : encoder.encode(lines[i] + '\n');                        // single line
      controller.enqueue(chunk);
      i = (i + 1 < lines.length && i % 2 === 0) ? i + 2 : i + 1;
    },
  });
  return new Response(stream, { status: 200, headers: { 'Content-Type': 'application/x-ndjson' } });
}

const events = [
  JSON.stringify({ type: 'log', ts: 1714432200000, level: 'info', message: 'robots.txt OK' }),
  JSON.stringify({ type: 'urls', batch: ['https://x.com/a', 'https://x.com/b'], total: 2 }),
  JSON.stringify({
    type: 'pathGroup',
    group: { id: 'pg_1', pattern: '/blog/*', urlCount: 412, sampleUrls: ['https://x.com/blog/a'] },
  }),
  JSON.stringify({
    type: 'classified',
    groupId: 'pg_1',
    contentDomain: 'marketing',
    confidence: 0.92,
    detectedVia: 'json-ld',
  }),
  JSON.stringify({ type: 'done', sessionId: 'sess_001' }),
];

describe('useDiscoveryStream', () => {
  beforeEach(() => { global.fetch = vi.fn(); });
  afterEach(() => { vi.restoreAllMocks(); });

  it('starts in idle state with empty arrays', () => {
    const { result } = renderHook(() => useDiscoveryStream());
    expect(result.current.status).toBe('idle');
    expect(result.current.logs).toEqual([]);
    expect(result.current.urlBatches).toEqual([]);
    expect(result.current.pathGroups).toEqual([]);
    expect(result.current.classifications).toEqual([]);
    expect(result.current.sessionId).toBeNull();
  });

  it('parses log events into the logs array', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([events[0]]));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.logs.length).toBe(1));
    expect(result.current.logs[0].message).toBe('robots.txt OK');
  });

  it('parses urls events into the urlBatches array', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([events[1]]));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.urlBatches.length).toBe(1));
    expect(result.current.urlBatches[0]).toEqual(['https://x.com/a', 'https://x.com/b']);
  });

  it('parses pathGroup events into the pathGroups array', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([events[2]]));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.pathGroups.length).toBe(1));
    expect(result.current.pathGroups[0].pattern).toBe('/blog/*');
  });

  it('parses classified events into the classifications array', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([events[3]]));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.classifications.length).toBe(1));
    expect(result.current.classifications[0].contentDomain).toBe('marketing');
    expect(result.current.classifications[0].confidence).toBe(0.92);
  });

  it('captures sessionId and transitions to status=done on the done event', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([events[4]]));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.status).toBe('done'));
    expect(result.current.sessionId).toBe('sess_001');
  });

  it('handles a full multi-event stream end-to-end', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse(events));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.status).toBe('done'));
    expect(result.current.logs.length).toBe(1);
    expect(result.current.urlBatches.length).toBe(1);
    expect(result.current.pathGroups.length).toBe(1);
    expect(result.current.classifications.length).toBe(1);
    expect(result.current.sessionId).toBe('sess_001');
  });

  it('moves to status=error when fetch returns non-ok', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(
      new Response('boom', { status: 500 })
    );
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.status).toBe('error'));
  });

  it('ignores malformed JSON lines without crashing the stream', async () => {
    const garbled = ['not-json', events[0], '{"incomplete":'];
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse(garbled));
    const { result } = renderHook(() => useDiscoveryStream());
    await act(async () => { await result.current.discover('https://x.com'); });
    await waitFor(() => expect(result.current.logs.length).toBe(1));
  });
});
  • [ ] Step 23.2.2: Run, expect FAIL
npx vitest run src/hooks/factory/__tests__/useDiscoveryStream.test.tsx

Expected: FAIL: Cannot find module '../useDiscoveryStream'.

  • [ ] Step 23.2.3: Implement src/hooks/factory/useDiscoveryStream.ts

Create src/hooks/factory/useDiscoveryStream.ts:

/**
 * useDiscoveryStream — SSE consumer for /api/factory/discover.
 *
 * Streaming pattern adapted from useAgentStudioMaverick.ts:162-246
 * (fetch → reader → decoder → line-buffer → JSON.parse). Each line
 * arriving over the wire is one JSON object with a `type` discriminator.
 *
 * State is split per event type to keep render cost proportional to the
 * channel that just changed (a 1000-URL urls event must not re-render
 * the logs list).
 */

import { useCallback, useRef, useState } from 'react';

// ─── Event payload shapes ────────────────────────────────────────────────────

export interface LogEntry {
  id: string;
  ts: number;
  level: 'info' | 'warn' | 'error';
  message: string;
}

export interface PathGroupSummary {
  id: string;
  pattern: string;
  urlCount: number;
  sampleUrls: string[];
}

export interface Classification {
  id: string;
  groupId: string;
  contentDomain: string;
  confidence: number;
  detectedVia: string;
}

export type DiscoveryStatus = 'idle' | 'streaming' | 'done' | 'error';

interface UseDiscoveryStreamResult {
  logs: LogEntry[];
  urlBatches: string[][];
  pathGroups: PathGroupSummary[];
  classifications: Classification[];
  status: DiscoveryStatus;
  sessionId: string | null;
  error: Error | null;
  discover: (url: string) => Promise<void>;
  abort: () => void;
}

/**
 * Parse one NDJSON line into a typed event. Returns null on malformed input
 * so the stream loop can skip and continue.
 */
function parseEventLine(line: string): { type: string; [k: string]: unknown } | null {
  try {
    const parsed = JSON.parse(line);
    if (parsed && typeof parsed === 'object' && typeof parsed.type === 'string') {
      return parsed as { type: string; [k: string]: unknown };
    }
    return null;
  } catch {
    return null;
  }
}

/**
 * SSE consumer for the discover endpoint. Returns per-channel state arrays
 * + a discover() trigger and an abort() escape hatch.
 */
export function useDiscoveryStream(): UseDiscoveryStreamResult {
  const [logs, setLogs] = useState<LogEntry[]>([]);
  const [urlBatches, setUrlBatches] = useState<string[][]>([]);
  const [pathGroups, setPathGroups] = useState<PathGroupSummary[]>([]);
  const [classifications, setClassifications] = useState<Classification[]>([]);
  const [status, setStatus] = useState<DiscoveryStatus>('idle');
  const [sessionId, setSessionId] = useState<string | null>(null);
  const [error, setError] = useState<Error | null>(null);

  const abortRef = useRef<AbortController | null>(null);

  const abort = useCallback(() => {
    abortRef.current?.abort();
    abortRef.current = null;
  }, []);

  const discover = useCallback(async (url: string): Promise<void> => {
    // Reset all channels for a fresh run
    setLogs([]);
    setUrlBatches([]);
    setPathGroups([]);
    setClassifications([]);
    setSessionId(null);
    setError(null);
    setStatus('streaming');

    const ctrl = new AbortController();
    abortRef.current = ctrl;

    try {
      const res = await fetch('/api/factory/discover', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ url }),
        signal: ctrl.signal,
      });
      if (!res.ok) {
        throw new Error(`discover stream failed: HTTP ${res.status}`);
      }
      if (!res.body) {
        throw new Error('discover stream: response has no body');
      }

      const reader = res.body.getReader();
      const decoder = new TextDecoder();
      let buffer = '';

      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          buffer += decoder.decode(value, { stream: true });
          const lines = buffer.split('\n');
          buffer = lines.pop() ?? '';

          for (const rawLine of lines) {
            const line = rawLine.trim();
            if (!line) continue;
            const evt = parseEventLine(line);
            if (!evt) continue;

            if (evt.type === 'log') {
              const entry: LogEntry = {
                id: `log_${evt.ts as number}_${Math.random().toString(36).slice(2, 8)}`,
                ts: Number(evt.ts) || Date.now(),
                level: (evt.level as LogEntry['level']) ?? 'info',
                message: String(evt.message ?? ''),
              };
              setLogs((prev) => [...prev, entry]);
            } else if (evt.type === 'urls') {
              const batch = Array.isArray(evt.batch) ? (evt.batch as string[]) : [];
              setUrlBatches((prev) => [...prev, batch]);
            } else if (evt.type === 'pathGroup') {
              const g = evt.group as PathGroupSummary | undefined;
              if (g && g.id) setPathGroups((prev) => [...prev, g]);
            } else if (evt.type === 'classified') {
              const c: Classification = {
                id: `cls_${evt.groupId as string}_${Date.now()}`,
                groupId: String(evt.groupId ?? ''),
                contentDomain: String(evt.contentDomain ?? ''),
                confidence: Number(evt.confidence ?? 0),
                detectedVia: String(evt.detectedVia ?? ''),
              };
              setClassifications((prev) => [...prev, c]);
            } else if (evt.type === 'done') {
              setSessionId(String(evt.sessionId ?? ''));
              setStatus('done');
            }
          }
        }
      } finally {
        reader.releaseLock();
      }

      // If the stream ended without a `done` event, leave status as 'streaming'
      // turned into 'done' only by the explicit `done` event. If the loop just
      // exhausted (server closed the connection) without one, mark done anyway.
      setStatus((cur) => (cur === 'streaming' ? 'done' : cur));
    } catch (err) {
      if ((err as Error).name === 'AbortError') {
        // user-initiated abort — leave channels alone, drop to idle
        setStatus('idle');
        return;
      }
      setError(err as Error);
      setStatus('error');
    }
  }, []);

  return {
    logs,
    urlBatches,
    pathGroups,
    classifications,
    status,
    sessionId,
    error,
    discover,
    abort,
  };
}
  • [ ] Step 23.2.4: Run discovery-stream tests, expect PASS
npx vitest run src/hooks/factory/__tests__/useDiscoveryStream.test.tsx

Expected: 9 tests pass.

  • [ ] Step 23.2.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 23.2.6: Commit
git add src/hooks/factory/useDiscoveryStream.ts src/hooks/factory/__tests__/useDiscoveryStream.test.tsx
git commit -m "feat(factory): useDiscoveryStream SSE hook (log/urls/pathGroup/classified/done)"

Task 23.3: useCrawlProgress: SSE consumer for /api/factory/crawl-progress

Files: - Create: src/hooks/factory/useCrawlProgress.ts - Create: src/hooks/factory/__tests__/useCrawlProgress.test.tsx

Reads the crawl-progress stream for one crawler. Event shapes (NDJSON, one JSON object per line):

{"type":"progress","crawlerId":"9f3ed...","crawled":38,"total":412,"success":35,"failed":0,"ignored":3}
{"type":"done","crawlerId":"9f3ed...","crawled":412,"total":412,"success":380,"failed":0,"ignored":32}
{"type":"timeout","crawlerId":"9f3ed...","reason":"30min ceiling reached"}

State: - stats: CrawlStats | null: most recent progress snapshot - status: 'idle' | 'streaming' | 'done' | 'timeout' | 'error' - error: Error | null

When crawlerId is null, the hook is dormant. When it changes (or the component unmounts), the previous stream is aborted.

  • [ ] Step 23.3.1: Write failing test: accumulates progress + closes on done

Create src/hooks/factory/__tests__/useCrawlProgress.test.tsx:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { renderHook, act, waitFor } from '@testing-library/react';
import { useCrawlProgress } from '../useCrawlProgress';

function makeStreamResponse(lines: string[]): Response {
  const encoder = new TextEncoder();
  let i = 0;
  const stream = new ReadableStream({
    pull(controller) {
      if (i >= lines.length) { controller.close(); return; }
      controller.enqueue(encoder.encode(lines[i] + '\n'));
      i += 1;
    },
  });
  return new Response(stream, { status: 200 });
}

describe('useCrawlProgress', () => {
  beforeEach(() => { global.fetch = vi.fn(); });
  afterEach(() => { vi.restoreAllMocks(); });

  it('does nothing when crawlerId is null', () => {
    const { result } = renderHook(() => useCrawlProgress(null));
    expect(global.fetch).not.toHaveBeenCalled();
    expect(result.current.status).toBe('idle');
    expect(result.current.stats).toBeNull();
  });

  it('updates stats on each progress event', async () => {
    const events = [
      JSON.stringify({ type: 'progress', crawlerId: 'c1', crawled: 10, total: 100, success: 9, failed: 0, ignored: 1 }),
      JSON.stringify({ type: 'progress', crawlerId: 'c1', crawled: 25, total: 100, success: 23, failed: 1, ignored: 1 }),
    ];
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse(events));

    const { result } = renderHook(() => useCrawlProgress('c1'));
    await waitFor(() => expect(result.current.stats?.crawled).toBe(25));
    expect(result.current.stats?.success).toBe(23);
    expect(result.current.status).toBe('streaming');
  });

  it('transitions to status=done on the done event with final stats', async () => {
    const events = [
      JSON.stringify({ type: 'progress', crawlerId: 'c1', crawled: 50, total: 100, success: 50, failed: 0, ignored: 0 }),
      JSON.stringify({ type: 'done', crawlerId: 'c1', crawled: 100, total: 100, success: 95, failed: 2, ignored: 3 }),
    ];
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse(events));

    const { result } = renderHook(() => useCrawlProgress('c1'));
    await waitFor(() => expect(result.current.status).toBe('done'));
    expect(result.current.stats?.crawled).toBe(100);
    expect(result.current.stats?.success).toBe(95);
  });

  it('transitions to status=timeout on the timeout event', async () => {
    const events = [
      JSON.stringify({ type: 'timeout', crawlerId: 'c1', reason: '30min ceiling reached' }),
    ];
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse(events));

    const { result } = renderHook(() => useCrawlProgress('c1'));
    await waitFor(() => expect(result.current.status).toBe('timeout'));
  });

  it('moves to status=error when fetch returns non-ok', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(
      new Response('boom', { status: 500 })
    );
    const { result } = renderHook(() => useCrawlProgress('c1'));
    await waitFor(() => expect(result.current.status).toBe('error'));
  });

  it('builds the request URL with the crawlerId query param', async () => {
    (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValueOnce(makeStreamResponse([]));
    renderHook(() => useCrawlProgress('crawler_xyz'));
    await waitFor(() => expect(global.fetch).toHaveBeenCalled());
    const calledUrl = (global.fetch as ReturnType<typeof vi.fn>).mock.calls[0][0];
    expect(String(calledUrl)).toContain('/api/factory/crawl-progress');
    expect(String(calledUrl)).toContain('crawlerId=crawler_xyz');
  });

  it('aborts the previous stream when crawlerId changes', async () => {
    const events1 = [JSON.stringify({ type: 'progress', crawlerId: 'c1', crawled: 1, total: 10, success: 1, failed: 0, ignored: 0 })];
    const events2 = [JSON.stringify({ type: 'progress', crawlerId: 'c2', crawled: 2, total: 20, success: 2, failed: 0, ignored: 0 })];
    (global.fetch as ReturnType<typeof vi.fn>)
      .mockResolvedValueOnce(makeStreamResponse(events1))
      .mockResolvedValueOnce(makeStreamResponse(events2));

    const { result, rerender } = renderHook(({ id }: { id: string | null }) => useCrawlProgress(id), {
      initialProps: { id: 'c1' as string | null },
    });
    await waitFor(() => expect(result.current.stats?.crawled).toBe(1));

    await act(async () => { rerender({ id: 'c2' }); });
    await waitFor(() => expect(result.current.stats?.crawled).toBe(2));
    expect(global.fetch).toHaveBeenCalledTimes(2);
  });
});
  • [ ] Step 23.3.2: Run, expect FAIL
npx vitest run src/hooks/factory/__tests__/useCrawlProgress.test.tsx

Expected: FAIL: Cannot find module '../useCrawlProgress'.

  • [ ] Step 23.3.3: Implement src/hooks/factory/useCrawlProgress.ts

Create src/hooks/factory/useCrawlProgress.ts:

/**
 * useCrawlProgress — SSE consumer for /api/factory/crawl-progress.
 *
 * One stream per crawlerId. Updates `stats` on every `progress` event,
 * transitions status to `done`, `timeout`, or `error` based on the
 * terminal event type. Aborts the previous stream when crawlerId
 * changes or the component unmounts (avoids zombie sockets — see
 * SOP rule 11).
 */

import { useEffect, useRef, useState } from 'react';

export interface CrawlStats {
  crawled: number;
  total: number;
  success: number;
  failed: number;
  ignored: number;
}

export type CrawlStatus = 'idle' | 'streaming' | 'done' | 'timeout' | 'error';

interface UseCrawlProgressResult {
  stats: CrawlStats | null;
  status: CrawlStatus;
  error: Error | null;
}

function parseEventLine(line: string): { type: string; [k: string]: unknown } | null {
  try {
    const parsed = JSON.parse(line);
    if (parsed && typeof parsed === 'object' && typeof parsed.type === 'string') {
      return parsed as { type: string; [k: string]: unknown };
    }
    return null;
  } catch {
    return null;
  }
}

function toStats(evt: { [k: string]: unknown }): CrawlStats {
  return {
    crawled: Number(evt.crawled ?? 0),
    total: Number(evt.total ?? 0),
    success: Number(evt.success ?? 0),
    failed: Number(evt.failed ?? 0),
    ignored: Number(evt.ignored ?? 0),
  };
}

/**
 * Subscribe to crawl-progress for a crawlerId. Pass null to disable.
 */
export function useCrawlProgress(crawlerId: string | null): UseCrawlProgressResult {
  const [stats, setStats] = useState<CrawlStats | null>(null);
  const [status, setStatus] = useState<CrawlStatus>('idle');
  const [error, setError] = useState<Error | null>(null);

  const abortRef = useRef<AbortController | null>(null);

  useEffect(() => {
    if (!crawlerId) {
      setStatus('idle');
      setStats(null);
      setError(null);
      return;
    }

    // Tear down any prior stream before starting a new one
    abortRef.current?.abort();
    const ctrl = new AbortController();
    abortRef.current = ctrl;

    setStats(null);
    setError(null);
    setStatus('streaming');

    const run = async () => {
      try {
        const url = `/api/factory/crawl-progress?crawlerId=${encodeURIComponent(crawlerId)}`;
        const res = await fetch(url, { method: 'GET', signal: ctrl.signal });
        if (!res.ok) {
          throw new Error(`crawl-progress stream failed: HTTP ${res.status}`);
        }
        if (!res.body) {
          throw new Error('crawl-progress stream: response has no body');
        }

        const reader = res.body.getReader();
        const decoder = new TextDecoder();
        let buffer = '';

        try {
          while (true) {
            const { done, value } = await reader.read();
            if (done) break;
            buffer += decoder.decode(value, { stream: true });
            const lines = buffer.split('\n');
            buffer = lines.pop() ?? '';

            for (const rawLine of lines) {
              const line = rawLine.trim();
              if (!line) continue;
              const evt = parseEventLine(line);
              if (!evt) continue;

              if (evt.type === 'progress') {
                setStats(toStats(evt));
              } else if (evt.type === 'done') {
                setStats(toStats(evt));
                setStatus('done');
              } else if (evt.type === 'timeout') {
                setStatus('timeout');
              }
            }
          }
        } finally {
          reader.releaseLock();
        }

        // Stream ended without a terminal event — treat as done
        setStatus((cur) => (cur === 'streaming' ? 'done' : cur));
      } catch (err) {
        if ((err as Error).name === 'AbortError') return; // unmount/replace; benign
        setError(err as Error);
        setStatus('error');
      }
    };

    run();

    return () => {
      ctrl.abort();
      abortRef.current = null;
    };
  }, [crawlerId]);

  return { stats, status, error };
}
  • [ ] Step 23.3.4: Run crawl-progress tests, expect PASS
npx vitest run src/hooks/factory/__tests__/useCrawlProgress.test.tsx

Expected: 7 tests pass.

  • [ ] Step 23.3.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 23.3.6: Commit
git add src/hooks/factory/useCrawlProgress.ts src/hooks/factory/__tests__/useCrawlProgress.test.tsx
git commit -m "feat(factory): useCrawlProgress SSE hook (progress/done/timeout)"

Task 23.4: Cluster validation: full regression + manual smoke

Files: - (Read-only verification)

  • [ ] Step 23.4.1: Run the full project test suite
npx vitest run

Expected: existing baseline + Specs 01–08 + ~22 new tests added by this spec, all GREEN.

Breakdown of new tests in Spec 09: - App route smoke: 1 - AdminFactory shell: 5 - useFactorySession: 5 - useDiscoveryStream: 9 - useCrawlProgress: 7

Total expected new: 27.

  • [ ] Step 23.4.2: Run full tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 23.4.3: Manual visual smoke (only if dev environment is available)
npm run dev:api &  # port 3005
npm run dev        # port 5173

Then in a browser navigate to http://localhost:5173/admin/factory. Expected: - Dark #0a0a0f background, framer-motion fade-in - Header reads "Crawler Factory" + indigo "SPIKE" badge + "BUILD → VERIFY → COMMIT" subtitle - Back arrow in top-left navigates to /admin - Body container is empty (no error overlays, no console errors)

If the smoke fails visually, stop and fix before declaring the spec done.

  • [ ] Step 23.4.4: Mark Spec 09 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 9 (Frontend Foundations) | ⏸ | route, shell, hooks |
+ | Cluster 9 (Frontend Foundations) | ✅ | route, shell, hooks |

Acceptance criteria: Spec 09 done means:

  1. ✅ /admin/factory route registered in src/App.tsx above the catch-all * route
  2. ✅ src/pages/AdminFactory.tsx exists and renders the shell: header, SPIKE badge, subtitle, back link, empty body container
  3. ✅ Visual conventions match Admin.tsx exactly: bg-[#0a0a0f], font-black uppercase tracking-tighter headers, indigo accent, framer-motion fade-up
  4. ✅ src/hooks/factory/useFactorySession.ts: TanStack Query, zod-validates, disabled when sessionId is null, exposes {session, isLoading, error, refetch}
  5. ✅ src/hooks/factory/useDiscoveryStream.ts: fetch+reader+decoder pattern, parses log/urls/pathGroup/classified/done event types, exposes {logs, urlBatches, pathGroups, classifications, status, sessionId, error, discover, abort}
  6. ✅ src/hooks/factory/useCrawlProgress.ts: fetch+reader+decoder pattern, handles progress/done/timeout event types, aborts on unmount/id change, exposes {stats, status, error}
  7. ✅ All 27 new tests pass; full project suite stays GREEN
  8. ✅ tsc --noEmit clean
  9. ✅ Per CodingSOPs: every module has docstring, every fallible call has try/catch, every external boundary uses zod (session) or shape-checked parsing (SSE events), every SSE hook releases its reader and aborts on unmount
  10. ✅ Manual smoke at /admin/factory shows shell with no console errors

Out of scope (handled by other specs)

  • DiscoveryPanel UI (URL input form + Submit button) → Spec 10
  • CategoryReview tree (pathGroup checkboxes + DomainBadge) → Spec 10
  • CategoryConfigurator drawer (sample preview + extractor editor + test crawl) → Spec 11
  • LiveCrawlMonitor blocks per crawler → Spec 12
  • CrawlerFactory orchestrator FSM that wires the components together → Spec 12
  • RollingLog + DomainBadge reusable components → Spec 10 (consume useDiscoveryStream.logs + .classifications)
  • API endpoints /api/factory/session, /api/factory/discover, /api/factory/crawl-progress → Spec 08
  • Session resume from ?session=<id> URL param → Spec 12 (orchestrator reads URL, calls useFactorySession)
  • E2E browser smoke covering the whole wizard → Spec 13 (smoke test cluster)