Crawler-Factory

Engineering-Specs/12-frontend-orchestrator.md

Crawler Factory: Spec 12: Frontend Live Monitor + Orchestrator Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Ship the live-crawl progress panel and the top-level FSM-driving orchestrator that wires every prior frontend cluster (Spec 09 hooks, Spec 10 discovery + category review + RollingLog, Spec 11 category configurator drawer) into a single resumable URL-driven flow at /admin/factory.

Architecture: LiveCrawlMonitor is a pure presentational component: one stacked block per active crawler, each block driven by useCrawlProgress(crawlerId) from Spec 09 (SSE polling against /api/factory/crawl-progress). CrawlerFactory is the FSM orchestrator: it reads ?session=<id> from the URL, loads session state via useFactorySession, switches the left column to the component matching session.status (null|discovering|categorized|configuring|crawling|done|error), and always renders the persistent RollingLog on the right column from the session's stored log entries. URL-as-state means the user can refresh, share, or come back tomorrow and resume exactly where they left off. The page shell AdminFactory.tsx (placeholder from Spec 09) is replaced with <CrawlerFactory />.

Tech Stack: React 18 + TypeScript (strict), react-router-dom v6 (useSearchParams for URL state), shadcn UI (Progress, Sheet, Card), framer-motion (existing /admin aesthetic: fade-up on mount), TanStack Query (already wired by useFactorySession), @testing-library/react + vitest for tests, MSW-free testing (mock the hooks directly per Spec 09 pattern).

Depends on: - Spec 09: hooks useFactorySession, useDiscoveryStream, useCrawlProgress; placeholder AdminFactory.tsx page shell - Spec 10: RollingLog, DomainBadge, DiscoveryPanel, CategoryReview - Spec 11: CategoryConfigurator drawer - Spec 08: /api/factory/session (CRUD) and /api/factory/crawl-progress (SSE) endpoints - Spec 01: FactorySession, PathGroup types from lib/factory/types.ts

Consumed by: - Spec 13: bundle + smoke test cluster (E2E flow validation, vault sync, docs)

Plan source: Projects/Crawler-Factory/00-Plan.md §2 (FSM), §5b (page layout), §7 Tasks 28, 29.


File structure

Path Responsibility
src/components/admin/factory/LiveCrawlMonitor.tsx Stacked progress blocks: one per active crawler. Pure presentation; subscribes via useCrawlProgress(crawlerId) per block.
src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx RTL tests: renders one block per crawler; renders header + progress + counters per block; handles empty list.
src/components/admin/factory/CrawlerFactory.tsx Top-level FSM orchestrator. Reads ?session=<id> from URL, loads session, switches left-column component on session.status, always renders RollingLog on right column.
src/components/admin/factory/__tests__/CrawlerFactory.test.tsx RTL tests: renders correct component per status; URL ?session=<id> resumes correctly; null → discovering → categorized → configuring → crawling → done flow.
src/pages/AdminFactory.tsx Modify: replace empty container from Spec 09 with <CrawlerFactory />. Header + framer-motion shell stays.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, UIUXDesignSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (already done by hooks reading API); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, for absent/sentinel values.
  2. No console.log. UI-side logging goes through the shared lib/utils/logger import only when a real signal is needed (errors caught at the boundary). Component bodies do not log routine state.
  3. Try/catch every fallible call. API errors surfaced from hooks are caught and routed to the error FSM branch: never silently swallowed.
  4. Functions ≤20 lines, ≤3 params. Sub-render helpers split out when a JSX block exceeds 20 lines or props exceed 3.
  5. Docstring on every exported component. One-line JSDoc explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. Component props are typed against FactorySession, PathGroup from lib/factory/types.ts.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock hooks in unit tests. vi.mock('@/hooks/factory/useFactorySession') etc.: no real network, no real timers.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for components and types, UPPER_SNAKE_CASE for module constants. Files: PascalCase.tsx (components), kebab-case.ts (utils). Tests: same name as subject + .test.tsx.
  10. Aesthetic conformance. Match /admin aesthetic exactly (Plan §5a): bg-[#0a0a0f], border-white/10, font-black uppercase tracking-tighter headers, mono font for IDs/URLs/timestamps, framer-motion fade-up on section mount.
  11. Accessibility. Every interactive element keyboard-reachable; ARIA labels on icon-only buttons; progress bars have aria-valuenow/aria-valuemin/aria-valuemax.
  12. URL is the source of truth. Session state is ?session=<id> query param. No client-only state for the session id. Resume on refresh works because the URL drives the session load.
  13. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.

Task 1: LiveCrawlMonitor: failing test for empty crawler list

Files: - Create: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

LiveCrawlMonitor receives crawlerIds: Array<{crawlerId, name, contentDomain, indexName}>. With an empty array it renders an empty-state message. With N entries it renders N stacked blocks. Each block reads its own progress via useCrawlProgress(crawlerId) (Spec 09) and shows: header (name + crawlerId mono + DomainBadge + indexName), shadcn Progress bar (0–100%), URLs/sec ticker, success/failed/ignored counters.

  • [ ] Step 1.1: Create the test file with empty-state test

Create src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { LiveCrawlMonitor } from '../LiveCrawlMonitor';

// Mock the Spec 09 hook before importing the component under test.
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
  useCrawlProgress: vi.fn(),
}));

import { useCrawlProgress } from '@/hooks/factory/useCrawlProgress';

const mockedUseCrawlProgress = vi.mocked(useCrawlProgress);

describe('LiveCrawlMonitor', () => {
  beforeEach(() => {
    mockedUseCrawlProgress.mockReset();
    mockedUseCrawlProgress.mockReturnValue({
      urlsCrawled: 0,
      urlsTotal: 0,
      urlsPerSec: 0,
      success: 0,
      failed: 0,
      ignored: 0,
      status: 'pending',
      error: null,
    });
  });

  it('renders empty-state copy when crawlerIds is empty', () => {
    render(<LiveCrawlMonitor crawlerIds={[]} />);
    expect(screen.getByText(/no active crawls/i)).toBeInTheDocument();
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

Expected: FAIL: Cannot find module '../LiveCrawlMonitor' (component not yet implemented).

  • [ ] Step 1.3: Implement LiveCrawlMonitor minimally to pass the empty-state test

Create src/components/admin/factory/LiveCrawlMonitor.tsx:

/**
 * LiveCrawlMonitor — stacked progress blocks, one per active crawler.
 *
 * Why: during a multi-crawler session, the user needs at-a-glance visibility
 * into every crawler the factory just created. Each block subscribes to its
 * own SSE stream via useCrawlProgress(crawlerId).
 */

import { motion } from 'framer-motion';
import { Progress } from '@/components/ui/progress';
import { useCrawlProgress } from '@/hooks/factory/useCrawlProgress';
import { DomainBadge } from './DomainBadge';
import type { ContentDomain } from '@/../lib/factory/types';

export interface LiveCrawlMonitorCrawler {
  crawlerId: string;
  name: string;
  contentDomain: ContentDomain;
  indexName: string;
}

export interface LiveCrawlMonitorProps {
  crawlerIds: LiveCrawlMonitorCrawler[];
}

/**
 * Renders one block per crawler. Empty list → empty-state copy.
 */
export function LiveCrawlMonitor({ crawlerIds }: LiveCrawlMonitorProps): JSX.Element {
  if (crawlerIds.length === 0) {
    return (
      <div
        className="rounded-lg border border-white/10 bg-[#0a0a0f] p-6 text-center"
        data-testid="live-crawl-monitor-empty"
      >
        <p className="text-sm font-bold uppercase tracking-widest text-white/40">
          No active crawls
        </p>
        <p className="mt-2 text-xs text-white/60">
          Start by configuring a category — crawler progress will appear here.
        </p>
      </div>
    );
  }

  return (
    <div className="flex flex-col gap-4" data-testid="live-crawl-monitor">
      {crawlerIds.map((c) => (
        <CrawlerBlock key={c.crawlerId} crawler={c} />
      ))}
    </div>
  );
}

interface CrawlerBlockProps {
  crawler: LiveCrawlMonitorCrawler;
}

function CrawlerBlock({ crawler }: CrawlerBlockProps): JSX.Element {
  const progress = useCrawlProgress(crawler.crawlerId);
  const pct =
    progress.urlsTotal > 0
      ? Math.min(100, Math.round((progress.urlsCrawled / progress.urlsTotal) * 100))
      : 0;

  return (
    <motion.div
      initial={{ opacity: 0, y: 8 }}
      animate={{ opacity: 1, y: 0 }}
      className="rounded-lg border border-white/10 bg-[#0a0a0f] p-4"
      data-testid={`crawler-block-${crawler.crawlerId}`}
    >
      <div className="flex items-center justify-between gap-3">
        <div className="flex items-center gap-3">
          <span className="text-sm font-black uppercase tracking-tighter text-white">
            {crawler.name}
          </span>
          <span className="font-mono text-[10px] text-white/40">{crawler.crawlerId}</span>
          <DomainBadge domain={crawler.contentDomain} confidence={null} />
        </div>
        <span className="font-mono text-[10px] text-white/40">{crawler.indexName}</span>
      </div>

      <div className="mt-3">
        <Progress
          value={pct}
          aria-valuenow={pct}
          aria-valuemin={0}
          aria-valuemax={100}
          aria-label={`${crawler.name} progress`}
        />
        <div className="mt-2 flex items-center justify-between text-[11px] text-white/60">
          <span data-testid={`progress-fraction-${crawler.crawlerId}`}>
            {progress.urlsCrawled} / {progress.urlsTotal} URLs ({pct}%)
          </span>
          <span data-testid={`urls-per-sec-${crawler.crawlerId}`}>
            {progress.urlsPerSec.toFixed(1)} URLs/sec
          </span>
        </div>
      </div>

      <div className="mt-3 flex items-center gap-4 text-[11px]">
        <span className="text-emerald-400" data-testid={`success-${crawler.crawlerId}`}>
          ✓ {progress.success} success
        </span>
        <span className="text-rose-400" data-testid={`failed-${crawler.crawlerId}`}>
          ✗ {progress.failed} failed
        </span>
        <span className="text-white/40" data-testid={`ignored-${crawler.crawlerId}`}>
          ⊘ {progress.ignored} ignored
        </span>
      </div>
    </motion.div>
  );
}
  • [ ] Step 1.4: Run empty-state test, expect PASS
npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

Expected: 1 test passes.

  • [ ] Step 1.5: Commit
git add src/components/admin/factory/LiveCrawlMonitor.tsx src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
git commit -m "feat(factory): LiveCrawlMonitor empty-state shell"

Task 2: LiveCrawlMonitor: render one block per crawler

Files: - Modify: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

  • [ ] Step 2.1: Append failing test for "renders one block per crawler"

Append to src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx (inside the same describe):

  it('renders one block per crawler with header info', () => {
    mockedUseCrawlProgress.mockReturnValue({
      urlsCrawled: 38,
      urlsTotal: 412,
      urlsPerSec: 4.2,
      success: 35,
      failed: 0,
      ignored: 3,
      status: 'running',
      error: null,
    });

    render(
      <LiveCrawlMonitor
        crawlerIds={[
          {
            crawlerId: '9f3ed-aaa',
            name: 'Blog',
            contentDomain: 'marketing',
            indexName: 'algoliacentral_marketing',
          },
          {
            crawlerId: '7a1bc-bbb',
            name: 'Customer Stories',
            contentDomain: 'customer-stories',
            indexName: 'algoliacentral_customers',
          },
        ]}
      />
    );

    expect(screen.getByTestId('crawler-block-9f3ed-aaa')).toBeInTheDocument();
    expect(screen.getByTestId('crawler-block-7a1bc-bbb')).toBeInTheDocument();
    expect(screen.getByText('Blog')).toBeInTheDocument();
    expect(screen.getByText('Customer Stories')).toBeInTheDocument();
    expect(screen.getByText('9f3ed-aaa')).toBeInTheDocument();
    expect(screen.getByText('algoliacentral_marketing')).toBeInTheDocument();
  });

  it('renders progress fraction, URLs/sec, and counters per block', () => {
    mockedUseCrawlProgress.mockReturnValue({
      urlsCrawled: 38,
      urlsTotal: 412,
      urlsPerSec: 4.2,
      success: 35,
      failed: 0,
      ignored: 3,
      status: 'running',
      error: null,
    });

    render(
      <LiveCrawlMonitor
        crawlerIds={[
          {
            crawlerId: '9f3ed-aaa',
            name: 'Blog',
            contentDomain: 'marketing',
            indexName: 'algoliacentral_marketing',
          },
        ]}
      />
    );

    expect(screen.getByTestId('progress-fraction-9f3ed-aaa')).toHaveTextContent(
      '38 / 412 URLs (9%)'
    );
    expect(screen.getByTestId('urls-per-sec-9f3ed-aaa')).toHaveTextContent('4.2 URLs/sec');
    expect(screen.getByTestId('success-9f3ed-aaa')).toHaveTextContent('35 success');
    expect(screen.getByTestId('failed-9f3ed-aaa')).toHaveTextContent('0 failed');
    expect(screen.getByTestId('ignored-9f3ed-aaa')).toHaveTextContent('3 ignored');
  });

  it('renders 0% progress when urlsTotal is 0', () => {
    mockedUseCrawlProgress.mockReturnValue({
      urlsCrawled: 0,
      urlsTotal: 0,
      urlsPerSec: 0,
      success: 0,
      failed: 0,
      ignored: 0,
      status: 'pending',
      error: null,
    });

    render(
      <LiveCrawlMonitor
        crawlerIds={[
          {
            crawlerId: 'p1',
            name: 'Pending',
            contentDomain: 'support',
            indexName: 'algoliacentral_support',
          },
        ]}
      />
    );

    const progressBar = screen.getByLabelText('Pending progress');
    expect(progressBar).toHaveAttribute('aria-valuenow', '0');
    expect(screen.getByTestId('progress-fraction-p1')).toHaveTextContent('0 / 0 URLs (0%)');
  });
  • [ ] Step 2.2: Run, expect PASS

The implementation from Task 1 already covers these: running them confirms.

npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx

Expected: 4 tests pass.

  • [ ] Step 2.3: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 2.4: Commit
git add src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
git commit -m "test(factory): LiveCrawlMonitor renders blocks + progress + counters"

Task 3: CrawlerFactory orchestrator: scaffold + null-session test

Files: - Create: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx - Create: src/components/admin/factory/CrawlerFactory.tsx

CrawlerFactory is the FSM driver. It reads ?session=<id> from the URL via useSearchParams. It loads session state via useFactorySession(sessionId). It picks the left-column component based on session.status:

status left column notes
null (no session id in URL) <DiscoveryPanel /> On done event from useDiscoveryStream, set ?session=<id> and continue.
'discovering' <DiscoveryPanel /> with active stream
'categorized' <CategoryReview /> On "Configure Selected", queue pathGroups + open drawer one at a time.
'configuring' <CategoryReview /> + <CategoryConfigurator /> overlay On commit, advance to next pathGroup; when queue empty, transition to 'crawling'.
'crawling' <LiveCrawlMonitor />
'done' <LiveCrawlMonitor /> + "Start new session" button
'error' error panel + retry/restart options

The right column ALWAYS renders <RollingLog /> driven by session log entries. Layout: 2-column grid (left 60%, right 40%).

  • [ ] Step 3.1: Write failing test for "no session id → renders DiscoveryPanel"

Create src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { MemoryRouter } from 'react-router-dom';

// ─── Mocks ──────────────────────────────────────────────────────────────────
vi.mock('@/hooks/factory/useFactorySession', () => ({
  useFactorySession: vi.fn(),
}));
vi.mock('@/hooks/factory/useDiscoveryStream', () => ({
  useDiscoveryStream: vi.fn(),
}));
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
  useCrawlProgress: vi.fn(() => ({
    urlsCrawled: 0,
    urlsTotal: 0,
    urlsPerSec: 0,
    success: 0,
    failed: 0,
    ignored: 0,
    status: 'pending',
    error: null,
  })),
}));

// Stub child components so each test asserts which one is rendered, not its content.
vi.mock('../DiscoveryPanel', () => ({
  DiscoveryPanel: () => <div data-testid="stub-discovery-panel" />,
}));
vi.mock('../CategoryReview', () => ({
  CategoryReview: () => <div data-testid="stub-category-review" />,
}));
vi.mock('../CategoryConfigurator', () => ({
  CategoryConfigurator: ({ pathGroupId }: { pathGroupId: string }) => (
    <div data-testid="stub-category-configurator">{pathGroupId}</div>
  ),
}));
vi.mock('../RollingLog', () => ({
  RollingLog: () => <div data-testid="stub-rolling-log" />,
}));

import { CrawlerFactory } from '../CrawlerFactory';
import { useFactorySession } from '@/hooks/factory/useFactorySession';
import { useDiscoveryStream } from '@/hooks/factory/useDiscoveryStream';

const mockedUseFactorySession = vi.mocked(useFactorySession);
const mockedUseDiscoveryStream = vi.mocked(useDiscoveryStream);

function renderWithUrl(url: string) {
  return render(
    <MemoryRouter initialEntries={[url]}>
      <CrawlerFactory />
    </MemoryRouter>
  );
}

describe('CrawlerFactory orchestrator', () => {
  beforeEach(() => {
    mockedUseFactorySession.mockReset();
    mockedUseDiscoveryStream.mockReset();
    mockedUseFactorySession.mockReturnValue({
      session: null,
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    mockedUseDiscoveryStream.mockReturnValue({
      logs: [],
      urls: [],
      pathGroups: [],
      isStreaming: false,
      sessionId: null,
      error: null,
      discover: vi.fn(),
    });
  });

  it('renders DiscoveryPanel + RollingLog when no session id in URL', () => {
    renderWithUrl('/admin/factory');
    expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
    expect(screen.getByTestId('stub-rolling-log')).toBeInTheDocument();
    expect(screen.queryByTestId('stub-category-review')).not.toBeInTheDocument();
    expect(screen.queryByTestId('stub-category-configurator')).not.toBeInTheDocument();
  });
});
  • [ ] Step 3.2: Run, expect FAIL
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

Expected: FAIL: Cannot find module '../CrawlerFactory'.

  • [ ] Step 3.3: Implement CrawlerFactory minimally: null-session branch only

Create src/components/admin/factory/CrawlerFactory.tsx:

/**
 * CrawlerFactory — top-level FSM orchestrator for /admin/factory.
 *
 * Why: the factory is a multi-stage flow (discover → categorize → configure
 * → crawl). State lives server-side in algoliacentral_factory_sessions; the
 * URL ?session=<id> is the resume key. This component reads the session and
 * renders the component matching session.status. RollingLog is always-on
 * in the right column for narrative continuity.
 */

import { useEffect, useMemo, useState } from 'react';
import { useSearchParams } from 'react-router-dom';
import { motion } from 'framer-motion';
import { useFactorySession } from '@/hooks/factory/useFactorySession';
import { useDiscoveryStream } from '@/hooks/factory/useDiscoveryStream';
import { DiscoveryPanel } from './DiscoveryPanel';
import { CategoryReview } from './CategoryReview';
import { CategoryConfigurator } from './CategoryConfigurator';
import { LiveCrawlMonitor, type LiveCrawlMonitorCrawler } from './LiveCrawlMonitor';
import { RollingLog } from './RollingLog';
import type { PathGroup } from '@/../lib/factory/types';

/**
 * Top-level FSM orchestrator. Reads ?session=<id> and switches the left
 * column to the component matching session.status.
 */
export function CrawlerFactory(): JSX.Element {
  const [searchParams, setSearchParams] = useSearchParams();
  const sessionId = searchParams.get('session');

  const { session, isLoading, error, refetch } = useFactorySession(sessionId);
  const discoveryStream = useDiscoveryStream();

  // When discovery completes, lift its sessionId into the URL so resume works.
  useEffect(() => {
    if (!sessionId && discoveryStream.sessionId) {
      setSearchParams({ session: discoveryStream.sessionId });
    }
  }, [sessionId, discoveryStream.sessionId, setSearchParams]);

  // Configurator drawer queue: array of pathGroup ids to walk in order.
  const [configQueue, setConfigQueue] = useState<string[]>([]);
  const activePathGroupId = configQueue[0] ?? null;

  const beginConfiguration = (selectedIds: string[]): void => {
    setConfigQueue(selectedIds);
  };

  const advanceQueue = (): void => {
    setConfigQueue((q) => q.slice(1));
    refetch();
  };

  const restart = (): void => {
    setSearchParams({});
    setConfigQueue([]);
  };

  const liveCrawlers = useMemo<LiveCrawlMonitorCrawler[]>(
    () => extractLiveCrawlers(session?.pathGroups ?? []),
    [session?.pathGroups]
  );

  const logs = session?.pathGroups ? [] : discoveryStream.logs;
  // Spec 10's RollingLog accepts the union of session-stored logs + the
  // active discovery stream's transient logs. We pass logs through unchanged.
  const rollingLogEntries = sessionId ? logs : discoveryStream.logs;

  return (
    <motion.div
      initial={{ opacity: 0, y: 12 }}
      animate={{ opacity: 1, y: 0 }}
      className="grid grid-cols-1 gap-6 lg:grid-cols-[3fr_2fr]"
      data-testid="crawler-factory"
    >
      <div className="min-w-0">
        <LeftColumn
          isLoading={isLoading}
          error={error}
          status={session?.status ?? null}
          sessionId={session?.objectID ?? sessionId ?? ''}
          pathGroups={session?.pathGroups ?? []}
          activePathGroupId={activePathGroupId}
          liveCrawlers={liveCrawlers}
          onConfigureSelected={beginConfiguration}
          onCommitDrawer={advanceQueue}
          onRestart={restart}
        />
      </div>
      <div className="min-w-0">
        <RollingLog entries={rollingLogEntries} />
      </div>
    </motion.div>
  );
}

interface LeftColumnProps {
  isLoading: boolean;
  error: Error | null;
  status: string | null;
  sessionId: string;
  pathGroups: PathGroup[];
  activePathGroupId: string | null;
  liveCrawlers: LiveCrawlMonitorCrawler[];
  onConfigureSelected: (selectedIds: string[]) => void;
  onCommitDrawer: () => void;
  onRestart: () => void;
}

function LeftColumn(props: LeftColumnProps): JSX.Element {
  if (props.isLoading) {
    return <LoadingPanel />;
  }
  if (props.error) {
    return <ErrorPanel error={props.error} onRestart={props.onRestart} />;
  }

  switch (props.status) {
    case null:
    case 'discovering':
      return <DiscoveryPanel />;
    case 'categorized':
      return (
        <CategoryReview
          pathGroups={props.pathGroups}
          onConfigureSelected={props.onConfigureSelected}
        />
      );
    case 'configuring': {
      const activePathGroup = props.activePathGroupId
        ? props.pathGroups.find((g) => g.id === props.activePathGroupId) ?? null
        : null;
      return (
        <>
          <CategoryReview
            pathGroups={props.pathGroups}
            onConfigureSelected={props.onConfigureSelected}
          />
          {activePathGroup && activePathGroup.detectedDomain && (
            <CategoryConfigurator
              sessionId={props.sessionId}
              pathGroupId={activePathGroup.id}
              pathGroup={activePathGroup}
              contentDomain={activePathGroup.detectedDomain}
              open={!!props.activePathGroupId}
              onClose={props.onCommitDrawer}
              onCommit={() => props.onCommitDrawer()}
            />
          )}
        </>
      );
    }
    case 'crawling':
      return <LiveCrawlMonitor crawlerIds={props.liveCrawlers} />;
    case 'done':
      return (
        <div className="flex flex-col gap-4">
          <LiveCrawlMonitor crawlerIds={props.liveCrawlers} />
          <button
            type="button"
            onClick={props.onRestart}
            className="self-start rounded-md border border-white/20 bg-white/5 px-4 py-2 text-xs font-bold uppercase tracking-widest text-white hover:bg-white/10"
            data-testid="restart-button"
          >
            Start new session
          </button>
        </div>
      );
    case 'error':
      return <ErrorPanel error={new Error('Session ended in error state.')} onRestart={props.onRestart} />;
    default:
      return <DiscoveryPanel />;
  }
}

function LoadingPanel(): JSX.Element {
  return (
    <div
      className="rounded-lg border border-white/10 bg-[#0a0a0f] p-6 text-center"
      data-testid="loading-panel"
    >
      <p className="text-sm font-bold uppercase tracking-widest text-white/40">Loading session…</p>
    </div>
  );
}

interface ErrorPanelProps {
  error: Error;
  onRestart: () => void;
}

function ErrorPanel({ error, onRestart }: ErrorPanelProps): JSX.Element {
  return (
    <div
      className="rounded-lg border border-rose-500/30 bg-rose-500/10 p-6"
      data-testid="error-panel"
    >
      <p className="text-sm font-black uppercase tracking-tighter text-rose-400">Something went wrong</p>
      <p className="mt-2 text-xs text-white/70">{error.message}</p>
      <button
        type="button"
        onClick={onRestart}
        className="mt-4 rounded-md border border-white/20 bg-white/5 px-3 py-1.5 text-[11px] font-bold uppercase tracking-widest text-white hover:bg-white/10"
        data-testid="error-restart-button"
      >
        Restart
      </button>
    </div>
  );
}

/**
 * Walk pathGroups and produce the LiveCrawlMonitor input. A pathGroup is
 * "live" once its blueprint has a crawlerId attached.
 */
function extractLiveCrawlers(pathGroups: PathGroup[]): LiveCrawlMonitorCrawler[] {
  const out: LiveCrawlMonitorCrawler[] = [];
  for (const pg of pathGroups) {
    const bp = pg.blueprint;
    if (!bp || !bp.crawlerId || !pg.detectedDomain) continue;
    out.push({
      crawlerId: bp.crawlerId,
      name: pg.pattern,
      contentDomain: pg.detectedDomain,
      indexName: bp.indexName,
    });
  }
  return out;
}
  • [ ] Step 3.4: Run null-session test, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx -t "no session id"

Expected: 1 test passes.

  • [ ] Step 3.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 3.6: Commit
git add src/components/admin/factory/CrawlerFactory.tsx src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "feat(factory): CrawlerFactory orchestrator scaffold + null-session branch"

Task 4: CrawlerFactory: discovering / categorized / configuring branches

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 4.1: Append failing tests for discovering, categorized, configuring

Append to src/components/admin/factory/__tests__/CrawlerFactory.test.tsx (inside the same describe):

  it('renders DiscoveryPanel when session.status is discovering', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({ status: 'discovering' }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
    expect(screen.queryByTestId('stub-category-review')).not.toBeInTheDocument();
  });

  it('renders CategoryReview when session.status is categorized', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({ status: 'categorized' }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('stub-category-review')).toBeInTheDocument();
    expect(screen.queryByTestId('stub-discovery-panel')).not.toBeInTheDocument();
    expect(screen.queryByTestId('stub-category-configurator')).not.toBeInTheDocument();
  });

  it('renders CategoryReview AND CategoryConfigurator overlay when status is configuring', async () => {
    const { user } = await import('@testing-library/user-event').then((m) => ({
      user: m.default.setup(),
    }));

    // Stage 1: status='categorized', user clicks "Configure Selected"
    let currentStatus: 'categorized' | 'configuring' = 'categorized';
    // pathGroups must carry detectedDomain so the orchestrator can satisfy
    // CategoryConfiguratorProps (pathGroup + contentDomain) when the drawer
    // opens for activePathGroupId='pg_blog'.
    const configuringPathGroups = [
      {
        id: 'pg_blog',
        pattern: '/blog/*',
        urlCount: 412,
        sampleUrls: [],
        detectedDomain: 'marketing' as const,
        detectionConfidence: 0.92,
        detectionMethod: 'cms' as const,
        selected: true,
      },
      {
        id: 'pg_docs',
        pattern: '/docs/*',
        urlCount: 80,
        sampleUrls: [],
        detectedDomain: 'documentation' as const,
        detectionConfidence: 0.9,
        detectionMethod: 'cms' as const,
        selected: true,
      },
    ];
    mockedUseFactorySession.mockImplementation(() => ({
      session: minimalSession({
        status: currentStatus,
        pathGroups: configuringPathGroups,
      }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    }));

    // Override CategoryReview stub to expose its onConfigureSelected callback.
    vi.doMock('../CategoryReview', () => ({
      CategoryReview: ({
        onConfigureSelected,
      }: {
        onConfigureSelected: (ids: string[]) => void;
      }) => (
        <button
          type="button"
          data-testid="stub-configure-selected-cta"
          onClick={() => onConfigureSelected(['pg_blog', 'pg_docs'])}
        >
          configure
        </button>
      ),
    }));

    vi.resetModules();
    const { CrawlerFactory: ReloadedCrawlerFactory } = await import('../CrawlerFactory');

    const { rerender } = render(
      <MemoryRouter initialEntries={['/admin/factory?session=sess_001']}>
        <ReloadedCrawlerFactory />
      </MemoryRouter>
    );

    await user.click(screen.getByTestId('stub-configure-selected-cta'));

    // After click + status flips to 'configuring' server-side
    currentStatus = 'configuring';
    rerender(
      <MemoryRouter initialEntries={['/admin/factory?session=sess_001']}>
        <ReloadedCrawlerFactory />
      </MemoryRouter>
    );

    expect(screen.getByTestId('stub-category-configurator')).toHaveTextContent('pg_blog');
  });

Add at the top of the file (under the imports, above describe):

import type { FactorySession } from '@/../lib/factory/types';

function minimalSession(overrides: Partial<FactorySession> = {}): FactorySession {
  return {
    objectID: 'sess_001',
    record_type: 'session',
    status: 'discovering',
    source_domain: 'algolia.com',
    source_url_root: 'https://www.algolia.com',
    createdAt: 1714432200000,
    updatedAt: 1714432200000,
    discovery: {
      sitemapUrls: [],
      urlCount: 0,
      urlShardCount: 0,
      schemaOrgCoverage: 0,
    },
    pathGroups: [],
    ...overrides,
  };
}
  • [ ] Step 4.2: Install @testing-library/user-event if not already present
node -e "console.log(require('@testing-library/user-event/package.json').version)" || npm install --save-dev @testing-library/user-event@^14.5.2

Expected: prints a version (already installed) OR installs and prints 14.5.x.

  • [ ] Step 4.3: Run, expect PASS for the new tests
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

Expected: 4 tests pass total (null-session + discovering + categorized + configuring queue).

  • [ ] Step 4.4: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 4.5: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx package.json package-lock.json
git commit -m "test(factory): orchestrator FSM transitions discovering → categorized → configuring"

Task 5: CrawlerFactory: crawling / done / error branches

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 5.1: Append failing tests for crawling, done, error, loading

Append to the same describe in src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:

  it('renders LiveCrawlMonitor when session.status is crawling', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({
        status: 'crawling',
        pathGroups: [
          {
            id: 'pg_blog',
            pattern: '/blog/*',
            urlCount: 412,
            sampleUrls: [],
            detectedDomain: 'marketing',
            detectionConfidence: 0.92,
            detectionMethod: 'json-ld',
            selected: true,
            blueprint: {
              indexName: 'algoliacentral_marketing',
              recordExtractor: 'crawler-configs/x.com/marketing-blog.js',
              crawlerConfig: {
                renderJavaScript: false,
                rateLimit: 8,
                pathsToMatch: ['/blog/*'],
              },
              crawlerId: '9f3ed-aaa',
            },
          },
        ],
      }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });

    renderWithUrl('/admin/factory?session=sess_001');
    // LiveCrawlMonitor itself is NOT stubbed in this test file — verify by its
    // own data-testid (rendered for any non-empty crawler list).
    expect(screen.getByTestId('live-crawl-monitor')).toBeInTheDocument();
    expect(screen.getByTestId('crawler-block-9f3ed-aaa')).toBeInTheDocument();
  });

  it('renders LiveCrawlMonitor + restart button when session.status is done', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({ status: 'done' }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('restart-button')).toBeInTheDocument();
    expect(screen.getByText(/start new session/i)).toBeInTheDocument();
  });

  it('renders ErrorPanel when session.status is error', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({ status: 'error' }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('error-panel')).toBeInTheDocument();
    expect(screen.getByTestId('error-restart-button')).toBeInTheDocument();
  });

  it('renders LoadingPanel when isLoading is true', () => {
    mockedUseFactorySession.mockReturnValue({
      session: null,
      isLoading: true,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('loading-panel')).toBeInTheDocument();
  });

  it('renders ErrorPanel when useFactorySession returns an error', () => {
    mockedUseFactorySession.mockReturnValue({
      session: null,
      isLoading: false,
      error: new Error('Network failure'),
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_001');
    expect(screen.getByTestId('error-panel')).toBeInTheDocument();
    expect(screen.getByText(/Network failure/i)).toBeInTheDocument();
  });
  • [ ] Step 5.2: Remove vi.mock('../LiveCrawlMonitor') from this test file

The "crawling" test asserts on the real LiveCrawlMonitor (so data-testid="live-crawl-monitor" is the real component's id). The current mock list in this file does NOT mock LiveCrawlMonitor: confirm by re-reading the imports section. If you accidentally added one, remove it. The real LiveCrawlMonitor uses the already-mocked useCrawlProgress hook, so no network is hit.

  • [ ] Step 5.3: Run all CrawlerFactory tests, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

Expected: 9 tests pass (null + discovering + categorized + configuring + crawling + done + error + loading + hook-error).

  • [ ] Step 5.4: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 5.5: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "test(factory): orchestrator covers crawling / done / error / loading branches"

Task 6: CrawlerFactory: URL ?session=<id> resume + discovery handoff

Files: - Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

  • [ ] Step 6.1: Append failing tests for URL resume + discovery → URL handoff

Append to the same describe:

  it('reads session id from ?session=<id> and passes it to useFactorySession', () => {
    mockedUseFactorySession.mockReturnValue({
      session: minimalSession({ status: 'categorized', objectID: 'sess_resume_42' }),
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory?session=sess_resume_42');
    expect(mockedUseFactorySession).toHaveBeenCalledWith('sess_resume_42');
    expect(screen.getByTestId('stub-category-review')).toBeInTheDocument();
  });

  it('passes null to useFactorySession when no ?session=<id> present', () => {
    mockedUseFactorySession.mockReturnValue({
      session: null,
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    renderWithUrl('/admin/factory');
    expect(mockedUseFactorySession).toHaveBeenCalledWith(null);
  });

  it('lifts discoveryStream.sessionId into ?session=<id> when discovery completes', () => {
    // No session in URL initially; discovery stream emits a sessionId.
    mockedUseFactorySession.mockReturnValue({
      session: null,
      isLoading: false,
      error: null,
      refetch: vi.fn(),
    });
    mockedUseDiscoveryStream.mockReturnValue({
      logs: [],
      urls: [],
      pathGroups: [],
      isStreaming: false,
      sessionId: 'sess_new_99',
      error: null,
      discover: vi.fn(),
    });

    // Use a custom router so we can read the resulting search params.
    function LocationProbe(): JSX.Element {
      const [params] = useSearchParams();
      return <div data-testid="probe">{params.get('session') ?? 'none'}</div>;
    }

    render(
      <MemoryRouter initialEntries={['/admin/factory']}>
        <CrawlerFactory />
        <LocationProbe />
      </MemoryRouter>
    );

    // useEffect runs after render; the URL should now carry session=sess_new_99.
    expect(screen.getByTestId('probe')).toHaveTextContent('sess_new_99');
  });

Add to the imports at the top of the test file (next to MemoryRouter):

import { useSearchParams } from 'react-router-dom';
  • [ ] Step 6.2: Run, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx

Expected: 12 tests pass total.

  • [ ] Step 6.3: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 6.4: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "test(factory): orchestrator resumes from ?session=<id> and lifts discovery sessionId into URL"

Task 7: Wire CrawlerFactory into AdminFactory page shell

Files: - Modify: src/pages/AdminFactory.tsx

The Spec 09 placeholder is an empty container under the page header. Replace its body with <CrawlerFactory />.

  • [ ] Step 7.1: Read the current AdminFactory.tsx to confirm its shape
cat src/pages/AdminFactory.tsx

Expected: a default-exported page component with header + framer-motion shell, body roughly <div className="container mx-auto..."><!-- placeholder --></div>. (Spec 09 left this empty container intentionally.)

  • [ ] Step 7.2: Replace the empty container body with <CrawlerFactory />

Open src/pages/AdminFactory.tsx. Inside the page's main content <div> (the one currently empty per Spec 09), import and render <CrawlerFactory />. Keep the existing header, framer-motion fade-up wrapper, and <NodeNetwork /> background untouched.

// Add at the top of the file (next to other imports):
import { CrawlerFactory } from '@/components/admin/factory/CrawlerFactory';

// Inside the page body, replace the empty placeholder container:
<div className="container mx-auto max-w-7xl px-6 py-10">
  <CrawlerFactory />
</div>

(If the existing container in your codebase uses different Tailwind classes, keep them: only swap the children.)

  • [ ] Step 7.3: Add the page-level smoke test

Create src/pages/__tests__/AdminFactory.test.tsx:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { MemoryRouter } from 'react-router-dom';

vi.mock('@/hooks/factory/useFactorySession', () => ({
  useFactorySession: vi.fn(() => ({
    session: null,
    isLoading: false,
    error: null,
    refetch: vi.fn(),
  })),
}));
vi.mock('@/hooks/factory/useDiscoveryStream', () => ({
  useDiscoveryStream: vi.fn(() => ({
    logs: [],
    urls: [],
    pathGroups: [],
    isStreaming: false,
    sessionId: null,
    error: null,
    discover: vi.fn(),
  })),
}));
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
  useCrawlProgress: vi.fn(() => ({
    urlsCrawled: 0,
    urlsTotal: 0,
    urlsPerSec: 0,
    success: 0,
    failed: 0,
    ignored: 0,
    status: 'pending',
    error: null,
  })),
}));
vi.mock('@/components/admin/factory/DiscoveryPanel', () => ({
  DiscoveryPanel: () => <div data-testid="stub-discovery-panel" />,
}));
vi.mock('@/components/admin/factory/CategoryReview', () => ({
  CategoryReview: () => <div data-testid="stub-category-review" />,
}));
vi.mock('@/components/admin/factory/CategoryConfigurator', () => ({
  CategoryConfigurator: () => <div data-testid="stub-category-configurator" />,
}));
vi.mock('@/components/admin/factory/RollingLog', () => ({
  RollingLog: () => <div data-testid="stub-rolling-log" />,
}));

import AdminFactory from '../AdminFactory';

describe('AdminFactory page', () => {
  it('mounts CrawlerFactory inside the page shell', () => {
    render(
      <MemoryRouter initialEntries={['/admin/factory']}>
        <AdminFactory />
      </MemoryRouter>
    );
    expect(screen.getByTestId('crawler-factory')).toBeInTheDocument();
    expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
    expect(screen.getByTestId('stub-rolling-log')).toBeInTheDocument();
  });
});
  • [ ] Step 7.4: Run the page test, expect PASS
npx vitest run src/pages/__tests__/AdminFactory.test.tsx

Expected: 1 test passes.

  • [ ] Step 7.5: Run all factory frontend tests + tsc
npx vitest run src/components/admin/factory/__tests__/ src/pages/__tests__/AdminFactory.test.tsx && npx tsc --noEmit

Expected: all factory frontend tests pass; tsc clean.

  • [ ] Step 7.6: Commit
git add src/pages/AdminFactory.tsx src/pages/__tests__/AdminFactory.test.tsx
git commit -m "feat(factory): mount CrawlerFactory inside AdminFactory page shell"

Task 8: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 8.1: Run the full project test suite
npx vitest run

Expected: prior baseline tests + new factory tests, all GREEN. New tests added by this spec: ~17 (4 LiveCrawlMonitor + 12 CrawlerFactory + 1 AdminFactory page).

  • [ ] Step 8.2: Run full tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 8.3: Manual smoke (optional, only if dev server running)
npm run dev

Then open http://localhost:5173/admin/factory. Expect: header + DiscoveryPanel + RollingLog. URL should have no query param yet.

Refresh with ?session=does-not-exist. Expect: ErrorPanel rendered (or LoadingPanel if the API takes a moment), with a "Restart" button that clears the URL param.

  • [ ] Step 8.4: Mark Spec 12 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 12 (Frontend live monitor + orchestrator) | ⏸ | LiveCrawlMonitor + CrawlerFactory FSM driver |
+ | Cluster 12 (Frontend live monitor + orchestrator) | ✅ | LiveCrawlMonitor + CrawlerFactory FSM driver |

Acceptance criteria: Spec 12 done means:

  1. src/components/admin/factory/LiveCrawlMonitor.tsx exists, exports LiveCrawlMonitor and types LiveCrawlMonitorProps, LiveCrawlMonitorCrawler
  2. ✅ Empty crawlerIds renders empty-state copy (No active crawls)
  3. ✅ Non-empty crawlerIds renders one block per crawler with: name, mono crawlerId, DomainBadge, indexName, shadcn Progress bar with proper aria attributes, URLs/sec, success/failed/ignored counters
  4. ✅ Each block subscribes to useCrawlProgress(crawlerId) (one hook call per crawlerId)
  5. src/components/admin/factory/CrawlerFactory.tsx exists and exports CrawlerFactory
  6. ✅ Reads ?session=<id> via useSearchParams and passes to useFactorySession(sessionId)
  7. ✅ FSM rendering verified by tests: null/discovering → DiscoveryPanel; categorized → CategoryReview; configuring → CategoryReview + CategoryConfigurator drawer; crawling → LiveCrawlMonitor; done → LiveCrawlMonitor + Restart; error → ErrorPanel; isLoading → LoadingPanel
  8. ✅ When discovery completes (useDiscoveryStream emits sessionId), URL is updated to ?session=<id> so refresh resumes correctly
  9. ✅ Configurator queue: clicking "Configure Selected" with N pathGroups walks them one at a time; each commit advances the queue
  10. ✅ Right column ALWAYS renders <RollingLog />; layout is 2-column grid (left ~60%, right ~40%) on lg breakpoint
  11. src/pages/AdminFactory.tsx mounts <CrawlerFactory /> (Spec 09 placeholder replaced)
  12. ✅ All new tests pass (~17 added by this spec)
  13. ✅ Full project test suite still GREEN
  14. tsc --noEmit clean
  15. ✅ Per CodingSOPs: every component has docstring, no console.log, no any, components ≤20-line functions where practical, mock-only-hooks unit tests
  16. ✅ Per UIUXDesignSOP / Plan §5a: aesthetic matches /admin (bg-[#0a0a0f], white/10 borders, font-black uppercase tracking-tighter, mono for IDs/URLs, framer-motion fade-up)

Out of scope (handled by other specs)

  • DiscoveryPanel, CategoryReview, RollingLog, DomainBadge implementations → Spec 10
  • CategoryConfigurator drawer + sub-components (SamplePreview, ConfigEditor, TestCrawlResults) → Spec 11
  • useFactorySession, useDiscoveryStream, useCrawlProgress hooks + page shell AdminFactory.tsx (without CrawlerFactory mount) → Spec 09
  • /api/factory/session and /api/factory/crawl-progress endpoints → Spec 08
  • FactorySession, PathGroup zod schemas → Spec 01
  • E2E browser smoke test (real backend hit) → Spec 13
  • Bundle build (scripts/bundle-api.mjs) + vault sync + Status.md toggle for the whole factory → Spec 13
  • The /admin/factory route registration in src/App.tsx → Spec 09 (already added there per Task 22)