Engineering-Specs/12-frontend-orchestrator.md
Crawler Factory: Spec 12: Frontend Live Monitor + Orchestrator Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Ship the live-crawl progress panel and the top-level FSM-driving orchestrator that wires every prior frontend cluster (Spec 09 hooks, Spec 10 discovery + category review + RollingLog, Spec 11 category configurator drawer) into a single resumable URL-driven flow at /admin/factory.
Architecture: LiveCrawlMonitor is a pure presentational component: one stacked block per active crawler, each block driven by useCrawlProgress(crawlerId) from Spec 09 (SSE polling against /api/factory/crawl-progress). CrawlerFactory is the FSM orchestrator: it reads ?session=<id> from the URL, loads session state via useFactorySession, switches the left column to the component matching session.status (null|discovering|categorized|configuring|crawling|done|error), and always renders the persistent RollingLog on the right column from the session's stored log entries. URL-as-state means the user can refresh, share, or come back tomorrow and resume exactly where they left off. The page shell AdminFactory.tsx (placeholder from Spec 09) is replaced with <CrawlerFactory />.
Tech Stack: React 18 + TypeScript (strict), react-router-dom v6 (useSearchParams for URL state), shadcn UI (Progress, Sheet, Card), framer-motion (existing /admin aesthetic: fade-up on mount), TanStack Query (already wired by useFactorySession), @testing-library/react + vitest for tests, MSW-free testing (mock the hooks directly per Spec 09 pattern).
Depends on:
- Spec 09: hooks useFactorySession, useDiscoveryStream, useCrawlProgress; placeholder AdminFactory.tsx page shell
- Spec 10: RollingLog, DomainBadge, DiscoveryPanel, CategoryReview
- Spec 11: CategoryConfigurator drawer
- Spec 08: /api/factory/session (CRUD) and /api/factory/crawl-progress (SSE) endpoints
- Spec 01: FactorySession, PathGroup types from lib/factory/types.ts
Consumed by: - Spec 13: bundle + smoke test cluster (E2E flow validation, vault sync, docs)
Plan source: Projects/Crawler-Factory/00-Plan.md §2 (FSM), §5b (page layout), §7 Tasks 28, 29.
File structure
| Path | Responsibility |
|---|---|
src/components/admin/factory/LiveCrawlMonitor.tsx |
Stacked progress blocks: one per active crawler. Pure presentation; subscribes via useCrawlProgress(crawlerId) per block. |
src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx |
RTL tests: renders one block per crawler; renders header + progress + counters per block; handles empty list. |
src/components/admin/factory/CrawlerFactory.tsx |
Top-level FSM orchestrator. Reads ?session=<id> from URL, loads session, switches left-column component on session.status, always renders RollingLog on right column. |
src/components/admin/factory/__tests__/CrawlerFactory.test.tsx |
RTL tests: renders correct component per status; URL ?session=<id> resumes correctly; null → discovering → categorized → configuring → crawling → done flow. |
src/pages/AdminFactory.tsx |
Modify: replace empty container from Spec 09 with <CrawlerFactory />. Header + framer-motion shell stays. |
SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, UIUXDesignSOPs.md)
Every file in this spec MUST follow these rules. They are not optional.
- Two-layer type safety. zod for runtime boundary validation (already done by hooks reading API);
tsc --noEmitstrict for write-time. Noany. UseT | null, notT | undefined, for absent/sentinel values. - No
console.log. UI-side logging goes through the sharedlib/utils/loggerimport only when a real signal is needed (errors caught at the boundary). Component bodies do not log routine state. - Try/catch every fallible call. API errors surfaced from hooks are caught and routed to the
errorFSM branch: never silently swallowed. - Functions ≤20 lines, ≤3 params. Sub-render helpers split out when a JSX block exceeds 20 lines or props exceed 3.
- Docstring on every exported component. One-line JSDoc explaining purpose. Comment WHY, not WHAT.
- No raw dicts crossing module boundaries. Component props are typed against
FactorySession,PathGroupfromlib/factory/types.ts. - TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
- Mock hooks in unit tests.
vi.mock('@/hooks/factory/useFactorySession')etc.: no real network, no real timers. - Naming. TypeScript:
camelCasefor vars/functions,PascalCasefor components and types,UPPER_SNAKE_CASEfor module constants. Files:PascalCase.tsx(components),kebab-case.ts(utils). Tests: same name as subject +.test.tsx. - Aesthetic conformance. Match
/adminaesthetic exactly (Plan §5a):bg-[#0a0a0f],border-white/10,font-black uppercase tracking-tighterheaders, mono font for IDs/URLs/timestamps, framer-motion fade-up on section mount. - Accessibility. Every interactive element keyboard-reachable; ARIA labels on icon-only buttons; progress bars have
aria-valuenow/aria-valuemin/aria-valuemax. - URL is the source of truth. Session state is
?session=<id>query param. No client-only state for the session id. Resume on refresh works because the URL drives the session load. - Conventional commits.
feat(factory):,test(factory):,chore(factory):. One logical change per commit.
Task 1: LiveCrawlMonitor: failing test for empty crawler list
Files:
- Create: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
LiveCrawlMonitor receives crawlerIds: Array<{crawlerId, name, contentDomain, indexName}>. With an empty array it renders an empty-state message. With N entries it renders N stacked blocks. Each block reads its own progress via useCrawlProgress(crawlerId) (Spec 09) and shows: header (name + crawlerId mono + DomainBadge + indexName), shadcn Progress bar (0–100%), URLs/sec ticker, success/failed/ignored counters.
- [ ] Step 1.1: Create the test file with empty-state test
Create src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx:
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { LiveCrawlMonitor } from '../LiveCrawlMonitor';
// Mock the Spec 09 hook before importing the component under test.
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
useCrawlProgress: vi.fn(),
}));
import { useCrawlProgress } from '@/hooks/factory/useCrawlProgress';
const mockedUseCrawlProgress = vi.mocked(useCrawlProgress);
describe('LiveCrawlMonitor', () => {
beforeEach(() => {
mockedUseCrawlProgress.mockReset();
mockedUseCrawlProgress.mockReturnValue({
urlsCrawled: 0,
urlsTotal: 0,
urlsPerSec: 0,
success: 0,
failed: 0,
ignored: 0,
status: 'pending',
error: null,
});
});
it('renders empty-state copy when crawlerIds is empty', () => {
render(<LiveCrawlMonitor crawlerIds={[]} />);
expect(screen.getByText(/no active crawls/i)).toBeInTheDocument();
});
});
- [ ] Step 1.2: Run, expect FAIL
npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
Expected: FAIL: Cannot find module '../LiveCrawlMonitor' (component not yet implemented).
- [ ] Step 1.3: Implement LiveCrawlMonitor minimally to pass the empty-state test
Create src/components/admin/factory/LiveCrawlMonitor.tsx:
/**
* LiveCrawlMonitor — stacked progress blocks, one per active crawler.
*
* Why: during a multi-crawler session, the user needs at-a-glance visibility
* into every crawler the factory just created. Each block subscribes to its
* own SSE stream via useCrawlProgress(crawlerId).
*/
import { motion } from 'framer-motion';
import { Progress } from '@/components/ui/progress';
import { useCrawlProgress } from '@/hooks/factory/useCrawlProgress';
import { DomainBadge } from './DomainBadge';
import type { ContentDomain } from '@/../lib/factory/types';
export interface LiveCrawlMonitorCrawler {
crawlerId: string;
name: string;
contentDomain: ContentDomain;
indexName: string;
}
export interface LiveCrawlMonitorProps {
crawlerIds: LiveCrawlMonitorCrawler[];
}
/**
* Renders one block per crawler. Empty list → empty-state copy.
*/
export function LiveCrawlMonitor({ crawlerIds }: LiveCrawlMonitorProps): JSX.Element {
if (crawlerIds.length === 0) {
return (
<div
className="rounded-lg border border-white/10 bg-[#0a0a0f] p-6 text-center"
data-testid="live-crawl-monitor-empty"
>
<p className="text-sm font-bold uppercase tracking-widest text-white/40">
No active crawls
</p>
<p className="mt-2 text-xs text-white/60">
Start by configuring a category — crawler progress will appear here.
</p>
</div>
);
}
return (
<div className="flex flex-col gap-4" data-testid="live-crawl-monitor">
{crawlerIds.map((c) => (
<CrawlerBlock key={c.crawlerId} crawler={c} />
))}
</div>
);
}
interface CrawlerBlockProps {
crawler: LiveCrawlMonitorCrawler;
}
function CrawlerBlock({ crawler }: CrawlerBlockProps): JSX.Element {
const progress = useCrawlProgress(crawler.crawlerId);
const pct =
progress.urlsTotal > 0
? Math.min(100, Math.round((progress.urlsCrawled / progress.urlsTotal) * 100))
: 0;
return (
<motion.div
initial={{ opacity: 0, y: 8 }}
animate={{ opacity: 1, y: 0 }}
className="rounded-lg border border-white/10 bg-[#0a0a0f] p-4"
data-testid={`crawler-block-${crawler.crawlerId}`}
>
<div className="flex items-center justify-between gap-3">
<div className="flex items-center gap-3">
<span className="text-sm font-black uppercase tracking-tighter text-white">
{crawler.name}
</span>
<span className="font-mono text-[10px] text-white/40">{crawler.crawlerId}</span>
<DomainBadge domain={crawler.contentDomain} confidence={null} />
</div>
<span className="font-mono text-[10px] text-white/40">{crawler.indexName}</span>
</div>
<div className="mt-3">
<Progress
value={pct}
aria-valuenow={pct}
aria-valuemin={0}
aria-valuemax={100}
aria-label={`${crawler.name} progress`}
/>
<div className="mt-2 flex items-center justify-between text-[11px] text-white/60">
<span data-testid={`progress-fraction-${crawler.crawlerId}`}>
{progress.urlsCrawled} / {progress.urlsTotal} URLs ({pct}%)
</span>
<span data-testid={`urls-per-sec-${crawler.crawlerId}`}>
{progress.urlsPerSec.toFixed(1)} URLs/sec
</span>
</div>
</div>
<div className="mt-3 flex items-center gap-4 text-[11px]">
<span className="text-emerald-400" data-testid={`success-${crawler.crawlerId}`}>
✓ {progress.success} success
</span>
<span className="text-rose-400" data-testid={`failed-${crawler.crawlerId}`}>
✗ {progress.failed} failed
</span>
<span className="text-white/40" data-testid={`ignored-${crawler.crawlerId}`}>
⊘ {progress.ignored} ignored
</span>
</div>
</motion.div>
);
}
- [ ] Step 1.4: Run empty-state test, expect PASS
npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
Expected: 1 test passes.
- [ ] Step 1.5: Commit
git add src/components/admin/factory/LiveCrawlMonitor.tsx src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
git commit -m "feat(factory): LiveCrawlMonitor empty-state shell"
Task 2: LiveCrawlMonitor: render one block per crawler
Files:
- Modify: src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
- [ ] Step 2.1: Append failing test for "renders one block per crawler"
Append to src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx (inside the same describe):
it('renders one block per crawler with header info', () => {
mockedUseCrawlProgress.mockReturnValue({
urlsCrawled: 38,
urlsTotal: 412,
urlsPerSec: 4.2,
success: 35,
failed: 0,
ignored: 3,
status: 'running',
error: null,
});
render(
<LiveCrawlMonitor
crawlerIds={[
{
crawlerId: '9f3ed-aaa',
name: 'Blog',
contentDomain: 'marketing',
indexName: 'algoliacentral_marketing',
},
{
crawlerId: '7a1bc-bbb',
name: 'Customer Stories',
contentDomain: 'customer-stories',
indexName: 'algoliacentral_customers',
},
]}
/>
);
expect(screen.getByTestId('crawler-block-9f3ed-aaa')).toBeInTheDocument();
expect(screen.getByTestId('crawler-block-7a1bc-bbb')).toBeInTheDocument();
expect(screen.getByText('Blog')).toBeInTheDocument();
expect(screen.getByText('Customer Stories')).toBeInTheDocument();
expect(screen.getByText('9f3ed-aaa')).toBeInTheDocument();
expect(screen.getByText('algoliacentral_marketing')).toBeInTheDocument();
});
it('renders progress fraction, URLs/sec, and counters per block', () => {
mockedUseCrawlProgress.mockReturnValue({
urlsCrawled: 38,
urlsTotal: 412,
urlsPerSec: 4.2,
success: 35,
failed: 0,
ignored: 3,
status: 'running',
error: null,
});
render(
<LiveCrawlMonitor
crawlerIds={[
{
crawlerId: '9f3ed-aaa',
name: 'Blog',
contentDomain: 'marketing',
indexName: 'algoliacentral_marketing',
},
]}
/>
);
expect(screen.getByTestId('progress-fraction-9f3ed-aaa')).toHaveTextContent(
'38 / 412 URLs (9%)'
);
expect(screen.getByTestId('urls-per-sec-9f3ed-aaa')).toHaveTextContent('4.2 URLs/sec');
expect(screen.getByTestId('success-9f3ed-aaa')).toHaveTextContent('35 success');
expect(screen.getByTestId('failed-9f3ed-aaa')).toHaveTextContent('0 failed');
expect(screen.getByTestId('ignored-9f3ed-aaa')).toHaveTextContent('3 ignored');
});
it('renders 0% progress when urlsTotal is 0', () => {
mockedUseCrawlProgress.mockReturnValue({
urlsCrawled: 0,
urlsTotal: 0,
urlsPerSec: 0,
success: 0,
failed: 0,
ignored: 0,
status: 'pending',
error: null,
});
render(
<LiveCrawlMonitor
crawlerIds={[
{
crawlerId: 'p1',
name: 'Pending',
contentDomain: 'support',
indexName: 'algoliacentral_support',
},
]}
/>
);
const progressBar = screen.getByLabelText('Pending progress');
expect(progressBar).toHaveAttribute('aria-valuenow', '0');
expect(screen.getByTestId('progress-fraction-p1')).toHaveTextContent('0 / 0 URLs (0%)');
});
- [ ] Step 2.2: Run, expect PASS
The implementation from Task 1 already covers these: running them confirms.
npx vitest run src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
Expected: 4 tests pass.
- [ ] Step 2.3: Run
tsc --noEmit
npx tsc --noEmit
Expected: clean.
- [ ] Step 2.4: Commit
git add src/components/admin/factory/__tests__/LiveCrawlMonitor.test.tsx
git commit -m "test(factory): LiveCrawlMonitor renders blocks + progress + counters"
Task 3: CrawlerFactory orchestrator: scaffold + null-session test
Files:
- Create: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
- Create: src/components/admin/factory/CrawlerFactory.tsx
CrawlerFactory is the FSM driver. It reads ?session=<id> from the URL via useSearchParams. It loads session state via useFactorySession(sessionId). It picks the left-column component based on session.status:
| status | left column | notes |
|---|---|---|
null (no session id in URL) |
<DiscoveryPanel /> |
On done event from useDiscoveryStream, set ?session=<id> and continue. |
'discovering' |
<DiscoveryPanel /> with active stream |
|
'categorized' |
<CategoryReview /> |
On "Configure Selected", queue pathGroups + open drawer one at a time. |
'configuring' |
<CategoryReview /> + <CategoryConfigurator /> overlay |
On commit, advance to next pathGroup; when queue empty, transition to 'crawling'. |
'crawling' |
<LiveCrawlMonitor /> |
|
'done' |
<LiveCrawlMonitor /> + "Start new session" button |
|
'error' |
error panel + retry/restart options |
The right column ALWAYS renders <RollingLog /> driven by session log entries. Layout: 2-column grid (left 60%, right 40%).
- [ ] Step 3.1: Write failing test for "no session id → renders DiscoveryPanel"
Create src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { MemoryRouter } from 'react-router-dom';
// ─── Mocks ──────────────────────────────────────────────────────────────────
vi.mock('@/hooks/factory/useFactorySession', () => ({
useFactorySession: vi.fn(),
}));
vi.mock('@/hooks/factory/useDiscoveryStream', () => ({
useDiscoveryStream: vi.fn(),
}));
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
useCrawlProgress: vi.fn(() => ({
urlsCrawled: 0,
urlsTotal: 0,
urlsPerSec: 0,
success: 0,
failed: 0,
ignored: 0,
status: 'pending',
error: null,
})),
}));
// Stub child components so each test asserts which one is rendered, not its content.
vi.mock('../DiscoveryPanel', () => ({
DiscoveryPanel: () => <div data-testid="stub-discovery-panel" />,
}));
vi.mock('../CategoryReview', () => ({
CategoryReview: () => <div data-testid="stub-category-review" />,
}));
vi.mock('../CategoryConfigurator', () => ({
CategoryConfigurator: ({ pathGroupId }: { pathGroupId: string }) => (
<div data-testid="stub-category-configurator">{pathGroupId}</div>
),
}));
vi.mock('../RollingLog', () => ({
RollingLog: () => <div data-testid="stub-rolling-log" />,
}));
import { CrawlerFactory } from '../CrawlerFactory';
import { useFactorySession } from '@/hooks/factory/useFactorySession';
import { useDiscoveryStream } from '@/hooks/factory/useDiscoveryStream';
const mockedUseFactorySession = vi.mocked(useFactorySession);
const mockedUseDiscoveryStream = vi.mocked(useDiscoveryStream);
function renderWithUrl(url: string) {
return render(
<MemoryRouter initialEntries={[url]}>
<CrawlerFactory />
</MemoryRouter>
);
}
describe('CrawlerFactory orchestrator', () => {
beforeEach(() => {
mockedUseFactorySession.mockReset();
mockedUseDiscoveryStream.mockReset();
mockedUseFactorySession.mockReturnValue({
session: null,
isLoading: false,
error: null,
refetch: vi.fn(),
});
mockedUseDiscoveryStream.mockReturnValue({
logs: [],
urls: [],
pathGroups: [],
isStreaming: false,
sessionId: null,
error: null,
discover: vi.fn(),
});
});
it('renders DiscoveryPanel + RollingLog when no session id in URL', () => {
renderWithUrl('/admin/factory');
expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
expect(screen.getByTestId('stub-rolling-log')).toBeInTheDocument();
expect(screen.queryByTestId('stub-category-review')).not.toBeInTheDocument();
expect(screen.queryByTestId('stub-category-configurator')).not.toBeInTheDocument();
});
});
- [ ] Step 3.2: Run, expect FAIL
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
Expected: FAIL: Cannot find module '../CrawlerFactory'.
- [ ] Step 3.3: Implement CrawlerFactory minimally: null-session branch only
Create src/components/admin/factory/CrawlerFactory.tsx:
/**
* CrawlerFactory — top-level FSM orchestrator for /admin/factory.
*
* Why: the factory is a multi-stage flow (discover → categorize → configure
* → crawl). State lives server-side in algoliacentral_factory_sessions; the
* URL ?session=<id> is the resume key. This component reads the session and
* renders the component matching session.status. RollingLog is always-on
* in the right column for narrative continuity.
*/
import { useEffect, useMemo, useState } from 'react';
import { useSearchParams } from 'react-router-dom';
import { motion } from 'framer-motion';
import { useFactorySession } from '@/hooks/factory/useFactorySession';
import { useDiscoveryStream } from '@/hooks/factory/useDiscoveryStream';
import { DiscoveryPanel } from './DiscoveryPanel';
import { CategoryReview } from './CategoryReview';
import { CategoryConfigurator } from './CategoryConfigurator';
import { LiveCrawlMonitor, type LiveCrawlMonitorCrawler } from './LiveCrawlMonitor';
import { RollingLog } from './RollingLog';
import type { PathGroup } from '@/../lib/factory/types';
/**
* Top-level FSM orchestrator. Reads ?session=<id> and switches the left
* column to the component matching session.status.
*/
export function CrawlerFactory(): JSX.Element {
const [searchParams, setSearchParams] = useSearchParams();
const sessionId = searchParams.get('session');
const { session, isLoading, error, refetch } = useFactorySession(sessionId);
const discoveryStream = useDiscoveryStream();
// When discovery completes, lift its sessionId into the URL so resume works.
useEffect(() => {
if (!sessionId && discoveryStream.sessionId) {
setSearchParams({ session: discoveryStream.sessionId });
}
}, [sessionId, discoveryStream.sessionId, setSearchParams]);
// Configurator drawer queue: array of pathGroup ids to walk in order.
const [configQueue, setConfigQueue] = useState<string[]>([]);
const activePathGroupId = configQueue[0] ?? null;
const beginConfiguration = (selectedIds: string[]): void => {
setConfigQueue(selectedIds);
};
const advanceQueue = (): void => {
setConfigQueue((q) => q.slice(1));
refetch();
};
const restart = (): void => {
setSearchParams({});
setConfigQueue([]);
};
const liveCrawlers = useMemo<LiveCrawlMonitorCrawler[]>(
() => extractLiveCrawlers(session?.pathGroups ?? []),
[session?.pathGroups]
);
const logs = session?.pathGroups ? [] : discoveryStream.logs;
// Spec 10's RollingLog accepts the union of session-stored logs + the
// active discovery stream's transient logs. We pass logs through unchanged.
const rollingLogEntries = sessionId ? logs : discoveryStream.logs;
return (
<motion.div
initial={{ opacity: 0, y: 12 }}
animate={{ opacity: 1, y: 0 }}
className="grid grid-cols-1 gap-6 lg:grid-cols-[3fr_2fr]"
data-testid="crawler-factory"
>
<div className="min-w-0">
<LeftColumn
isLoading={isLoading}
error={error}
status={session?.status ?? null}
sessionId={session?.objectID ?? sessionId ?? ''}
pathGroups={session?.pathGroups ?? []}
activePathGroupId={activePathGroupId}
liveCrawlers={liveCrawlers}
onConfigureSelected={beginConfiguration}
onCommitDrawer={advanceQueue}
onRestart={restart}
/>
</div>
<div className="min-w-0">
<RollingLog entries={rollingLogEntries} />
</div>
</motion.div>
);
}
interface LeftColumnProps {
isLoading: boolean;
error: Error | null;
status: string | null;
sessionId: string;
pathGroups: PathGroup[];
activePathGroupId: string | null;
liveCrawlers: LiveCrawlMonitorCrawler[];
onConfigureSelected: (selectedIds: string[]) => void;
onCommitDrawer: () => void;
onRestart: () => void;
}
function LeftColumn(props: LeftColumnProps): JSX.Element {
if (props.isLoading) {
return <LoadingPanel />;
}
if (props.error) {
return <ErrorPanel error={props.error} onRestart={props.onRestart} />;
}
switch (props.status) {
case null:
case 'discovering':
return <DiscoveryPanel />;
case 'categorized':
return (
<CategoryReview
pathGroups={props.pathGroups}
onConfigureSelected={props.onConfigureSelected}
/>
);
case 'configuring': {
const activePathGroup = props.activePathGroupId
? props.pathGroups.find((g) => g.id === props.activePathGroupId) ?? null
: null;
return (
<>
<CategoryReview
pathGroups={props.pathGroups}
onConfigureSelected={props.onConfigureSelected}
/>
{activePathGroup && activePathGroup.detectedDomain && (
<CategoryConfigurator
sessionId={props.sessionId}
pathGroupId={activePathGroup.id}
pathGroup={activePathGroup}
contentDomain={activePathGroup.detectedDomain}
open={!!props.activePathGroupId}
onClose={props.onCommitDrawer}
onCommit={() => props.onCommitDrawer()}
/>
)}
</>
);
}
case 'crawling':
return <LiveCrawlMonitor crawlerIds={props.liveCrawlers} />;
case 'done':
return (
<div className="flex flex-col gap-4">
<LiveCrawlMonitor crawlerIds={props.liveCrawlers} />
<button
type="button"
onClick={props.onRestart}
className="self-start rounded-md border border-white/20 bg-white/5 px-4 py-2 text-xs font-bold uppercase tracking-widest text-white hover:bg-white/10"
data-testid="restart-button"
>
Start new session
</button>
</div>
);
case 'error':
return <ErrorPanel error={new Error('Session ended in error state.')} onRestart={props.onRestart} />;
default:
return <DiscoveryPanel />;
}
}
function LoadingPanel(): JSX.Element {
return (
<div
className="rounded-lg border border-white/10 bg-[#0a0a0f] p-6 text-center"
data-testid="loading-panel"
>
<p className="text-sm font-bold uppercase tracking-widest text-white/40">Loading session…</p>
</div>
);
}
interface ErrorPanelProps {
error: Error;
onRestart: () => void;
}
function ErrorPanel({ error, onRestart }: ErrorPanelProps): JSX.Element {
return (
<div
className="rounded-lg border border-rose-500/30 bg-rose-500/10 p-6"
data-testid="error-panel"
>
<p className="text-sm font-black uppercase tracking-tighter text-rose-400">Something went wrong</p>
<p className="mt-2 text-xs text-white/70">{error.message}</p>
<button
type="button"
onClick={onRestart}
className="mt-4 rounded-md border border-white/20 bg-white/5 px-3 py-1.5 text-[11px] font-bold uppercase tracking-widest text-white hover:bg-white/10"
data-testid="error-restart-button"
>
Restart
</button>
</div>
);
}
/**
* Walk pathGroups and produce the LiveCrawlMonitor input. A pathGroup is
* "live" once its blueprint has a crawlerId attached.
*/
function extractLiveCrawlers(pathGroups: PathGroup[]): LiveCrawlMonitorCrawler[] {
const out: LiveCrawlMonitorCrawler[] = [];
for (const pg of pathGroups) {
const bp = pg.blueprint;
if (!bp || !bp.crawlerId || !pg.detectedDomain) continue;
out.push({
crawlerId: bp.crawlerId,
name: pg.pattern,
contentDomain: pg.detectedDomain,
indexName: bp.indexName,
});
}
return out;
}
- [ ] Step 3.4: Run null-session test, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx -t "no session id"
Expected: 1 test passes.
- [ ] Step 3.5: Run
tsc --noEmit
npx tsc --noEmit
Expected: clean.
- [ ] Step 3.6: Commit
git add src/components/admin/factory/CrawlerFactory.tsx src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "feat(factory): CrawlerFactory orchestrator scaffold + null-session branch"
Task 4: CrawlerFactory: discovering / categorized / configuring branches
Files:
- Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
- [ ] Step 4.1: Append failing tests for discovering, categorized, configuring
Append to src/components/admin/factory/__tests__/CrawlerFactory.test.tsx (inside the same describe):
it('renders DiscoveryPanel when session.status is discovering', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({ status: 'discovering' }),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
expect(screen.queryByTestId('stub-category-review')).not.toBeInTheDocument();
});
it('renders CategoryReview when session.status is categorized', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({ status: 'categorized' }),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('stub-category-review')).toBeInTheDocument();
expect(screen.queryByTestId('stub-discovery-panel')).not.toBeInTheDocument();
expect(screen.queryByTestId('stub-category-configurator')).not.toBeInTheDocument();
});
it('renders CategoryReview AND CategoryConfigurator overlay when status is configuring', async () => {
const { user } = await import('@testing-library/user-event').then((m) => ({
user: m.default.setup(),
}));
// Stage 1: status='categorized', user clicks "Configure Selected"
let currentStatus: 'categorized' | 'configuring' = 'categorized';
// pathGroups must carry detectedDomain so the orchestrator can satisfy
// CategoryConfiguratorProps (pathGroup + contentDomain) when the drawer
// opens for activePathGroupId='pg_blog'.
const configuringPathGroups = [
{
id: 'pg_blog',
pattern: '/blog/*',
urlCount: 412,
sampleUrls: [],
detectedDomain: 'marketing' as const,
detectionConfidence: 0.92,
detectionMethod: 'cms' as const,
selected: true,
},
{
id: 'pg_docs',
pattern: '/docs/*',
urlCount: 80,
sampleUrls: [],
detectedDomain: 'documentation' as const,
detectionConfidence: 0.9,
detectionMethod: 'cms' as const,
selected: true,
},
];
mockedUseFactorySession.mockImplementation(() => ({
session: minimalSession({
status: currentStatus,
pathGroups: configuringPathGroups,
}),
isLoading: false,
error: null,
refetch: vi.fn(),
}));
// Override CategoryReview stub to expose its onConfigureSelected callback.
vi.doMock('../CategoryReview', () => ({
CategoryReview: ({
onConfigureSelected,
}: {
onConfigureSelected: (ids: string[]) => void;
}) => (
<button
type="button"
data-testid="stub-configure-selected-cta"
onClick={() => onConfigureSelected(['pg_blog', 'pg_docs'])}
>
configure
</button>
),
}));
vi.resetModules();
const { CrawlerFactory: ReloadedCrawlerFactory } = await import('../CrawlerFactory');
const { rerender } = render(
<MemoryRouter initialEntries={['/admin/factory?session=sess_001']}>
<ReloadedCrawlerFactory />
</MemoryRouter>
);
await user.click(screen.getByTestId('stub-configure-selected-cta'));
// After click + status flips to 'configuring' server-side
currentStatus = 'configuring';
rerender(
<MemoryRouter initialEntries={['/admin/factory?session=sess_001']}>
<ReloadedCrawlerFactory />
</MemoryRouter>
);
expect(screen.getByTestId('stub-category-configurator')).toHaveTextContent('pg_blog');
});
Add at the top of the file (under the imports, above describe):
import type { FactorySession } from '@/../lib/factory/types';
function minimalSession(overrides: Partial<FactorySession> = {}): FactorySession {
return {
objectID: 'sess_001',
record_type: 'session',
status: 'discovering',
source_domain: 'algolia.com',
source_url_root: 'https://www.algolia.com',
createdAt: 1714432200000,
updatedAt: 1714432200000,
discovery: {
sitemapUrls: [],
urlCount: 0,
urlShardCount: 0,
schemaOrgCoverage: 0,
},
pathGroups: [],
...overrides,
};
}
- [ ] Step 4.2: Install
@testing-library/user-eventif not already present
node -e "console.log(require('@testing-library/user-event/package.json').version)" || npm install --save-dev @testing-library/user-event@^14.5.2
Expected: prints a version (already installed) OR installs and prints 14.5.x.
- [ ] Step 4.3: Run, expect PASS for the new tests
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
Expected: 4 tests pass total (null-session + discovering + categorized + configuring queue).
- [ ] Step 4.4: Run
tsc --noEmit
npx tsc --noEmit
Expected: clean.
- [ ] Step 4.5: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx package.json package-lock.json
git commit -m "test(factory): orchestrator FSM transitions discovering → categorized → configuring"
Task 5: CrawlerFactory: crawling / done / error branches
Files:
- Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
- [ ] Step 5.1: Append failing tests for crawling, done, error, loading
Append to the same describe in src/components/admin/factory/__tests__/CrawlerFactory.test.tsx:
it('renders LiveCrawlMonitor when session.status is crawling', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({
status: 'crawling',
pathGroups: [
{
id: 'pg_blog',
pattern: '/blog/*',
urlCount: 412,
sampleUrls: [],
detectedDomain: 'marketing',
detectionConfidence: 0.92,
detectionMethod: 'json-ld',
selected: true,
blueprint: {
indexName: 'algoliacentral_marketing',
recordExtractor: 'crawler-configs/x.com/marketing-blog.js',
crawlerConfig: {
renderJavaScript: false,
rateLimit: 8,
pathsToMatch: ['/blog/*'],
},
crawlerId: '9f3ed-aaa',
},
},
],
}),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
// LiveCrawlMonitor itself is NOT stubbed in this test file — verify by its
// own data-testid (rendered for any non-empty crawler list).
expect(screen.getByTestId('live-crawl-monitor')).toBeInTheDocument();
expect(screen.getByTestId('crawler-block-9f3ed-aaa')).toBeInTheDocument();
});
it('renders LiveCrawlMonitor + restart button when session.status is done', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({ status: 'done' }),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('restart-button')).toBeInTheDocument();
expect(screen.getByText(/start new session/i)).toBeInTheDocument();
});
it('renders ErrorPanel when session.status is error', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({ status: 'error' }),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('error-panel')).toBeInTheDocument();
expect(screen.getByTestId('error-restart-button')).toBeInTheDocument();
});
it('renders LoadingPanel when isLoading is true', () => {
mockedUseFactorySession.mockReturnValue({
session: null,
isLoading: true,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('loading-panel')).toBeInTheDocument();
});
it('renders ErrorPanel when useFactorySession returns an error', () => {
mockedUseFactorySession.mockReturnValue({
session: null,
isLoading: false,
error: new Error('Network failure'),
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_001');
expect(screen.getByTestId('error-panel')).toBeInTheDocument();
expect(screen.getByText(/Network failure/i)).toBeInTheDocument();
});
- [ ] Step 5.2: Remove
vi.mock('../LiveCrawlMonitor')from this test file
The "crawling" test asserts on the real LiveCrawlMonitor (so data-testid="live-crawl-monitor" is the real component's id). The current mock list in this file does NOT mock LiveCrawlMonitor: confirm by re-reading the imports section. If you accidentally added one, remove it. The real LiveCrawlMonitor uses the already-mocked useCrawlProgress hook, so no network is hit.
- [ ] Step 5.3: Run all CrawlerFactory tests, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
Expected: 9 tests pass (null + discovering + categorized + configuring + crawling + done + error + loading + hook-error).
- [ ] Step 5.4: Run
tsc --noEmit
npx tsc --noEmit
Expected: clean.
- [ ] Step 5.5: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "test(factory): orchestrator covers crawling / done / error / loading branches"
Task 6: CrawlerFactory: URL ?session=<id> resume + discovery handoff
Files:
- Modify: src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
- [ ] Step 6.1: Append failing tests for URL resume + discovery → URL handoff
Append to the same describe:
it('reads session id from ?session=<id> and passes it to useFactorySession', () => {
mockedUseFactorySession.mockReturnValue({
session: minimalSession({ status: 'categorized', objectID: 'sess_resume_42' }),
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory?session=sess_resume_42');
expect(mockedUseFactorySession).toHaveBeenCalledWith('sess_resume_42');
expect(screen.getByTestId('stub-category-review')).toBeInTheDocument();
});
it('passes null to useFactorySession when no ?session=<id> present', () => {
mockedUseFactorySession.mockReturnValue({
session: null,
isLoading: false,
error: null,
refetch: vi.fn(),
});
renderWithUrl('/admin/factory');
expect(mockedUseFactorySession).toHaveBeenCalledWith(null);
});
it('lifts discoveryStream.sessionId into ?session=<id> when discovery completes', () => {
// No session in URL initially; discovery stream emits a sessionId.
mockedUseFactorySession.mockReturnValue({
session: null,
isLoading: false,
error: null,
refetch: vi.fn(),
});
mockedUseDiscoveryStream.mockReturnValue({
logs: [],
urls: [],
pathGroups: [],
isStreaming: false,
sessionId: 'sess_new_99',
error: null,
discover: vi.fn(),
});
// Use a custom router so we can read the resulting search params.
function LocationProbe(): JSX.Element {
const [params] = useSearchParams();
return <div data-testid="probe">{params.get('session') ?? 'none'}</div>;
}
render(
<MemoryRouter initialEntries={['/admin/factory']}>
<CrawlerFactory />
<LocationProbe />
</MemoryRouter>
);
// useEffect runs after render; the URL should now carry session=sess_new_99.
expect(screen.getByTestId('probe')).toHaveTextContent('sess_new_99');
});
Add to the imports at the top of the test file (next to MemoryRouter):
import { useSearchParams } from 'react-router-dom';
- [ ] Step 6.2: Run, expect PASS
npx vitest run src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
Expected: 12 tests pass total.
- [ ] Step 6.3: Run
tsc --noEmit
npx tsc --noEmit
Expected: clean.
- [ ] Step 6.4: Commit
git add src/components/admin/factory/__tests__/CrawlerFactory.test.tsx
git commit -m "test(factory): orchestrator resumes from ?session=<id> and lifts discovery sessionId into URL"
Task 7: Wire CrawlerFactory into AdminFactory page shell
Files:
- Modify: src/pages/AdminFactory.tsx
The Spec 09 placeholder is an empty container under the page header. Replace its body with <CrawlerFactory />.
- [ ] Step 7.1: Read the current AdminFactory.tsx to confirm its shape
cat src/pages/AdminFactory.tsx
Expected: a default-exported page component with header + framer-motion shell, body roughly <div className="container mx-auto..."><!-- placeholder --></div>. (Spec 09 left this empty container intentionally.)
- [ ] Step 7.2: Replace the empty container body with
<CrawlerFactory />
Open src/pages/AdminFactory.tsx. Inside the page's main content <div> (the one currently empty per Spec 09), import and render <CrawlerFactory />. Keep the existing header, framer-motion fade-up wrapper, and <NodeNetwork /> background untouched.
// Add at the top of the file (next to other imports):
import { CrawlerFactory } from '@/components/admin/factory/CrawlerFactory';
// Inside the page body, replace the empty placeholder container:
<div className="container mx-auto max-w-7xl px-6 py-10">
<CrawlerFactory />
</div>
(If the existing container in your codebase uses different Tailwind classes, keep them: only swap the children.)
- [ ] Step 7.3: Add the page-level smoke test
Create src/pages/__tests__/AdminFactory.test.tsx:
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen } from '@testing-library/react';
import '@testing-library/jest-dom';
import { MemoryRouter } from 'react-router-dom';
vi.mock('@/hooks/factory/useFactorySession', () => ({
useFactorySession: vi.fn(() => ({
session: null,
isLoading: false,
error: null,
refetch: vi.fn(),
})),
}));
vi.mock('@/hooks/factory/useDiscoveryStream', () => ({
useDiscoveryStream: vi.fn(() => ({
logs: [],
urls: [],
pathGroups: [],
isStreaming: false,
sessionId: null,
error: null,
discover: vi.fn(),
})),
}));
vi.mock('@/hooks/factory/useCrawlProgress', () => ({
useCrawlProgress: vi.fn(() => ({
urlsCrawled: 0,
urlsTotal: 0,
urlsPerSec: 0,
success: 0,
failed: 0,
ignored: 0,
status: 'pending',
error: null,
})),
}));
vi.mock('@/components/admin/factory/DiscoveryPanel', () => ({
DiscoveryPanel: () => <div data-testid="stub-discovery-panel" />,
}));
vi.mock('@/components/admin/factory/CategoryReview', () => ({
CategoryReview: () => <div data-testid="stub-category-review" />,
}));
vi.mock('@/components/admin/factory/CategoryConfigurator', () => ({
CategoryConfigurator: () => <div data-testid="stub-category-configurator" />,
}));
vi.mock('@/components/admin/factory/RollingLog', () => ({
RollingLog: () => <div data-testid="stub-rolling-log" />,
}));
import AdminFactory from '../AdminFactory';
describe('AdminFactory page', () => {
it('mounts CrawlerFactory inside the page shell', () => {
render(
<MemoryRouter initialEntries={['/admin/factory']}>
<AdminFactory />
</MemoryRouter>
);
expect(screen.getByTestId('crawler-factory')).toBeInTheDocument();
expect(screen.getByTestId('stub-discovery-panel')).toBeInTheDocument();
expect(screen.getByTestId('stub-rolling-log')).toBeInTheDocument();
});
});
- [ ] Step 7.4: Run the page test, expect PASS
npx vitest run src/pages/__tests__/AdminFactory.test.tsx
Expected: 1 test passes.
- [ ] Step 7.5: Run all factory frontend tests + tsc
npx vitest run src/components/admin/factory/__tests__/ src/pages/__tests__/AdminFactory.test.tsx && npx tsc --noEmit
Expected: all factory frontend tests pass; tsc clean.
- [ ] Step 7.6: Commit
git add src/pages/AdminFactory.tsx src/pages/__tests__/AdminFactory.test.tsx
git commit -m "feat(factory): mount CrawlerFactory inside AdminFactory page shell"
Task 8: Cluster validation: full regression
Files: - (Read-only verification)
- [ ] Step 8.1: Run the full project test suite
npx vitest run
Expected: prior baseline tests + new factory tests, all GREEN. New tests added by this spec: ~17 (4 LiveCrawlMonitor + 12 CrawlerFactory + 1 AdminFactory page).
- [ ] Step 8.2: Run full tsc
npx tsc --noEmit
Expected: clean.
- [ ] Step 8.3: Manual smoke (optional, only if dev server running)
npm run dev
Then open http://localhost:5173/admin/factory. Expect: header + DiscoveryPanel + RollingLog. URL should have no query param yet.
Refresh with ?session=does-not-exist. Expect: ErrorPanel rendered (or LoadingPanel if the API takes a moment), with a "Restart" button that clears the URL param.
- [ ] Step 8.4: Mark Spec 12 done in
Status.md
In the vault Projects/Crawler-Factory/Status.md, change:
- | Cluster 12 (Frontend live monitor + orchestrator) | ⏸ | LiveCrawlMonitor + CrawlerFactory FSM driver |
+ | Cluster 12 (Frontend live monitor + orchestrator) | ✅ | LiveCrawlMonitor + CrawlerFactory FSM driver |
Acceptance criteria: Spec 12 done means:
- ✅
src/components/admin/factory/LiveCrawlMonitor.tsxexists, exportsLiveCrawlMonitorand typesLiveCrawlMonitorProps,LiveCrawlMonitorCrawler - ✅ Empty
crawlerIdsrenders empty-state copy (No active crawls) - ✅ Non-empty
crawlerIdsrenders one block per crawler with: name, mono crawlerId, DomainBadge, indexName, shadcnProgressbar with proper aria attributes, URLs/sec, success/failed/ignored counters - ✅ Each block subscribes to
useCrawlProgress(crawlerId)(one hook call per crawlerId) - ✅
src/components/admin/factory/CrawlerFactory.tsxexists and exportsCrawlerFactory - ✅ Reads
?session=<id>viauseSearchParamsand passes touseFactorySession(sessionId) - ✅ FSM rendering verified by tests: null/discovering → DiscoveryPanel; categorized → CategoryReview; configuring → CategoryReview + CategoryConfigurator drawer; crawling → LiveCrawlMonitor; done → LiveCrawlMonitor + Restart; error → ErrorPanel; isLoading → LoadingPanel
- ✅ When discovery completes (useDiscoveryStream emits
sessionId), URL is updated to?session=<id>so refresh resumes correctly - ✅ Configurator queue: clicking "Configure Selected" with N pathGroups walks them one at a time; each commit advances the queue
- ✅ Right column ALWAYS renders
<RollingLog />; layout is 2-column grid (left ~60%, right ~40%) onlgbreakpoint - ✅
src/pages/AdminFactory.tsxmounts<CrawlerFactory />(Spec 09 placeholder replaced) - ✅ All new tests pass (~17 added by this spec)
- ✅ Full project test suite still GREEN
- ✅
tsc --noEmitclean - ✅ Per CodingSOPs: every component has docstring, no
console.log, noany, components ≤20-line functions where practical, mock-only-hooks unit tests - ✅ Per UIUXDesignSOP / Plan §5a: aesthetic matches
/admin(bg-[#0a0a0f], white/10 borders, font-black uppercase tracking-tighter, mono for IDs/URLs, framer-motion fade-up)
Out of scope (handled by other specs)
- DiscoveryPanel, CategoryReview, RollingLog, DomainBadge implementations → Spec 10
- CategoryConfigurator drawer + sub-components (SamplePreview, ConfigEditor, TestCrawlResults) → Spec 11
useFactorySession,useDiscoveryStream,useCrawlProgresshooks + page shellAdminFactory.tsx(without CrawlerFactory mount) → Spec 09/api/factory/sessionand/api/factory/crawl-progressendpoints → Spec 08FactorySession,PathGroupzod schemas → Spec 01- E2E browser smoke test (real backend hit) → Spec 13
- Bundle build (
scripts/bundle-api.mjs) + vault sync +Status.mdtoggle for the whole factory → Spec 13 - The
/admin/factoryroute registration insrc/App.tsx→ Spec 09 (already added there per Task 22)