Crawler-Factory

Engineering-Specs/01-foundations.md

Crawler Factory: Spec 01: Foundations Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Establish the data foundation for the Crawler Factory: Domain Schema Standard (DSS) registry, zod schemas (record union, session, blueprint), sharded session store, and blueprint store: so every downstream cluster can validate boundaries and persist state.

Architecture: Pure data + I/O layer. No business logic. DSS is a typed registry of 9 content domains mapped to schema.org types + Algolia config. Zod schemas validate every boundary (file I/O, API, inter-module handoffs). Session storage is sharded across 4 record types (session, url_shard, sample, log) in the algoliacentral_factory_sessions index: uncapped URL counts via streaming. Blueprints land in algoliacentral_factory_blueprints with an agent_slot field for the future specialist-agent factory.

Tech Stack: TypeScript (strict), zod 3.25, algoliasearch v5, vitest, pino logger via lib/utils/logger.ts, fast-xml-parser (added here for downstream specs).

Depends on: Nothing. This is the root cluster. Consumed by: Specs 02–13 (every other cluster imports types/DSS/stores from here).

Plan source: Projects/Crawler-Factory/00-Plan.md §1, §3, §4, §6, §7 Tasks 0–2, 6–7, §19f principles.

File structure

Path	Responsibility
`package.json`	Add `fast-xml-parser` dep (used by Spec 02 walker)
`scripts/bundle-api.mjs`	Add `fast-xml-parser` to externals so Vercel doesn't bundle it
`crawler-configs/.gitkeep`	Reserve directory for version-controlled recordExtractor copies (Spec 06 writes here)
`lib/factory/dss.ts`	DSS registry: 9 content domains × schema.org types × Algolia config
`lib/factory/types.ts`	Zod schemas: record discriminated union, session, blueprint, log entry
`lib/factory/session-store.ts`	Sharded session CRUD against `algoliacentral_factory_sessions`
`lib/factory/blueprint-store.ts`	Blueprint CRUD against `algoliacentral_factory_blueprints`
`lib/factory/__tests__/dss.test.ts`	DSS registry tests (one per content domain + helpers)
`lib/factory/__tests__/types.test.ts`	Zod schema tests (per-domain + union + session + blueprint)
`lib/factory/__tests__/session-store.test.ts`	Session store tests (mocked algoliasearch client)
`lib/factory/__tests__/blueprint-store.test.ts`	Blueprint store tests (mocked algoliasearch client)

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

Two-layer type safety. zod for runtime boundary validation (every public API + file I/O); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful.
Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:dss'); (or appropriate context). No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
Try/catch every fallible call. Every external (Algolia API, fetch) wrapped. Catch specific error first, generic last. Always log before rethrow with full context.
Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object.
Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
No raw dicts crossing module boundaries. Everything is a zod-validated type.
TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
Mock external services in unit tests. Use vitest's vi.mock for algoliasearch. Real Algolia calls live in integration tests gated by env vars.
Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
Conventional commits. feat(factory):, test(factory):, chore(factory):, etc. One logical change per commit.

Task 0: Add `fast-xml-parser` dependency + bundler externals + crawler-configs directory

Files: - Modify: package.json: dependencies section - Modify: scripts/bundle-api.mjs: externals array - Create: crawler-configs/.gitkeep

[ ] Step 0.1: Install fast-xml-parser

npm install fast-xml-parser@^4.5.0

Expected: package.json updated; node_modules/fast-xml-parser/ exists; package-lock.json updated.

[ ] Step 0.2: Verify installed version

node -e "console.log(require('fast-xml-parser/package.json').version)"

Expected output: 4.5.x or higher.

[ ] Step 0.3: Add to bundler externals

Open scripts/bundle-api.mjs. Find the existing externals array (a list of npm packages that should NOT be bundled by esbuild). Add 'fast-xml-parser' alphabetically.

// Example — locate the existing externals array and add fast-xml-parser
const externals = [
  '@google/generative-ai',
  'algoliasearch',
  'cheerio',
  'fast-xml-parser',   // ← ADD THIS LINE (alphabetical)
  'pino',
  'uuid',
];

[ ] Step 0.4: Verify bundler still runs

node scripts/bundle-api.mjs

Expected: completes without error; existing api/*.mjs bundles regenerate.

[ ] Step 0.5: Create crawler-configs/ directory with .gitkeep

mkdir -p crawler-configs && touch crawler-configs/.gitkeep

[ ] Step 0.6: Commit

git add package.json package-lock.json scripts/bundle-api.mjs crawler-configs/.gitkeep
git commit -m "chore(factory): add fast-xml-parser dep + reserve crawler-configs/ dir"

Task 1: DSS registry: Domain Schema Standard

Files: - Create: lib/factory/dss.ts - Create: lib/factory/__tests__/dss.test.ts

The DSS is a typed registry. Each content domain maps to: - indexName: the Algolia index it writes to - schemaOrgTypes: schema.org @type values that should classify into this domain - algoliaConfig: searchableAttributes, customRanking, attributesForFaceting - description: one-line semantic descriptor used by LLM classification fallback

The 9 default domains are: marketing, support, education, technical, customer-stories, product-catalog, events, legal, social. (See 00-Plan.md §4a for the rationale.)

[ ] Step 1.1: Write failing test for getDss('marketing') shape

Create lib/factory/__tests__/dss.test.ts:

import { describe, it, expect } from 'vitest';
import { DSS, getDss, dssBySchemaOrgType, listContentDomains } from '../dss';

describe('DSS registry', () => {
  describe('getDss', () => {
    it('returns marketing entry with expected shape', () => {
      const entry = getDss('marketing');
      expect(entry.indexName).toBe('algoliacentral_marketing');
      expect(entry.schemaOrgTypes).toContain('BlogPosting');
      expect(entry.schemaOrgTypes).toContain('NewsArticle');
      expect(entry.schemaOrgTypes).toContain('Article');
      expect(entry.algoliaConfig.searchableAttributes).toContain('headline');
      expect(entry.algoliaConfig.searchableAttributes).toContain('articleBody');
      expect(entry.algoliaConfig.customRanking).toEqual(
        expect.arrayContaining([expect.stringMatching(/desc\(datePublished\)/)])
      );
      expect(entry.description).toMatch(/marketing|news|blog/i);
    });

    it('throws on unknown domain', () => {
      // @ts-expect-error — runtime check; TS prevents this at compile time
      expect(() => getDss('made-up-domain')).toThrow();
    });
  });
});

[ ] Step 1.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/dss.test.ts

Expected: FAIL: Cannot find module '../dss' or similar. (Module not yet implemented.)

[ ] Step 1.3: Implement dss.ts with marketing entry first

Create lib/factory/dss.ts:

/**
 * Domain Schema Standard (DSS) — typed registry mapping content domains
 * to schema.org types, Algolia indices, and per-domain config.
 *
 * Why: schema.org alone covers <25% of useful classifications across the web
 * (see 00-Plan.md §14). DSS adds CMS-fingerprint-aware mappings + index tuning
 * derived from per-domain user-search behavior.
 */

import { createLogger } from '../utils/logger';

const log = createLogger('factory:dss');

export type ContentDomain =
  | 'marketing'
  | 'support'
  | 'education'
  | 'technical'
  | 'customer-stories'
  | 'product-catalog'
  | 'events'
  | 'legal'
  | 'social';

export interface DssEntry {
  indexName: string;
  schemaOrgTypes: readonly string[];
  algoliaConfig: {
    searchableAttributes: readonly string[];
    customRanking: readonly string[];
    attributesForFaceting: readonly string[];
  };
  description: string;
}

export const DSS: Readonly<Record<ContentDomain, DssEntry>> = Object.freeze({
  marketing: {
    indexName: 'algoliacentral_marketing',
    schemaOrgTypes: [
      'BlogPosting',
      'NewsArticle',
      'OpinionArticle',
      'Article',
      'AnnouncementPage',
      'SpecialAnnouncement',
    ],
    algoliaConfig: {
      searchableAttributes: ['headline', 'articleBody', 'keywords'],
      customRanking: ['desc(datePublished)', 'desc(view_count)'],
      attributesForFaceting: ['articleSection', 'author', 'year'],
    },
    description: 'Marketing, news, blog, announcement content',
  },
  support: {
    indexName: 'algoliacentral_support',
    schemaOrgTypes: ['TechArticle', 'FAQPage', 'Question', 'Answer', 'HowTo'],
    algoliaConfig: {
      searchableAttributes: ['name', 'articleBody', 'acceptedAnswer'],
      customRanking: ['desc(lastReviewed)', 'desc(helpful_count)'],
      attributesForFaceting: ['productAffected', 'severity', 'status'],
    },
    description: 'Support articles, FAQs, troubleshooting, help center',
  },
  education: {
    indexName: 'algoliacentral_education',
    schemaOrgTypes: ['Course', 'LearningResource', 'HowTo', 'Guide', 'VideoObject'],
    algoliaConfig: {
      searchableAttributes: ['name', 'description', 'articleBody', 'transcript'],
      customRanking: ['desc(datePublished)', 'desc(enrollment_count)'],
      attributesForFaceting: ['educationalLevel', 'learningResourceType', 'language', 'duration'],
    },
    description: 'Courses, tutorials, learning resources, training material',
  },
  technical: {
    indexName: 'algoliacentral_technical',
    schemaOrgTypes: ['TechArticle', 'SoftwareSourceCode', 'SoftwareApplication', 'WebAPI'],
    algoliaConfig: {
      searchableAttributes: ['name', 'articleBody', 'programmingLanguage', 'codeSampleType'],
      customRanking: ['desc(version)', 'desc(view_count)'],
      attributesForFaceting: ['programmingLanguage', 'framework', 'version', 'apiEndpoint'],
    },
    description: 'Technical docs, API references, developer guides, code samples',
  },
  'customer-stories': {
    indexName: 'algoliacentral_customers',
    schemaOrgTypes: ['Article', 'Review', 'Person', 'Organization'],
    algoliaConfig: {
      searchableAttributes: ['headline', 'articleBody', 'about.name', 'quotes'],
      customRanking: ['desc(datePublished)', 'desc(prominence)'],
      attributesForFaceting: ['industry', 'companySize', 'region', 'useCase'],
    },
    description: 'Customer case studies, success stories, testimonials',
  },
  'product-catalog': {
    indexName: 'algoliacentral_products',
    schemaOrgTypes: ['Product', 'Offer', 'Service', 'Brand'],
    algoliaConfig: {
      searchableAttributes: ['name', 'description', 'brand'],
      customRanking: ['desc(aggregateRating)', 'desc(popularity)'],
      attributesForFaceting: ['brand', 'category', 'priceRange', 'rating'],
    },
    description: 'Product detail pages, offerings, services, brand pages',
  },
  events: {
    indexName: 'algoliacentral_events',
    schemaOrgTypes: ['Event', 'EducationEvent', 'BusinessEvent', 'Conference'],
    algoliaConfig: {
      searchableAttributes: ['name', 'description'],
      customRanking: ['asc(startDate)'],
      attributesForFaceting: ['eventType', 'location', 'year'],
    },
    description: 'Conferences, webinars, training events, business events',
  },
  legal: {
    indexName: 'algoliacentral_legal',
    schemaOrgTypes: ['DigitalDocument', 'Legislation'],
    algoliaConfig: {
      searchableAttributes: ['name', 'text'],
      customRanking: ['desc(version)', 'desc(datePublished)'],
      attributesForFaceting: ['jurisdiction', 'documentType'],
    },
    description: 'Terms of service, privacy policies, legal documents',
  },
  social: {
    indexName: 'algoliacentral_social',
    schemaOrgTypes: ['VideoObject', 'Comment', 'Conversation'],
    algoliaConfig: {
      searchableAttributes: ['text', 'transcript'],
      customRanking: ['desc(uploadDate)', 'desc(engagement)'],
      attributesForFaceting: ['platform', 'channel', 'sentiment'],
    },
    description: 'Social media posts, video transcripts, community comments',
  },
});

/**
 * Get the DSS entry for a given content domain. Throws if domain is unknown.
 */
export function getDss(domain: ContentDomain): DssEntry {
  const entry = DSS[domain];
  if (!entry) {
    log.error({ event: 'getDss_unknown_domain', domain }, 'Unknown content domain');
    throw new Error(`Unknown content domain: ${domain}`);
  }
  return entry;
}

/**
 * Reverse lookup — given a schema.org @type, return the content domain it maps to.
 * Returns null if no DSS entry claims this type.
 */
export function dssBySchemaOrgType(schemaType: string): ContentDomain | null {
  for (const domain of Object.keys(DSS) as ContentDomain[]) {
    if (DSS[domain].schemaOrgTypes.includes(schemaType)) {
      return domain;
    }
  }
  return null;
}

/**
 * Returns the canonical ordered list of content domains.
 */
export function listContentDomains(): ContentDomain[] {
  return Object.keys(DSS) as ContentDomain[];
}

[ ] Step 1.4: Run test, expect PASS

npx vitest run lib/factory/__tests__/dss.test.ts

Expected: 2 tests pass (getDss returns marketing entry + throws on unknown domain).

[ ] Step 1.5: Add tests for the remaining 8 domains and the helpers

Append to lib/factory/__tests__/dss.test.ts:

describe('DSS registry — all 9 default domains', () => {
  const cases: Array<[string, string, string]> = [
    ['support', 'algoliacentral_support', 'TechArticle'],
    ['education', 'algoliacentral_education', 'Course'],
    ['technical', 'algoliacentral_technical', 'SoftwareSourceCode'],
    ['customer-stories', 'algoliacentral_customers', 'Review'],
    ['product-catalog', 'algoliacentral_products', 'Product'],
    ['events', 'algoliacentral_events', 'Event'],
    ['legal', 'algoliacentral_legal', 'DigitalDocument'],
    ['social', 'algoliacentral_social', 'VideoObject'],
  ];

  it.each(cases)('domain %s has indexName %s and includes type %s', (domain, expectedIndex, expectedType) => {
    const entry = getDss(domain as Parameters<typeof getDss>[0]);
    expect(entry.indexName).toBe(expectedIndex);
    expect(entry.schemaOrgTypes).toContain(expectedType);
    expect(entry.description).toBeTruthy();
    expect(entry.algoliaConfig.searchableAttributes.length).toBeGreaterThan(0);
    expect(entry.algoliaConfig.customRanking.length).toBeGreaterThan(0);
  });
});

describe('dssBySchemaOrgType', () => {
  it('maps BlogPosting → marketing', () => {
    expect(dssBySchemaOrgType('BlogPosting')).toBe('marketing');
  });

  it('maps Course → education', () => {
    expect(dssBySchemaOrgType('Course')).toBe('education');
  });

  it('maps Product → product-catalog', () => {
    expect(dssBySchemaOrgType('Product')).toBe('product-catalog');
  });

  it('returns null for unknown @type', () => {
    expect(dssBySchemaOrgType('NonExistentSchemaType')).toBeNull();
  });
});

describe('listContentDomains', () => {
  it('returns all 9 default domains', () => {
    const domains = listContentDomains();
    expect(domains).toHaveLength(9);
    expect(domains).toContain('marketing');
    expect(domains).toContain('support');
    expect(domains).toContain('social');
  });
});

[ ] Step 1.6: Run, expect PASS for all DSS tests

npx vitest run lib/factory/__tests__/dss.test.ts

Expected: 13 tests pass total.

[ ] Step 1.7: Run tsc to confirm strict type-safety

npx tsc --noEmit

Expected: clean (existing 407-test baseline still passes; no new errors).

[ ] Step 1.8: Commit

git add lib/factory/dss.ts lib/factory/__tests__/dss.test.ts
git commit -m "feat(factory): add Domain Schema Standard registry with 9 content domains"

Task 2: Zod schemas: record discriminated union, session, blueprint, log

Files: - Create: lib/factory/types.ts - Create: lib/factory/__tests__/types.test.ts

This module defines every type that crosses a module boundary in the factory. Per CodingSOPs §7, no raw unknown/any is allowed; everything is zod-validated.

[ ] Step 2.1: Write failing test for MarketingRecordSchema

Create lib/factory/__tests__/types.test.ts:

import { describe, it, expect } from 'vitest';
import {
  MarketingRecordSchema,
  SupportRecordSchema,
  FactoryRecordSchema,
  FactorySessionSchema,
  BlueprintSchema,
  LogEntrySchema,
} from '../types';

describe('MarketingRecordSchema', () => {
  it('accepts a minimal valid marketing record', () => {
    const valid = {
      objectID: 'abc123',
      url: 'https://www.algolia.com/blog/how-to-search/',
      content_domain: 'marketing' as const,
      schema_org_type: 'BlogPosting',
      detected_via: 'json-ld' as const,
      detection_confidence: 0.92,
      language_code: 'en',
      source_url_root: 'https://www.algolia.com',
      crawled_at: 1714432200000,
      is_chunk: false as const,
      status: 'indexed' as const,
      headline: 'How to Search',
      articleBody: 'A long article body. '.repeat(20),
      datePublished: '2026-04-30T00:00:00Z',
    };
    expect(() => MarketingRecordSchema.parse(valid)).not.toThrow();
  });

  it('rejects a marketing record with content_domain mismatch', () => {
    const invalid = {
      objectID: 'abc123',
      url: 'https://www.algolia.com/blog/how-to-search/',
      content_domain: 'support' as const, // wrong
      schema_org_type: 'BlogPosting',
      detected_via: 'json-ld' as const,
      detection_confidence: 0.92,
      language_code: 'en',
      source_url_root: 'https://www.algolia.com',
      crawled_at: 1714432200000,
      is_chunk: false as const,
      status: 'indexed' as const,
      headline: 'How to Search',
      articleBody: 'A long article body. '.repeat(20),
    };
    expect(() => MarketingRecordSchema.parse(invalid)).toThrow();
  });

  it('rejects when articleBody is too short', () => {
    const invalid = {
      objectID: 'abc123',
      url: 'https://www.algolia.com/blog/how-to-search/',
      content_domain: 'marketing' as const,
      schema_org_type: 'BlogPosting',
      detected_via: 'json-ld' as const,
      detection_confidence: 0.92,
      language_code: 'en',
      source_url_root: 'https://www.algolia.com',
      crawled_at: 1714432200000,
      is_chunk: false as const,
      status: 'indexed' as const,
      headline: 'How to Search',
      articleBody: 'too short',
    };
    expect(() => MarketingRecordSchema.parse(invalid)).toThrow();
  });
});

[ ] Step 2.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/types.test.ts

Expected: FAIL: Cannot find module '../types'.

[ ] Step 2.3: Implement lib/factory/types.ts with base + marketing schemas

Create lib/factory/types.ts:

/**
 * Zod schemas for every type that crosses a module boundary in the factory.
 *
 * Why: per CodingSOPs §7, no raw dicts cross boundaries. Every record, session,
 * blueprint, and log entry is validated at the edge.
 */

import { z } from 'zod';

// ─── Detection metadata (shared across all records) ─────────────────────────

export const DetectedViaEnum = z.enum([
  'json-ld',
  'microdata',
  'rdfa',
  'opengraph',
  'heuristic',
  'url',
  'llm',
  'cms',
]);
export type DetectedVia = z.infer<typeof DetectedViaEnum>;

export const ContentDomainEnum = z.enum([
  'marketing',
  'support',
  'education',
  'technical',
  'customer-stories',
  'product-catalog',
  'events',
  'legal',
  'social',
]);
export type ContentDomain = z.infer<typeof ContentDomainEnum>;

// ─── Base record schema (every domain extends this) ─────────────────────────

const RecordBaseSchema = z.object({
  objectID: z.string().min(1),
  url: z.string().url(),
  schema_org_type: z.string(),
  detected_via: DetectedViaEnum,
  detection_confidence: z.number().min(0).max(1),
  language_code: z.string().default('en'),
  source_url_root: z.string().url(),
  crawled_at: z.number().int().positive(),
  is_chunk: z.literal(false),
  status: z.literal('indexed'),
});

// ─── Per-domain record schemas ──────────────────────────────────────────────

export const MarketingRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('marketing'),
  headline: z.string().min(1),
  articleBody: z.string().min(200),
  datePublished: z.string().optional(),
  dateModified: z.string().optional(),
  author: z.array(z.string()).optional(),
  articleSection: z.string().optional(),
  keywords: z.array(z.string()).optional(),
  image: z.string().url().optional(),
});
export type MarketingRecord = z.infer<typeof MarketingRecordSchema>;

export const SupportRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('support'),
  name: z.string().min(1),
  articleBody: z.string().min(200),
  acceptedAnswer: z.string().optional(),
  lastReviewed: z.string().optional(),
  productAffected: z.array(z.string()).optional(),
  severity: z.enum(['low', 'medium', 'high', 'critical']).optional(),
  supportLevel: z.enum(['self-serve', 'assisted', 'enterprise']).optional(),
});
export type SupportRecord = z.infer<typeof SupportRecordSchema>;

export const EducationRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('education'),
  name: z.string().min(1),
  description: z.string().min(50),
  learningResourceType: z.string().optional(),
  educationalLevel: z.string().optional(),
  timeRequired: z.string().optional(),
  hasCourseInstance: z.array(z.string()).optional(),
  transcript: z.string().optional(),
});
export type EducationRecord = z.infer<typeof EducationRecordSchema>;

export const TechnicalRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('technical'),
  name: z.string().min(1),
  articleBody: z.string().min(200),
  programmingLanguage: z.string().optional(),
  version: z.string().optional(),
  applicationCategory: z.string().optional(),
  codeSampleType: z.string().optional(),
  apiEndpoint: z.string().optional(),
});
export type TechnicalRecord = z.infer<typeof TechnicalRecordSchema>;

export const CustomerStoryRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('customer-stories'),
  headline: z.string().min(1),
  articleBody: z.string().min(200),
  about: z.object({ name: z.string(), industry: z.string().optional() }).optional(),
  industry: z.string().optional(),
  companySize: z.string().optional(),
  region: z.string().optional(),
  metrics: z.array(z.string()).optional(),
  quotes: z.array(z.string()).optional(),
});
export type CustomerStoryRecord = z.infer<typeof CustomerStoryRecordSchema>;

export const ProductRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('product-catalog'),
  name: z.string().min(1),
  description: z.string().min(50),
  sku: z.string().optional(),
  brand: z.string().optional(),
  category: z.string().optional(),
  offers: z
    .object({
      price: z.number().optional(),
      priceCurrency: z.string().optional(),
      availability: z.string().optional(),
    })
    .optional(),
  aggregateRating: z.number().optional(),
  image: z.array(z.string().url()).optional(),
  variants: z
    .array(
      z.object({
        color: z.string().optional(),
        size: z.string().optional(),
        sku: z.string().optional(),
        available: z.boolean().optional(),
      })
    )
    .optional(),
});
export type ProductRecord = z.infer<typeof ProductRecordSchema>;

export const EventRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('events'),
  name: z.string().min(1),
  description: z.string().min(50),
  startDate: z.string(),
  endDate: z.string().optional(),
  location: z.string().optional(),
  organizer: z.string().optional(),
  eventType: z.string().optional(),
  recordingUrl: z.string().url().optional(),
});
export type EventRecord = z.infer<typeof EventRecordSchema>;

export const LegalRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('legal'),
  name: z.string().min(1),
  text: z.string().min(200),
  version: z.string().optional(),
  datePublished: z.string().optional(),
  jurisdiction: z.string().optional(),
  documentType: z.string().optional(),
});
export type LegalRecord = z.infer<typeof LegalRecordSchema>;

export const SocialRecordSchema = RecordBaseSchema.extend({
  content_domain: z.literal('social'),
  text: z.string().min(50),
  transcript: z.string().optional(),
  uploadDate: z.string().optional(),
  channel: z.string().optional(),
  viewCount: z.number().optional(),
  commentCount: z.number().optional(),
  platform: z.string().optional(),
  sentiment: z.enum(['positive', 'neutral', 'negative']).optional(),
});
export type SocialRecord = z.infer<typeof SocialRecordSchema>;

// ─── Discriminated union — used wherever a "factory record" crosses a boundary

export const FactoryRecordSchema = z.discriminatedUnion('content_domain', [
  MarketingRecordSchema,
  SupportRecordSchema,
  EducationRecordSchema,
  TechnicalRecordSchema,
  CustomerStoryRecordSchema,
  ProductRecordSchema,
  EventRecordSchema,
  LegalRecordSchema,
  SocialRecordSchema,
]);
export type FactoryRecord = z.infer<typeof FactoryRecordSchema>;

// ─── Log entry (per session) ────────────────────────────────────────────────

export const LogEntrySchema = z.object({
  ts: z.number().int().positive(),
  level: z.enum(['info', 'warn', 'error']),
  message: z.string().min(1),
  meta: z.record(z.unknown()).optional(),
});
export type LogEntry = z.infer<typeof LogEntrySchema>;

// ─── Path group (output of path-grouper, per session) ───────────────────────

export const PathGroupSchema = z.object({
  id: z.string().min(1),
  pattern: z.string().min(1),
  urlCount: z.number().int().nonnegative(),
  sampleUrls: z.array(z.string().url()).max(10),
  detectedDomain: ContentDomainEnum.nullable(),
  detectionConfidence: z.number().min(0).max(1).nullable(),
  detectionMethod: DetectedViaEnum.nullable(),
  selected: z.boolean().default(false),
  blueprint: z
    .object({
      indexName: z.string().min(1),
      recordExtractor: z.string().min(1),
      crawlerConfig: z.object({
        renderJavaScript: z.boolean(),
        rateLimit: z.number().int().positive(),
        pathsToMatch: z.array(z.string()),
      }),
      sandboxResults: z.array(z.unknown()).optional(),
      realTestResults: z.array(z.unknown()).optional(),
      crawlerId: z.string().optional(),
      reindexTaskId: z.string().optional(),
    })
    .optional(),
});
export type PathGroup = z.infer<typeof PathGroupSchema>;

// ─── Session main record ────────────────────────────────────────────────────

export const FactorySessionSchema = z.object({
  objectID: z.string().min(1),
  record_type: z.literal('session'),
  status: z.enum(['discovering', 'categorized', 'configuring', 'crawling', 'done', 'error']),
  source_domain: z.string().min(1),
  source_url_root: z.string().url(),
  createdAt: z.number().int().positive(),
  updatedAt: z.number().int().positive(),
  discovery: z.object({
    sitemapUrls: z.array(z.string()),
    urlCount: z.number().int().nonnegative(),
    urlShardCount: z.number().int().nonnegative(),
    schemaOrgCoverage: z.number().min(0).max(1),
  }),
  pathGroups: z.array(PathGroupSchema),
});
export type FactorySession = z.infer<typeof FactorySessionSchema>;

// ─── Blueprint (one per crawler created) ────────────────────────────────────

export const BlueprintSchema = z.object({
  objectID: z.string().min(1),
  content_domain: ContentDomainEnum,
  index_name: z.string().min(1),
  source_domain: z.string().min(1),
  path_groups: z.array(z.string()),
  schema_org_types: z.array(z.string()),
  recordExtractor_path: z.string().min(1),
  algolia_config: z.object({
    searchableAttributes: z.array(z.string()),
    customRanking: z.array(z.string()),
    attributesForFaceting: z.array(z.string()),
  }),
  agent_slot: z.string().nullable(),
  created_at: z.number().int().positive(),
  status: z.enum(['live', 'paused', 'archived']),
});
export type Blueprint = z.infer<typeof BlueprintSchema>;

[ ] Step 2.4: Run marketing tests, expect PASS

npx vitest run lib/factory/__tests__/types.test.ts -t MarketingRecordSchema

Expected: 3 tests pass.

[ ] Step 2.5: Add tests for the discriminated union and remaining schemas

Append to lib/factory/__tests__/types.test.ts:

describe('FactoryRecordSchema discriminated union', () => {
  it('accepts a marketing record via the union', () => {
    const valid = {
      objectID: 'm1',
      url: 'https://x.com/blog/post',
      content_domain: 'marketing' as const,
      schema_org_type: 'BlogPosting',
      detected_via: 'cms' as const,
      detection_confidence: 0.85,
      language_code: 'en',
      source_url_root: 'https://x.com',
      crawled_at: 1714432200000,
      is_chunk: false as const,
      status: 'indexed' as const,
      headline: 'Hello',
      articleBody: 'A '.repeat(120),
    };
    expect(() => FactoryRecordSchema.parse(valid)).not.toThrow();
  });

  it('discriminates correctly — passing marketing fields to support shape fails', () => {
    const invalid = {
      objectID: 's1',
      url: 'https://x.com/help',
      content_domain: 'support' as const,
      schema_org_type: 'TechArticle',
      detected_via: 'cms' as const,
      detection_confidence: 0.9,
      language_code: 'en',
      source_url_root: 'https://x.com',
      crawled_at: 1714432200000,
      is_chunk: false as const,
      status: 'indexed' as const,
      // missing required `name`; using marketing-flavored `headline` instead
      headline: 'Wrong field',
      articleBody: 'A '.repeat(120),
    };
    expect(() => FactoryRecordSchema.parse(invalid)).toThrow();
  });
});

describe('LogEntrySchema', () => {
  it('accepts a minimal info log', () => {
    const valid = { ts: 1714432200000, level: 'info' as const, message: 'started' };
    expect(() => LogEntrySchema.parse(valid)).not.toThrow();
  });

  it('rejects empty message', () => {
    const invalid = { ts: 1714432200000, level: 'info' as const, message: '' };
    expect(() => LogEntrySchema.parse(invalid)).toThrow();
  });
});

describe('FactorySessionSchema', () => {
  it('accepts a minimal session', () => {
    const valid = {
      objectID: 'sess_001',
      record_type: 'session' as const,
      status: 'discovering' as const,
      source_domain: 'algolia.com',
      source_url_root: 'https://www.algolia.com',
      createdAt: 1714432200000,
      updatedAt: 1714432200000,
      discovery: {
        sitemapUrls: ['https://www.algolia.com/sitemap.xml'],
        urlCount: 0,
        urlShardCount: 0,
        schemaOrgCoverage: 0,
      },
      pathGroups: [],
    };
    expect(() => FactorySessionSchema.parse(valid)).not.toThrow();
  });
});

describe('BlueprintSchema', () => {
  it('accepts a minimal blueprint with null agent_slot', () => {
    const valid = {
      objectID: 'crawler_xyz',
      content_domain: 'marketing' as const,
      index_name: 'algoliacentral_marketing',
      source_domain: 'algolia.com',
      path_groups: ['/blog/*'],
      schema_org_types: ['BlogPosting'],
      recordExtractor_path: 'crawler-configs/algolia.com/marketing-blog.js',
      algolia_config: {
        searchableAttributes: ['headline', 'articleBody'],
        customRanking: ['desc(datePublished)'],
        attributesForFaceting: ['articleSection'],
      },
      agent_slot: null,
      created_at: 1714432200000,
      status: 'live' as const,
    };
    expect(() => BlueprintSchema.parse(valid)).not.toThrow();
  });
});

[ ] Step 2.6: Run all type tests

npx vitest run lib/factory/__tests__/types.test.ts

Expected: all tests pass.

[ ] Step 2.7: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 2.8: Commit

git add lib/factory/types.ts lib/factory/__tests__/types.test.ts
git commit -m "feat(factory): add zod schemas for record union, session, blueprint, log"

Task 3: Sharded session store

Files: - Create: lib/factory/session-store.ts - Create: lib/factory/__tests__/session-store.test.ts

The session store wraps the Algolia client to read/write four record types in algoliacentral_factory_sessions: - record_type='session': main session metadata - record_type='url_shard': batches of 1000 discovered URLs - record_type='sample': sampled HTML pages per pathGroup - record_type='log': narrative log entries

Reads filter on parent_id:<sessionId> then split by record_type facet.

[ ] Step 3.1: Write failing test for saveSession

Create lib/factory/__tests__/session-store.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import type { FactorySession } from '../types';

// Mock algoliasearch BEFORE importing the module under test
const mockSaveObject = vi.fn();
const mockGetObject = vi.fn();
const mockSearchSingleIndex = vi.fn();
const mockSetSettings = vi.fn();

vi.mock('algoliasearch', () => ({
  algoliasearch: () => ({
    saveObject: mockSaveObject,
    getObject: mockGetObject,
    searchSingleIndex: mockSearchSingleIndex,
    setSettings: mockSetSettings,
  }),
}));

import { SessionStore } from '../session-store';

const minimalSession = (overrides: Partial<FactorySession> = {}): FactorySession => ({
  objectID: 'sess_001',
  record_type: 'session',
  status: 'discovering',
  source_domain: 'algolia.com',
  source_url_root: 'https://www.algolia.com',
  createdAt: 1714432200000,
  updatedAt: 1714432200000,
  discovery: {
    sitemapUrls: ['https://www.algolia.com/sitemap.xml'],
    urlCount: 0,
    urlShardCount: 0,
    schemaOrgCoverage: 0,
  },
  pathGroups: [],
  ...overrides,
});

describe('SessionStore.saveSession', () => {
  beforeEach(() => {
    mockSaveObject.mockReset();
    mockGetObject.mockReset();
    mockSearchSingleIndex.mockReset();
  });

  it('writes the main session record with record_type=session', async () => {
    mockSaveObject.mockResolvedValue({ taskID: 1, objectID: 'sess_001' });
    const store = new SessionStore('app-id', 'admin-key');
    await store.saveSession(minimalSession());
    expect(mockSaveObject).toHaveBeenCalledWith({
      indexName: 'algoliacentral_factory_sessions',
      body: expect.objectContaining({
        objectID: 'sess_001',
        record_type: 'session',
        source_domain: 'algolia.com',
      }),
    });
  });

  it('rejects when the input fails zod validation', async () => {
    const store = new SessionStore('app-id', 'admin-key');
    // @ts-expect-error — runtime validation
    await expect(store.saveSession({ objectID: 'bad', record_type: 'session' })).rejects.toThrow();
  });
});

[ ] Step 3.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/session-store.test.ts

Expected: FAIL: Cannot find module '../session-store'.

[ ] Step 3.3: Implement lib/factory/session-store.ts

Create lib/factory/session-store.ts:

/**
 * Sharded session store — read/write factory session state in
 * algoliacentral_factory_sessions across 4 record types.
 *
 * Sharding scheme:
 *   record_type='session'   — main metadata (one per sessionId)
 *   record_type='url_shard' — batches of 1000 discovered URLs
 *   record_type='sample'    — sampled HTML pages per pathGroup
 *   record_type='log'       — narrative log entries
 *
 * Why sharded: a single Algolia record has 100KB limit; 1M-URL sites
 * exceed it. See 00-Plan.md §4d.
 */

import { algoliasearch, type SearchClient } from 'algoliasearch';
import { createLogger } from '../utils/logger';
import {
  FactorySessionSchema,
  LogEntrySchema,
  type FactorySession,
  type LogEntry,
} from './types';

const log = createLogger('factory:session-store');

const INDEX = 'algoliacentral_factory_sessions';
const URL_BATCH_SIZE = 1000;

export class SessionStore {
  private client: SearchClient;

  constructor(appId: string, apiKey: string) {
    this.client = algoliasearch(appId, apiKey);
  }

  /**
   * Persist the main session record (record_type='session').
   * Validates input with FactorySessionSchema before writing.
   */
  async saveSession(session: FactorySession): Promise<void> {
    const validated = FactorySessionSchema.parse(session);
    try {
      await this.client.saveObject({ indexName: INDEX, body: validated });
      log.info({ event: 'saveSession', sessionId: validated.objectID }, 'session saved');
    } catch (err) {
      log.error(
        { event: 'saveSession_failed', sessionId: validated.objectID, err: String(err) },
        'failed to save session'
      );
      throw err;
    }
  }

  /**
   * Read the main session record by sessionId. Returns null if not found.
   */
  async getSession(sessionId: string): Promise<FactorySession | null> {
    try {
      const obj = await this.client.getObject({ indexName: INDEX, objectID: sessionId });
      return FactorySessionSchema.parse(obj);
    } catch (err) {
      if (String(err).includes('does not exist')) return null;
      log.error({ event: 'getSession_failed', sessionId, err: String(err) }, 'getSession error');
      throw err;
    }
  }

  /**
   * Append a batch of URLs as url_shard records (1000 per shard).
   * Returns the new total shard count.
   */
  async appendUrls(sessionId: string, urls: string[]): Promise<number> {
    if (urls.length === 0) return 0;
    let count = 0;
    for (let i = 0; i < urls.length; i += URL_BATCH_SIZE) {
      const batch = urls.slice(i, i + URL_BATCH_SIZE);
      const shardId = `${sessionId}:urls:${Date.now()}_${i}`;
      try {
        await this.client.saveObject({
          indexName: INDEX,
          body: {
            objectID: shardId,
            record_type: 'url_shard',
            parent_id: sessionId,
            urls: batch,
            shardIndex: i / URL_BATCH_SIZE,
          },
        });
        count += 1;
      } catch (err) {
        log.error(
          { event: 'appendUrls_failed', sessionId, shardIndex: i, err: String(err) },
          'failed to write url_shard'
        );
        throw err;
      }
    }
    log.info({ event: 'appendUrls', sessionId, batches: count, total_urls: urls.length });
    return count;
  }

  /**
   * Append a structured log entry (record_type='log').
   */
  async appendLog(sessionId: string, entry: LogEntry): Promise<void> {
    const validated = LogEntrySchema.parse(entry);
    const objectID = `${sessionId}:log:${validated.ts}`;
    try {
      await this.client.saveObject({
        indexName: INDEX,
        body: {
          ...validated,
          objectID,
          record_type: 'log',
          parent_id: sessionId,
        },
      });
    } catch (err) {
      log.error({ event: 'appendLog_failed', sessionId, err: String(err) }, 'failed to write log');
      throw err;
    }
  }

  /**
   * Add a sampled page record (record_type='sample').
   * `html` should be truncated to ≤80KB upstream.
   */
  async addSample(
    sessionId: string,
    pathGroupId: string,
    urlHash: string,
    sample: { url: string; html: string; status: number; structureAnalysis?: unknown }
  ): Promise<void> {
    const objectID = `${sessionId}:sample:${urlHash}`;
    try {
      await this.client.saveObject({
        indexName: INDEX,
        body: {
          objectID,
          record_type: 'sample',
          parent_id: sessionId,
          pathGroupId,
          ...sample,
        },
      });
    } catch (err) {
      log.error({ event: 'addSample_failed', sessionId, urlHash, err: String(err) }, 'failed to write sample');
      throw err;
    }
  }

  /**
   * Read all sample records for a given pathGroup of a session.
   * Filters by parent_id + record_type='sample' + pathGroupId.
   * Used by Spec 08 generate-extractor.ts to load samples for structure analysis.
   */
  async getSamples(
    sessionId: string,
    pathGroupId: string
  ): Promise<Array<{ url: string; html: string; status: number; structureAnalysis?: unknown }>> {
    try {
      const result = await this.client.searchSingleIndex({
        indexName: INDEX,
        searchParams: {
          query: '',
          filters: `parent_id:${sessionId} AND record_type:sample AND pathGroupId:${pathGroupId}`,
          hitsPerPage: 100,
        },
      });
      return (result.hits as Array<Record<string, unknown>>).map((h) => ({
        url: h.url as string,
        html: h.html as string,
        status: h.status as number,
        structureAnalysis: h.structureAnalysis,
      }));
    } catch (err) {
      log.error(
        { event: 'getSamples_failed', sessionId, pathGroupId, err: String(err) },
        'failed to read samples'
      );
      throw err;
    }
  }
}

[ ] Step 3.4: Run saveSession tests, expect PASS

npx vitest run lib/factory/__tests__/session-store.test.ts -t saveSession

Expected: 2 tests pass.

[ ] Step 3.5: Add tests for appendUrls, appendLog, addSample, getSession

Append to lib/factory/__tests__/session-store.test.ts:

describe('SessionStore.appendUrls', () => {
  beforeEach(() => mockSaveObject.mockReset());

  it('writes one url_shard per 1000 URLs', async () => {
    mockSaveObject.mockResolvedValue({ taskID: 1, objectID: '' });
    const urls = Array.from({ length: 2500 }, (_, i) => `https://x.com/p/${i}`);
    const store = new SessionStore('app-id', 'admin-key');
    const count = await store.appendUrls('sess_001', urls);
    expect(count).toBe(3); // 2500 / 1000 = 3 batches (1000, 1000, 500)
    expect(mockSaveObject).toHaveBeenCalledTimes(3);
  });

  it('returns 0 and writes nothing for empty input', async () => {
    const store = new SessionStore('app-id', 'admin-key');
    const count = await store.appendUrls('sess_001', []);
    expect(count).toBe(0);
    expect(mockSaveObject).not.toHaveBeenCalled();
  });
});

describe('SessionStore.appendLog', () => {
  beforeEach(() => mockSaveObject.mockReset());

  it('writes a log record with record_type=log and parent_id', async () => {
    mockSaveObject.mockResolvedValue({ taskID: 1 });
    const store = new SessionStore('app-id', 'admin-key');
    await store.appendLog('sess_001', { ts: 1714432200000, level: 'info', message: 'hello' });
    expect(mockSaveObject).toHaveBeenCalledWith({
      indexName: 'algoliacentral_factory_sessions',
      body: expect.objectContaining({
        record_type: 'log',
        parent_id: 'sess_001',
        message: 'hello',
      }),
    });
  });
});

describe('SessionStore.getSession', () => {
  beforeEach(() => {
    mockGetObject.mockReset();
  });

  it('returns null when the record does not exist', async () => {
    mockGetObject.mockRejectedValue(new Error('ObjectID does not exist'));
    const store = new SessionStore('app-id', 'admin-key');
    const result = await store.getSession('missing');
    expect(result).toBeNull();
  });
});

[ ] Step 3.6: Run all session-store tests

npx vitest run lib/factory/__tests__/session-store.test.ts

Expected: all pass.

[ ] Step 3.7: Run tsc --noEmit

npx tsc --noEmit

Expected: clean.

[ ] Step 3.8: Commit

git add lib/factory/session-store.ts lib/factory/__tests__/session-store.test.ts
git commit -m "feat(factory): sharded session store (session/url_shard/sample/log)"

Task 4: Blueprint store

Files: - Create: lib/factory/blueprint-store.ts - Create: lib/factory/__tests__/blueprint-store.test.ts

Writes algoliacentral_factory_blueprints. One blueprint per crawler created. The agent_slot field is initially null; the future specialist-agent factory populates it. See 00-Plan.md §15.

[ ] Step 4.1: Write failing tests

Create lib/factory/__tests__/blueprint-store.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';
import type { Blueprint } from '../types';

const mockSaveObject = vi.fn();
const mockGetObject = vi.fn();
const mockSearchSingleIndex = vi.fn();

vi.mock('algoliasearch', () => ({
  algoliasearch: () => ({
    saveObject: mockSaveObject,
    getObject: mockGetObject,
    searchSingleIndex: mockSearchSingleIndex,
  }),
}));

import { BlueprintStore } from '../blueprint-store';

const minimalBlueprint = (overrides: Partial<Blueprint> = {}): Blueprint => ({
  objectID: 'crawler_001',
  content_domain: 'marketing',
  index_name: 'algoliacentral_marketing',
  source_domain: 'algolia.com',
  path_groups: ['/blog/*'],
  schema_org_types: ['BlogPosting'],
  recordExtractor_path: 'crawler-configs/algolia.com/marketing-blog.js',
  algolia_config: {
    searchableAttributes: ['headline'],
    customRanking: ['desc(datePublished)'],
    attributesForFaceting: ['author'],
  },
  agent_slot: null,
  created_at: 1714432200000,
  status: 'live',
  ...overrides,
});

describe('BlueprintStore', () => {
  beforeEach(() => {
    mockSaveObject.mockReset();
    mockGetObject.mockReset();
    mockSearchSingleIndex.mockReset();
  });

  it('saveBlueprint writes to algoliacentral_factory_blueprints', async () => {
    mockSaveObject.mockResolvedValue({ taskID: 1, objectID: 'crawler_001' });
    const store = new BlueprintStore('app', 'key');
    await store.saveBlueprint(minimalBlueprint());
    expect(mockSaveObject).toHaveBeenCalledWith({
      indexName: 'algoliacentral_factory_blueprints',
      body: expect.objectContaining({ objectID: 'crawler_001', agent_slot: null }),
    });
  });

  it('getBlueprint returns null when not found', async () => {
    mockGetObject.mockRejectedValue(new Error('ObjectID does not exist'));
    const store = new BlueprintStore('app', 'key');
    const result = await store.getBlueprint('missing');
    expect(result).toBeNull();
  });

  it('listBlueprintsByContentDomain filters by domain', async () => {
    mockSearchSingleIndex.mockResolvedValue({
      hits: [{ ...minimalBlueprint(), content_domain: 'support' }],
      nbHits: 1,
    });
    const store = new BlueprintStore('app', 'key');
    const results = await store.listBlueprintsByContentDomain('support');
    expect(results).toHaveLength(1);
    expect(mockSearchSingleIndex).toHaveBeenCalledWith({
      indexName: 'algoliacentral_factory_blueprints',
      searchParams: expect.objectContaining({
        filters: 'content_domain:support',
      }),
    });
  });
});

[ ] Step 4.2: Run, expect FAIL

npx vitest run lib/factory/__tests__/blueprint-store.test.ts

Expected: FAIL: module not found.

[ ] Step 4.3: Implement lib/factory/blueprint-store.ts

Create lib/factory/blueprint-store.ts:

/**
 * Blueprint store — one record per crawler created. The agent_slot field is
 * the link to the future specialist-agent factory (00-Plan.md §15).
 */

import { algoliasearch, type SearchClient } from 'algoliasearch';
import { createLogger } from '../utils/logger';
import { BlueprintSchema, type Blueprint, type ContentDomain } from './types';

const log = createLogger('factory:blueprint-store');

const INDEX = 'algoliacentral_factory_blueprints';

export class BlueprintStore {
  private client: SearchClient;

  constructor(appId: string, apiKey: string) {
    this.client = algoliasearch(appId, apiKey);
  }

  async saveBlueprint(bp: Blueprint): Promise<void> {
    const validated = BlueprintSchema.parse(bp);
    try {
      await this.client.saveObject({ indexName: INDEX, body: validated });
      log.info(
        { event: 'saveBlueprint', crawlerId: validated.objectID, domain: validated.content_domain },
        'blueprint saved'
      );
    } catch (err) {
      log.error(
        { event: 'saveBlueprint_failed', crawlerId: validated.objectID, err: String(err) },
        'failed to save blueprint'
      );
      throw err;
    }
  }

  async getBlueprint(crawlerId: string): Promise<Blueprint | null> {
    try {
      const obj = await this.client.getObject({ indexName: INDEX, objectID: crawlerId });
      return BlueprintSchema.parse(obj);
    } catch (err) {
      if (String(err).includes('does not exist')) return null;
      log.error({ event: 'getBlueprint_failed', crawlerId, err: String(err) }, 'getBlueprint error');
      throw err;
    }
  }

  async listBlueprintsByContentDomain(domain: ContentDomain): Promise<Blueprint[]> {
    try {
      const result = await this.client.searchSingleIndex({
        indexName: INDEX,
        searchParams: {
          query: '',
          filters: `content_domain:${domain}`,
          hitsPerPage: 1000,
        },
      });
      return (result.hits as unknown[]).map((h) => BlueprintSchema.parse(h));
    } catch (err) {
      log.error(
        { event: 'listByDomain_failed', domain, err: String(err) },
        'failed to list blueprints by domain'
      );
      throw err;
    }
  }

  async listAllBlueprints(): Promise<Blueprint[]> {
    try {
      const result = await this.client.searchSingleIndex({
        indexName: INDEX,
        searchParams: { query: '', hitsPerPage: 1000 },
      });
      return (result.hits as unknown[]).map((h) => BlueprintSchema.parse(h));
    } catch (err) {
      log.error({ event: 'listAll_failed', err: String(err) }, 'failed to list all blueprints');
      throw err;
    }
  }
}

[ ] Step 4.4: Run all blueprint-store tests

npx vitest run lib/factory/__tests__/blueprint-store.test.ts

Expected: 3 tests pass.

[ ] Step 4.5: Run all factory tests + tsc

npx vitest run lib/factory/__tests__/ && npx tsc --noEmit

Expected: all factory tests pass; tsc clean.

[ ] Step 4.6: Commit

git add lib/factory/blueprint-store.ts lib/factory/__tests__/blueprint-store.test.ts
git commit -m "feat(factory): blueprint store with agent_slot field for future agent factory"

Files: - (Read-only verification + repository note)

[ ] Step 5.1: Run the full project test suite

npx vitest run

Expected: 407 baseline tests + 23+ new factory tests, all GREEN.

[ ] Step 5.2: Run full tsc

npx tsc --noEmit

Expected: clean.

[ ] Step 5.3: Note the Algolia facet configuration step

Spec 06 (Crawler API + Index Management) is responsible for calling setSettings on algoliacentral_factory_sessions and algoliacentral_factory_blueprints to configure facets: - algoliacentral_factory_sessions: attributesForFaceting: ['filterOnly(record_type)', 'filterOnly(parent_id)'] - algoliacentral_factory_blueprints: attributesForFaceting: ['filterOnly(content_domain)', 'filterOnly(status)']

Spec 01 does NOT do this: it simply assumes the indices exist with correct settings. Spec 06's index-manager.ts runs the setSettings step idempotently on first use.

This is a deliberate split: Spec 01 is pure data + I/O without index management.

[ ] Step 5.4: Sync vault + push branch (only if part of a larger commit batch)

# Optional — only if you're at a clean checkpoint
git push origin rc3-phoenix

[ ] Step 5.5: Mark Spec 01 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 1 (Foundations) | ⏸ | DSS, types, session/blueprint stores |
+ | Cluster 1 (Foundations) | ✅ | DSS, types, session/blueprint stores |

Acceptance criteria: Spec 01 done means:

✅ fast-xml-parser installed and externalized in bundler
✅ crawler-configs/ directory exists (empty, .gitkeep only)
✅ lib/factory/dss.ts exists with 9 default content domains, getDss, dssBySchemaOrgType, listContentDomains
✅ lib/factory/types.ts exports zod schemas: 9 record schemas, FactoryRecordSchema discriminated union, LogEntrySchema, PathGroupSchema, FactorySessionSchema, BlueprintSchema
✅ lib/factory/session-store.ts exports SessionStore class with saveSession, getSession, appendUrls, appendLog, addSample, getSamples
✅ lib/factory/blueprint-store.ts exports BlueprintStore class with saveBlueprint, getBlueprint, listBlueprintsByContentDomain, listAllBlueprints
✅ All new tests pass (~23+ tests)
✅ Full project test suite still GREEN
✅ tsc --noEmit clean
✅ Per CodingSOPs: every module has logger, every public function has docstring, every fallible call has try/catch, every boundary uses zod.

Out of scope (handled by other specs)

Index creation + facet configuration → Spec 06 (index-manager.ts)
Streaming reads of url_shards → Spec 02 (sitemap.ts + discover endpoint, which writes to the store; reads happen in Spec 11 frontend)
Sample structure analysis → Spec 05
Content classification of pathGroups → Spec 04
Real Algolia round-trip integration tests → Spec 13 (smoke test cluster)

Engineering-Specs/01-foundations.md

Crawler Factory: Spec 01: Foundations Implementation Plan

File structure

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Task 0: Add fast-xml-parser dependency + bundler externals + crawler-configs directory

Task 1: DSS registry: Domain Schema Standard

Task 2: Zod schemas: record discriminated union, session, blueprint, log

Task 3: Sharded session store

Task 4: Blueprint store

Task 5: Cluster validation: full regression + facet config note

Acceptance criteria: Spec 01 done means:

Out of scope (handled by other specs)

Task 0: Add `fast-xml-parser` dependency + bundler externals + crawler-configs directory