Dashboard/Crawler-Factory/Engineering-Specs/06-crawler-client-index-manager.md
Crawler-Factory

Engineering-Specs/06-crawler-client-index-manager.md

Crawler Factory: Spec 06: Crawler API Client + Index Manager Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Port the Python CrawlerClient (packages/crawl/src/crawl/manage_seeder.py:34-102) to TypeScript and add an IndexManager that idempotently creates/updates per-domain Algolia indices using DSS-derived settings: so downstream API endpoints (Specs 08+) can provision crawlers and their target indices through one cohesive interface.

Architecture: Two thin classes. CrawlerClient is a typed fetch wrapper around the Algolia Crawler REST API (https://crawler.algolia.com/api/1) with Basic-auth (base64(userId:apiKey)). IndexManager wraps the algoliasearch v5 client to ensure-or-update per-domain indices (settings pulled from lib/factory/dss.ts) and the two factory meta-indices (algoliacentral_factory_sessions, algoliacentral_factory_blueprints). neuralSearch is intentionally NOT enabled on cold-start indices: the Algolia API rejects it without prior Insights events; we set a neuralPending flag for the next-run promotion (Spec 13).

Tech Stack: TypeScript (strict), zod 3.25 for boundary validation, algoliasearch v5, native fetch (no httpx/axios), vitest with vi.mock for algoliasearch and vi.spyOn(globalThis, 'fetch') for the crawler client, pino logger via lib/utils/logger.ts.

Depends on: Spec 01: imports getDss, listContentDomains, ContentDomain from lib/factory/dss.ts. Consumed by: Spec 08 (/api/factory/create-crawler), Spec 09 (/api/factory/test-real), Spec 11 (frontend status panel via stats), Spec 13 (neural promotion smoke).

Plan source: Projects/Crawler-Factory/00-Plan.md §3c (per-domain index design), §7 Tasks 12–13 (TS Crawler client + Index manager), §16k Decision 3 (per-domain indices for multi-brand sites: each content domain × tenant gets its own index, never a shared one), §4e (agent_slot and blueprint metadata), §15 (cold-start neural deferral).

Python reference: packages/crawl/src/crawl/manage_seeder.py:34-102 defines the source-of-truth API call shapes (paths, headers, request bodies, response keys). The TypeScript port mirrors method-for-method.

File structure

Path Responsibility
lib/factory/crawler-client.ts Typed wrapper around the Algolia Crawler REST API. Auth, createCrawler, patchConfig, crawlUrls, startReindex, getStatus, getStats. Native fetch.
lib/factory/index-manager.ts Idempotent index creation + settings reconciliation against DSS. Handles both per-domain content indices and the two factory meta-indices. Cold-start neural deferral.
lib/factory/__tests__/crawler-client.test.ts Unit tests with mocked global fetch. One per public method + auth-header + 4xx error case.
lib/factory/__tests__/index-manager.test.ts Unit tests with mocked algoliasearch. Idempotent ensure, drift-update, cold-start neural-pending, factory meta-indices.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (request bodies, response payloads, env reads); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when absence is meaningful.
  2. Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:crawler-client'); (or factory:index-manager). No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
  3. Try/catch every fallible call. fetch, algoliasearch.*, process.env reads when required. Catch specific first, generic last. Always log before rethrow with full context (sanitize auth headers: never log the api key or the full Authorization header).
  4. Functions ≤20 lines, ≤3 params. More than 3 params → use a typed config object.
  5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. Crawler API responses go through zod parse before returning to callers. DSS reads return DssEntry, not Record<string, unknown>.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock external services in unit tests. Use vi.mock for algoliasearch and vi.spyOn(globalThis, 'fetch') for the crawler API. Real round-trip lives in Spec 13 smoke tests, gated by env vars.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
  10. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.
  11. Env var discipline. ALGOLIA_CRAWLER_USER_ID and ALGOLIA_CRAWLER_API_KEY are constructor-injected (not read inside class methods), so unit tests don't have to manipulate process.env. The construction site (Spec 08 endpoints) does the env read.

Task 1: TS CrawlerClient: class skeleton + Basic auth header

Files: - Create: lib/factory/crawler-client.ts - Create: lib/factory/__tests__/crawler-client.test.ts

The Python reference (manage_seeder.py:34-46) builds the auth header once at construction:

self._auth_header = "Basic " + base64.b64encode(f"{user_id}:{api_key}".encode()).decode()

Mirror that exactly.

  • [ ] Step 1.1: Write failing test for the auth header format

Create lib/factory/__tests__/crawler-client.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { CrawlerClient } from '../crawler-client';

const mockFetch = vi.fn();

beforeEach(() => {
  vi.stubGlobal('fetch', mockFetch);
  mockFetch.mockReset();
});

afterEach(() => {
  vi.unstubAllGlobals();
});

describe('CrawlerClient — auth header', () => {
  it('builds a Basic header from base64(userId:apiKey)', async () => {
    mockFetch.mockResolvedValue({
      ok: true,
      status: 200,
      json: async () => ({ crawlerId: 'c1' }),
    });
    const client = new CrawlerClient('user-abc', 'key-xyz');
    await client.createCrawler({ name: 'demo', config: { appId: 'a', apiKey: 'k', indexPrefix: 'p' } });
    const expected = 'Basic ' + Buffer.from('user-abc:key-xyz').toString('base64');
    const callInit = mockFetch.mock.calls[0][1] as RequestInit;
    expect((callInit.headers as Record<string, string>).Authorization).toBe(expected);
    expect((callInit.headers as Record<string, string>)['Content-Type']).toBe('application/json');
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts

Expected: FAIL: Cannot find module '../crawler-client'.

  • [ ] Step 1.3: Implement skeleton + auth + createCrawler

Create lib/factory/crawler-client.ts:

/**
 * TypeScript port of the Python CrawlerClient
 * (packages/crawl/src/crawl/manage_seeder.py:34-102).
 *
 * Why: Spec 08 endpoints run on Vercel Node — we cannot import Python.
 * The wire format and method shapes are 1-for-1 with the reference.
 */

import { z } from 'zod';
import { createLogger } from '../utils/logger';

const log = createLogger('factory:crawler-client');

const CRAWLER_BASE = 'https://crawler.algolia.com/api/1';

// ─── Request/response zod schemas ───────────────────────────────────────────

const CreateCrawlerResponseSchema = z.object({
  crawlerId: z.string().min(1),
});

const TaskResponseSchema = z.object({
  taskId: z.string().min(1),
});

const StatusResponseSchema = z.object({
  id: z.string(),
  name: z.string(),
  reindexing: z.boolean().optional(),
  config: z.record(z.unknown()).optional(),
}).passthrough();

const StatsResponseSchema = z.record(z.unknown());

export interface CreateCrawlerInput {
  /** Human-readable crawler name (becomes part of the dashboard URL). */
  name: string;
  /** Initial crawler config block — passed through to PATCH /config later. */
  config: Record<string, unknown>;
}

export interface CrawlerStatus {
  id: string;
  name: string;
  reindexing?: boolean;
  config?: Record<string, unknown>;
  [k: string]: unknown;
}

/**
 * Thin typed wrapper around the Algolia Crawler REST API.
 * Auth: Basic base64(userId:apiKey) — separate from Algolia search credentials.
 */
export class CrawlerClient {
  private readonly authHeader: string;

  constructor(userId: string, apiKey: string) {
    if (!userId || !apiKey) {
      throw new Error('CrawlerClient requires both userId and apiKey');
    }
    // Buffer is available in Node 18+; Vercel functions use Node 20.
    const token = Buffer.from(`${userId}:${apiKey}`).toString('base64');
    this.authHeader = `Basic ${token}`;
  }

  /**
   * Create a new crawler. POST /1/crawlers — returns { crawlerId }.
   */
  async createCrawler(input: CreateCrawlerInput): Promise<{ crawlerId: string }> {
    const body = JSON.stringify(input);
    const res = await this.request('POST', '/crawlers', body);
    const parsed = CreateCrawlerResponseSchema.parse(res);
    log.info({ event: 'createCrawler', crawlerId: parsed.crawlerId, name: input.name }, 'crawler created');
    return parsed;
  }

  // (later steps add patchConfig, crawlUrls, startReindex, getStatus, getStats)

  /**
   * Internal request helper — issues the fetch, handles 4xx/5xx, logs failures.
   * Sanitizes auth out of error logs.
   */
  private async request(method: string, path: string, body?: string): Promise<unknown> {
    const url = `${CRAWLER_BASE}${path}`;
    let res: Response;
    try {
      res = await fetch(url, {
        method,
        headers: {
          Authorization: this.authHeader,
          'Content-Type': 'application/json',
        },
        body,
      });
    } catch (err) {
      log.error({ event: 'request_network_error', method, path, err: String(err) }, 'crawler API network error');
      throw err;
    }
    if (!res.ok) {
      const text = await res.text().catch(() => '');
      log.error(
        { event: 'request_http_error', method, path, status: res.status, body: text.slice(0, 500) },
        'crawler API HTTP error'
      );
      throw new Error(`Crawler API ${method} ${path} failed: ${res.status} ${text.slice(0, 200)}`);
    }
    try {
      return await res.json();
    } catch (err) {
      log.error({ event: 'request_json_error', method, path, err: String(err) }, 'crawler API non-JSON response');
      throw err;
    }
  }
}
  • [ ] Step 1.4: Run, expect PASS
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t auth

Expected: 1 test passes.

  • [ ] Step 1.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient skeleton with Basic auth + createCrawler"

Task 2: CrawlerClient.patchConfig

Files: - Modify: lib/factory/crawler-client.ts - Modify: lib/factory/__tests__/crawler-client.test.ts

Python reference (manage_seeder.py:54-61):

def patch_config(self, crawler_id: str, patch: dict[str, Any]) -> None:
    resp = self._http.patch(
        f"{_CRAWLER_BASE}/crawlers/{crawler_id}/config",
        json=patch,
    )
    resp.raise_for_status()
  • [ ] Step 2.1: Write failing test for patchConfig

Append to lib/factory/__tests__/crawler-client.test.ts:

describe('CrawlerClient.patchConfig', () => {
  it('PATCHes /1/crawlers/{id}/config with the given patch body', async () => {
    mockFetch.mockResolvedValue({ ok: true, status: 200, json: async () => ({}) });
    const client = new CrawlerClient('u', 'k');
    await client.patchConfig('crawler_xyz', { startUrls: ['https://x.com/'] });
    expect(mockFetch).toHaveBeenCalledTimes(1);
    const [url, init] = mockFetch.mock.calls[0];
    expect(url).toBe('https://crawler.algolia.com/api/1/crawlers/crawler_xyz/config');
    expect((init as RequestInit).method).toBe('PATCH');
    expect((init as RequestInit).body).toBe(JSON.stringify({ startUrls: ['https://x.com/'] }));
  });

  it('throws on 4xx with body fragment in error message', async () => {
    mockFetch.mockResolvedValue({
      ok: false,
      status: 400,
      text: async () => '{"error":"bad startUrls"}',
    });
    const client = new CrawlerClient('u', 'k');
    await expect(client.patchConfig('crawler_xyz', { startUrls: [] })).rejects.toThrow(/400/);
  });
});
  • [ ] Step 2.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t patchConfig

Expected: FAIL: client.patchConfig is not a function.

  • [ ] Step 2.3: Add patchConfig to crawler-client.ts

Insert after createCrawler:

  /**
   * Partial-update a crawler's config. PATCH /1/crawlers/{id}/config.
   * Only keys present in `patch` are changed.
   */
  async patchConfig(crawlerId: string, patch: Record<string, unknown>): Promise<void> {
    await this.request('PATCH', `/crawlers/${encodeURIComponent(crawlerId)}/config`, JSON.stringify(patch));
    log.info({ event: 'patchConfig', crawlerId, keys: Object.keys(patch) }, 'crawler config patched');
  }
  • [ ] Step 2.4: Run patchConfig tests, expect PASS
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t patchConfig

Expected: 2 tests pass.

  • [ ] Step 2.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient.patchConfig with 4xx error handling"

Task 3: CrawlerClient.crawlUrls

Files: - Modify: lib/factory/crawler-client.ts - Modify: lib/factory/__tests__/crawler-client.test.ts

Python reference (manage_seeder.py:73-91): POST /1/crawlers/{id}/urls/crawl with {urls, save} → returns {taskId}. Rate limit: 500 req/24h (Algolia Crawler API limit; documented for callers, not enforced here).

  • [ ] Step 3.1: Write failing test for crawlUrls

Append to lib/factory/__tests__/crawler-client.test.ts:

describe('CrawlerClient.crawlUrls', () => {
  it('POSTs urls + save flag and returns taskId', async () => {
    mockFetch.mockResolvedValue({
      ok: true,
      status: 200,
      json: async () => ({ taskId: 'task_1' }),
    });
    const client = new CrawlerClient('u', 'k');
    const taskId = await client.crawlUrls('crawler_xyz', ['https://x.com/a', 'https://x.com/b'], true);
    expect(taskId).toBe('task_1');
    const [url, init] = mockFetch.mock.calls[0];
    expect(url).toBe('https://crawler.algolia.com/api/1/crawlers/crawler_xyz/urls/crawl');
    expect((init as RequestInit).method).toBe('POST');
    expect(JSON.parse((init as RequestInit).body as string)).toEqual({
      urls: ['https://x.com/a', 'https://x.com/b'],
      save: true,
    });
  });

  it('defaults save to false when omitted', async () => {
    mockFetch.mockResolvedValue({ ok: true, status: 200, json: async () => ({ taskId: 't2' }) });
    const client = new CrawlerClient('u', 'k');
    await client.crawlUrls('crawler_xyz', ['https://x.com/a']);
    const init = mockFetch.mock.calls[0][1] as RequestInit;
    expect(JSON.parse(init.body as string).save).toBe(false);
  });
});
  • [ ] Step 3.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t crawlUrls

Expected: FAIL: client.crawlUrls is not a function.

  • [ ] Step 3.3: Add crawlUrls

Insert in crawler-client.ts:

  /**
   * Trigger an immediate crawl of specific URLs. POST /1/crawlers/{id}/urls/crawl.
   * Returns the Algolia taskId. `save=true` persists URLs to extraUrls config.
   * Algolia rate-limits this endpoint to 500 requests per 24 hours per crawler.
   */
  async crawlUrls(crawlerId: string, urls: string[], save: boolean = false): Promise<string> {
    const body = JSON.stringify({ urls, save });
    const res = await this.request(
      'POST',
      `/crawlers/${encodeURIComponent(crawlerId)}/urls/crawl`,
      body
    );
    const parsed = TaskResponseSchema.parse(res);
    log.info(
      { event: 'crawlUrls', crawlerId, count: urls.length, save, taskId: parsed.taskId },
      'crawl_urls submitted'
    );
    return parsed.taskId;
  }
  • [ ] Step 3.4: Run, expect PASS
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t crawlUrls

Expected: 2 tests pass.

  • [ ] Step 3.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient.crawlUrls returns Algolia taskId"

Task 4: CrawlerClient.startReindex

Files: - Modify: lib/factory/crawler-client.ts - Modify: lib/factory/__tests__/crawler-client.test.ts

Python reference (manage_seeder.py:93-99): POST /1/crawlers/{id}/reindex → returns {taskId}.

  • [ ] Step 4.1: Write failing test

Append:

describe('CrawlerClient.startReindex', () => {
  it('POSTs to /reindex and returns taskId', async () => {
    mockFetch.mockResolvedValue({
      ok: true,
      status: 200,
      json: async () => ({ taskId: 'reindex_99' }),
    });
    const client = new CrawlerClient('u', 'k');
    const taskId = await client.startReindex('crawler_xyz');
    expect(taskId).toBe('reindex_99');
    const [url, init] = mockFetch.mock.calls[0];
    expect(url).toBe('https://crawler.algolia.com/api/1/crawlers/crawler_xyz/reindex');
    expect((init as RequestInit).method).toBe('POST');
  });
});
  • [ ] Step 4.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t startReindex

Expected: FAIL.

  • [ ] Step 4.3: Add startReindex

Insert:

  /**
   * Start a full crawl. POST /1/crawlers/{id}/reindex. Returns the taskId.
   */
  async startReindex(crawlerId: string): Promise<string> {
    const res = await this.request('POST', `/crawlers/${encodeURIComponent(crawlerId)}/reindex`);
    const parsed = TaskResponseSchema.parse(res);
    log.info({ event: 'startReindex', crawlerId, taskId: parsed.taskId }, 'reindex started');
    return parsed.taskId;
  }
  • [ ] Step 4.4: Run, expect PASS
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t startReindex

Expected: 1 test passes.

  • [ ] Step 4.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient.startReindex"

Task 5: CrawlerClient.getStatus (with optional withConfig)

Files: - Modify: lib/factory/crawler-client.ts - Modify: lib/factory/__tests__/crawler-client.test.ts

Python reference (manage_seeder.py:47-52): GET /1/crawlers/{id}?withConfig=true. The TS port exposes withConfig as an option on the method (default false to keep payloads small for status-only polls).

  • [ ] Step 5.1: Write failing tests

Append:

describe('CrawlerClient.getStatus', () => {
  it('GETs /1/crawlers/{id} without withConfig by default', async () => {
    mockFetch.mockResolvedValue({
      ok: true,
      status: 200,
      json: async () => ({ id: 'crawler_xyz', name: 'demo', reindexing: false }),
    });
    const client = new CrawlerClient('u', 'k');
    const status = await client.getStatus('crawler_xyz');
    expect(status.id).toBe('crawler_xyz');
    expect(mockFetch.mock.calls[0][0]).toBe('https://crawler.algolia.com/api/1/crawlers/crawler_xyz');
  });

  it('appends ?withConfig=true when option set', async () => {
    mockFetch.mockResolvedValue({
      ok: true,
      status: 200,
      json: async () => ({ id: 'crawler_xyz', name: 'demo', config: { startUrls: [] } }),
    });
    const client = new CrawlerClient('u', 'k');
    const status = await client.getStatus('crawler_xyz', { withConfig: true });
    expect(status.config).toBeDefined();
    expect(mockFetch.mock.calls[0][0]).toBe(
      'https://crawler.algolia.com/api/1/crawlers/crawler_xyz?withConfig=true'
    );
  });
});
  • [ ] Step 5.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t getStatus

Expected: FAIL.

  • [ ] Step 5.3: Add getStatus

Insert:

  /**
   * Fetch crawler details. GET /1/crawlers/{id}.
   * Pass `{ withConfig: true }` to also receive the full crawler config block.
   */
  async getStatus(
    crawlerId: string,
    options: { withConfig?: boolean } = {}
  ): Promise<CrawlerStatus> {
    const qs = options.withConfig ? '?withConfig=true' : '';
    const res = await this.request('GET', `/crawlers/${encodeURIComponent(crawlerId)}${qs}`);
    return StatusResponseSchema.parse(res);
  }
  • [ ] Step 5.4: Run, expect PASS
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t getStatus

Expected: 2 tests pass.

  • [ ] Step 5.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient.getStatus with optional withConfig"

Task 6: CrawlerClient.getStats

Files: - Modify: lib/factory/crawler-client.ts - Modify: lib/factory/__tests__/crawler-client.test.ts

GET /1/crawlers/{id}/stats/urls: returns crawl progress shape (varies by Algolia; we return as Record<string, unknown> and let Spec 11 frontend parse against its own zod). The Python reference doesn't have this method: it's a TS-only addition because the frontend Crawl Monitor (Spec 11) needs it.

  • [ ] Step 6.1: Write failing test

Append:

describe('CrawlerClient.getStats', () => {
  it('GETs /1/crawlers/{id}/stats/urls and returns the raw payload', async () => {
    const payload = { count: 1234, errors: 2, indexed: 1232 };
    mockFetch.mockResolvedValue({ ok: true, status: 200, json: async () => payload });
    const client = new CrawlerClient('u', 'k');
    const stats = await client.getStats('crawler_xyz');
    expect(stats).toEqual(payload);
    expect(mockFetch.mock.calls[0][0]).toBe(
      'https://crawler.algolia.com/api/1/crawlers/crawler_xyz/stats/urls'
    );
  });

  it('throws on 401 with sanitized message (no auth header leak)', async () => {
    mockFetch.mockResolvedValue({
      ok: false,
      status: 401,
      text: async () => 'Unauthorized',
    });
    const client = new CrawlerClient('u', 'k');
    await expect(client.getStats('crawler_xyz')).rejects.toThrow(/401/);
    // Asserting the error message does NOT contain the api key
    try {
      await client.getStats('crawler_xyz');
    } catch (err) {
      expect(String(err)).not.toContain('u:k');
      expect(String(err)).not.toContain('Basic ');
    }
  });
});
  • [ ] Step 6.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/crawler-client.test.ts -t getStats

Expected: FAIL.

  • [ ] Step 6.3: Add getStats

Insert:

  /**
   * Fetch URL-level stats for a crawler.
   * GET /1/crawlers/{id}/stats/urls — schema varies; we surface as a record.
   */
  async getStats(crawlerId: string): Promise<Record<string, unknown>> {
    const res = await this.request('GET', `/crawlers/${encodeURIComponent(crawlerId)}/stats/urls`);
    return StatsResponseSchema.parse(res);
  }
  • [ ] Step 6.4: Run all crawler-client tests + tsc
npx vitest run lib/factory/__tests__/crawler-client.test.ts && npx tsc --noEmit

Expected: all crawler-client tests GREEN; tsc clean.

  • [ ] Step 6.5: Commit
git add lib/factory/crawler-client.ts lib/factory/__tests__/crawler-client.test.ts
git commit -m "feat(factory): CrawlerClient.getStats + sanitized auth in error logs"

Task 7: IndexManager: class skeleton + ensureIndex happy path

Files: - Create: lib/factory/index-manager.ts - Create: lib/factory/__tests__/index-manager.test.ts

Per §16k Decision 3 (multi-brand federated corporates), each content domain on a tenant gets its OWN index, never shared across tenants. The IndexManager assumes the getDss(domain).indexName already encodes the canonical per-domain path; callers (Spec 08) are responsible for passing the right ContentDomain. We do NOT do tenant prefixing here; that is a higher-level concern flagged for v2.

ensureIndex(domain) is idempotent: - Missing index → call setSettings with the DSS-derived block; Algolia auto-creates the index on first setSettings. - Existing index, settings drift from DSS → call setSettings with the DSS-derived block. - Existing index, settings match → no-op.

Drift detection compares only the three keys we manage (searchableAttributes, customRanking, attributesForFaceting). Other operator-set settings (e.g., ranking, replicas) are left untouched.

  • [ ] Step 7.1: Write failing test for the missing-index case

Create lib/factory/__tests__/index-manager.test.ts:

import { describe, it, expect, vi, beforeEach } from 'vitest';

const mockGetSettings = vi.fn();
const mockSetSettings = vi.fn();

vi.mock('algoliasearch', () => ({
  algoliasearch: () => ({
    getSettings: mockGetSettings,
    setSettings: mockSetSettings,
  }),
}));

import { IndexManager } from '../index-manager';

beforeEach(() => {
  mockGetSettings.mockReset();
  mockSetSettings.mockReset();
});

describe('IndexManager.ensureIndex — cold start', () => {
  it('creates the index via setSettings when getSettings throws "Index does not exist"', async () => {
    mockGetSettings.mockRejectedValue(new Error('Index algoliacentral_marketing does not exist'));
    mockSetSettings.mockResolvedValue({ taskID: 1 });

    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureIndex('marketing');

    expect(result.created).toBe(true);
    expect(result.updated).toBe(false);
    expect(result.neuralPending).toBe(true);
    expect(mockSetSettings).toHaveBeenCalledTimes(1);
    expect(mockSetSettings).toHaveBeenCalledWith({
      indexName: 'algoliacentral_marketing',
      indexSettings: expect.objectContaining({
        searchableAttributes: expect.arrayContaining(['headline', 'articleBody']),
        customRanking: expect.arrayContaining([expect.stringMatching(/desc\(datePublished\)/)]),
        attributesForFaceting: expect.arrayContaining(['articleSection']),
      }),
    });
    // neuralSearch must NOT be set on cold start
    const setCall = mockSetSettings.mock.calls[0][0] as { indexSettings: Record<string, unknown> };
    expect(setCall.indexSettings).not.toHaveProperty('mode');
    expect(setCall.indexSettings).not.toHaveProperty('neuralSearch');
  });
});
  • [ ] Step 7.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/index-manager.test.ts

Expected: FAIL: Cannot find module '../index-manager'.

  • [ ] Step 7.3: Implement index-manager.ts skeleton + ensureIndex cold-start path

Create lib/factory/index-manager.ts:

/**
 * IndexManager — idempotent per-domain Algolia index creation + settings reconciliation.
 *
 * Why: per 00-Plan.md §3c, every content domain has its own index with DSS-derived
 * settings. This module is the single chokepoint for ensure-or-update so endpoints
 * (Spec 08) and the future agent factory (§15) can rely on a consistent surface.
 *
 * Cold-start neural deferral: per 00-Plan.md §3c, neuralSearch is enabled by default
 * but the Algolia API rejects it on a brand-new index without prior Insights events.
 * We log a warning and return `{ neuralPending: true }`; Spec 13 promotes neural once
 * events exist.
 */

import { algoliasearch, type SearchClient } from 'algoliasearch';
import { createLogger } from '../utils/logger';
import { getDss, type ContentDomain } from './dss';

const log = createLogger('factory:index-manager');

const FACTORY_SESSIONS_INDEX = 'algoliacentral_factory_sessions';
const FACTORY_BLUEPRINTS_INDEX = 'algoliacentral_factory_blueprints';

export interface EnsureResult {
  /** Index was created on this call (no prior settings existed). */
  created: boolean;
  /** Existing index had drift; settings were updated on this call. */
  updated: boolean;
  /** neuralSearch was deferred — caller (or Spec 13) must promote later. */
  neuralPending: boolean;
}

/**
 * Idempotent index lifecycle manager. Wraps algoliasearch v5 client.
 */
export class IndexManager {
  private client: SearchClient;

  constructor(appId: string, apiKey: string) {
    if (!appId || !apiKey) {
      throw new Error('IndexManager requires both appId and apiKey');
    }
    this.client = algoliasearch(appId, apiKey);
  }

  /**
   * Ensure the per-domain index exists with DSS-derived settings.
   * Idempotent — safe to call repeatedly.
   */
  async ensureIndex(contentDomain: ContentDomain): Promise<EnsureResult> {
    const dss = getDss(contentDomain);
    const desired = {
      searchableAttributes: [...dss.algoliaConfig.searchableAttributes],
      customRanking: [...dss.algoliaConfig.customRanking],
      attributesForFaceting: [...dss.algoliaConfig.attributesForFaceting],
    };

    let existing: Record<string, unknown> | null = null;
    try {
      existing = (await this.client.getSettings({ indexName: dss.indexName })) as Record<string, unknown>;
    } catch (err) {
      if (this.isMissingIndexError(err)) {
        existing = null;
      } else {
        log.error(
          { event: 'ensureIndex_getSettings_failed', indexName: dss.indexName, err: String(err) },
          'getSettings failed (non-missing error)'
        );
        throw err;
      }
    }

    if (existing === null) {
      await this.applySettings(dss.indexName, desired);
      log.warn(
        { event: 'ensureIndex_neural_pending', indexName: dss.indexName, contentDomain },
        'neuralSearch deferred — index has no Insights events yet'
      );
      return { created: true, updated: false, neuralPending: true };
    }

    // Settings already exist → check drift on the three keys we manage.
    return { created: false, updated: false, neuralPending: false };
  }

  /**
   * Internal: detect "Index does not exist" errors across algoliasearch v5 shapes.
   */
  private isMissingIndexError(err: unknown): boolean {
    const msg = String(err);
    return msg.includes('does not exist') || msg.includes('IndexNotFoundError');
  }

  /**
   * Internal: setSettings with try/catch + log.
   */
  private async applySettings(
    indexName: string,
    settings: {
      searchableAttributes: string[];
      customRanking: string[];
      attributesForFaceting: string[];
    }
  ): Promise<void> {
    try {
      await this.client.setSettings({ indexName, indexSettings: settings });
      log.info({ event: 'applySettings', indexName, keys: Object.keys(settings) }, 'settings applied');
    } catch (err) {
      log.error(
        { event: 'applySettings_failed', indexName, err: String(err) },
        'setSettings failed'
      );
      throw err;
    }
  }
}
  • [ ] Step 7.4: Run, expect PASS
npx vitest run lib/factory/__tests__/index-manager.test.ts -t cold

Expected: 1 test passes.

  • [ ] Step 7.5: Commit
git add lib/factory/index-manager.ts lib/factory/__tests__/index-manager.test.ts
git commit -m "feat(factory): IndexManager with cold-start ensureIndex + neural deferral"

Task 8: IndexManager.ensureIndex drift detection + idempotent no-op

Files: - Modify: lib/factory/index-manager.ts - Modify: lib/factory/__tests__/index-manager.test.ts

When the index already exists, compare current searchableAttributes, customRanking, attributesForFaceting against DSS. If any differ, call setSettings with the DSS-derived block. Otherwise, no-op.

Drift comparison is order-sensitive on customRanking and searchableAttributes (these affect ranking) and order-insensitive on attributesForFaceting (Algolia treats facet config as a set).

  • [ ] Step 8.1: Write failing tests for drift + no-op

Append to lib/factory/__tests__/index-manager.test.ts:

describe('IndexManager.ensureIndex — existing index', () => {
  it('is a no-op when settings already match DSS', async () => {
    mockGetSettings.mockResolvedValue({
      searchableAttributes: ['headline', 'articleBody', 'keywords'],
      customRanking: ['desc(datePublished)', 'desc(view_count)'],
      attributesForFaceting: ['articleSection', 'author', 'year'],
    });
    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureIndex('marketing');
    expect(result.created).toBe(false);
    expect(result.updated).toBe(false);
    expect(result.neuralPending).toBe(false);
    expect(mockSetSettings).not.toHaveBeenCalled();
  });

  it('updates settings when customRanking drifts (order matters)', async () => {
    mockGetSettings.mockResolvedValue({
      searchableAttributes: ['headline', 'articleBody', 'keywords'],
      customRanking: ['desc(view_count)', 'desc(datePublished)'], // wrong order
      attributesForFaceting: ['articleSection', 'author', 'year'],
    });
    mockSetSettings.mockResolvedValue({ taskID: 1 });

    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureIndex('marketing');
    expect(result.created).toBe(false);
    expect(result.updated).toBe(true);
    expect(result.neuralPending).toBe(false);
    expect(mockSetSettings).toHaveBeenCalledTimes(1);
  });

  it('treats attributesForFaceting drift as set-equality (any-order match = no-op)', async () => {
    mockGetSettings.mockResolvedValue({
      searchableAttributes: ['headline', 'articleBody', 'keywords'],
      customRanking: ['desc(datePublished)', 'desc(view_count)'],
      attributesForFaceting: ['year', 'author', 'articleSection'], // reordered
    });
    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureIndex('marketing');
    expect(result.updated).toBe(false);
    expect(mockSetSettings).not.toHaveBeenCalled();
  });

  it('updates when attributesForFaceting set-content differs', async () => {
    mockGetSettings.mockResolvedValue({
      searchableAttributes: ['headline', 'articleBody', 'keywords'],
      customRanking: ['desc(datePublished)', 'desc(view_count)'],
      attributesForFaceting: ['articleSection'], // missing two
    });
    mockSetSettings.mockResolvedValue({ taskID: 1 });
    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureIndex('marketing');
    expect(result.updated).toBe(true);
    expect(mockSetSettings).toHaveBeenCalledTimes(1);
  });
});
  • [ ] Step 8.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/index-manager.test.ts -t "existing index"

Expected: 4 tests fail (drift + no-op logic not yet implemented).

  • [ ] Step 8.3: Add drift detection to ensureIndex

Replace the existing !== null branch in ensureIndex:

    if (existing === null) {
      await this.applySettings(dss.indexName, desired);
      log.warn(
        { event: 'ensureIndex_neural_pending', indexName: dss.indexName, contentDomain },
        'neuralSearch deferred — index has no Insights events yet'
      );
      return { created: true, updated: false, neuralPending: true };
    }

    // Drift detection on the three keys we manage.
    const drift = this.detectDrift(existing, desired);
    if (drift) {
      await this.applySettings(dss.indexName, desired);
      log.info(
        { event: 'ensureIndex_drift_corrected', indexName: dss.indexName, drift },
        'settings drift corrected'
      );
      return { created: false, updated: true, neuralPending: false };
    }

    return { created: false, updated: false, neuralPending: false };
  }

  /**
   * Internal: detect drift on the three managed keys.
   * Returns a brief reason-string for logging, or null when in sync.
   * - searchableAttributes / customRanking — order-sensitive
   * - attributesForFaceting — set-equality (any order)
   */
  private detectDrift(
    existing: Record<string, unknown>,
    desired: { searchableAttributes: string[]; customRanking: string[]; attributesForFaceting: string[] }
  ): string | null {
    const exSearchable = (existing.searchableAttributes as string[] | undefined) ?? [];
    const exRanking = (existing.customRanking as string[] | undefined) ?? [];
    const exFacets = (existing.attributesForFaceting as string[] | undefined) ?? [];

    if (!arraysEqualOrdered(exSearchable, desired.searchableAttributes)) return 'searchableAttributes';
    if (!arraysEqualOrdered(exRanking, desired.customRanking)) return 'customRanking';
    if (!arraysEqualUnordered(exFacets, desired.attributesForFaceting)) return 'attributesForFaceting';
    return null;
  }

Add at the bottom of the file (module-level helpers):

function arraysEqualOrdered(a: readonly string[], b: readonly string[]): boolean {
  if (a.length !== b.length) return false;
  for (let i = 0; i < a.length; i += 1) {
    if (a[i] !== b[i]) return false;
  }
  return true;
}

function arraysEqualUnordered(a: readonly string[], b: readonly string[]): boolean {
  if (a.length !== b.length) return false;
  const sa = new Set(a);
  for (const v of b) {
    if (!sa.has(v)) return false;
  }
  return true;
}
  • [ ] Step 8.4: Run, expect PASS
npx vitest run lib/factory/__tests__/index-manager.test.ts

Expected: all 5 ensureIndex tests pass.

  • [ ] Step 8.5: Commit
git add lib/factory/index-manager.ts lib/factory/__tests__/index-manager.test.ts
git commit -m "feat(factory): IndexManager drift detection + idempotent no-op"

Task 9: IndexManager.ensureFactoryMetaIndices

Files: - Modify: lib/factory/index-manager.ts - Modify: lib/factory/__tests__/index-manager.test.ts

Spec 01 §5.3 explicitly defers facet configuration of algoliacentral_factory_sessions and algoliacentral_factory_blueprints to this spec. Settings:

  • algoliacentral_factory_sessions: attributesForFaceting: ['filterOnly(record_type)', 'filterOnly(parent_id)']: these power the sharded reads in Spec 01's SessionStore.
  • algoliacentral_factory_blueprints: attributesForFaceting: ['filterOnly(content_domain)', 'filterOnly(status)']: these power listBlueprintsByContentDomain in Spec 01's BlueprintStore.

Both are idempotent: created on first call, drift-corrected after.

  • [ ] Step 9.1: Write failing tests

Append:

describe('IndexManager.ensureFactoryMetaIndices', () => {
  it('creates both meta-indices on cold start with the correct facet config', async () => {
    mockGetSettings.mockRejectedValue(new Error('Index does not exist'));
    mockSetSettings.mockResolvedValue({ taskID: 1 });

    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureFactoryMetaIndices();

    expect(result.sessions.created).toBe(true);
    expect(result.blueprints.created).toBe(true);
    expect(mockSetSettings).toHaveBeenCalledTimes(2);

    const callArgs = mockSetSettings.mock.calls.map((c) => c[0] as { indexName: string; indexSettings: Record<string, unknown> });
    const sessionsCall = callArgs.find((c) => c.indexName === 'algoliacentral_factory_sessions');
    const blueprintsCall = callArgs.find((c) => c.indexName === 'algoliacentral_factory_blueprints');
    expect(sessionsCall?.indexSettings.attributesForFaceting).toEqual([
      'filterOnly(record_type)',
      'filterOnly(parent_id)',
    ]);
    expect(blueprintsCall?.indexSettings.attributesForFaceting).toEqual([
      'filterOnly(content_domain)',
      'filterOnly(status)',
    ]);
  });

  it('is a no-op when both meta-indices already have the right facets', async () => {
    mockGetSettings.mockImplementation(async ({ indexName }: { indexName: string }) => {
      if (indexName === 'algoliacentral_factory_sessions') {
        return { attributesForFaceting: ['filterOnly(record_type)', 'filterOnly(parent_id)'] };
      }
      return { attributesForFaceting: ['filterOnly(content_domain)', 'filterOnly(status)'] };
    });
    const manager = new IndexManager('app-id', 'admin-key');
    const result = await manager.ensureFactoryMetaIndices();
    expect(result.sessions.created).toBe(false);
    expect(result.sessions.updated).toBe(false);
    expect(result.blueprints.created).toBe(false);
    expect(result.blueprints.updated).toBe(false);
    expect(mockSetSettings).not.toHaveBeenCalled();
  });
});
  • [ ] Step 9.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/index-manager.test.ts -t ensureFactoryMetaIndices

Expected: FAIL: method not implemented.

  • [ ] Step 9.3: Add ensureFactoryMetaIndices + helper for facet-only drift

Insert into the IndexManager class (above the private helpers):

  /**
   * Ensure the two factory meta-indices exist with the correct facet config.
   * Idempotent. Called once per app boot (or by Spec 13 smoke).
   */
  async ensureFactoryMetaIndices(): Promise<{ sessions: EnsureResult; blueprints: EnsureResult }> {
    const sessions = await this.ensureFacetOnly(FACTORY_SESSIONS_INDEX, [
      'filterOnly(record_type)',
      'filterOnly(parent_id)',
    ]);
    const blueprints = await this.ensureFacetOnly(FACTORY_BLUEPRINTS_INDEX, [
      'filterOnly(content_domain)',
      'filterOnly(status)',
    ]);
    return { sessions, blueprints };
  }

  /**
   * Internal: ensure a meta-index has exactly the given attributesForFaceting list.
   * Other settings are not touched. Order-sensitive (matches what Algolia returns).
   */
  private async ensureFacetOnly(indexName: string, facets: string[]): Promise<EnsureResult> {
    let existing: Record<string, unknown> | null = null;
    try {
      existing = (await this.client.getSettings({ indexName })) as Record<string, unknown>;
    } catch (err) {
      if (this.isMissingIndexError(err)) {
        existing = null;
      } else {
        log.error({ event: 'ensureFacetOnly_getSettings_failed', indexName, err: String(err) }, 'getSettings failed');
        throw err;
      }
    }
    if (existing === null) {
      try {
        await this.client.setSettings({ indexName, indexSettings: { attributesForFaceting: facets } });
        log.info({ event: 'ensureFacetOnly_created', indexName, facets }, 'meta index created');
      } catch (err) {
        log.error({ event: 'ensureFacetOnly_setSettings_failed', indexName, err: String(err) }, 'setSettings failed');
        throw err;
      }
      return { created: true, updated: false, neuralPending: false };
    }
    const exFacets = (existing.attributesForFaceting as string[] | undefined) ?? [];
    if (arraysEqualOrdered(exFacets, facets)) {
      return { created: false, updated: false, neuralPending: false };
    }
    try {
      await this.client.setSettings({ indexName, indexSettings: { attributesForFaceting: facets } });
      log.info({ event: 'ensureFacetOnly_drift_corrected', indexName, facets }, 'meta index facets updated');
    } catch (err) {
      log.error({ event: 'ensureFacetOnly_setSettings_failed', indexName, err: String(err) }, 'setSettings failed');
      throw err;
    }
    return { created: false, updated: true, neuralPending: false };
  }
  • [ ] Step 9.4: Run, expect PASS
npx vitest run lib/factory/__tests__/index-manager.test.ts -t ensureFactoryMetaIndices

Expected: 2 tests pass.

  • [ ] Step 9.5: Run all factory tests + tsc
npx vitest run lib/factory/__tests__/ && npx tsc --noEmit

Expected: all factory tests GREEN; tsc clean.

  • [ ] Step 9.6: Commit
git add lib/factory/index-manager.ts lib/factory/__tests__/index-manager.test.ts
git commit -m "feat(factory): IndexManager.ensureFactoryMetaIndices for sessions+blueprints"

Task 10: 4xx error handling test for IndexManager

Files: - Modify: lib/factory/__tests__/index-manager.test.ts

Confirms that non-"missing-index" errors from getSettings (auth failure, rate limit, network) propagate rather than being silently treated as cold start.

  • [ ] Step 10.1: Write failing test

Append:

describe('IndexManager — error propagation', () => {
  it('propagates non-missing errors from getSettings', async () => {
    mockGetSettings.mockRejectedValue(new Error('Invalid Application-ID or API key'));
    const manager = new IndexManager('app-id', 'admin-key');
    await expect(manager.ensureIndex('marketing')).rejects.toThrow(/Invalid Application-ID/);
    expect(mockSetSettings).not.toHaveBeenCalled();
  });

  it('propagates setSettings failure', async () => {
    mockGetSettings.mockRejectedValue(new Error('Index does not exist'));
    mockSetSettings.mockRejectedValue(new Error('Rate limit exceeded'));
    const manager = new IndexManager('app-id', 'admin-key');
    await expect(manager.ensureIndex('marketing')).rejects.toThrow(/Rate limit/);
  });
});
  • [ ] Step 10.2: Run, expect PASS (logic already correct)
npx vitest run lib/factory/__tests__/index-manager.test.ts -t "error propagation"

Expected: 2 tests pass: the existing implementation already throws on non-missing errors. (If they fail, that's a real bug in Task 7/8 implementation; fix before moving on.)

  • [ ] Step 10.3: Commit
git add lib/factory/__tests__/index-manager.test.ts
git commit -m "test(factory): pin IndexManager error propagation behavior"

Task 11: Cluster validation: full regression + tsc

Files: (read-only verification)

  • [ ] Step 11.1: Run the full project test suite
npx vitest run

Expected: Spec 01 baseline (407 + ~23 factory tests from Spec 01) + Spec 06 additions (~16+ new tests): all GREEN.

  • [ ] Step 11.2: Run full tsc
npx tsc --noEmit

Expected: clean.

  • [ ] Step 11.3: Confirm no console.log and no leaked api-key strings
grep -RIn "console\." lib/factory/crawler-client.ts lib/factory/index-manager.ts || echo "no console calls"
grep -RIn "api[_-]key\|apiKey" lib/factory/crawler-client.ts | grep -v "import\|: string\|//\|/\*\|param" || echo "no leaked api key references in logs"

Expected: both lines emit "no ..." sentinel (or only show legitimate parameter declarations).

  • [ ] Step 11.4: Mark Spec 06 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 6 (Crawler Client + Index Manager) | ⏸ | TS port + idempotent index lifecycle |
+ | Cluster 6 (Crawler Client + Index Manager) | ✅ | TS port + idempotent index lifecycle |

Acceptance criteria: Spec 06 done means:

  1. ✅ lib/factory/crawler-client.ts exports CrawlerClient class with: createCrawler, patchConfig, crawlUrls, startReindex, getStatus, getStats.
  2. ✅ Auth header is Basic base64(userId:apiKey): built once at construction, never rebuilt per request.
  3. ✅ Base URL is https://crawler.algolia.com/api/1 (constant, not env-driven).
  4. ✅ All crawler API responses pass through zod parse before being returned to callers.
  5. ✅ 4xx/5xx HTTP errors throw with sanitized message: error message MUST NOT contain the api key, the userId, or the full Authorization header.
  6. ✅ lib/factory/index-manager.ts exports IndexManager class with: ensureIndex(contentDomain), ensureFactoryMetaIndices().
  7. ✅ ensureIndex is idempotent: cold start → create + neuralPending; existing+matching → no-op; existing+drift → update.
  8. ✅ Drift comparison: order-sensitive on searchableAttributes/customRanking; set-equality on attributesForFaceting.
  9. ✅ Cold-start neuralSearch is NOT enabled; neuralPending: true is returned and logged.
  10. ✅ ensureFactoryMetaIndices configures algoliacentral_factory_sessions (filterOnly(record_type), filterOnly(parent_id)) and algoliacentral_factory_blueprints (filterOnly(content_domain), filterOnly(status)).
  11. ✅ All new tests pass (~16+ tests across crawler-client + index-manager).
  12. ✅ Full project test suite still GREEN.
  13. ✅ tsc --noEmit clean.
  14. ✅ Per CodingSOPs: every module has logger, every public method has docstring, every fallible call has try/catch, every boundary uses zod.

Out of scope (handled by other specs)

  • /api/factory/create-crawler and the rest of the factory endpoints → Spec 08 (consumes both classes built here).
  • /api/factory/test-real (test crawler reuse + sample-URL crawls) → Spec 09 (consumes CrawlerClient).
  • /api/factory/crawl-progress SSE endpoint that polls getStats → Spec 11 (frontend) wires Spec 09's polling endpoint.
  • neuralSearch promotion (flipping neuralPending: true once Insights events accrue) → Spec 13 (smoke + post-deploy promotion job).
  • Tenant-prefixed indices ({tenant}_marketing instead of algoliacentral_marketing) for multi-brand customers per §16k Decision 3: v1 sticks with the canonical DSS-derived names; tenant prefixing is a v2 enhancement deliberately deferred (§6 v2).
  • Real Algolia round-trip integration tests for both classes → Spec 13 (gated by env vars).
  • Reindex-in-progress safety (patch_config_safe from the Python reference) → Spec 09 will layer this on top of getStatus({withConfig: true}) + patchConfig. The TS CrawlerClient intentionally exposes the primitive only.