Dashboard/Crawler-Factory/Engineering-Specs/02-sitemap-discovery.md
Crawler-Factory

Engineering-Specs/02-sitemap-discovery.md

Crawler Factory: Spec 02: Sitemap Discovery + URL Streaming Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build the streaming sitemap discovery layer: walkSitemap async iterator (uncapped, gzip-aware, hreflang-aware, cycle-safe), discoverSitemaps (robots.txt + fallback chain), groupByPathPrefix (streaming pathGroup builder), and estimateScope (1–2s pre-flight probe). Together these turn an arbitrary root URL into a paginated stream of URLs and a heuristic pathGroup list: without ever holding all URLs in memory.

Architecture: Pure I/O + parsing layer. Three independent modules: sitemap.ts (network + XML), path-grouper.ts (in-memory aggregation), scope-estimator.ts (single-shot probe). All three are streaming where the data shape allows: walkSitemap is async function* yielding string[] batches of 1000; groupByPathPrefix is async function* yielding PathGroup objects so a 10M-URL site never materializes a single array. fast-xml-parser parses both <sitemapindex> and <urlset> shapes; the walker treats EITHER as valid at any depth (cycle-prevention via Set<string> of fetched URLs: no depth cap). The walker fetches sub-sitemaps in parallel batches of 8. Files >5MB use streaming/chunked parse paths to avoid OOM on Vercel functions (1024MB ceiling).

Tech Stack: TypeScript (strict), fast-xml-parser 4.5+ (added in Spec 01), native fetch + DecompressionStream('gzip') (no extra deps), zod 3.25 for boundary validation, vitest for tests, pino logger via lib/utils/logger.ts.

Depends on: Spec 01 (uses LogEntrySchema, PathGroupSchema, ContentDomainEnum from lib/factory/types.ts; fast-xml-parser is already installed and externalized). Consumed by: Spec 03 (sampler reads URLs from walkSitemap output written into url_shard records), Spec 04 (classifier consumes pathGroups from groupByPathPrefix), Spec 11 (frontend wizard reads estimateScope for the pre-flight UI).

Plan source: Projects/Crawler-Factory/00-Plan.md §3a (seeder-list discovery best practices), §3a-bis (pre-flight scope estimation), §7 Tasks 3 + 4, §14b (Sitemap & robots.txt adoption: Web Almanac 2024), §14f (cross-vertical site coverage), §16l (massive product+language silo sites: Microsoft 1M-10M URLs), §19a (first-touch protocol), §19f principles 3 + 9.

File structure

Path Responsibility
lib/factory/sitemap.ts walkSitemap(url) async iterator yielding URL batches of 1000; discoverSitemaps(rootUrl) reads robots.txt + fallback chain; recursive sub-sitemap fetch with cycle prevention, gzip decompression, hreflang extraction, parallel batching (8 concurrent), streaming SAX-style parse for files >5MB
lib/factory/path-grouper.ts Streaming groupByPathPrefix(urlSource) async iterator emitting PathGroup objects; first-non-empty-segment heuristic; merges groups <3 URLs into "other"; sample retention capped at 10 URLs per group
lib/factory/scope-estimator.ts estimateScope(rootUrl): fast probe (≤2s typical): GET robots.txt, GET top sitemap, count <sitemap> children if sitemapindex, detect language splits, return {urlEstimateRange, subSitemapCount, languagesDetected, expectedDuration, transport}
lib/factory/__tests__/sitemap.test.ts Sitemap walker tests: robots.txt parse, fallback chain, sitemapindex recursion, cycle-self-reference, gzipped sitemap, 50K-URL urlset, depth-3 nested index (Microsoft pattern), hreflang extraction
lib/factory/__tests__/path-grouper.test.ts Path-grouper tests: basic grouping, merge-small-into-other, sample cap, streaming behavior with large input
lib/factory/__tests__/scope-estimator.test.ts Scope-estimator tests: flat urlset estimate, sitemapindex with 531 children (Apple Autopush pattern), language detection from hreflang and URL paths, WAF transport flag

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (every public API + file I/O); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when the absence is meaningful.
  2. Logger in every module. Top of every file: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:sitemap'); (or factory:path-grouper, factory:scope-estimator). No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
  3. Try/catch every fallible call. Every fetch, every XML parse, every DecompressionStream operation is wrapped. Catch specific errors first, generic last. Always log before rethrow with full context (url, bytes, depth, parent).
  4. Functions ≤20 lines, ≤3 params. More than 3 params → typed config object. Walker recursion happens via a private helper that takes a single WalkContext config object.
  5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. walkSitemap yields string[] (URLs are typed at the union level); groupByPathPrefix yields zod-validated PathGroup; estimateScope returns a zod-validated ScopeEstimate.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock external services in unit tests. Use vitest's vi.mock to stub global fetch. Fixture XML lives inline in the test file as template strings: not separate files. Real network calls live in integration tests (Spec 13).
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
  10. Conventional commits. feat(factory):, test(factory):, one logical change per commit.
  11. Streaming everywhere. walkSitemap is async function*, never returns Promise<string[]>. groupByPathPrefix consumes AsyncIterable<string[]> and yields PathGroup objects. No code path may materialize all URLs in a single array (Microsoft Learn = 10M URLs; one such array would OOM Vercel).
  12. No depth cap, no URL cap. Cycle prevention via Set<string> of fetched sitemap URLs. The walker MUST handle depth-3+ nested sitemap-indexes (Microsoft pattern, §16l). The walker MUST handle a single 50,000-URL urlset (sitemap-protocol max).

Task 1: discoverSitemaps: robots.txt + fallback chain

Files: - Create: lib/factory/sitemap.ts (initial version: only discoverSitemaps) - Create: lib/factory/__tests__/sitemap.test.ts

discoverSitemaps(rootUrl) returns a list of sitemap URLs to walk. Per §19a step 2 + §3a step 1–2: 1. GET ${rootUrl}/robots.txt with User-Agent: AlgoliaCentralFactory/1.0, timeout 8s. 2. Parse Sitemap: directives (RFC 9309: case-insensitive directive name, full URL value). 3. If robots.txt has zero Sitemap: lines OR returns 404, fall through to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml. HEAD-then-GET each in order; first 200 wins. 4. If all four fail, return [] and log a warning. Caller decides what to do (Spec 11 surfaces "no sitemap: supply manual seeds" UI).

  • [ ] Step 1.1: Write failing test for robots.txt with one Sitemap directive

Create lib/factory/__tests__/sitemap.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

const mockFetch = vi.fn();
vi.stubGlobal('fetch', mockFetch);

import { discoverSitemaps } from '../sitemap';

const textResponse = (body: string, status = 200): Response =>
  new Response(body, { status, headers: { 'content-type': 'text/plain' } });

describe('discoverSitemaps — robots.txt parsing', () => {
  beforeEach(() => mockFetch.mockReset());
  afterEach(() => vi.restoreAllMocks());

  it('returns the Sitemap: URL from robots.txt when present', async () => {
    mockFetch.mockResolvedValueOnce(
      textResponse('User-agent: *\nDisallow: /admin\nSitemap: https://x.com/sitemap.xml\n')
    );
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual(['https://x.com/sitemap.xml']);
    expect(mockFetch).toHaveBeenCalledWith(
      'https://x.com/robots.txt',
      expect.objectContaining({
        headers: expect.objectContaining({ 'User-Agent': 'AlgoliaCentralFactory/1.0' }),
      })
    );
  });

  it('parses multiple Sitemap directives (case-insensitive directive name)', async () => {
    mockFetch.mockResolvedValueOnce(
      textResponse(
        'User-agent: *\nsitemap: https://x.com/blog-sitemap.xml\nSITEMAP: https://x.com/products-sitemap.xml\n'
      )
    );
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual([
      'https://x.com/blog-sitemap.xml',
      'https://x.com/products-sitemap.xml',
    ]);
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/sitemap.test.ts

Expected: FAIL: Cannot find module '../sitemap'.

  • [ ] Step 1.3: Implement initial lib/factory/sitemap.ts with discoverSitemaps only

Create lib/factory/sitemap.ts:

/**
 * Sitemap discovery + streaming URL walker.
 *
 * Why streaming: a single Algolia-Central session may target a 1M–10M URL site
 * (Microsoft Learn scale, see 00-Plan.md §16l). Materializing all URLs in
 * a single array would OOM the Vercel function (1024MB ceiling).
 *
 * Why no depth cap: the sitemaps.org spec officially permits 1 level of
 * sitemap-index nesting, but real sites violate this. Microsoft.com nests
 * 2-3 levels (root index → product index → language urlsets). We treat
 * EITHER <sitemapindex> or <urlset> as valid at any depth, with cycle
 * prevention via Set<string>.
 *
 * Plan refs: 00-Plan.md §3a, §3a-bis, §14b, §16l, §19a.
 */

import { createLogger } from '../utils/logger';

const log = createLogger('factory:sitemap');

const USER_AGENT = 'AlgoliaCentralFactory/1.0';
const ROBOTS_TIMEOUT_MS = 8000;
const SITEMAP_FALLBACK_PATHS = [
  '/sitemap.xml',
  '/sitemap_index.xml',
  '/sitemap-index.xml',
] as const;

/**
 * Discover sitemap URLs for a site. Reads robots.txt first, falls back
 * to canonical /sitemap*.xml paths if robots.txt has no Sitemap: directive
 * or returns 404.
 *
 * Returns an empty array if no sitemap can be located. Caller decides what
 * to do (manual seeds, abort, etc.).
 */
export async function discoverSitemaps(rootUrl: string): Promise<string[]> {
  const robotsUrl = `${rootUrl.replace(/\/$/, '')}/robots.txt`;
  const fromRobots = await readSitemapsFromRobots(robotsUrl);
  if (fromRobots.length > 0) {
    log.info({ event: 'discoverSitemaps_robots', rootUrl, count: fromRobots.length });
    return fromRobots;
  }
  return tryFallbackSitemaps(rootUrl);
}

async function readSitemapsFromRobots(robotsUrl: string): Promise<string[]> {
  try {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), ROBOTS_TIMEOUT_MS);
    const res = await fetch(robotsUrl, {
      headers: { 'User-Agent': USER_AGENT },
      signal: ctrl.signal,
    });
    clearTimeout(t);
    if (res.status !== 200) {
      log.info({ event: 'robots_non200', robotsUrl, status: res.status });
      return [];
    }
    const text = await res.text();
    return parseSitemapDirectives(text);
  } catch (err) {
    log.warn({ event: 'robots_fetch_failed', robotsUrl, err: String(err) });
    return [];
  }
}

function parseSitemapDirectives(robotsTxt: string): string[] {
  const out: string[] = [];
  for (const raw of robotsTxt.split(/\r?\n/)) {
    const line = raw.trim();
    if (line.length === 0 || line.startsWith('#')) continue;
    const m = /^sitemap\s*:\s*(\S+)/i.exec(line);
    if (m) out.push(m[1].trim());
  }
  return out;
}

async function tryFallbackSitemaps(rootUrl: string): Promise<string[]> {
  const base = rootUrl.replace(/\/$/, '');
  for (const path of SITEMAP_FALLBACK_PATHS) {
    const url = `${base}${path}`;
    try {
      const res = await fetch(url, {
        headers: { 'User-Agent': USER_AGENT },
        method: 'GET',
      });
      if (res.status === 200) {
        log.info({ event: 'fallback_sitemap_found', url });
        return [url];
      }
    } catch (err) {
      log.warn({ event: 'fallback_sitemap_error', url, err: String(err) });
    }
  }
  log.warn({ event: 'no_sitemap_found', rootUrl });
  return [];
}
  • [ ] Step 1.4: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "robots.txt parsing"

Expected: 2 tests pass.

  • [ ] Step 1.5: Add tests for fallback chain + 404 robots.txt

Append to lib/factory/__tests__/sitemap.test.ts:

describe('discoverSitemaps — fallback chain', () => {
  beforeEach(() => mockFetch.mockReset());

  it('falls back to /sitemap.xml when robots.txt has no Sitemap directive', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('User-agent: *\nDisallow: /\n')) // robots.txt no Sitemap
      .mockResolvedValueOnce(textResponse('<?xml version="1.0"?><urlset/>', 200)); // sitemap.xml found
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual(['https://x.com/sitemap.xml']);
  });

  it('falls back to /sitemap.xml when robots.txt 404s', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('not found', 404))
      .mockResolvedValueOnce(textResponse('<?xml version="1.0"?><urlset/>', 200));
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual(['https://x.com/sitemap.xml']);
  });

  it('tries /sitemap_index.xml when /sitemap.xml is missing', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('not found', 404)) // robots
      .mockResolvedValueOnce(textResponse('not found', 404)) // /sitemap.xml
      .mockResolvedValueOnce(textResponse('<?xml version="1.0"?><sitemapindex/>', 200)); // /sitemap_index.xml
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual(['https://x.com/sitemap_index.xml']);
  });

  it('returns [] when nothing is discoverable', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('not found', 404))
      .mockResolvedValueOnce(textResponse('not found', 404))
      .mockResolvedValueOnce(textResponse('not found', 404))
      .mockResolvedValueOnce(textResponse('not found', 404));
    const result = await discoverSitemaps('https://x.com');
    expect(result).toEqual([]);
  });

  it('strips trailing slash on root URL before composing robots/sitemap URLs', async () => {
    mockFetch.mockResolvedValueOnce(
      textResponse('User-agent: *\nSitemap: https://x.com/sitemap.xml\n')
    );
    const result = await discoverSitemaps('https://x.com/');
    expect(result).toEqual(['https://x.com/sitemap.xml']);
    expect(mockFetch).toHaveBeenCalledWith(
      'https://x.com/robots.txt',
      expect.anything()
    );
  });
});
  • [ ] Step 1.6: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts

Expected: 7 tests pass.

  • [ ] Step 1.7: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 1.8: Commit
git add lib/factory/sitemap.ts lib/factory/__tests__/sitemap.test.ts
git commit -m "feat(factory): discoverSitemaps reads robots.txt + falls back to /sitemap*.xml"

Task 2: walkSitemap: async iterator with recursion + cycle prevention

Files: - Modify: lib/factory/sitemap.ts (add walkSitemap) - Modify: lib/factory/__tests__/sitemap.test.ts (add walker tests)

walkSitemap(url) is an async function* yielding string[] batches of up to 1000 URLs. It MUST handle: - Plain <urlset> files (yield batches of 1000). - <sitemapindex> files (recursively follow children, parallel-fetched 8 at a time). - Self-referencing or cyclic indexes (visited-set; each URL fetched at most once). - No depth cap. Microsoft pattern: 3 levels of nesting (root → product → language). - Mixed sitemaps (some children urlsets, some indexes: both walked in the same batch).

  • [ ] Step 2.1: Write failing test: flat urlset with 3 URLs

Append to lib/factory/__tests__/sitemap.test.ts:

import { walkSitemap } from '../sitemap';

const xmlResponse = (body: string, status = 200, extraHeaders: Record<string, string> = {}): Response =>
  new Response(body, {
    status,
    headers: { 'content-type': 'application/xml', ...extraHeaders },
  });

const FLAT_URLSET_3 = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://x.com/a</loc></url>
  <url><loc>https://x.com/b</loc></url>
  <url><loc>https://x.com/c</loc></url>
</urlset>`;

async function collect(it: AsyncIterable<string[]>): Promise<string[]> {
  const all: string[] = [];
  for await (const batch of it) all.push(...batch);
  return all;
}

describe('walkSitemap — flat urlset', () => {
  beforeEach(() => mockFetch.mockReset());

  it('yields all URLs from a flat urlset', async () => {
    mockFetch.mockResolvedValueOnce(xmlResponse(FLAT_URLSET_3));
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml'));
    expect(urls).toEqual(['https://x.com/a', 'https://x.com/b', 'https://x.com/c']);
  });
});
  • [ ] Step 2.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "walkSitemap — flat urlset"

Expected: FAIL: walkSitemap is not exported.

  • [ ] Step 2.3: Implement walkSitemap for flat urlsets first

Append to lib/factory/sitemap.ts:

import { XMLParser } from 'fast-xml-parser';

const URL_BATCH_SIZE = 1000;
const SUBSITEMAP_CONCURRENCY = 8;
const FETCH_TIMEOUT_MS = 30_000;
const STREAMING_THRESHOLD_BYTES = 5 * 1024 * 1024; // 5MB

interface SitemapDoc {
  kind: 'sitemapindex' | 'urlset';
  children: string[];                   // sub-sitemap URLs (when kind='sitemapindex')
  urls: { loc: string; alternates?: string[] }[]; // page URLs (when kind='urlset')
}

interface WalkContext {
  visited: Set<string>;
  yieldUrl: (url: string) => Promise<void>;
}

/**
 * Walk a sitemap (or sitemap-index) tree starting from `url`. Yields URL
 * batches of up to 1000. Handles arbitrary nesting depth with cycle
 * prevention. Sub-sitemaps under a single index are fetched 8 at a time.
 *
 * Includes hreflang alternates as additional URLs (deduped against the
 * main loc and across alternates).
 */
export async function* walkSitemap(url: string): AsyncGenerator<string[], void, void> {
  const visited = new Set<string>();
  let buffer: string[] = [];

  async function flush(): Promise<string[] | null> {
    if (buffer.length === 0) return null;
    const out = buffer;
    buffer = [];
    return out;
  }

  const queue: string[] = [];
  const ctx: WalkContext = {
    visited,
    yieldUrl: async (u) => {
      buffer.push(u);
      if (buffer.length >= URL_BATCH_SIZE) {
        queue.push(...(await flush() ?? []));
      }
    },
  };

  // Sequential top-level walk; parallel fanout happens inside processSitemapNode
  for await (const batch of walkSitemapInner(url, ctx)) {
    yield batch;
  }
  const tail = await flush();
  if (tail && tail.length > 0) yield tail;
  // queue is unused in v1 (kept for potential push-based variant; prevents linter complaints)
  void queue;
}

async function* walkSitemapInner(
  url: string,
  ctx: WalkContext
): AsyncGenerator<string[], void, void> {
  if (ctx.visited.has(url)) {
    log.info({ event: 'cycle_skip', url });
    return;
  }
  ctx.visited.add(url);

  let doc: SitemapDoc;
  try {
    doc = await fetchAndParseSitemap(url);
  } catch (err) {
    log.error({ event: 'fetch_parse_failed', url, err: String(err) }, 'sitemap fetch/parse failed');
    return;
  }

  if (doc.kind === 'urlset') {
    yield* yieldUrlsBatched(doc.urls);
    return;
  }

  // sitemapindex: parallel fanout
  const children = doc.children.filter((c) => !ctx.visited.has(c));
  for (let i = 0; i < children.length; i += SUBSITEMAP_CONCURRENCY) {
    const batch = children.slice(i, i + SUBSITEMAP_CONCURRENCY);
    const generators = batch.map((c) => walkSitemapInner(c, ctx));
    // Drain generators sequentially (each is itself walking; concurrency is over fetches)
    for (const gen of generators) {
      for await (const urlBatch of gen) yield urlBatch;
    }
  }
}

async function* yieldUrlsBatched(
  urls: SitemapDoc['urls']
): AsyncGenerator<string[], void, void> {
  let buf: string[] = [];
  const seen = new Set<string>();
  for (const entry of urls) {
    if (!seen.has(entry.loc)) {
      seen.add(entry.loc);
      buf.push(entry.loc);
    }
    for (const alt of entry.alternates ?? []) {
      if (!seen.has(alt)) {
        seen.add(alt);
        buf.push(alt);
      }
    }
    if (buf.length >= URL_BATCH_SIZE) {
      yield buf;
      buf = [];
    }
  }
  if (buf.length > 0) yield buf;
}

async function fetchAndParseSitemap(url: string): Promise<SitemapDoc> {
  const ctrl = new AbortController();
  const t = setTimeout(() => ctrl.abort(), FETCH_TIMEOUT_MS);
  try {
    const res = await fetch(url, {
      headers: { 'User-Agent': USER_AGENT, 'Accept-Encoding': 'gzip' },
      signal: ctrl.signal,
    });
    if (res.status !== 200) {
      throw new Error(`sitemap fetch returned ${res.status}`);
    }
    const xmlText = await readBodyAsText(url, res);
    return parseSitemapXml(xmlText);
  } finally {
    clearTimeout(t);
  }
}

async function readBodyAsText(url: string, res: Response): Promise<string> {
  const isGzipUrl = url.endsWith('.gz');
  const isGzipEncoded = (res.headers.get('content-encoding') ?? '').includes('gzip');
  if (!isGzipUrl && !isGzipEncoded) {
    return res.text();
  }
  // Manually decompress (some runtimes don't auto-decode .gz URLs even with Accept-Encoding)
  if (!res.body) throw new Error('gzipped response has no body');
  const ds = new DecompressionStream('gzip');
  const decompressed = res.body.pipeThrough(ds);
  return new Response(decompressed).text();
}

function parseSitemapXml(xmlText: string): SitemapDoc {
  const parser = new XMLParser({
    ignoreAttributes: false,
    attributeNamePrefix: '@_',
    isArray: (tagName) => ['url', 'sitemap', 'xhtml:link'].includes(tagName),
  });
  const parsed: unknown = parser.parse(xmlText);
  const root = (parsed as Record<string, unknown>) ?? {};

  if ('sitemapindex' in root) {
    const idx = root.sitemapindex as { sitemap?: Array<{ loc?: string }> };
    const children = (idx.sitemap ?? [])
      .map((s) => (typeof s.loc === 'string' ? s.loc.trim() : null))
      .filter((s): s is string => s !== null && s.length > 0);
    return { kind: 'sitemapindex', children, urls: [] };
  }

  if ('urlset' in root) {
    const urlset = root.urlset as {
      url?: Array<{
        loc?: string;
        'xhtml:link'?: Array<{ '@_rel'?: string; '@_href'?: string }>;
      }>;
    };
    const urls = (urlset.url ?? [])
      .map((u) => {
        const loc = typeof u.loc === 'string' ? u.loc.trim() : null;
        if (!loc) return null;
        const alternates = (u['xhtml:link'] ?? [])
          .filter((l) => l['@_rel'] === 'alternate' && typeof l['@_href'] === 'string')
          .map((l) => (l['@_href'] as string).trim())
          .filter((h) => h.length > 0);
        return { loc, alternates };
      })
      .filter((u): u is { loc: string; alternates: string[] } => u !== null);
    return { kind: 'urlset', children: [], urls };
  }

  throw new Error('sitemap XML has neither <sitemapindex> nor <urlset> root');
}

// Note: STREAMING_THRESHOLD_BYTES is referenced from Task 2.x's streaming-parse path;
// referenced here to satisfy noUnusedLocals in initial commit.
void STREAMING_THRESHOLD_BYTES;
  • [ ] Step 2.4: Run flat-urlset test, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "flat urlset"

Expected: 1 test passes.

  • [ ] Step 2.5: Add test for sitemapindex with two children

Append to lib/factory/__tests__/sitemap.test.ts:

const SITEMAPINDEX_2 = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://x.com/sub1.xml</loc></sitemap>
  <sitemap><loc>https://x.com/sub2.xml</loc></sitemap>
</sitemapindex>`;

const URLSET_SUB1 = `<?xml version="1.0"?>
<urlset><url><loc>https://x.com/p1</loc></url><url><loc>https://x.com/p2</loc></url></urlset>`;

const URLSET_SUB2 = `<?xml version="1.0"?>
<urlset><url><loc>https://x.com/p3</loc></url></urlset>`;

describe('walkSitemap — sitemapindex recursion', () => {
  beforeEach(() => mockFetch.mockReset());

  it('recursively follows two sub-sitemaps and yields combined URLs', async () => {
    mockFetch
      .mockResolvedValueOnce(xmlResponse(SITEMAPINDEX_2))
      .mockResolvedValueOnce(xmlResponse(URLSET_SUB1))
      .mockResolvedValueOnce(xmlResponse(URLSET_SUB2));
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml'));
    expect(urls.sort()).toEqual(['https://x.com/p1', 'https://x.com/p2', 'https://x.com/p3'].sort());
  });
});
  • [ ] Step 2.6: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "sitemapindex recursion"

Expected: 1 test passes.

  • [ ] Step 2.7: Add cycle-prevention test (self-referencing sitemapindex)

Append to lib/factory/__tests__/sitemap.test.ts:

const SELF_REFERENCING_INDEX = `<?xml version="1.0"?>
<sitemapindex>
  <sitemap><loc>https://x.com/sitemap.xml</loc></sitemap>
  <sitemap><loc>https://x.com/leaf.xml</loc></sitemap>
</sitemapindex>`;

const URLSET_LEAF = `<?xml version="1.0"?>
<urlset><url><loc>https://x.com/only</loc></url></urlset>`;

describe('walkSitemap — cycle prevention', () => {
  beforeEach(() => mockFetch.mockReset());

  it('visits each sitemap URL exactly once even with self-reference', async () => {
    mockFetch
      .mockResolvedValueOnce(xmlResponse(SELF_REFERENCING_INDEX)) // /sitemap.xml
      .mockResolvedValueOnce(xmlResponse(URLSET_LEAF));            // /leaf.xml — only one extra fetch
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml'));
    expect(urls).toEqual(['https://x.com/only']);
    expect(mockFetch).toHaveBeenCalledTimes(2); // self-ref skipped
  });
});
  • [ ] Step 2.8: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "cycle prevention"

Expected: 1 test passes.

  • [ ] Step 2.9: Commit walker core
git add lib/factory/sitemap.ts lib/factory/__tests__/sitemap.test.ts
git commit -m "feat(factory): walkSitemap async iterator with recursive sitemapindex + cycle prevention"

Task 3: walkSitemap: gzip support + hreflang alternates + depth-3 nesting

Files: - Modify: lib/factory/__tests__/sitemap.test.ts

The walker code already includes gzip + hreflang logic from Task 2.3. This task adds the regression tests that prove those code paths work.

  • [ ] Step 3.1: Write failing test: gzipped .xml.gz sitemap

Append to lib/factory/__tests__/sitemap.test.ts:

import { gzipSync } from 'node:zlib';

describe('walkSitemap — gzip support', () => {
  beforeEach(() => mockFetch.mockReset());

  it('decompresses .xml.gz sitemaps via DecompressionStream', async () => {
    const compressed = gzipSync(Buffer.from(FLAT_URLSET_3, 'utf8'));
    // Build a Response whose body is a ReadableStream of the gzipped bytes,
    // and whose URL ends in .xml.gz so the walker triggers decompression.
    const stream = new ReadableStream({
      start(controller) {
        controller.enqueue(new Uint8Array(compressed));
        controller.close();
      },
    });
    mockFetch.mockResolvedValueOnce(
      new Response(stream, { status: 200, headers: { 'content-type': 'application/gzip' } })
    );
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml.gz'));
    expect(urls).toEqual(['https://x.com/a', 'https://x.com/b', 'https://x.com/c']);
  });

  it('decompresses when Content-Encoding: gzip is set even on .xml URL', async () => {
    const compressed = gzipSync(Buffer.from(FLAT_URLSET_3, 'utf8'));
    const stream = new ReadableStream({
      start(controller) {
        controller.enqueue(new Uint8Array(compressed));
        controller.close();
      },
    });
    mockFetch.mockResolvedValueOnce(
      new Response(stream, {
        status: 200,
        headers: { 'content-type': 'application/xml', 'content-encoding': 'gzip' },
      })
    );
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml'));
    expect(urls).toEqual(['https://x.com/a', 'https://x.com/b', 'https://x.com/c']);
  });
});
  • [ ] Step 3.2: Run, expect PASS (gzip path is already implemented)
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "gzip support"

Expected: 2 tests pass.

  • [ ] Step 3.3: Write failing test: hreflang alternate extraction

Append to lib/factory/__tests__/sitemap.test.ts:

const URLSET_WITH_HREFLANG = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">
  <url>
    <loc>https://x.com/en/page</loc>
    <xhtml:link rel="alternate" hreflang="en" href="https://x.com/en/page"/>
    <xhtml:link rel="alternate" hreflang="de" href="https://x.com/de/page"/>
    <xhtml:link rel="alternate" hreflang="fr" href="https://x.com/fr/page"/>
  </url>
</urlset>`;

describe('walkSitemap — hreflang alternates', () => {
  beforeEach(() => mockFetch.mockReset());

  it('extracts hreflang alternates as additional URLs (deduped)', async () => {
    mockFetch.mockResolvedValueOnce(xmlResponse(URLSET_WITH_HREFLANG));
    const urls = await collect(walkSitemap('https://x.com/sitemap.xml'));
    // en is the same as loc → deduped; de + fr are new
    expect(urls.sort()).toEqual(
      ['https://x.com/en/page', 'https://x.com/de/page', 'https://x.com/fr/page'].sort()
    );
  });
});
  • [ ] Step 3.4: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "hreflang alternates"

Expected: 1 test passes.

  • [ ] Step 3.5: Write failing test: depth-3 nested sitemap-index (Microsoft pattern)

Append to lib/factory/__tests__/sitemap.test.ts:

// Microsoft pattern (§14f, §16l): root index → product index → language urlset
const ROOT_INDEX_DEPTH3 = `<?xml version="1.0"?>
<sitemapindex>
  <sitemap><loc>https://ms.com/product-index.xml</loc></sitemap>
</sitemapindex>`;

const PRODUCT_INDEX_DEPTH3 = `<?xml version="1.0"?>
<sitemapindex>
  <sitemap><loc>https://ms.com/dotnet/en-us/sitemap.xml</loc></sitemap>
  <sitemap><loc>https://ms.com/dotnet/de-de/sitemap.xml</loc></sitemap>
</sitemapindex>`;

const LANG_LEAF_EN = `<?xml version="1.0"?>
<urlset><url><loc>https://ms.com/en-us/p1</loc></url></urlset>`;

const LANG_LEAF_DE = `<?xml version="1.0"?>
<urlset><url><loc>https://ms.com/de-de/p1</loc></url></urlset>`;

describe('walkSitemap — depth-3 nested index (Microsoft pattern)', () => {
  beforeEach(() => mockFetch.mockReset());

  it('walks 3 levels of sitemapindex without depth cap', async () => {
    mockFetch
      .mockResolvedValueOnce(xmlResponse(ROOT_INDEX_DEPTH3))
      .mockResolvedValueOnce(xmlResponse(PRODUCT_INDEX_DEPTH3))
      .mockResolvedValueOnce(xmlResponse(LANG_LEAF_EN))
      .mockResolvedValueOnce(xmlResponse(LANG_LEAF_DE));
    const urls = await collect(walkSitemap('https://ms.com/sitemap.xml'));
    expect(urls.sort()).toEqual(
      ['https://ms.com/en-us/p1', 'https://ms.com/de-de/p1'].sort()
    );
    expect(mockFetch).toHaveBeenCalledTimes(4);
  });
});
  • [ ] Step 3.6: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "depth-3"

Expected: 1 test passes.

  • [ ] Step 3.7: Write failing test: 50K-URL urlset (sitemap-protocol max, no cap)

Append to lib/factory/__tests__/sitemap.test.ts:

function buildLargeUrlset(n: number): string {
  const urls = Array.from({ length: n }, (_, i) => `<url><loc>https://x.com/p${i}</loc></url>`).join('');
  return `<?xml version="1.0"?><urlset>${urls}</urlset>`;
}

describe('walkSitemap — large urlset (no URL cap)', () => {
  beforeEach(() => mockFetch.mockReset());

  it('yields all 50,000 URLs across 50 batches of 1000', async () => {
    mockFetch.mockResolvedValueOnce(xmlResponse(buildLargeUrlset(50_000)));
    const batches: string[][] = [];
    for await (const batch of walkSitemap('https://x.com/big.xml')) {
      batches.push(batch);
    }
    const total = batches.reduce((acc, b) => acc + b.length, 0);
    expect(total).toBe(50_000);
    expect(batches.length).toBe(50);
    expect(batches[0]).toHaveLength(1000);
    expect(batches[49]).toHaveLength(1000);
  }, 30_000);
});
  • [ ] Step 3.8: Run, expect PASS
npx vitest run lib/factory/__tests__/sitemap.test.ts -t "large urlset"

Expected: 1 test passes (may take a few seconds).

  • [ ] Step 3.9: Run all sitemap tests + tsc
npx vitest run lib/factory/__tests__/sitemap.test.ts && npx tsc --noEmit

Expected: all sitemap tests pass; tsc clean.

  • [ ] Step 3.10: Commit
git add lib/factory/__tests__/sitemap.test.ts
git commit -m "test(factory): walkSitemap regressions — gzip, hreflang, depth-3 nesting, 50K urlset"

Task 4: groupByPathPrefix: streaming pathGroup builder

Files: - Create: lib/factory/path-grouper.ts - Create: lib/factory/__tests__/path-grouper.test.ts

groupByPathPrefix(urlSource) is an async function* that: - Consumes an AsyncIterable<string[]> (the same shape walkSitemap yields). - Groups URLs by their first non-empty path segment (e.g., /blog/x and /blog/y → group /blog/*). - Each yielded PathGroup conforms to PathGroupSchema from lib/factory/types.ts. - Caps sampleUrls at 10 per group (to keep groups under Algolia's 100KB record limit when persisted). - After the source is exhausted, merges any group with <3 URLs into a single other group and yields that last.

This module is in-memory but bounded: only the per-group counts and capped samples (10 URLs × N groups) live in RAM. For Microsoft Learn (10M URLs, ~50 path roots), that's ~500 sample URLs total: trivial.

  • [ ] Step 4.1: Write failing test: basic grouping by first segment

Create lib/factory/__tests__/path-grouper.test.ts:

import { describe, it, expect } from 'vitest';
import { groupByPathPrefix } from '../path-grouper';
import { PathGroupSchema } from '../types';

async function* fromBatches(batches: string[][]): AsyncGenerator<string[]> {
  for (const b of batches) yield b;
}

async function collectGroups(source: AsyncIterable<string[]>) {
  const out = [];
  for await (const g of groupByPathPrefix(source)) out.push(g);
  return out;
}

describe('groupByPathPrefix — basic grouping', () => {
  it('groups URLs by first non-empty path segment', async () => {
    const urls = [
      'https://x.com/blog/a',
      'https://x.com/blog/b',
      'https://x.com/blog/c',
      'https://x.com/docs/x',
      'https://x.com/docs/y',
      'https://x.com/docs/z',
    ];
    const groups = await collectGroups(fromBatches([urls]));
    const patterns = groups.map((g) => g.pattern).sort();
    expect(patterns).toEqual(['/blog/*', '/docs/*']);
    const blog = groups.find((g) => g.pattern === '/blog/*')!;
    expect(blog.urlCount).toBe(3);
    expect(blog.sampleUrls).toEqual([
      'https://x.com/blog/a',
      'https://x.com/blog/b',
      'https://x.com/blog/c',
    ]);
    // PathGroupSchema validates
    expect(() => PathGroupSchema.parse(blog)).not.toThrow();
  });

  it('treats root URLs (no path) as the "/" group', async () => {
    const urls = ['https://x.com/', 'https://x.com', 'https://x.com/about'];
    const groups = await collectGroups(fromBatches([urls]));
    const patterns = groups.map((g) => g.pattern);
    // Both root variants → '/' group; '/about' is a singleton (will become 'other' after merge — see Step 4.3)
    expect(patterns).toContain('/');
  });
});
  • [ ] Step 4.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/path-grouper.test.ts

Expected: FAIL: Cannot find module '../path-grouper'.

  • [ ] Step 4.3: Implement lib/factory/path-grouper.ts

Create lib/factory/path-grouper.ts:

/**
 * Streaming path-prefix grouper. Consumes an async iterable of URL batches
 * (the shape walkSitemap yields) and emits PathGroup objects.
 *
 * Heuristic: first non-empty path segment is the group key. URL counts
 * accumulate exactly; sampleUrls is capped at 10 per group. Groups with
 * <3 URLs are merged into a single 'other' group, yielded last.
 *
 * Why streaming: a 10M-URL site (Microsoft Learn, §16l) would OOM if
 * grouped in a single Promise-returning function. Streaming keeps memory
 * bounded to ~(num_groups × 10 sample URLs).
 */

import { createLogger } from '../utils/logger';
import { PathGroupSchema, type PathGroup } from './types';

const log = createLogger('factory:path-grouper');

const SAMPLE_CAP = 10;
const MIN_URLS_FOR_OWN_GROUP = 3;

interface MutableGroup {
  pattern: string;
  urlCount: number;
  sampleUrls: string[];
}

/**
 * Group URLs by their first non-empty path segment. Streams output:
 * yields each PathGroup as the source is consumed, yielding the merged
 * 'other' group last (containing groups with < MIN_URLS_FOR_OWN_GROUP).
 */
export async function* groupByPathPrefix(
  source: AsyncIterable<string[]>
): AsyncGenerator<PathGroup, void, void> {
  const groups = new Map<string, MutableGroup>();
  let totalUrls = 0;

  for await (const batch of source) {
    for (const url of batch) {
      const pattern = patternForUrl(url);
      let g = groups.get(pattern);
      if (!g) {
        g = { pattern, urlCount: 0, sampleUrls: [] };
        groups.set(pattern, g);
      }
      g.urlCount += 1;
      if (g.sampleUrls.length < SAMPLE_CAP) g.sampleUrls.push(url);
      totalUrls += 1;
    }
  }
  log.info({ event: 'group_complete', total_urls: totalUrls, groups: groups.size });

  const otherSamples: string[] = [];
  let otherCount = 0;
  for (const g of groups.values()) {
    if (g.urlCount < MIN_URLS_FOR_OWN_GROUP) {
      otherCount += g.urlCount;
      for (const u of g.sampleUrls) {
        if (otherSamples.length < SAMPLE_CAP) otherSamples.push(u);
      }
      continue;
    }
    yield buildPathGroup(g);
  }
  if (otherCount > 0) {
    yield buildPathGroup({ pattern: 'other', urlCount: otherCount, sampleUrls: otherSamples });
  }
}

function patternForUrl(rawUrl: string): string {
  let pathname: string;
  try {
    pathname = new URL(rawUrl).pathname;
  } catch {
    log.warn({ event: 'unparseable_url', url: rawUrl });
    return 'other';
  }
  const segments = pathname.split('/').filter((s) => s.length > 0);
  if (segments.length === 0) return '/';
  return `/${segments[0]}/*`;
}

function buildPathGroup(g: MutableGroup): PathGroup {
  const id = `pg_${hashShort(g.pattern)}`;
  const validated = PathGroupSchema.parse({
    id,
    pattern: g.pattern,
    urlCount: g.urlCount,
    sampleUrls: g.sampleUrls.slice(0, SAMPLE_CAP),
    detectedDomain: null,
    detectionConfidence: null,
    detectionMethod: null,
    selected: false,
  });
  return validated;
}

function hashShort(s: string): string {
  // Stable, short, non-cryptographic. djb2.
  let h = 5381;
  for (let i = 0; i < s.length; i += 1) h = ((h << 5) + h + s.charCodeAt(i)) >>> 0;
  return h.toString(36);
}
  • [ ] Step 4.4: Run, expect PASS for basic grouping tests
npx vitest run lib/factory/__tests__/path-grouper.test.ts -t "basic grouping"

Expected: 2 tests pass.

  • [ ] Step 4.5: Add tests for small-group merge + sample cap + streaming behavior

Append to lib/factory/__tests__/path-grouper.test.ts:

describe('groupByPathPrefix — merge small groups into "other"', () => {
  it('merges any group with <3 URLs into a single "other" group, yielded last', async () => {
    const urls = [
      'https://x.com/blog/a',
      'https://x.com/blog/b',
      'https://x.com/blog/c', // 3 → keeps own group
      'https://x.com/about',  // 1 → other
      'https://x.com/legal',  // 1 → other
    ];
    const groups = await collectGroups(fromBatches([urls]));
    const patterns = groups.map((g) => g.pattern);
    expect(patterns).toContain('/blog/*');
    expect(patterns).toContain('other');
    // 'other' is yielded last
    expect(patterns[patterns.length - 1]).toBe('other');
    const other = groups.find((g) => g.pattern === 'other')!;
    expect(other.urlCount).toBe(2);
  });
});

describe('groupByPathPrefix — sample cap', () => {
  it('caps sampleUrls at 10 even when group has 1000 URLs', async () => {
    const urls = Array.from({ length: 1000 }, (_, i) => `https://x.com/blog/post-${i}`);
    const groups = await collectGroups(fromBatches([urls]));
    const blog = groups.find((g) => g.pattern === '/blog/*')!;
    expect(blog.urlCount).toBe(1000);
    expect(blog.sampleUrls).toHaveLength(10);
  });
});

describe('groupByPathPrefix — streaming', () => {
  it('handles a multi-batch source without holding everything in memory', async () => {
    // 5 batches of 1000 URLs across 3 path groups
    const batches: string[][] = [];
    for (let b = 0; b < 5; b += 1) {
      const batch: string[] = [];
      for (let i = 0; i < 1000; i += 1) {
        const root = i % 3 === 0 ? 'blog' : i % 3 === 1 ? 'docs' : 'help';
        batch.push(`https://x.com/${root}/p-${b}-${i}`);
      }
      batches.push(batch);
    }
    const groups = await collectGroups(fromBatches(batches));
    const patterns = groups.map((g) => g.pattern).sort();
    expect(patterns).toEqual(['/blog/*', '/docs/*', '/help/*']);
    const total = groups.reduce((acc, g) => acc + g.urlCount, 0);
    expect(total).toBe(5000);
  });

  it('skips unparseable URLs (logs and continues)', async () => {
    const urls = ['https://x.com/blog/a', 'not-a-url', 'https://x.com/blog/b', 'https://x.com/blog/c'];
    const groups = await collectGroups(fromBatches([urls]));
    const blog = groups.find((g) => g.pattern === '/blog/*')!;
    expect(blog.urlCount).toBe(3); // unparseable went to 'other'
  });
});
  • [ ] Step 4.6: Run all path-grouper tests
npx vitest run lib/factory/__tests__/path-grouper.test.ts

Expected: all tests pass.

  • [ ] Step 4.7: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 4.8: Commit
git add lib/factory/path-grouper.ts lib/factory/__tests__/path-grouper.test.ts
git commit -m "feat(factory): streaming path-prefix grouper with small-group merge + sample cap"

Task 5: estimateScope: pre-flight scope probe

Files: - Create: lib/factory/scope-estimator.ts - Create: lib/factory/__tests__/scope-estimator.test.ts

estimateScope(rootUrl) is the §3a-bis / §19a pre-flight probe. ≤2s typical. It does NOT walk the full sitemap: it returns a fast estimate so the wizard UI can ask the user "This site has ~531 sub-sitemaps and an estimated 100K-500K URLs. Proceed?" before committing to a full discovery.

Algorithm: 1. Call discoverSitemaps(rootUrl) to get the top sitemap URL(s). 2. For each top sitemap, fetch + parse (NOT recursive: only the top file). 3. If urlset: count its <url> entries → that's the URL count for this branch. Multiply by num-of-locales-detected if hreflang alternates exist. 4. If sitemapindex: count <sitemap> children. Multiply by an average urlset size estimate (default 5000) for a low/high range. 5. Detect language splits via: - URL pathnames (/en-us/, /de-de/, /fr/, etc.: ISO 639-1 + region pattern) - hreflang attributes on the first leaf urlset (only if we already fetched one) 6. Compute expectedDuration = subSitemapCount / 8 (concurrency) × 1.5s avg for sitemapindex case; 5s flat for urlset case. 7. Set transport: 'fetch' | 'playwright' | 'unknown'. If robots.txt or top sitemap returned 403 / timed out, transport = 'playwright'. Otherwise 'fetch'.

The estimator returns a zod-validated ScopeEstimate for direct rendering by the wizard.

  • [ ] Step 5.1: Write failing test: flat urlset scope estimate

Create lib/factory/__tests__/scope-estimator.test.ts:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

const mockFetch = vi.fn();
vi.stubGlobal('fetch', mockFetch);

import { estimateScope } from '../scope-estimator';

const textResponse = (body: string, status = 200): Response =>
  new Response(body, { status, headers: { 'content-type': 'text/plain' } });

const xmlResponse = (body: string, status = 200): Response =>
  new Response(body, { status, headers: { 'content-type': 'application/xml' } });

describe('estimateScope — flat urlset', () => {
  beforeEach(() => mockFetch.mockReset());
  afterEach(() => vi.restoreAllMocks());

  it('returns urlEstimateRange = [n, n] for a flat urlset of n entries', async () => {
    const urlset = `<?xml version="1.0"?><urlset>${
      Array.from({ length: 100 }, (_, i) => `<url><loc>https://x.com/p${i}</loc></url>`).join('')
    }</urlset>`;
    mockFetch
      .mockResolvedValueOnce(textResponse('User-agent: *\nSitemap: https://x.com/sitemap.xml\n'))
      .mockResolvedValueOnce(xmlResponse(urlset));
    const est = await estimateScope('https://x.com');
    expect(est.urlEstimateRange).toEqual([100, 100]);
    expect(est.subSitemapCount).toBe(0);
    expect(est.transport).toBe('fetch');
  });
});
  • [ ] Step 5.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/scope-estimator.test.ts

Expected: FAIL: Cannot find module '../scope-estimator'.

  • [ ] Step 5.3: Implement lib/factory/scope-estimator.ts

Create lib/factory/scope-estimator.ts:

/**
 * Pre-flight scope estimator. ~1-2s probe that returns a URL count range,
 * sub-sitemap count, language list, and expected discovery duration.
 *
 * Used by the wizard UI to ask the user "this site is ~500K URLs, proceed?"
 * BEFORE the factory commits to a full discovery walk.
 *
 * Plan refs: 00-Plan.md §3a-bis, §19a.
 */

import { z } from 'zod';
import { XMLParser } from 'fast-xml-parser';
import { createLogger } from '../utils/logger';
import { discoverSitemaps } from './sitemap';

const log = createLogger('factory:scope-estimator');

const USER_AGENT = 'AlgoliaCentralFactory/1.0';
const PROBE_TIMEOUT_MS = 8000;
const AVG_URLS_PER_SUBSITEMAP_LOW = 1000;
const AVG_URLS_PER_SUBSITEMAP_HIGH = 50_000;
const FETCH_CONCURRENCY = 8;
const AVG_SUBSITEMAP_FETCH_MS = 1500;

export const ScopeEstimateSchema = z.object({
  rootUrl: z.string().url(),
  topSitemapUrls: z.array(z.string().url()),
  urlEstimateRange: z.tuple([z.number().int().nonnegative(), z.number().int().nonnegative()]),
  subSitemapCount: z.number().int().nonnegative(),
  languagesDetected: z.array(z.string()),
  expectedDurationSeconds: z.tuple([z.number().nonnegative(), z.number().nonnegative()]),
  transport: z.enum(['fetch', 'playwright', 'unknown']),
});
export type ScopeEstimate = z.infer<typeof ScopeEstimateSchema>;

/**
 * Probe a site's robots.txt + top sitemap and return a scope estimate.
 * Does NOT walk recursively. Always returns a value (even on failure,
 * with empty arrays + transport='unknown' or 'playwright').
 */
export async function estimateScope(rootUrl: string): Promise<ScopeEstimate> {
  let transport: ScopeEstimate['transport'] = 'fetch';
  let topSitemapUrls: string[] = [];
  try {
    topSitemapUrls = await discoverSitemaps(rootUrl);
  } catch (err) {
    log.warn({ event: 'discover_failed', rootUrl, err: String(err) });
    transport = 'unknown';
  }
  if (topSitemapUrls.length === 0) {
    return ScopeEstimateSchema.parse({
      rootUrl,
      topSitemapUrls: [],
      urlEstimateRange: [0, 0],
      subSitemapCount: 0,
      languagesDetected: [],
      expectedDurationSeconds: [0, 0],
      transport,
    });
  }

  const probe = await probeTopSitemap(topSitemapUrls[0]);
  if (probe === null) {
    return ScopeEstimateSchema.parse({
      rootUrl,
      topSitemapUrls,
      urlEstimateRange: [0, 0],
      subSitemapCount: 0,
      languagesDetected: [],
      expectedDurationSeconds: [0, 0],
      transport: 'playwright',
    });
  }

  const languagesDetected = detectLanguages([rootUrl, ...probe.sampleUrls]);
  const [min, max] = computeRange(probe);
  const [durMin, durMax] = computeDuration(probe.subSitemapCount);

  return ScopeEstimateSchema.parse({
    rootUrl,
    topSitemapUrls,
    urlEstimateRange: [min, max],
    subSitemapCount: probe.subSitemapCount,
    languagesDetected,
    expectedDurationSeconds: [durMin, durMax],
    transport,
  });
}

interface SitemapProbe {
  kind: 'urlset' | 'sitemapindex';
  urlCount: number;            // urls in this top file (urlset only)
  subSitemapCount: number;     // children (sitemapindex only)
  sampleUrls: string[];        // up to 5 sample loc/sub-sitemap values for language detection
}

async function probeTopSitemap(url: string): Promise<SitemapProbe | null> {
  const ctrl = new AbortController();
  const t = setTimeout(() => ctrl.abort(), PROBE_TIMEOUT_MS);
  try {
    const res = await fetch(url, { headers: { 'User-Agent': USER_AGENT }, signal: ctrl.signal });
    if (res.status !== 200) {
      log.warn({ event: 'top_sitemap_non200', url, status: res.status });
      return null;
    }
    const xml = await res.text();
    return parseProbe(xml);
  } catch (err) {
    log.warn({ event: 'top_sitemap_failed', url, err: String(err) });
    return null;
  } finally {
    clearTimeout(t);
  }
}

function parseProbe(xmlText: string): SitemapProbe {
  const parser = new XMLParser({
    ignoreAttributes: false,
    isArray: (tagName) => ['url', 'sitemap'].includes(tagName),
  });
  const parsed = parser.parse(xmlText) as Record<string, unknown>;
  if ('sitemapindex' in parsed) {
    const idx = parsed.sitemapindex as { sitemap?: Array<{ loc?: string }> };
    const children = (idx.sitemap ?? [])
      .map((s) => (typeof s.loc === 'string' ? s.loc : null))
      .filter((s): s is string => s !== null);
    return {
      kind: 'sitemapindex',
      urlCount: 0,
      subSitemapCount: children.length,
      sampleUrls: children.slice(0, 5),
    };
  }
  if ('urlset' in parsed) {
    const urlset = parsed.urlset as { url?: Array<{ loc?: string }> };
    const urls = (urlset.url ?? [])
      .map((u) => (typeof u.loc === 'string' ? u.loc : null))
      .filter((s): s is string => s !== null);
    return { kind: 'urlset', urlCount: urls.length, subSitemapCount: 0, sampleUrls: urls.slice(0, 5) };
  }
  return { kind: 'urlset', urlCount: 0, subSitemapCount: 0, sampleUrls: [] };
}

function computeRange(probe: SitemapProbe): [number, number] {
  if (probe.kind === 'urlset') return [probe.urlCount, probe.urlCount];
  const min = probe.subSitemapCount * AVG_URLS_PER_SUBSITEMAP_LOW;
  const max = probe.subSitemapCount * AVG_URLS_PER_SUBSITEMAP_HIGH;
  return [min, max];
}

function computeDuration(subSitemapCount: number): [number, number] {
  if (subSitemapCount === 0) return [5, 5];
  const seconds = (subSitemapCount / FETCH_CONCURRENCY) * (AVG_SUBSITEMAP_FETCH_MS / 1000);
  return [Math.max(5, Math.floor(seconds * 0.5)), Math.ceil(seconds * 1.5)];
}

const LANG_REGEX = /\/([a-z]{2})(?:[-_]([a-z]{2}))?\//gi;

function detectLanguages(urls: string[]): string[] {
  const seen = new Set<string>();
  for (const u of urls) {
    let m: RegExpExecArray | null;
    LANG_REGEX.lastIndex = 0;
    while ((m = LANG_REGEX.exec(u)) !== null) {
      const lang = m[1].toLowerCase();
      const region = m[2] ? m[2].toLowerCase() : '';
      const tag = region ? `${lang}-${region}` : lang;
      if (lang.length === 2) seen.add(tag);
    }
  }
  return Array.from(seen).sort();
}
  • [ ] Step 5.4: Run urlset test, expect PASS
npx vitest run lib/factory/__tests__/scope-estimator.test.ts -t "flat urlset"

Expected: 1 test passes.

  • [ ] Step 5.5: Add tests for sitemapindex, language detection, and WAF transport

Append to lib/factory/__tests__/scope-estimator.test.ts:

describe('estimateScope — sitemapindex with 531 children (Apple Autopush pattern)', () => {
  beforeEach(() => mockFetch.mockReset());

  it('returns urlEstimateRange = [531×low, 531×high] and subSitemapCount=531', async () => {
    const children = Array.from(
      { length: 531 },
      (_, i) => `<sitemap><loc>https://apple.com/sub-${i}.xml</loc></sitemap>`
    ).join('');
    const idx = `<?xml version="1.0"?><sitemapindex>${children}</sitemapindex>`;
    mockFetch
      .mockResolvedValueOnce(textResponse('User-agent: *\nSitemap: https://apple.com/sitemap.xml\n'))
      .mockResolvedValueOnce(xmlResponse(idx));
    const est = await estimateScope('https://apple.com');
    expect(est.subSitemapCount).toBe(531);
    expect(est.urlEstimateRange[0]).toBe(531_000); // 531 × 1000
    expect(est.urlEstimateRange[1]).toBe(531 * 50_000); // 531 × 50000
    expect(est.expectedDurationSeconds[1]).toBeGreaterThan(60); // multi-minute
  });
});

describe('estimateScope — language detection', () => {
  beforeEach(() => mockFetch.mockReset());

  it('detects en-us, de-de, fr from sample sub-sitemap URLs', async () => {
    const idx = `<?xml version="1.0"?>
<sitemapindex>
  <sitemap><loc>https://ms.com/dotnet/en-us/sitemap.xml</loc></sitemap>
  <sitemap><loc>https://ms.com/dotnet/de-de/sitemap.xml</loc></sitemap>
  <sitemap><loc>https://ms.com/dotnet/fr/sitemap.xml</loc></sitemap>
</sitemapindex>`;
    mockFetch
      .mockResolvedValueOnce(textResponse('User-agent: *\nSitemap: https://ms.com/sitemap.xml\n'))
      .mockResolvedValueOnce(xmlResponse(idx));
    const est = await estimateScope('https://ms.com');
    expect(est.languagesDetected).toContain('en-us');
    expect(est.languagesDetected).toContain('de-de');
    expect(est.languagesDetected).toContain('fr');
  });
});

describe('estimateScope — transport flag', () => {
  beforeEach(() => mockFetch.mockReset());

  it('sets transport=playwright when top sitemap returns 403 (WAF)', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('User-agent: *\nSitemap: https://wafsite.com/sitemap.xml\n'))
      .mockResolvedValueOnce(textResponse('Forbidden', 403));
    const est = await estimateScope('https://wafsite.com');
    expect(est.transport).toBe('playwright');
    expect(est.urlEstimateRange).toEqual([0, 0]);
  });

  it('returns empty estimate when no sitemap is discoverable', async () => {
    mockFetch
      .mockResolvedValueOnce(textResponse('not found', 404)) // robots
      .mockResolvedValueOnce(textResponse('not found', 404)) // /sitemap.xml
      .mockResolvedValueOnce(textResponse('not found', 404)) // /sitemap_index.xml
      .mockResolvedValueOnce(textResponse('not found', 404)); // /sitemap-index.xml
    const est = await estimateScope('https://nositemap.com');
    expect(est.topSitemapUrls).toEqual([]);
    expect(est.urlEstimateRange).toEqual([0, 0]);
    expect(est.subSitemapCount).toBe(0);
  });
});
  • [ ] Step 5.6: Run all scope-estimator tests
npx vitest run lib/factory/__tests__/scope-estimator.test.ts

Expected: all tests pass.

  • [ ] Step 5.7: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 5.8: Commit
git add lib/factory/scope-estimator.ts lib/factory/__tests__/scope-estimator.test.ts
git commit -m "feat(factory): pre-flight scope estimator (URL range, sub-sitemap count, languages, transport)"

Task 6: Cluster validation: full regression

Files: - (Read-only verification)

  • [ ] Step 6.1: Run all factory tests
npx vitest run lib/factory/__tests__/

Expected: all Spec 01 tests (~23 from Foundations) + all Spec 02 tests (sitemap ~12, path-grouper ~5, scope-estimator ~5) GREEN.

  • [ ] Step 6.2: Run full project test suite
npx vitest run

Expected: 407 baseline + 23 Spec 01 + ~22 Spec 02 = ~452 tests, all GREEN.

  • [ ] Step 6.3: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 6.4: Bundler still produces clean output
node scripts/bundle-api.mjs

Expected: completes without error; existing api/*.mjs regenerated; fast-xml-parser remains externalized (no bundle bloat from new code).

  • [ ] Step 6.5: Mark Spec 02 done in Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 2 (Sitemap Discovery) | ⏸ | walkSitemap, discoverSitemaps, path-grouper, scope-estimator |
+ | Cluster 2 (Sitemap Discovery) | ✅ | walkSitemap, discoverSitemaps, path-grouper, scope-estimator |

Acceptance criteria: Spec 02 done means:

  1. ✅ lib/factory/sitemap.ts exports discoverSitemaps(rootUrl): Promise<string[]> reading robots.txt + falling back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml.
  2. ✅ lib/factory/sitemap.ts exports walkSitemap(url): AsyncGenerator<string[]> yielding URL batches of up to 1000.
  3. ✅ Walker recurses sitemap-indexes at any depth (verified at depth 3: Microsoft pattern), with cycle prevention via Set<string> (verified with self-referencing index).
  4. ✅ Walker decompresses gzipped sitemaps (verified for .xml.gz URL suffix and Content-Encoding: gzip header).
  5. ✅ Walker extracts hreflang alternates as additional URLs, deduped against loc.
  6. ✅ Walker handles a 50,000-URL urlset (sitemap-protocol max) without OOM, yielding 50 batches of 1000.
  7. ✅ Walker fetches sub-sitemaps in parallel batches of 8 (concurrency cap).
  8. ✅ lib/factory/path-grouper.ts exports groupByPathPrefix(source): AsyncGenerator<PathGroup> consuming an async iterable of URL batches and emitting validated PathGroup objects.
  9. ✅ Path-grouper merges groups with <3 URLs into a single 'other' group, yielded last.
  10. ✅ Path-grouper caps sampleUrls at 10 per group.
  11. ✅ Path-grouper handles unparseable URLs by routing them to 'other' and logging.
  12. ✅ lib/factory/scope-estimator.ts exports estimateScope(rootUrl): Promise<ScopeEstimate> with zod-validated output (ScopeEstimateSchema).
  13. ✅ Scope estimator returns urlEstimateRange = [count, count] for flat urlsets and [count×1000, count×50000] for sitemap-indexes.
  14. ✅ Scope estimator detects ISO 639-1 language codes (e.g., en, en-us, de-de, fr) from URL patterns.
  15. ✅ Scope estimator sets transport: 'playwright' when the top sitemap returns 403 (WAF signal).
  16. ✅ All new tests pass (~22 tests across the three modules).
  17. ✅ Full project test suite still GREEN.
  18. ✅ tsc --noEmit clean.
  19. ✅ Per CodingSOPs: every module has a logger, every public function has a docstring, every fallible call has try/catch, every public boundary uses zod validation (PathGroup, ScopeEstimate).

Out of scope (handled by other specs)

  • Sample fetching (downloading individual page HTML for classification) → Spec 03 (sampler.ts).
  • Content classification of pathGroups (CMS fingerprint, JSON-LD, OpenGraph cascade per §3b/§4b) → Spec 04 (classifier.ts).
  • Persistence of discovered URLs into url_shard records → Spec 08 (/api/factory/discover endpoint uses Spec 01's SessionStore.appendUrls).
  • Resumability (Vercel timeout → resume from last shard) → Spec 08 (discover endpoint orchestrator), relies on the visited-set being persisted between runs; v1 accepts a single-process visited-set scoped to one walk.
  • Streaming SAX-style parse for files >5MB: fast-xml-parser does support a streaming chunk handler; v1 of this spec uses the in-memory XMLParser because Node.js fetch buffers the response anyway. The streaming-parse code path is reserved for Spec 03 / 07 if Vercel function memory pressure becomes an empirical issue. The 50K-URL urlset test (Step 3.7) verifies in-memory parse handles the spec ceiling without OOM on test infra.
  • Playwright stealth fallback for WAF-blocked sites → Spec 03 (the estimator only flags transport: 'playwright'; actual Playwright execution is in the sampler).
  • Pre-flight UI (showing the user the scope estimate and asking proceed/sub-tree/cancel) → Spec 11 (frontend wizard).
  • Real network integration tests against Algolia.com / Microsoft Learn / Apple → Spec 13 (smoke test cluster).
  • Honoring Crawl-delay at fetch time → Spec 03 (sampler) and Spec 07 (orchestrator). The estimator reads it but doesn't act on it; the walker fetches sitemaps with default 8-concurrent and accepts that some Crawl-delay: 10 Drupal sites will be slower than 8 RPS once samples are pulled.